This section describes the Champion/Challenger feature in ModelOp Center, and how to use it to help determine the best model method or approach to promote to production.

Table of Contents

Introduction

Data Science requires experimentation using a variety of methods, approaches, frameworks, and potentially even coding languages. While most data scientists will conduct their experimentation in their model development environment, data scientists--and the managerial and governance reviewers--often want to review that a candidate model for production performs better than any currently running version of the model. The Champion/Challenger feature in ModelOp Center presents a side-by side comparison of the performance of different models--or versions of the same model--to help users analyze which model or model version is best suited for production usage.

The following phases lay the groundwork for doing a Champion/Challenger comparison.

1. Define the evaluation metrics for the model. See Model Monitoring: Overview.

2. Automate evaluation tests - this is done either by manually running metrics jobs (see Running a Metrics Job Manually) or through an automated MLC Process. You can build an MLC Process to automatically execute metrics tests for a particular version of a model. See Model Lifecycle Management Overview for more information.

3. Conduct a side-by-side comparison of the test results in the Champion/Challenger page of the Command Center (see next section).

Champion/Challenger Comparison

For the previously generated metrics results (see steps 1 & 2 above), there are multiple ways to conduct a Champion/Challenger comparison:

Compare two distinct Model Implementations

To compare two distinct Model Implementations:

1. Select Inventory from the main menu

2. Toggle to “Implementations”

3. Select the “Enable champion challenger” toggle button. This will expose the “Champion/Challenger” column, allowing the user compare two or models models using the “Champion/Challenger” check boxes.

Open

4. Select the models and click the “Compare Models” button

5. The Champion/Challenger wizard will open to guide the user through the next steps

Compare two versions of the same Model Implementation

To compare two version (snapshots) of the same Model Implementation:

1. Open the specific Model Implementation page of interest

2. In the “Snapshots” section, click the “Enable champion challenger” button

3. Select the models and click the “Compare Models” button

4. The Champion/Challenger wizard will open to guide the user through the next steps

Open

Champion/Challenger Wizard

1. The Champion/Challenger wizard will open to guide the user through the next steps

2. Choose “Select Additional Models for Comparison” to add other models in the comparison, or select NEXT.

3. Choose the specific Snapshots (aka Versions) of the models that you want to compare by selecting the “+” sign under the Add column. Click “Next Step” when done.

4. On the “Test Results” screen, select the specific test results for which you want to compare and hit the “Compare Results” button.

5. The Champion/Challenger test results comparison screen will open to allow for a side-by-side comparison for those selected models.

Next Article: Model Monitoring: Overview >