Introduction

The three Critical Requirements for a ModelOps Tech Solution, relative to models in production, are:

• Ability to deploy, monitor and govern any model across all enterprise AI, regardless of their development tool or runtime environment

• Ability to abstract the complexity of the enterprise AI stack, driving agility and scale in the enterprise’s operationalization of models in business

• Ability to automate the model life cycle in the enterprise with repeatability, resilience and scale

ModelOps answer to these requirements is the ModelOp Command Center, where day-to-day operations, model monitoring, alert and notification response, and model retraining activities happen.

Modeling in the ModelOp Command Center takes place in three distinct phases. These phases may be done by different people with different roles in different locations. The phases include:

1. Preparation

2. Test and deploy a model

3. Monitor and update a model

Preparation

The Model: Metrics

The ModelOps Command Center design offers you the freedom to use the most effective model development tool for each application and use case.

Evaluating your model algorithm is an essential part of any model deployment. Data scientists choose evaluation metrics to determine the accuracy and statistical performance of the model. The choice of metric depends on the objective and the problem you are trying to solve. Some common metrics used in ModelOps sample models and examples include:

• The F1 score 

• SHAP values

• The ROC Curve

• The AUC

A metrics function in the model can help automate and execute back tests of models. A metrics function can either be specified with a #modelop.metrics smartcomment before the function definition or selected within the UI after the model source code is registered.

There are two ways to create a metrics job manually:

• In the Command Center, use the “Create a New Batch Job”. See Model Batch Jobs and Tests for details.

• From the CLI. See Model Monitoring and Metrics for details.

The Model Life Cycle Process (MLC Process)

The MLC Process automates and regulates the models within the ModelOp Center. Each MLC Process is defined in the Camunda Modeler as a BPM file. The MLC Process can to models of a variety of scope . They can apply to an individual model or a set of models based on the team or language or framework they employ. easily modified to comply with governmental or industry regulations will have different requirements for compliance reporting than a similar (or even the same) model deployed in an unregulated application

MLC Processes leverage the standard elements of a Camunda BPM asset:

• Signal events - events that initiate the process, triggered when a model is changed or based on a timer

• Tasks - can be a user task, operator leveraging MOC functionality, etc.

• Gateway - decision logic to control the flow based on model meta-data

The full documentation set of Camunda is available at https://camunda.com/bpmn/reference/.

Prepare a Runtime Environment

You can use the runtime environment provided by ModelOp, or ModelOp can integrate with runtime environment of your choice (e.g. Spark).

When you prepare a runtime, you configure it to be discoverable by the MLC Process operator, to encode it’s messages in the proper format, and to let the MLC Process know which model it supposed to go there.

You prepare the runtime by accessing the engine in the Runtimes section of the Command Center and providing the following details:

• A name for the engine. You will put this name in the MLC Process service task

• Endpoint type: REST or Kafka

• Encoding: Avro Binary, CSV, jason, Json Binary Message Pack, or UTF-8 Binary

• ‘Tag’ the engine with the name of the model you will deploy in that engine.

For details about preparing a Runtime environment, see https://modelop.atlassian.net/wiki/spaces/VDP/pages/edit-v2/909082752

Register the Model with the Command Center

The Command Center has to know about your model before you can manage, test, and deploy it. This is done by registering the model.

For details about how to register your model using the Command Center, the command line interface, or a Jupyter plug in, see Register a Model .

Deploy and Test a Model

In general, the first place you deploy a model is in a development runtime engine so you can test it and modify it accordingly. For details on how to deploy a model into a runtime, see https://modelop.atlassian.net/wiki/spaces/VDP/pages/edit-v2/909082752.

Most model tests are operationalized in an MLC Process using ModelOp’s Smart Tags to automate execution of training, testing, and/or scoring throughout the model’s life cycle. For details about the Smart Tags, see Register a Model.

You can also be run Batch Jobs manually in the Command Center or from the command line. For details about Batch Jobs, see Model Batch Jobs and Tests.

You can execute a Champion/Challenger test to compare the results of two models. For details about the Champion/Challenger feature, see Champion/Challenger Model Comparison.

When you are satisfied with your model, you can build an MLC process to automate the promotion to a production environment.

Monitor and Update a Model

The ModelOp Command Center displays a single screen with access to all of the tools to monitor and manage your models. This section describes the components of the Command Center which appear on the Dashboard. The components include the three panes in the middle of the screen, and the four icons in the column on the left.

Models Pane

On the Models pane, you can see the current status of the models in your system. The Models pane shows:

• The number of models that are currently registered. This is the number of models being managed.  

• The number of models that are currently deployed in an MLC Engine

• The number of model tests that have failed

• The number of those failures that are errors in the model as opposed to system or MLC errors

You can click on a category that is populated to see details. For instance, if Test Failures shows the number 2, you can click on the number to see details about the failures.

Processes Pane

The Processes pane displays how many MLC processes are currently deployed.

The MLC Processes

Each MLC Process has built in monitoring tasks that report events and errors to the Command Center. This pane displays the following:

• The number of defined MLC Processes

• The number of reported incidents

• The number of active MLC Processes

• The number of MLC Process errors

Each model can have only one MLC Process deployed with it.

Engines Pane

The Engines pane displays information about the runtime engines in the system. These are the environments where models are deployed.

This pane displays how many engines are available, active, idle, and if there are any engine errors.

Tasks and Alerts Icon

The Tasks and Alerts screen has two tabs, My Tasks and Approvals.

My Tasks

Tasks are defined as User Tasks in the MLC Process. They can be configured to assign tasks to a specific person, and to one of the categories in this section of the ModelOp Command Center.

Tasks are things a user has to do - such as approve a model or acknowledge a failed test - so that the MLC Process can move forward.

The My Tasks page filters tasks in two categories:

• All Open Tasks

• Tasks in Progress

Approvals

• My active Tasks - # outstanding

• ds_team - awaiting assignment - # outstanding

• python_approvers - awaiting assignment - # outstanding

• r_approvers - awaiting assignment - # outstanding

• other_approvers - awaiting assignment - 0 outstanding

Runtimes Icon

The Runtimes screen has the following tabs:

Models Icon

Tabs under the Models icon are Model Details, Model Tests and Champion/Challenger tabs.

Model Details

The following table describes the details available under the Model Details tab. These fields are available only after you select a model from the list.

Model Tests

Lists the models in the system. Click a model to access the results of tests on that model.

Click on a test to see visualizations of the tests. The visualizations available depend on the tests built into the MCL Process.

Champion/Challenger

Champion/Challenger is an industry standard technique to compare the performance of competing model strategies, or the performance of a recently updated model against a previous version of the same model.

Lists the models in the system. Click a model to access the results of tests on that model . Choose and choose two tests two compare them against each other.