Introduction
The three Critical Requirements for a ModelOps Tech Solution, relative to models in production, are:
Ability to deploy, monitor and govern any model across all enterprise AI, regardless of their development tool or runtime environment
Ability to abstract the complexity of the enterprise AI stack, driving agility and scale in the enterprise’s operationalization of models in business
Ability to automate the model life cycle in the enterprise with repeatability, resilience and scale
ModelOps answer to these requirements is the ModelOp Command Center, where day-to-day operations, model monitoring, alert and notification response, and model retraining activities happen.
Modeling in the ModelOp Command Center takes place in three distinct phases. These phases may be done by different people with different roles in different locations. The phases include:
Preparation
Test and deploy a model
Monitor and update a model
Preparation
The Model: Metrics
The ModelOps Command Center design offers you the freedom to use the most effective model development tool for each application and use case.
Evaluating your model algorithm is an essential part of any model deployment. Data scientists choose evaluation metrics to determine the accuracy and statistical performance of the model. The choice of metric depends on the objective and the problem you are trying to solve. Some common metrics used in ModelOps sample models and examples include:
The F1 score
SHAP values
The ROC Curve
The AUC
A metrics function in the model can help automate and execute back tests of models. A metrics function can either be specified with a #modelop.metrics smartcomment before the function definition or selected within the UI after the model source code is registered.
There are two ways to create a metrics job manually:
In the Command Center, use the “Create a New Batch Job”. See Model Batch Jobs and Tests for details.
From the CLI. See Model Monitoring and Metrics for details.
The Model Life Cycle Process (MLC Process)
The MLC Process automates and regulates the models within the ModelOp Center. Each MLC Process is defined in the Camunda Modeler as a BPM file. The MLC Process can to models of a variety of scope . They can apply to an individual model or a set of models based on the team or language or framework they employ. easily modified to comply with governmental or industry regulations will have different requirements for compliance reporting than a similar (or even the same) model deployed in an unregulated application
MLC Processes leverage the standard elements of a Camunda BPM asset:
Signal events - events that initiate the process, triggered when a model is changed or based on a timer
Tasks - can be a user task, operator leveraging MOC functionality, etc.
Gateway - decision logic to control the flow based on model meta-data
The full documentation set of Camunda is available at https://camunda.com/bpmn/reference/.
Prepare a Runtime Environment
You can use the runtime environment provided by ModelOp, or ModelOp can integrate with runtime environment of your choice (e.g. Spark).
When you prepare a runtime, you configure it to be discoverable by the MLC Process operator, to encode it’s messages in the proper format, and to let the MLC Process know which model it supposed to go there.
You prepare the runtime by accessing the engine in the Runtimes section of the Command Center and providing the following details:
A name for the engine. You will put this name in the MLC Process service task
Endpoint type: REST or Kafka
Encoding: Avro Binary, CSV, jason, Json Binary Message Pack, or UTF-8 Binary
‘Tag’ the engine with the name of the model you will deploy in that engine.
For details about preparing a Runtime environment, see https://modelop.atlassian.net/wiki/spaces/VDP/pages/edit-v2/909082752
Register the Model with the Command Center
The Command Center has to know about your model before you can manage, test, and deploy it. This is done by registering the model.
For details about how to register your model using the Command Center, the command line interface, or a Jupyter plug in, see Register a Model .
Deploy and Test a Model
In general, the first place you deploy a model is in a development runtime engine so you can test it and modify it accordingly. For details on how to deploy a model into a runtime, see https://modelop.atlassian.net/wiki/spaces/VDP/pages/edit-v2/909082752.
Most model tests are operationalized in an MLC Process using ModelOp’s Smart Tags to automate execution of training, testing, and/or scoring throughout the model’s life cycle. For details about the Smart Tags, see Register a Model.
You can also be run Batch Jobs manually in the Command Center or from the command line. For details about Batch Jobs, see Model Batch Jobs and Tests.
You can execute a Champion/Challenger test to compare the results of two models. For details about the Champion/Challenger feature, see Champion/Challenger Model Comparison.
When you are satisfied with your model, you can build an MLC process to automate the promotion to a production environment.
Introduction
The ModelOp Command Center displays a single screen with access to all of the tools to monitor and manage your models. This section describes the components of the Command Center which appear The components on the Dashboard . The components include the three panes in the middle of the screen, the notifications pane below, and the four icons in the column menu on the left.
Models Pane
On the Models pane, you can see the current status of the models in your system. The Models pane shows:
The number of models that are currently registered. This is the number of models being managed.
The number of models that are currently deployed running in an MLC EngineModelOp Runtime
The number of model tests that have failed
The number of those failures that are errors in the model as opposed to system or MLC errors
You can click on a category that is populated to filter the notifications to see details related to that category. For instance, if Test Failures shows the number 2, you can click on the number to see details about the filter the notifications below to just those associated with test failures.
Processes Pane
The Processes pane displays how many MLC processes are currently deployed and any incidents or errors associated with the MLC processes.
The MLC Processes
Each MLC Process has built in monitoring tasks that report events and errors to the Command Center. This pane displays the following:
The number of defined MLC Processes
The number of reported incidents
The number of active MLC Processes
The number of MLC Process errors
Each model can have only one MLC Process deployed with it.
Engines
Runtimes Pane
The Engines Runtimes pane displays information about the runtime engines in runtimes available to the system. These are the environments where models are deployed.
This pane displays how many engines runtimes are available, active, idle, and if there are any engine errors.
Tasks and Alerts Iconruntime errors.
Notifications Pane
The notifications pane displays a real-time list of all notifications across the ModelOp Center system. This includes informational notifications as well as alerts that require immediate attention. Per the prior sections, the notifications pane can be filtered by clicking one of the items in the Models, Processes, or Runtimes pane.
Main Menu Bar
The Main Menu Bar is meant to provide quick access to all of the Models, MLC Processes, Runtimes, and Tasks across the enterprise ModelOps program. See below for details of each section.
Tasks and Alerts
The Tasks and Alerts screen has two tabs, My Tasks and Approvals.
My Tasks
Tasks are defined as User Tasks in the MLC Process. They can be are typically configured to assign tasks to a specific person, and to one of the categories in this section of the ModelOp Command Centergroup. A person within this group can then decide to take ownership of that task and execute it to completion.
Tasks are things a user has to do - such as approve a model or acknowledge a failed test - so that the MLC Process can move forward.
The My Tasks page filters tasks in two categories:
All Open Tasks
Tasks in Progress
Approvals
My active Tasks - # outstanding
ds_team - awaiting assignment - # outstanding
python_approvers - awaiting assignment - # outstanding
r_approvers - awaiting assignment - # outstanding
other_approvers - awaiting assignment - 0 outstanding
Runtimes
IconThe Runtimes screen has the following tabs:
Jobs | Displays how many jobs are running, how many have failed, how many were successful and the total. |
Runtime Dashboard | Displays high level status for all currently deployed runtime enginesruntimes, each in it’s own pane. Click on the pane for access to details about that Runtime engine. Tabs under the Runtime Dashboard pane include:
|
Engine Runtime Stats | Lists all available enginesruntimes. Click through an engine a runtime for details about that engineruntime. |
Models
IconTabs under the Models icon are Model Details, Model Tests and Champion/Challenger tabs.
Model Details
The following table describes the details available under the Model Details tab. These fields are available only after you select a model from the list.
Model Info | Displays the name of the model, the description, and any tags applied to the model. |
Model Source | Source code that defines the model. Beginning statement initializes the model, loads any attachments into the engine with the model. If you retrain the model and update the attachment, the model is automatically updated. |
Attachments | Attachments are typically training artifacts that get loaded into the model, but can be any other file required to execute the model. |
Input Schemas | Defines the parameters of what is passed to the model. |
Output Schemas | Defines the parameters and format of what is passed from the model. |
Model Functions | Access to view the some details about the functions of a modelDefines the entry points into the model for initialization, training, metrics testing, and scoring. |
Model Platform | Lists the dependencies used by the model. |
Model Tests
Lists the models in the system. Click a model to access the results of tests on that model.
Click on a test to see visualizations of the tests. The visualizations available depend on the tests built into the MCL a MLC Process.
Champion/Challenger
Champion/Challenger is an industry standard technique to compare the performance of competing model strategies, or the performance of a recently updated model against a previous version of the same model.
Lists the models in the system. Click a model to access the results of tests on that model . Choose and choose two tests two compare them against each other.
See the Champion/Challenger article for more details.
Model Life Cycle
The Model Life Cycle screen has two tabs, MLC Overview and MLC Processes.
The MLC Overview tab displays information about all the activities being handled by the MLC Manager.
Active MLC Processes
Total Incidents
Total Tasks
Runtime information
Oldest Tasks information
The MLC Processes tab displays the diagrams of the MLC Processes that are currently deployed into the MLC Manager.
