Introduction
Modeling in the ModelOp Command Center takes place in three distinct phases. These phases may be done by different people with different roles in different locations. The phases include:
Preparation
Testing and deploying a model
Monitoring and updating the model
Preparation
The model
You can use any model factory to build your model. ModelOps is agnostic when it comes to the source of the model.
Metrics
Evaluating your model algorithm is an essential part of any model deployment. Data scientists choose evaluation metrics to determine the accuracy and statistical performance of the model. The choice of metric depends on the objective and the problem you are trying to solve. Some common metrics used in ModelOps sample models and examples include:
The F1 score
SHAP values
The ROC Curve
The AUC
A metrics function in the model can help automate and execute back tests of models. A metrics function can either be specified with a #modelop.metrics smartcomment before the function definition or selected within the UI after the model source code is registered.
There are two ways to create a metrics job manually:
In the Command Center, use the “Create a New Batch Job”. See Model Batch Jobs and Tests for details.
From the CLI. See Model Monitoring and Metrics for details.
The Model Life Cycle Process
The MLC Process automates and regulates the models within the ModelOp Center. Each MLC Process is defined in the Camunda Modeler as a BPM file. The MLC Process can to models of a variety of scope . They can apply to an individual model or a set of models based on the team or language or framework they employ. easily modified to comply with governmental or industry regulations will have different requirements for compliance reporting than a similar (or even the same) model deployed in an unregulated application
MLC Processes leverage the standard elements of a Camunda BPM asset:
Signal events - events that initiate the process, triggered when a model is changed or based on a timer
Tasks - can be a user task, operator leveraging MOC functionality, etc.
Gateway - decision logic to control the flow based on model meta-data
The full documentation set of Camunda is available at https://camunda.com/bpmn/reference/.
The Runtime Environment
ModelOps provides a runtime environment where you deploy your model for testing.
Register a model that has already been defined by a data scientist in a model factory
· Define and test the metrics, code executions, bias, and governance of the model
· Deploy the model to a ModelOp runtime environment, or set up their own runtime
· Promote the model to a production environment
· Monitor the deployed model
· Modify the model when