Introduction

Modeling in the ModelOp Command Center takes place in three distinct phases. These phases may be done by different people with different roles in different locations. The phases include:

1. Preparation

2. Testing and deploying a model

3. Monitoring and updating the model

Preparation

The model

You can use any model factory to build your model. ModelOps is agnostic when it comes to the source of the model.

Metrics

Data Evaluating your model algorithm is an essential part of any model deployment. Data scientists choose evaluation metrics to determine the accuracy and statistical performance of the model. The choice of metric depends on the type of model and the implementation plan for the modelobjective and the problem you are trying to solve. Some common metrics used in ModelOps sample models and examples include:

• The F1 score 

• SHAP values

• The ROC Curve

• The AUC

A metrics function in the model can help automate and execute back tests of models. A metrics function can either be specified with a #modelop.metrics smartcomment before the function definition or selected within the UI after the model source code is registered.

There are two ways to create a metrics job manually:

• In the Command Center, use the “Create a New Batch Job”. See Model Batch Jobs and Tests for details.

• From the CLI. See Model Monitoring and Metrics for details.

moc job create testjob <model-name> <input-file-name> <output-file-name> optional-flags.

This command yields a UUID for the job. The raw json results of the job can be found with the command

moc job result <uuid>.

For more information, see the Model Monitoring and Metrics  page.

The Model Life Cycle Process

The Model Life Cycle Process

The MLC Process automates and regulates the models within the ModelOp Center. Each MLC Process is defined in the Camunda Modeler as a BPM file. The MLC Process can to models of a variety of scope . They can apply to an individual model or a set of models based on the team or language or framework they employ. easily modified to comply with governmental or industry regulations will have different requirements for compliance reporting than a similar (or even the same) model deployed in an unregulated application

MLC Processes leverage the standard elements of a Camunda BPM asset:

• Signal events - events that initiate the process, triggered when a model is changed or based on a timer

• Tasks - can be a user task, operator leveraging MOC functionality, etc.

• Gateway - decision logic to control the flow based on model meta-data

The full documentation set of Camunda is available at https://camunda.com/bpmn/reference/.

The Runtime Environment

ModelOps provides a runtime environment where you deploy your model for testing.  

·         Register a model that has already been defined by a data scientist in a model factory

·         Define and test the metrics, code executions, bias, and governance of the model

·         Deploy the model to a ModelOp runtime environment, or set up their own runtime

·         Promote the model to a production environment

·         Monitor the deployed model

·         Modify the model when