...

ModelOp Center provides a number of Drift monitors out of the box, but also allows you to write your own drift monitor. The subsequent sections describe how to add a drift monitor (assuming an out-of-the-box monitor) and the detailed makeup of a drift monitor for multiple types of models.

Adding

...

Statistical Monitors

As background on the terminology and concepts used in the below, please read the Monitoring Concepts section of the Model overview documentation.

...

1. As mentioned in the Monitoring Concepts article, ModelOp Center uses decision tables to define the thresholds within which the model should operate for the given monitor.

2. The first step is to define these thresholds. For this tutorial, we will leverage the example Performance-test.dmn decision table. This assumes that the out-of-the-box metrics function in the Consumer Credit Default example model is used, which outputs AUC, ROC, F1, amongst others. Specifically, this decision table ensures that the F1 and AUC from the Consumer Linear Demo model are within specification.

3. Image RemovedImage Added

4. Save the files locally to your machine.

...

1. Schedule. Monitors can be scheduled to run using your preferred enterprise scheduling capability (Control-M, Airflow, Autosys, etc.)

   1. While the details will depend on the specific scheduling software, at the highest level, the user simply needs to create a REST call to the ModelOp Center API. Here are the steps:

      1. Obtain the Model snapshot’s unique ID, which can be obtained from the Model snapshot screen. Simply copy the ID from the URL bar:

         1. Example:

            Image RemovedImage Added

      2. Within the scheduler, configure the REST call to ModelOp Center’s automation engine to trigger the monitor for your model:

         1. Obtain a valid auth token

         2. Make a call to the ModelOp Center API to initiate the monitor

         3. Example:

            Image Removed

         4. Code Block
            { "name": "com.modelop.mlc.definitions.Signals_MODEL_BACK_TEST", "variables": { "MODEL_ID": { "value": "FILL-IN-SNAPSHOT-GUID" } } }

2. Monitoring Execution: once the scheduler triggers the monitoring job, the relevant model life cycle will initiated the specific monitor, which likely includes:

   1. Preparing the monitoring job with all artifacts necessary to run the job

   2. Creating the monitoring job

   3. Parsing the results into viewable test results

   4. Comparing the results against the thresholds in the decision table

   5. Taking action, which could include creating a notification and/or opening up an incident in JIRA/ServiceNow/etc.

   6. 

...

1. All monitor job results are persisted and can be viewed directly by clicking the specific “result” in the “Model Tests” section of the model snapshot page:

   1. Image RemovedImage Added

Statistical Monitor Details

...