...

Monitoring incoming data for statistical drift is necessary to track whether assumptions made during model development are still valid in a production setting. For instance, a data scientist may assume that the values of a particular feature are normally distributed or the choice of encoding of a certain categorical variable may have been made with a certain multinomial distribution in mind. Tests should be run routinely against batches of live data and compared against the distribution of the training data to ensure that these assumptions are still valid, and if the tests fail, then appropriate alerts are raised for the data scientist or ModelOps engineer to investigate.

ModelOp Center provides a number of Drift monitors out of the box, but also allows you to write your own drift monitor. The subsequent sections describe how to add a drift monitor (assuming an out of the box monitor) and the detailed makeup of a drift monitor for multiple types of models.

Adding Drift Monitors

To add a drift monitor to your model, you will add an existing “associated” model to your model. Below are the steps to accomplish this. For tutorial purposes, these instructions use all OOTB and OTS content provided by ModelOp.

1. Define KPIs and thresholds for model 

   1. Edit the provided Data-drift.dmn file to reflect your desired tolerance for data drift

   2. Image Added

   3. Repeat for the provided Concept-drift.dmn file Performance-test.dmn

   4. Save the files locally to your machine.

2. Associate Monitor models to snapshot

   1. Navigate to the specific model snapshot

3. Using the associated models widget, create a data drift association

   1. Image Added

   2. Use the provided data and the DMN you made in step 2.

      1. Use the provided data and the DMN you made in step 2.

      2. Image Added

4. Before leaving the Model snapshot screen, copy the ID from the URL bar, you’ll need this for later

   1. Image Added

5. To test, run a monitoring jobs manually

   1. Make a REST call to MOC’s automation engine to trigger a data drift detection job on your model

      1. Obtain a valid auth token

      2. Make a call to the MLC API to initiate the monitor:

...

Drift Monitor Details

As the same data set may serve several models, you can write one drift detection model to associate to several models. This association is made during the Model Lifecycle process. The drift model can compare the training data of the associated models to a given batch of data. The following is a simple example:

...

If the training data is too large to fit in memory, you can save summary statistics about the training data and save those as, e.g., a pickle file and read those statistics in during the init function of the drift model. The metrics function can contain other statistical tests to compare those statistics to the statistics of the incoming batch.

Spark

...

Drift Model Details

A similar drift detection method may be used for PySpark models with HDFS assets by parsing the HDFS asset URLs from the parameters of the metrics function. The following is a simple example:

...