...
| Table of Contents |
|---|
Introduction
Monitoring incoming data a model for its statistical drift performance is necessary to track whether assumptions made during model development are still valid in a production setting. For instance, a data scientist may assume that the values of a particular feature are normally distributed or the choice of encoding of a certain categorical variable may have been made with a certain multinomial distribution in mind. Tests the model is producing good output (inferences/scores) as compared to actual ground truth. These statistical metrics provide excellent insight into the predictive power of the model, including helping to identify degradation in the model’s ability to predict correctly. These statistical monitors should be run routinely against batches of live labeled data and compared against the distribution of the training data to ensure that these assumptions are still valid, and if the tests fail, then original metrics produced during training to ensure that the model is performing within specification. If the production statistical metrics deviate beyond a set threshold, then the appropriate alerts are raised for the data scientist or ModelOps engineer to investigate.
ModelOp Center provides a number of Drift statistical monitors out of the box, but also allows you to write your own drift monitorcustom metrics to monitor the statistical performance of the model. The subsequent sections describe how to add a drift statistical monitor (assuming an out-of-the-box monitor) and the detailed makeup of a drift statistical monitor for multiple types of models.
...
All monitor job results are persisted and can be viewed directly by clicking the specific “result” in the “Model Tests” section of the model snapshot page:
Statistical Monitor Details
...