...
ModelOp Center v2.4 introduces a new comprehensive monitoring package designed to help enterprises implement monitoring across ALL models, regardless of the language, framework, model runtime (e.g. SageMaker, Spark, Docker, etc.), or environment (on-prem, cloud, or hybrid). Specific features include:
Monitoring Automation
OOTB Monitors
Custom Monitors
Alerting with Custom Thresholds
OOTB Remediation Paths
Integrated scheduling: ability to automatically monitor ANY model, using either custom monitors or the out-of-the-box ModelOp monitors, and have the results analyzed against thresholds, as well as persisted for auditability. Furthermore, ModelOp orchestrates any required remediation paths to ensure that proper resolution is obtained.
Out-of-the-Box Monitors: provides a comprehensive set of monitors out-of-the-box, including ability to monitor statistical performance (classification, regression, etc), data drift (both numerical and categoricals), model concept drift, ethical fairness, population and characteristic stability, and data integrity / volumetrics. These out-of-the-box monitors work for any model, across batch and online models.
Custom Monitors: while ModelOp’s out-of-the-box monitors are comprehensive, users are also able to add any custom monitors across one or more models. These custom monitors can leverage the same automation to conduct comparisons against thresholds and orchestrated remediation actions.
Alerting with Custom Thresholds: ability to define thresholds and even business rules around monitors, including the ability to define warning vs. error thresholds across multiple monitoring metrics and incorporated model metadata.
Automated Remediation Paths: customizable remediation paths via model life cycles (MLC’s) that orchestrate the specific actions that should occur based on the results of monitoring, which may involve automatically opening a ServiceNow ticket for the operations team to address the ticket, or opening a Jira for a data scientist to investigate a statistical issue, or in worst cases, rollback a model to a prior designated version. The exact paths taken in the MLC are persisted in the model inventory for long-term auditability.
Integrated scheduling: provides an out-of-the-box scheduler to initiate monitoring jobs. ModelOp Center also allows for the use of external schedulers or other triggers (e.g. API calls) to initiate monitoring execution.