V2.4 Release Notes

Owned by Dave Trier
Nov 10, 2022
3 min read

What’s New:

 

AWS SageMaker Integration

ModelOp Center provides seamless integration with AWS SageMaker to allow data scientists to design, build, test, and run models in SageMaker, while providing enterprise-wide model governance and life cycle management. Specific features include:

  • Model-Centric Inventory:

    • Inventory: provides a detailed, model-centric view of all elements, artifacts, jobs, deployments, etc. of a model developed in SageMaker, inclusive of all versions of a given model over time.

    • Register: allows for importing a model developed in SageMaker into ModelOp Center, including the model’s related Training Jobs, Batch Transform Jobs, Endpoint configurations, Endpoints. Provides automated syncing with the related SageMaker artifacts.

  • Model Risk / Governance:

    • Productionization: orchestrates the entire life cycle of a SageMaker model, inclusive of all automated testing, SecOps reviews, compliance/governance reviews, as well as deployment of a SageMaker model to a SageMaker endpoint.

    • Model Risk: provides a detailed audit trail of every step in a SageMaker’s model lifecycle—from registration, to approvals, to tests, to deployment into Production and on-going monitoring.

  • Model Execution:

    • Enables execution of a Sagemaker transform job from ModelOp Center manually and via a Model Life Cycle (MLC).

    • Provides visibility into the operational status of all SageMaker endpoint deployments--and alert upon any issues--as part of an enterprise-wide operational control across all model runtimes.

  • Monitoring/Metrics:

    • Parses and persists metrics from a SageMaker training job.

    • Allows execution of back testing, drift and concept drift monitoring, volumetrics, ethical fairness testing, and stability testing for SageMaker deployed models automatically.

    • Enables threshold-based notifications and alerting, with integrations into ServiceNow, JIRA, and other operational systems.

Monitoring Updates

ModelOp Center v2.4 introduces a new comprehensive monitoring package designed to help enterprises implement monitoring across ALL models, regardless of the language, framework, model runtime (e.g. SageMaker, Spark, Docker, etc.), or environment (on-prem, cloud, or hybrid). Specific features include:

  • Monitoring Automation: ability to automatically monitor ANY model, using either custom monitors or the out-of-the-box ModelOp monitors, and have the results analyzed against thresholds, as well as persisted for auditability. Furthermore, ModelOp orchestrates any required remediation paths to ensure that proper resolution is obtained.

  • Out-of-the-Box Monitors: provides a comprehensive set of monitors out-of-the-box, including ability to monitor statistical performance (classification, regression, etc), data drift (both numerical and categoricals), model concept drift, ethical fairness, population and characteristic stability, and data integrity / volumetrics. These out-of-the-box monitors work for any model, across batch and online models.

  • Custom Monitors: while ModelOp’s out-of-the-box monitors are comprehensive, users are also able to add any custom monitors across one or more models. These custom monitors can leverage the same automation to conduct comparisons against thresholds and orchestrated remediation actions.

  • Alerting with Custom Thresholds: ability to define thresholds and even business rules around monitors, including the ability to define warning vs. error thresholds across multiple monitoring metrics and incorporated model metadata.

  • Automated Remediation Paths: customizable remediation paths via model life cycles (MLC’s) that orchestrate the specific actions that should occur based on the results of monitoring, which may involve automatically opening a ServiceNow ticket for the operations team to address the ticket, or opening a Jira for a data scientist to investigate a statistical issue, or in worst cases, rollback a model to a prior designated version. The exact paths taken in the MLC are persisted in the model inventory for long-term auditability.

  • Integrated scheduling: provides an out-of-the-box scheduler to initiate monitoring jobs. ModelOp Center also allows for the use of external schedulers or other triggers (e.g. API calls) to initiate monitoring execution.