Example Workflows - Default Model Implementations

Owned by Dave Trier
Last updated: Aug 07, 2024
3 min read

This article describes how ModelOp Center’s MLC’s can be used to orchestrate and automate the Governance - Default Implementations.

Table of Contents

Overview

The ModelOp Life Cycle Manager (MLC Manager) orchestrates and automates the the various steps involved in conducting Model Completeness validation, Model Risk evaluation and Governance processes.

Pre-Requisites

The following are the prerequisites for leveraging this Governance - Default Implementations MLC:

  • The Jira project must be configured with Done and Rejected statuses in the workflow

  • Attachments must be enabled for the Jira project

  • Standard Risk Tests model must be present with a valid snapshot

  • Business model must be deployed in production

  • Business model must have appropriate assets with the BASELINE_DATA and COMPARATOR_DATA asset roles

  • Deployment stage one: Engine environment name must be "validation" + model tag (if any)

  • Deployment stage two: Engine environment name must be "prod" + model tag (if any)

Process Overview

The process is triggered for the Default Business model implementation. The Governance - Default Implementations MLC runs Standard Risk Tests on the snapshot and generates test results. These results are used to generate a model review document for validation by a model reviewer. When the document review is approved the MLC updates the snapshot is Marked as Deployed in Production.

 

 

  1. Scheduler - the process is triggered by Use Case Registeration MLC based on ModelType “Python, R, C++ and others”.

  2. Model PIA Approval - the process creates PIA Approval Request Jira ticket and continues when the ticket is moved to Done status.

  3. Create Model Snapshot - A snapshot of the business model is created

  4. Model Completeness Check - Process validates the model has all the required assets i-e Readme asset, Input schema, Output schema, Metrics test asset, Baseline data asset, Comparator data asset, Training data asset and Test result comparator asset (dmn)

  5. Run Standard Risk Tests - If the assets are present in the model the MLC runs the Standard Risk Tests and creates a notification if any of the monitors in Standard Risk Tests fail to run. If the test job fails, a Jira ticket is created with the failed job error message. A test can be re-run if the Jira ticket is moved to Done.

  6. Approval Based on Test Results - A Jira ticket is created with document generated from the model test results for validation. If ticket is moved to Done the flow continues and adds the generated document to the model snapshot.

  7. Validation Environment Approval - Jira ticket is created for validation deployment approval.

  8. Marked as Deployed- If ticket is moved to Done the snapshot will Mark as Deployed in Validation

  9. Production Approval - Jira ticket is created for production deployment approval

  10. Marked as Deployed - If ticket is moved to Done the snapshot will Mark as Deployed in Production

  11. Error handling -

    1. An error is logged if the flow is unable to get the business model

    2. An error notification is created for the following:
      i. Standard Risk Test snapshot is not found.
      ii. the first Jira ticket with document has been moved to Rejected.
      iii. any other exception occurs in the flow.

Next Article: Example Workflows - Sagemaker Model Implementations >