This article describes how ModelOp Center’s MLC’s can be used to orchestrate and automate the annual review process
Table of Contents
Overview
The ModelOp Life Cycle Manager (MLC Manager) orchestrates and automates the the various steps involved in conducting an annual review of a model for Model Risk and Governance processes. This involves …
Pre-Requisites
The following are the pre-requisites for leveraging this Annual Review Process MLC:
Jira project must be setup with Done and Rejected status in workflow
Attachment on the Jira project must be enabled.
Standard Risk Tests model must be present with a valid snapshot
Business model must be deployed in production
Business model must have baseline data and comparator data assets.
Process Overview
Process is triggered for the annual review of a snapshot of a business model(s) when a business model is deployed to production and the expiration date of the snapshot is within the 30 days of the current date. MLC runs Standard Risk Tests on the snapshot and generate the test results. It puts all these results into a model review document to be sent to a model reviewer for validation. If the review is validated, the flow updates the snapshot custom metadata with the annual review date and time and the validator of the ticket. Finally, it updates the expiration date of the snapshot.
Scheduler - process is triggered by a scheduler (com.modelop.mlc.definitions.Signals_deployable_model_annual_review) and get deployed model(s) in production.
Model Initial Checkpoint - the process continues if the model’s snapshot is within the 30 days of expiration date and is not already validated or pending for validation.
Run Standard Risk Tests - the process validates if the model has baseline data and comparator data assets in order to run the Standard Risk Tests on that model. If the assets are present in the model the MLC runs the Standard Risk Tests and notifies if any of the monitor has failed to run. If the test job fails, a Jira ticket is created with the failed job error message. A test can be re-run if the Jira ticket is moved to “Done”
Approval Based on Test Results - Jira ticket is created with the model test results for validation. If ticket is moved to “Done” the flow continues and adds the generated document to the model snapshot.
Annual Validation - Jira ticket is created with the test result document for the senior analyst for the final sign-off.
Update Snapshot Metadata- If ticket is moved to “Done” snapshot’s custom metadata is updated with the sign-off date and validator’s info and updates snapshot’s expiration date. If the ticket is moved to “Rejected” the custom metadata is updated with Annual Model Review: Rejected.
Error handling -
An error is logged if the flow is unable to read the model(s) in production
An error notification is created for the following:
i. if Standard Risk Test snapshot is not found.
ii. if the first Jira ticket with document has been moved to rejected.
iii. if any other exception occurs in the flow.
Related Articles
Next Article: Operationalizing Models: Overview >