...

The following are the prerequisites for leveraging this Annual Review Process Sagemaker Model Implementation MLC:

• The Jira project must be configured with Done and Rejected statuses in the workflow

• Attachments must be enabled for the Jira project

• Standard Risk Tests model must be present with a valid snapshot

• Business model must be deployed in production

• Business model must have appropriate assets with the BASELINE_DATA and COMPARATOR_DATA asset roles

• The Sagemaker model should already have existing "Endpoint Configuration" that will be used in the Snapshot for deployment

Process Overview

The process is triggered for the annual review of a snapshot of a business model(s) when a business model is deployed to production and the expiration date of the snapshot is within the 30 days of the current date. The annual review Sagemaker Business model implementation. The Sagemaker Model Implementations MLC runs Standard Risk Tests on the snapshot and generates test results. These results are used to generate a model review document for validation by a model reviewer. When the document review is approved the MLC updates the snapshot’s custom metadata with the annual review date and time and the validator of the ticket. Finally, the MLC updates the expiration date of the snapshot.

...

snapshot is Marked as Deployed in Production.

Image Added

1. Scheduler - the process is triggered by a signal (com.modelop.mlc.definitions.Signals_deployable_model_annual_review) and retrieves a deployed model(s) in production.Model Initial Checkpoint - the process continues if the model’s snapshot is within the 30 days of expiration date and is not already validated or pending for validation.Use Case Registeration MLC based on ModelType “SAGEMAKER”.

2. Model PIA Approval - the process creates PIA Approval Request Jira ticket and continues when the ticket is moved to Done status.

3. Create Model Snapshot - A snapshot of the business model is created

4. Run Standard Risk Tests - the process validates if the model assets with the BASELINE_DATA and COMPARATOR_DATA assets roles in preparation to run Standard Risk Tests on the model. If the assets are present in the model the MLC runs the Standard Risk Tests and creates a notification if any of the monitors in Standard Risk Tests fail to run. If the test job fails, a Jira ticket is created with the failed job error message. A test can be re-run if the Jira ticket is moved to Done.

5. Approval Based on Test Results - A Jira ticket is created with document generated from the model test results for validation. If ticket is moved to Done the flow continues and adds the generated document to the model snapshot.

6. Annual Validation Production Approval - Jira ticket is created with the test result document for the senior analyst for the final sign-off. Update Snapshot Metadatafor production deployment approval.

7. Marked as Deployed- If ticket is moved to Done the snapshot 's custom metadata is updated with the sign-off date and the validator's info. The snapshot's expiration date is also updated. If the ticket is moved to Rejected the custom metadata is updated with Annual Model Review: Rejected.will Mark as Deployed in Production

8. Error handling -

   1. An error is logged if the flow is unable to read get the business model(s) in production

   2. An error notification is created for the following:
      i. Standard Risk Test snapshot is not found.
      ii. the first Jira ticket with document has been moved to Rejected.
      iii. any other exception occurs in the flow.

...