Example Workflows - LLM Model Implementations

Owned by Dave Trier
Last updated: Aug 07, 2024
3 min read

This article describes how ModelOp Center’s MLC’s can be used to orchestrate and automate the Governance - LLM Implementations process.

Table of Contents

Overview

The ModelOp Life Cycle Manager (MLC Manager) orchestrates and automates the the various steps involved in associating an LLM model to a registered use case .

Pre-Requisites

The following are the prerequisites for leveraging this Governance - LLM Implementations MLC:

  • The Jira project must be configured with Done and Rejected statuses in the workflow

  • Attachments must be enabled for the Jira project

  • Model named "LLM Standardized Tests" must be present with a valid snapshot.

  • Deployment stage : Engine environment name must be "prod" + model tag (if any).

  • Business model must have appropriate assets with the TEST_DATA and RAILS asset roles

Process Overview

This is a full LLM implementation of a model lifecycle. It begins with the gathering of the use case, and review of its risk to the business. It then creates a development ticket, and after implementation registration it makes sure the submission is complete. Then second line will review all generated documentation, before seeking final approval from a risk perspective. It then moves onto secops review, along with a deployment into a test environment for a week. After a final review of the validation environment performance, it will be deployed to production.

 

 

  1. Scheduler - the process is triggered by Use Case Registration MLC if the implementation model methodology is LLM.

  2. Jira PIA Approval: the process creates a Jira ticket for PIA approval and continues after the ticket is moved to appropriate exit status.

  3. Model Completeness Check - the process validates if the following assets are available .

    - Readme asset - Input schema - Output schema - Metrics test asset - Rails Test Questions asset - Rails File asset - Training data asset - Test result comparator data asset (dmn)

  4. Run LLM Standard Risk Tests - the process validates if the model assets with the Training data, Rails File Asset and Rails Test Question assets in preparation to run LLM Standard Risk Tests on the model. If the assets are present in the model the MLC runs the Standard Risk Tests and creates a notification if any of the monitors in Standard Risk Tests fail to run. If the test job fails, a Jira ticket is created with the failed job error message. A test can be re-run if the Jira ticket is moved to Done.

  5. Approval Based on Test Results - A Jira ticket is created with document generated from the model test results for validation. If ticket is moved to Done the flow continues and adds the generated document to the model snapshot.

  6. Approval for Deployment - Jira ticket is created to proceed with the model deployment.

  7. Deploy to Production - If ticket is moved to Done the snapshot is deployed to production environment stage runtime.

  8. Error handling - A notification with appropriate error message is created if any step fails in the mlc.

Next Article: Example Workflows - Vendor Model Implementations >

Â