V3.2 Release Notes

Owned by Dave Trier
Apr 01, 2024
7 min read

What’s New:

ModelOp Center v3.2 introduces support for Generative AI models, including Large Language Models (LLM’s), providing comprehensive AI governance for these transformational models. Additionally, v3.2 introduces support fine-grained access controls, allowing for setting read/write/execute permissions at the model, snapshot, notification, job, and test result level across groups.

Governance & Security:

  • Generative AI Inventory: added support for classifying models as Generative AI (including LLM’s) as well as creating Generative AI ensembles, including managing langchain models, prompt templates, embedding models, LLM’s, and validation code (guardrails)

  • Generative AI Asset Management: added support for prompt templates, RAILS, etc. for Generative AI asset tracking

  • Granular Entity-Level Security: extending the current group-based isolation security model to provide fine-grain access control at a given entity-level (e.g. Snapshot, Job), such that read/write/execute privileges can be set at an individual entity level, as desired.

  • Google Cloud Storage Buckets: added support for managing technical artifacts (model binaries/weights, etc.) in Google Cloud Storage Buckets

 

Test, Monitor, & Visualize

  • REST-based Data Set Support: added support to pull model-specific data sets (e.g. training data, production data) via REST, allowing integration with existing REST-based data management systems

  • Testing/Monitoring Updates:

    • Added support for Rank Order Break

    • Added support for performance metrics for Probability of Default for Credit Models

    • Added support for longitudinal tracking of metrics

  • External Monitors: added support for collecting existing metrics for a model that are calculated from external monitors, with the ability to automate threshold comparison and remediation pathways.

 

General:

  • User Experience: minor User Interface enhancements including:

    • MLC tracking enhancements

    • Native Dashboard updates for usability

  • Model Archival: ability to archive a model snapshot and all its related artifacts, allowing for more clean visibility into active models while maintaining auditability.

 

Specific Details:

 

Full List of Enhancements:

  • Archive: Added ability to archive a Model Snapshot, which also archives the Snapshot’s related jobs, notifications, test results, and deployments

  • UI:

    • In the Compliance Overview page, updated the graphs to allow for drill-down into each of the individual charts

    • In the Monitoring Wizard, added the ability to specify parameters that can be used within the monitoring code

    • In the Model Import dialog, added ability to see the friendly user group name instead of groupID for environments using AzureAD

    • When adding a tag, the UI now stores the tags in UPPERCASE for consistency and usage by MLC’s, etc.

    • Updated the notifications pane in the Business Model and Snapshot pages to show all Job notifications for a given model/snapshot

    • On the Business Model Inventory page, updated the filters to (1) allow for viewing all models related to a given group (2) toggle to only show models submitted by the current user

    • Updated all data selector dialogs to ensure that the dates are cleared properly

  • MLC:

    • Added ability to pass a Map object in a new optional input variable BUILD_PARAMETERS_MAP to the Create Jenkins Job delegate. This allows the Map to be submitted to Jenkins as an individual parameter.

    • Simplified the OOTB Job Handling MLC by removing the subprocess, providing maintainability and performance improvements

    • Added ability to upload Jira or ServiceNOW attachments as model documents on a Snapshot

    • Added support to trigger an MLC based on a specific file change

  • Google Cloud Storage Buckets: added support for managing technical artifacts (model binaries/weights, etc.) in Google Cloud Storage Buckets

    • Added ability to specify the region for the bucket

  • AWS SageMaker:

    • Addressed AWS Rate Exceeded issue more gracefully by optimizing the calls from the ModelOp Sagemaker service and also handling the rate exceeded messages gracefully with proper retries

    • Optimized the SageMaker model import to import basic information immediately and queuing all of the SageMaker transform/other jobs to be imported as a background process. This allows the user to view the basic information about the SageMaker model immediately without having to wait for all the other jobs to import

    • Added ability to create an AWS SageMaker snapshot directly from the UI, which will use the most recent AWS SageMaker Endpoint Configuration OR will create a default Endpoint Configuration, if one does not already exist for the model

    • Updated websocket header security settings to address API syncing issues with the latest SageMaker API’s

  • Jira / ServiceNOW:

    • within an MLC, added new capability to upload Jira or ServiceNOW attachments directly to a Business Model as assets. The new delegates are ApplyJiraAttachmentToStoredModel and ApplyServiceNowAttachmentToStoredModel, respectively

    • In an MLC, added ability to set "assignee" for a Jira ticket

    • Added ability to set Model Approval Type in an MLC

    • When pushing a Jira or ServiceNOW attachment back to a model snapshot, added the ability to append a timestamp to the uploaded document’s name

  • Jupyter:

    • If a Jupyter notebook is in a git repository, updated the import functionality to import the notebook as an external file instead of storing the notebook contents in Mongo for performance and maintainability

  • ModelOp Runtime:

    • REST-based Data Set Support: added support to pull model-specific data sets (e.g. training data, production data) via REST, allowing integration with existing REST-based data management systems

      • Added support to provide a configurable timeout for REST calls

    • When loading large assets, added a startup state to indicate that the Engine “jet” has been created but is not fully ready yet

  • Dashboard & Tests/Monitors:

    • Updated the default Dashboard model to only pull the latest Model Test Result for a given model category

    • Updated the Inference count on the Dashboard model to pull from the Volumetrics record count, which will be more commonly available across ALL models

    • Added the ability to calculate Monitoring metrics for a specific time window (e.g. current day/week/month/quarter) automatically for a business model snapshot WITHOUT having to manually change the exact dates for each monitoring execution

  • ModelOp Python SDK:

    • Added the ability to pull Model Approval Notifications by storedModelID and by deployableModelID from the ModelOp Python SDK for custom monitors

    • Added environment variables to configure the following ModelOp Python SDK parameters:

      • MOC_VERIFY_SSL ( boolean )

      • MOC_ALLOW_REDIRECTS ( boolean )

      • {{MOC_TIMEOUT_SECONDS}} ( seconds )

  • Security:

  • Installation / Configuration:

    • Added support to configure different external asset repositories (e.g. AWS s3 vs. Azure Blob Store) for different groups

    • Added support to only show a sub-set of the total groups coming inside an end-user token based on a REGEX pattern

    • Added support to define separate basic authentication credentials for a proxy server used within the enterprise

    • Updated MLC Server configurations to provide proactive cleanup of the MLC HISTORY data in the MLC Server database

    • Updated Helm installation support for Vault integration

 

Bug Fixes:

  • Azure:

    • Addressed an issue with non-AWS external repositories when pushing a Snapshot’s attachments to a Jira ticket

    • Addressed an issue with non-AWS external repositories when pushing Jira attachments back to a Snapshot.

  • Snapshots & Jobs

    • Updated the logic to find compatible runtimes to exclude runtimes that have an online deployment

  • MLC’s:

    • Updated the OOTB MLC’s to automatically add the default OOTB monitors (drift, stability, concept drift, performance) upon deployment

    • Updated the runtime matching error message for the default deployment MLC

    • Updated the Model Review Notification creation to only allow either a deployable model or stored model to be specified to avoid confusion

    • Addressed issue with setting the Model Approval Type in an MLC

    • Updated error handling of Jira and ServiceNOW External Task Workers to gracefully exit the task if there is an unrecognized, unrecoverable issue

    • In the External Task Workers, added fix to check the processTask function to make sure the execution is still alive, and if not, kill the task

  • UI:

    • Addressed an issue with the tags drop-down filter on the Assets page

    • Addressed an issue with modifying the association role on an associated model in the UI

    • Updated the Run Monitor functionality on the Monitors tab of the Snapshot to only allow for one monitor of the same name to run

    • Updated SSL Timeout configuration to address infrequent 500 issue typically due to a customer environment load balancer timeout issue

    • When re-running a training job, addressed an issue where an extra asset was appended from the prior run

Security Fixes/Patches: