V3.3 Release Notes

Owned by Dave Trier
Last updated: Jul 17, 2024
10 min read

What’s New:

ModelOp Center version 3.3 elevates the ModelOp user experience and interface, and extends support for generative AI, enabling enterprises to quickly harness the power of Large Language Models (LLMs) while safeguarding the business. New users will be able to get up to speed even more quickly and have additional intuitive tools at their fingertips. Version 3.3 includes major enhancements that will help ModelOp customers:

  • AI Governance Inventory & Comprehensive Use Case Management — Version 3.3 introduces an enhanced AI Governance inventory, making it straightforward for users to register and customize new AI use cases, including those involving generative AI, and quickly bulk import existing models.

  • AI Governance Score & Automated Compliance Controls — The new ModelOp AI Governance Score offers a rapid assessment tool — designed for leaders and executives — to ensure continuous adherence to governance and regulatory policies, including for third-party vendors and embedded models.

  • Enhanced Reporting on AI Governance Adherence — With enhanced search capabilities, users can swiftly navigate AI initiatives across the organization, generate comprehensive AI use case reports, maintain oversight of AI use cases, and perform critical comparisons of model versions focusing on performance, fairness, bias, and more.

Release Details:

Full List of Enhancements:

  • Governance Scores: added ability to rapidly calculate Governance adherence via the ModelOp Governance Score. This includes:

    • Governance Score Calculation: ability to calculate the Governance Score for a Use Case, Model Implementation, and/or Snapshot for a given Use Case

    • Missing/Failing Factors: ability to see the list of items that lowered the score

    • Rule Definition: ability to define the specific rules that are to be executed during Governance Score calculation

  • Inventory & Reporting:

    • Filtering by PII/PHI: added ability to search by implementations that have PII or PHI

    • Custom Search: added ability search for and report on Use Cases and Model Implementations in the inventory based on criteria that is contained in Custom Form Information or Custom Metadata

    • Export to Excel: added ability to generate a report of all Use Cases or Model Implementations that match a given criteria. This includes ability to export the resultant set to Excel for offline analysis.

    • Compliance Report: enhanced the visualization for custom metadata within the compliance report page of a given implementation

  • Risk Management: added ability to create and manage model risks directly within ModelOp (as opposed to using an existing enterprise system such as Jira). While most enterprises have an existing task/risk management system, this embedded capability is useful for those organizations that lack such a system, or cannot integrate directly with such a system. Specific features include:

    • Priority & due date: ability to set priority and due date

    • Comments: ability to add comments to a given risk

    • Resolve: ability to resolve a risk

    • Additional details: ability to add additional details such as Risk Category, Risk Rating, Compensating Controls, Recommendation, and Documentation

  • Model Card Generation: added support for generating model cards for a given Use Case. Specific features include:

    • Template: out-of-the-box template based on the Hugging Face model card

    • Charts: ability to embed charts/graphs within a model card

    • Sharing: ability to share a Model Card with colleagues

    • Download: ability to download a Model Card as a PDF

    • History: ability to persist model cards at very times throughout a Use Case’s history

  • Custom Forms: added ability to create custom forms that can be displayed in the UI to collect specific information about Use Cases or Model Implementations. This includes:

    • New Use Case Wizard: ability to include custom forms with the “Add a New Use Case” wizard, allowing for the collection of specific information for a given Use Case.

    • New Implementation Wizard: ability to include custom forms with the “Add a New Implementation” wizard, allowing for the collection of specific information for a given Implementation. Note that an Implementation Form can be created for each Model Type + Model Methodology combination.

    • Use Case Page: ability to view and modify information in the custom form, for a given Use Case

    • Implementation Page: ability to view and modify information in the custom form, for a given Implementation

    • Configuration: ability to create new custom forms and configure them with the specific fields (text, drop-down, radio, etc.) within the form

  • My Work: added new page to display the specific tasks, risks, issues for my group’s use cases and model implementations. This includes:

    • Summary statistics about Use Cases and Model Implementations in the system

    • Open risks/issues by priority

    • List of open risks/issues/tasks with sorting and filtering

  • Other UI Updates:

    • Use Cases Experience: enhanced the user experience to focus on the Use Case, which is business challenge/opportunity that AI/ML is solving. This includes:

      • Separated the Use Case from Model Implementations, as there may be multiple approaches (“implementations”) to address the use case. Note that ModelOp v3.3 has explicit terminology to separate Use Cases (e.g. Research Report Summarizer) vs. Model Implementation (e.g. LLM vs. NLP).

      • Added a new wizard to add a Use Case

      • Added a new Use Case page that summarizes all items related to a Use Case across all Implementations within the Use Case page: Risks, Issues, Notifications, Custom Information/Metadata, Governance Score, Metrics, Documentation, Approvals, Model Cards

      • Added a new wizard to add a new or existing Model Implementation to a Use Case

      • Added support for viewing Metrics over Time, including viewing metrics over time across snapshots for a given Implementation

      • Added support for ensembles, which are composed of multiple implementations to address a Use Case

    • Model Schema:

      • Added support to specify the fields that contain PII or PHI for input or output data for a model

    • Test Results Page:

      • Added support for Generic Scatter Plots and Donut Charts

      • Updated the Generic Line Chart and Generic Scatter Plot to only show the Top 10 features by default (instead of all of them)

      • Added specific Pass/Fail messages for a given model test result

      • Added support for clickable hyperlinks within a table or other values within the model test result

      • Added a link to the job details that generated the model test result

      • Updated axis label ticks on charts to improve readability, especially for labels that have long names

      • Added support to specific a fixed value for min/max values for the Y-Axis

    • Notification Details: added new Notification details page to provide more information about a given Risk, Issue, Task, Review, or general notification. This includes support for internally managed risks

    • Markdown File Viewer: added ability to view Markdown files directly within the ModelOp UI

    • New Use Case/Model Registration: When adding a new Use Case/Model, updated the UI to only show the Groups to which the user belongs.

    • Asset Uploader:

      • Enhanced the asset upload capability with upload status tracking, full-folder upload, size limit, and error messages.

      • Updated the asset uploader so that any source code files that are added to the system are added as embedded source.

    • MLC Overview: added indicators as to whether an MLC was suspended or active

  • MLC:

    • For a ModelReviewNotification, made it optional to attach documents (previously it was required)

    • Added ability to add any type of asset (not just a Document) to a ticket created by the MLC

    • Added a new external task to check if all Risk Notifications of a certain severity are resolved before allowing the MLC to proceed

  • Dashboard & Tests/Monitors:

    • Metrics over Time: updated all of the existing out-of-the-box metrics to support calculating metrics over time, based on the specific transaction timestamp within the business data itself. For example, if it is a fraud model, each record that the model creates has a transaction processing timestamp, the metrics (e.g. performance metrics) would be calculated for each day based on that transaction timestamp.

      • Note that a new role within the Schema called “predictionDate” was added to allow a user to specify which field in the schema is the predictionDate field.

    • NLP Tests/Monitors: provided out-of-the-box tests/monitors specifically for NLP. This includes tests for Sentiment Analysis, Top Words by Type, PII Leakage Analysis, and Semantic Textual Similarity

    • LLM Tests/Monitors: provided out-of-the-box tests/monitors specifically for LLM’s. This includes tests for Accuracy, Fact Checking Analysis, Rails File Validation, Prompt File Validation, Fairness Analysis, Sentiment Analysis, Top Words by Type, PII Leakage Analysis, and Semantic Textual Similarity

    • Ethical Fairness/Bias Tests: added ability to pass Reference Groups per Protected Class

    • Dashboard Process:

      • Added support to assign a prefix and suffix (e.g. unit label) on the top header cards

      • Updated out-of-the-box dashboard MLC to support environments that do not contain Jira

      • Updated logic for how the Dashboard MLC’s GetModelRequiredAssets delegate uses the required_assets.json to obtain the inputs for Dashboard execution

    • DMN Evaluation: added support for leveraging DMN files within the monitor’s repository.

  • Documentation Generation:

    • In the Generate Documentation wizard, added support for Document Templates that contain special characters in the filename

    • Added support for Documentation Generation with a Test Result without requiring Jira/ServiceNOW.

  • AWS SageMaker:

    • Added support to assume Role by Group when integrating with AWS SageMaker environments

    • Added support for “undeploying” a SageMaker model directly from the ModelOp UI

  • Spark Runtime:

    • Added support for the Spark Runtime where multiple assets can have the same filename. This can occur when running drift jobs or similar in Spark and the business model’s asset can have the same name as the monitor’s asset.

    • Updated the Create a Job wizard to automatically select the Spark Runtime type by default.

  • ModelOp Runtime:

    • Added support for IPv6.

    • Enhanced asset loading to support loading very large binaries onto the ModelOp runtime (e.g. when loading a large GGUF for LLM’s).

    • When loading a model onto the ModelOp runtime (e.g. for batch jobs), optimized the model loading to only load the model’s assets of type: MODEL_SOURCE, WEIGHTS_FILE, MODEL_BINARY_FILE, MODEL_SCHEMA, REQUIRED_LIBRARIES, {{GREAT_EXPECTATIONS_TEST}}, {{TEST_RESULT_COMPARATOR}}, {{DASHBOARD_RESULT_COMPARATOR}}

  • Jira/ServiceNOW:

    • Enhanced support for Jira Cloud integration to allow for setting Jira custom fields within a Jira Cloud environment

  • Jupyter:

    • Added support for RWX for onboarding models via the ModelOp Juptyer plugin

  • Security:

    • AzureAD:

      • Upgraded AzureADCustomGroupsConfig and AzureADReactiveCustomGroupsServiceImpl to receive the default WebClient exposed by GW that is able to use proxy values if defined.

      • Updated AzureAD integration to allow for displaying the AzureAD group names, instead of group id’s.

    • SAML2: added ability to obtain refresh tokens to avoid token expiration.

    • RWX:

      • Added ability to define default RWX permissions to the groups to which an end-user belongs.

      • For new group initialization, added ability to customize RWX privileges for the groups owned by users, so that a user belonging to a group will get specific privileges for that group.

      • Added support for configuring RWX privileges on Notifications (Risks, Issues, Approvals, etc.).

      • Added support for Admin to be able to manage permissions for the groups to which they belong.

    • Login:

      • Added support for JWT OAuth2 tokens carrying over hundreds of groups associated to a given user, to avoid getting Http 431 errors.

      • In secured mode, based on configuration, added ability to allow users without any groups to access MOC.

    • Regex for Group Visibility: added ability to apply a REGEX filter to the authenticated user groups contained inside the token, so that only groups that match the REGEX are selectable inside MOC.

    • Runtime Security: added ability to have ModelOp Runtimes request and validate incoming Opaque tokens from the ESG using POST instead of GET methods.

  • Installation / Configuration:

    • CORS Configuration: added ability to configure default Spring CORS rules through properties files, so that I can leverage CORS configurations to restrict/allow access.

  • Other:

    • Updated ModelOp Center to use Spring-boot v2.7.10

 

Bug Fixes:

  • Git:

    • Addressed issue where GitSync process fails when assets contain special characters on the filename

    • Addressed issue where metadata.py is being processed as metadata.json

    • Addressed GitSync issue when the local repository gets into MERGING state

  • Dashboard & Tests/Monitors

    • Addressed issue in Dashboard output, if there was no Test Data available on the business model

  • CLI:

    • Addressed issue with immutability when creating a batch job that uses SQL assets

  • UI:

    • Job Creation Wizard: addressed issue with the Job Creation Wizard where if you enter a filename and select the extension from the drop down in the job wizard, when you go the next step, it only shows the name without the extension.

    • Addressed issue when creating a new snapshot with existing associated models (e.g. monitors) where an asset changed on the base business model

  • ModelOp Python SDK:

    • Addressed issue with Python SDK that was adding an extra space to the scopes, before requesting a token.

Security Fixes/Patches:

NOTE: many of the below are NOT related to ModelOp software, but rather related to dependencies