V3.2 Release Notes

Owned by Dave Trier
Apr 01, 2024
7 min read

What’s New:

ModelOp Center v3.2 introduces support for Generative AI models, including Large Language Models (LLM’s), providing comprehensive AI governance for these transformational models. Additionally, v3.2 introduces support fine-grained access controls, allowing for setting read/write/execute permissions at the model, snapshot, notification, job, and test result level across groups.

Governance & Security:

  • Generative AI Inventory: added support for classifying models as Generative AI (including LLM’s) as well as creating Generative AI ensembles, including managing langchain models, prompt templates, embedding models, LLM’s, and validation code (guardrails)

  • Generative AI Asset Management: added support for prompt templates, RAILS, etc. for Generative AI asset tracking

  • Granular Entity-Level Security: extending the current group-based isolation security model to provide fine-grain access control at a given entity-level (e.g. Snapshot, Job), such that read/write/execute privileges can be set at an individual entity level, as desired.

  • Google Cloud Storage Buckets: added support for managing technical artifacts (model binaries/weights, etc.) in Google Cloud Storage Buckets

 

Test, Monitor, & Visualize

  • REST-based Data Set Support: added support to pull model-specific data sets (e.g. training data, production data) via REST, allowing integration with existing REST-based data management systems

  • Testing/Monitoring Updates:

    • Added support for Rank Order Break

    • Added support for performance metrics for Probability of Default for Credit Models

    • Added support for longitudinal tracking of metrics

  • External Monitors: added support for collecting existing metrics for a model that are calculated from external monitors, with the ability to automate threshold comparison and remediation pathways.

 

General:

  • User Experience: minor User Interface enhancements including:

    • MLC tracking enhancements

    • Native Dashboard updates for usability

  • Model Archival: ability to archive a model snapshot and all its related artifacts, allowing for more clean visibility into active models while maintaining auditability.

 

Specific Details:

 

Full List of Enhancements:

  • Archive: Added ability to archive a Model Snapshot, which also archives the Snapshot’s related jobs, notifications, test results, and deployments

  • UI:

    • In the Compliance Overview page, updated the graphs to allow for drill-down into each of the individual charts

    • In the Monitoring Wizard, added the ability to specify parameters that can be used within the monitoring code

    • In the Model Import dialog, added ability to see the friendly user group name instead of groupID for environments using AzureAD

    • When adding a tag, the UI now stores the tags in UPPERCASE for consistency and usage by MLC’s, etc.

    • Updated the notifications pane in the Business Model and Snapshot pages to show all Job notifications for a given model/snapshot

    • On the Business Model Inventory page, updated the filters to (1) allow for viewing all models related to a given group (2) toggle to only show models submitted by the current user

    • Updated all data selector dialogs to ensure that the dates are cleared properly

  • MLC:

    • Added ability to pass a Map object in a new optional input variable BUILD_PARAMETERS_MAP to the Create Jenkins Job delegate. This allows the Map to be submitted to Jenkins as an individual parameter.

    • Simplified the OOTB Job Handling MLC by removing the subprocess, providing maintainability and performance improvements

    • Added ability to upload Jira or ServiceNOW attachments as model documents on a Snapshot

    • Added support to trigger an MLC based on a specific file change

  • Google Cloud Storage Buckets: added support for managing technical artifacts (model binaries/weights, etc.) in Google Cloud Storage Buckets

    • Added ability to specify the region for the bucket

  • AWS SageMaker:

    • Addressed AWS Rate Exceeded issue more gracefully by optimizing the calls from the ModelOp Sagemaker service and also handling the rate exceeded messages gracefully with proper retries

    • Optimized the SageMaker model import to import basic information immediately and queuing all of the SageMaker transform/other jobs to be imported as a background process. This allows the user to view the basic information about the SageMaker model immediately without having to wait for all the other jobs to import

    • Added ability to create an AWS SageMaker snapshot directly from the UI, which will use the most recent AWS SageMaker Endpoint Configuration OR will create a default Endpoint Configuration, if one does not already exist for the model

    • Updated websocket header security settings to address API syncing issues with the latest SageMaker API’s

  • Jira / ServiceNOW:

    • within an MLC, added new capability to upload Jira or ServiceNOW attachments directly to a Business Model as assets. The new delegates are ApplyJiraAttachmentToStoredModel and ApplyServiceNowAttachmentToStoredModel, respectively

    • In an MLC, added ability to set "assignee" for a Jira ticket

    • Added ability to set Model Approval Type in an MLC

    • When pushing a Jira or ServiceNOW attachment back to a model snapshot, added the ability to append a timestamp to the uploaded document’s name

  • Jupyter:

    • If a Jupyter notebook is in a git repository, updated the import functionality to import the notebook as an external file instead of storing the notebook contents in Mongo for performance and maintainability

  • ModelOp Runtime:

    • REST-based Data Set Support: added support to pull model-specific data sets (e.g. training data, production data) via REST, allowing integration with existing REST-based data management systems

      • Added support to provide a configurable timeout for REST calls

    • When loading large assets, added a startup state to indicate that the Engine “jet” has been created but is not fully ready yet

  • Dashboard & Tests/Monitors:

    • Updated the default Dashboard model to only pull the latest Model Test Result for a given model category

    • Updated the Inference count on the Dashboard model to pull from the Volumetrics record count, which will be more commonly available across ALL models

    • Added the ability to calculate Monitoring metrics for a specific time window (e.g. current day/week/month/quarter) automatically for a business model snapshot WITHOUT having to manually change the exact dates for each monitoring execution

  • ModelOp Python SDK:

    • Added the ability to pull Model Approval Notifications by storedModelID and by deployableModelID from the ModelOp Python SDK for custom monitors

    • Added environment variables to configure the following ModelOp Python SDK parameters:

      • MOC_VERIFY_SSL ( boolean )

      • MOC_ALLOW_REDIRECTS ( boolean )

      • {{MOC_TIMEOUT_SECONDS}} ( seconds )

  • Security:

  • Installation / Configuration:

    • Added support to configure different external asset repositories (e.g. AWS s3 vs. Azure Blob Store) for different groups

    • Added support to only show a sub-set of the total groups coming inside an end-user token based on a REGEX pattern

    • Added support to define separate basic authentication credentials for a proxy server used within the enterprise

    • Updated MLC Server configurations to provide proactive cleanup of the MLC HISTORY data in the MLC Server database

    • Updated Helm installation support for Vault integration

 

Bug Fixes:

  • Azure:

    • Addressed an issue with non-AWS external repositories when pushing a Snapshot’s attachments to a Jira ticket

    • Addressed an issue with non-AWS external repositories when pushing Jira attachments back to a Snapshot.

  • Snapshots & Jobs

    • Updated the logic to find compatible runtimes to exclude runtimes that have an online deployment

  • MLC’s:

    • Updated the OOTB MLC’s to automatically add the default OOTB monitors (drift, stability, concept drift, performance) upon deployment

    • Updated the runtime matching error message for the default deployment MLC

    • Updated the Model Review Notification creation to only allow either a deployable model or stored model to be specified to avoid confusion

    • Addressed issue with setting the Model Approval Type in an MLC

    • Updated error handling of Jira and ServiceNOW External Task Workers to gracefully exit the task if there is an unrecognized, unrecoverable issue

    • In the External Task Workers, added fix to check the processTask function to make sure the execution is still alive, and if not, kill the task

  • UI:

    • Addressed an issue with the tags drop-down filter on the Assets page

    • Addressed an issue with modifying the association role on an associated model in the UI

    • Updated the Run Monitor functionality on the Monitors tab of the Snapshot to only allow for one monitor of the same name to run

    • Updated SSL Timeout configuration to address infrequent 500 issue typically due to a customer environment load balancer timeout issue

    • When re-running a training job, addressed an issue where an extra asset was appended from the prior run

Security Fixes/Patches:

  • Addressed CVE-2022-40152: A flaw was found in the FasterXML/woodstox package-. This flaw allows an attacker to cause a denial of service (DoS) in its target via XML serialization. An attacker may benefit from the parser sending a malicious input that may cause a crash. This vulnerability is only relevant for users making use of the DTD parsing functionality.

  • Addressed CVE-2022-1471: vulnerability and compatibility issue with snake_yaml 2.0 and spring boot 2.6

  • Addressed CVE-2022-25647: A flaw was found in gson, which is vulnerable to Deserialization of Untrusted Data via the writeReplace() method in internal classes. This issue may lead to availability attacks.

  • Addressed CVE-2021-40690: All versions of Apache Santuario - XML Security for Java prior to 2.2.3 and 2.1.7 are vulnerable to an issue where the "secureValidation" property is not passed correctly when creating a KeyInfo from a KeyInfoReference element. This allows an attacker to abuse an XPath Transform to extract any local .xml files in a RetrievalMethod element.

  • Addressed CVE-2020-11988: Apache XmlGraphics Commons 2.4 is vulnerable to server-side request forgery, caused by improper input validation by the XMPParser. By using a specially-crafted argument, an attacker could exploit this vulnerability to cause the underlying server to make arbitrary GET requests.

  • Addressed CVE-2022-41704: Affected versions of this package are vulnerable to XML External Entity (XXE) Injection. {{NodePickerPanel}} loads External DTDs.

  • Addressed CVE-2022-34169: The Apache Xalan Java XSLT library is vulnerable to an integer truncation issue when processing malicious XSLT stylesheets. This can be used to corrupt Java class files generated by the internal XSLTC compiler and execute arbitrary Java bytecode.

  • Addressed SNYK-JAVA-XALAN-2953385: Xalan is a XSLT processor for transforming XML documents into HTML, text, or other XML document types. It implements XSL Transformations (XSLT) Version 1.0 and XML Path Language (XPath) Version 1.0 and can be used from the command line, in an applet or a servlet, or as a module in other program. Affected versions of this package are vulnerable to Arbitrary Code Execution when processing malicious XSLT stylesheets, due to an integer truncation issue. This allows attackers to corrupt Java class files generated by the internal XSLTC compiler and execute arbitrary Java bytecode.

  • Addressed CVE-2023-24998: Apache Commons FileUpload before 1.5 does not limit the number of request parts to be processed resulting in the possibility of an attacker triggering a DoS with a malicious upload or series of uploads. Note that, like all of the file upload limits, the new configuration option (FileUploadBase#setFileCountMax) is not enabled by default and must be explicitly configured.

  • Addressed CVE-2022-40146: Server-Side Request Forgery (SSRF) vulnerability in Batik of Apache XML Graphics allows an attacker to access files using a Jar url. This issue affects Apache XML Graphics Batik 1.14.

  • Addressed org.apache.pdfbox:pdfbox DoS issue. PDFBox is an open source Java tool for working with PDF documents. Affected versions of this package are vulnerable to Denial of Service (DoS) via a crafted PDF file that may trigger an infinite loop during parsing.

  • Addressed SNYK-JAVA-ORGAPACHEXMLGRAPHICS-3031729: Affected versions of this package are vulnerable to Server-side Request Forgery (SSRF) by allowing an attacker to fetch external resources.