V3.3 Release Notes
- Dave Trier
What’s New:
ModelOp Center version 3.3 elevates the ModelOp user experience and interface, and extends support for generative AI, enabling enterprises to quickly harness the power of Large Language Models (LLMs) while safeguarding the business. New users will be able to get up to speed even more quickly and have additional intuitive tools at their fingertips. Version 3.3 includes major enhancements that will help ModelOp customers:
AI Governance Inventory & Comprehensive Use Case Management — Version 3.3 introduces an enhanced AI Governance inventory, making it straightforward for users to register and customize new AI use cases, including those involving generative AI, and quickly bulk import existing models.
AI Governance Score & Automated Compliance Controls — The new ModelOp AI Governance Score offers a rapid assessment tool — designed for leaders and executives — to ensure continuous adherence to governance and regulatory policies, including for third-party vendors and embedded models.
Enhanced Reporting on AI Governance Adherence — With enhanced search capabilities, users can swiftly navigate AI initiatives across the organization, generate comprehensive AI use case reports, maintain oversight of AI use cases, and perform critical comparisons of model versions focusing on performance, fairness, bias, and more.
Release Details:
Full List of Enhancements:
Governance Scores: added ability to rapidly calculate Governance adherence via the ModelOp Governance Score. This includes:
Governance Score Calculation: ability to calculate the Governance Score for a Use Case, Model Implementation, and/or Snapshot for a given Use Case
Missing/Failing Factors: ability to see the list of items that lowered the score
Rule Definition: ability to define the specific rules that are to be executed during Governance Score calculation
Inventory & Reporting:
Filtering by PII/PHI: added ability to search by implementations that have PII or PHI
Custom Search: added ability search for and report on Use Cases and Model Implementations in the inventory based on criteria that is contained in Custom Form Information or Custom Metadata
Export to Excel: added ability to generate a report of all Use Cases or Model Implementations that match a given criteria. This includes ability to export the resultant set to Excel for offline analysis.
Compliance Report: enhanced the visualization for custom metadata within the compliance report page of a given implementation
Risk Management: added ability to create and manage model risks directly within ModelOp (as opposed to using an existing enterprise system such as Jira). While most enterprises have an existing task/risk management system, this embedded capability is useful for those organizations that lack such a system, or cannot integrate directly with such a system. Specific features include:
Priority & due date: ability to set priority and due date
Comments: ability to add comments to a given risk
Resolve: ability to resolve a risk
Additional details: ability to add additional details such as Risk Category, Risk Rating, Compensating Controls, Recommendation, and Documentation
Model Card Generation: added support for generating model cards for a given Use Case. Specific features include:
Template: out-of-the-box template based on the Hugging Face model card
Charts: ability to embed charts/graphs within a model card
Sharing: ability to share a Model Card with colleagues
Download: ability to download a Model Card as a PDF
History: ability to persist model cards at very times throughout a Use Case’s history
Custom Forms: added ability to create custom forms that can be displayed in the UI to collect specific information about Use Cases or Model Implementations. This includes:
New Use Case Wizard: ability to include custom forms with the “Add a New Use Case” wizard, allowing for the collection of specific information for a given Use Case.
New Implementation Wizard: ability to include custom forms with the “Add a New Implementation” wizard, allowing for the collection of specific information for a given Implementation. Note that an Implementation Form can be created for each Model Type + Model Methodology combination.
Use Case Page: ability to view and modify information in the custom form, for a given Use Case
Implementation Page: ability to view and modify information in the custom form, for a given Implementation
Configuration: ability to create new custom forms and configure them with the specific fields (text, drop-down, radio, etc.) within the form
My Work: added new page to display the specific tasks, risks, issues for my group’s use cases and model implementations. This includes:
Summary statistics about Use Cases and Model Implementations in the system
Open risks/issues by priority
List of open risks/issues/tasks with sorting and filtering
Other UI Updates:
Use Cases Experience: enhanced the user experience to focus on the Use Case, which is business challenge/opportunity that AI/ML is solving. This includes:
Separated the Use Case from Model Implementations, as there may be multiple approaches (“implementations”) to address the use case. Note that ModelOp v3.3 has explicit terminology to separate Use Cases (e.g. Research Report Summarizer) vs. Model Implementation (e.g. LLM vs. NLP).
Added a new wizard to add a Use Case
Added a new Use Case page that summarizes all items related to a Use Case across all Implementations within the Use Case page: Risks, Issues, Notifications, Custom Information/Metadata, Governance Score, Metrics, Documentation, Approvals, Model Cards
Added a new wizard to add a new or existing Model Implementation to a Use Case
Added support for viewing Metrics over Time, including viewing metrics over time across snapshots for a given Implementation
Added support for ensembles, which are composed of multiple implementations to address a Use Case
Model Schema:
Added support to specify the fields that contain PII or PHI for input or output data for a model
Test Results Page:
Added support for Generic Scatter Plots and Donut Charts
Updated the Generic Line Chart and Generic Scatter Plot to only show the Top 10 features by default (instead of all of them)
Added specific Pass/Fail messages for a given model test result
Added support for clickable hyperlinks within a table or other values within the model test result
Added a link to the job details that generated the model test result
Updated axis label ticks on charts to improve readability, especially for labels that have long names
Added support to specific a fixed value for min/max values for the Y-Axis
Notification Details: added new Notification details page to provide more information about a given Risk, Issue, Task, Review, or general notification. This includes support for internally managed risks
Markdown File Viewer: added ability to view Markdown files directly within the ModelOp UI
New Use Case/Model Registration: When adding a new Use Case/Model, updated the UI to only show the Groups to which the user belongs.
Asset Uploader:
Enhanced the asset upload capability with upload status tracking, full-folder upload, size limit, and error messages.
Updated the asset uploader so that any source code files that are added to the system are added as embedded source.
MLC Overview: added indicators as to whether an MLC was suspended or active
MLC:
For a ModelReviewNotification, made it optional to attach documents (previously it was required)
Added ability to add any type of asset (not just a Document) to a ticket created by the MLC
Added a new external task to check if all Risk Notifications of a certain severity are resolved before allowing the MLC to proceed
Dashboard & Tests/Monitors:
Metrics over Time: updated all of the existing out-of-the-box metrics to support calculating metrics over time, based on the specific transaction timestamp within the business data itself. For example, if it is a fraud model, each record that the model creates has a transaction processing timestamp, the metrics (e.g. performance metrics) would be calculated for each day based on that transaction timestamp.
Note that a new role within the Schema called “predictionDate” was added to allow a user to specify which field in the schema is the predictionDate field.
NLP Tests/Monitors: provided out-of-the-box tests/monitors specifically for NLP. This includes tests for Sentiment Analysis, Top Words by Type, PII Leakage Analysis, and Semantic Textual Similarity
LLM Tests/Monitors: provided out-of-the-box tests/monitors specifically for LLM’s. This includes tests for Accuracy, Fact Checking Analysis, Rails File Validation, Prompt File Validation, Fairness Analysis, Sentiment Analysis, Top Words by Type, PII Leakage Analysis, and Semantic Textual Similarity
Ethical Fairness/Bias Tests: added ability to pass Reference Groups per Protected Class
Dashboard Process:
Added support to assign a prefix and suffix (e.g. unit label) on the top header cards
Updated out-of-the-box dashboard MLC to support environments that do not contain Jira
Updated logic for how the Dashboard MLC’s GetModelRequiredAssets delegate uses the required_assets.json to obtain the inputs for Dashboard execution
DMN Evaluation: added support for leveraging DMN files within the monitor’s repository.
Documentation Generation:
In the Generate Documentation wizard, added support for Document Templates that contain special characters in the filename
Added support for Documentation Generation with a Test Result without requiring Jira/ServiceNOW.
AWS SageMaker:
Added support to assume Role by Group when integrating with AWS SageMaker environments
Added support for “undeploying” a SageMaker model directly from the ModelOp UI
Spark Runtime:
Added support for the Spark Runtime where multiple assets can have the same filename. This can occur when running drift jobs or similar in Spark and the business model’s asset can have the same name as the monitor’s asset.
Updated the Create a Job wizard to automatically select the Spark Runtime type by default.
ModelOp Runtime:
Added support for IPv6.
Enhanced asset loading to support loading very large binaries onto the ModelOp runtime (e.g. when loading a large GGUF for LLM’s).
When loading a model onto the ModelOp runtime (e.g. for batch jobs), optimized the model loading to only load the model’s assets of type: MODEL_SOURCE, WEIGHTS_FILE, MODEL_BINARY_FILE, MODEL_SCHEMA, REQUIRED_LIBRARIES, {{GREAT_EXPECTATIONS_TEST}}, {{TEST_RESULT_COMPARATOR}}, {{DASHBOARD_RESULT_COMPARATOR}}
Jira/ServiceNOW:
Enhanced support for Jira Cloud integration to allow for setting Jira custom fields within a Jira Cloud environment
Jupyter:
Added support for RWX for onboarding models via the ModelOp Juptyer plugin
Security:
AzureAD:
Upgraded AzureADCustomGroupsConfig and AzureADReactiveCustomGroupsServiceImpl to receive the default WebClient exposed by GW that is able to use proxy values if defined.
Updated AzureAD integration to allow for displaying the AzureAD group names, instead of group id’s.
SAML2: added ability to obtain refresh tokens to avoid token expiration.
RWX:
Added ability to define default RWX permissions to the groups to which an end-user belongs.
For new group initialization, added ability to customize RWX privileges for the groups owned by users, so that a user belonging to a group will get specific privileges for that group.
Added support for configuring RWX privileges on Notifications (Risks, Issues, Approvals, etc.).
Added support for Admin to be able to manage permissions for the groups to which they belong.
Login:
Added support for JWT OAuth2 tokens carrying over hundreds of groups associated to a given user, to avoid getting Http 431 errors.
In secured mode, based on configuration, added ability to allow users without any groups to access MOC.
Regex for Group Visibility: added ability to apply a REGEX filter to the authenticated user groups contained inside the token, so that only groups that match the REGEX are selectable inside MOC.
Runtime Security: added ability to have ModelOp Runtimes request and validate incoming Opaque tokens from the ESG using POST instead of GET methods.
Installation / Configuration:
CORS Configuration: added ability to configure default Spring CORS rules through properties files, so that I can leverage CORS configurations to restrict/allow access.
Other:
Updated ModelOp Center to use Spring-boot v2.7.10
Bug Fixes:
Git:
Addressed issue where GitSync process fails when assets contain special characters on the filename
Addressed issue where metadata.py is being processed as metadata.json
Addressed GitSync issue when the local repository gets into MERGING state
Dashboard & Tests/Monitors
Addressed issue in Dashboard output, if there was no Test Data available on the business model
CLI:
Addressed issue with immutability when creating a batch job that uses SQL assets
UI:
Job Creation Wizard: addressed issue with the Job Creation Wizard where if you enter a filename and select the extension from the drop down in the job wizard, when you go the next step, it only shows the name without the extension.
Addressed issue when creating a new snapshot with existing associated models (e.g. monitors) where an asset changed on the base business model
ModelOp Python SDK:
Addressed issue with Python SDK that was adding an extra space to the scopes, before requesting a token.
Security Fixes/Patches:
NOTE: many of the below are NOT related to ModelOp software, but rather related to dependencies
Addressed CVE-2024-22243 & CVE-2024-22259 - Spring framework: URL Parsing with Host validation.
Addressed CVE-2024-25710 & CVE-2024-26308 - Commons-compress DoS caused by infinite loop and potential OutOfMemoryIssue
Addressed CVE-2024-22257 - spring-security: Broken Access Control With Direct Use of AuthenticatedVoter
Addressed CVE-2024-1597: Postgres JDBC SQL Injection Vulnerability
Issue Link: https://avd.aquasec.com/nvd/2024/cve-2024-1597
Addressed CVE-2023-20873 - Spring-boot: Security bypass with Wildcard Pattern Matching on Cloud Foundry
Addressed CVE-2023-34053 - io.micrometer:micrometer-core classpath vulnerable DoS
Addressed CVE-2023-34054 AND CVE-2023-34062 - Reactor netty HTTP Server DoS vulnerability
Addressed CVE-2022-36944 - Scala vulnerability fix in mlc-service
Issue Link: https://nvd.nist.gov/vuln/detail/cve-2022-36944
Addressed CVE-2019-10202 - Jackson vulnerability fix in mlc-service
Issue Link: https://nvd.nist.gov/vuln/detail/cve-2019-10202
Addressed CVE-2015-3253 - Groovy vulnerability fix in mlc-service
Issue Link: https://nvd.nist.gov/vuln/detail/cve-2015-3253
Addressed CVE-2021-42392, CVE-2022-2322, CVE-2021-23463 - Remove dependencies for H2 to address vulnerabilities in mlc-service
Addressed CVE-2022-3171, CVE-2022-3509, CVE-2022-3510 - Upgrade com.google.protobuf:protobuf-java to resolve vulnerabilities
Addressed CVE-2020-8908 -A temp directory creation vulnerability exists in all versions of Guava, allowing an attacker with access to the machine to potentially access data in a temporary directory created by the Guava API
Addressed CVE-2023-2976 - Upgrade Guava to 32+
Issue Link: CVE-2023-2976
Addressed CVE-2021-31684 - Upgrade net.minidev:json-smart to prevent denial of service
Issue Link: CVE-2021-31684
Addressed CVE-2023-1370 - json-smart - Can cause a stack exhaustion (stack overflow) and crash the software.
Addressed CVE-2023-43642 - Upgrade snappy-java dependency to 1.1.10.5
Issue Link: https://nvd.nist.gov/vuln/detail/CVE-2023-43642
Addressed CVE-2023-4586 - A vulnerability was found in the Hot Rod client. This security issue occurs as the Hot Rod client does not enable hostname validation when using TLS, possibly resulting in a man-in-the-middle (MITM) attack.
Issue Link: CVE-2023-4586
Addressed CVE-2023-44487 - The HTTP/2 protocol allows a denial of service (server resource consumption) because request cancellation can reset many streams quickly, as exploited in the wild in August through October 2023.
Addressed CVE-2023-34034 - Using "**" as a pattern in Spring Security configuration for WebFlux creates a mismatch in pattern matching between Spring Security and Spring WebFlux, and the potential for a security bypass
Issue Link: CVE-2023-34034
Addressed CVE-2023-39410 - Vulnerability at org.apache.avro:avro:1.10.2
Issue Link: CVE-2023-39410
Upgrade version of spark-core_2.12 to resolve vulnerabilities listed
Addressed CVE-2023-1436 - Upgrade Jettison to version 1.5.4
Issue Link: CVE-2023-1436