Introduction

Enterprises have invested millions of dollars into Data Science efforts over the past few years, with many enterprises realizing early successes in their programs. However, these enterprises typically have a plethora of different model development tools, languages/frameworks, environments, and varying operational systems across the disparate teams. Each team typically takes their own approach to operating, managing, and governing their Data Science programs. Additionally, few teams track the business value that these Data Science models are unlocking. The result is that executives have little-to-no visibility into (a) all Data Science models used for production business decisioning (b) the return on investment for those models (c) the health and status of these models.

ModelOp Center’s executive dashboard allows enterprises to overcome these challenges, providing visibility into the business value of each model, as well as the operational, IT, risk, and data science KPI’s for all models across the enterprise, regardless of where the model was developed, the language/framework used, the environment in which it runs, etc.

Open

Dashboard Overview

Key Concepts

1. “Model in Production”: ModelOp Center considers a “model in production” as a given model snapshot in “state”=DEPLOYED to a runtime that has the “inProduction” flag set to true. Typically, a model life cycle (MLC) is what orchestrates the process of putting a model in production, whereby the MLC sets the snapshot’s state to “DEPLOYED” and also pushes the snapshot to the appropriate runtime(s) that have the “inProduction” flag set to true. A user can verify that a model is in production by navigating to the snapshot page and viewing the deployments section.

   1. Open

      

2. “Open Priority Issue”: ModelOp Center considers an “open priority issue” as any notification of “severity”=CRITICAL or HIGH that has an attached ticket (JIRA, ServiceNOW, etc) of elevated priority. A user can verify that a given Model has an Open Priority Issue by navigating to the model snapshot page and viewing if there are any “Associated Tickets” of severity=Error and status=(not closed).

Dashboard Details:

The ModelOp Center Executive Dashboard is divided into 3 main sections:

1. Summary: provides cumulative statistics such as the number of models in production, cumulative ROI of all models, and cumulative usage (inferences)

2. Individual Model Status: for each production model, provides a summary of the current business value, open priority issues, as well as a “heatmap” of the current status of all major health indicators against KPI’s.

3. Issues: provides insights into the list of open issues by Business Unit as well as the breakdown of the issue types

Dashboard Section Details

The Summary and Issues section are standard across all ModelOp Center implementations. However, the “Per Model Status” section can be customized to the specific requirements of a given implementation. See the “Default Dashboard” section below for examples of how to customize the dashboard.

Summary Section Metrics

The Summary section contains the following information:

1. Cumulative Value: this value is computed by summing the dollar amount for each “production model” contained in the “Business KPI” field of each production model in the “Per Model Status” section. Note that this cumulative value only sums numerical dollar values within the “Business KPI” field of each model – it will ignore all other KPI metrics. This field is dynamically calculated upon dashboard load.

2. Daily Inferences: this value is computed by summing the daily inferences for each “production model” contained in the “Daily Inferences” field of each production model in the “Per Model Status” section. Note that this cumulative value only sums numerical values within the “Daily Inferences” field of each model – it will ignore non-numerical values. This field is dynamically calculated upon dashboard load.

3. Models in Production: per the “Key Concepts” section, ModelOp Center considers a “model in production” as a given model snapshot in “state”=DEPLOYED to a runtime that has the “inProduction” flag set to true. This field is dynamically obtained upon Dashboard load by querying ModelOp Center’s “Business Model” inventory to determine all models matching the aforementioned criteria.

4. Open Priority Issues: per the “Key Concepts” section, ModelOp Center considers an “open priority issue” as any notification of “severity”=CRITICAL or HIGH that has an attached ticket (JIRA, ServiceNOW, etc) of elevated priority. This field is dynamically obtained upon Dashboard load by querying ModelOp Center’s Notifications API to determine matching criteria.

Individual Model Status

The center section displays the status for each individual model “in production”. This section is configurable (see below); however, the information produced by the Default Dashboard model include the following:

1. Deployed Models By Business KPI: the name of the Business model and the name of the specific snapshot that is currently “in production.” This field is dynamically obtained upon Dashboard load by querying ModelOp Center’s “Business Model” inventory to determine all models matching the aforementioned criteria.

2. Business Unit: the business unit to which the model belongs. This information is pulled dynamically from the “Model Organization” metadata element of a Business model.

3. Business KPI: the cumulative business value for the Business model. This information is calculated by the “Dashboard Model” that is run regularly for each Business model that is “in production.”

4. Open Priority Issues: the number of “open priority issues” for the given Business Model and the trend over the last 30 days. Per the “Key Concepts” section, ModelOp Center considers an “open priority issue” as any notification of “severity”=CRITICAL or HIGH that has an attached ticket (JIRA, ServiceNOW, etc) of elevated priority. This field is dynamically obtained upon Dashboard load by querying ModelOp Center’s Notifications API to determine matching criteria.

5. Heatmap Values: this section provides a red/green/yellow status of the key health indicators for a model. These metrics are calculated by the “Dashboard Model” and evaluated against thresholds using the “Dashboard Model’s” DMN file to produce the specific red/green/yellow/gray status. To note, by default, GRAY indicates that the “Dashboard model” could not process that particular metric. The metrics included in the default Heatmap include:

   1. Characteristic Stability: calculates the characteristic stability index for each feature and compares the max value against the thresholds (from the DMN) to determine the status.

   2. Performance Monitor: calculates the performance metrics (e.g. auc or rmse) for the model using ground truth. Compares against the thresholds (from the DMN) to determine the status.

   3. Ethical Fairness: calculates the maximum and minimum proportional parity for each protected class and compares the max and min values against the thresholds (from the DMN) to determine the status.

   4. Data Drift: calculates the p-value from a kolmogorov_smirnov test for each feature and compares the max value against the thresholds (from the DMN) to determine the status.

   5. Output Integrity: determines that all input records received a corresponding output inference/score using a unique identifier to ensure the model produced the appropriate output for all inputs.

   6. Concept Drift: calculates the p-value from a kolmogorov_smirnov test for the output score column(s) and compares the max value against the thresholds (from the DMN) to determine the status.

6. Daily Inferences: the count of inferences processed by the given Business model over the period. This information is calculated by the “Dashboard Model” that is run regularly for each Business model that is “in production.”

Issues Section

The bottom section provides insights into the list of open issues by Business Unit as well as the breakdown of the issue types:

1. Issues by Business Unit: displays the count of issues for each day over the last 30 days, grouped by Business Unit. This field is calculated by the “Dashboard Model” for each Business Model in production (in the “notificationsTimelineYTD” section of the Dashboard model test result). Upon page load, the ModelOp Center UI dynamically aggregates all open priority issues across all models and displays the count for each day, grouped by Business Unit.

2. Issues by Type: displays the breakdown of issues by issue type (in percentage form). This field is calculated by the “Dashboard Model” for each Business Model in production (in the “notificationsGroupedByTypeYTD” section of the Dashboard model test result). Upon page load, the ModelOp Center UI dynamically aggregates all issue types across all models and displays the aggregate percentage breakdown by issue type.

Running the Dashboard

Scheduled Runs

By default, the “Dashboard model” runs on a regular basis that is triggered by a built-in scheduler.

To View the Current Schedule

1. Select “Monitors” from the main menu

2. Select the “Scheduler” monitor from the “Featured Monitors” section

3. Click the most recent snapshot from “Snapshots” section:

4. Click on the “Monitoring” tab to view the current schedule:

To Modify the Current Schedule:

1. Select the pencil icon in the Scheduler details:

2. Select “Schedules” step:

3. Click on the first schedule item to view the details of that schedule:

4. Make the modifications to the schedule, using the “Advanced” tab (default for the Dashboard model) OR the “Standard” option:

   1. Advanced (cron expression) Option:

   2. Standard Option: selecting the “Minutes”, “Hourly”, “Daily”, “Weekly”, “Monthly”, or “Yearly” tabs

Run On-demand (UI)

To run a monitor on-demand from the UI:

1. Select “Monitors” from the main menu

2. Select the “Scheduler” monitor from the “Featured Monitors” section

3. Click the most recent snapshot from “Snapshots” section:

4. Click on the “Monitoring” tab to view the current schedule:

5. Click the “play” icon to run the “Dashboard model” manually:

6. A request to run the “Dashboard model will be made:

7. Once the “Dashboard model” starts, the User is directed to the “Jobs” page to view the status of the “Dashboard Model” job runs:

   1. Note that a “Dashboard model” job will be created for each business model that is “deployed in production” (see Dashboard Terminology section above)

Dashboard Model Execution Process Details

Once the scheduler triggers the Dashboard Model run, the corresponding Dashboard MLC will be initiated (by default, this MLC is called “Dashboard Process“). The sequence of events include:

1. Get a list of all Business Models that are “deployed in production” (see Dashboard Terminology section above)

2. For each Business Model that is “deployed in production”, the MLC will:

   1. The process will take the data and other assets from the Business Model and pass them to the “Dashboard model” to produce all of the metrics for the given Business Model

   2. After the job(s) is complete a test result (including the heatMap) will be generated for the Business Model.

   3. If a DMN with the name "dashboard_model.dmn" is found on the dashboard model assets, it is used to produce the “heatmap” items and other pass/fail metrics for the Business Model.

      1. If there are failures according to the thresholds, a notification in ModelOp Center will be produced with the failures.

   4. If there are errors in executing the “Dashboard model”, a notification in ModelOp Center will be produced.

These steps can be summarized in the following Model Life Cycle (MLC):

Dashboard Monitor Results and Notifications

Default Output

For a given Business Model, the Dashboard Test Results are listed under the Test Results table:

View an Individual Dashboard Test Result

Click on the “View” icon will display the Test Result details, including the Test Result notifications, graphical output, and raw output:

1. Test Result Notifications: provides a list of related notifications. This could include:

   1. Execution Errors: when a given “Dashboard model” run could not execute successfully.

      1. Typically this is due to a system issue, such as the ModelOp runtime used to run the “Dashboard model” had an issue.

      2. To troubleshoot, click on the “Jobs” tab of the Business Model snapshot. Click on the latest “Dashboard model” job run to view the logs from the Dashboard model job run.

   2. Individual Monitor Issue: when a given monitor (e.g. a monitor that calculates drift) had did not run successfully or had to be skipped:

      1. Typically this is due to missing inputs for that given monitor

      2. To troubleshoot, click on the “Jobs” tab of the Business Model snapshot. Click on the latest “Dashboard model” job run to view the logs from the Dashboard model job run.

2. Graphical Test Results: shows a table view of all the output metrics from the Default Dashboard model, which includes the following items by default:

   1. Numerical Metrics: Business Value, Data Drift, Statistical Performance, Stability, Ethical Fairness, Inferences:

   2. Heatmap: contains the list of items that appear in the “heatmap” and their status (typically Green, Yellow, Red):

3. Raw Test Results: contains the detailed list (json) of all output from the “Dashboard model”:

Installing the Dashboard Model - Initial Install & Setup

By default, ModelOp Center will create and configure the default “Dashboard model” during ModelOp Center installation. Below are more details on this process and troubleshooting steps, as required.

Dashboard Model Installation

To configure the dashboard the following steps need to be taken:

• Add repository attributes to model manager application.yaml (See below for example)

• Compose the changes (using manifest compose) to the manifest files making sure to include --ootb latest and --scheduler latest flags

• Redeploy the changes (using install deploy)

The list of the repositories can be multiple models and for each provide the following attributes:

• repositoryRemote: The git clone URL

• repositoryBranch: the branch of the git repository

• createBaseSnapshot: (optional) if “true” it will create a snapshot after finishing the import

Below is an example that would be placed in moc-builder/enviornment/[ENV_NAME]/data/model-manager/application.yaml

(Note: Line 7 should be replaced with your the a private repository value in order to have proper permissions)

You will need to restart the Model-Manage for change to take effect

Dashboard Scheduler

Upon installation of the environment, the default MLC’s should be loaded in the ModelOp Center instance, two of which are referring to the Executive Dashboard.

• Dashboard Process - Is executed on schedule (or on demand) to generate Dashboard test results.

• Dashboard Init Scheduler - Provides an easy auto-configured schedule out of the box.

Dashboard Scheduler Automated Setup

This MLC runs automatically only once in the lifetime of the environment and is meant to provide an easy to auto-configure schedule for the Dashboard Process out of the box. The result of this execution can be observed on the “Scheduler” model in the “Monitoring” tab where a new associated should be present.

Dashboard Scheduler Manual Setup

Alternatively, if you want to manually add a schedule for the Dashboard Process, the following steps are pre-configured if the “Dashboard Init Scheduler” MLC was successfully run:

1. Navigate to the “Scheduler” model, and select the latest snapshot (or create one if none present).

2. Go to the “Monitoring” tab and click on “Add” a “Monitor”.

3. Select the “Scheduler” model and skip until the “Schedules” step.

4. Add a Schedule with the signal name com.modelop.mlc.definitions.Signals_DASHBOARD_MODEL and the desired repetition.

Dashboard Scheduler Troubleshooting

If the “Scheduler” model is missing, please look at the “environment install” instructions above to correctly add the “scheduler” flag during install.

If the “Scheduler” model was missing during first ever startup of the environment it will not automatically run again. If it is desired to re-run the automatic setup of the dashboard please look at the “Model Lifecycle list” and verify that the “Dashboard Init Scheduler” is present. If missing please reach out to a ModelOp representative and load into the environment.
To run the automatic scheduler again please run the following command, make sure to provide the correct gateway url and the oauth2 token if the environment requires one.

If the “Dashboard Init Scheduler” MLC was present but was not able to fully run (most likely because the “Scheduler” model was not present) it is possible to pick up where it left off after successfully importing the model.

Look into the active running instances of the “Dashboard Init Scheduler” MLC process and verify if there are any

After retrying (if the issue has been fixed) we can finally observe that the “Scheduler” model has a new association with the schedule pre-configured as it was intended originally.

Configuring the Dashboard Model

Dashboard Monitor Assumptions

To run the “Dashboard model”, there are a few requirements:

1. A Dashboard model and related “batch deployment of the Dashboard model” have been created (a default version is created with ModelOp Center installation)

2. A Dashboard scheduler has been created (a default version is created with ModelOp Center installation)

3. A Dashboard model threshold file (e.g. dashboard_model.dmn) is present on the “Dashboard model”

4. One or more Business Models are configured correctly (see the Included Monitors section for more details on prerequisites):

   1. Contains an extended schema asset that specifies the appropriate fields for drift, labeled fields, score fields, etc.

   2. Contains the requisite model metadata for business value calculation.

   3. “Baseline” data asset present

   4. Production “Comparator” data asset present

   5. Business Model is “deployed in production”

Included Monitors

The default “Dashboard model” contains a number of individual monitors that are commonly used across all models across the enterprise. While the “Dashboard model” can be customized per the needs of the business, the below tables provide details of the default monitors and metrics calculated, as well as the inputs required:

Business KPI & Inferences

Heatmap Monitors

Configuring Dashboard Thresholds

Defining Thresholds for the Dashboard

As mentioned in the Monitoring Concepts article, ModelOp Center uses decision tables to define the thresholds within which the model should operate. This applies for the “Dashboard Model” as well. However, the decision table for the “Dashboard Model” uses a few specific features to populate the UI dynamically. For this tutorial, we will leverage the default dashboard_model_demo.dmn decision table. Specifically, this decision table ensures that all typical metrics for example classification models (e.g. German Credit Model) are within specification.

• Decision Table Inputs:

  • The input column names on the decision tables can be any of the element names in the “Dashboard Model” model test results (see Running the Dashboard Model section):

  • The input column values for the Dashboard Model are typically a specific value for the given metric OR the MAX or MIN of a given metric across all features or characteristics (e.g. for Data Drift, the max p-value for a KS test across all input features). Using MAX and MIN thresholds allows for providing a consistent way to evaluate metric thresholds across all features for a given model (as opposed to having to define thresholds for each individual feature)

• Decision Table Output:

  • “monitor_name”: the category for the specific metric that is being considered. A Customer can choose any categories they want by simply filling in their desired category in this decision table column with the specific column name called “monitor_name”

  • “color”: the resulting status color that will be displayed in the Executive Dashboard “heatmap” for that “monitor name” based on the thresholds defined in the decision table inputs. Using this fictitious example, if the max p-value for data drift is greater than 2, then the Executive Dashboard heatmap will display a “Red” value for the “Data Drift” monitor category.

Uploading Dashboard Thresholds Changes

Modifications to the “Default Dashboard” thresholds can be made by updating the “dashboard_model.dmn” file in the “Default Dashboard” git repository. To find the backing git repository for the “Default Dashboard” model:

1. Navigate to Monitors from the main menu

2. Select the “Default Dashboard” monitor from the list

3. Select the “Repository” tab. The git repository URL and branch are listed here:

4. To make the changes to the “Default Dashboard” thresholds, simply check in the relevant changes to the “dashboard_model.dmn” into the git repository and branch listed in the prior step.

5. Once ModelOp Center performs its regular git sync (or the User selects the “Sync GIT” option in the Repository tab of step 3, the updated “dashboard_model.dmn” file should be brought into ModelOp Center.

6. Confirm by selecting the “Assets” tab of the “Default Dashboard” monitor and click on the “dashboard_model.dmn” file to get more details:

7. Select the “view source” icon to look for the changes for confirmation.