Introduction

Organizations need visibility into how frequently models are being used, and to keep track of distributions of input and output data. ModelOp Center provides volumetric monitors out-of-the-box (OOTB) but also allows you to write your own custom metrics to monitor your model. The subsequent sections describe how to add a volumetric monitor, assuming an OOTB monitor.

Adding Volumetric Monitors

As background on the terminology and concepts used below, please read the Monitoring Concepts section of the Model overview documentation.

To add volumetric monitoring to your business model, you will add an existing “Monitor” to a snapshot (deployable model) of the business model under consideration. Below are the steps to accomplish this. For tutorial purposes, these instructions use all out-of-the-box and publicly available content provided by ModelOp, focusing on the German Credit Model and its related assets.

The "German Credit Data" dataset classifies people described by a set of attributes as good or bad credit risks. Volumetric monitoring can be used to govern the model in various ways including verifying that the number of input records match the number of output records and matching identifiers in input records to output records.

Associate a Monitor to a Snapshot of a Business Model

In MOC, navigate to the business model to be monitored. In our case, that’s the German Credit Model.
Navigate to the specific snapshot of the business model. If no snapshots exist, create one.
On the Monitoring tab, click on + Add, then click on Monitor
Search for (or select) the Volumetric Monitor: Summary from the list of OOTB monitors.
Select a snapshot of the monitor. By default, a snapshot is created for each OOTB monitor
On the Input Assets page, you’ll notice that the only asset that is required is sample data. This is because this volumetric monitor computes metrics on 1 dataset only, and thus does not do a comparison to a baseline/reference dataset. For our example, select df_sample_scored.json as the Sample Data Asset. Since the file is already an asset of the business model, we can find it under Select Existing
We will skip adding a threshold on this monitor. For examples on how to add a threshold, please visit this page.
The last step in adding a monitor is adding an optional schedule. To do so, click on ADD A SCHEDULE. The Schedule Name field is free-form. The Signal Name field is a dropdown. Choose a signal that corresponds to your ticketing system (Jira, ServiceNow). Lastly, set the frequency of the monitoring job. This can be done either by the wizard or by entering a cron expression. Note: schedules are optional; a monitor may be run on-demand from the business model’s snapshot page, or by a curl command.
On the Review page click SAVE

To run a monitor on demand, click on COPY CURL TO RUN JOB EXTERNALLY. The CURL command can then be run from the application of your choosing.

Run a Monitor On-demand (UI)

To run a monitor on-demand from the UI, navigate to the business model’s snapshot page and click the play button next to the monitor of interest. A monitoring job will be initiated, and you will be redirected to the corresponding job page once the job is created.

Schedule a Monitor DIY (CURL)

Monitors can be scheduled to run using your preferred enterprise scheduling capability (Control-M, Airflow, Autosys, etc.) While the details will depend on the specific scheduling software, at the highest level, the user simply needs to create a REST call to the ModelOp Center API. Here are the steps:

Obtain the Business Model snapshot’s UUID. This can be found, for instance, in the URL of the snapshot page, as shown in this example:
Similarly, obtain the Monitoring Model snapshot’s UUID.
Within the scheduler, configure the REST call to ModelOp Center’s automation engine to trigger the monitor for your model:
1. Obtain a valid auth token
2. Make a call (POST) to the ModelOp Center API to initiate the monitor. The endpoint is
```
<MOC_INSTANCE_URL>/mlc-service/rest/signal
```
3. The body should contain references to the Model Life Cycle (MLC) being triggered, as well as the business model and monitor snapshots, as shown below:
```
{
    "name": "com.modelop.mlc.definitions.Signals_Run_Associated_Model_Jira",
    "variables": {
        "DEPLOYABLE_MODEL_ID" : {
            "value": <UUID_of_business_model_snapshot_as_a_string>
        },
        "ASSOCIATED_MODEL_ID": {
            "value": <UUID_of_monitoring_model_snapshot_as_a_string>
        }
    }
}
```

This process is made easier by copying the CURL command provided at the last step of the monitoring wizard

The copied command will look something like this:

curl 'http://localhost:8090/mlc-service/rest/signalResponsive' -H 'Accept: application/json, text/plain, /' -H 'Content-Type: application/json' -X POST -H 'Authorization: Bearer <token>' --data-raw '{"name":"com.modelop.mlc.definitions.Signals_Run_Associated_Model_Jira","variables":{"DEPLOYABLE_MODEL_ID":{"value":"23282688-62a6-47ae-8603-16f380efca57"},"ASSOCIATED_MODEL_ID":{"value":"1dc64c1e-3634-4e2e-b37d-71d04a9ee5ef"}}}'

Monitoring Execution

Once the scheduler triggers the signal, the corresponding MLC (listening to that signal) will be initiated. The sequence of events include:

Preparing the monitoring job with all artifacts necessary to run the job
Creating the monitoring job
Parsing the results into viewable test results
Comparing the results against the thresholds in the decision table
Taking action, which could include creating a notification and/or opening up an incident in JIRA/ServiceNow/etc.

These steps can be summarized in the following Model Life Cycle (MLC)

Monitoring Results and Notifications

Sample Standard Output of a Volumetric Monitor

Monitoring Test Results are listed under the Test Results table:

Upon clicking on the “View” icon, you’ll have two options for looking at test results: graphical (under “Test Results”), and raw (under “Raw Results”).

Visual elements

Numerical Summary Table
Categorical Summary Table

Raw Results

The “Raw Results” tab shows a clickable (expandable and collapsable) JSON representation of the test results

To get a JSON file of the test results,

Navigate to the “Jobs” tab of the snapshot and click on “Details” next to the monitoring job of interest
Click on “Download File” under “Outputs”

Click here to see the contents of the file.

{
    "volumetrics": [
        {
            "test_name": "Summary",
            "test_category": "volumetrics",
            "test_type": "summary",
            "test_id": "volumetrics_summary",
            "values": {
                "numerical_summary": {
                    "DURATION_MONTHS": {
                        "count": 200.0,
                        "mean": 21.54,
                        "std": 12.0755,
                        "min": 6.0,
                        "25%": 12.0,
                        "50%": 18.0,
                        "75%": 24.0,
                        "max": 60.0
                    },
                    "CREDIT_AMOUNT": {
                        "count": 200.0,
                        "mean": 3394.795,
                        "std": 2790.5621,
                        "min": 343.0,
                        "25%": 1407.5,
                        "50%": 2411.5,
                        "75%": 4219.75,
                        "max": 14555.0
                    },
                    "INSTALLMENT_RATE": {
                        "count": 200.0,
                        "mean": 2.88,
                        "std": 1.1455,
                        "min": 1.0,
                        "25%": 2.0,
                        "50%": 3.0,
                        "75%": 4.0,
                        "max": 4.0
                    },
                    "PRESENT_RESIDENCE_SINCE": {
                        "count": 200.0,
                        "mean": 2.785,
                        "std": 1.1556,
                        "min": 1.0,
                        "25%": 2.0,
                        "50%": 3.0,
                        "75%": 4.0,
                        "max": 4.0
                    },
                    "AGE_YEARS": {
                        "count": 200.0,
                        "mean": 34.1,
                        "std": 10.2393,
                        "min": 20.0,
                        "25%": 26.0,
                        "50%": 32.0,
                        "75%": 39.0,
                        "max": 65.0
                    },
                    "NUMBER_EXISTING_CREDITS": {
                        "count": 200.0,
                        "mean": 1.405,
                        "std": 0.6106,
                        "min": 1.0,
                        "25%": 1.0,
                        "50%": 1.0,
                        "75%": 2.0,
                        "max": 4.0
                    },
                    "NUMBER_PEOPLE_LIABLE": {
                        "count": 200.0,
                        "mean": 1.17,
                        "std": 0.3766,
                        "min": 1.0,
                        "25%": 1.0,
                        "50%": 1.0,
                        "75%": 1.0,
                        "max": 2.0
                    }
                },
                "categorical_summary": {
                    "CHECKING_STATUS": {
                        "count": 200,
                        "unique": 4,
                        "top": "A14",
                        "freq": 81
                    },
                    "CREDIT_HISTORY": {
                        "count": 200,
                        "unique": 5,
                        "top": "A32",
                        "freq": 111
                    },
                    "PURPOSE": {
                        "count": 200,
                        "unique": 10,
                        "top": "A43",
                        "freq": 57
                    },
                    "SAVINGS_ACCOUNT": {
                        "count": 200,
                        "unique": 5,
                        "top": "A61",
                        "freq": 118
                    },
                    "PRESENT_EMPLOYMENT_SINCE": {
                        "count": 200,
                        "unique": 5,
                        "top": "A73",
                        "freq": 64
                    },
                    "DEBTORS_GUARANTORS": {
                        "count": 200,
                        "unique": 3,
                        "top": "A101",
                        "freq": 182
                    },
                    "PROPERTY": {
                        "count": 200,
                        "unique": 4,
                        "top": "A123",
                        "freq": 68
                    },
                    "INSTALLMENT_PLANS": {
                        "count": 200,
                        "unique": 3,
                        "top": "A143",
                        "freq": 160
                    },
                    "HOUSING": {
                        "count": 200,
                        "unique": 3,
                        "top": "A152",
                        "freq": 141
                    },
                    "JOB": {
                        "count": 200,
                        "unique": 4,
                        "top": "A173",
                        "freq": 125
                    },
                    "TELEPHONE": {
                        "count": 200,
                        "unique": 2,
                        "top": "A191",
                        "freq": 125
                    },
                    "FOREIGN_WORKER": {
                        "count": 200,
                        "unique": 2,
                        "top": "A201",
                        "freq": 191
                    },
                    "GENDER": {
                        "count": 200,
                        "unique": 2,
                        "top": "male",
                        "freq": 128
                    }
                }
            }
        }
    ]
}

Sample Monitoring Notification

Notifications arising from monitoring jobs can be found under the corresponding model test result

If a ticketing system is configured in ModelOp Center, such as Jira, a ticket will be written when an ERROR occurs (as in above), and a link to the ticket will be available next to the notification. In the example above, since there were no thresholds set on the monitor, the job automatically passed.

Volumetric Monitors Details

Model Assumptions

Business Models considered for volumetric monitoring have a few requirements:

An extended schema asset for the input data.
Input data contains at least one numerical column or one categorical column.

Model Execution

During execution, volumetric monitors execute the following steps:

init function extracts extended input schema (corresponding to the BUSINESS_MODEL being monitored) from job JSON.
Monitoring parameters are set based on the schema above. numerical_columns and categorical_columns are determined accordingly.
metrics function runs a summary volumetric test.
Test result is returned under the list of volumetric tests.

For a deeper look at OOTB volumetric monitors, see the GitHub READMEs:

Volumetrics Monitoring