Introduction

Organizations need visibility into how frequently models are being used, and to keep track of distributions of input and output data. ModelOp Center provides volumetric monitors out-of-the-box (OOTB) but also allows you to write your own custom volumetric metrics to monitor your model. The subsequent sections describe how to add a volumetric monitor, assuming an OOTB monitor.

Adding Volumetric Monitors

As background on the terminology and concepts used below, please read the Monitoring Concepts section of the Model overview documentation.

To add volumetric monitoring to your business model, you will add an existing “Monitor” to a snapshot (deployable model) of the business model under consideration. Below are the steps to accomplish this. For tutorial purposes, these instructions use all out-of-the-box and publicly available content provided by ModelOp, focusing on the German Credit Model and its related assets.

The "German Credit Data" dataset classifies people described by a set of attributes as good or bad credit risks. Volumetric monitoring can be used to govern the model in various ways including verifying that the number of input records match the number of output records and matching identifiers in input records to output records.

Associate a Monitor to a Snapshot of a Business Model

In MOC, navigate to the business model to be monitored. In our case, that’s the German Credit Model.
Navigate to the specific snapshot of the business model. If no snapshots exist, create one.
1. On the Monitors widget click on + Add
2. Search for (or select) the Volumetrics Monitor: Summary from the list of OOTB monitors.
3. Select a snapshot of the monitor. By default, a snapshot is created for each OOTB monitor.
4. On the Input Assets page, you’ll notice that the only asset that is required is sample data. This is because this volumetrics monitor computes metrics on 1 dataset only, and thus does not do a comparison to a baseline/reference dataset. For our example, select df_sample_scored.json as the Sample Data Asset. Since the file is already an asset of the business model, we can find it under Select Existing.
5. If the business model does not have a .dmn asset, the user may upload on from a local directory during the monitor association process. More on thresholds and decision tables in the next section.
6. The last step in adding a monitor is adding an optional schedule. To do so, click on ADD A SCHEDULE. The Schedule Name field is free-form. The Signal Name field is a dropdown. Choose a signal that corresponds to your ticketing system (Jira, ServiceNow). Lastly, set the frequency of the monitoring job. This can be done either by the wizard or by entering a cron expression.
7. On the Review page click SAVE.

Monitoring Results and Notifications

Sample Standard Output of Performance Monitors

The output of the performance monitoring job can be viewed by clicking on the monitor from “Model Test Results” as shown in the previous section. In this section, you can view the results in a graphical format or in the raw format.

Raw JSON:

{
    "volumetrics": [
        {
            "test_name": "Summary",
            "test_category": "volumetrics",
            "test_type": "summary",
            "test_id": "volumetrics_summary",
            "values": {
                "numerical_summary": {
                    "DURATION_MONTHS": {
                        "count": 200.0,
                        "mean": 21.54,
                        "std": 12.0755,
                        "min": 6.0,
                        "25%": 12.0,
                        "50%": 18.0,
                        "75%": 24.0,
                        "max": 60.0
                    },
                    "CREDIT_AMOUNT": {
                        "count": 200.0,
                        "mean": 3394.795,
                        "std": 2790.5621,
                        "min": 343.0,
                        "25%": 1407.5,
                        "50%": 2411.5,
                        "75%": 4219.75,
                        "max": 14555.0
                    },
                    "INSTALLMENT_RATE": {
                        "count": 200.0,
                        "mean": 2.88,
                        "std": 1.1455,
                        "min": 1.0,
                        "25%": 2.0,
                        "50%": 3.0,
                        "75%": 4.0,
                        "max": 4.0
                    },
                    "PRESENT_RESIDENCE_SINCE": {
                        "count": 200.0,
                        "mean": 2.785,
                        "std": 1.1556,
                        "min": 1.0,
                        "25%": 2.0,
                        "50%": 3.0,
                        "75%": 4.0,
                        "max": 4.0
                    },
                    "AGE_YEARS": {
                        "count": 200.0,
                        "mean": 34.1,
                        "std": 10.2393,
                        "min": 20.0,
                        "25%": 26.0,
                        "50%": 32.0,
                        "75%": 39.0,
                        "max": 65.0
                    },
                    "NUMBER_EXISTING_CREDITS": {
                        "count": 200.0,
                        "mean": 1.405,
                        "std": 0.6106,
                        "min": 1.0,
                        "25%": 1.0,
                        "50%": 1.0,
                        "75%": 2.0,
                        "max": 4.0
                    },
                    "NUMBER_PEOPLE_LIABLE": {
                        "count": 200.0,
                        "mean": 1.17,
                        "std": 0.3766,
                        "min": 1.0,
                        "25%": 1.0,
                        "50%": 1.0,
                        "75%": 1.0,
                        "max": 2.0
                    }
                },
                "categorical_summary": {
                    "CHECKING_STATUS": {
                        "count": 200,
                        "unique": 4,
                        "top": "A14",
                        "freq": 81
                    },
                    "CREDIT_HISTORY": {
                        "count": 200,
                        "unique": 5,
                        "top": "A32",
                        "freq": 111
                    },
                    "PURPOSE": {
                        "count": 200,
                        "unique": 10,
                        "top": "A43",
                        "freq": 57
                    },
                    "SAVINGS_ACCOUNT": {
                        "count": 200,
                        "unique": 5,
                        "top": "A61",
                        "freq": 118
                    },
                    "PRESENT_EMPLOYMENT_SINCE": {
                        "count": 200,
                        "unique": 5,
                        "top": "A73",
                        "freq": 64
                    },
                    "DEBTORS_GUARANTORS": {
                        "count": 200,
                        "unique": 3,
                        "top": "A101",
                        "freq": 182
                    },
                    "PROPERTY": {
                        "count": 200,
                        "unique": 4,
                        "top": "A123",
                        "freq": 68
                    },
                    "INSTALLMENT_PLANS": {
                        "count": 200,
                        "unique": 3,
                        "top": "A143",
                        "freq": 160
                    },
                    "HOUSING": {
                        "count": 200,
                        "unique": 3,
                        "top": "A152",
                        "freq": 141
                    },
                    "JOB": {
                        "count": 200,
                        "unique": 4,
                        "top": "A173",
                        "freq": 125
                    },
                    "TELEPHONE": {
                        "count": 200,
                        "unique": 2,
                        "top": "A191",
                        "freq": 125
                    },
                    "FOREIGN_WORKER": {
                        "count": 200,
                        "unique": 2,
                        "top": "A201",
                        "freq": 191
                    },
                    "GENDER": {
                        "count": 200,
                        "unique": 2,
                        "top": "male",
                        "freq": 128
                    }
                }
            }
        }
    ]
}

Volumetrics Monitors Details

Model Assumptions

Business Models considered for volumetric monitoring have a few requirements:

An extended schema asset for the input data.
Input data contains at least one numerical column or one categorical column.

Model Execution

During execution, volumetric monitors execute the following steps:

init function extracts extended input schema (corresponding to the BUSINESS_MODEL being monitored) from job JSON.
Monitoring parameters are set based on the schema above. numerical_columns and categorical_columns are determined accordingly.
metrics function runs a summary volumetrics test.
Test result is returned under the list of volumetrics tests.

For a deeper look at OOTB volumetric monitors, see the GitHub READMEs:

Next Article: Model Governance: Standard Model Definition >