Page Comparison

...

You can monitor models using MLC Processes by automatically running Model Batch Jobs and Tests on a model. You can run Batch Jobs on a schedule or based on new labeled or ground truth data becoming available. For example, you can run a Batch Metrics Job using the Cron Triggered Run Back Test MLC to calculate the statistical performance and/or determine if the model has started to produce ethically biased predictions, and then use decision criteria to determine which action to take. A common pattern is to generate an alert into ModelOp Center for the ModelOp Support Team to triage.

...

The Cron Triggered Back Test is a simple monitor that runs a test against a new set of labeled data for a given model.

...

1. Start Event - a triggered signal event initiates the monitor. This signal (com.modelop.mlc.definitions.Signals_MODEL_BACK_TESTRun_Back_Test_Jira or com.modelop.mlc.definitions.Signals_Run_Back_Test_ServiceNow) can be triggered by a rest API providing the variables used during the process.

2. Get model - based on the MODEL_ID signal variable, the process will fetch the snapshot

3. Get data - Based on the variables provided using the signal, the input/output will be decided. If the signal has INPUT_FILE the process will use as input to the job or find an asset with TEST_DATA role on the model. Similarly, if OUTPUT_FILE is provided with the signal, it will be used for storing the job’s output. Otherwise, it will create an embedded output file and use it for the job.

4. Run and Analyze test - runs a Metrics Test batch job to evaluate the model with the new data. Analyze test results - based Based on the results of the test, if the model has an associated DMN file, it will be used to determine success criteria.General error and failure handling - if certain conditions are not met or if errors happen during the process Jira notifications

5. will happen Test Passed - generates a notification stating that the test passed

6. Test Failed - if test fails a Jira/ServiceNow ticket is created. The details of the model, including all of the core information about the model, the changes to the model and the test results with failure, are passed on to the reviewer on the Jira/Servicenow ticket.

7. Error Handling - The following are the scenarios if any error or exception occurs in the process

a. If there is an exception while running a test job a jira/servicenow review ticket is created and all the exception details are passed on to the reviewer.

b. If there is any exception or error occurs during the process a notification is generated with a failure reason to notify the user.

For more details on how to run this MLC see the article on triggering metrics tests.

Run Batch Model Job MLC Process

...