Page Comparison

...

1. Model Submitted - a deployable model object has been created, which is a snapshot in time of the model with the typical goal of moving the model into production. Clicking “Create New Snapshot” in the Model Details page triggers the start of this MLC process.

2. Training - based on a metadata flag, a Training Job can be initiated to train the model. A service task automatically polls checking for the Training Job to finish before proceeding.

3. Testing - based on a metadata flag “run_test” (read from a snapshot level tag), an automated, reproducible Metrics Job executes, and then the results are persisted in ModelOp Center.

4. Approval Based on Test Results - based on a metadata flag “jira” (read from a snapshot level tag), the previous test results are analyzed if a DMN file is associated with the model. The details of the model, including all of the core information about the model, the changes to the model and the test results, are passed on to the reviewer on the Jira ticket.

5. Model Deployment - the MLC Process finds a ModelOp Runtime that is tagged with the appropriate tag, and attempts to deploy the model onto that runtime. Or if the matching runtime has no endpoints defined, then the model will be deployed as batch, leaving it ready for scoring on the matching runtime at the time of batch execution.

6. Error handling - when models are rejected, errors occur running the tests, or the model fails to deploy, the process creates Jira tasks to review the reasons for failure so they can take the appropriate actions.

   1. A rejection Jira ticket is created with the details for the reviewer if:

      1. We intended to run a Metrics Test Job (step 3) the model is missing test data

      2. The job execution failed

      3. The analyzed test results didn’t pass the DMN criteria

      4. The generated Jira review was rejected

   2. An error Jira ticket is created if other general errors are found during the process of deployment.

...

Run Back Test MLC Process

The Cron Triggered Run Back Test is a simple monitor that runs a test against a new set of labeled data for a given model.

...

1. Start Event - a triggered signal event initiates the monitor. This signal (com.modelop.mlc.definitions.Signals_Run_Back_Test_Jira or / com.modelop.mlc.definitions.Signals_Run_Back_Test_ServiceNow) can be triggered by a rest API providing the variables used during the process.

2. Get model - based on the MODEL_ID signal variable, the process will fetch the snapshot

3. Get data - Based on the variables provided using the signal, the input/output will be decided. If the signal has INPUT_FILE the process will use as input to the job or find an asset with TEST_DATA role on the model. Similarly, if OUTPUT_FILE is provided with the signal, it will be used for storing the job’s output. Otherwise, it will create an embedded output file and use it for the job.

4. Run and Analyze test - runs a Metrics Test batch job to evaluate the model with the new data. Based on the results of the test, if the model has an associated DMN file, it will be used to determine success criteria.

5. Test Passed - generates a notification stating that the test passed

6. Test Failed - if test fails a Jira/ServiceNow ticket is created. The details of the model, including all of the core information about the model, the changes to the model and the test results with failure, are passed on to the reviewer on the Jira/Servicenow ticket.

7. Error Handling - The following are the scenarios if any error or exception occurs in the process

a. If there is an exception while running a test job a jiraJira/servicenow Servicenow review ticket is created and all the exception details are passed on to the reviewer.

...