...

This section describes some specific examples of MLC Processes in detail.

...

Deploy with run test and Jira MLC Process

On Model Changed Deploy with run test and Jira is an MLC Process that incorporates several of the patterns described earlier in a single MLC Process for managing the creation of a model, or changes to an existing model.

...

1. Model Submitted - a deployable model object has been created, which is a snapshot in time of the model with the typical goal of moving the model into production. Clicking Submit “Create New Snapshot” in the Model Details page triggers the start of this MLC process.

2. Training - based on a metadata flag, a Training Job can be initiated to train the model. A polling service task automatically checks polls checking for the Training Job to finish before proceeding.

3. Testing - based on a metadata flag “run_test” (read from a snapshot level tag), an automated, reproducible Metrics Job executes, and then the results are persisted in ModelOp Center.

4. Approval Based on Type Test Results - based on the metadata, you can route the model to the correct group to approve the changesa metadata flag “jira” (read from a snapshot level tag), the previous test results are analyzed if a DMN file is associated with the model. The details of the model, including all of the core information about the model, the changes to the model and the test results, are passed on to the reviewer on the Jira ticket.

5. Model Deployment - the MLC Process finds a ModelOp Runtime that is tagged with the appropriate tag, and attempts to deploy the model onto that runtime. If there are any errors during deployment, a task is created to review the errorOr if the matching runtime has no endpoints defined, then the model will be deployed as batch, leaving it ready for scoring on the matching runtime at the time of batch execution.

6. Error handling - when models are rejected, errors occur running the tests, or the model fails to deploy, the process creates human Jira tasks to review the reasons for failure so they can take the appropriate actions.

Daily Backtesting MLC Process

...

1. .

   1. A rejection Jira ticket is created with the details for the reviewer if:

      1. We intended to run a Metrics Test Job (step 3) the model is missing test data

      2. The job execution failed

      3. The analyzed test results didn’t pass the DMN criteria

      4. The generated Jira review was rejected

   2. An error Jira ticket is created if other general errors are found during the process of deployment.

Cron Triggered Back Test MLC Process

The Cron Triggered Back Test is a simple monitor that runs a test against a new set of labeled data from the night beforefor a given model.

...

1. Start Event - a timed start triggered signal event initiates the monitor every morning at a specific time.Run Test -next, a Test Job evaluates . This signal (com.modelop.mlc.definitions.Signals_MODEL_BACK_TEST) can be triggered by a rest API providing the variables used during the process.

2. Get model - based on the MODEL_ID signal variable, the process will fetch the snapshot

3. Get data - Based on the variables provided using the signal, the input/output will be decided. If the signal has INPUT_FILE the process will use as input to the job or find an asset with TEST_DATA role on the model. Similarly, if OUTPUT_FILE is provided with the signal, it will be used for storing the job’s output. Otherwise, it will create an embedded output file and use it for the job.

4. Run test - runs a Metrics Test batch job to evaluate the model with the new data.

5. Parse Test Results - parses the raw test results into a standardized format and captures the data within Model Manager for future review.

6. Exceeded Threshold - if the the business threshold is not met, an alert is sent to the support team to investigateAnalyze test results - based on the results of the test, if the model has an associated DMN file, it will be used to determine success criteria.

7. General error and failure handling - if certain conditions are not met or if errors happen during the process Jira notifications will happen

For more details on how to run this MLC see this: https://modelop.atlassian.net/wiki/spaces/DV21/pages/1865416723/Triggering+Model+Test+MLCs#Back-Test

Run Batch Model Job MLC Process

Run Batch Model Job is an MLC process that will trigger the execution of a scoring job on a batch deployed model. In order for this MLC to pick up correctly, a model must have been deployed as batch.

...

1. Start event - a triggered signal event initiates the execution. This signal (com.modelop.mlc.definitions.Signals_DEPLOYED_BATCH_JOB) should contain a variable with the TAG of the model (model service tag) to run.

2. Get Deployed Model - based on the tag given in the input signal, the process finds the most recent batch deployed model in “deployed” (active) state using the MODEL_STAGE (if provided) to match the deployment target of this execution.

3. Set inputs and outputs in order - Creates the input and output parameters from the provided signal variables.

4. Get compatible runtime - Finds the matching target runtime to run this scoring job.

5. Run job - runs the Scoring batch job to obtain model’s inference given the data.

6. Error handling - if certain conditions are not met an error is raised and Jira notification will be created.

Create & Deploy a New MLC Process Using Camunda Modeler

1. Download and install Camunda Modeler on your local machine. Go to https://camunda.com/download/modeler/

2. Create a new BPMN. Go to https://docs.camunda.org/manual/7.13/modeler/bpmn/ for a quickstart on BPMN modeling.

3. When the MLC Process is ready, select the Deploy icon to put it in the MLC Manager. Note: while the URL will be highly dependent on your environment’s exact setup, it likely uses a path such as http://<moc-base-url>/mlc-service/rest

...

...

3(a). If your target url is behind an oauth2 secured environment the Camunda Modeler will request for an authentication method. Please provide a bearer token by logging into your secured ModelOp Center.

curl --location --request POST 'http://<moc-base-url>/gocli/token' \
--header 'Content-Type: application/x-www-form-urlencoded' \
--data-urlencode 'username=<user>' \
--data-urlencode 'password=<password>'

From the above command replace the <moc-base-url> and provide its respective <user> and <password>.
Retrieve the “access_token” from the response of the command above and paste it in the “Token” field presented.

...

4. Verify that your new MLC Process is registered with MLC Manager. Go to the Command Center and click the Models icon in the left column.

...

• Operationalizing Models: Overview

• Operationalize a Model - REST

• Operationalize a Model - Batch

• Champion/Challenger Model Comparison

Next Article: Operationalizing Models: Overview >