Operationalizing Models: Batch

Owned by Dave Trier
Jun 15, 2022
9 min read

Batch Job Overview

Batch Job Types

A Batch Job is an execution of a model against a set of input records for scoring or for testing. You can build Batch Jobs into an MLC Process, execute a job manually from the Command Center, or run a Batch Job from the command line. The main types of Batch Jobs are detailed in the following table.

Type

Description 

Scoring Job

Executes the Scoring Function to yield predictions for each of the records in the input data. This can be used for conducting testing or for production batch scoring jobs.

Metrics Job

Executes the Metric Function against labeled test data. This yields efficacy metrics and/or bias detection and interpretability metrics.

Training Job

Executes the Training Function to train or re-train a model. The output is typically a trained model artifact or other type of attachment.

Batch Job Scenarios

You can use Batch Jobs for several different scenarios in a Model’s Life Cycle as detailed in the following table.

Scenario

Job Type

Description

Scenario

Job Type

Description

Testing a Model

Scoring Job

Use the Scoring Job to score test data so that you can conduct functional, performance, or system testing of the model execution code. You can enable or disable schema checking during this test.

Model Back-Test/Evaluation

Metrics Job

The Metrics Job executes the Metrics Function against labeled test data to generate evaluation metrics such as F1, a Confusion Matrix, ROC Curve, AUC, etc. See Model Model Efficacy Metrics and Monitoring for more information.

Ethical Fairness Detection

Metrics Job

Use the Metrics Job to run the Metrics Function against labeled data to generate metrics that detect ethical fairness. See Model Governance: Bias & Interpretability for more information.

Re-Training/Refresh

Training Job

When new labeled data is available, use a Training Job to create a new trained model artifact. This can be automated in an MLC Process. See Model Lifecycle Manager: Automation for more information.

Input & Output Data Sets for Batch Jobs

Batch Jobs require input data in order to run. You can upload your input data, provide a reference to an S3 or HDFS location for your data, or specify a SQL statement to define your input data set. These options are available via the ModelOp Center UI, MLC, CLI, or API.

Manually Create a Batch Job in the Command Center

Initiate Job Creation

Jobs can be created from 2 locations:

  1. Within the “Jobs” tab of a specific Model Snapshot:

  2. From the Jobs list page:

 

Job Creation Wizard

After clicking the “Create Job”, the “Job Creation Wizard” will be opened which will guide the user through completing the following:

  1. Select the model for which you want to run a batch job.

 

2. Select the specific snapshot (version) for your batch job.

 

3. Propose a Runtime for your batch job, or elect to allow an MLC to find the best runtime to run the job

 

4. Choose the type of Job (Training, Scoring, Metrics).

 

5. Optionally select to use input and/or output schemas for the job.

 

6. Add Input Assets, either by uploading, providing a file reference to an S3 or HDFS location, or provide a SQL statement.

 

7. Specify the Output Asset(s), either by providing a file reference to an S3 or HDFS location, provide a SQL statement, or specify to embed the output.

 

8. Review the Job information and select “Run Job” when ready.

 

 

Create a Batch Job from the CLI

  1. Install the ModelOp CLI if it is not already installed. See the ModelOp CLI Reference for install instructions.

  2. Type moc job create [batchjob | testjob | trainingjob] <deployable-or-deployed-model-uuid> input.json output.json Where:

    1. batchjob is a Scoring Job as described earlier in this article

    2. testjob is a Metrics Job as described earlier in this article

    3. trainingjob is a Training Job as described earlier in this article

    4. deployable-or-deployed-model-uuid is the uuid of a model already registered with the ModelOp Center (see the ModelOp CLI Reference for how to find these uuids)

    5. input.json is the name of the data set to run the job against

    6. output.json is the name of the output file

  3. Type moc job result <uuid> where <uuid> is the unique identifier generated by the command in the previous step. If the output file is embedded, the results are displayed in the terminal. If the job utilizes an external file asset (S3) for the output, then the results will yield a link to the S3 object where the results are placed.

 

Operationalize a Model - Batch Deployment in a ModelOp Runtime

The following are the steps to deploy a model as Batch in a ModelOp Runtime:

Pre-Requisites:

Prior to any model deployment, the following steps should be conducted by the Enterprise AI Architect and ModelOps team.

  1. Runtime Environments/Stages. Define the Runtime Environments/Stages required for operationalizing models for a given team and/or class of models. For example, if this team requires a model to be deployed and tested in Development, then SIT, then UAT, before being promoted to Production.

  2. Operationalization MLC. The Enterprise AI Architect and/or ModelOps Engineer will create the model life cycles (MLC’s) required for operationalization models for that team and/or class of models. This operationalization MLC will take into account the technical and governance requirements as a model is operationalized, which may include running tests, obtaining approvals, promoting the model to the next higher Runtime Environment (e.g. from Dev to QA).

Note, each Business Unit / Team may have different requirements and therefore these steps may be updated or repeated as the overall program progresses.

Steps:

Prepare Runtimes.

  1. Identify the target Runtimes across the requisite Environments. For each target Runtime, complete the following:

    1. Add “Environment/Stage Tags”. Based on the environments/stages required (see pre-requisites), add the necessary “environment/stage tag” to the runtime.

      1. Example: add a “DEV” tag to the Runtime in their development environment, an “SIT” tag to the Runtime in their SIT environment, a “UAT” tag to the Runtime in their UAT environment, and ultimately a “PROD” tag to the Runtime in their Prod environment

    2. Add “Model Service Tags”. The Model “Service” tag will be used to identify that this specific runtime is designed to be a target runtime for that particular model. Add the appropriate “Model Service Tag” to the runtime.

      1. Example: add a “cc-fraud” Model Service Tag to the runtime for a 3rd party credit card model to the “Dev”, “SIT”, “UAT”, and “Prod” runtimes.

 

Create a Snapshot of the model.

  1. From the Production Model Inventory, open the specific model that you would like to snapshot. Select the “Create Snapshot” button from the top right.

 

2. Optionally, add a Name and Description for the snapshot. Most important, add the same “Model Service Tag” that was added to the requisite Runtimes (see above). This will ensure that the MLC will know the correct Runtimes to which the model should be deployed throughout the life cycle.

Example: add a “credit” Model Service Tag to the snapshot.

 

 

3. Enter target runtime selection criteria by entering in text for the “Tags” section or enter a “Runtime name”. The list of matching runtimes will be filtered below based on the selections. NOTE: for Batch deployments, the runtimes should NOT have endpoints assigned to them. Ensure to select runtimes that do not have endpoints.

 

4. Review the Snapshot details. When ready, select “Create Snapshot”.

 

5. Upon successful snapshot creation, the Model Snapshot page will be opened for the newly created snapshot. Typically an MLC is triggered upon snapshot, and will be shown immediately:

 

The appropriate Model Life Cycle will automatically begin orchestrating all of the steps necessary to operationalize the model. This includes deployment of the model as REST into the designated ModelOp runtimes or the best matching available ModelOp runtime.

 

6. Note, model life cycle updates and any errors in the model life cycle execution will be populated in the “Snapshot Notifications” of the Snapshot “Overview” tab. As well, depending on the environment configuration, Jira and/or ServiceNow tickets may also be raised in the event of an issue. Tickets related to the model will appear in the “Associated Tickets” pane of the Snapshot “Overview” tab.

 

 

Operationalize a Model - Batch Deployment in a Spark Runtime

The following are the steps to deploy a model as Batch in a Spark Runtime:

Pre-Requisites:

Prior to any model deployment, the following steps should be conducted by the Enterprise AI Architect and ModelOps team.

  1. Spark Runtime Environments/Stages. Define the Spark Runtime Environments/Stages required for operationalizing models for a given team and/or class of models. For example, if this team requires a model to be deployed and tested in a Development Spark cluster/environment, then SIT, then UAT, before being promoted to Production, then Development/SIT/UAT/Production Spark Runtimes must be defined.

  2. Operationalization MLC. The Enterprise AI Architect and/or ModelOps Engineer will create the model life cycles (MLC’s) required for operationalization models for that team and/or class of models. This operationalization MLC will take into account the technical and governance requirements as a model is operationalized, which may include running tests, obtaining approvals, promoting the model to the next higher Runtime Environment (e.g. from Dev to QA).

Note, each Business Unit / Team may have different requirements and therefore these steps may be updated or repeated as the overall program progresses.

Steps:

Prepare Runtimes.

  1. Identify the target Runtimes across the requisite Environments. For each target Runtime, complete the following:

    1. Add “Environment/Stage Tags”. Based on the environments/stages required (see pre-requisites), add the necessary “environment/stage tag” to the runtime.

      1. Example: add a “DEV” tag to the Runtime in their development environment, an “SIT” tag to the Runtime in their SIT environment, a “UAT” tag to the Runtime in their UAT environment, and ultimately a “PROD” tag to the Runtime in their Prod environment

    2. Add “Model Service Tags”. The Model “Service” tag will be used to identify that this specific runtime is designed to be a target runtime for that particular model. Add the appropriate “Model Service Tag” to the runtime.

      1. Example: add a “cc-fraud” Model Service Tag to the runtime for a 3rd party credit card model to the “Dev”, “SIT”, “UAT”, and “Prod” runtimes.

Create a Snapshot of the model.

  1. From the Production Model Inventory, open the specific model that you would like to snapshot. Select the “Create Snapshot” button from the top right.

 

  1. Optionally, add a Name and Description for the snapshot. Most important, add the same “Model Service Tag” that was added to the requisite Runtimes (see above). This will ensure that the MLC will know the correct Runtimes to which the model should be deployed throughout the life cycle.

  2. Example: add a “cc-fraud” Model Service Tag to the snapshot.

 

3. For Spark models, select the “Apache Spark” Runtime type and/or specific Spark Runtimes. Enter target runtime selection criteria by entering in text for the “Tags” section or enter a “Runtime name”. The list of matching runtimes will be filtered below based on the selections.

 

4. Review the Snapshot details. When ready, select “Create Snapshot”.

 

5. Upon successful snapshot creation, the Model Snapshot page will be opened for the newly created snapshot. Typically an MLC is triggered upon snapshot, and will be shown immediately:

The appropriate Model Life Cycle will automatically begin orchestrating all of the steps necessary to operationalize the model. This includes deployment of the model as REST into the designated ModelOp runtimes or the best matching available ModelOp runtime.

 

6. Note, model life cycle updates and any errors in the model life cycle execution will be populated in the “Snapshot Notifications” of the Snapshot “Overview” tab. As well, depending on the environment configuration, Jira and/or ServiceNow tickets may also be raised in the event of an issue. Tickets related to the model will appear in the “Associated Tickets” pane of the Snapshot “Overview” tab.

 

 

Next Article: Champion/Challenger Model Comparison >