ModelOp Runtime Test Utility

Owned by Dave Trier
Last updated: May 30, 2023 by Nate Bradac (Unlicensed)
6 min read

ModelOp Center provides a CLI-based “test utility” to allow users to test their model within a ModelOp Runtime without officially registering the model with ModelOp Center. This is particularly useful if there is complex code in your model, custom integrations or data pipelines, or custom dependencies that need to be tested for compatibility.

Introduction

ModelOp CLI provides runtime test utility under iterative command. The utility provides users to provide a way to test the model code along with the assets using local files on the ModelOp runtime.

CLI Installation

Please follow the steps mentioned here to download and set up the ModelOp CLI : Install CLI

Interacting directly with a runtime in 'secured' mode via the iterative command requires admin credentials.

Overview of commands

The help for the moc commands is presented in Linux man page format.

  • Type help to get general information about command-line man pages.

  • Type help <command> to get information about a specific command.

The following moc commands are available under iterative command:

  • iterative

    • engine

    • job

    • logs

    • model

Command details

iterative

Synopsis

moc iterative [--help | --version] moc iterative <sub-command>

Description

User moc iterative command to test model source code and assets on ModelOp Runtime. moc iterative has sub-commands to run different types of jobs and deploy the model as REST. Before using any commands, the CLI needs to know which engine to use for the commands. To set the target engine for the command please see the engine use command. Description for the sub-commands is as shown below.

engine

Synopsis

moc iterative engine [--help] moc iterative engine <command> [--help] moc iterative engine reset [--help] moc iterative engine score [--help] moc iterative engine status [--help] moc iterative engine use <engine URL|name> [flags]

Description

Use moc iterative engine commands to interact with the ModelOp runtime. It has the following sub-commands that can be used:

reset

Description

Use this command to reset the target runtime set using the moc iterative engine use command.

Options

The command does not take any options

Arguments

The command does not take any arguments

Examples

moc iterative engine reset

score

Description

Use this command to send records for scoring to the target runtime set using the moc iterative engine use command.

Options

--input

Specify a file to use as the source of the input records. When not set, input records are read from stdin.

Arguments

The command does not take any arguments

Examples

status

Description

Use this command to know the status of the target runtime set using the moc iterative engine use command.

Options

The command does not take any options

Arguments

The command does not take any arguments

Examples

use

Description

Use this command to set the target ModelOp runtime. All the commands under the moc iterative commands are run to the runtime set during this time. Set the target runtime using a URL pointing to ModelOp runtime engine or a name of a known engine set up using the moc init command. If the engine is behind a secured gateway, use --token flag to provide a JWT token that can be used for the authentication.

Options

--token

JWT token for authentication (If runtime is behind secured gateway)

Arguments

engine URL | name

Either a URL pointing to ModelOp Runtime or the name of a known runtime (set via moc init)

Examples

job

Synopsis

Description

Use moc iterative job commands to create and run different jobs on ModelOP runtime.

create

Description

Run different types of jobs on ModelOp runtime. The command allows the creation of 3 types of jobs: batch job, test job (metrics job), and training jobs. The options and arguments are shared between commands to run these 3 types of jobs.

The command requires the path to the primary source code as the FIRST argument. The primary source code file should have a tagged function depending on the type of job to run. (# modelop.score for batch job, # modelop.train for training job and # modelop.metrics for test job). Additional assets required by the model (additional source code files, model-binary, etc.) can be provided with the subsequent arguments. Optionally, a path to the directory can be provided in addition to the path for the primary source code file. The CLI will process each file in that directory recursively and add assets to the model during which any duplicate files will be ignored (matching the same path and filename).

NOTE: make sure to structure source code files in the same way required by the model as ModelOp runtime needs the relative path of the additional source code files to the primary source code file.

If the model schemas were not part of the directory, schemas can be provided by using --schema-in and --schema-out flags.

The input file(s) to the job can be provided using the --input flag and the Output file(s) can be provided using the --output flag. The output file is not required to be created beforehand. If the output file already exists, the output of the job will be written to that file. Otherwise, a new file will be created at the given path and the results will be stored in the newly created file.

Options

--input

Provide input for the job. The flag can be used multiple times to provide multiple input files.

--logs

Print logs for the job

--output

Specify output location for the job. The flag can be used multiple times to provide multiple output file locations.

--schema-in

Provide input schema for the model. The flag can be used multiple times to provide multiple schemas.

--schema-out

Provide output schema for the model. The flag can be used multiple times to provide multiple schemas.

Arguments

batchjob

Create a model batch scoring job (MODEL_BATCH_JOB)

testjob

Create a model batch metrics job (MODEL_BATCH_TEST_JOB)

trainingjob

Create a model training job (MODEL_BATCH_TRAINING_JOB)

primary sourcecode

Provide the path to the primary source code file

Examples

logs

Synopsis

Description

Print logs for the target runtime.

Options

-f, --follow

Specify if the logs should be streamed

-t, --tail

Lines of the recent log file to display. (default 100)

Arguments

The command does not take any arguments

Examples

model

Synopsis

Description

Use the command to deploy the model to ModelOp runtime as REST.

deploy

Description

Create and deploy a model in ModelOp runtime as REST. The command requires the path to the primary source code as the FIRST argument. The primary source code file should have a tagged function depending on the type of job to run. (# modelop.score for batch job, # modelop.train for training job and # modelop.metrics for test job). Additional assets required by the model (additional source code files, model-binary, etc.) can be provided with the subsequent arguments. Optionally, a path to the directory can be provided in addition to the path for the primary source code file. The CLI will process each file in that directory recursively and add assets to the model during which any duplicate files will be ignored (matching the same path and filename).
NOTE: make sure to structure source code files in the same way required by the model as ModelOp runtime needs the relative path of the additional source code files to the primary source code file.

If the model schemas were not part of the directory, schemas can be provided by using --schema-in and --schema-out flags.

Options

--schema-in

Provide input schema for the model. The flag can be used multiple times to provide multiple schemas.

--schema-out

Provide output schema for the model. The flag can be used multiple times to provide multiple schemas.

Arguments

primary sourcecode

Provide the path to the primary source code file

Examples