ModelOp Runtime Test Utility
- Dave Trier
- Antron Gladney
ModelOp Center provides a CLI-based “test utility” to allow users to test their model within a ModelOp Runtime without officially registering the model with ModelOp Center. This is particularly useful if there is complex code in your model, custom integrations or data pipelines, or custom dependencies that need to be tested for compatibility.
Introduction
ModelOp CLI provides runtime test utility under iterative command. The utility provides users to provide a way to test the model code along with the assets using local files on the ModelOp runtime.
CLI Installation
Please follow the steps mentioned here to download and set up the ModelOp CLI : Install CLI
Interacting directly with a runtime in 'secured' mode via the iterative command requires admin credentials.
Overview of commands
The help for the moc commands is presented in Linux man page format.
Type
helpto get general information about command-line man pages.Type
help <command>to get information about a specific command.
The following moc commands are available under iterative command:
iterative
engine
job
logs
model
Command details
iterative
Synopsis
moc iterative [--help | --version]
moc iterative <sub-command>Description
User moc iterative command to test model source code and assets on ModelOp Runtime. moc iterative has sub-commands to run different types of jobs and deploy the model as REST. Before using any commands, the CLI needs to know which engine to use for the commands. To set the target engine for the command please see the engine use command. Description for the sub-commands is as shown below.
engine
Synopsis
moc iterative engine [--help]
moc iterative engine <command> [--help]
moc iterative engine reset [--help]
moc iterative engine score [--help]
moc iterative engine status [--help]
moc iterative engine use <engine URL|name> [flags]Description
Use moc iterative engine commands to interact with the ModelOp runtime. It has the following sub-commands that can be used:
reset
Description
Use this command to reset the target runtime set using the moc iterative engine use command.
Options
The command does not take any options
Arguments
The command does not take any arguments
Examples
moc iterative engine resetscore
Description
Use this command to send records for scoring to the target runtime set using the moc iterative engine use command.
Options
--input
Specify a file to use as the source of the input records. When not set, input records are read from stdin.
Arguments
The command does not take any arguments
Examples
status
Description
Use this command to know the status of the target runtime set using the moc iterative engine use command.
Options
The command does not take any options
Arguments
The command does not take any arguments
Examples
use
Description
Use this command to set the target ModelOp runtime. All the commands under the moc iterative commands are run using the runtime set during this time. Set the target runtime using a URL pointing to ModelOp runtime engine or the name of a known engine set up using the moc init command. If the engine is behind a secured gateway, use --token flag to provide a JWT token that can be used for the authentication.
Options
--token
JWT token for authentication (If runtime is behind secured gateway)
Arguments
engine URL | name
Either a URL pointing to ModelOp Runtime or the name of a known runtime (set via moc init)
Examples
job
Synopsis
Description
Use moc iterative job commands to create and run different jobs on ModelOp runtime.
create
Description
Run different types of jobs on ModelOp runtime. The command allows the creation of 3 types of jobs: batch job, test job (metrics job), and training jobs. The options and arguments are shared between commands to run these 3 types of jobs.
The command requires the path to the primary source code as the FIRST argument. The primary source code file should have a tagged function depending on the type of job to run. (# modelop.score for batch job, # modelop.train for training job and # modelop.metrics for test job). Additional assets required by the model (additional source code files, model-binary, etc.) can be provided with the subsequent arguments. Optionally, a path to the directory can be provided in addition to the path for the primary source code file. The CLI will process each file in that directory recursively and add assets to the model during which any duplicate files will be ignored (matching the same path and filename).
NOTE: make sure to structure source code files in the same way required by the model as ModelOp runtime needs the relative path of the additional source code files to the primary source code file.
If the model schemas were not part of the directory, schemas can be provided by using --schema-in and --schema-out flags.
The input file(s) to the job can be provided using the --input flag and the Output file(s) can be provided using the --output flag. The output file is not required to be created beforehand. If the output file already exists, the output of the job will be written to that file. Otherwise, a new file will be created at the given path and the results will be stored in the newly created file.
Options
--input
Provide input for the job. The flag can be used multiple times to provide multiple input files.
--logs
Print logs for the job
--output
Specify output location for the job. The flag can be used multiple times to provide multiple output file locations.
--schema-in
Provide input schema for the model. The flag can be used multiple times to provide multiple schemas.
--schema-out
Provide output schema for the model. The flag can be used multiple times to provide multiple schemas.
Arguments
batchjob
Create a model batch scoring job (MODEL_BATCH_JOB)
testjob
Create a model batch metrics job (MODEL_BATCH_TEST_JOB)
trainingjob
Create a model training job (MODEL_BATCH_TRAINING_JOB)
primary sourcecode
Provide the path to the primary source code file
Examples
logs
Synopsis
Description
Print logs for the target runtime.
Options
-f, --follow
Specify if the logs should be streamed
-t, --tail
Lines of the recent log file to display. (default 100)
Arguments
The command does not take any arguments
Examples
model
Synopsis
Description
Use the command to deploy the model to ModelOp runtime as REST.
deploy
Description
Create and deploy a model in ModelOp runtime as REST. The command requires the path to the primary source code as the FIRST argument. The primary source code file should have a tagged function depending on the type of job to run. (# modelop.score for batch job, # modelop.train for training job and # modelop.metrics for test job). Additional assets required by the model (additional source code files, model-binary, etc.) can be provided with the subsequent arguments. Optionally, a path to the directory can be provided in addition to the path for the primary source code file. The CLI will process each file in that directory recursively and add assets to the model during which any duplicate files will be ignored (matching the same path and filename).
NOTE: make sure to structure source code files in the same way required by the model as ModelOp runtime needs the relative path of the additional source code files to the primary source code file.
If the model schemas were not part of the directory, schemas can be provided by using --schema-in and --schema-out flags.
Options
--schema-in
Provide input schema for the model. The flag can be used multiple times to provide multiple schemas.
--schema-out
Provide output schema for the model. The flag can be used multiple times to provide multiple schemas.
Arguments
primary sourcecode
Provide the path to the primary source code file
Examples