Anatomy of the ModelOp Runtime

At a high level, the ModelOp Runtime consists of 3 different components: the stream, the jet, and the manifold. Streams deal with the I/O, the jet is a wrapper around the model code itself, and the manifold is responsible for stitching everything together.

Streams

The runtime supports a variety of different streams, including REST, Kafka, File, S3, and ODBC. Input and output streams are typically configured independently, the main exception here being when we want a roundtrip REST call to return with the results from synchronously processed POSTed input.

Jet

The jet consists of two main pieces. There is some common engine code that handles communication with the underlying model runner, and then there is the language specific model runner code that is dynamically started/stopped by the jet. The engine code and the language specific model runner code communicate via a Unix domain socket.

Manifold

The manifold is responsible for managing the lifecycles of the various streams and the underlying jet that is running the model code.

Configuration Options

The engine is configured via several mechanisms. The preferred mechanism is via spring configuration through spring cloud config server. There are also a number of (deprecated) environment variables that affect the behavior, as well as an application.json file that can be configured with credentials the engine may need when interacting with external services.

Environment Variables

    - path: ""
      authenticationType: "BASIC"
      accessKeyId: "my_key"
      secretAccessKey: "my_secret"

    - path: "s3.us-east-2.amazonaws.com"
      roleArnEnvVariable: "TEST_ROLE_ARN"
      webIdentityTokenFileEnvVariable: "TEST_TOKEN_FILE"
      sessionPrefix: "testprefixABC"

  - host: "host1"
    username: "username1"
    password: "password1"

moc-builder

When leveraging moc-builder to generate a MOC environment, your environment’s config/config.json file can be updated with the desired values for the aforementioned environment variables. A subsequent compose command will use the values in the config.json when building kubernetes manifests.

application.json

{
  "sql-credentials": [
    {
      "host": "sql-server-abc",
      "username": "my_user",
      "password": "my_password"
    },
    {
      "host": "sql-other-server-def",
      "username": {"from_file": "/config/the-username"},
      "password": {"from_env": "THE_ENV_VAR_WHERE_MY_PASSWORD_IS"}
    }
  ],
  "s3-credentials": {
    "storedCredentials": [
      {
        "path": "",
        "authenticationType": "BASIC",
        "accessKeyId" : "my_key",
        "secretAccessKey" : "my_secret"
      }
    ],
    "eksServiceAccountCredentials": [
      {
        "path": "s3.us-east-2.amazonaws.com",
        "roleArnEnvVariable" : "TEST_ROLE_ARN",
        "webIdentityTokenFileEnvVariable" : "TEST_TOKEN_FILE",
        "sessionPrefix" : "testprefixABC"
      }
    ]
  }
}

The ‘sql-credentials’ section allows username and password credentials to be associated with specific sql database hosts. Credentials can be hardcoded directly in the config file, located in a file, or located in an environment variable.

The ‘s3-credentials’ section allows credentials to be hardcoded directly (under storedCredentials) or determined dynamically using the AWS STS service. The credentials are chosen based on the longest matching path.

API’s

The following are a list of commonly used runtime APIs. The provided curl commands assume the runtime is running at localhost:8003. When communicating with a runtime through a MOC gateway, the localhost:8003 will likely look like this: gateway:8090/engine-name, where engine-name is the name the runtime used when it registered with the registry via the env variable: MODELOP_ENGINE_NAME.

Reset the engine

Reset the engine, causing the current model to be unloaded, and the various endpoints reset. This will also cause the current batch job to complete immediately, and move on to the next batch job.

curl -X POST http://localhost:8003/2/engine/reset

Load a deployed model

Load the deployed model contained in the ‘my-deployable-model.json’ file.

curl -X PUT --data-binary "@my-deployable-model.json" -H "Content-Type: application/json" http://localhost:8003/api/deployedModels/{UUID}

There is an optional wait_for_init query parameter for the /api/deployedModels/{UUID} endpoint. The default value of true means to not return from the REST call until the model’s init function has completed successfully. A value of false means to return immediately once the model and the specified endpoints have been loaded, but before the init function returns.

Load a batch job

Load a batch job into the runtime.

curl -X POST --data-binary "@my-batch-job.json" -H "Content-Type: application/json" http://localhost:8003/api/modelBatchJobs/{UUID}

Delete a batch job

Delete the batch job with the specified UUID.

curl -X DELETE http://localhost:8003/api/modelBatchJobs/{UUID}

List all batch jobs

Return all the batch jobs known to the runtime.

curl http://localhost:8003/api/modelBatchJobs/

List a specific batch job

Return the batch job with the specified UUID.

curl http://localhost:8003/api/modelBatchJobs/{UUID}

Score data via the REST endpoint

Submit the contents of input.json to the runtime. Note that in this example, the timeout has been set to 100 seconds, overriding the default set by the MODELOP_ROUNDTRIP_DEFAULT_TIMEOUT environment variable.

curl -X POST --data-binary "@input.json" -H "Content-Type: application/json" http://localhost:8003/api/roundtrip/0/1?timeout=100000

Get current engine status

Retrieve information about the currently running model, attached streams, and overall state of the engine.

curl http://localhost:8003/1/job/status

Sample response:

{
  "streams": [
    {
      "slot": 0,
      "name": "rest",
      "eof": false,
      "descriptor": {
        "Transport": {
          "secured": false,
          "roundtrip": true,
          "port": 9003,
          "Type": "REST",
          "Mode": "simple"
        },
        "Schema": "$inherit",
        "Name": "rest",
        "Encoding": "JSON",
        "Description": "Application Restful Endpoint"
      }
    },
    {
      "slot": 1,
      "name": "rest_out",
      "eof": false,
      "descriptor": {
        "Transport": {
          "Type": "synthetic"
        },
        "Schema": "$inherit",
        "Encoding": "JSON"
      }
    }
  ],
  "state": "running",
  "model": {
    "source": "[model source code...]",
    "snapshots": "none",
    "slots": [
      {
        "slot": 0,
        "schema": null,
        "recordsets": false,
        "action": "action"
      },
      {
        "slot": 1,
        "schema": null,
        "recordsets": false,
        "action": null
      }
    ],
    "name": "test_model.py",
    "mtype": "python3",
    "attachments": []
  },
  "jets": [
    {
      "sandbox": "123472369",
      "pid": 55,
      "busy": false
    }
  ],
  "initialTags": [],
  "group": "modelop"
}

Model Definition Injection

Both the python and R runners support an optional parameter for the init function. Normally, the init function is defined with 0 arguments, but if a single argument is defined, the runner will populate that argument with a ‘model definition’ object. In python, this object is a dictionary. In R, this object is a list. In both cases, the object has two keys. There will be a ‘rawJson’ key that contains the unparsed json string that contains either the job or the deployedModel object in its serialized form. There will also be either a ‘job’ or a ‘deployedModel' key, the value of which will either be a deserialized job or deployedModel object. In python, the deserialized job or deployedModel will be the result of calling json.loads() on the contents of the ‘rawJson’ field. In R, it will be the result of calling the jsonlite fromJSON function.

Python:

def begin(model_def):
  print(model_def['rawJson'])
  print(model_def['job'])

R:

begin <- function(model_def) {
  print(model_def$rawJson)
  print(model_def$deployedModel)
}

To see an example of the above, go to https://github.com/merhi-odg/model_definition_test.

Adding Endpoints to the ModelOp Runtime

To add endpoints to the ModelOp Runtime:

1. Open the ModelOp Runtime to which you want to add endpoints

2. Select the “Create Endpoint” button for either the input or output endpoints

3. Select to add an endpoint, either REST or Kafka

4. Add a Name, optional Description, Encoding type, and optional port. Select “Save Endpoint”

5. Repeat for the Output Endpoint, as required.

Adding Tags to a ModelOp Runtime

1. Open the ModelOp Runtime to which you want to add tag(s).

2. In the “tags” section begin typing the tag that you would like to add:

3. Once done typing, either hit enter on your keyboard or click your mouse in any other location on the screen.

4. Select the “Update Engine” button in the top right to save changes.