Anatomy of the ModelOp Runtime

At a high level, the ModelOp Runtime consists of 3 different components: the stream, the jet, and the manifold. Streams deal with the I/O, the jet is a wrapper around the model code itself, and the manifold is responsible for stitching everything together.

Streams

The runtime supports a variety of different streams, including REST, Kafka, File, S3, and ODBC. Input and output streams are typically configured independently, the main exception here being when we want a roundtrip REST call to return with the results from synchronously processed POSTed input.

Jet

The jet consists of two main pieces. There is some common engine code that handles communication with the underlying model runner, and then there is the language specific model runner code that is dynamically started/stopped by the jet. The engine code and the language specific model runner code communicate via a Unix domain socket.

Manifold

The manifold is responsible for managing the lifecycles of the various streams and the underlying jet that is running the model code.

Configuration Options

The engine is configured via several mechanisms. The preferred mechanism is via spring configuration through spring cloud config server. There are also a number of (deprecated) environment variables that affect the behavior, as well as an application.json file that can be configured with credentials the engine may need when interacting with external services.

Environment Variables

Spring Config	Deprecated Environment Variable	Example Values	Description

Spring Config	Deprecated Environment Variable	Example Values	Description
modelop.runtime.log-level	MODELOP_LOG_LEVEL	debug, info (default), warning, error	Log level of the engine
modelop.gateway-url	MODELOP_GATEWAY_LOCATION	http://gateway:8090 gateway:8090	The host:port of the MOC gateway
spring.application.name	MODELOP_ENGINE_NAME	engine-3	The name of the runtime that will be used to identify it in the registry
spring.cloud.config.uri		http://sccs:1234	The uri of the spring cloud config server from whence to acquire configuration (MUST BE SET LOCALLY)
spring.profiles.active		profile1,profile2,profile3	The spring profiles that contain the desired configuration (MUST BE SET LOCALLY)
modelop.runtime.group	MODELOP_RUNTIME_GROUP	modelop	The group name that owns the runtime.
modelop.runtime.initial-tags	MODELOP_RUNTIME_INITIAL_TAGS	“tag1, tag2”	A comma separated list of runtime tags that MOC should apply to the engine when it is first discovered.
server.port	MODELOP_API_PORT	8003 (default)	The port that the runtime will listen on
modelop.runtime.roundtrip-default-timeout	MODELOP_ROUNDTRIP_DEFAULT_TIMEOUT	5000 (default)	The default timeout (in milliseconds) for roundtrip REST calls
eureka.instance.ip-address	MODELOP_ENGINE_IP	0.0.0.0	The IP address the engine should use to identify itself to the registry
eureka.instance.hostname	MODELOP_ENGINE_HOSTNAME	engine-3	The hostname the engine should use to identify itself to the registry
eureka.instance.instanceId			The instance id to register with eureka (NOT TYPICALLY SET)
eureka.instance.health-check-url		{hostname:port}/1/health	Health check url (NOT TYPICALLY SET)
eureka.instance.home-page-url		{hostname:port}/	Root api url (NOT TYPICALLY SET)
eureka.instance.status-page-url		{hostname:port}/1/platforms	Status check url (NOT TYPICALLY SET)
server.ssl.enabled	MODELOP_ENGINE_USE_HTTPS	true, false (default)	Indicates whether the engine uses SSL
eureka.client.service-url.defaultZone	MODELOP_EUREKA_HOSTNAME and MODELOP_EUREKA_PORT	http://registry:8761	url indicating the registry’s hostname and port
	MODELOP_EUREKA_HOSTNAME	registry	The hostname of the registry
	MODELOP_EUREKA_PORT	8761 (default)	The port on which the registry is listening
modelop.runtime.log-location	MODELOP_RUNTIME_LOG_LOCATION	/tmp/logs	The directory where engine logs are stored
modelop.runtime.log-file-max-size	MODELOP_RUNTIME_LOG_FILE_MAX_SIZE	2097152	Max engine log file size
logstash.location	MODELOP_LOGSTASH_LOCATION	logstash:8093	The host:port where logstash messages should be sent. NOTE: the port is a UDP port, not a TCP port.
modelop.runtime.config-file	MODELOP_RUNTIME_CONFIG_FILE	/config/application.json (default)	The location of the application.json config file
reporting-service.websocket.url	MODELOP_REPORTING_SERVICE_MESSAGES_PATH	http://gateway:8090/reporting-service/messages	The url where internal messages should be sent
	MODELOP_REPORTING_SERVICE_MESSAGES_PATH	/reporting-service/messages (default)	The path on the configured gateway to send internal messages
modelop.provider.token-uri	MODELOP_OAUTH_SERVER_URL	https://host.of.oauth.server/as/token.oauth2	The URI of the oauth server (if present)
modelop.runtime.client-id	MODELOP_OAUTH_CLIENT_ID	moc-runtime	The OAuth client id (populates the ‘client_id’ header)
modelop.runtime.client-secret	MODELOP_OAUTH_CLIENT_SECRET		The OAuth client secret (populates the ‘client_secret’) header)
modelop.runtime.scope	MODELOP_OAUTH_SCOPE		The OAuth client scope (populates the ‘scope’ header)
modelop.runtime.default-listener-idle-timeout	MODELOP_ENGINE_DEFAULT_LISTENER_IDLE_TIMEOUT	60000 (default), infinity	Number of milliseconds (or ‘infinity’) to wait before timing out incoming REST calls on the default control port (the one specified by MODELOP_API_PORT)
modelop.runtime.roundtrip-listener-idle-timeout	MODELOP_ENGINE_ROUNDTRIP_LISTENER_IDLE_TIMEOUT	60000 (default), infinity	Just like MODELOP_ENGINE_DEFAULT_LISTENER_IDLE_TIMEOUT, but for the roundtrip REST listener
modelop.runtime.allow-dynamic-library-installation		true (default)	Whether or not to attempt to install libraries identified by assets with a role of REQUIRED_LIBRARIES
modelop.runtime.max-input-file-size-bytes	MODELOP_RUNTIME_MAX_INPUT_FILE_SIZE_BYTES	1048576000	The max size of a file being used as an input to a batch job
modelop.runtime.log-input-to-stomp		false (default)	Whether or not to log input records to the STOMP queue
modelop.runtime.log-output-to-stomp		false (default)	Whether or not to log output records to the STOMP queue
modelop.runtime.odbc-enable-bulk-inserts	MODELOP_ODBC_ENABLE_BULK_INSERTS	true (default)	Whether or not to convert individual insert commands into bulk insert commands
modelop.runtime.s3-credentials.stored-credentials		`- path: "" authenticationType: "BASIC" accessKeyId: "my_key" secretAccessKey: "my_secret"`	A list of credentials
modelop.runtime.s3-credentials.eks-service-account-credentials		`- path: "s3.us-east-2.amazonaws.com" roleArnEnvVariable: "TEST_ROLE_ARN" webIdentityTokenFileEnvVariable: "TEST_TOKEN_FILE" sessionPrefix: "testprefixABC"`	A list of credentials
modelop.runtime.s3-https-proxy	MODELOP_S3_HTTPS_PROXY	proxy_host:port	The https proxy to use when talking to s3
modelop.runtime.s3-http-proxy	MODELOP_S3_HTTP_PROXY	proxy_host:port	The http proxy to use when talking to s3
modelop.runtime.sts-https-proxy	MODELOP_AWS_STS_HTTPS_PROXY	proxy_host:port	The https proxy to use when talking to sts
modelop.runtime.sts-http-proxy	MODELOP_AWS_STS_HTTP_PROXY	proxy_host:port	The http proxy to use when talking to sts
modelop.runtime.sql-credentials		`- host: "host1" username: "username1" password: "password1"`	A list of credentials
modelop.runtime.monitoring-package		false (default)	Set to ‘true’ to enable the monitoring package
	MODELOP_RUNTIME_PIP3_FREEZE		Set to ‘true’ to run pip freeze before and after installing python requirements attached to a model
	MODELOP_RUNTIME_PIP3_UPGRADE		Set to ‘true’ to attempt to upgrade pip before installing python requirements attached to a model
modelop.runtime.in-production		false (default)	Informs the initial value of the Engine inProduction field
modelop.runtime.environment-name			Informs the initial value of the Engine environmentName field
modelop.runtime.pre-3.0-startup		false (default)	Set to ‘true’ to use the older/deprecated mechanism for retrieving a deployed model upon runtime startup. Newer runtimes, by default, use an API on Model Manage that doesn’t exist on pre 3.0 versions.

moc-builder

When leveraging moc-builder to generate a MOC environment, your environment’s config/config.json file can be updated with the desired values for the aforementioned environment variables. A subsequent compose command will use the values in the config.json when building kubernetes manifests.

application.json

{
  "sql-credentials": [
    {
      "host": "sql-server-abc",
      "username": "my_user",
      "password": "my_password"
    },
    {
      "host": "sql-other-server-def",
      "username": {"from_file": "/config/the-username"},
      "password": {"from_env": "THE_ENV_VAR_WHERE_MY_PASSWORD_IS"}
    }
  ],
  "s3-credentials": {
    "storedCredentials": [
      {
        "path": "",
        "authenticationType": "BASIC",
        "accessKeyId" : "my_key",
        "secretAccessKey" : "my_secret"
      }
    ],
    "eksServiceAccountCredentials": [
      {
        "path": "s3.us-east-2.amazonaws.com",
        "roleArnEnvVariable" : "TEST_ROLE_ARN",
        "webIdentityTokenFileEnvVariable" : "TEST_TOKEN_FILE",
        "sessionPrefix" : "testprefixABC"
      }
    ]
  },
  "azure-credentials": {
    "storedCredentials": [
      {
        "account": "my_account",
        "authenticationType": "SHARED_KEY",
        "path": "",
        "sharedKey": "mysharedkey=="
      }
    ]
  }
}

The ‘sql-credentials’ section allows username and password credentials to be associated with specific sql database hosts. Credentials can be hardcoded directly in the config file, located in a file, or located in an environment variable.

The ‘s3-credentials’ section allows credentials to be hardcoded directly (under storedCredentials) or determined dynamically using the AWS STS service. The credentials are chosen based on the longest matching path.

API’s

The following are a list of commonly used runtime APIs. The provided curl commands assume the runtime is running at localhost:8003. When communicating with a runtime through a MOC gateway, the localhost:8003 will likely look like this: gateway:8090/engine-name, where engine-name is the name the runtime used when it registered with the registry via the env variable: MODELOP_ENGINE_NAME.

Reset the engine

Reset the engine, causing the current model to be unloaded, and the various endpoints reset. This will also cause the current batch job to complete immediately, and move on to the next batch job.

curl -X POST http://localhost:8003/2/engine/reset

Load a deployed model

Load the deployed model contained in the ‘my-deployable-model.json’ file.

curl -X PUT --data-binary "@my-deployable-model.json" -H "Content-Type: application/json" http://localhost:8003/api/deployedModels/{UUID}

There is an optional wait_for_init query parameter for the /api/deployedModels/{UUID} endpoint. The default value of true means to not return from the REST call until the model’s init function has completed successfully. A value of false means to return immediately once the model and the specified endpoints have been loaded, but before the init function returns.

Load a batch job

Load a batch job into the runtime.

curl -X POST --data-binary "@my-batch-job.json" -H "Content-Type: application/json" http://localhost:8003/api/modelBatchJobs/{UUID}

Delete a batch job

Delete the batch job with the specified UUID.

curl -X DELETE http://localhost:8003/api/modelBatchJobs/{UUID}

List all batch jobs

Return all the batch jobs known to the runtime.

curl http://localhost:8003/api/modelBatchJobs/

List a specific batch job

Return the batch job with the specified UUID.

curl http://localhost:8003/api/modelBatchJobs/{UUID}

Score data via the REST endpoint

Submit the contents of input.json to the runtime. Note that in this example, the timeout has been set to 100 seconds, overriding the default set by the MODELOP_ROUNDTRIP_DEFAULT_TIMEOUT environment variable.

curl -X POST --data-binary "@input.json" -H "Content-Type: application/json" http://localhost:8003/api/roundtrip/0/1?timeout=100000

Get current engine status

Retrieve information about the currently running model, attached streams, and overall state of the engine.

curl http://localhost:8003/1/job/status

Sample response:

{
  "streams": [
    {
      "slot": 0,
      "name": "rest",
      "eof": false,
      "descriptor": {
        "Transport": {
          "secured": false,
          "roundtrip": true,
          "port": 9003,
          "Type": "REST",
          "Mode": "simple"
        },
        "Schema": "$inherit",
        "Name": "rest",
        "Encoding": "JSON",
        "Description": "Application Restful Endpoint"
      }
    },
    {
      "slot": 1,
      "name": "rest_out",
      "eof": false,
      "descriptor": {
        "Transport": {
          "Type": "synthetic"
        },
        "Schema": "$inherit",
        "Encoding": "JSON"
      }
    }
  ],
  "state": "running",
  "model": {
    "source": "[model source code...]",
    "snapshots": "none",
    "slots": [
      {
        "slot": 0,
        "schema": null,
        "recordsets": false,
        "action": "action"
      },
      {
        "slot": 1,
        "schema": null,
        "recordsets": false,
        "action": null
      }
    ],
    "name": "test_model.py",
    "mtype": "python3",
    "attachments": []
  },
  "jets": [
    {
      "sandbox": "123472369",
      "pid": 55,
      "busy": false
    }
  ],
  "initialTags": [],
  "group": "modelop"
}

Model Definition Injection

Both the python and R runners support an optional parameter for the init function. Normally, the init function is defined with 0 arguments, but if a single argument is defined, the runner will populate that argument with a ‘model definition’ object. In python, this object is a dictionary. In R, this object is a list. In both cases, the object has two keys. There will be a ‘rawJson’ key that contains the unparsed json string that contains either the job or the deployedModel object in its serialized form. There will also be either a ‘job’ or a ‘deployedModel' key, the value of which will either be a deserialized job or deployedModel object. In python, the deserialized job or deployedModel will be the result of calling json.loads() on the contents of the ‘rawJson’ field. In R, it will be the result of calling the jsonlite fromJSON function.

Python:

def begin(model_def):
  print(model_def['rawJson'])
  print(model_def['job'])

R:

begin <- function(model_def) {
  print(model_def$rawJson)
  print(model_def$deployedModel)
}

To see an example of the above, go to https://github.com/merhi-odg/model_definition_test.

Adding Endpoints to the ModelOp Runtime

To add endpoints to the ModelOp Runtime:

Open the ModelOp Runtime to which you want to add endpoints

2. Select the “Create Endpoint” button for either the input or output endpoints

3. Click the button on the right to add either an Input endpoint or Output endpoint. Choose either REST or Kafka:

REST Endpoints

Add a Name, optional Description, Encoding type, and optional port. Select “Save Endpoint”

2. Repeat for the Output Endpoint, as required.

Kafka Endpoints

Kafka endpoints require several fields, as detailed in the screenshot and table below.

Field	Type	Description	Example

Field	Type	Description	Example
Name	`string`	A user-supplied name for the endpoint	“MyKafkaEndpoint1”
Description	`string`	A user-supplied description for the endpoint	“Kafka Endpoint for consumer transactions”
Encoding	`drop-down`	The encoding used for serialization/deserialization of the kafka messages	json
BootstrapServers	`string`	The name and port of the Kafka bootstrap servers.	"kafka1:9002"
Group (Consumer Group) (optional)	`string`	To provide a horizontally scalable solution, a Consumer Group can be used to allow load balancing of requests across multiple ModelOp runtimes. By defining a Consumer Group, ModelOp Center ensures that only one ModelOp runtime in the Consumer Group services a given message.	"fastscore-1"
Principal (optional)	`string`	An authenticated user in a secure cluster	"kafka/kafka@REALM"
Topic	`string`	The Kafka topic to consume or push to, depending on if input vs. output endpoint	"MyKafkaTopic"
Keyfile (optional)	`string`	The location of the keytab file containing pairs of Kerberos principals and encrypted keys	"/fastscore.keytab"