Anatomy of the ModelOp Runtime

At a high level, the ModelOp Runtime consists of 3 different components: the stream, the jet, and the manifold. Streams deal with the I/O, the jet is a wrapper around the model code itself, and the manifold is responsible for stitching everything together.

Streams

The runtime supports a variety of different streams, including REST, Kafka, File, S3, and ODBC. Input and output streams are typically configured independently, the main exception here being when we want a roundtrip REST call to return with the results from synchronously processed POSTed input.

Jet

The jet consists of two main pieces. There is some common engine code that handles communication with the underlying model runner, and then there is the language specific model runner code that is dynamically started/stopped by the jet. The engine code and the language specific model runner code communicate via a Unix domain socket.

Manifold

The manifold is responsible for managing the lifecycles of the various streams and the underlying jet that is running the model code.

Configuration Options

The engine is configured via several mechanisms. The preferred mechanism is via spring configuration through spring cloud config server (via the application-runtime.yaml file). There are also a number of (deprecated) environment variables that affect the behavior, as well as a (deprecated) application.json file.

Environment Variables

Spring Config	Deprecated Environment Variable	Example Values	Description

Spring Config	Deprecated Environment Variable	Example Values	Description
modelop.runtime.log-level	MODELOP_LOG_LEVEL	debug, info (default), warning, error	Log level of the engine
modelop.gateway-url	MODELOP_GATEWAY_LOCATION	http://gateway:8090 gateway:8090	The host:port of the MOC gateway
spring.application.name	MODELOP_ENGINE_NAME	engine-3	The name of the runtime that will be used to identify it in the registry
spring.cloud.config.uri		http://sccs:1234	The uri of the spring cloud config server from whence to acquire configuration (MUST BE SET LOCALLY)
spring.profiles.active		profile1,profile2,profile3	The spring profiles that contain the desired configuration (MUST BE SET LOCALLY)
modelop.runtime.group	MODELOP_RUNTIME_GROUP	modelop	The group name that owns the runtime.
modelop.runtime.initial-tags	MODELOP_RUNTIME_INITIAL_TAGS	“tag1, tag2”	A comma separated list of runtime tags that MOC should apply to the engine when it is first discovered.
server.port	MODELOP_API_PORT	8003 (default)	The port that the runtime will listen on
server.eureka_port		8003 (default, uses server.port)	The port that the runtime instructs eureka to use. Useful for accessing a runtime from a different cluster or namespace using an ingress.
modelop.runtime.roundtrip-default-timeout	MODELOP_ROUNDTRIP_DEFAULT_TIMEOUT	5000 (default)	The default timeout (in milliseconds) for roundtrip REST calls
modelop.runtime.retain-assets:		“true”, “false”	(boolean) Setting to keep the input files to a given Job in the temporary directory on the engine pod.
modelop.runtime.display-records		“true”, “false” (default)	Log records rejected by a schema.
eureka.instance.ip-address	MODELOP_ENGINE_IP	0.0.0.0	The IP address the engine should use to identify itself to the registry
eureka.instance.hostname	MODELOP_ENGINE_HOSTNAME	engine-3	The hostname the engine should use to identify itself to the registry
eureka.instance.instanceId			The instance id to register with eureka (NOT TYPICALLY SET)
eureka.instance.health-check-url		{hostname:port}/1/health	Health check url (NOT TYPICALLY SET)
eureka.instance.home-page-url		{hostname:port}/	Root api url (NOT TYPICALLY SET)
eureka.instance.status-page-url		{hostname:port}/1/platforms	Status check url (NOT TYPICALLY SET)
server.ssl.enabled	MODELOP_ENGINE_USE_HTTPS	true, false (default)	Indicates whether the engine uses SSL
eureka.client.service-url.defaultZone	MODELOP_EUREKA_HOSTNAME and MODELOP_EUREKA_PORT	http://registry:8761/eureka	url indicating the registry’s hostname and port, as well as a trailing ‘/eureka’.
	MODELOP_EUREKA_HOSTNAME	registry	The hostname of the registry
	MODELOP_EUREKA_PORT	8761 (default)	The port on which the registry is listening
modelop.runtime.log-location	MODELOP_RUNTIME_LOG_LOCATION	/tmp/logs	The directory where engine logs are stored
modelop.runtime.log-file-max-size	MODELOP_RUNTIME_LOG_FILE_MAX_SIZE	2097152	Max engine log file size
logstash.location	MODELOP_LOGSTASH_LOCATION	logstash:8093	The host:port where logstash messages should be sent. NOTE: the port is a UDP port, not a TCP port.
	MODELOP_RUNTIME_CONFIG_FILE	/config/application.json (default)	The location of the application.json config file.
	MODELOP_RUNTIME_BOOTSTRAP_FILE	/config/bootstrap.json (default)	The location the bootstrap.json config file.
reporting-service.websocket.url	MODELOP_REPORTING_SERVICE_MESSAGES_PATH	http://gateway:8090/reporting-service/messages	The url where internal messages should be sent
	MODELOP_REPORTING_SERVICE_MESSAGES_PATH	/reporting-service/messages (default)	The path on the configured gateway to send internal messages
modelop.provider.token-uri	MODELOP_OAUTH_SERVER_URL	https://host.of.oauth.server/as/token.oauth2	The URI of the oauth server (if present)
modelop.runtime.client-id	MODELOP_OAUTH_CLIENT_ID	moc-runtime	The OAuth client id (populates the ‘client_id’ header)
modelop.runtime.client-secret	MODELOP_OAUTH_CLIENT_SECRET		The OAuth client secret (populates the ‘client_secret’) header)
modelop.runtime.scope	MODELOP_OAUTH_SCOPE		The OAuth client scope (populates the ‘scope’ header)
modelop.runtime.default-listener-idle-timeout	MODELOP_ENGINE_DEFAULT_LISTENER_IDLE_TIMEOUT	60000 (default), infinity	Number of milliseconds (or ‘infinity’) to wait before timing out incoming REST calls on the default control port (the one specified by MODELOP_API_PORT)
modelop.runtime.roundtrip-listener-idle-timeout	MODELOP_ENGINE_ROUNDTRIP_LISTENER_IDLE_TIMEOUT	60000 (default), infinity	Just like MODELOP_ENGINE_DEFAULT_LISTENER_IDLE_TIMEOUT, but for the roundtrip REST listener
modelop.runtime.rest-client-timeout		5000 (default)	Just like MODELOP_ENGINE_DEFAULT_LISTENER_IDLE_TIMEOUT, but for the REST client streams to respond.
modelop.runtime.stream-start-timeout		20000 (default)	Number of milliseconds to wait before timing out a stream startup.
modelop.runtime.allow-dynamic-library-installation		true (default)	Whether or not to attempt to install libraries identified by assets with a role of REQUIRED_LIBRARIES
modelop.runtime.max-input-file-size-bytes	MODELOP_RUNTIME_MAX_INPUT_FILE_SIZE_BYTES	1048576000	The max size of a file being used as an input to a batch job
modelop.runtime.publish-input-to-stomp		false (default)	Whether or not to log input records to the STOMP queue
modelop.runtime.publish-output-to-stomp		false (default)	Whether or not to log output records to the STOMP queue
modelop.runtime.odbc-enable-bulk-inserts	MODELOP_ODBC_ENABLE_BULK_INSERTS	true (default)	Whether or not to convert individual insert commands into bulk insert commands
modelop.runtime.s3-credentials.stored-credentials		`- path: "" authenticationType: "BASIC" accessKeyId: "my_key" secretAccessKey: "my_secret"`	A list of credentials
modelop.runtime.s3-credentials.eks-service-account-credentials		`- path: "s3.us-east-2.amazonaws.com" roleArnEnvVariable: "TEST_ROLE_ARN" webIdentityTokenFileEnvVariable: "TEST_TOKEN_FILE" sessionPrefix: "testprefixABC"` Or `- path: "s3.us-east-2.amazonaws.com" roleArn: "arn:aws:iam::account_number:role/TEST_S3_ROLE" webIdentityTokenFile: "/path/to/.web-id-token-file" sessionPrefix: "testprefixABC"`	A list of credentials
modelop.runtime.azure-credentials.stored-credentials			A list of credentials
modelop.runtime.kafka-credentials (as of 3.1.1)			‘bootstrap’ must match one of the bootstrap servers in order to activate this credentials block. Furthermore, ‘client-type’ and ‘topic’ can (they are optional) be used to further restrict when this credentials block is activated. Remaining config options are passed to the underlying kafka client and are outlined here: erlkaf/CONFIGURATION.md at master · silviucpp/erlkaf under ‘Client global configs’.
modelop.runtime.kafka-config (as of 3.1.1)			‘bootstrap’, ‘client-type’, and ‘topic’ behave as they do for `modelop.runtime.kafka-credentials`. The `batch_size` option sets the batch size when activating the config block for a consumer. Remaining config options are passed to the underlying kafka client and are outlined here: https://github.com/silviucpp/erlkaf/blob/master/CONFIGURATION.md#topic-configuration-properties under ‘Topic configuration properties’.
https_proxy (must be set as an environment variable)	https_proxy	scheme://username:password@host:port	The username:password is optional. The scheme should either be `https` or `wss` and indicates whether the runtime will use TLS/SSL to connect to the proxy.
http_proxy (must be set as an environment variable)	http_proxy	scheme://username:password@host:port	The username:password is optional. The scheme should either be `http` or `ws` and indicates whether the runtime will use not use TLS/SSL to connect to the proxy.
no_proxy (must be set as an environment variable)	no_proxy	host	The `no_proxy` variable should be a comma separated list of hostnames or regex expressions that, if they match the hostname, the proxy will not be used.
modelop.runtime.sql-credentials			A list of credentials
modelop.runtime.monitoring-package		false (default)	Set to ‘true’ to enable the monitoring package
	MODELOP_RUNTIME_PIP3_FREEZE		Set to ‘true’ to run pip freeze before and after installing python requirements attached to a model
	MODELOP_RUNTIME_PIP3_UPGRADE		Set to ‘true’ to attempt to upgrade pip before installing python requirements attached to a model
modelop.runtime.in-production		false (default)	Informs the initial value of the Engine inProduction field
modelop.runtime.environment-name			Informs the initial value of the Engine environmentName field
modelop.runtime.pre-3.0-startup		false (default)	Set to ‘true’ to use the older/deprecated mechanism for retrieving a deployed model upon runtime startup. Newer runtimes, by default, use an API on Model Manage that doesn’t exist on pre 3.0 versions.
modelop.runtime.odbc-timeout		5000	Timeout for ODBC calls
modelop.runtime.use-auth-header-for-stomp		true (default)	Add an authorization header to the websocket connection.
modelop.runtime.force-oauth-credentials-into-runner-env		false (default)	Enables the modelop.client-secret in the python runner environment for use in models. By default, client-secret is removed from runner environments.
modelop.runtime.asset-roles-to-exclude		"MODEL_TEST_SOURCE,TRAINING_DATA,TEST_DATA,COMPARATOR_DATA,BASELINE_DATA,LICENSE_FILE,MODEL_RUNTIME_CONTAINER,MODEL_METADATA,MODEL_DOCUMENTATION,EXTERNAL_ASSETS" (default)	A string of asset roles. Assets with those roles will not be copied into the runtime when running a job on a model.
modelop.runtime.s3.server-side-encryption.encryption		aws:kms	Set to `aws:kms` when writing to encrypted s3 buckets configured with the `DenyIncorrectEncryptionHeader` policy.
management.endpoint.loggers.enabled		false (default)	Enables actuator/loggers endpoint.
management.endpoint.env.enabled		false (default)	Enables actuator/env endpoint.

application.json (deprecated in favor of spring cloud config via the application-runtime.yaml file)

The ‘sql-credentials’ section allows username and password credentials to be associated with specific sql database hosts. Credentials can be hardcoded directly in the config file, located in a file, or located in an environment variable.

The ‘s3-credentials’ section allows credentials to be hardcoded directly (under storedCredentials) or determined dynamically using the AWS STS service. The credentials are chosen based on the longest matching path.

application-runtime.yaml

API’s

The following are a list of commonly used runtime APIs. The provided curl commands assume the runtime is running at localhost:8003. When communicating with a runtime through a MOC gateway, the localhost:8003 will likely look like this: gateway:8090/engine-name, where engine-name is the name the runtime used when it registered with the registry via the env variable: MODELOP_ENGINE_NAME.

Reset the engine

Reset the engine, causing the current model to be unloaded, and the various endpoints reset. This will also cause the current batch job to complete immediately, and move on to the next batch job.

curl -X POST http://localhost:8003/2/engine/reset

Load a deployed model

Load the deployed model contained in the ‘my-deployable-model.json’ file.

curl -X PUT --data-binary "@my-deployable-model.json" -H "Content-Type: application/json" http://localhost:8003/api/deployedModels/{UUID}

There is an optional wait_for_init query parameter for the /api/deployedModels/{UUID} endpoint. The default value of true means to not return from the REST call until the model’s init function has completed successfully. A value of false means to return immediately once the model and the specified endpoints have been loaded, but before the init function returns.

Load a batch job

Load a batch job into the runtime.

curl -X POST --data-binary "@my-batch-job.json" -H "Content-Type: application/json" http://localhost:8003/api/modelBatchJobs/{UUID}

Delete a batch job

Delete the batch job with the specified UUID.

curl -X DELETE http://localhost:8003/api/modelBatchJobs/{UUID}

List all batch jobs

Return all the batch jobs known to the runtime.

curl http://localhost:8003/api/modelBatchJobs/

List a specific batch job

Return the batch job with the specified UUID.

curl http://localhost:8003/api/modelBatchJobs/{UUID}

Score data via the REST endpoint

Submit the contents of input.json to the runtime. Note that in this example, the timeout has been set to 100 seconds, overriding the default set by the MODELOP_ROUNDTRIP_DEFAULT_TIMEOUT environment variable.

curl -X POST --data-binary "@input.json" -H "Content-Type: application/json" http://localhost:8003/api/roundtrip/0/1?timeout=100000

Get current engine status

Retrieve information about the currently running model, attached streams, and overall state of the engine.

curl http://localhost:8003/1/job/status

Sample response:

Model Definition Injection

Both the python and R runners support an optional parameter for the init function. Normally, the init function is defined with 0 arguments, but if a single argument is defined, the runner will populate that argument with a ‘model definition’ object. In python, this object is a dictionary. In R, this object is a list. In both cases, the object has two keys. There will be a ‘rawJson’ key that contains the unparsed json string that contains either the job or the deployedModel object in its serialized form. There will also be either a ‘job’ or a ‘deployedModel' key, the value of which will either be a deserialized job or deployedModel object. In python, the deserialized job or deployedModel will be the result of calling json.loads() on the contents of the ‘rawJson’ field. In R, it will be the result of calling the jsonlite fromJSON function.

Python:

R:

To see an example of the above, go to https://github.com/merhi-odg/model_definition_test.

Adding Endpoints to the ModelOp Runtime

To add endpoints to the ModelOp Runtime:

Open the ModelOp Runtime to which you want to add endpoints

2. Select the “Create Endpoint” button for either the input or output endpoints

3. Click the button on the right to add either an Input endpoint or Output endpoint. Choose either REST or Kafka:

REST Endpoints

Add a Name, optional Description, Encoding type, and optional port. Select “Save Endpoint”

2. Repeat for the Output Endpoint, as required.

Kafka Endpoints

Kafka endpoints require several fields, as detailed in the screenshot and table below.

Field	Type	Description	Example

Field	Type	Description	Example
Name	`string`	A user-supplied name for the endpoint	“MyKafkaEndpoint1”
Description	`string`	A user-supplied description for the endpoint	“Kafka Endpoint for consumer transactions”
Encoding	`drop-down`	The encoding used for serialization/deserialization of the kafka messages	json
BootstrapServers	`string`	The name and port of the Kafka bootstrap servers.	"kafka1:9002"
Group (Consumer Group) (optional)	`string`	To provide a horizontally scalable solution, a Consumer Group can be used to allow load balancing of requests across multiple ModelOp runtimes. By defining a Consumer Group, ModelOp Center ensures that only one ModelOp runtime in the Consumer Group services a given message.	"fastscore-1"
Principal (optional)	`string`	An authenticated user in a secure cluster	"kafka/kafka@REALM"
Topic	`string`	The Kafka topic to consume or push to, depending on if input vs. output endpoint	"MyKafkaTopic"
Keyfile (optional)	`string`	The location of the keytab file containing pairs of Kerberos principals and encrypted keys	"/fastscore.keytab"