ModelOp Runtime Details
- Dave Trier
- Nate Bradac (Unlicensed)
Anatomy of the ModelOp Runtime
At a high level, the ModelOp Runtime consists of 3 different components: the stream, the jet, and the manifold. Streams deal with the I/O, the jet is a wrapper around the model code itself, and the manifold is responsible for stitching everything together.
Streams
The runtime supports a variety of different streams, including REST, Kafka, File, S3, and ODBC. Input and output streams are typically configured independently, the main exception here being when we want a roundtrip REST call to return with the results from synchronously processed POSTed input.
Jet
The jet consists of two main pieces. There is some common engine code that handles communication with the underlying model runner, and then there is the language specific model runner code that is dynamically started/stopped by the jet. The engine code and the language specific model runner code communicate via a Unix domain socket.
Manifold
The manifold is responsible for managing the lifecycles of the various streams and the underlying jet that is running the model code.
Configuration Options
The engine is configured via several mechanisms. The preferred mechanism is via spring configuration through spring cloud config server. There are also a number of (deprecated) environment variables that affect the behavior, as well as an application.json file that can be configured with credentials the engine may need when interacting with external services.
Environment Variables
Spring Config | Deprecated Environment Variable | Example Values | Description |
|---|---|---|---|
modelop.runtime.log-level | MODELOP_LOG_LEVEL | debug, info (default), warning, error | Log level of the engine |
modelop.gateway-url | MODELOP_GATEWAY_LOCATION | http://gateway:8090 gateway:8090 | The host:port of the MOC gateway |
spring.application.name | MODELOP_ENGINE_NAME | engine-3 | The name of the runtime that will be used to identify it in the registry |
spring.cloud.config.uri |
| http://sccs:1234 | The uri of the spring cloud config server from whence to acquire configuration (MUST BE SET LOCALLY) |
spring.profiles.active |
| profile1,profile2,profile3 | The spring profiles that contain the desired configuration (MUST BE SET LOCALLY) |
modelop.runtime.group | MODELOP_RUNTIME_GROUP | modelop | The group name that owns the runtime. |
modelop.runtime.initial-tags | MODELOP_RUNTIME_INITIAL_TAGS | “tag1, tag2” | A comma separated list of runtime tags that MOC should apply to the engine when it is first discovered. |
server.port | MODELOP_API_PORT | 8003 (default) | The port that the runtime will listen on |
modelop.runtime.roundtrip-default-timeout | MODELOP_ROUNDTRIP_DEFAULT_TIMEOUT | 5000 (default) | The default timeout (in milliseconds) for roundtrip REST calls |
modelop.runtime.retain-assets: |
| “true”, “false” | (boolean) Setting to keep the input files to a given Job in the temporary directory on the engine pod. |
eureka.instance.ip-address | MODELOP_ENGINE_IP | 0.0.0.0 | The IP address the engine should use to identify itself to the registry |
eureka.instance.hostname | MODELOP_ENGINE_HOSTNAME | engine-3 | The hostname the engine should use to identify itself to the registry |
eureka.instance.instanceId |
|
| The instance id to register with eureka (NOT TYPICALLY SET) |
eureka.instance.health-check-url |
| {hostname:port}/1/health | Health check url (NOT TYPICALLY SET) |
eureka.instance.home-page-url |
| {hostname:port}/ | Root api url (NOT TYPICALLY SET) |
eureka.instance.status-page-url |
| {hostname:port}/1/platforms | Status check url (NOT TYPICALLY SET) |
server.ssl.enabled | MODELOP_ENGINE_USE_HTTPS | true, false (default) | Indicates whether the engine uses SSL |
eureka.client.service-url.defaultZone | MODELOP_EUREKA_HOSTNAME and MODELOP_EUREKA_PORT | http://registry:8761/eureka | url indicating the registry’s hostname and port, as well as a trailing ‘/eureka’. |
| MODELOP_EUREKA_HOSTNAME | registry | The hostname of the registry |
| MODELOP_EUREKA_PORT | 8761 (default) | The port on which the registry is listening |
modelop.runtime.log-location | MODELOP_RUNTIME_LOG_LOCATION | /tmp/logs | The directory where engine logs are stored |
modelop.runtime.log-file-max-size | MODELOP_RUNTIME_LOG_FILE_MAX_SIZE | 2097152 | Max engine log file size |
logstash.location | MODELOP_LOGSTASH_LOCATION | logstash:8093 | The host:port where logstash messages should be sent. NOTE: the port is a UDP port, not a TCP port. |
modelop.runtime.config-file | MODELOP_RUNTIME_CONFIG_FILE | /config/application.json (default) | The location of the application.json config file |
reporting-service.websocket.url | MODELOP_REPORTING_SERVICE_MESSAGES_PATH | http://gateway:8090/reporting-service/messages | The url where internal messages should be sent |
| MODELOP_REPORTING_SERVICE_MESSAGES_PATH | /reporting-service/messages (default) | The path on the configured gateway to send internal messages |
modelop.provider.token-uri | MODELOP_OAUTH_SERVER_URL | The URI of the oauth server (if present) | |
modelop.runtime.client-id | MODELOP_OAUTH_CLIENT_ID | moc-runtime | The OAuth client id (populates the ‘client_id’ header) |
modelop.runtime.client-secret | MODELOP_OAUTH_CLIENT_SECRET |
| The OAuth client secret (populates the ‘client_secret’) header) |
modelop.runtime.scope | MODELOP_OAUTH_SCOPE |
| The OAuth client scope (populates the ‘scope’ header) |
modelop.runtime.default-listener-idle-timeout | MODELOP_ENGINE_DEFAULT_LISTENER_IDLE_TIMEOUT | 60000 (default), infinity | Number of milliseconds (or ‘infinity’) to wait before timing out incoming REST calls on the default control port (the one specified by MODELOP_API_PORT) |
modelop.runtime.roundtrip-listener-idle-timeout | MODELOP_ENGINE_ROUNDTRIP_LISTENER_IDLE_TIMEOUT | 60000 (default), infinity | Just like MODELOP_ENGINE_DEFAULT_LISTENER_IDLE_TIMEOUT, but for the roundtrip REST listener |
modelop.runtime.allow-dynamic-library-installation |
| true (default) | Whether or not to attempt to install libraries identified by assets with a role of REQUIRED_LIBRARIES |
modelop.runtime.max-input-file-size-bytes | MODELOP_RUNTIME_MAX_INPUT_FILE_SIZE_BYTES | 1048576000 | The max size of a file being used as an input to a batch job |
modelop.runtime.log-input-to-stomp |
| false (default) | Whether or not to log input records to the STOMP queue |
modelop.runtime.log-output-to-stomp |
| false (default) | Whether or not to log output records to the STOMP queue |
modelop.runtime.odbc-enable-bulk-inserts | MODELOP_ODBC_ENABLE_BULK_INSERTS | true (default) | Whether or not to convert individual insert commands into bulk insert commands |
modelop.runtime.s3-credentials.stored-credentials |
| - path: ""
authenticationType: "BASIC"
accessKeyId: "my_key"
secretAccessKey: "my_secret" | A list of credentials |
modelop.runtime.s3-credentials.eks-service-account-credentials |
| - path: "s3.us-east-2.amazonaws.com"
roleArnEnvVariable: "TEST_ROLE_ARN"
webIdentityTokenFileEnvVariable: "TEST_TOKEN_FILE"
sessionPrefix: "testprefixABC" | A list of credentials |
modelop.runtime.kafka-credentials (as of 3.1.1) |
| - bootstrap: my.kafka.server
client-type: producer
topic: my-topic
security_protocol: ssl
ssl_ca_location: /ssl-stuff/ca-cert
ssl_keystore_location: /ssl-stuff/kafka.client.keystore.pkcs12
ssl_keystore_password: "password | ‘bootstrap’ must match one of the bootstrap servers in order to activate this credentials block. Furthermore, ‘client-type’ and ‘topic’ can (they are optional) be used to further restrict when this credentials block is activated. Remaining config options are passed to the underlying kafka client and are outlined here: https://github.com/silviucpp/erlkaf/blob/master/CONFIGURATION.md under ‘Client global configs’. |
modelop.runtime.kafka-config (as of 3.1.1) |
| ‘bootstrap’, ‘client-type’, and ‘topic’ behave as they do for | |
modelop.runtime.s3-https-proxy | MODELOP_S3_HTTPS_PROXY | proxy_host:port | The https proxy to use when talking to s3 |
modelop.runtime.s3-http-proxy | MODELOP_S3_HTTP_PROXY | proxy_host:port | The http proxy to use when talking to s3 |
modelop.runtime.sts-https-proxy | MODELOP_AWS_STS_HTTPS_PROXY | proxy_host:port | The https proxy to use when talking to sts |
modelop.runtime.sts-http-proxy | MODELOP_AWS_STS_HTTP_PROXY | proxy_host:port | The http proxy to use when talking to sts |
modelop.runtime.sql-credentials |
| A list of credentials | |
modelop.runtime.monitoring-package |
| false (default) | Set to ‘true’ to enable the monitoring package |
| MODELOP_RUNTIME_PIP3_FREEZE |
| Set to ‘true’ to run pip freeze before and after installing python requirements attached to a model |
| MODELOP_RUNTIME_PIP3_UPGRADE |
| Set to ‘true’ to attempt to upgrade pip before installing python requirements attached to a model |
modelop.runtime.in-production |
| false (default) | Informs the initial value of the Engine inProduction field |
modelop.runtime.environment-name |
|
| Informs the initial value of the Engine environmentName field |
modelop.runtime.pre-3.0-startup |
| false (default) | Set to ‘true’ to use the older/deprecated mechanism for retrieving a deployed model upon runtime startup. Newer runtimes, by default, use an API on Model Manage that doesn’t exist on pre 3.0 versions. |
modelop.runtime.odbc-timeout |
| 5000 | Timeout for ODBC calls |
modelop.runtime.s3.server-side-encryption.encryption |
| aws:kms | Set to |
moc-builder
When leveraging moc-builder to generate a MOC environment, your environment’s config/config.json file can be updated with the desired values for the aforementioned environment variables. A subsequent compose command will use the values in the config.json when building kubernetes manifests.
application.json
The ‘sql-credentials’ section allows username and password credentials to be associated with specific sql database hosts. Credentials can be hardcoded directly in the config file, located in a file, or located in an environment variable.
The ‘s3-credentials’ section allows credentials to be hardcoded directly (under storedCredentials) or determined dynamically using the AWS STS service. The credentials are chosen based on the longest matching path.
API’s
The following are a list of commonly used runtime APIs. The provided curl commands assume the runtime is running at localhost:8003. When communicating with a runtime through a MOC gateway, the localhost:8003 will likely look like this: gateway:8090/engine-name, where engine-name is the name the runtime used when it registered with the registry via the env variable: MODELOP_ENGINE_NAME.
Reset the engine
Reset the engine, causing the current model to be unloaded, and the various endpoints reset. This will also cause the current batch job to complete immediately, and move on to the next batch job.
curl -X POST http://localhost:8003/2/engine/reset
Load a deployed model
Load the deployed model contained in the ‘my-deployable-model.json’ file.
curl -X PUT --data-binary "@my-deployable-model.json" -H "Content-Type: application/json" http://localhost:8003/api/deployedModels/{UUID}
There is an optional wait_for_init query parameter for the /api/deployedModels/{UUID} endpoint. The default value of true means to not return from the REST call until the model’s init function has completed successfully. A value of false means to return immediately once the model and the specified endpoints have been loaded, but before the init function returns.
Load a batch job
Load a batch job into the runtime.
curl -X POST --data-binary "@my-batch-job.json" -H "Content-Type: application/json" http://localhost:8003/api/modelBatchJobs/{UUID}
Delete a batch job
Delete the batch job with the specified UUID.
curl -X DELETE http://localhost:8003/api/modelBatchJobs/{UUID}
List all batch jobs
Return all the batch jobs known to the runtime.
curl http://localhost:8003/api/modelBatchJobs/
List a specific batch job
Return the batch job with the specified UUID.
curl http://localhost:8003/api/modelBatchJobs/{UUID}
Score data via the REST endpoint
Submit the contents of input.json to the runtime. Note that in this example, the timeout has been set to 100 seconds, overriding the default set by the MODELOP_ROUNDTRIP_DEFAULT_TIMEOUT environment variable.
curl -X POST --data-binary "@input.json" -H "Content-Type: application/json" http://localhost:8003/api/roundtrip/0/1?timeout=100000
Get current engine status
Retrieve information about the currently running model, attached streams, and overall state of the engine.
curl http://localhost:8003/1/job/status
Sample response:
Model Definition Injection
Both the python and R runners support an optional parameter for the init function. Normally, the init function is defined with 0 arguments, but if a single argument is defined, the runner will populate that argument with a ‘model definition’ object. In python, this object is a dictionary. In R, this object is a list. In both cases, the object has two keys. There will be a ‘rawJson’ key that contains the unparsed json string that contains either the job or the deployedModel object in its serialized form. There will also be either a ‘job’ or a ‘deployedModel' key, the value of which will either be a deserialized job or deployedModel object. In python, the deserialized job or deployedModel will be the result of calling json.loads() on the contents of the ‘rawJson’ field. In R, it will be the result of calling the jsonlite fromJSON function.
Python:
R:
To see an example of the above, go to https://github.com/merhi-odg/model_definition_test.
Adding Endpoints to the ModelOp Runtime
To add endpoints to the ModelOp Runtime:
Open the ModelOp Runtime to which you want to add endpoints
2. Select the “Create Endpoint” button for either the input or output endpoints
3. Click the button on the right to add either an Input endpoint or Output endpoint. Choose either REST or Kafka:
REST Endpoints
Add a Name, optional Description, Encoding type, and optional port. Select “Save Endpoint”
2. Repeat for the Output Endpoint, as required.
Kafka Endpoints
Kafka endpoints require several fields, as detailed in the screenshot and table below.
Field | Type | Description | Example |
|---|---|---|---|
Name |
| A user-supplied name for the endpoint | “MyKafkaEndpoint1” |
Description |
| A user-supplied description for the endpoint | “Kafka Endpoint for consumer transactions” |
Encoding |
| The encoding used for serialization/deserialization of the kafka messages | json |
BootstrapServers |
| The name and port of the Kafka bootstrap servers. | "kafka1:9002" |
Group (Consumer Group) (optional) |
| To provide a horizontally scalable solution, a Consumer Group can be used to allow load balancing of requests across multiple ModelOp runtimes. By defining a Consumer Group, ModelOp Center ensures that only one ModelOp runtime in the Consumer Group services a given message. | "fastscore-1" |
Principal (optional) |
| An authenticated user in a secure cluster | "kafka/kafka@REALM" |
Topic |
| The Kafka topic to consume or push to, depending on if input vs. output endpoint | "MyKafkaTopic" |
Keyfile (optional) |
| The location of the keytab file containing pairs of Kerberos principals and encrypted keys | "/fastscore.keytab" |
2. Repeat for the Output Endpoint, as required.
Adding files to a runtime container for purposes of Kerberos or mTLS authentication
Files can be added to a k8s secret or configmap, and then the secret or configmap can be mounted to a specific location in the runtime container’s file system.
Create a secret or configmap that contains the necessary files.
This can be done using
kubectl:kubectl create configmap [configmap_name] --from-file [path/to/directory]This can be done via
helmPlace the files you want in your configmap inside a subdirectory of the helm chart:
moc/my-files/some.fileIn the helm values/override file:
Mount the secret or configmap on a runtime:
This will create a configmap called my-configmap that contains a file called some.file and will mount that file into "my-runtime" such that inside the runtime’s container there will exist a file at /path/to/files/in/container/some.file.
Adding Tags to a ModelOp Runtime
Open the ModelOp Runtime to which you want to add tag(s).
In the “tags” section begin typing the tag that you would like to add:
3. Once done typing, either hit enter on your keyboard or click your mouse in any other location on the screen.
4. Select the “Update Engine” button in the top right to save changes.