ModelOp Runtime Details
Anatomy of the ModelOp Runtime
At a high level, the ModelOp Runtime consists of 3 different components: the stream, the jet, and the manifold. Streams deal with the I/O, the jet is a wrapper around the model code itself, and the manifold is responsible for stitching everything together.
Streams
The runtime supports a variety of different streams, including REST, Kafka, File, S3, and ODBC. Input and output streams are typically configured independently, the main exception here being when we want a roundtrip REST call to return with the results from synchronously processed POSTed input.
Jet
The jet consists of two main pieces. There is some common engine code that handles communication with the underlying model runner, and then there is the language specific model runner code that is dynamically started/stopped by the jet. The engine code and the language specific model runner code communicate via a Unix domain socket.
Manifold
The manifold is responsible for managing the lifecycles of the various streams and the underlying jet that is running the model code.
Configuration Options
The engine is configured via two main mechanisms. There are a number of environment variables that affect the behavior, as well as an application.json file that can be configured with credentials the engine may need when interacting with external services.
Environment Variables
Variable | Example Values | Description |
|---|---|---|
MODELOP_LOG_LEVEL | debug, info (default), warning, error | Log level of the engine |
MODELOP_GATEWAY_LOCATION | gateway:8090 | The host:port of the MOC gateway |
MODELOP_ENGINE_NAME | engine-3 | The name of the runtime that will be used to identify it in the registry |
MODELOP_RUNTIME_GROUP | modelop | The group name that owns the runtime. |
MODELOP_RUNTIME_INITIAL_TAGS | “tag1, tag2” | A comma separated list of runtime tags that MOC should apply to the engine when it is first discovered. |
MODELOP_API_PORT | 8003 (default) | The port that the runtime will listen on |
MODELOP_ROUNDTRIP_DEFAULT_TIMEOUT | 5000 (default) | The default timeout (in milliseconds) for roundtrip REST calls |
MODELOP_ENGINE_IP | 0.0.0.0 | The IP address the engine should use to identify itself to the registry |
MODELOP_ENGINE_HOSTNAME | engine-3 | The hostname the engine should use to identify itself to the registry |
MODELOP_ENGINE_USE_HTTPS | true, false (default) | Indicates whether the engine uses SSL |
MODELOP_EUREKA_HOSTNAME | registry | The hostname of the registry |
MODELOP_EUREKA_PORT | 8761 (default) | The port on which the registry is listening |
MODELOP_RUNTIME_LOG_LOCATION | /tmp/logs | The directory where engine logs are stored |
MODELOP_LOGSTASH_LOCATION | logstash:8093 | The host:port where logstash messages should be sent. NOTE: the port is a UDP port, not a TCP port. |
MODELOP_RUNTIME_CONFIG_FILE | /config/application.json (default) | The location of the application.json config file |
MODELOP_REPORTING_SERVICE_MESSAGES_PATH | /reporting-service/messages (default) | The path on the configured gateway to send internal messages |
MODELOP_OAUTH_SERVER_URL | The URI of the oauth server (if present) | |
MODELOP_OAUTH_CLIENT_ID | moc-runtime | The OAuth client id (populates the ‘client_id’ header) |
MODELOP_OAUTH_CLIENT_SECRET |
| The OAuth client secret (populates the ‘client_secret’) header) |
MODELOP_OAUTH_SCOPE |
| The OAuth client scope (populates the ‘scope’ header) |
MODELOP_ENGINE_DEFAULT_LISTENER_IDLE_TIMEOUT | 60000 (default), infinity | Number of milliseconds (or ‘infinity’) to wait before timing out incoming REST calls on the default control port (the one specified by MODELOP_API_PORT) |
MODELOP_ENGINE_ROUNDTRIP_LISTENER_IDLE_TIMEOUT | 60000 (default), infinity | Just like MODELOP_ENGINE_DEFAULT_LISTENER_IDLE_TIMEOUT, but for the roundtrip REST listener |
moc-builder
When leveraging moc-builder to generate a MOC environment, your environment’s config/config.json file can be updated with the desired values for the aforementioned environment variables. A subsequent compose command will use the values in the config.json when building kubernetes manifests.
application.json
{
"sql-credentials": [
{
"host": "sql-server-abc",
"username": "my_user",
"password": "my_password"
},
{
"host": "sql-other-server-def",
"username": {"from_file": "/config/the-username"},
"password": {"from_env": "THE_ENV_VAR_WHERE_MY_PASSWORD_IS"}
}
],
"s3-credentials": {
"storedCredentials": [
{
"path": "",
"authenticationType": "BASIC",
"accessKeyId" : "my_key",
"secretAccessKey" : "my_secret"
}
],
"eksServiceAccountCredentials": [
{
"path": "s3.us-east-2.amazonaws.com",
"roleArnEnvVariable" : "TEST_ROLE_ARN",
"webIdentityTokenFileEnvVariable" : "TEST_TOKEN_FILE",
"sessionPrefix" : "testprefixABC"
}
]
}
}The ‘sql-credentials’ section allows username and password credentials to be associated with specific sql database hosts. Credentials can be hardcoded directly in the config file, located in a file, or located in an environment variable.
The ‘s3-credentials’ section allows credentials to be hardcoded directly (under storedCredentials) or determined dynamically using the AWS STS service. The credentials are chosen based on the longest matching path.
API’s
The following are a list of commonly used runtime APIs. The provided curl commands assume the runtime is running at localhost:8003. When communicating with a runtime through a MOC gateway, the localhost:8003 will likely look like this: gateway:8090/engine-name, where engine-name is the name the runtime used when it registered with the registry via the env variable: MODELOP_ENGINE_NAME.
Reset the engine
Reset the engine, causing the current model to be unloaded, and the various endpoints reset. This will also cause the current batch job to complete immediately, and move on to the next batch job.
curl -X POST http://localhost:8003/2/engine/reset
Load a deployed model
Load the deployed model contained in the ‘my-deployable-model.json’ file.
curl -X PUT --data-binary "@my-deployable-model.json" -H "Content-Type: application/json" http://localhost:8003/api/deployedModels/{UUID}
There is an optional wait_for_init query parameter for the /api/deployedModels/{UUID} endpoint. The default value of true means to not return from the REST call until the model’s init function has completed successfully. A value of false means to return immediately once the model and the specified endpoints have been loaded, but before the init function returns.
Load a batch job
Load a batch job into the runtime.
curl -X POST --data-binary "@my-batch-job.json" -H "Content-Type: application/json" http://localhost:8003/api/modelBatchJobs/{UUID}
Delete a batch job
Delete the batch job with the specified UUID.
curl -X DELETE http://localhost:8003/api/modelBatchJobs/{UUID}
List all batch jobs
Return all the batch jobs known to the runtime.
curl http://localhost:8003/api/modelBatchJobs/
List a specific batch job
Return the batch job with the specified UUID.
curl http://localhost:8003/api/modelBatchJobs/{UUID}
Score data via the REST endpoint
Submit the contents of input.json to the runtime. Note that in this example, the timeout has been set to 100 seconds, overriding the default set by the MODELOP_ROUNDTRIP_DEFAULT_TIMEOUT environment variable.
curl -X POST --data-binary "@input.json" -H "Content-Type: application/json" http://localhost:8003/api/roundtrip/0/1?timeout=100000
Get current engine status
Retrieve information about the currently running model, attached streams, and overall state of the engine.
curl http://localhost:8003/1/job/status
Sample response:
{
"streams": [
{
"slot": 0,
"name": "rest",
"eof": false,
"descriptor": {
"Transport": {
"secured": false,
"roundtrip": true,
"port": 9003,
"Type": "REST",
"Mode": "simple"
},
"Schema": "$inherit",
"Name": "rest",
"Encoding": "JSON",
"Description": "Application Restful Endpoint"
}
},
{
"slot": 1,
"name": "rest_out",
"eof": false,
"descriptor": {
"Transport": {
"Type": "synthetic"
},
"Schema": "$inherit",
"Encoding": "JSON"
}
}
],
"state": "running",
"model": {
"source": "[model source code...]",
"snapshots": "none",
"slots": [
{
"slot": 0,
"schema": null,
"recordsets": false,
"action": "action"
},
{
"slot": 1,
"schema": null,
"recordsets": false,
"action": null
}
],
"name": "test_model.py",
"mtype": "python3",
"attachments": []
},
"jets": [
{
"sandbox": "123472369",
"pid": 55,
"busy": false
}
],
"initialTags": [],
"group": "modelop"
}Model Definition Injection
Both the python and R runners support an optional parameter for the init function. Normally, the init function is defined with 0 arguments, but if a single argument is defined, the runner will populate that argument with a ‘model definition’ object. In python, this object is a dictionary. In R, this object is a list. In both cases, the object has two keys. There will be a ‘rawJson’ key that contains the unparsed json string that contains either the job or the deployedModel object in its serialized form. There will also be either a ‘job’ or a ‘deployedModel' key, the value of which will either be a deserialized job or deployedModel object. In python, the deserialized job or deployedModel will be the result of calling json.loads() on the contents of the ‘rawJson’ field. In R, it will be the result of calling the jsonlite fromJSON function.
Python:
def begin(model_def):
print(model_def['rawJson'])
print(model_def['job'])R:
To see an example of the above, go to https://github.com/merhi-odg/model_definition_test.
Adding Endpoints to the ModelOp Runtime
To add endpoints to the ModelOp Runtime:
Open the ModelOp Runtime to which you want to add endpoints
Select the “Create Endpoint” button for either the input or output endpoints
3. Select to add an endpoint, either REST or Kafka
4. Add a Name, optional Description, Encoding type, and optional port. Select “Save Endpoint”
5. Repeat for the Output Endpoint, as required.
Adding Tags to a ModelOp Runtime
Open the ModelOp Runtime to which you want to add tag(s).
In the “tags” section begin typing the tag that you would like to add:
3. Once done typing, either hit enter on your keyboard or click your mouse in any other location on the screen.
4. Select the “Update Engine” button in the top right to save changes.