Introduction
ModelOp Center provides a standard framework for defining a model for deployment. Enterprise model deployment requires data science experiments that need to be packaged as software assets for deployment. Model Development Notebooks cannot be deployed into Production. ModelOp Center provides the abstractions required to deploy, monitor, and govern a model.
Standard Model Definition in ModelOp Center
Of the several assets that define how a model will perform upon deployment, only a Model Source with a Scoring Function is required to initially register a model. The next section describes the available functions.
The list of available abstractions includes:
Model Source - Code that is called as the model is deployed, scored, trained, or validated. The Model Source can be backed with a git repo where it can be versioned and managed. See the next section for details.
Model Functions - Special functions within the Model Source which the runtime uses for specific tasks. See the next section for details.
Attachment - This abstraction allows data scientists to load in different versions of the trained model artifact without changing the code used for scoring. consists of external files to be utilized during prediction or scoring. The contents of the attachment get extracted into the current working directory of the model source. include any files or binaries that will be referenced during scoring including the trained model artifact.
Schemas - define the data structure and features the model uses on its input as well as its output. The schema can reject the record received for scoring in order to prevent the model from erring. This also serves as the contract between the data pipeline and the model, as well as the data scientist and the data engineer.
Model Platform - details the dependencies the model requires once deployed. This metadata is used to determine which runtime will be used during deployment of the model. Note: this information is automatically captured based on the Jupyter Notebook environment if using the Jupyter Notebook plugin (see Integrate with Jupyter).
See Model Governance: Standard Model Definition for additional detail.
Functions within Model Source
Within the Source Code, you can designate the specific entry points, or functions, into the model code. The Scoring Function is the only function required for deployment. The other functions define the other steps of a model’s life cycle and operationalization.
These functions are executed at different portions of the Model Life Cycle utilizing Batch Jobs. Batch Jobs can be executed manually as part of testing or automated within an MLC Process. This provides a flexible and scalable framework for deploying Models into Business. Batch Jobs including scoring, test, and training can be run manually or automated using an MLC Process.
The function examples in the following sections are a walk through of the Model Source for a lasso regression model that detects emails that are forwarded outside the organization. These functions can be mapped from the Model Source using the Command Center or using smart tag comments. The smart tag comments are called out in red in the following sections.
See ModelOp Life Cycle Model Lifecycle Manager: Automation for more information about MLC Processes.
See Model Batch Jobs and Tests for details about Batch Jobs.
Init Function
The Init Function is executed when the model is initially deployed into the runtime. It initializes the model and loads any dependencies for scoring. If the model uses an attachment, the Init Function typically loads the trained model artifact for scoring. In this example of a regression model, the model artifacts are loaded from the pickle file with the trained model weights.
# modelop.init
| Code Block |
|---|
def begin():
global lasso_model_artifacts
lasso_model_artifacts = pickle.load(open('lasso_model_artifacts.pkl', 'rb'))
nltk.download('averaged_perceptron_tagger')
pass |
Scoring Function
The Scoring Function yields predictions from input records. It executes when a record is sent to the endpoint of the runtime the model is deployed in. It can also be called using a Scoring Batch Job in the Command Center, and from the CLI. In this example of a regression model, the action function is tokenizing an incoming email, applying a bag-of-words transformation, applying a TFIDF vectorizer to the bag of words, and then using a LASSO regularized logistic regression to produce a classification.
# modelop.score
| Code Block | ||
|---|---|---|
| ||
def action(x):
lasso_model = lasso_model_artifacts['lasso_model']
dictionary = lasso_model_artifacts['dictionary']
threshold = lasso_model_artifacts['threshold']
tfidf_model = lasso_model_artifacts['tfidf_model']
x = pd.DataFrame(x, index=[0])
print(x) sys.stdout.flush()
cleaned = preprocess(x.content)
corpus = cleaned.apply(dictionary.doc2bow)
corpus_sparse = gensim.matutils.corpus2csc(corpus).transpose()
corpus_sparse_padded = pad_sparse_matrix(sp_mat = corpus_sparse,
length=corpus_sparse.shape[0],
width = len(dictionary))
tfidf_vectors = tfidf_model.transform(corpus_sparse_padded)
probabilities = lasso_model.predict_proba(tfidf_vectors)[:,1]
predictions = pd.Series(probabilities > threshold, index=x.index).astype(int)
output = pd.concat([x, predictions], axis=1)
output.columns = ['content', 'id', 'prediction']
output = output.to_dict(orient='records')
yield print(output)
yield output
def matrix_to_dicts(matrix, labels):
cm = []
for idx, label in enumerate(labels):
cm.append(dict(zip(labels, matrix[idx, :].tolist())))
return cm
output |
Metrics Function
The Metrics Function calculates metrics around the model’s performance based on labelled data, including mathematical (back-test), bias, and interpretability. It executes when a Test Batch Job is run on the model. The Metrics Function calculates a confusion matrix, ROC curve, AUC, F2 , ROC.
For more details on how to write monitoring metrics for a model, see Model Efficacy Metrics and Monitoring .
The Metrics Function can also be used to calculate bias and Interpretability as discussed in Model Governance: Bias & Interpretability .
# modelop.metrics
| Code Block |
|---|
def metrics(x):
lasso_model = lasso_model_artifacts['lasso_model']
dictionary = lasso_model_artifacts['dictionary']
threshold = lasso_model_artifacts['threshold']
tfidf_model = lasso_model_artifacts['tfidf_model']
actuals = x.flagged
cleaned = preprocess(x.content)
corpus = cleaned.apply(dictionary.doc2bow)
corpus_sparse = gensim.matutils.corpus2csc(corpus).transpose()
corpus_sparse_padded = pad_sparse_matrix(sp_mat = corpus_sparse,
length=corpus_sparse.shape[0],
width = len(dictionary))
tfidf_vectors = tfidf_model.transform(corpus_sparse_padded)
probabilities = lasso_model.predict_proba(tfidf_vectors)[:,1]
predictions = pd.Series(probabilities > threshold, index=x.index).astype(int)
confusion_matrix = sklearn.metrics.confusion_matrix(actuals, predictions)
fpr,tpr,thres = sklearn.metrics.roc_curve(actuals, predictions)
auc_val = sklearn.metrics.auc(fpr, tpr)
f2_score = sklearn.metrics.fbeta_score(actuals, predictions, beta=2)
roc_curve = [{'fpr': x[0], 'tpr':x[1]} for x in list(zip(fpr, tpr))]
labels = ['Compliant', 'Non-Compliant']
cm = matrix_to_dicts(confusion_matrix, labels)
test_results = dict(roc_curve=roc_curve,
auc=auc_val,
f2_score=f2_score,
confusion_matrix=cm)
yield test_results |
Training Function
The Training Function train and retrain the model. It executes when a Training Batch Job is called on the model. The Training Function also provides context and traceability on the model origins and collaboration between data scientiststhe model origins and collaboration between data scientists. Any files written to the directory “outputDir” as in the example below will be written to S3 as an External File Asset.
# modelop.training
| Code Block |
|---|
def train(data):
y_train = data.flagged
removed_proper_nouns = data.content.astype(str).apply(remove_proper_nouns)
CUSTOM_FILTERS = [lambda x: x.lower(),
gensim.parsing.preprocessing.strip_tags,
gensim.parsing.preprocessing.strip_punctuation]
removed_punctuation = removed_proper_nouns.apply(functools.partial(gensim.parsing.preprocess_string, filters=CUSTOM_FILTERS))
stemmer = nltk.stem.porter.PorterStemmer()
#Remove stop words, words of length less than 2, and words with non-alphabet characters.
cleaned = removed_punctuation.apply(lambda x: list(map(gensim.parsing.preprocessing.remove_stopwords, x)))
cleaned = cleaned.apply(lambda x: list(filter(lambda y: len(y) > 1, x)))
cleaned = cleaned.apply(lambda x: list(filter(lambda y: y.isalpha(), x)))
cleaned = cleaned.apply(lambda x: list(map(stemmer.stem, x)))
#Create a dictionary (key, value pairs of ids with words which appear in the corpus.
dictionary = gensim.corpora.dictionary.Dictionary(documents=cleaned)
dictionary.filter_extremes(no_below=5, no_above=0.4)
# Produce a sparse bag-of-words matrix from the word-document frequency counts
corpus = cleaned.apply(dictionary.doc2bow).to_list()
corpus_sparse = gensim.matutils.corpus2csc(corpus).transpose()
# Train a tf-idf transformer and transform the training data
tfidf_model = sklearn.feature_extraction.text.TfidfTransformer()
train_tfidf = tfidf_model.fit_transform(train_corpus_sparse)
# Define and fit a logistic regression model
logreg = sklearn.linear_model.LogisticRegression(penalty='l1', class_weight='balanced', max_iter=2500, random_state=740189)
logreg_model = logreg.fit(X=train_tfidf, y=y_train)
lasso_model_artifacts = dict(lasso_model = logreg_model,
dictionary = dictionary,
= dictionary tfidf_model = tfidf_model,
tfidf_modelthreshold = tfidf_model,thresh)
threshold = thresh)
pickle.dump(lasso_model_artifacts, with open('outputDir/lasso_model_artifactsartifact.pkl', 'wb') as f:
pickle.dump(lasso_model_artifacts, f)
pass |
Related Articles
ModelOp Life Cycle Model Lifecycle Manager: Automation
Next Article: Register a Model >