Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 3 Next »

ModelOp Center seamlessly integrates with existing enterprise source code management (SCM) systems, such as Bitbucket, Github, and Git, to allow enterprises to leverage existing IT investments.

Table of Contents

Introduction

ModelOp Center allows integration with Git platforms in order to externalize the management of model assets as well as allow for a distributed development of such resources. This in place, allows for the development of such assets to be done from the platform (IDE) of choice of the Data Scientist providing greater flexibility using a widely accepted technology such as Git.

ModelOp Center allows different levels of interaction with Git:

  1. No Git interaction. Model assets are not stored in a Git repository but rather are stored in a local database.

  2. Local Git repository. It is possible to create and work in a local Git repository which will be used by ModelOp Center to commit changes to assets. A remote git repository can be added to the asset at any point.

  3. Remote Git repository. We can also configure a remote git repository to sync external changes back into ModelOp Center working directory. It is also possible optionally enable/disable the feature to push to upstream.

Remote import

Given a remote repository url to clone, ModelOp Center clones down the repo and creates a “Model” with all files represented as assets within it. All non-source files will be turned into external assets and loaded into the configured external file asset repository instance (e.g. S3 / MinIO). All source assets will be loaded into the “Model” instance, and have their repository information loaded appropriately.

  • Source files are automatically determined (by file extension):

    • Model source: "py", "py3", "c", "r", "pfa", "ipynb", "ppfa", "java", "m"

    • Test result comparator: "dmn", "cmmn"

  • Non-source files are considered any other type of files.

ModelOp Center allows to import a git repository through ModelOp Center Dashboard

  1. Click on the “Models” menu item.

  2. On the top right corner click on “Import”.

  3. Provide the details of the “Remote Repository Url” and “Branch”.
    Please note for the URL to extract the one depending on your Git platform, generally you can find this URL by going to the “Clone” button and selecting the “HTTPS clone” option.

  4. Click on “Import model from Git”

For the above steps to work on private repositories with login restrictions please see the sections below

Git integration setup

In order to be able to configure a private remote repository for any model in ModelOp Center, it’s necessary to go through certain configuration settings

Git credentials

In order for ModelOp Center to know which username and password to use to connect to the remote repository, we have the following properties:

Container: "model-manager":

model-manage.git.username=<The git username>
model-manage.git.password=<The username's passphrase>

The above configuration will be used to access all remote repositories from ModelOp Center.

Repo settings

ModelOp Center also provides additional parameters, so that we can configure the behavior of our local repository.

Container: "model-manager":

model-manage.repo.push=false   # Enable/Disable to push back to the remote repository after an update has been applied to an asset in ModelOp Center
model-manage.repo.change-poll-rate=120000   # Time interval between remote repository polls for updates.
model-manage.repo.base-directory=/tmp/model-manage-repos   # Default container's base for git repositories

Git Config

It is possible to add git-specific configurations to the way ModelOp Center’s git behaves. Please refer to git scm for more details on the git configuration and how it can be used to achieve certain behavior.

 Useful git config syntax terminology for the purposes of mapping git configuration into ModelOp Center

Assume a property key of the form 'section.subsection.variable=value', the following is used as guideline:

  • A section is considered to be everything up to the first dot of the param key.

  • A subsection is considered everything in between the first and the last dots of the param key. This is optional and may be null if the key contains only one dot (i.e. 'section.variable' as opposed to 'section.subsection.variable')

  • A variable name is considered everything from the last dot onwards.

  • A variable value is the param value itself. Repeated values are ok and are considered multivars in gitconfig. These should be presented in yaml as list or its equivalent properties (key.1, key.2).

See more about gitconfig syntax here.

The idea behind ModelOp Center’s git config customization is that any git config variable can be mapped in the following way.

The example desired git config file being the following:

  [http]
    sslVerify = true
  [http "https://example.com/scm/demo/repo.git"]
    sslVerify = false
  [format]
    notes = foo
    notes = bar

Then the expected ModelOp Center configuration would be the following:

Container: "model-manager"
  model-manage.git.config.http.sslVerify=true
  model-manage.git.config.http.https://example.com/scm/demo/repo.git.sslVerify=false
  model-manage.git.config.format.notes.0=foo
  model-manage.git.config.format.notes.1=bar

Please refer to git-scm for more comprehensive list of variables that can be set through git config.

Load on startup DEMO ONLY

Optionally you can configure to automatically import upon startup, mainly for demo purposes.

  model-manage.git.load-on-startup.repos.0.repositoryRemote="https://github.com/modelop/first-repo.git"
  model-manage.git.load-on-startup.repos.0.repositoryBranch="master"
  model-manage.git.load-on-startup.repos.1.repositoryRemote="https://github.com/modelop/second-repo.git"
  model-manage.git.load-on-startup.repos.1.repositoryBranch="master"
  ...

View asset details

ModelOp Center Dashboard

You can view the model’s assets repository configuration in the ModelOp Center Dashboard

  1. Click on the “Models” menu item.

  2. From the list on the main panel, select the desired model to inspect.

  3. From the sub-menus on the left, click on “<> Sources”

  4. On the main panel, select the “Metadata” tab on the title

  5. On the left panel, you can inspect the various source files as needed.

Jupyter Notebook plugin

After having loaded a ModelOp Center model on Jupyter Notebook by either registering or opening a model, you can view a ModelOp specific Cell Toolbar button.

  1. On Jupyter click on the menu “View”

  2. Click on “Cell Toolbar”

  3. Select the “ModelOp Model” toolbar option.

  4. On the desired model asset (cell) click on the toolbar button “Asset Details”. This button will be visible for ModelOp registered models only.

    1. If the selected asset didn’t have repository configured previously, it will allow the user to configure it, similarly as when registering the model for the first time.

    2. If the selected asset already had a repository configured it will allow to modify the “Remote URL” and “Branch” for the asset.

v2.1 Register a Model

  • No labels