Upgrading from MOC 2.5 to Helm-Based Installs

Owned by Mark LeMonnier
Apr 19, 2023
6 min read

Upgrading moc from 2.5 → 3.1.4

  1. Used moc-builder (version 4) to provision and deploy 2.5
    Steps taken:
    - in the moc-builder folder executed
    python3 src/moc_builder.py env create --fresh oauth-2.5
    cp resources/external_pf.crt environments/oauth-2.5/certs
    - Copied the file config-2.5.yaml to environments/oauth-2.5/ renaming to config.yaml and overwriting the existing file
    - Executed the command to compose:
    python3 src/moc_builder.py manifest compose oauth-2.4 docker-desktop --oauth --services jenkins jira jupyter rstudio --data minio mongodb mysql postgres --log-level DEBUG --ootb latest
    - Finally did an install with:
    python3 src/moc_builder.py install deploy oauth-2.5 -f --log-level DEBUG
    - Added the following indexes to mongoDB:
    db.job.createIndex( { createdDate: 1 } )

    db.job.createIndex( { jobStatus: 1 } )

    db.job.createIndex( { jobType: 1 } )

    db.job.createIndex( { "model.id": 1 } )

    db.job.createIndex( { "referenceModel.id": 1 } )

      db.notification.createIndex( { createdDate: 1 } )

      db.modelTestResult.createIndex( { "deployedModel.deployableModel.id" : 1 } )

    db.modelTestResult.createIndex( { "job.id" : 1 } )

    db.modelTestResult.createIndex( { createdDate : 1 } )

      db.deployableModel.createIndex( { "storedModel.id": 1 } )

    db.deployableModel.createIndex( { createdDate: 1 } )

      db.storedModels.createIndex( { createdDate: 1 } )
    - Ran some integration tests logged in and tested to make sure things were ok.

  2. Deleted all deployments except for MongoDB and Postgres
    kubectl delete deployments engine-1 engine-2   engine-test  gateway jenkins jira jupyter minio mlc-manager model-manager mysql registry reporting-service rstudio spark-runtime-1

  3. Upgrade to 3.1.4
    Backup

    Database Backup
    Before you begin the upgrade process you will need to first backup your current Mongo and PostgreSQL databases. Please see instructions here on how to backup your databases:
    How To Backup/Restore ModelOp Center's Databases

    K8 Resources backup
    Back up your current environment be executing:
    kubectl get deployments,services,configmaps,secrets -l app=modelop-center -o yaml > moc-config-backup.yaml

  4. Shut down MOC
    Prior to version 3.0 ModelOp Center was deployed using MOC builder. From 3.0 forward ModelOp center is now deployed using our helm charts.

    As a result, before you can upgrade one must first shutdown the current running instance of ModelOp Center running in your cluster as well as delete all K8 resources e.t.c originally provisioned using MOC builder.

    To shutdown moc execute the following:
    Delete deployments
    kubectl delete deployments engine-1 engine-2 engine-3 engine-4 engine-5 engine-6 engine-7 engine-8 gateway jenkins jira jupyter minio mlc-manager registry reporting-service rstudio spark-runtime-1 model-manager

    Delete configMaps
    kubectl delete configMap ootb

    Delete Services
    kubectl delete services engine-1 engine-2 engine-3 engine-4 engine-5 engine-6 engine-7 engine-8 gateway jenkins jira jupyter minio mlc-manager registry reporting-service rstudio spark-runtime-1 model-manager  

  5. Execute DB Migration script
    When upgrading MOC from 2.5 -> 3.1.4 an update of Camunda from 7.13 to 7.17 is required
    To Migrate Camunda from 7.13 to 7.17, we would need to execute the available database patch script on your postgresql db: postgres-camunda-schema-upgrade.sql

    In addition you would also need to execute the following DB script on your mongoDB database:
    db.deployableModel.aggregate(

         [

             { $project: { "mlc" : { $map: { input: "$modelMLCS", as: "mlc", in: {$mergeObjects: "$$mlc" } } }, "deployableModelId": "$_id"} },

             { $match: { "mlc.0": { $exists: true } } },

             { $unwind: "$mlc" },

             { $project: { "_id": 0, "processInstance": "$mlc.processInstance", "activityInstances": "$mlc.activityInstances", "deployableModelId" : "$deployableModelId", "historyType" : "DEPLOYABLE_MODEL_HISTORY"} },

             { $out: "tempDeployableModelMLC" }

         ])

     

    db.jobHistory.aggregate(

         [

             { $project: { "mlc" : { $map: { input: "$jobMLCS", as: "mlc", in: {$mergeObjects: "$$mlc" } } }, "jobId": "$jobId"} },

             { $match: { "mlc.0": { $exists: true } } },

             { $unwind: "$mlc" },

             { $project: { "_id": 0, "processInstance": "$mlc.processInstance", "activityInstances": "$mlc.activityInstances", "jobId" : "$jobId", "historyType" : "JOB_HISTORY"} },

             { $out: "tempJobMLC" }

         ])

     

    db.deployedModel.aggregate(

         [

             { $match: { "modelLifecycleProcessInstance": { $exists: true }}},

             { $project: { "_id": 0, "processInstance": "$modelLifecycleProcessInstance", "activityInstances": "$modelLifecycleActivityInstances", "deployedModelId": "$_id", "historyType": "DEPLOYED_MODEL_HISTORY"}},

             { $out: "tempDeployedModelMLC"}

         ]

    )

     

    db.tempDeployableModelMLC.find().forEach(function (mlc) {

         db.modelMLC.insert(mlc)

    })

    db.tempDeployedModelMLC.find().forEach(function (mlc) {

         db.modelMLC.insert(mlc)

    })

    db.tempJobMLC.find().forEach(function (mlc) {

         db.modelMLC.insert(mlc)

    })

  6. Install Modelop Center using helm
    Next step would be to install helm. See the following link on steps to install ModelOp Center using helm:
    Installing ModelOp Center Using Helm

  7. Update helm overrides file
    When installing with helm you have the ability to configure modelOp center. Put in the appropriate configuration settings related to your environment as well as specifying the version to install 3.1.4

  8. MOC Validation
    Once MOC comes up login and validate things work as expected.

 


Upgrading moc from 2.4.1-rc-11 → 2.5

  1. Used moc-builder (version 4) to provision and deploy 2.4
    Steps taken:
    - in the moc-builder folder executed
    python3 src/moc_builder.py env create --fresh oauth-2.4
    cp resources/external_pf.crt environments/oauth-2.4/certs
    - Copied the file config-2.4.yaml to environments/oauth-2.4/ renaming to config.yaml and overwriting the existing file
    - Executed the command to compose:
    python3 src/moc_builder.py manifest compose oauth-2.4 docker-desktop --oauth --services jenkins jira jupyter rstudio --data minio mongodb mysql postgres --log-level DEBUG --ootb latest
    - Finally did an install with:
    python3 src/moc_builder.py install deploy oauth-2.4 -f --log-level DEBUG
    - Added the following indexes to mongoDB:
    db.job.createIndex( { createdDate: 1 } )

    db.job.createIndex( { jobStatus: 1 } )

    db.job.createIndex( { jobType: 1 } )

    db.job.createIndex( { "model.id": 1 } )

    db.job.createIndex( { "referenceModel.id": 1 } )

      db.notification.createIndex( { createdDate: 1 } )

      db.modelTestResult.createIndex( { "deployedModel.deployableModel.id" : 1 } )

    db.modelTestResult.createIndex( { "job.id" : 1 } )

    db.modelTestResult.createIndex( { createdDate : 1 } )

      db.deployableModel.createIndex( { "storedModel.id": 1 } )

    db.deployableModel.createIndex( { createdDate: 1 } )

      db.storedModels.createIndex( { createdDate: 1 } )
    - Ran some integration tests logged in and tested to make sure things were ok.

  2. Deleted all deployments except for MongoDB and Postgres
    kubectl delete deployments engine-1 engine-2   engine-test  gateway jenkins jira jupyter minio mlc-manager model-manager mysql registry reporting-service rstudio spark-runtime-1

  3. Upgrade to 2.5
    Executed the command:
    python3 src/moc_builder.py env create --fresh oauth-2.5
    cp resources/external_pf.crt environments/oauth-2.5/certs
    - Copied the file config-2.5.yaml to environments/oauth-2.5/ renaming to config.yaml and overwriting the existing file
    - executed the command:
    python3 src/moc_builder.py manifest compose oauth-2.5 docker-desktop  --oauth  --services jenkins jira jupyter rstudio --data minio mongodb mysql postgres  --log-level DEBUG --ootb latest
    * Since we want to keep the mongoDB we need to modify the mongodb.yaml file in environments/oauth-2.5/manifests/ and override the existing file that was created.

    Copy mongodb.yaml environments/oauth-2.5/manifests/
    - Finally install with:
    python3 src/moc_builder.py install deploy oauth-2.5 -f --log-level DEBUG

    - Added the following indexes to mongoDB:
    db.deployableModel.createIndex( { createdDate: -1, group : -1 }, { name: "createdDate_-1_group_-1" } )

    db.deployableModel.createIndex( { "modelMLCS.processInstance.rootProcessInstanceId": 1, group : 1 }, { name: "modelMLCS.processInstance.rootProcessInstanceId_1_group_1" } )

    db.deployableModel.createIndex( { "associatedModels.schedules.active": 1, "lastModifiedDate" : 1, group : 1 }, { name: "associatedModels.schedules.active_1_lastModifiedDate_1_group_1" } )

     

    db.deployedModel.createIndex( { createdDate: -1, group : -1 }, { name: "createdDate_-1_group_-1" } )

    db.deployedModel.createIndex( { "engine.name": 1, "state" : 1, group : 1 }, { name: "engine.name_1_state_1_group_1" } )

    db.deployedModel.createIndex( {  group : 1 }, { name: "group_1" } )

    db.deployedModel.createIndex( { deployTime: -1, group : -1 }, { name: "deployTime_-1_group_-1" } )

     

    db.job.createIndex( {  "model.id" : 1, group : 1 }, { name: "modelId_and_group" } )

    db.job.createIndex( {  "referenceModel.id" : 1, group : 1 }, { name: "referenceModel_and_group" } )

    db.job.createIndex( {  createdDate : -1, group : 1 }, { name: "createdDate_and_group" } )

    db.job.createIndex( {  jobStatus : 1, group : 1 , jobType : 1,deployableModelId : 1 }, { name: "status_group_type_deployableModel" } )

     

    db.jobHistory.createIndex( {  jobId : 1, group : 1 }, { name: "jobId_and_group" } )

    db.jobHistory.createIndex( {  "jobMLCS.processInstance.rootProcessInstanceId" : 1, group : 1 }, { name: "rootProcId_and_group" } )

    db.jobHistory.createIndex( {  group : 1 }, { name: "group" } )

  4. Login and you should now be running 2.5.
    - Performed some tests.