ML Runtimes Known Issues and Limitations
You might run into some known issues while using ML Runtimes.
Adding a new ML Runtimes when using a custom root certificate might generate error messages
When trying to add new ML Runtimes, a number of error messages might appear in various places when using a custom root certificate. For example, you might see: "Could not fetch the image metadata." certificate signed by unknown authority". This is caused by the runtime-puller pods not having access to the custom root certificate that is in use.
- Create a directory at any location and copy the certificate information to that
#mkdir -p /certs/
- Copy your customer docker registry certificates and place them under the path which
you created in the previous step and include all your global ca Certificates as well.
The default path should be similar to:
# cp -rp <your custom docker registry Certificates> /certs/ca.crt # cat /etc/ssl/certs/ca-bundle.crt >> /certs/ca.crt #chmod 777 -R /certs
- Edit your deployment of runtime manager and add the new mount.
Do not delete any existing objects.
#kubectl edit deployment runtime-manager
- Under VolumeMounts add the following lines. Note: The text is space sensitive.
- mountPath: /etc/ssl/certs/ca-certificates.crt name: mycert subPath: ca.crt
Under Volumes add the following text in the same edit:
- hostPath: path: /certs/ type: "" name: mycert
- Save your changes:
Once saved, you will receive the message "deployment.apps/runtime-manager edited" and the pod will be restarted with your new changes.
# more /etc/cdsw/patches/default/deployments/runtime-manager.yaml apiVersion: apps/v1 kind: Deployment metadata: name: runtime-manager namespace: default spec: template: spec: containers: - name: runtime-manager volumeMounts: - mountPath: /etc/ssl/certs/ca-certificates.crt <----- do not modify, this is the path inside the pod name: mycert subPath: ca.crt <----- filename on each hosts (as the pod has no nodeaffinity) volumes: - hostPath: path: /var/tmp <----- directory name on each hosts type: "" name: mycert
- Create the following path if it does not
- Place the file runtime-manager.yaml in the path you created to make the changes permanent after restart.
Cloudera Bug: DSE-20530
Starting a worker from Runtime session will create a worker with engine image
This issue is resolved in ML Runtimes 2021.09 when used with the latest ML Workspace versions.
For workers to function properly with ML Runtimes, please use ML Runtimes 2021.09 or later with CML Workspace version of 2.0.22 or later.
Cloudera Bug: DSE-17126
Disable Scala runtimes in models, experiments and applications runtime selection
Scala Runtimes should not appear as an option for Models, Experiments, and Applications in the user interface. Currently Scala Runtimes only support Session and Jobs.
Cloudera Bug: DSE-17981
Some bugs present for R in the legacy engine persist in ML Runtimes
Some bugs that were present for R in the legacy engine persist in ML runtimes.
Cloudera Bug: DSE-14447
Code completion in Workbench editor does not work in ML Runtimes
Code completion in the Workbench editor does not work when using Runtimes. It does work in the console on the right hand side, and it does work in both the editor and console when using Legacy Engines.
Limitations for Spark support in ML Runtimes
- ML Runtimes running Python 3.8 kernel does not support running Spark 2.x and Spark 1.x.