Known Issues

You might run into some known issues while using Cloudera Machine Learning on Private Cloud.

On-premise CDSW incompatibility with Python 3-based Cloudera Manager

Not all versions of the on-premise Cloudera Data Science Workbench (CDSW) are compatible with Python 3-based Cloudera Manager (CM). As Private Cloud Data Services (PvC DS) 1.5.2 requires CM 7.11.3 which is Python 3-based, it is recommended not to upgrade an older version of CM to 7.11.3 (or other Python 3-based CMs) that has a running instance of on-premise CDSW.

Upgrade CDSW to CDSW 1.10.5, as it is Python 3-based and works with Python 3 CM. Please refer to CDSW 1.10.5 documentation for more details.

DSE-32887: Runtime Updates can not be enabled from the Site administration UI on PVC

With this bug present on PVC 1.5.2 and 1.5.3, Runtime repositories added on the Site Administration>Runtimes page will not be scanned for new Runtimes. Adding Runtime repositories to CML was a new feature in 1.5.2. Unfortunately with this bug, the feature is completely unusable.

Manually enable Runtime updates with an API call using the legacy API key of the admin user. That will let CML check the registered runtime registries for new runtimes every 24 hours. The syntax of the API call is as follows:

curl 'https://<CML_URL>/api/v1/site/config' \
-X 'PATCH' \
-u <ADMIN_APIv1_KEY>: \
-H 'content-type: application/json' \ 
-d '{"enable_runtime_updates":true}'

DSE-33636: Workloads unable to start up after changing default hadoopCLI addon

Changing the default Hadoop CLI Runtime Addon causes jobs, models, and application workloads to be unable to start up.

Open affected workload settings.
Update the workload (this updates the Hadoop CLI Addon associated with the workload to the default one.)
For Jobs: update.
For Applications: update and restart.
For Models: deploy a new build.

Please see Machine Learning Site administration>Disable Addons for related tasks.

ENGSEC-22794: CML Sessions fail randomly with status 34

CML Sessions fail randomly with status 34 for fresh CML PVC 1.5.2 workspace installs with Base cluster timezone set to a non-UTC timezone.

Edit the CML workspace web deployment to force the timezone to UTC:

kubectl edit deployments web -n mlx 

- command:
        - /usr/bin/bash
        - -c
        - TZ=UTC /home/cdswint/services/web/pod-bootstrap.sh 2>&1 | tee /var/fluentbit/log/web.log

Note the TZ=UTC before pod-bootstrap.sh command

OPSX-4603: Buildkit in ECS in CML PrivateCloud

Issue: This release introduces BuildKit in ECS for building images of models and experiments. BuildKit is a replacement for Docker, which was previously used to build images of CML's models and experiments in ECS. Buildkit will be only for OS RHEL8.x and CentOS 8.x.

Buildkit in CML Private Cloud 1.5.2 is a Tech Preview feature. Hence, having Docker installed on the nodes/hosts is still mandatory for models and experiments to work smoothly. Upcoming release will be completely eliminating the dependency of Docker on the nodes.

None.

DSE-32285: Migration: Migrated models are failing due to image pull errors

Issue: After CDSW to CML migration (on-premises) via full-fledged migration tool, migrated models on CML Private Cloud Workspace fails on initial deployment. This is because the initial model deployment tries to pull images from on-premises's registry.

: Re-deploy the migrated model. As this involves the build and deploy process, the image will be built, pushed to the CML Private Cloud Workspace's configured registry, and then the same image will be consumed for further usage.

DSE-28768: Spark Pushdown is not working with Scala 2.11 runtime

Issue: Scala and R are not supported for Spark Pushdown.

None.

DSE-32304 On CML Private Cloud ECS terminal and ssh connections can terminate

Issue: In Private Cloud ECS, CML Terminal and SSH connections can terminate after an uncertain amount of time, usually after 4-10 minutes. This issue affects customers using local IDEs to work with CML, as well as any customer application using a websocket connection.

None.