What's New in ML Runtimes version 2024.05.1

Review the major features and updates for Machine Learning Runtimes.

This release is available with Machine Learning Runtimes version 2024.05.1.

New features

Cloudera Copilot is an AI-powered coding assistant designed for seamless integration within JupyterLab Machine Learning Runtimes. With its chat interface and comprehensive code completion features, Cloudera Copilot enhances the development experience for machine learning projects. It offers compatibility with both custom models deployed in Cloudera AI Inference Service and Amazon Bedrock models, providing developers with flexibility and efficiency in their workflows.
JupyterLab is upgraded to 4.1.5 for Python 3.8 and for higher releases.
Workbench R 4.4 Runtime is available.
JupyterLab, PBJ Workbench Editor, and Nvidia GPU Runtime variants with Python 3.8 and higher versions are upgraded from CUDA 11.8 and cuDNN 8.7 versions to CUDA 12.3 and cuDNN 9.0 versions. It is recommmended to install TensorFlow with the [and-cuda] option, to install TensorFlow's NVIDIA CUDA library dependencies as well, as the current latest TensorFlow version (2.16.1) still has dependencies on cuDNN8.

Due to the known TensorFlow issue https://github.com/tensorflow/tensorflow/issues/63362 for installing TensorFlow 2.16.1 it is also needed to set the LD_LIBRARY_PATH environment variable to /home/cdsw/.local/lib/python3.X/site-packages/nvidia/cudnn/lib/:$LD_LIBRARY_PATH where python3.X stands for the used Python version, for example python3.9.

Python was missing from Scala Runtime, which could lead to warnings or the Scala engine not starting at all. Python interpreter is now added to Scala Runtime.
With the upgrade of the PY4J library version to 0.10.9.5, the SPARK-37004 bug in Spark workloads is fixed, making those workloads more robust against kernel interruptions.

This release includes numerous improvements related to Common Vulnerability and Exposures (CVE).