Cloudera Docs

ML Runtimes What's New

This section lists major features and updates for Machine Learning (ML)
 Runtimes.

This release is available with ML Runtimes version 2024.05.1.

New Features

  • Cloudera Copilot is an AI-powered coding assistant designed for seamless integration 
 within JupyterLab ML Runtimes. With its chat interface and comprehensive code completion
 features, Cloudera Copilot enhances the development experience for machine learning
 projects. It offers compatibility with both custom models deployed in Cloudera AI
 Inference Service and Amazon Bedrock models, providing developers with flexibility and
 efficiency in their workflows.
  • JupyterLab is upgraded to 4.1.5 for Python 3.8 and for higher releases.
  • Workbench R 4.4 Runtime is available.

  • JupyterLab, PBJ Workbench Editor, and Nvidia GPU Runtime variants with Python 3.8 and
 higher versions are upgraded from CUDA 11.8 and cuDNN 8.7 versions to CUDA 12.3 and
 cuDNN 9.0 versions. It is recommmended to install TensorFlow with the [and-cuda] option, to install TensorFlow's NVIDIA CUDA library
 dependencies as well, as the current latest TensorFlow version (2.16.1) still has 
dependencies on cuDNN8.

    Due to the known TensorFlow issue https://github.com/tensorflow/tensorflow/issues/63362, for installing TensorFlow 2.16.1 it is also needed to set the 
LD_LIBRARY_PATH environment variable to
 /home/cdsw/.local/lib/python3.X/site-packages/nvidia/cudnn/lib/:$LD_LIBRARY_PATH
 where python3.X stands for the used Python version, for example
 python3.9.

    


Fixed issues

  • Python was missing from Scala Runtime, which could lead to warnings or the Scala engine
 not starting at all. Python interpreter is now added to Scala Runtime.
  • With the upgrade of the PY4J library version to 0.10.9.5, the SPARK-37004 bug in Spark
 workloads is fixed, making those workloads more robust against kernel interruptions.

Improvements

  • This release includes numerous improvements related to Common Vulnerability and
 Exposures (CVE).