Cloudera Docs

Using Conda Runtime

Users now can create their own Python or R Conda environments within their Cloudera AI Projects that they can use in the JupyterLab editor.

Creating Python Conda environment

Environments, installed packages, and kernel specifications are persisted on the Project file system. An example flow for creating a new Python Conda environment is the following:

  1. Register the Conda environment with the following command:
    conda create --name myenv python=3.10
conda activate myenv
conda install ipykernel
ipython kernel install --user --name=myenv
  2. Restart the session after executing the commands in the terminal and you can create notebooks in the registered Conda environment.

JupyterLab Conda Runtime (Technical Preview)

You might run into some known issues while using JupyterLab Conda Runtime.
Sessions
When starting a Notebook or a Console for a specific environment, the installed packages will be available and the interpreter used to evaluate the contents of the Notebook or Console will be the one installed in the environment. However, the Conda environment is not "activated" in these sessions, therefore commands like !which python will return with the base Python interpreter on the Runtime. The recommended ways to modify a Conda environments or install packages are the following:
  • conda commands must be used with the -n or --name argument to specify the environment, for example conda -n myenv install pandas
  • When installing packages with pip, use the %pip magic to install packages in the active kernel’s environment, for example %pip install pandas
Applications and Jobs
To start an Application or Job, first create a launcher Python script containing the following line: !source activate <conda_env_name> && python <job / application script.py>
When starting the Application or Job, select the launcher script as the "Script".
Models
Models are currently not supported for the Conda Runtime.
Spark
Spark is not supported in JupyterLab Notebooks and Consoles.
Spark workloads are supported in activated Conda environments in JupyterLab Terminals, or in Jobs or Applications.
The CDSW libraries for Python and R are not available for the Conda Runtimes.