Using Conda with Cloudera Data Science Workbench

Cloudera Data Science Workbench recommends using pip for package management along with a requirements.txt file (as described in the previous section).

Cloudera Data Science Workbench recommends using pip for package management along with a requirements.txt file (as described in the previous section). However, for users that prefer Conda, the default engine in Cloudera Data Science Workbench includes two environments called python2.7, and python3.6. These environments are added to sys.path, depending on the version of Python selected when you launch a new session.

In Python 2 and Python 3 sessions and attached terminals, Cloudera Data Science Workbench automatically sets the CONDA_DEFAULT_ENV and CONDA_PREFIX environment variables to point to Conda environments under /home/cdsw/.conda.

However, Cloudera Data Science Workbench does not automatically configure Conda to pin the actual Python version. Therefore if you are using Conda to install a package, you must specify the version of Python. For example, to use Conda to install the feather-format package into the python3.6 environment, run the following command in the Workbench command prompt:

!conda install -y -c conda-forge python=3.6.1 feather-format

To install a package into the python2.7 environment, run:

!conda install -y -c conda-forge python=2.7.11 feather-format

Note that on sys.path, pip packages have precedence over conda packages.

note

If your project is using an older base engine image (version 3 and lower), you will need to specify both the Python version as well as the Conda environment. For example:
```
!conda install -y -c conda-forge --name python3.6 python=3.6.1 feather-format
```
The Conda environment is also required when you create an extensible engine using Conda (as described in the following section).
Cloudera Data Science Workbench does not automatically configure a Conda environment for R and Scala sessions and attached terminals. If you want to use Conda to install packages from an R or Scala session or terminal, you must manually configure Conda to install packages into the desired environment.