Installing Additional ML Runtimes Packages
Cloudera Machine Learning Runtimes are preloaded with a few common packages and libraries for Python. However, a key feature of Cloudera Machine Learning is the ability of different projects to install and use libraries pinned to specific versions, just as you would on your local computer.
Generally, Cloudera recommends you install all required packages locally into your project. This will ensure you have the exact versions you want and that these libraries will not be upgraded when Cloudera upgrades the ML Runtimes image. You only need to install libraries and packages once per project. From then on, they are available to any new ML Runtimes you spawn throughout the lifetime of the project.
You can install additional libraries and packages from the workbench, using either the command prompt or the terminal.
- Navigate to your project's Overview page. Click Open Workbench and launch a session.
- At the command prompt (see Native Workbench Console and Editor) in the bottom right, enter the command to install the package. Some examples using Python have been provided.
# Installing from console using ! shell operator and pip3: !pip3 install beautifulsoup4 # Installing from terminal pip3 install beautifulsoup4
For a Python project, you can specify a list of the packages you want
requirements.txt file that lives in your project.
The packages can be installed all at once using pip/pip3.
- Create a new file called
requirements.txtfile within your project:
- To install the packages in a Python 3 ML Runtimes, run the following command in the
workbench command prompt.
!pip3 install -r requirements.txt