Customized engine images

This topic explains how custom engines work and when they shall be used.

By default, Cloudera AI engines are preloaded with a few common packages and libraries for R, Python, and Scala. In addition to these, Cloudera AI also allows you to install any other packages or libraries that are required by your projects. However, directly installing a package to a project as described above might not always be feasible. For example, packages that require root access to be installed, or that must be installed to a path outside /home/cdsw (outside the project mount), cannot be installed directly from the workbench.

For such circumstances, Cloudera AI allows you to extend the base Docker image and create a new Docker image with all the libraries and packages you require. Site administrators can then include this new image in the allowlist for use in projects, and project administrators set the new white-listed image to be used as the default engine image for their projects. For an end-to-end example of this process, see End-to-End Example: MeCab.

Note that this approach can also be used to accelerate project setup across the deployment. For example, if you want multiple projects on your deployment to have access to some common dependencies (package or software or driver) out of the box, or even if a package just has a complicated setup, it might be easier to simply provide users with an engine that has already been customized for their project(s).

Related Resources

• The Cloudera Engineering Blog post on Customizing Docker Images in Cloudera Maching Learning describes an end-to-end example on how to build and publish a customized Docker image and use it as an engine in Cloudera AI.
• For an example of how to extend the base engine image to include Conda, see Installing Additional Packages.