Configuring YARN Docker Containers Support

You can configure YARN to run Docker containers.

Docker containerization provides isolation and enables you to run multiple versions of the same applications side-by-side. For example, you can have a stable production version of an application while you evaluate a test version.

On the other hand, using Docker containers introduces a new layer of virtualization, thus creates some overhead compared to regular containers.

Running Docker containers on YARN works very similar to running regular containers. Containers have access to files that are localized for the container as well as logging.

To facilitate the use of YARN features, the following rules need to be followed:

The processes in the containers must run as the user submitting the application (or the local-user in insecure mode).
The mount allowlist must include the yarn.local.dirs so that the files needed for the application are available in the container. This is ensured by Cloudera Manager.