Manage Docker containers on YARN
Docker containerization makes it easier to package and distribute applications, allowing you to focus on running and fine-tuning applications. It also significantly reduces the ‘time-to-deployment’ and the ‘time-to-insight’. You can configure YARN to manage Docker containers.
Docker containerization provides isolation and enables you to run multiple versions of the same applications side-by-side. For example, you can have a stable production version of an application while you evaluate a test version.
On the other hand, using Docker containers introduces a new layer of virtualization, thus creates some overhead compared to regular containers.
Running Docker containers on YARN works very similar to running regular containers. Containers have access to files that are localized for the container as well as logging.
The processes in the containers must run as the user submitting the application (or the local-user in insecure mode).
The mount whitelist must include the yarn.local.dirs so that the files needed for the application are available in the container. This is ensured by Cloudera Manager.