Cloudera Docs

Increasing Docker storage

Cloudera requires a dedicated disk for docker (the "docker block device") on every single CDSW node. This this disk should be of size 1TB. You can increase the Docker storage space for CDSW.

If the Docker storage on any of the CDSW nodes reaches 75% capacity or more, you may see issues with pods being evicted from those nodes with error messages including but not limited to "The node had condition: [DiskPressure]." You can view the free space with the docker info command.

To increase the Docker storage space, complete the following steps:

  1. Remove existing docker devices
    1. On all nodes that will get new disks, run the file located at /opt/cloudera/parcels/CDSW/scripts/teardown-docker.sh This will remove all of the exiting docker devices on the host that it is ran on.
  2. Option 1 - Add the same disk on every worker node
    1. Add a new, empty disk to all CDSW worker nodes.
      This disk does not need to be 1TB. Each node should have 1TB of docker space but that can be split across multiple disks.
    2. Ensure the new disks are added to the same location on all CDSW worker nodes, such as: /dev/sdc.
    3. Navigate to Cloudera Manager > CDSW > Configuration > then search for Docker block device and add the new location:
      This location must be the same on every worker node and must have a disk. If this does not happen, one or more of the worker nodes will fail to start.
    4. Save the new setting, then stop CDSW.
    5. Run Prepare Node on all nodes, then restart CDSW from Cloudera Manager. Copy any existing Legacy Engine docker images to all nodes.
  3. Option 2 - Add different disks to different worker nodes
    Using Role Groups (see Role Groups) in Cloudera Manager, you can add disks to worker nodes in different locations.
    1. Navigate to Cloudera Manager > CDSW and add a new Role Group.
      For instance, if you have a set of worker nodes that will use /dev/sdc for the new disk, and another set of worker nodes that will use /dev/vdb, you can create two role groups named "sdc" and "vdb." Ensure that these role groups are assigned to the right set of specific nodes.
    2. Edit the Docker Block device per the above procedure but you can have different disks for different nodes. Example:
      In this example, every worker nodes has a disk in /dev/vdb. The worker nodes that are using the "worker3" role group have an additional disk added at /dev/sde, while the nodes using "workers1-2" role group have the disk at /dev/sdc. Any nodes using the default role group ("Docker daemon default group") have a disk only in /dev/vdb.
    3. Save the new setting, then stop CDSW.
    4. Run Prepare Node on all nodes, then restart CDSW from Cloudera Manager. Copy any existing Legacy Engine docker images to all nodes.