How YARN mounts an external volume using CSI

To access an external volume on a Docker container in YARN, your application must specify information about the external volume as a resource in a service specification file and submit the same to YARN using the YARN services API.

The following steps explain how YARN receives the request to access an already provisioned external volume, launches a Docker container, and mounts the external volume:

1. YARN receives the service specification JSON file containing the volume resource information through the YARN services API.

   The following example is an excerpt from a specification file that requests a vcore, 1024 MB of memory, and a 1-GB volume from an external Ozone storage system that interacts with YARN through an S3 API:

   
       "resource": {
         "cpus": 1,
         "memory": "1024",
         "additional": {
           "yarn.io/csi-volume": {
             "value": 1,
             "unit": "Gi",
             "attributes" : {
               "volume.id" : "csi-volume-01",
               "driver.name" : "ch.ctrox.csi.s3-driver",
               "volume.mount" : "/mnt/data",
               "access.mode": "SINGLE_NODE_WRITTER",
               "read.only" : "false"
             }
           }
         }
       }
       

   In this example, csi-volume-01 is the name of the volume provisioned in Ozone, ch.ctrox.csi.s3-driver is the CSI driver, and /mnt/data is the expected mount point for the volume in the YARN Docker container.

2. The ResourceManager receives the resource request with the volume resource information. The ResourceManager interacts with the CSI driver on the cluster node running the controller service to verify whether YARN can meet the capacity requirements of the external volume.
3. If YARN can meet the capacity requirements of the requested volume, the ResourceManager allocates a container for the request.
4. The ApplicationMaster launches the container for the volume on the assigned NodeManager.
5. Before starting the container, the NodeManager interacts with the CSI driver to publish the requested volume on the corresponding node and binds the volume to the Docker container through the specified mount point.
6. The container starts, which enables the client applications to access data in the external volume through the specified mount path.