Options to Run Distributed Shell and GPU

You can run the distributed shell by specifying resources other than memory and vCores.

• Use the following command to run the distributed shell and GPU without a Docker container:

  yarn jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
    -jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
    -shell_command /usr/local/nvidia/bin/nvidia-smi \
    -container_resources memory-mb=3072,vcores=1,yarn.io/gpu=2 \
    -num_containers 2

  You receive output similar to the following:

  +-----------------------------------------------------------------------------+
  | NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
  |-------------------------------+----------------------+----------------------+
  | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
  | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
  |===============================+======================+======================|
  |   0  Tesla P100-PCIE...  Off  | 0000:04:00.0     Off |                    0 |
  | N/A   30C    P0    24W / 250W |      0MiB / 12193MiB |      0%      Default |
  +-------------------------------+----------------------+----------------------+
  |   1  Tesla P100-PCIE...  Off  | 0000:82:00.0     Off |                    0 |
  | N/A   34C    P0    25W / 250W |      0MiB / 12193MiB |      0%      Default |
  +-------------------------------+----------------------+----------------------+
  
  +-----------------------------------------------------------------------------+
  | Processes:                                                       GPU Memory |
  |  GPU       PID  Type  Process name                               Usage      |
  |=============================================================================|
  |  No running processes found                                                 |
  +-----------------------------------------------------------------------------+
  

• Use the following command to run the distributed shell and GPU with a Docker container:

  yarn jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
         -jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
         -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker \
         -shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=<docker-image-name> \
         -shell_command nvidia-smi \
         -container_resources memory-mb=3072,vcores=1,yarn.io/gpu=2 \
         -num_containers 2