Managing Data Operating System
Also available as:
PDF
loading table of contents...

Options to Run Distributed Shell and GPU

You can run the distributed shell by specifying resources other than memory and vCores.

  • Use the following command to run the distributed shell and GPU without a Docker container:
    yarn jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
      -jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
      -shell_command /usr/local/nvidia/bin/nvidia-smi \
      -container_resources memory-mb=3072,vcores=1,yarn.io/gpu=2 \
      -num_containers 2
    You receive output similar to the following:
    +-----------------------------------------------------------------------------+
    | NVIDIA-SMI 375.66                 Driver Version: 375.66                    |
    |-------------------------------+----------------------+----------------------+
    | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
    | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
    |===============================+======================+======================|
    |   0  Tesla P100-PCIE...  Off  | 0000:04:00.0     Off |                    0 |
    | N/A   30C    P0    24W / 250W |      0MiB / 12193MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+
    |   1  Tesla P100-PCIE...  Off  | 0000:82:00.0     Off |                    0 |
    | N/A   34C    P0    25W / 250W |      0MiB / 12193MiB |      0%      Default |
    +-------------------------------+----------------------+----------------------+
    
    +-----------------------------------------------------------------------------+
    | Processes:                                                       GPU Memory |
    |  GPU       PID  Type  Process name                               Usage      |
    |=============================================================================|
    |  No running processes found                                                 |
    +-----------------------------------------------------------------------------+
    
  • Use the following command to run the distributed shell and GPU with a Docker container:
    yarn jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
           -jar <path/to/hadoop-yarn-applications-distributedshell.jar> \
           -shell_env YARN_CONTAINER_RUNTIME_TYPE=docker \
           -shell_env YARN_CONTAINER_RUNTIME_DOCKER_IMAGE=<docker-image-name> \
           -shell_command nvidia-smi \
           -container_resources memory-mb=3072,vcores=1,yarn.io/gpu=2 \
           -num_containers 2