Chapter 9. Using the YARN REST APIs to Manage Applications

This chapter describes how to use the YARN REST APIs to submit, monitor, and kill applications.

Get an Application ID

You can use the New Application API to get an application ID, which can then be used to submit an application. For example:

curl -v -X POST 'http://localhost:8088/ws/v1/cluster/apps/new-application'

The response returns the application ID, and also includes the maximum resource capabilities available on the cluster. For example:

    {
    application-id: application_1409421698529_0012", 
    "maximum-resource-capability":{"memory":"8192","vCores":"32"}
    } 
  

Set Up an Application .json File

Before you submitting an application, you must set up a .json file with the parameters required by the application. This is analogous to creating your own ApplicationMaster. The application .json file contains all of the fields you are required to submit in order to launch the application.

The following is an example of an application .json file:

 { 
    "application-id":"application_1404203615263_0001", 
    "application-name":"test", 
    "am-container-spec":
    { 
       "local-resources":
       { 
          "entry":
         [
            { 
               "key":"AppMaster.jar", 
               "value":
               { 
                  "resource":"hdfs://hdfs-namenode:9000/user/testuser/DistributedShell/demo-app/AppMaster.jar", 
                  "type":"FILE", 
                  "visibility":"APPLICATION", 
                  "size": "43004", 
                  "timestamp": "1405452071209"
               }
            }
          ]
       }, 
       "commands":
       { 
          "command":"{{JAVA_HOME}}/bin/java -Xmx10m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 10 --container_vcores 1 --num_containers 1 --priority 0 1><LOG_DIR>/AppMaster.stdout 2><LOG_DIR>/AppMaster.stderr"
       }, 
       "environment":
       { 
          "entry":
          [ 
             { 
                "key": "DISTRIBUTEDSHELLSCRIPTTIMESTAMP", 
                "value": "1405459400754"
             },
             {
                "key": "CLASSPATH", 
                "value": "{{CLASSPATH}}<CPS>./*<CPS>{{HADOOP_CONF_DIR}}<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/*<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*<CPS>./log4j.properties"
             },
             { 
                "key": "DISTRIBUTEDSHELLSCRIPTLEN", 
                "value": "6"
             },
             { 
                "key": "DISTRIBUTEDSHELLSCRIPTLOCATION", 
                "value": "hdfs://hdfs-namenode:9000/user/testuser/demo-app/shellCommands"
             } 
          ]
       }
    }, 
    "unmanaged-AM":"false", 
    "max-app-attempts":"2", 
    "resource":
    { 
       "memory":"1024", 
       "vCores":"1"
    }, 
    "application-type":"YARN", 
    "keep-containers-across-application-attempts":"false"
  }
 

Submit an Application

You can use the Submit Application API to submit applications. For example:

curl -v -X POST -d @example-submit-app.json -H "Content-type: application/json"'http://localhost:8088/ws/v1/cluster/apps'

After you submit an application the response includes the following field:

HTTP/1.1 202 Accepted

The response also includes the Location field, which you can use to get the status of the application (app ID). The following is an example of a returned Location code:

Location: http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012

Monitor an Application

You can use the Application State API to query the application state. To return only the state of a running application, use the following command format:

curl 'http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012/state'

You can also use the value of the Location field (returned in the application submission response) to check the application status. For example:

curl -v 'http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012'

You can use the following command format to check the logs:

yarn logs -appOwner 'dr.who' -applicationId application_1409421698529_0012 | less

Kill an Application

You can also use the Application State API to kill an application by using a PUT operation to set the application state to KILLED. For example:

curl -v -X PUT -d '{"state": "KILLED"}''http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012'

Access the Apache YARN REST API Specification

For more information, see the Apache YARN REST APIs documentation.