Using the YARN REST APIs to Manage Applications

Important

In a non-secure cluster, you must append a request with ?user.name=<user>. Example: Get application data Without ?user.name=<user>: curl http://localhost:19888/jobhistory/job/job_1516861688424_0001 Access denied: User null does not have permission to view job job_1516861688424_0001 With ?user.name=<user>: curl http://localhost:19888/jobhistory/job/job_1516861688424_0001?user.name=hrt_1 {"job":{"submitTime":1516863297896,"startTime":1516863310110,"finishTime":1516863330610, "id":"job_1516861688424_0001","name":"Sleepjob","queue":"default","user":"hrt_1", "state":"SUCCEEDED","mapsTotal":1,"mapsCompleted":1,"reducesTotal":1,"reducesCompleted":1, "uberized":false,"diagnostics":"","avgMapTime":10387,"avgReduceTime":536,"avgShuffleTime":4727, "avgMergeTime":27,"failedReduceAttempts":0,"killedReduceAttempts":0,"successfulReduceAttempts":1, "failedMapAttempts":0,"killedMapAttempts":0,"successfulMapAttempts":1,"acls":[{"name":"mapreduce.job.acl- view-job","value":" "},{"name":"mapreduce.job.acl-modify-job","value":" "}]}}

Get an Application ID

You can use the New Application API to get an application ID, which can then be used to submit an application. For example:

curl -v -X POST 'http://localhost:8088/ws/v1/cluster/apps/new-application'

The response returns the application ID, and also includes the maximum resource capabilities available on the cluster. For example:

    {
    application-id: application_1409421698529_0012", 
    "maximum-resource-capability":{"memory":"8192","vCores":"32"}
    } 
  

Set Up an Application .json File

Before you submit an application, you must set up a .json file with the parameters required by the application. This is analogous to creating your own ApplicationMaster. The application .json file contains all of the fields you are required to submit in order to launch the application.

The following is an example of an application .json file:

 { 
    "application-id":"application_1404203615263_0001", 
    "application-name":"test", 
    "am-container-spec":
    { 
       "local-resources":
       { 
          "entry":
         [
            { 
               "key":"AppMaster.jar", 
               "value":
               { 
                  "resource":"hdfs://hdfs-namenode:9000/user/testuser/DistributedShell/demo-app/AppMaster.jar", 
                  "type":"FILE", 
                  "visibility":"APPLICATION", 
                  "size": "43004", 
                  "timestamp": "1405452071209"
               }
            }
          ]
       }, 
       "commands":
       { 
          "command":"{{JAVA_HOME}}/bin/java -Xmx10m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 10 --container_vcores 1 --num_containers 1 --priority 0 1><LOG_DIR>/AppMaster.stdout 2><LOG_DIR>/AppMaster.stderr"
       }, 
       "environment":
       { 
          "entry":
          [ 
             { 
                "key": "DISTRIBUTEDSHELLSCRIPTTIMESTAMP", 
                "value": "1405459400754"
             },
             {
                "key": "CLASSPATH", 
                "value": "{{CLASSPATH}}<CPS>./*<CPS>{{HADOOP_CONF_DIR}}<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/*<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*<CPS>./log4j.properties"
             },
             { 
                "key": "DISTRIBUTEDSHELLSCRIPTLEN", 
                "value": "6"
             },
             { 
                "key": "DISTRIBUTEDSHELLSCRIPTLOCATION", 
                "value": "hdfs://hdfs-namenode:9000/user/testuser/demo-app/shellCommands"
             } 
          ]
       }
    }, 
    "unmanaged-AM":"false", 
    "max-app-attempts":"2", 
    "resource":
    { 
       "memory":"1024", 
       "vCores":"1"
    }, 
    "application-type":"YARN", 
    "keep-containers-across-application-attempts":"false"
  }
 

Submit an Application

You can use the Submit Application API to submit applications. For example:

curl -v -X POST -d @example-submit-app.json -H "Content-type: application/json"'http://localhost:8088/ws/v1/cluster/apps'

After you submit an application the response includes the following field:

HTTP/1.1 202 Accepted

The response also includes the Location field, which you can use to get the status of the application (app ID). The following is an example of a returned Location code:

Location: http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012

Monitor an Application

You can use the Application State API to query the application state. To return only the state of a running application, use the following command format:

curl 'http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012/state'

You can also use the value of the Location field (returned in the application submission response) to check the application status. For example:

curl -v 'http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012'

You can use the following command format to check the logs:

yarn logs -appOwner 'dr.who' -applicationId application_1409421698529_0012 | less

Kill an Application

You can also use the Application State API to kill an application by using a PUT operation to set the application state to KILLED. For example:

curl -v -X PUT -d '{"state": "KILLED"}''http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012'