Use the YARN REST APIs to manage applications
You can use the YARN REST APIs to submit, monitor, and kill applications.
Get an Application ID
You can use the New Application API to get an application ID, which can then be used to submit an application. For example:
curl -v -X POST 'http://localhost:8088/ws/v1/cluster/apps/new-application'
The response returns the application ID, and also includes the maximum resource capabilities available on the cluster. For example:
{ application-id: application_1409421698529_0012", "maximum-resource-capability":{"memory":"8192","vCores":"32"} }
Set Up an Application .json File
Before you submit an application, you must set up a .json
file with the parameters required by the application. This is analogous to creating your own
ApplicationMaster. The application .json
file contains all of the fields
you are required to submit in order to launch the application.
The following is an example of an application .json
file:
{ "application-id":"application_1404203615263_0001", "application-name":"test", "am-container-spec": { "local-resources": { "entry": [ { "key":"AppMaster.jar", "value": { "resource":"hdfs://hdfs-namenode:9000/user/testuser/DistributedShell/demo-app/AppMaster.jar", "type":"FILE", "visibility":"APPLICATION", "size": "43004", "timestamp": "1405452071209" } } ] }, "commands": { "command":"{{JAVA_HOME}}/bin/java -Xmx10m org.apache.hadoop.yarn.applications.distributedshell.ApplicationMaster --container_memory 10 --container_vcores 1 --num_containers 1 --priority 0 1><LOG_DIR>/AppMaster.stdout 2><LOG_DIR>/AppMaster.stderr" }, "environment": { "entry": [ { "key": "DISTRIBUTEDSHELLSCRIPTTIMESTAMP", "value": "1405459400754" }, { "key": "CLASSPATH", "value": "{{CLASSPATH}}<CPS>./*<CPS>{{HADOOP_CONF_DIR}}<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/*<CPS>{{HADOOP_COMMON_HOME}}/share/hadoop/common/lib/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/*<CPS>{{HADOOP_HDFS_HOME}}/share/hadoop/hdfs/lib/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/*<CPS>{{HADOOP_YARN_HOME}}/share/hadoop/yarn/lib/*<CPS>./log4j.properties" }, { "key": "DISTRIBUTEDSHELLSCRIPTLEN", "value": "6" }, { "key": "DISTRIBUTEDSHELLSCRIPTLOCATION", "value": "hdfs://hdfs-namenode:9000/user/testuser/demo-app/shellCommands" } ] } }, "unmanaged-AM":"false", "max-app-attempts":"2", "resource": { "memory":"1024", "vCores":"1" }, "application-type":"YARN", "keep-containers-across-application-attempts":"false" }
Submit an Application
You can use the Submit Application API to submit applications. For example:
curl -v -X POST -d @example-submit-app.json -H "Content-type: application/json"'http://localhost:8088/ws/v1/cluster/apps'
After you submit an application the response includes the following field:
HTTP/1.1 202 Accepted
The response also includes the Location field, which you can use to get the status of the application (app ID). The following is an example of a returned Location code:
Location: http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012
Monitor an Application
You can use the Application State API to query the application state. To return only the state of a running application, use the following command format:
curl 'http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012/state'
You can also use the value of the Location field (returned in the application submission response) to check the application status. For example:
curl -v 'http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012'
You can use the following command format to check the logs:
yarn logs -appOwner 'dr.who' -applicationId application_1409421698529_0012 | less
Kill an Application
PUT
operation
to set the application state to KILLED
. For
example:curl -v -X PUT -H 'Accept: application/json' -H 'Content-Type: application/json' -d '{"state": "KILLED"}' 'http://localhost:8088/ws/v1/cluster/apps/application_1409421698529_0012/state'