Run Docker on YARN using the YARN services API
You can deploy a load balanced web server pair on a single node HDP cluster using the YARN Services API.
-
Create the following Yarnfile and save it to
/tmp/httpd.json
:{ "name":"httpd-service", "version":"1.0.0", "lifetime":"3600", "configuration":{ "properties":{ "docker.network":"bridge" } }, "components":[ { "name":"httpd", "number_of_containers":2, "artifact":{ "id":"centos/httpd-24-centos7:latest", "type":"DOCKER" }, "launch_command":"/usr/bin/run-httpd", "resource":{ "cpus":1, "memory":"1024" }, "readiness_check":{ "type":"HTTP", "properties":{ "url":"http://${THIS_HOST}:8080" } }, "configuration":{ "files":[ { "type":"TEMPLATE", "dest_file":"/var/www/html/index.html", "properties":{ "content":"<html><header><title>Title</title></header><body>Hello from ${COMPONENT_INSTANCE_NAME}!</body></html>" } } ] } }, { "name":"httpd-proxy", "number_of_containers":1, "dependencies":[ "httpd" ], "artifact":{ "id":"centos/httpd-24-centos7:latest", "type":"DOCKER" }, "launch_command":"/usr/bin/run-httpd", "resource":{ "cpus":1, "memory":"1024" }, "configuration":{ "files":[ { "type":"TEMPLATE", "dest_file":"/etc/httpd/conf.d/httpd-proxy.conf", "src_file":"httpd-proxy-no-dns.conf" } ] } } ] }
-
Copy the HTTPD proxy configuration to HDFS.
This configuration will be used to configure the HTTPD proxy container to load balance traffic between two backend HTTPD servers.
hdfs dfs -copyFromLocal /usr/hdp/current/hadoop-yarn-client/yarn-service-examples/httpd-no-dns/httpd-proxy-no-dns.conf
-
Submit the HTTPD application using the YARN CLI:
yarn app -launch httpdservice /tmp/httpd.json
-
Once the application has launched, get the IP address for the HTTPD proxy container to validate that the backend servers have successfully started.
The components need to reach a STABLE state and the containers a READY state in the status output before proceeding with next step.
yarn app -status httpdservice
Noteusejq
orpython -m json.tool
to pretty print the output.The output will appear as follows.
{ "name":"httpdservice", "id":"application_1525954234278_0006", "lifetime":3549, "components":[ { "name":"httpd", "dependencies":[ ], "artifact":{ "id":"centos/httpd-24-centos7:latest", "type":"DOCKER" }, "resource":{ "cpus":1, "memory":"1024", "additional":{ } }, "state":"STABLE", "configuration":{ "properties":{ "docker.network":"bridge" }, "env":{ }, "files":[ { "type":"TEMPLATE", "properties":{ "content":"<html><header><title>Title</title></header><body>Hello from ${COMPONENT_INSTANCE_NAME}!</body></html>" }, "dest_file":"/var/www/html/index.html" } ] }, "quicklinks":[ ], "containers":[ { "id":"container_1525954234278_0006_01_000002", "ip":"172.17.0.2", "hostname":"httpd-0.httpdservice.user.domain", "state":"READY", "launch_time":1525980550770, "bare_host":"node.domain", "component_instance_name":"httpd-0" }, { "id":"container_1525954234278_0006_01_000003", "ip":"172.17.0.3", "hostname":"httpd-1.httpdservice.user.domain", "state":"READY", "launch_time":1525980551772, "bare_host":"node.domain", "component_instance_name":"httpd-1" } ], "readiness_check":{ "type":"HTTP", "properties":{ "url":"http://${THIS_HOST}:8080" } }, "launch_command":"/usr/bin/run-httpd", "number_of_containers":2, "run_privileged_container":false }, { "name":"httpd-proxy", "dependencies":[ "httpd" ], "artifact":{ "id":"centos/httpd-24-centos7:latest", "type":"DOCKER" }, "resource":{ "cpus":1, "memory":"1024", "additional":{ } }, "state":"STABLE", "configuration":{ "properties":{ "docker.network":"bridge" }, "env":{ }, "files":[ { "type":"TEMPLATE", "properties":{ }, "dest_file":"/etc/httpd/conf.d/httpd-proxy.conf", "src_file":"httpd-proxy-no-dns.conf" } ] }, "quicklinks":[ ], "containers":[ { "id":"container_1525954234278_0006_01_000005", "ip":"172.17.0.4", "hostname":"httpd-proxy-0.httpdservice.user.domain", "state":"READY", "launch_time":1525980579818, "bare_host":"node.domain", "component_instance_name":"httpd-proxy-0" } ], "launch_command":"/usr/bin/run-httpd", "number_of_containers":1, "run_privileged_container":false } ], "configuration":{ "properties":{ "docker.network":"bridge" }, "env":{ }, "files":[ ] }, "state":"STARTED", "quicklinks":{ }, "version":"1.0.0", "kerberos_principal":{ } }
Find the details regarding the httpd-proxy-0 instance and record the IP address associated with the container. In the example above, the IP is 172.17.0.4.
-
Using curl, or a similar HTTP client, perform a GET request to the HTTPD proxy on port 8080, using the IP address obtained in the previous step.
curl http://172.17.0.4:8080/
The requests should be routed to both backend HTTPD servers, as seen in the following output:
[user@node ~]$ curl http://172.17.0.4:8080/ <html><header><title>Title</title></header><body>Hello from httpd-0!</body></html>
[user@node ~]$ curl http://172.17.0.4:8080/ <html><header><title>Title</title></header><body>Hello from httpd-1!</body></html>
[user@node ~]$ curl http://172.17.0.4:8080/ <html><header><title>Title</title></header><body>Hello from httpd-0!</body></html>
[user@node ~]$ curl http://172.17.0.4:8080/ <html><header><title>Title</title></header><body>Hello from httpd-1!</body></html>