Managing Data Operating System
Also available as:
PDF
loading table of contents...

Run Docker on YARN using the YARN services API

You can deploy a load balanced web server pair on a single node HDP cluster using the YARN Services API.

Note that the networking setup here is not appropriate for a multi-node cluster. See the Networking Recommendations for more.
  1. Create the following Yarnfile and save it to /tmp/httpd.json:
    
     {  
       "name":"httpd-service",
       "version":"1.0.0",
       "lifetime":"3600",
       "configuration":{  
          "properties":{  
             "docker.network":"bridge"
          }
       },
       "components":[  
          {  
             "name":"httpd",
             "number_of_containers":2,
             "artifact":{  
                "id":"centos/httpd-24-centos7:latest",
                "type":"DOCKER"
             },
             "launch_command":"/usr/bin/run-httpd",
             "resource":{  
                "cpus":1,
                "memory":"1024"
             },
             "readiness_check":{  
                "type":"HTTP",
                "properties":{  
                   "url":"http://${THIS_HOST}:8080"
                }
             },
             "configuration":{  
                "files":[  
                   {  
                      "type":"TEMPLATE",
                      "dest_file":"/var/www/html/index.html",
                      "properties":{  
                         "content":"<html><header><title>Title</title></header><body>Hello from ${COMPONENT_INSTANCE_NAME}!</body></html>"
                      }
                   }
                ]
             }
          },
          {  
             "name":"httpd-proxy",
             "number_of_containers":1,
             "dependencies":[  
                "httpd"
             ],
             "artifact":{  
                "id":"centos/httpd-24-centos7:latest",
                "type":"DOCKER"
             },
             "launch_command":"/usr/bin/run-httpd",
             "resource":{  
                "cpus":1,
                "memory":"1024"
             },
             "configuration":{  
                "files":[  
                   {  
                      "type":"TEMPLATE",
                      "dest_file":"/etc/httpd/conf.d/httpd-proxy.conf",
                      "src_file":"httpd-proxy-no-dns.conf"
                   }
                ]
             }
          }
       ]
    }
                           
                       
  2. Copy the HTTPD proxy configuration to HDFS. This configuration will be used to configure the HTTPD proxy container to load balance traffic between two backend HTTPD servers.

    hdfs dfs -copyFromLocal /usr/hdp/current/hadoop-yarn-client/yarn-service-examples/httpd-no-dns/httpd-proxy-no-dns.conf

  3. Submit the HTTPD application using the YARN CLI:

    yarn app -launch httpdservice /tmp/httpd.json

  4. Once the application has launched, get the IP address for the HTTPD proxy container to validate that the backend servers have successfully started. The components need to reach a STABLE state and the containers a READY state in the status output before proceeding with next step.

    yarn app -status httpdservice

    Note
    Note
    use jq or python -m json.tool to pretty print the output.

    The output will appear as follows.

    
    {  
       "name":"httpdservice",
       "id":"application_1525954234278_0006",
       "lifetime":3549,
       "components":[  
          {  
             "name":"httpd",
             "dependencies":[  
    
             ],
             "artifact":{  
                "id":"centos/httpd-24-centos7:latest",
                "type":"DOCKER"
             },
             "resource":{  
                "cpus":1,
                "memory":"1024",
                "additional":{  
    
                }
             },
             "state":"STABLE",
             "configuration":{  
                "properties":{  
                   "docker.network":"bridge"
                },
                "env":{  
    
                },
                "files":[  
                   {  
                      "type":"TEMPLATE",
                      "properties":{  
                         "content":"<html><header><title>Title</title></header><body>Hello from ${COMPONENT_INSTANCE_NAME}!</body></html>"
                      },
                      "dest_file":"/var/www/html/index.html"
                   }
                ]
             },
             "quicklinks":[  
    
             ],
             "containers":[  
                {  
                   "id":"container_1525954234278_0006_01_000002",
                   "ip":"172.17.0.2",
                   "hostname":"httpd-0.httpdservice.user.domain",
                   "state":"READY",
                   "launch_time":1525980550770,
                   "bare_host":"node.domain",
                   "component_instance_name":"httpd-0"
                },
                {  
                   "id":"container_1525954234278_0006_01_000003",
                   "ip":"172.17.0.3",
                   "hostname":"httpd-1.httpdservice.user.domain",
                   "state":"READY",
                   "launch_time":1525980551772,
                   "bare_host":"node.domain",
                   "component_instance_name":"httpd-1"
                }
             ],
             "readiness_check":{  
                "type":"HTTP",
                "properties":{  
                   "url":"http://${THIS_HOST}:8080"
                }
             },
             "launch_command":"/usr/bin/run-httpd",
             "number_of_containers":2,
             "run_privileged_container":false
          },
          {  
             "name":"httpd-proxy",
             "dependencies":[  
                "httpd"
             ],
             "artifact":{  
                "id":"centos/httpd-24-centos7:latest",
                "type":"DOCKER"
             },
             "resource":{  
                "cpus":1,
                "memory":"1024",
                "additional":{  
    
                }
             },
             "state":"STABLE",
             "configuration":{  
                "properties":{  
                   "docker.network":"bridge"
                },
                "env":{  
    
                },
                "files":[  
                   {  
                      "type":"TEMPLATE",
                      "properties":{  
    
                      },
                      "dest_file":"/etc/httpd/conf.d/httpd-proxy.conf",
                      "src_file":"httpd-proxy-no-dns.conf"
                   }
                ]
             },
             "quicklinks":[  
    
             ],
             "containers":[  
                {  
                   "id":"container_1525954234278_0006_01_000005",
                   "ip":"172.17.0.4",
                   "hostname":"httpd-proxy-0.httpdservice.user.domain",
                   "state":"READY",
                   "launch_time":1525980579818,
                   "bare_host":"node.domain",
                   "component_instance_name":"httpd-proxy-0"
                }
             ],
             "launch_command":"/usr/bin/run-httpd",
             "number_of_containers":1,
             "run_privileged_container":false
          }
       ],
       "configuration":{  
          "properties":{  
             "docker.network":"bridge"
          },
          "env":{  
    
          },
          "files":[  
    
          ]
       },
       "state":"STARTED",
       "quicklinks":{  
    
       },
       "version":"1.0.0",
       "kerberos_principal":{  
    
       }
    }
             

    Find the details regarding the httpd-proxy-0 instance and record the IP address associated with the container. In the example above, the IP is 172.17.0.4.

  5. Using curl, or a similar HTTP client, perform a GET request to the HTTPD proxy on port 8080, using the IP address obtained in the previous step.

    curl http://172.17.0.4:8080/

    The requests should be routed to both backend HTTPD servers, as seen in the following output:

    [user@node ~]$ curl http://172.17.0.4:8080/ <html><header><title>Title</title></header><body>Hello from httpd-0!</body></html>

    [user@node ~]$ curl http://172.17.0.4:8080/ <html><header><title>Title</title></header><body>Hello from httpd-1!</body></html>

    [user@node ~]$ curl http://172.17.0.4:8080/ <html><header><title>Title</title></header><body>Hello from httpd-0!</body></html>

    [user@node ~]$ curl http://172.17.0.4:8080/ <html><header><title>Title</title></header><body>Hello from httpd-1!</body></html>