Cloudera Docs

Prerequisites

You must first authenticate and obtain the cookies to send a request.

  • Authenticate and obtain cookie
    • Log in to Cloudera Manager and get the cookie to send a request. This generates a session cookie that must be provided with every request. If you are using an HTTP client such as Postman, you can copy the cookie value from your browser and add it as a header to your requests. You can also authenticate and obtain the cookie using the command line using cURL, as follows: 
      curl -i -k -v -c cookies.txt -u admin:admin
http://<cm_host>:7180/api/v41/clusters
    • This command saves the resulting cookie in cookies.txt. You can use the cookie to make the requests. Run the following command to make the initial validation request. 
      curl -v -k -b cookies.txt -u <username>:<password> -H 'Content-Type: application/json' -X POST -H 'referer:
http://<cm_host>:7180/cmf/clusters/Cluster%201/queue-manager/'
http://<cm_host>:7180/cmf/clusters/Cluster%201/queue-manager-api/api/v1/environments/dev/clusters/Cluster%201/resources/scheduler/partitions/default/queues/root.default -d '<request_body>'
  • Secure cluster (SSL and Kerberos enabled)

    If you are using a cluster secured with SSL and Kerberos, obtain the required certificates.

    1. Create a directory on your machine to copy the certificates from the Queue Manager node. The following command creates the apiCerts directory. 
      mkdir apiCerts
    2. SSH to your Queue Manager node to obtain the WebApp folder information. 
      cd /var/run/cloudera-scm-agent/process/
    3. Copy the certificates from the Queue Manager node to the “apiCerts” directory. 
      scp -i <your_keypair>
root@<qm_host>:/var/run/cloudera-scm-agent/process/{12}-queuemanager-QUEUEMANAGER_WEBAPP/cm-auto-host_cert_chain.pem apiCerts/
    4. Copy the following certificates from the Queue Manager node. 
      scp -i <your_keypair> 
root@<qm_host>:/var/run/cloudera-scm-agent/process/12-queuemanager-QUEUEMANAGER_WEBAPP/cm-auto-global_cacerts.pem apiCerts/

scp -i <your_keypair> 
root@<qm_host>:/var/run/cloudera-scm-agent/process/12-queuemanager-QUEUEMANAGER_WEBAPP/cm-auto-host_cert_chain.pem apiCerts/

scp -i <your_keypair> 
root@<qm_host>:/var/run/cloudera-scm-agent/process/12-queuemanager-QUEUEMANAGER_WEBAPP/cm-auto-host_key.pem apiCerts/

scp -i <your_keypair> 
root@<qm_host>:/var/run/cloudera-scm-agent/process/12-queuemanager-QUEUEMANAGER_WEBAPP/cm-auto-host_key.pw apiCerts/
    5. Decrypt the certificate so that you can use it later in the cURL call. 
      openssl rsa -passin file:cm-auto-host_key.pw -in cm-auto-host_key.pem -out
cm-auto-host_key_decrypted.pem
chmod 644 cm-auto-host_key_decrypted.pem
    6. Save the cookie for the API calls. 
      curl --key cm-auto-host_key_decrypted.pem --cert cm-auto-host_cert_chain.pem --cacert cm-auto-global_cacerts.pem -v -c cookies_ssl.txt -u admin:admin https://<cm_host>:7183/api/v41/clusters
  • Initialization

    Before using Queue Manager REST APIs, you must initialize the back-end with information about the cluster. This is done using a POST request with the cluster information obtained from the Cloudera Manager, which you can find in various places in the Cloudera Manager UI.

    The information can also be found in session storage after navigating to the Queue Manager UI. Refer your browser documentation for information about viewing session storage. For example, to view session storage in Chrome, see View Session Storage.
    1. Open the YARN Queue Manager user interface. This must be done once in order to create the first initialization request.
    2. Get the cluster details:
      curl -v -X GET --key cm-auto-host_key_decrypted.pem --cert cm-auto-host_cert_chain.pem --cacert cm-auto-global_cacerts.pem -b cookies_ssl.txt -u <username>:<password> -H 'Content-Type: application/json' -H 'Referer: https://<cm_host>:7183/cmf/clusters/Cluster%201/queue-manager/' https://<cm_host>:7183/cmf/clusters/Cluster%201/queue-manager-api/api/v1/environments/dev/clusters/Cluster%201

      This is the output for the cluster information called, Cluster 1.

    3. Using the the cluster output information above, create a JSON file called, “init_request.json” and save it in the following format below:
      {
  "provisioner": "CLOUDERA_MANAGER",
  "provisionerUrl": "<provisionerUrl from above cluster info output>",
  "clusters": [
    {
      "rmUrl": <activeRMUrl from above cluster info output>,
      "clusterId": <id from  above cluster info output>,
      "yarnServiceName": <yarnServiceName from above cluster info output>,
      "rmRoleConfigGroup": <rmRoleConfigGroup from above cluster info output>
    }
  ]
}
    4. Initialize the cluster:
      curl -v -X POST --key cm-auto-host_key_decrypted.pem --cert cm-auto-host_cert_chain.pem --cacert cm-auto-global_cacerts.pem -b cookies_ssl.txt -u <username>:<password> -H 'Content-Type: application/json' -H 'Referer: https://<cm_host>:7183/cmf/clusters/Cluster%201/queue-manager/' https://<cm_host>:7183/cmf/clusters/Cluster%201/queue-manager-api/api/v1/environments/dev/clusters -d @init_request.json
      This curl command and init_request.json file can be used to initialize a cluster when Queue Manager services restart, specifically Queue Manager web app service.

      Sample request:

      You can use a comma separated list to add multiple URLs of the Resource Manager.

      POST http://my-cluster-1.my-cluster.root.hwx.site:7180/cmf/clusters/Cluster%201/queue-manager-api/api/v1/environments/dev/clusters

{
  "provisioner": "CLOUDERA_MANAGER",
  "provisionerUrl": "my-cluster-1.my-cluster.root.hwx.site:7180",
  "clusters": [
    {
      "rmUrl": "my-cluster-2.my-cluster.root.hwx.site:7180",
      "clusterId": "Cluster 1",
      "yarnServiceName": "YARN-1",
      "rmRoleConfigGroup": "YARN-1-RESOURCEMANAGER-BASE"
    }
  ]
}
  • Validation
    You can check to ensure that your setup is correct.
    1. Make the following request to verify that the Queue Manager API REST setup is correct.
      curl -v -X GET --key cm-auto-host_key_decrypted.pem --cert cm-auto-host_cert_chain.pem --cacert cm-auto-global_cacerts.pem -b cookies_ssl.txt -u <username>:<password> -H 'Content-Type: application/json' -H 'Referer: https://<cm_host>:7183/cmf/clusters/Cluster%201/queue-manager/' https://<cm_host>:7183/cmf/clusters/Cluster%201/queue-manager-api/api/v1/environments/dev/clusters/Cluster%201/resources/scheduler/partitions/default/queues