Cloudera Docs

Creating a Cloudera Data Engineering job using the API

You can create a job in Cloudera Data Engineering (CDE) using the CDE jobs API endpoint.

Request an access token and save it as an environment variable to use in API calls. For instructions, see Getting a Cloudera Data Engineering API access token.

  1. Determine the API URL for the virtual cluster you want to access using the API:
    1. In the Cloudera Data Platform (CDP) console, click the Data Engineering tile. The CDE Home page displays.
    2. In the Virtual Clusters section, navigate to the virtual cluster for which you want to interact with, and click Cluster Details.
    3. Click JOBS API URL to copy the URL.
      For example: https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1
  2. Submit the job creation request using the API to the /jobs endpoint.
    The JSON payload to create a job is structured as follows:
    {
  "name": "demoJob",
  "spark": {
    "className": "com.example.demoJobMainClass",
    "file": "local:/path/to/demoJobJar"
  },
  "type": "spark"
}
    curl -H "Authorization: Bearer ${CDE_TOKEN}" <jobs_api_url>/jobs \
-H "Content-Type: application/json" \
-X POST -d "{\"name\":\"demoJob\",\"spark\":{\"className\":\"com.example.demoJobMainClass\",\"file\":\"local:/path/to/demoJobJar\"},\"type\":\"spark\"}"
  3. Verify the job was created. You can view job details using the /jobs/<jobName> endpoint:
    curl -H "Authorization: Bearer ${CDE_TOKEN}" -H "Content-Type: application/json" -X GET "https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/jobs/demoJob"