Creating a Cloudera Data Engineering job using the API

You can create a job in Cloudera Data Engineering (CDE) using the CDE jobs API endpoint.

Request an access token and save it as an environment variable to use in API calls. For instructions, see Getting a Cloudera Data Engineering API access token.

Determine the API URL for the virtual cluster you want to access using the API:
1. Navigate to the Cloudera Data Engineering Overview page.
2. In the CDE Services column, select the environment containing the virtual cluster you want to interact with using the API.
3. In the Virtual Clusters column on the right, click the Cluster Details icon for the virtual cluster you want to interact with.
4. Copy the URL under JOBS API URL.
  For example: https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1

Submit the job creation request using the API to the /jobs endpoint.

The JSON payload to create a job is structured as follows:

{
  "name": "demoJob",
  "spark": {
    "className": "com.example.demoJobMainClass",
    "file": "local:/path/to/demoJobJar"
  },
  "type": "spark"
}

curl -H "Authorization: Bearer ${CDE_TOKEN}" <jobs_api_url>/jobs \
-H "Content-Type: application/json" \
-X POST -d "{\"name\":\"demoJob\",\"spark\":{\"className\":\"com.example.demoJobMainClass\",\"file\":\"local:/path/to/demoJobJar\"},\"type\":\"spark\"}"

Verify the job was created. You can view job details using the /jobs/<jobName> endpoint:

curl -H "Authorization: Bearer ${CDE_TOKEN}" -H "Content-Type: application/json" -X GET "https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/jobs/demoJob"

Creating a Cloudera Data Engineering job using the API

We want your opinion

How can we improve this page?