Creating a Cloudera Data Engineering job using the API

You can create a job in Cloudera Data Engineering using the Cloudera Data Engineering jobs API endpoint.

Request an access token and save it as an environment variable to use in API calls. For instructions, see Getting a Cloudera Data Engineering API access token.

Determine the API URL for the virtual cluster you want to access using the API:
1. In the Cloudera console, click the Data Engineering tile. The Cloudera Data Engineering Home page displays.
2. In the Virtual Clusters section, navigate to the virtual cluster for which you want to interact with, and click Cluster Details.
3. Click JOBS API URL to copy the URL.
  For example: https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1

Submit the job creation request using the API to the /jobs endpoint.

The JSON payload to create a job is structured as follows:

{
  "name": "demoJob",
  "spark": {
    "className": "com.example.demoJobMainClass",
    "file": "local:/path/to/demoJobJar"
  },
  "type": "spark"
}

curl -H "Authorization: Bearer ${CDE_TOKEN}" <jobs_api_url>/jobs \
-H "Content-Type: application/json" \
-X POST -d "{\"name\":\"demoJob\",\"spark\":{\"className\":\"com.example.demoJobMainClass\",\"file\":\"local:/path/to/demoJobJar\"},\"type\":\"spark\"}"

Verify the job was created. You can view job details using the /jobs/<jobName> endpoint:

curl -H "Authorization: Bearer ${CDE_TOKEN}" -H "Content-Type: application/json" -X GET "https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/jobs/demoJob"

Creating a Cloudera Data Engineering job using the API

We want your opinion

How can we improve this page?