Creating a Cloudera Data Engineering job using the API

You can create a job in Cloudera Data Engineering (CDE) using the CDE jobs API endpoint.

Request an access token and save it as an environment variable to use in API calls. For instructions, see Getting a Cloudera Data Engineering API access token.

Determine the API URL for the virtual cluster you want to access using the API:
1. In the Cloudera Data Platform (CDP) console, click the Data Engineering tile and click Overview
2. In the CDE Services column, select the environment containing the virtual cluster you want to interact with using the API.
3. In the Virtual Clusters column on the right, click the Cluster Details icon for the virtual cluster you want to interact with.
4. Click JOBS API URL to copy the URL.
  For example: https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1

Submit the job creation request using the API to the /jobs endpoint.

The JSON payload to create a job is structured as follows:

{
  "name": "demoJob",
  "spark": {
    "className": "com.example.demoJobMainClass",
    "file": "local:/path/to/demoJobJar"
  },
  "type": "spark"
}

curl -H "Authorization: Bearer ${CDE_TOKEN}" <jobs_api_url>/jobs \
-H "Content-Type: application/json" \
-X POST -d "{\"name\":\"demoJob\",\"spark\":{\"className\":\"com.example.demoJobMainClass\",\"file\":\"local:/path/to/demoJobJar\"},\"type\":\"spark\"}"

Verify the job was created. You can view job details using the /jobs/<jobName> endpoint:

curl -H "Authorization: Bearer ${CDE_TOKEN}" -H "Content-Type: application/json" -X GET "https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/jobs/demoJob"