Creating a Cloudera Data Engineering job using the API

You can create a job in Cloudera Data Engineering using the Cloudera Data Engineering jobs API endpoint.

Request an access token and save it as an environment variable to use in API calls. For instructions, see Getting a Cloudera Data Engineering API access token.

important

The user interface for  Cloudera Data Engineering 1.17 and above has been updated. The left-hand menu was updated to provide easy access to commonly used pages. The steps below will vary slightly, for example, the  Overview page has been replaced with the  Home page. To view Cloudera Data Engineering Services, click  Administration in the left-hand menu. The new home page still  displays Virtual Clusters, but now includes quick-access links located at the top for the  following categories: Jobs, Resources, and  Download & Docs.

If you're using secrets or if you want to access workload secrets, refer to Managing workload secrets with Cloudera Data Engineering using the API linked below.

Determine the API URL for the virtual cluster you want to access using the API:
1. In the Cloudera console, click the Data Engineering tile and click Overview
2. In the Cloudera Data Engineering Services column, select the environment containing the virtual cluster you want to interact with using the API.
3. In the Virtual Clusters column on the right, click the Cluster Details icon for the virtual cluster you want to interact with.
4. Click JOBS API URL to copy the URL.
  For example: https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1

Submit the job creation request using the API to the /jobs endpoint.

The JSON payload to create a job is structured as follows:

{
  "name": "demoJob",
  "spark": {
    "className": "com.example.demoJobMainClass",
    "file": "local:/path/to/demoJobJar"
  },
  "type": "spark"
}

curl -H "Authorization: Bearer ${CDE_TOKEN}" <jobs_api_url>/jobs \
-H "Content-Type: application/json" \
-X POST -d "{\"name\":\"demoJob\",\"spark\":{\"className\":\"com.example.demoJobMainClass\",\"file\":\"local:/path/to/demoJobJar\"},\"type\":\"spark\"}"

Verify the job was created. You can view job details using the /jobs/<jobName> endpoint:

curl -H "Authorization: Bearer ${CDE_TOKEN}" -H "Content-Type: application/json" -X GET "https://pmjkrgn5.cde-czlmkz4y.na-01.xvp2-7p8o.cloudera.site/dex/api/v1/jobs/demoJob"