Creating a Cloudera Data Engineering resource using the API

A resource in Cloudera Data Engineering (CDE) is a named collection of files referenced by a job. The files can include application code, configuration files, and even Python virtual environment specifications (requirements.txt). You can create a resource using the CDE API.

As with all API calls, make sure you have a valid access token. For instructions, see Getting a Cloudera Data Engineering API access token.

  1. Create a JSON file describing the resource, using the structure for the type of resource you want to create.
    The JSON payload for a resource is structured as follows:
    files resource type
    {
       "name":"<resourceName>",
       "type":"files",
       "retentionPolicy":"keep_indefinitely"
    }
    python-env resource type
    {
       "name":"<resourceName>",
       "type":"python-env",
       "retentionPolicy":"keep_indefinitely",
       "pythonEnv":{
          "pythonVersion":"python3"
       }
    }
    custom-runtime-image resource type
    {
      "customRuntimeImage": {
        "credential": "string",
        "engine": "string",
        "image": "string",
        "modified": "string"
      },
      "name": "<resourceName>",
      "retentionPolicy": "keep_indefinitely",
      "type": "custom-runtime-image"
    }
  2. Create the resource by submitting a POST request to the resources endpoint. The JSON filename is referenced using the @/path/to/filename.json convention. In this example, the JSON filename is fileResource.json, and describes a files type resource named example-job-files.
    curl <jobs_api_url>/resources
    -H "Authorization: Bearer ${CDE_TOKEN}" \
    -H "Content-Type: application/json" \
    -X POST -d @${HOME}/fileResource.json
  3. Verify that the resource was creating by requesting the resource details from the /resources/<resourceName> endpoint:
    curl <jobs_api_url>/example-job-files \
    -H "Authorization: Bearer ${CDE_TOKEN}" \
    -H "Content-Type: application/json" \
    -X GET