A resource in Cloudera Data Engineering (CDE) is a
named collection of files referenced by a job. The files can include
application code, configuration files, and even Python virtual environment
specifications (requirements.txt
). You can create a
resource using the CDE API.
- Create a JSON file describing the resource, using the structure
for the type of resource you want to create.
The JSON
payload for a resource is structured as follows:
files
resource type
-
{
"name":"<resourceName>",
"type":"files",
"retentionPolicy":"keep_indefinitely"
}
python-env
resource type
-
{
"name":"<resourceName>",
"type":"python-env",
"retentionPolicy":"keep_indefinitely",
"pythonEnv":{
"pythonVersion":"python3"
}
}
custom-runtime-image
resource type
-
{
"customRuntimeImage": {
"credential": "string",
"engine": "string",
"image": "string",
"modified": "string"
},
"name": "<resourceName>",
"retentionPolicy": "keep_indefinitely",
"type": "custom-runtime-image"
}
- Create the resource by submitting a
POST
request to the resources
endpoint. The JSON filename
is referenced using the @/path/to/filename.json
convention. In this example, the JSON filename is
fileResource.json
, and describes a
files
type resource named
example-job-files
.
curl <jobs_api_url>/resources
-H "Authorization: Bearer ${CDE_TOKEN}" \
-H "Content-Type: application/json" \
-X POST -d @${HOME}/fileResource.json
- Verify that the resource was creating by requesting the
resource details from the
/
resources/<resourceName>
endpoint:
curl <jobs_api_url>/example-job-files \
-H "Authorization: Bearer ${CDE_TOKEN}" \
-H "Content-Type: application/json" \
-X GET