Cloudera Data Engineering resources
A resource in Cloudera Data Engineering (CDE) is a named collection of
files used by a job or a session. Resources can include application code, configuration files,
custom Docker images, and Python virtual environment specifications
(requirements.txt
).
Resources are associated with virtual clusters. A resource can be used by
multiple jobs, and jobs can use multiple resources.The resource types
supported by CDE are files
, python-env
,
and custom-runtime-image
.
files
- An arbitrary collection of files that a job can reference. The
application code for the job, including any necessary configuration
files or supporting libraries, can be stored in a
files
resource. Files can be uploaded to and removed from a resource as needed. python-env
- A defined virtual Python environment that a job runs in. The only
file that can be uploaded to a
python-env
resource is arequirements.txt
file. When you associate apython-env
resource with a job, the job runs within a Python virtual environment built according to therequirements.txt
specification. custom-runtime-image
- A Docker container image. When you run a job using a
custom-runtime-image
resource, the executors that are launched use your custom image.