Associating a Python virtual environment with a Cloudera Data Engineering job
You can associate the Python virtual environment with a CDE job at the time of creation, or you can update an existing job.
Before you begin
- Download and configure the CDE CLI.
- Create a Python virtual environment CDE resource.
- Create a CDE job.
Steps
- Using the CDE CLI, run the
cde job update
command to associate a Python virtual environment with the job.cde job update --name pyspark-example --python-env-resource-name cde-python-env-resource
Before you begin
- Create a Python virtual environment CDE resource.
- Create a CDE job.
Steps
- In the Cloudera Data Platform (CDP) management console, click the Data Engineering tile and click Overview.
- In the CDE Services column, select the service containing the virtual cluster and job you want to configure.
- In the Virtual Clusters column on the right, click the View Jobs icon for the virtual cluster containing the job you want to configure.
- Click Jobs in the left menu.
- Click the job you want to modify.
- Go to the Configuration tab.
- Click Edit.
- In the Python Environment section, click Select Python Environment.
- Select the Python virtual environment resource you want to use, and then click Select Resource.
- At the bottom of the page, click Update and Run to run the job immediately, or click the drop-down arrow on the button and select Update to update the job without running it.