Cloudera Docs

Associating a Python virtual environment with a Cloudera Data Engineering job

You can associate the Python virtual environment with a CDE job at the time of creation, or you can update an existing job.

Before you begin

  • Download and configure the CDE CLI.
  • Create a Python virtual environment CDE resource.
  • Create a CDE job.

Steps

  1. Using the CDE CLI, run the cde job update command to associate a Python virtual environment with the job. 
    cde job update --name pyspark-example --python-env-resource-name cde-python-env-resource

Before you begin

  • Create a Python virtual environment CDE resource.
  • Create a CDE job.

Steps

  1. In the Cloudera Data Platform (CDP) console, click the Data Engineering tile. The CDE Home page displays.
  2. Click Job Runs on the left navigation menu. The Job Runs page displays.
  3. Using the dropdown menu, select the virtual cluster containing the application you want to manage.
  4. Click on the name of the job you want to modify.
  5. Go to the Configuration tab.
  6. Click Edit.
  7. In the Python Environment section, click Select Python Environment.
  8. Select the Python virtual environment resource you want to use, and then click Select Resource.
  9. At the bottom of the page, click Update and Run to run the job immediately, or click the drop-down arrow on the button and select Update to update the job without running it.