Cloudera Docs

Associating a Python virtual environment with a Cloudera Data Engineering job

You can associate the Python virtual environment with a CDE job at the time of creation, or you can update an existing job.

Before you begin

  • Download and configure the CDE CLI.
  • Create a Python virtual environment CDE resource.
  • Create a CDE job.

Steps

  1. Using the CDE CLI, run the cde job update command to associate a Python virtual environment with the job. 
    cde job update --name pyspark-example --python-env-resource-name cde-python-env-resource

Before you begin

  • Create a Python virtual environment CDE resource.
  • Create a CDE job.

Steps

  1. Go to the Cloudera Data Engineering Overview page by clicking the Data Engineering tile in the Cloudera Data Platform (CDP) management console.
  2. In the CDE Services column, select the service containing the virtual cluster and job you want to configure.
  3. In the Virtual Clusters column on the right, click the View Jobs icon for the virtual cluster containing the job you want to configure.
  4. Click Jobs in the left menu.
  5. Click the job you want to modify.
  6. Go to the Configuration tab.
  7. Click Edit.
  8. In the Python Environment section, click Select Python Environment.
  9. Select the Python virtual environment resource you want to use, and then click Select Resource.
  10. At the bottom of the page, click Update and Run to run the job immediately, or click the drop-down arrow on the button and select Update to update the job without running it.