Cloudera Docs

Updating Python virtual environment resources

Currently, Python virtual environments cannot be updated. Instead, create a new Python virtual environment resource and update the job to reference the new resource.

Before you begin

  • Download and configure the CDE CLI.
  • Create a new requirements.txt file specifying the Python package and version dependencies required by your Cloudera Data Engineering job.

Steps

  1. Create a new python-env resource.
    cde resource create --name new-cde-python-env-resource --type python-env --python-version python3
  2. Upload the new or updated requirements.txt file to the new resource.
    cde resource upload --name new-cde-python-env-resource --local-path ${HOME}/requirements.txt
  3. Update the Cloudera Data Engineering job to specify the new resource.
    cde job update --name pyspark-example --python-env-resource-name new-cde-python-env-resource

Before you begin

  • Create a new requirements.txt file specifying the Python package and version dependencies required by your Cloudera Data Engineering job.

Steps

  1. Create a new Python virtual environment using the new requirements.txt file, following the instructions in Creating a Python virtual environment resource .
  2. Update the Cloudera Data Engineering job to reference the new Python virtual environment, following the instructions in Associating a Python virtual environment with a Cloudera Data Engineering job .