Using the cde-env tool

You can run the spark-submit command after installing the cde-env tool. You can use the cde-env tool even without administrator privileges.

You can activate the specific profile you want to use by running the following command:
$ cde-env.sh activate -p <profile-name>

For example, to alternatively run the the same spark-submit command either against YARN or one of the CDE virtual cluster, you can activate the relevant profile:

  1. Run spark jobs on YARN by activating the yarn profile. yarn is a reserved profile name here.
    $ cde-env.sh activate -p yarn 
  2. Run spark jobs on a CDE virtual cluster configured by CDE CLI profile named vc-1:
    $ cde-env.sh activate -p vc-1

Switching profiles back and forth this way lets you run the same spark-submit command either against YARN or one of the CDE virtual clusters.