Using the cde-env tool
You can run the spark-submit
command after installing the
cde-env
tool. You can use the cde-env
tool even without
administrator privileges.
You can activate the specific profile you want to use by running the following command:
$ cde-env.sh activate -p <profile-name>
For example, to alternatively run the the same spark-submit
command either
against YARN or one of the CDE virtual cluster, you can activate the relevant profile:
- Run spark jobs on YARN by activating the
yarn
profile.yarn
is a reserved profile name here.$ cde-env.sh activate -p yarn
- Run spark jobs on a CDE virtual cluster configured by CDE CLI profile named
vc-1
:$ cde-env.sh activate -p vc-1
Switching profiles back and forth this way lets you run the same spark-submit
command either against YARN or one of the CDE virtual clusters.