Using the cde-env tool
You can run the
spark-submit command after installing the
cde-env tool. You can use the
cde-env tool even without
$ cde-env.sh activate -p <profile-name>
For example, to alternatively run the the same
spark-submit command either
against YARN or one of the CDE virtual cluster, you can activate the relevant profile:
- Run spark jobs on YARN by activating the
yarnis a reserved profile name here.
$ cde-env.sh activate -p yarn
- Run spark jobs on a CDE virtual cluster configured by CDE CLI profile named
$ cde-env.sh activate -p vc-1
Switching profiles back and forth this way lets you run the same
command either against YARN or one of the CDE virtual clusters.