Known Issues and Limitations

This page lists the current known issues and limitations that you might run into while using the cde-env tool.

• Limited to spark-submit commands and does not include spark-shell, pyspark, and sparksql.
• When activating a profile using the cde-env.sh script, there is no validation yet on whether such profile exists. However, if a profile does not exist, it will display an error when running the spark-submit command.
• The following spark-submit flags are not yet supported in CDE:

  • --archives

  • --exclude-packages

  • --driver-class-path

  • --driver-library-path

  • --driver-java-options

    You are instead suggested to create CDE jobs to handle the above mentioned scenarios.

• Using the profile yarn is not supported in the container version of the migration tool.