Known Issues and Limitations

This page lists the current known issues and limitations that you might run into while using the cde-env tool.

  • Limited to spark-submit commands and does not include spark-shell, pyspark, and sparksql.
  • When activating a profile using the cde-env.sh script, there is no validation yet on whether such profile exists. However, if a profile does not exist, it will display an error when running the spark-submit command.
  • The following spark-submit flags are not yet supported in CDE:
    • --archives

    • --exclude-packages

    • --driver-class-path

    • --driver-library-path

    • --driver-java-options

      You are instead suggested to create CDE jobs to handle the above mentioned scenarios.

  • Using the profile yarn is not supported in the container version of the migration tool.