CDE CLI Spark flag reference
The Cloudera Data Engineering command Spark flag reference is shown below.
--application-file: application main file --class: application main class --arg: Spark argument --conf: Spark configuration (format key=value) (can be repeated) --min-executors: minimum number of executors --max-executors: maximum number of executors --initial-executors: initial number of executors --executor-cores: number of cores per executor --executor-memory: memory per executor --driver-memory: memory for driver --driver-cores: number of driver cores --spark-name: Spark application name --file: additional file additional file (can be repeated) (will be merged with --files, if provided) --files: additional files (comma-separated list) (will be merged with all --file) --jar: additional jar (can be repeated) (will be merged with --jars, if provided) --jars: additional jars (comma-separated list) (will be merged with all --jar) --py-file: additional Python file (can be repeated) (will be merged with --py-file, if provided) --py-files: additional Python files (comma-separated list) (will be merged with all --py-file) --packages: additional dependencies as comma-separated list of Maven coordinates --repositories: additional repositories/resolvers for retrieving the --packages dependencies --python-env-resource-name: Python environment resource name --python-version: Python version ("python3" or "python2") --log-level: log level for Spark containers (TRACE, DEBUG, INFO, WARN, ERROR, FATAL, OFF) --enable-analysis: enables Spark analysis (see 'Analysis' UI tab for a job run)