CDE CLI Spark flag reference
The Cloudera Data Engineering (CDE) command Spark flag reference is shown below.
--application-file: application main file
--class: application main class
--arg: Spark argument
--conf: Spark configuration (format key=value) (can be repeated)
--min-executors: minimum number of executors
--max-executors: maximum number of executors
--initial-executors: initial number of executors
--executor-cores: number of cores per executor
--executor-memory: memory per executor
--driver-memory: memory for driver
--driver-cores: number of driver cores
--spark-name: Spark application name
--file: additional file additional file (can be repeated) (will be merged with --files, if provided)
--files: additional files (comma-separated list) (will be merged with all --file)
--jar: additional jar (can be repeated) (will be merged with --jars, if provided)
--jars: additional jars (comma-separated list) (will be merged with all --jar)
--py-file: additional Python file (can be repeated) (will be merged with --py-file, if provided)
--py-files: additional Python files (comma-separated list) (will be merged with all --py-file)
--packages: additional dependencies as comma-separated list of Maven coordinates
--repositories: additional repositories/resolvers for retrieving the --packages dependencies
--python-env-resource-name: Python environment resource name
--python-version: Python version ("python3" or "python2")
--log-level: log level for Spark containers (TRACE, DEBUG, INFO, WARN, ERROR, FATAL, OFF)
--enable-analysis: enables Spark analysis (see 'Analysis' UI tab for a job run)