Spark

Cloudera supports Spark on YARN-managed deployments for a more flexible and consistent resource management approach.

When running under Spark, the number of executors (YARN containers) can be specified when submitting the Spark job. By default, dynamic allocation is enabled but can be disabled by setting the spark.dynamicAllocation.enabled parameter in Cloudera Manager to false. If you specify the --num-executors option in the job, then the dynamic allocation is disabled implicitly. For more information on Spark configuration and management, see Configuring Apache Spark.

Deploying Spark in a standalone mode is not supported.