Cloudera supports Spark on YARN-managed deployments for a more flexible and consistent resource management approach.
When running under Spark, the number of executors (YARN containers) can be specified when
submitting the Spark job. By default, dynamic allocation is enabled but can be disabled by
setting the spark.dynamicAllocation.enabled parameter in Cloudera Manager to
false. If you specify the
--num-executors option in the job,
then the dynamic allocation is disabled implicitly. For more information on Spark configuration
and management, see Configuring Apache Spark.
Deploying Spark in a standalone mode is not supported.