Configuring CDS 2.x Powered by Apache Spark 2

This topic describes how to set Spark 2 environment variables, manage package dependencies for Spark 2 jobs, and how to configure logging.

Spark Configuration Files
Cloudera Data Science Workbench supports configuring Spark 2 properties on a per project basis with the spark-defaults.conf file.
Managing Memory Available for Spark Drivers
By default, the amount of memory allocated to Spark driver processes is set to a 0.8 fraction of the total memory allocated for the engine container.
Managing Dependencies for Spark 2 Jobs
As with any Spark job, you can add external packages to the executor on startup.
Spark Logging Configuration
Cloudera Data Science Workbench allows you to update Spark’s internal logging configuration on a per-project basis. Spark 2 uses Apache Log4j, which can be configured through a properties file.
Running Spark Jobs on an HDP Cluster
Execution errors might occur due to inadequate resource threshold values set for YARN.