Configuring Spark application properties in
spark-defaults.conf
Specify properties in the spark-defaults.conf
file in
the form property=value
.
To create a comment, add a hash mark ( # ) at the beginning of a line. You cannot add comments to the end or middle of a line.
This example shows an example excerpt of a
spark-defaults.conf
file:
spark.authenticate=false spark.driver.log.dfsDir=/user/spark/driverLogs spark.driver.log.persistToDfs.enabled=true spark.dynamicAllocation.enabled=true spark.dynamicAllocation.executorIdleTimeout=60 spark.dynamicAllocation.minExecutors=0 spark.dynamicAllocation.schedulerBacklogTimeout=1 spark.eventLog.enabled=true spark.io.encryption.enabled=false spark.lineage.enabled=true spark.network.crypto.enabled=false spark.serializer=org.apache.spark.serializer.KryoSerializer spark.shuffle.service.enabled=true spark.shuffle.service.port=7337 spark.ui.enabled=true spark.ui.killEnabled=true spark.yarn.access.hadoopFileSystems=s3a://bucket1,s3a://bucket2
Cloudera recommends placing configuration properties that you want to use for every application in spark-defaults.conf. See Application Properties for more information.