Spark Logging Configuration
Cloudera Data Science Workbench allows you to update Spark’s internal logging configuration on a per-project basis. Spark 2 uses Apache Log4j, which can be configured through a properties file.
log4j.properties file found in the
root of your project will be appended to the existing Spark logging properties for every
session and job. To specify a custom location, set the environmental variable
LOG4J_CONFIG to the file location relative to your
The Log4j documentation has more details on logging options.
Increasing the log level or pushing logs to an alternate location for troublesome jobs can be very helpful for debugging. For example, this is a
log4j.propertiesfile in the root of a project that sets the logging level to INFO for Spark jobs.
PySpark logging levels should be set as follows:
And Scala logging levels should be set as: