Spark Logging Configuration
Cloudera Data Science Workbench allows you to update Spark’s internal logging configuration on a per-project basis. Spark 2 uses Apache Log4j, which can be configured through a properties file.
log4j.properties file found in the
root of your project will be appended to the existing Spark logging properties for every
session and job. To specify a custom location, set the environmental variable
LOG4J_CONFIG to the file location relative to your
The Log4j documentation has more details on logging options.
log4j.propertiesfile in the root of a project that sets the logging level to INFO for Spark jobs.