Cloudera Docs

Managing Spark Driver Logs

You can change the default directory for Spark driver logs, or disable the log collection.

The Spark service collects Spark driver logs when Spark applications are run in YARN-client mode or with the Spark Shell. This feature is enabled by default, and the logs are persisted to an HDFS directory and included in YARN Diagnostic Bundles. The default directory for the logs is /user/spark/driverLogs.
  1. In the Cloudera Manager Admin Console, go to the Spark service.
  2. Select the Configuration page.
  3. Configure the default log directory, or disable the log collection:
    • To configure the default log directory, search for the Spark Driver Log Location configuration and change the directory specified in the field.
    • To disable the log collection, search for the Persist Driver Logs to Dfs configuration.
  4. Save the changes.
  5. Deploy the updated client configuration.