Log aggregation properties

You can configure YARN log aggregation for your cluster by specifying values of specific properties depending on your requirements.

Name Property Default value Description
Enable Log Aggregation yarn.log-aggregation-enable Enabled

Specifies if log aggregation is enabled.

If log aggregation is disabled, YARN container logs are not aggregated.

Log Aggregation Retention Period yarn.log-aggregation.retain-seconds 7 days

Specifies how long aggregation logs are kept before they are deleted.

Log Aggregation Status Timeout yarn.log-aggregation-status.time-out.ms 10 minutes

Specifies the maximum amount of time that the NodeManager has for reporting a container's log aggregation status. If no log aggregation status is sent by the NodeManager within the allotted time, the ResourceManager reports a TIME_OUT log aggregation status for that container.

Supported Log Aggregation File Formats yarn.log-aggregation.file-formats IFile and TFile

Specifies which log file formats are supported. The first file format in the list is used to write the aggregated logs. TFile format is always added to the end of the list, to support backward compatibility.

Remote App Log Directory for TFile Format yarn.log-aggregation.TFile.remote-app-log-dir empty

Specifies the path of the directory where application logs are stored after an application is completed if TFile format is selected for writing. This configuration overwrites the one given in NodeManager level (yarn.nodemanager.remote-app-log-dir).

This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read.

Remote App Log Directory Suffix for TFile Format yarn.log-aggregation.TFile.remote-app-log-dir-suffix empty

The remote log directory is created at {remote-app-log-dir}/${user}/{thisParam} if TFile format is selected for writing. This configuration overwrites the one given in NodeManager level (yarn.nodemanager.remote-app-log-dir-suffix).

This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read.

Remote App Log Directory for IFile Format yarn.log-aggregation.IFile.remote-app-log-dor empty

Specifies the path of the directory where application logs are stored after an application is completed if IFile format is given as the file format for writing. This configuration overwrites the one given in NodeManager level (yarn.nodemanager.remote-app-log-dir).

This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read.

Remote App Log Directory Suffix for IFile Format yarn.log-aggregation.IFile.remote-app-log-dir-suffix empty

The remote log directory is created at {remote-app-log-dir}/${user}/{thisParam} if IFile format is selected for writing. This configuration overwrites the one given in NodeManager level (yarn.nodemanager.remote-app-log-dir-suffix).

This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read.

Log Aggregation Compression Type yarn.nodemanager.log-aggregation.compression-type none

Specifies the TFile compression type used to compress aggregated logs. Its value can be: gz, lzo or none.

Remote App Log Directory yarn.nodemanager.remote-app-log-dir /tmp/logs

Specifies the path of the directory where application logs are stored after an application is completed.

This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read.

Remote App Log Directory Suffix yarn.nodemanager.remote-app-log-dir-siffux logs

The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}.

This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read.

Log Aggregation Roll Monitoring Interval yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds -1 seconds

Defines how often NodeManagers wake up to upload log files.

If this value is 0 or less than 0, the logs are uploaded when the application is completed.

If this value is positive but less than the value of yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds.min an error message is displayed in Cloudera Manager and the value of yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds.min is used instead of the one that you set.

Hard Limit for Log Aggregation Roll Monitoring Intervall yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds.min 1 hours

Defines the hard minimum value for yarn.nodemanager.log-aggregation.roll-monitoring-interval-seconds, if Log Aggregation Roll Monitoring Interval is set to a positive value.

If you set this value less than the default 1 hour an error message is displayed as this can impact the performance of the NodeManager for larger clusters.

Log Aggregation Policy yarn.nodemanager.log-aggregation.policy.class All

Specifies the types of container logs that are uploaded during the log aggregation. Valid values are: Only AM, AM or Failed, All, Only Failed, Failed or Killed, None, or Sample.