Log aggregation properties
You can configure YARN log aggregation for your cluster by specifying values of specific properties depending on your requirements.
Name | Property | Default value | Description |
---|---|---|---|
Enable Log Aggregation | yarn.log-aggregation-enable |
Enabled |
Specifies if log aggregation is enabled. If log aggregation is disabled, YARN container logs are not aggregated. |
Log Aggregation Retention Period | yarn.log-aggregation.retain-seconds |
7 days |
Specifies how long aggregation logs are kept before they are deleted. |
Log Aggregation Status Timeout | yarn.log-aggregation-status.time-out.ms |
10 minutes |
Specifies the maximum amount of time that the NodeManager has for reporting a container's log aggregation status. If no log aggregation status is sent by the NodeManager within the allotted time, the ResourceManager reports a TIME_OUT log aggregation status for that container. |
Supported Log Aggregation File Formats | yarn.log-aggregation.file-formats |
IFile and TFile |
Specifies which log file formats are supported. The first file format in the list is used to write the aggregated logs. TFile format is always added to the end of the list, to support backward compatibility. |
Remote App Log Directory for TFile Format | yarn.log-aggregation.TFile.remote-app-log-dir |
empty |
Specifies the path of the directory where application logs are stored after an application is completed if TFile format is selected for writing. This configuration overwrites the one given in NodeManager level (yarn.nodemanager.remote-app-log-dir). This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read. |
Remote App Log Directory Suffix for TFile Format | yarn.log-aggregation.TFile.remote-app-log-dir-suffix |
empty |
The remote log directory is created at {remote-app-log-dir}/${user}/{thisParam} if TFile format is selected for writing. This configuration overwrites the one given in NodeManager level (yarn.nodemanager.remote-app-log-dir-suffix). This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read. |
Remote App Log Directory for IFile Format | yarn.log-aggregation.IFile.remote-app-log-dor |
empty |
Specifies the path of the directory where application logs are stored after an application is completed if IFile format is given as the file format for writing. This configuration overwrites the one given in NodeManager level (yarn.nodemanager.remote-app-log-dir). This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read. |
Remote App Log Directory Suffix for IFile Format | yarn.log-aggregation.IFile.remote-app-log-dir-suffix |
empty |
The remote log directory is created at {remote-app-log-dir}/${user}/{thisParam} if IFile format is selected for writing. This configuration overwrites the one given in NodeManager level (yarn.nodemanager.remote-app-log-dir-suffix). This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read. |
Log Aggregation Compression Type | yarn.nodemanager.log-aggregation.compression-type |
none |
Specifies the TFile compression type used to compress aggregated logs. Its value can be: gz, lzo or none. |
Remote App Log Directory | yarn.nodemanager.remote-app-log-dir |
/tmp/logs |
Specifies the path of the directory where application logs are stored after an application is completed. This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read. |
Remote App Log Directory Suffix | yarn.nodemanager.remote-app-log-dir-suffix |
logs |
The remote log dir will be created at {yarn.nodemanager.remote-app-log-dir}/${user}/{thisParam}. This property is not backward compatible. If you change its value later, the logs of applications that are already finished cannot be read. |
Log Aggregation Policy | yarn.nodemanager.log-aggregation.policy.class |
All |
Specifies the types of container logs that are uploaded during the log aggregation. Valid values are: Only AM, AM or Failed, All, Only Failed, Failed or Killed, None, or Sample. |