Upgrading Log Rotation Configuration

Ambari 2.5.0 provides s a simplified log rotation configuration. These changes will be made automatically during your next stack upgrade, but are not automatically made during the Ambari upgrade. After upgrading Ambari from version 2.x to 2.5.0, if you want to utilize the simplified log rotation configuration, you must update configurations for all services in your cluster, using the following steps:

Steps

ZooKeeper
1. In Ambari Web, browse to ZooKeeper > Configs.
2. Scroll down to Custom zookeeper-log4j.
3. In Custom zookeeper-log4j, click Add Property.
4. In Add Property, type the following properties and values:
  zookeeper_log_max_backup_size=10
  zookeeper_log_number_of_backup_files=10
  For example:
5. Click Add.
6. Browse to Advanced zookeeper-log4j.
7. In Advanced zookeeper-log4j content section, find and replace the following properties and values:
  Find: log4j.appender.ROLLINGFILE.MaxFileSize=<value>
  Replace: log4j.appender.ROLLINGFILE.MaxFileSize={{zookeeper_log_max_backup_size}}MB
  Find: #log4j.appender.ROLLINGFILE.MaxBackupIndex=<value>
  Replace: #log4j.appender.ROLLINGFILE.MaxBackupIndex={{zookeeper_log_number_of_backup_files}}
  For example:
8. In Configs, click Save.
  For example:
9. Restart ZooKeeper, as prompted.
Kafka
1. In Ambari Web, browse to Kafka > Configs.
2. Scroll down to Custom Kafka-log4j.
3. In Custom Kafka-log4j, click Add Property.
4. In Add Property, type the following properties and values:
  kafka_log_maxfilesize=256
  kafka_log_maxbackupindex=20
  controller_log_maxfilesize=256
  controller_log_maxbackupindex=20
5. Click Add.
6. Browse to Advanced kafka-log4j.
7. In Advanced kafka-log4j content section, find and replace the following properties and values:
  Find: log4j.appender.kafkaAppender=org.apache.log4j.DailyRollingFileAppender
  Add: log4j.appender.kafkaAppender.MaxFileSize = {{kafka_log_maxfilesize}}MB
  Add: log4j.appender.kafkaAppender.MaxBackupIndex = {{kafka_log_maxbackupindex}}
  Find: log4j.appender.controllerAppender=org.apache.log4j.DailyRollingFileAppender
  Add: log4j.appender.controllerAppender.MaxFileSize = {{controller_log_maxfilesize}}MB
  Add: log4j.appender.controllerAppender.MaxBackupIndex = {{controller_log_maxbackupindex}}
8. In Configs, click Save.
9. Restart Kafka, as prompted.
Knox
1. In Ambari Web, browse to Knox > Configs.
2. Scroll down to Custom gateway-log4j.
3. In Custom gateway-log4j, click Add Property.
4. In Add Property, type the following properties and values:
  knox_gateway_log_maxfilesize=256
  knox_gateway_log_maxbackupindex=20
5. Click Add.
6. Browse to Advanced gateway-log4j.
7. In Advanced gateway-log4j content section, find and replace the following properties and values:
  Find: log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender
  Add: log4j.appender.drfa.MaxFileSize = {{knox_gateway_log_maxfilesize}}MB
  Add: log4j.appender.drfa.MaxBackupIndex = {{knox_gateway_log_maxbackupindex}}
8. In Configs, click Save.
9. Restart Knox.
Atlas
1. In Ambari Web, browse to Atlas > Configs > Advanced.
2. Scroll down to Custom Atlas-log4j.
3. In Custom Atlas-log4j, click Add Property.
4. In Add Property, type the following properties and values:
  atlas_log_max_backup_size=256
  atlas_log_number_of_backup_files=20
5. Click Add.
6. Browse to Advanced atlas-log4j.
7. In Advanced atlas-log4j content section, find and replace the following properties and values:
  Find: <appender name="FILE" class="org.apache.log4j.DailyRollingFileAppender">
  Add: <param name="MaxFileSize" value="{{atlas_log_max_backup_size}}MB" >
  Add: <param name="MaxBackupIndex" value="{{atlas_log_number_of_backup_files}}" >
8. In Configs, click Save.
9. Restart Atlas, as prompted.
Ranger
1. In Ambari Web, browse to Ranger > Configs > Advanced.
2. Scroll down to Custom admin-log4j.
3. In Custom admin-log4j, click Add Property.
4. In Add Property, type the following properties and values:
  ranger_xa_log_maxfilesize=256
  ranger_xa_log_maxbackupindex=20
5. Click Add.
6. Browse to Advanced admin-log4j.
7. In Advanced admin-log4j content section, find and replace the following properties and values:
  Find: log4j.appender.xa_log_appender=org.apache.log4j.DailyRollingFileAppender
  Add: log4j.appender.xa_log_appender.MaxFileSize={{ranger_xa_log_maxfilesize}}MB
  Add: log4j.appender.xa_log_appender.MaxBackupIndex={{ranger_xa_log_maxbackupindex}}
8. Scroll down to Custom usersync-log4j.
9. In Custom usersync-log4j, click Add Property.
10. In Add Property, type the following properties and values:
  ranger_usersync_log_maxfilesize=256
  ranger_usersync_log_number_of_backup_files=20
11. Click Add.
12. Browse to Advanced usersync-log4j.
13. In Advanced usersync-log4j content section, find and replace the following properties and values:
  Find: log4j.appender.logFile=org.apache.log4j.DailyRollingFileAppender
  Add: log4j.appender.logFile.MaxFileSize = {{ranger_usersync_log_maxfilesize}}MB
  Add: log4j.appender.logFile.MaxBackupIndex = {{ranger_usersync_log_number_of_backup_files}}
14. Scroll down to Custom tagsync-log4j.
15. In Custom tagsync-log4j, click Add Property.
16. In Add Property, type the following properties and values:
  ranger_tagsync_log_maxfilesize=256
  ranger_tagsync_log_number_of_backup_files=20
17. Click Add.
18. Browse to Advanced tagsync-log4j.
19. In Advanced tagsync-log4j content section, find and replace the following properties and values:
  Find: log4j.appender.logFile=org.apache.log4j.DailyRollingFileAppender
  Add: log4j.appender.logFile.MaxFileSize = {{ranger_tagsync_log_maxfilesize}}MB
  Add: log4j.appender.logFile.MaxBackupIndex = {{ranger_tagsync_log_number_of_backup_files}}
20. In Configs, click Save.
21. Restart Ranger, as prompted.
Ranger-KMS
1. In Ambari Web, browse to Ranger-KMS > Configs > Advanced.
2. Scroll down to Custom kms-log4j.
3. In Custom kms-log4j, click Add Property.
4. In Add Property, type the following properties and values:
  ranger_kms_log_maxfilesize=256
  ranger_kms_log_maxbackupindex=20
  ranger_kms_audit_log_maxfilesize=256
  ranger_kms_audit_log_maxbackupindex=20
5. Click Add.
6. Browse to Advanced kms-log4j .
7. In Advanced kms-log4j content section, find and replace the following properties and values:
  Find: log4j.appender.kms=org.apache.log4j.DailyRollingFileAppender
  Add: log4j.appender.kms.MaxFileSize = {{ranger_kms_log_maxfilesize}}MB
  Add: log4j.appender.kms.MaxBackupIndex = {{ranger_kms_log_maxbackupindex}}
  Find: log4j.appender.kms-audit=org.apache.log4j.DailyRollingFileAppender
  Add: log4j.appender.kms-audit.MaxFileSize={{ranger_kms_audit_log_maxfilesize}}MB
  Add: log4j.appender.kms-audit.MaxBackupIndex = {{ranger_kms_audit_log_maxbackupindex}}
8. In Configs, click Save.
9. Restart Ranger-KMS.
HBase
1. In Ambari Web, browse to HBase > Configs > Advanced.
2. Scroll down to Custom hbase-log4j.
3. In Custom hbase-log4j, click Add Property.
4. In Add Property, type the following properties and values:
  hbase_log_maxfilesize=256
  hbase_log_maxbackupindex=20
  hbase_security_log_maxfilesize=256
  hbase_security_log_maxbackupindex=20
5. Click Add.
6. Browse to Advanced hbase-log4j .
7. In Advanced hbase-log4j content section, find and replace the following properties and values:
  Find: hbase.log.maxfilesize=<value>MB
  Replace: hbase.log.maxfilesize={{hbase_log_maxfilesize}}MB
  Find: hbase.log.maxbackupindex=<value>
  Replace: hbase.log.maxbackupindex={{hbase_log_maxbackupindex}}
  Find: hbase.security.log.maxfilesize=<value>MB
  Replace: hbase.security.log.maxfilesize={{hbase_security_log_maxfilesize}}MB
  Find: hbase.security.log.maxbackupindex=<value>
  Replace: hbase.security.log.maxbackupindex={{hbase_security_log_maxbackupindex}}
8. In Configs, click Save.
9. Restart HBase.
Hive
1. In Ambari Web, browse to Hive > Configs > Advanced.
2. Scroll down to Custom hive-log4j.
3. In Custom hive-log4j, click Add Property.
4. In Add Property, type the following properties and values:
  hive_log_maxfilesize=256
  hive_log_maxbackupindex=30
5. Click Add.
6. Browse to Advanced hive-log4j .
7. In Advanced hive-log4j content section, find and replace the following properties and values:
  Find: #log4j.appender.DRFA.MaxBackupIndex=<value>
  Replace: #log4j.appender.DRFA.MaxBackupIndex= {{hive_log_maxbackupindex}}
  Find: log4j.appender.DRFA.MaxBackupIndex=<value>
  Add: log4j.appender.DRFA.MaxFileSize = {{hive_log_maxfilesize}}MB
8. Scroll down to Custom llap-daemon-log4j.
9. In Custom llap-daemon-log4j, click Add Property.
10. In Add Property, type the following properties and values:
  hive_llap_log_maxfilesize=256
  hive_log_maxbackupindex=240
11. Click Add.
12. Browse to llap-daemon-log4j.
13. In llap-daemon-log4j content section, find and replace the following properties and values:
  Find: property.llap.daemon.log.maxfilesize =<value>MB
  Replace: property.llap.daemon.log.maxfilesize = {{hive_llap_log_maxfilesize}}MB
  Find: property.llap.daemon.log.maxbackupindex=<value>
  Replace: property.llap.daemon.log.maxbackupindex={{hive_llap_log_maxbackupindex}}
14. Scroll down to Custom webhcat-log4j.
15. In Custom webhcat-log4j, click Add Property.
16. In Add Property, type the following properties and values:
  webhcat_log_maxfilesize=256
  webhcat_log_maxbackupindex=240
17. Click Add.
18. Browse to webhcat-log4j .
19. In webhcat-log4j content section, find and replace the following properties and values:
  Find: log4j.appender.standard = org.apache.log4j.DailyRollingFileAppender
  Add: log4j.appender.standard.MaxFileSize = {{webhcat_log_maxfilesize}}MB
  Add: log4j.appender.standard.MaxBackupIndex = {{webhcat_log_maxbackupindex}}
20. In Configs, click Save.
21. Restart Hive, as prompted.
Storm
1. In Ambari Web, browse to Storm > Configs.
2. Scroll down to Custom cluster-log4j property.
3. In Custom cluster-log4j property, click Add Property.
4. In Add Property, type the following properties and values:
  storm_a1_maxfilesize=100
  storm_a1_maxbackupindex=9
5. Click Add.
6. Browse to Advanced storm-cluster-log4j .
7. In Advanced storm-cluster-log4j content section, find and replace the following properties and values:
  Find: In RollingFile="A1"<SizeBasedTriggeringPolicy size=”<value>MB"/>
  Replace: <SizeBasedTriggeringPolicy size="{{storm_a1_maxfilesize}}MB"/>
  Find: In RollingFile="A1"<DefaultRolloverStrategy max="<value>"/>
  Replace: <DefaultRolloverStrategy max="{{storm_a1_maxbackupindex}}"/>
8. Scroll down to Custom worker-log4j property.
9. In Custom worker-log4j property, click Add Property.
10. In Add Property, type the following properties and values:
  storm_wrkr_a1_maxfilesize=100
  storm_wrkr_a1_maxbackupindex=9
  storm_wrkr_out_maxfilesize=100
  storm_wrkr_out_maxbackupindex=4
  storm_wrkr_err_maxfilesize=100
  storm_wrkr_err_maxbackupindex=4
11. Click Add.
12. Browse to Advanced storm-worker-log4j .
13. In Advanced storm-worker-log4j content section, find and replace the following properties and values:
  Find: In RollingFile="A1"<SizeBasedTriggeringPolicy size="<value> MB"/>
  Replace: <SizeBasedTriggeringPolicy size="{{storm_wrkr_a1_maxfilesize}} MB"/>
  Find: In RollingFile="A1"<DefaultRolloverStrategy max="<value>"/>
  Replace: <DefaultRolloverStrategy max="{{storm_wrkr_a1_maxbackupindex}}"/>
  Find: In RollingFile="STDOUT"<SizeBasedTriggeringPolicy size="<value>" MB/>
  Replace: <SizeBasedTriggeringPolicy size="{{storm_wrkr_out_maxfilesize}} MB"/>
  Find: In RollingFile="STDOUT"<DefaultRolloverStrategy max="<value>"/>
  Replace: <DefaultRolloverStrategy max="{{storm_wrkr_out_maxbackupindex}}"/>
  Find: In RollingFile=”STDERR”<SizeBasedTriggeringPolicy size=”<value>” MB/>
  Replace: <SizeBasedTriggeringPolicy size="{{storm_wrkr_err_maxfilesize}} MB"/>
  Find: In RollingFile="STDOUT"<DefaultRolloverStrategy max="<value>"/>
  Replace: <DefaultRolloverStrategy max="{{storm_wrkr_err_maxbackupindex}}"/>
14. In Configs, click Save.
15. Restart Storm, as prompted.
Falcon
1. In Ambari Web, browse to Falcon > Configs.
2. Scroll down to Custom falcon-log4j.
3. In Custom falcon-log4j, click Add Property.
4. In Add Property, type the following properties and values:
  falcon_log_maxfilesize=256
  falcon_log_maxbackupindex=20
  falcon_security_log_maxfilesize=256
  falcon_security_log_maxbackupindex=20
5. Click Add.
6. Browse to Advanced falcon-log4j .
7. In Advanced falcon-log4j content section, find and replace the following properties and values:
  Find: <appender name="FILE" class="org.apache.log4j.DailyRollingFileAppender">
  Add: <param name="MaxFileSize" value="{{falcon_log_maxfilesize}}MB" />
  Add: <param name="MaxBackupIndex" value="{{falcon_log_maxbackupindex}}" />
  Find: <appendername="SECURITY"class="org.apache.log4j.DailyRollingFileAppender">
  Add: <param name="MaxFileSize" value="{{falcon_security_log_maxfilesize}}MB"/>
  Add: <param name="MaxBackupIndex" value="{{falcon_security_log_maxbackupindex}}"/>
8. In Configs, click Save.
9. Restart Falcon.
Oozie
1. In Ambari Web, browse to Oozie > Configs.
2. Scroll down to Custom oozie-log4j.
3. In Custom oozie-log4j, click Add Property.
4. In Add Property, type the following properties and values:
  oozie_log_maxhistory=720
5. Click Add.
6. Browse to Advanced oozie-log4j .
7. In Advanced oozie-log4j content section, find and replace the following properties and values:
  Find: log4j.appender.oozie.RollingPolicy.MaxHistory=720
  Replace: log4j.appender.oozie.RollingPolicy.MaxHistory={{oozie_log_maxhistory}}
8. In Configs, click Save.
9. Restart Oozie, as prompted.
HDFS
1. In Ambari Web, browse to HDFS > Configs > Advanced.
2. Scroll down to Custom HDFS-log4j.
3. In Custom HDFS-log4j, click Add Property.
4. In Add Property, type the following properties and values:
  hadoop_security_log_max_backup_size = 256
  hadoop_security_log_number_of_backup_files = 20
  hadoop_log_max_backup_size = 256
  hadoop_log_number_of_backup_files = 10
5. Click Add.
6. Browse to Advanced hdfs-log4j .
7. In Advanced hdfs-log4j content section, find and replace the following properties and values:
  Find: hadoop.security.log.maxfilesize= <value>MB
  Replace: hadoop.security.log.maxfilesize={{hadoop_security_log_max_backup_size}}MB
  Find: hadoop.security.log.maxbackupindex=20
  Replace: hadoop.security.log.maxbackupindex={{hadoop_security_log_number_of_backup_files}}
  Find: log4j.appender.RFA.MaxFileSize=<value>MB
  Replace: log4j.appender.RFA.MaxFileSize={{hadoop_log_max_backup_size}}MB
  Find: log4j.appender.RFA.MaxBackupIndex=<value>
  Replace: log4j.appender.RFA.MaxBackupIndex={{hadoop_log_number_of_backup_files}}
8. In Configs, click Save.
9. Restart HDFS, as prompted.
YARN
1. In Ambari Web, browse to YARN > Configs > Advanced.
2. Scroll down to Custom YARN-log4j property.
3. In Custom YARN-log4j property, click Add Property.
4. In Add Property, type the following properties and values:
  yarn_rm_summary_log_max_backup_size=256
  yarn_rm_summary_log_number_of_backup_files=20
5. Click Add.
6. Browse to Advanced yarn-log4j .
7. In Advanced yarn-log4j content section, find and replace the following properties and values:
  Find: log4j.appender.RMSUMMARY.MaxFileSize=<value>MB
  Replace: log4j.appender.RMSUMMARY.MaxFileSize={{yarn_rm_summary_log_max_backup_size}}MB
  Find: log4j.appender.RMSUMMARY.MaxBackupIndex=<value>
  Replace: log4j.appender.RMSUMMARY.MaxBackupIndex={{yarn_rm_summary_log_number_of_backup_files}}
8. In Configs, click Save.
9. Restart YARN, as prompted.

​Upgrading Log Rotation Configuration

Upgrading Log Rotation Configuration