Cloudera Docs » 2.6.2.2 » Apache Ambari Upgrade
Apache Ambari Upgrade
Also available as:
PDF

​Upgrading Log Rotation Configuration

Starting with Ambari 2.6.0, log rotation configuration capability is enabled. Ambari 2.6.0 provides s a simplified log rotation configuration. These changes will be made automatically during your next stack upgrade, but are not automatically made during the Ambari upgrade. After upgrading Ambari from version 2.x to 2.6.0, if you want to utilize the simplified log rotation configuration, you must update configurations for all services in your cluster, using the following steps:

Steps

  1. ZooKeeper

    1. In Ambari Web, browse to ZooKeeper > Configs.

    2. Scroll down to Custom zookeeper-log4j.

    3. In Custom zookeeper-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      zookeeper_log_max_backup_size=10

      zookeeper_log_number_of_backup_files=10

      For example:

    5. Click Add.

    6. Browse to Advanced zookeeper-log4j.

    7. In Advanced zookeeper-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.ROLLINGFILE.MaxFileSize=<value>

      Replace: log4j.appender.ROLLINGFILE.MaxFileSize={{zookeeper_log_max_backup_size}}MB

      Find: #log4j.appender.ROLLINGFILE.MaxBackupIndex=<value>

      Replace: log4j.appender.ROLLINGFILE.MaxBackupIndex={{zookeeper_log_number_of_backup_files}}

      For example:

    8. In Configs, click Save.

      For example:

    9. Restart ZooKeeper, as prompted.

  2. Kafka

    1. In Ambari Web, browse to Kafka > Configs.

    2. Scroll down to Custom Kafka-log4j.

    3. In Custom Kafka-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      kafka_log_maxfilesize=256

      kafka_log_maxbackupindex=20

      controller_log_maxfilesize=256

      controller_log_maxbackupindex=20

    5. Click Add.

    6. Browse to Advanced kafka-log4j.

    7. In Advanced kafka-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.kafkaAppender=<appender type>

      Replace: log4j.appender.kafkaAppender=org.apache.log4j.RollingFileAppender

      Add: log4j.appender.kafkaAppender.MaxFileSize = {{kafka_log_maxfilesize}}MB

      Add: log4j.appender.kafkaAppender.MaxBackupIndex = {{kafka_log_maxbackupindex}}

      Find: log4j.appender.controllerAppender=org.apache.log4j.RollingFileAppender

      Add: log4j.appender.controllerAppender.MaxFileSize = {{controller_log_maxfilesize}}MB

      Add: log4j.appender.controllerAppender.MaxBackupIndex = {{controller_log_maxbackupindex}}

    8. In Configs, click Save.

    9. Restart Kafka, as prompted.

  3. Knox

    1. In Ambari Web, browse to Knox > Configs.

    2. Scroll down to Custom gateway-log4j.

    3. In Custom gateway-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      knox_gateway_log_maxfilesize=256

      knox_gateway_log_maxbackupindex=20

    5. Click Add.

    6. Browse to Advanced gateway-log4j.

    7. In Advanced gateway-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.drfa=org.apache.log4j.DailyRollingFileAppender

      Replace: log4j.appender.drfa=org.apache.log4j.RollingFileAppender

      Add: log4j.appender.drfa.MaxFileSize = {{knox_gateway_log_maxfilesize}}MB

      Add: log4j.appender.drfa.MaxBackupIndex = {{knox_gateway_log_maxbackupindex}}

    8. In Configs, click Save.

    9. Restart Knox.

  4. Atlas

    1. In Ambari Web, browse to Atlas > Configs > Advanced.

    2. Scroll down to Custom Atlas-log4j.

    3. In Custom Atlas-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      atlas_log_max_backup_size=256

      atlas_log_number_of_backup_files=20

    5. Click Add.

    6. Browse to Advanced atlas-log4j.

    7. In Advanced atlas-log4j content section, find and replace the following properties and values:

      Find: <appender name="FILE" class="org.apache.log4j.DailyRollingFileAppender”>

      Replace: <appender name="FILE" class="org.apache.log4j.RollingFileAppender”>

      Add: <param name="MaxFileSize" value="{{atlas_log_max_backup_size}}MB" >

      Add: <param name="MaxBackupIndex" value="{{atlas_log_number_of_backup_files}}" >

    8. In Configs, click Save.

    9. Restart Atlas, as prompted.

  5. Ranger

    1. In Ambari Web, browse to Ranger > Configs > Advanced.

    2. Scroll down to Custom admin-log4j.

    3. In Custom admin-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      ranger_xa_log_maxfilesize=256

      ranger_xa_log_maxbackupindex=20

    5. Click Add.

    6. Browse to Advanced admin-log4j.

    7. In Advanced admin-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.xa_log_appender=<appender type>

      Replace: log4j.appender.xa_log_appender=org.apache.log4j.RollingFileAppender

      Add: log4j.appender.xa_log_appender.MaxFileSize={{ranger_xa_log_maxfilesize}}MB

      Add: log4j.appender.xa_log_appender.MaxBackupIndex={{ranger_xa_log_maxbackupindex}}

    8. Scroll down to Custom usersync-log4j.

    9. In Custom usersync-log4j, click Add Property.

    10. In Add Property, type the following properties and values:

      ranger_usersync_log_maxfilesize=256

      ranger_usersync_log_number_of_backup_files=20

    11. Click Add.

    12. Browse to Advanced usersync-log4j.

    13. In Advanced usersync-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.logFile=<appender type>

      Replace: log4j.appender.logFile=org.apache.log4j.RollingFileAppender

      Add: log4j.appender.logFile.MaxFileSize = {{ranger_usersync_log_maxfilesize}}MB

      Add: log4j.appender.logFile.MaxBackupIndex = {{ranger_usersync_log_number_of_backup_files}}

    14. Scroll down to Custom tagsync-log4j.

    15. In Custom tagsync-log4j, click Add Property.

    16. In Add Property, type the following properties and values:

      ranger_tagsync_log_maxfilesize=256

      ranger_tagsync_log_number_of_backup_files=20

    17. Click Add.

    18. Browse to Advanced tagsync-log4j.

    19. In Advanced tagsync-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.logFile=<appender type>

      Replace: log4j.appender.logFile=org.apache.log4j.RollingFileAppender

      Add: log4j.appender.logFile.MaxFileSize = {{ranger_tagsync_log_maxfilesize}}MB

      Add: log4j.appender.logFile.MaxBackupIndex = {{ranger_tagsync_log_number_of_backup_files}}

    20. In Configs, click Save.

    21. Restart Ranger, as prompted.

  6. Ranger-KMS

    1. In Ambari Web, browse to Ranger-KMS > Configs > Advanced.

    2. Scroll down to Custom kms-log4j.

    3. In Custom kms-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      ranger_kms_log_maxfilesize=256

      ranger_kms_log_maxbackupindex=20

      ranger_kms_audit_log_maxfilesize=256

      ranger_kms_audit_log_maxbackupindex=20

    5. Click Add.

    6. Browse to Advanced kms-log4j .

    7. In Advanced kms-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.kms=<appender type>

      Replace: log4j.appender.kms=org.apache.log4j.RollingFileAppender

      Add: log4j.appender.kms.MaxFileSize = {{ranger_kms_log_maxfilesize}}MB

      Add: log4j.appender.kms.MaxBackupIndex = {{ranger_kms_log_maxbackupindex}}

      Find: log4j.appender.kms-audit=org.apache.log4j.RollingFileAppender

      Add: log4j.appender.kms-audit.MaxFileSize={{ranger_kms_audit_log_maxfilesize}}MB

      Add: log4j.appender.kms-audit.MaxBackupIndex = {{ranger_kms_audit_log_maxbackupindex}}

    8. In Configs, click Save.

    9. Restart Ranger-KMS.

  7. HBase

    1. In Ambari Web, browse to HBase > Configs > Advanced.

    2. Scroll down to Custom hbase-log4j.

    3. In Custom hbase-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      hbase_log_maxfilesize=256

      hbase_log_maxbackupindex=20

      hbase_security_log_maxfilesize=256

      hbase_security_log_maxbackupindex=20

    5. Click Add.

    6. Browse to Advanced hbase-log4j .

    7. In Advanced hbase-log4j content section, find and replace the following properties and values:

      Find: hbase.log.maxfilesize=<value>MB

      Replace: hbase.log.maxfilesize={{hbase_log_maxfilesize}}MB

      Find: hbase.log.maxbackupindex=<value>

      Replace: hbase.log.maxbackupindex={{hbase_log_maxbackupindex}}

      Find: hbase.security.log.maxfilesize=<value>MB

      Replace: hbase.security.log.maxfilesize={{hbase_security_log_maxfilesize}}MB

      Find: hbase.security.log.maxbackupindex=<value>

      Replace: hbase.security.log.maxbackupindex={{hbase_security_log_maxbackupindex}}

    8. In Configs, click Save.

    9. Restart HBase.

  8. Hive

    1. In Ambari Web, browse to Hive > Configs > Advanced.

    2. Scroll down to Custom hive-log4j.

    3. In Custom hive-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      hive_log_maxfilesize=256

      hive_log_maxbackupindex=30

    5. Click Add.

    6. Browse to Advanced hive-log4j .

    7. In Advanced hive-log4j content section, find and replace the following properties and values:

      Find:log4j.appender.DRFA=org.apache.log4j.DailyRollingFileAppender

      Replace: log4j.appender.DRFA=org.apache.log4j.RollingFileAppender

      Find: #log4j.appender.DRFA.MaxBackupIndex=<value>

      Replace: log4j.appender.DRFA.MaxBackupIndex={{hive_log_maxbackupindex}}

    8. Scroll down to Custom llap-daemon-log4j.

    9. In Custom llap-daemon-log4j, click Add Property.

    10. In Add Property, type the following properties and values:

      hive_llap_log_maxfilesize=256

      hive_log_maxbackupindex=240

    11. Click Add.

    12. Browse to llap-daemon-log4j.

    13. In llap-daemon-log4j content section, find and replace the following properties and values:

      Find: property.llap.daemon.log.maxfilesize =<value>MB

      Replace: property.llap.daemon.log.maxfilesize = {{hive_llap_log_maxfilesize}}MB

      Find: property.llap.daemon.log.maxbackupindex=<value>

      Replace: property.llap.daemon.log.maxbackupindex={{hive_llap_log_maxbackupindex}}

    14. Scroll down to Custom webhcat-log4j.

    15. In Custom webhcat-log4j, click Add Property.

    16. In Add Property, type the following properties and values:

      webhcat_log_maxfilesize=256

      webhcat_log_maxbackupindex=240

    17. Click Add.

    18. Browse to webhcat-log4j .

    19. In webhcat-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.standard = org.apache.log4j.RollingFileAppender

      Add: log4j.appender.standard.MaxFileSize = {{webhcat_log_maxfilesize}}MB

      Add: log4j.appender.standard.MaxBackupIndex = {{webhcat_log_maxbackupindex}}

    20. In Configs, click Save.

    21. Restart Hive, as prompted.

  9. Storm

    1. In Ambari Web, browse to Storm > Configs.

    2. Scroll down to Custom cluster-log4j property.

    3. In Custom cluster-log4j property, click Add Property.

    4. In Add Property, type the following properties and values:

      storm_a1_maxfilesize=100

      storm_a1_maxbackupindex=9

    5. Click Add.

    6. Browse to Advanced storm-cluster-log4j .

    7. In Advanced storm-cluster-log4j content section, find and replace the following properties and values:

      Find: In RollingFile="A1"<SizeBasedTriggeringPolicy size=”<value>MB"/>

      Replace: <SizeBasedTriggeringPolicy size="{{storm_a1_maxfilesize}}MB"/>

      Find: In RollingFile="A1"<DefaultRolloverStrategy max="<value>"/>

      Replace: <DefaultRolloverStrategy max="{{storm_a1_maxbackupindex}}"/>

    8. Scroll down to Custom worker-log4j property.

    9. In Custom worker-log4j property, click Add Property.

    10. In Add Property, type the following properties and values:

      storm_wrkr_a1_maxfilesize=100

      storm_wrkr_a1_maxbackupindex=9

      storm_wrkr_out_maxfilesize=100

      storm_wrkr_out_maxbackupindex=4

      storm_wrkr_err_maxfilesize=100

      storm_wrkr_err_maxbackupindex=4

    11. Click Add.

    12. Browse to Advanced storm-worker-log4j .

    13. In Advanced storm-worker-log4j content section, find and replace the following properties and values:

      Find: In RollingFile="A1"<SizeBasedTriggeringPolicy size="<value> MB"/>

      Replace: <SizeBasedTriggeringPolicy size="{{storm_wrkr_a1_maxfilesize}} MB"/>

      Find: In RollingFile="A1"<DefaultRolloverStrategy max="<value>"/>

      Replace: <DefaultRolloverStrategy max="{{storm_wrkr_a1_maxbackupindex}}"/>

      Find: In RollingFile="STDOUT"<SizeBasedTriggeringPolicy size="<value>" MB/>

      Replace: <SizeBasedTriggeringPolicy size="{{storm_wrkr_out_maxfilesize}} MB"/>

      Find: In RollingFile="STDOUT"<DefaultRolloverStrategy max="<value>"/>

      Replace: <DefaultRolloverStrategy max="{{storm_wrkr_out_maxbackupindex}}"/>

      Find: In RollingFile=”STDERR”<SizeBasedTriggeringPolicy size=”<value>” MB/>

      Replace: <SizeBasedTriggeringPolicy size="{{storm_wrkr_err_maxfilesize}} MB"/>

      Find: In RollingFile="STDOUT"<DefaultRolloverStrategy max="<value>"/>

      Replace: <DefaultRolloverStrategy max="{{storm_wrkr_err_maxbackupindex}}"/>

    14. In Configs, click Save.

    15. Restart Storm, as prompted.

  10. Falcon

    1. In Ambari Web, browse to Falcon > Configs.

    2. Scroll down to Custom falcon-log4j.

    3. In Custom falcon-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      falcon_log_maxfilesize=256

      falcon_log_maxbackupindex=20

      falcon_security_log_maxfilesize=256

      falcon_security_log_maxbackupindex=20

    5. Click Add.

    6. Browse to Advanced falcon-log4j .

    7. In Advanced falcon-log4j content section, find and replace the following properties and values:

      Find: <appender name="FILE" class="org.apache.log4j.DailyRollingFileAppender”>

      Replace: <appender name="FILE" class="org.apache.log4j.RollingFileAppender”>

      Add: <param name="MaxFileSize" value="{{falcon_log_maxfilesize}}MB" />

      Add: <param name="MaxBackupIndex" value="{{falcon_log_maxbackupindex}}" />

      Find: <appendername="SECURITY"class="org.apache.log4j.RollingFileAppender">

      Add: <param name="MaxFileSize" value="{{falcon_security_log_maxfilesize}}MB"/>

      Add: <param name="MaxBackupIndex" value="{{falcon_security_log_maxbackupindex}}"/>

    8. In Configs, click Save.

    9. Restart Falcon.

  11. Oozie

    1. In Ambari Web, browse to Oozie > Configs.

    2. Scroll down to Custom oozie-log4j.

    3. In Custom oozie-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      oozie_log_maxhistory=720

    5. Click Add.

    6. Browse to Advanced oozie-log4j .

    7. In Advanced oozie-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.oozie.RollingPolicy.MaxHistory=720

      Replace: log4j.appender.oozie.RollingPolicy.MaxHistory={{oozie_log_maxhistory}}

    8. In Configs, click Save.

    9. Restart Oozie, as prompted.

  12. HDFS

    1. In Ambari Web, browse to HDFS > Configs > Advanced.

    2. Scroll down to Custom HDFS-log4j.

    3. In Custom HDFS-log4j, click Add Property.

    4. In Add Property, type the following properties and values:

      hadoop_security_log_max_backup_size = 256

      hadoop_security_log_number_of_backup_files = 20

      hadoop_log_max_backup_size = 256

      hadoop_log_number_of_backup_files = 10

    5. Click Add.

    6. Browse to Advanced hdfs-log4j .

    7. In Advanced hdfs-log4j content section, find and replace the following properties and values:

      Find: hadoop.security.log.maxfilesize= <value>MB

      Replace: hadoop.security.log.maxfilesize={{hadoop_security_log_max_backup_size}}MB

      Find: hadoop.security.log.maxbackupindex=20

      Replace: hadoop.security.log.maxbackupindex={{hadoop_security_log_number_of_backup_files}}

      Find: log4j.appender.RFA.MaxFileSize=<value>MB

      Replace: log4j.appender.RFA.MaxFileSize={{hadoop_log_max_backup_size}}MB

      Find: log4j.appender.RFA.MaxBackupIndex=<value>

      Replace: log4j.appender.RFA.MaxBackupIndex={{hadoop_log_number_of_backup_files}}

    8. In Configs, click Save.

    9. Restart HDFS, as prompted.

  13. YARN

    1. In Ambari Web, browse to YARN > Configs > Advanced.

    2. Scroll down to Custom YARN-log4j property.

    3. In Custom YARN-log4j property, click Add Property.

    4. In Add Property, type the following properties and values:

      yarn_rm_summary_log_max_backup_size=256

      yarn_rm_summary_log_number_of_backup_files=20

    5. Click Add.

    6. Browse to Advanced yarn-log4j .

    7. In Advanced yarn-log4j content section, find and replace the following properties and values:

      Find: log4j.appender.RMSUMMARY.MaxFileSize=<value>MB

      Replace: log4j.appender.RMSUMMARY.MaxFileSize={{yarn_rm_summary_log_max_backup_size}}MB

      Find: log4j.appender.RMSUMMARY.MaxBackupIndex=<value>

      Replace: log4j.appender.RMSUMMARY.MaxBackupIndex={{yarn_rm_summary_log_number_of_backup_files}}

    8. In Configs, click Save.

    9. Restart YARN, as prompted.

© 2012–2021 by Cloudera, Inc.
Document licensed under the Creative Commons Attribution ShareAlike 4.0 License.
Cloudera.com | Documentation | Support | Community