Broker log management
Learn more about the configuration properties related to broker log management, as well as the ways you can monitor log directory space.
Kafka brokers save their data as log segments in a directory. The logs are rotated depending on the size and time settings.
The most common log retention settings to adjust for your cluster are shown below. These are accessible in Cloudera Manager via the tab.
log.dirs: The location for the Kafka data (that is, topic directories and log segments).log.retention.{ms|minutes|hours}: The retention period for the entire log. Any older log segments are removed.log.retention.bytes: The retention size for the entire log.
In addition, there are many more configuration properties available for fine-tuning broker log management. For more information look for the following properties in theBroker Configs section in the upstream Apache Kafka documentation.
log.dirslog.flush.*log.retention.*log.roll.*log.segment.*
You can monitor the log directory space of the Kafka Brokers on the Log Directory Free Capacity chart on the page.
This chart relies on the Broker Log Directory Free Capacity Check trigger. The check is triggered if the capacity of any log directory falls below 10%. The trigger is automatically created for all newly deployed Kafka services. For existing services, you need to create the trigger using the Create Kafka Log Directory Free Capacity Check action on the page following an upgrade.
