Record management
Learn how you can manage records.
There are two pieces to record management, log segments and log cleaner.
As part of the general data storage, Kafka rolls logs periodically based on size or time limits. Once either limit is hit, a new log segment is created with the all new data being placed there, while older log segments should generally no longer change. This helps limit the risk of data loss or corruption to a single segment instead of the entire log.
log.roll.{ms|hours}: The time period for each log segment. Once the current segment is older than this value, it goes through log segment rotation.log.segment.bytes: The maximum size for a single log segment.
There is an alternative to simply removing log segments for a partition. There is another feature based on the log cleaner. When the log cleaner is enabled, individual records in older log segments can be managed differently:
log.cleaner.enable: This is a global setting in Kafka to enable the log cleaner.cleanup.policy: This is a per-topic property that is usually set at topic creation time. There are two valid values for this property,deleteandcompact.log.cleaner.min.compaction.lag.ms: This is the retention period for the “head” of the log. Only records outside of this retention period will be compacted by the log cleaner.
The compact policy, also called log compaction, assumes that the "most
recent Kafka record is important." Some examples include tracking a current email address or
tracking a current mailing address. With log compaction, older records with the same key are
removed from a log segment and the latest one is kept. This effectively removes some offsets
from the partition.
