Log cleaner

An overview on configuration properties related to log compaction.

The following cluster-wide configuration settings can be used to fine tune log compaction:

• log.cleaner.threads controls how many background threads are responsible for log compaction. Increasing this value improves performance of log compaction at the cost of increased I/O activity.
• log.cleaner.io.max.bytes.per.second throttles log cleaner’s I/O activity so that the sum of its read and write is less than this value on average.
• log.cleaner.dedupe.buffer.size specifies memory used for log compaction across all cleaner threads.
• log.cleaner.io.buffer.size controls total memory used for log cleaner I/O buffers across all cleaner threads.
• log.cleaner.min.compaction.lag.ms controls how long messages are left uncompacted.
• log.cleaner.io.buffer.load.factor controls log cleaner’s load factor for the dedupe buffer. Increasing this value allows the system to clean more logs at once but increases hash collisions.
• log.cleaner.backoff.ms controls how long to wait until the next check if there is no log to compact.