An overview on configuration properties related to log compaction.
The following cluster-wide configuration settings can be used to fine tune log compaction:
log.cleaner.threadscontrols how many background threads are responsible for log compaction. Increasing this value improves performance of log compaction at the cost of increased I/O activity.
log.cleaner.io.max.bytes.per.secondthrottles log cleaner’s I/O activity so that the sum of its read and write is less than this value on average.
log.cleaner.dedupe.buffer.sizespecifies memory used for log compaction across all cleaner threads.
log.cleaner.io.buffer.sizecontrols total memory used for log cleaner I/O buffers across all cleaner threads.
log.cleaner.min.compaction.lag.mscontrols how long messages are left uncompacted.
log.cleaner.io.buffer.load.factorcontrols log cleaner’s load factor for the dedupe buffer. Increasing this value allows the system to clean more logs at once but increases hash collisions.
log.cleaner.backoff.mscontrols how long to wait until the next check if there is no log to compact.