Flushing data to disk
Flushing data from memory to disk relieves memory pressure and can improve read
performance by switching from a write-optimized, row-oriented in-memory format in the MemRowSet
, to a read-optimized, column-oriented
format on disk.
Background tasks that flush data include FlushMRSOp
and FlushDeltaMemStoresOp
. The metrics associated
with these operations have the prefix flush_mrs
and flush_dms
,
respectively.
With Kudu 1.4, the maintenance manager aggressively schedules flushes of in-memory data when memory consumption crosses 60 percent of the configured process-wide memory limit. The backpressure mechanism which begins to throttle client writes was also adjusted to not begin throttling until memory consumption reaches 80 percent of the configured limit. These two changes together result in improved write throughput, more consistent latency, and fewer timeouts due to memory exhaustion.