Kudu has a hard and soft memory limit. The hard memory limit is the maximum amount a
Kudu process is allowed to use, and is controlled by the
--memory_limit_hard_bytes flag. The soft memory limit is a percentage of the hard
memory limit, controlled by the flag
memory_limit_soft_percentage and with a default value of 80%, that determines the
amount of memory a process may use before it will start rejecting some write
If the logs or RPC traces contain messages such as the following example, then Kudu is rejecting writes due to memory back pressure. This may result in write timeouts.
Service unavailable: Soft memory limit exceeded (at 96.35% of capacity)
There are several ways to relieve the memory pressure on Kudu:
If the host has more memory available for Kudu, increase
Increase the rate at which Kudu can flush writes from memory to disk by increasing the number of disks or increasing the number of maintenance manager threads
--maintenance_manager_num_threads. Generally, the recommended ratio of maintenance manager threads to data directories is 1:3.
Reduce the volume of writes flowing to Kudu on the application side.
Finally, in Kudu versions 1.7 and lower, check the value of the
--block_cache_capacity_mb setting. This setting determines the
maximum size of Kudu's block cache. While a higher value can help with read and write
performance, setting it too high as a percentage of the
--memory_limit_hard_bytes setting is harmful. Do not raise
--memory_pressure_percentage (default 60%) of
--memory_limit_hard_bytes, as this will cause Kudu to flush
aggressively even if write throughput is low. The recommended value for
--block_cache_capacity_mb is below the following:
With the defaults, this means the
not exceed 30% of
In Kudu 1.8 and higher, servers will refuse to start if the block cache capacity exceeds the memory pressure threshold.