Verifying if a memory limit is sufficient

After configuring an appropriate memory limit with --memory_limit_hard_bytes, run a workload and monitor the Kudu tablet server process’s RAM usage. The memory usage should stay around 50-75% of the hard limit, with occasional spikes above 75% but below 100%. If the tablet server runs above 75% consistently, the memory limit should be increased.

Additionally, it’s also useful to monitor the logs for memory rejections, which look like:

Service unavailable: Soft memory limit exceeded (at 96.35% of capacity)

And watch the memory rejections metrics:

leader_memory_pressure_rejections
follower_memory_pressure_rejections
transaction_memory_pressure_rejections

Occasional rejections due to memory pressure are fine and act as backpressure to clients. Clients will transparently retry operations. However, no operations should time out.