Troubleshooting performance issues
This topic helps you to troubleshoot issues and improve performance using Kudu
tracing, memory limits, block size cache, heap sampling, and name service cache daemon
(nscd
).
Kudu tracing The Kudu master and tablet server daemons include built-in support for tracing based on the open source Chromium Tracing framework. You can use tracing to diagnose latency issues or other problems on Kudu servers. Memory limits Kudu has a hard and soft memory limit. The hard memory limit is the maximum amount a Kudu process is allowed to use, and is controlled by the --memory_limit_hard_bytes
flag. The soft memory limit is a percentage of the hard memory limit, controlled by the flag memory_limit_soft_percentage
and with a default value of 80%, that determines the amount of memory a process may use before it will start rejecting some write operations.Block cache size Kudu uses an LRU cache for recently read data. On workloads that scan a subset of the data repeatedly, raising the size of this cache can offer significant performance benefits. To increase the amount of memory dedicated to the block cache, increase the value of the --block_cache_capacity_mb
flag. The default is 512 MiB.Heap sampling For advanced debugging of memory usage, administrators may enable heap sampling on Kudu daemons. This allows Kudu developers to associate memory usage with the specific lines of code and data structures responsible. When reporting a bug related to memory usage or an apparent memory leak, heap profiling can give quantitative data to pinpoint the issue.Slow name resolution and nscd For better scalability on nodes hosting many replicas, we recommend that you use nscd
(name service cache daemon) to cache both DNS name resolution and static name resolution (via /etc/hosts
).