For advanced debugging of memory usage, administrators may enable heap sampling on Kudu daemons. This allows Kudu developers to associate memory usage with the specific lines of code and data structures responsible. When reporting a bug related to memory usage or an apparent memory leak, heap profiling can give quantitative data to pinpoint the issue.
To enable heap sampling on a Kudu daemon, pass the flag
--heap-sample-every-n-bytes=524588. If heap sampling is enabled, the
current sampled heap occupancy can be retrieved over HTTP by visiting
http://master.example.com:8051/pprof/heap. The output is a
machine-readable dump of the stack traces with their associated heap usage.
Rather than visiting the heap profile page directly in a web browser, it is typically
more useful to use the
pprof tool that is distributed as part of the
gperftools open source project. For example, a developer with a local
build tree can use the following command to collect the sampled heap usage and output
an SVG diagram:
thirdparty/installed/uninstrumented/bin/pprof -svg 'http://localhost:8051/pprof/heap' > /tmp/heap.svg
The resulting SVG may be visualized in a web browser or sent to the Kudu community to help troubleshoot memory occupancy issues.