Reporting Kudu crashes using breakpad
Kudu uses the Google breakpad library to generate a minidump whenever Kudu experiences a crash. A minidump file contains important debugging information about the process that crashed, including shared libraries loaded and their versions, a list of threads running at the time of the crash, the state of the processor registers and a copy of the stack memory for each thread, and CPU and operating system version information. These minidumps are typically only a few MB in size and are generated even if core dump generation is disabled. Currently, generating minidumps is only possible on Linux deployments.
By default, Kudu stores its minidumps in a subdirectory of the configured glog directory
called minidumps
. This location can be customized by setting the
--minidump_path
flag. Kudu will retain only a certain number of
minidumps before deleting the older ones, in an effort to avoid filling up the disk with
minidump files. The maximum number of minidumps that will be retained can be controlled
by setting the --max_minidumps
gflag.
Minidumps contain information specific to the binary that created them and are therefore not useful without access to the exact binary that crashed, or a very similar binary.
Kudu developers can access the minidump tools in their development environment because
they are installed as part of the Kudu thirdparty build. They can be found in the Kudu
development environment under uninstrumented/bin
. For example,
thirdparty/installed/uninstrumented/bin/minidump-2-core
.
If minidumps are enabled, it is possible to force Kudu to create a minidump without
killing the process. To do that, send a USR1
signal to the
kudu-tserver
or kudu-master
process. For example:
sudo pkill -USR1 kudu-tserver
Viewing the minidump stack trace with the GNU debugger
Although a minidump contains no heap information, it does contain thread and stack information. You can convert a minidump to a core file to view it with GDB.
To convert the minidump (.dmp
file) to a core file:
minidump-2-core -o 02cb4a97-ee37-6454-73a9d9cb-590c7dde.core \
02cb4a97-ee37-6454-73a9d9cb-590c7dde.dmp
To view the core file with GDB (on a parcel deployment):
gdb /opt/cloudera/parcels/KUDU/lib/kudu/sbin-release/kudu-master \
-s /opt/cloudera/parcels/KUDU/lib/debug/usr/lib/kudu/sbin-release/kudu-master.debug \
02cb4a97-ee37-6454-73a9d9cb-590c7dde.core