Installing and Configuring KDump on Debian Jessie
Having kdump enabled on a server provides a number of benefits, not least that in the event of a kernel panic you can collect a core-dump to help investigations into the root cause. It may simply be bad luck, but my experience with Debian Jessie has been that JournalD is absolutely hopeless in the event of a kernel panic.
Pre SystemD we used to (sometimes) get a backtrace written out to a log, even a partial backtrace could help point investigations into a rough direction, but even with JournalD configured to pass through to rsyslogd those traces just don't seem to be appearing (which to be fair, might be because of the nature of the panic rather than the fault of journald).
This documentation details the steps required to install and configure KDump on Debian Jessie
Installation
Installation is straight-forward, everything we need is in the repos
apt-get install kdump-tools crash kexec-tools makedumpfile `uname -r`-dbg
We've just installed kdump and the tools needed to process coredumps. The call to uname is used so that we can obtain an unstripped version of the currently installed kernel for use with crash
Configuration
KDump on Debian is a little odd - on other distro's, once installed it's automatically enabled (aside from one change to the kernel commandline). On Debian, it's configured to be disabled, so we need to enable it
nano /etc/default/kdump-tools
# Set USE_KDUMP=1
# Save and exit
Next, we need to amend the Kernel commandline to specify how much RAM to reserve for the crash kernel
nano /etc/default/grub
# Find the line beginning GRUB_CMDLINE_LINUX_DEFAULT and append crashkernel=128M. e.g.
GRUB_CMDLINE_LINUX_DEFAULT="quiet crashkernel=128M"
# Save and exit
update-grub
The next step is to reboot the system so that kdump can load
reboot
Test Panic
There's little sense in having to wait for a natural panic to occur to see whether our configuration worked, so we're going to force one
echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger
You should see the kernel panic occur, then shortly after you should see kdump take control and start writing a coredump out (by default to /var/crash). Once the coredump is written, the system will reboot itself.
Reading the Coredump
Once the system has rebooted, we can test reading the coredump with crash. If you look in /var/crash you should see a datestamped directory.
cd /var/crash/[directory name]
for i in *.dump
do
crash /usr/lib/debug/vmlinux-`uname -r` $i
done
Once crash has loaded, type bt to get a backtrace, dmesg to view the kernel ring buffer etc - most of crash's functionality should be entirely familiar to anyone used to using gdb