Installing and Configuring KDump on Debian Jessie

Having kdump enabled on a server provides a number of benefits, not least that in the event of a kernel panic you can collect a core-dump to help investigations into the root cause. It may simply be bad luck, but my experience with Debian Jessie has been that JournalD is absolutely hopeless in the event of a kernel panic.

Pre SystemD we used to (sometimes) get a backtrace written out to a log, even a partial backtrace could help point investigations into a rough direction, but even with JournalD configured to pass through to rsyslogd those traces just don't seem to be appearing (which to be fair, might be because of the nature of the panic rather than the fault of journald).

This documentation details the steps required to install and configure KDump on Debian Jessie

 

 

Installation

Installation is straight-forward, everything we need is in the repos

apt-get install kdump-tools crash kexec-tools makedumpfile `uname -r`-dbg

We've just installed kdump and the tools needed to process coredumps. The call to uname is used so that we can obtain an unstripped version of the currently installed kernel for use with crash

 

Configuration

KDump on Debian is a little odd - on other distro's, once installed it's automatically enabled (aside from one change to the kernel commandline). On Debian, it's configured to be disabled, so we need to enable it

nano /etc/default/kdump-tools

# Set USE_KDUMP=1
# Save and exit

Next, we need to amend the Kernel commandline to specify how much RAM to reserve for the crash kernel

nano /etc/default/grub
# Find the line beginning GRUB_CMDLINE_LINUX_DEFAULT and append crashkernel=128M. e.g.
GRUB_CMDLINE_LINUX_DEFAULT="quiet crashkernel=128M"

# Save and exit
update-grub

The next step is to reboot the system so that kdump can load

reboot

 

Test Panic

There's little sense in having to wait for a natural panic to occur to see whether our configuration worked, so we're going to force one

echo 1 > /proc/sys/kernel/sysrq
echo c > /proc/sysrq-trigger

You should see the kernel panic occur, then shortly after you should see kdump take control and start writing a coredump out (by default to /var/crash). Once the coredump is written, the system will reboot itself.

 

Reading the Coredump

Once the system has rebooted, we can test reading the coredump with crash. If you look in /var/crash you should see a datestamped directory.

cd /var/crash/[directory name]
for i in *.dump
do
   crash /usr/lib/debug/vmlinux-`uname -r` $i
done

Once crash has loaded, type bt to get a backtrace, dmesg to view the kernel ring buffer etc - most of crash's functionality should be entirely familiar to anyone used to using gdb