Kernel Modules missing on Rasberry Pi

This, almost certainly, was a mess of my own making, but as I didn't find any answers with web searches I thought it was worth documenting for anyone else who sets a similar time bomb for themselves.

I've got some Raspberry Pi's which use NFS for their root partition. They used to be PXE booted, but at some point starting failing to boot so some time back I put a SD card back in for the /boot partition.

This, I suspect, was probably my undoing.

The Pi's have been working fine since, but I wanted to install Docker onto one of them. Although it installed, Docker failed to start, logging the following

Oct 09 22:45:43 redim-4-search-pi dockerd[3534]: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to
Oct 09 22:45:43 redim-4-search-pi dockerd[3534]: modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.19.75-v8+
Oct 09 22:45:43 redim-4-search-pi dockerd[3534]: iptables v1.6.0: can't initialize iptables table `nat': Table does not exist (do you need to insmod?)
Oct 09 22:45:43 redim-4-search-pi dockerd[3534]: Perhaps iptables or your kernel needs to be upgraded.
Oct 09 22:45:43 redim-4-search-pi dockerd[3534]:  (exit status 3)
Oct 09 22:45:43 redim-4-search-pi systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE

On examination, there is no modules directory for the kernel version I'm currently running

root@redim-4-search-pi:~# uname -r
4.19.75-v8+
root@redim-4-search-pi:~# ls /lib/modules/
4.19.66+  4.19.66-v7+

This post details the steps I took to resolve this issue

 

Understanding the issue

This issue isn't specific to Docker - it's just that it highlighted it. On startup, Docker tries to add a bunch of iptables rules to handle NAT and the like.

On that Raspberry Pi iptables can't be used at all because of the missing modules

iptables -L -v -n
modprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.75-v8+/modules.dep.bin'
modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.19.75-v8+
iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?)
Perhaps iptables or your kernel needs to be upgraded.

The underlying issue is actually more severe - we have no modules for the current kernel, so it's not just iptables that'd be affected.

 

The Cause

In my case, the issue will be the workaround I used when the Pi's started failing to PXE boot (it happened to just 1 Pi initially so I was reasonably sure it was to do with the Pi itself, others failed to reboot at later points).

I'm not sure where I got the image I used - there's a reasonable chance I downloaded a fresh image - but the end result is that the boot partition contains a kernel version that doesn't exist on the root disk

 

Resolution

The first temptation is to use apt-get to install a new kernel, but whilst it's the right idea, it's the wrong command.

Before going any further, back up your data.

We need the RPI kernel, so instead we need to run

apt-get -y install rpi-update
rpi-update 

Following a reboot, we have a new kernel version and the modules

root@redim-4-search-pi:~# uname -r
5.10.63-v7+

root@redim-4-search-pi:~# ls /lib/modules/
4.19.66+  4.19.66-v7+  5.10.63+  5.10.63-v7+  5.10.63-v7l+  5.10.63-v8+

And just for completeness, we can run iptables

root@redim-4-search-pi:~# iptables -L -v -n | head -n1
Chain INPUT (policy ACCEPT 19227 packets, 5966K bytes)

docker also started successfully during the reboot.

 

Conclusion

This really isn't something you should experience unless you've been fiddling/doing odd things. But then, the Raspberry Pi is for fiddling.

The correct answer is probably to do a reinstall so that you get back to a known good state, but I had some stuff I wanted to experiment with first.