This, almost certainly, was a mess of my own making, but as I didn't find any answers with web searches I thought it was worth documenting for anyone else who sets a similar time bomb for themselves.
I've got some Raspberry Pi's which use NFS for their root partition. They used to be PXE booted, but at some point starting failing to boot so some time back I put a SD card back in for the
This, I suspect, was probably my undoing.
The Pi's have been working fine since, but I wanted to install Docker onto one of them. Although it installed, Docker failed to start, logging the following
Oct 09 22:45:43 redim-4-search-pi dockerd: failed to start daemon: Error initializing network controller: error obtaining controller instance: failed to Oct 09 22:45:43 redim-4-search-pi dockerd: modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.19.75-v8+ Oct 09 22:45:43 redim-4-search-pi dockerd: iptables v1.6.0: can't initialize iptables table `nat': Table does not exist (do you need to insmod?) Oct 09 22:45:43 redim-4-search-pi dockerd: Perhaps iptables or your kernel needs to be upgraded. Oct 09 22:45:43 redim-4-search-pi dockerd: (exit status 3) Oct 09 22:45:43 redim-4-search-pi systemd: docker.service: Main process exited, code=exited, status=1/FAILURE
On examination, there is no modules directory for the kernel version I'm currently running
root@redim-4-search-pi:~# uname -r 4.19.75-v8+ root@redim-4-search-pi:~# ls /lib/modules/ 4.19.66+ 4.19.66-v7+
This post details the steps I took to resolve this issue
Understanding the issue
This issue isn't specific to Docker - it's just that it highlighted it. On startup, Docker tries to add a bunch of
iptables rules to handle NAT and the like.
On that Raspberry Pi iptables can't be used at all because of the missing modules
iptables -L -v -nmodprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/4.19.75-v8+/modules.dep.bin' modprobe: FATAL: Module ip_tables not found in directory /lib/modules/4.19.75-v8+ iptables v1.6.0: can't initialize iptables table `filter': Table does not exist (do you need to insmod?) Perhaps iptables or your kernel needs to be upgraded.
The underlying issue is actually more severe - we have no modules for the current kernel, so it's not just
iptables that'd be affected.
In my case, the issue will be the workaround I used when the Pi's started failing to PXE boot (it happened to just 1 Pi initially so I was reasonably sure it was to do with the Pi itself, others failed to reboot at later points).
I'm not sure where I got the image I used - there's a reasonable chance I downloaded a fresh image - but the end result is that the boot partition contains a kernel version that doesn't exist on the root disk
The first temptation is to use
apt-get to install a new kernel, but whilst it's the right idea, it's the wrong command.
Before going any further, back up your data.
We need the RPI kernel, so instead we need to run
apt-get -y install rpi-update rpi-update
Following a reboot, we have a new kernel version and the modules
root@redim-4-search-pi:~# uname -r 5.10.63-v7+ root@redim-4-search-pi:~# ls /lib/modules/ 4.19.66+ 4.19.66-v7+ 5.10.63+ 5.10.63-v7+ 5.10.63-v7l+ 5.10.63-v8+
And just for completeness, we can run
root@redim-4-search-pi:~# iptables -L -v -n | head -n1 Chain INPUT (policy ACCEPT 19227 packets, 5966K bytes)
docker also started successfully during the reboot.
This really isn't something you should experience unless you've been fiddling/doing odd things. But then, the Raspberry Pi is for fiddling.
The correct answer is probably to do a reinstall so that you get back to a known good state, but I had some stuff I wanted to experiment with first.