Using multiple swap partitions in a specific order on Linux

It's possible, on Linux, to have multiple swap partitions and/or swapfiles so that your swapspace is spread across multiple physical devices.

It's also sometimes desirable to set an order of priority for these, so that paging uses the fastest underlying storage first.

This documentation details how to set up a mix of swap files and partitions and tell the kernel how to prioritise it's swapsources Linux.

My use-case

Wanting multiple swap sources sounds like quite an obscure thing to require in the modern age, so I thought it'd be useful to quickly detail the use-case that led me to this.

I have a Lenovo Thinkpad X1 Carbon, it's a lovely bit of hardware.

Unfortunately, some poor decisions have collided: Lenovo decided to solder the RAM in, and at purchase time the laptop was under-specced with 16GB RAM (can always upgrade it right? not like it's Apple who solder stuff in. Oh wait...).

However, the SSD is blazing fast. In the old days, you knew you were swapping because your system started being sluggish - that's just not the case here.

The inevitable result is that I tend to first find out that I'm in swap when I've exhausted it (and available RAM). Waiting for OOMKiller to return (most of) your system to you isn't the smoothest of workflows. Because there's only 16GB of RAM, it happens all too frequently.

In an ideal world, you'd upgrade the RAM, but ... yeah, thanks Lenovo...

A secondary solution would be to expand the swapfile. But, all that's really doing is moving the goal-posts, I'd still only find out I've exhausted things when the system crunches to a halt.

What I wanted to implement, instead, was a secondary, slower swap source using a flash drive.

Initial swaps get the benefit of the NVMe (giving almost seamless performance) but as we move further in, we should switch to another slower source so that I get some of the traditional swap lag as a warning that things are about to go wrong.

By default, the Linux kernel will load-balance across multiple swap sources, which isn't what I want. I need the system to exhaust the pool of fast swap storage before moving onto the slower safety net.

Creating Swap

In this example, we'll set up a swapfile and a partition - there's no requirement to mix types, you can use just files or partitions.

Check what swap you currently have

# cat /proc/swaps 
Filename                Type        Size        Used        Priority
/swapfile                               file        2096636 13056       -2

Become root

sudo -i

Disable the swap

swapoff /swapfile

Let's expand our pagefile to 4GB whilst here

fallocate -l 4G /swapfile
mkswap /swapfile

Next we'll create a swap partition on our USB stick

lsblk # get the device/partition name, mine is sdd1
mkswap /dev/sdd1

The output of mkswap will provide a UUID

mkswap: /dev/sda1: warning: wiping old ext4 signature.
Setting up swapspace version 1, size = 28.7 GiB (30765199360 bytes)
no label, UUID=833085e3-7ab8-4341-8cb5-cc51d5d9f7d0

We'll use this in fstab as device naming can change between reboots.

Enabling and prioritising swap

Next, we need to edit /etc/fstab to add our swap sources, and specify the priority in which they should be used

vi /etc/fstab

/swapfile                                 none            swap    sw,pri=2              0       0
UUID=833085e3-7ab8-4341-8cb5-cc51d5d9f7d0   none    swap    sw,nofail,pri=1 0   0

The highest priority wins, so in this example, /swapfile will be used before the flash drive.

We've include nofail for the flash drive so that if it isn't present (or has failed etc) the system won't abort booting.

We then just need to tell the system to put both into use

swapon -a

We should now be able to see both swap sources in use, as well as their priority

# cat /proc/swaps 
Filename                Type        Size        Used        Priority
/swapfile                               file        4194300     0       2
/dev/sda1                               partition   30044140    0       1

The system should also automatically bring both online at boot.

Conclusion

It's incredibly simple to specify the order in which swap sources should be used - in fact, it's harder to search for the information than it is to put it in place: the kernel will honour the ordering specified in fstab with the pri option.

Although it's not been done in this example, you can specify multiple sources with the same priority, so if you wanted to balance writes across one tier, you could do something like

/swapfile                                 none            swap    sw,pri=3              0       0
UUID=833085e3-7ab8-4341-8cb5-cc51d5d9f7d0   none    swap    sw,nofail,pri=2 0   0
UUID=abcde-f01  none    swap    sw,nofail,pri=1 0   0
UUID=01234-567  none    swap    sw,nofail,pri=1 0   0

Which would cause the kernel to balance across devices for the final tier of swap sources.