Re: deadlock when swapping to encrypted swapfile

[Robert Beckett <bob.beckett@collabora.com>]

 ---- On Tue, 09 Sep 2025 15:37:09 +0100  Mikulas Patocka <mpatocka@redhat.com> wrote --- 
 > 
 > 
 > On Mon, 8 Sep 2025, Robert Beckett wrote:
 > 
 > > Hi,
 > > 
 > > While testing resiliency of encrypted swap using dmcrypt we encounter easily reproducible deadlocks.
 > > The setup is a simple 1GB encrypted swap file [1] with a little mem chewer program [2] to consume all ram.
 > > 
 > > [1] Swap file setup
 > > ```
 > > $ swapoff /home/swapfile
 > > $ echo 'swap /home/swapfile /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256' >> /etc/crypttab
 > > $ systemctl daemon-reload
 > > $ systemctl start systemd-cryptsetup@swap.service
 > > $ swapon /dev/mapper/swap
 > > ```
 > 
 > I have tried to swap on encrypted block device and it worked for me.
 > 
 > I've just realized that you are swapping to a file with the loopback 
 > driver on the top of it and with the dm-crypt device on the top of the 
 > loopback device.
 > 
 > This can't work in principle - the problem is that the filesystem needs to 
 > allocate memory when you write to it, so it deadlocks when the machine 
 > runs out of memory and needs to write back some pages. There is no easy 
 > fix - fixing this would require major rewrite of the VFS layer.
 > 
 > When you swap to a file directly, the kernel bypasses the filesystem, so 
 > it should work - but when you put encryption on the top of a file, there 
 > is no way how to bypass the filesystem.
 > 
 > So, I suggest to create a partition or a logical volume for swap and put 
 > dm-crypt on the top of it.
 > 
 > Mikulas
 > 
 > 

Yeah, unfortunately we are currently restricted to using a swapfile due to many units already shipped with that.
We have longer term plans to dynamically allocate the swapfiles as neded based on a new query for estimated size
required for hibernation etc. Moving to swap partition is just not viable currently.

I tried halving /sys/module/dm_mod/parameters/swap_bios but it didn't help, which based on your more recent
reply is not unexpected.

I have a work around for now, which is to run a userland earlyoom daemon. That seems to get in and oomkill in time.
I guess another option would be to have the swapfile in a luks encrypted partition, but that equally is not viable for
steamdeck currently.

However, I'm still interested in the longer term solution of fixing the kernel so that it can handle scenarios
like this no matter how ill advised they may be. Telling users not to do something seems like a bad solution :)

Do you have any ideas about the unreliable kernel oomkiller stepping in? I definitely fill ram and swap, seems like
it should be firing.

Thanks

Bob