Re: deadlock when swapping to encrypted swapfile

[Mikulas Patocka <mpatocka@redhat.com>]

On Mon, 8 Sep 2025, Robert Beckett wrote:

> Hi,
> 
> While testing resiliency of encrypted swap using dmcrypt we encounter easily reproducible deadlocks.
> The setup is a simple 1GB encrypted swap file [1] with a little mem chewer program [2] to consume all ram.
> 
> [1] Swap file setup
> ```
> $ swapoff /home/swapfile
> $ echo 'swap /home/swapfile /dev/urandom swap,cipher=aes-cbc-essiv:sha256,size=256' >> /etc/crypttab
> $ systemctl daemon-reload
> $ systemctl start systemd-cryptsetup@swap.service
> $ swapon /dev/mapper/swap
> ```

I have tried to swap on encrypted block device and it worked for me.

I've just realized that you are swapping to a file with the loopback 
driver on the top of it and with the dm-crypt device on the top of the 
loopback device.

This can't work in principle - the problem is that the filesystem needs to 
allocate memory when you write to it, so it deadlocks when the machine 
runs out of memory and needs to write back some pages. There is no easy 
fix - fixing this would require major rewrite of the VFS layer.

When you swap to a file directly, the kernel bypasses the filesystem, so 
it should work - but when you put encryption on the top of a file, there 
is no way how to bypass the filesystem.

So, I suggest to create a partition or a logical volume for swap and put 
dm-crypt on the top of it.

Mikulas