Re: Deleting large amounts of data causes system freeze due to OOM.

[fdavidl073rnovn@tutanota.com]

Sep 27, 2023, 04:53 by quwenruo.btrfs@gmx.com:

>
> The compression is the easily way to create tons of small file extents
> (the limit of a compressed extent is only 128K).
>
> Furthermore, each file extent would need an in-memory structure (struct
> extent_map, for a debug kernel, it's 122 bytes) to cache the contents.
>
> Thus for a 8TiB file with all compressed file extents at their max size
> (pretty common if it's only for backup).
> Then we still have 512M file extents.
>
> Just multiple that by 122, you can see how this go crazy.
>
> But still, if you're only deleting the file, the result shouldn't go
> this crazy, as deleting itself won't try to read the file extents thus
> no such cache.
>
> However as long as we start doing read/write, the cache can go very
> large, especially if you use compress, and only get released when the
> whole inode get released from kernel.
>
> On the other hand, if you go uncompressed data, the maximum file extent
> size is enlarged to 128M (a 1024x increase), thus a huge reduce in the
> number of extents.
>
> In the long run I guess we need some way to release the extent_map when
> low on memory.
> But for now, I'm afraid I don't have better suggestion other than
> turning off compression and defrag the compressed files using newer
> kernel (v6.2 and newer).
>
> In v6.2, there is a patch to prevent defrag from populating the extent
> map cache, thus it won't take all the memory just by defrag.
> And with all those files converted from compression, I believe the
> situation would be greatly improved.
>
> Thanks,
> Qu
>
The backup itself is gone and will need to be re-sent. If I'm understanding things properly then by mounting the btrfs device for the backup without compression and enforcing send protocol one it should be written uncompressed which will avoid the issue correct?

I was also looking at the source code and it seems relatively straight forward to change BTRFS_MAX_COMPRESSED and BTRFS_MAX_UNCOMPRESSED to SZ_128M or somewhere in between like SZ_8M. Do you have any thoughts on how well that might work?

Do you have any idea on how complicated the long term fix is or when it might added? v6.8 maybe?

Thank you for your prompt responses. Sending the backup again will take some days but I will email you to tell you if disabling compression fixes the issue.

Sincerely,
David