Re: Compressed files & the page cache

Qu Wenruo
2025-07-16T00:57:29Z

在 2025/7/16 06:10, Matthew Wilcox 写道:
> I've started looking at how the page cache can help filesystems handle
> compressed data better. Feedback would be appreciated! I'll probably
> say a few things which are obvious to anyone who knows how compressed
> files work, but I'm trying to be explicit about my assumptions.
>
> First, I believe that all filesystems work by compressing fixed-size
> plaintext into variable-sized compressed blocks. This would be a good
> point to stop reading and tell me about counterexamples.

I don't think it's the case for btrfs, unless your "fixed-size" means
block size, and in that case, a single block won't be compressed at all...

In btrfs, we support compressing the plaintext from 2 blocks to 128KiB
(the 128KiB limit is an artificial one).

>
> From what I've been reading in all your filesystems is that you want to
> allocate extra pages in the page cache in order to store the excess data
> retrieved along with the page that you're actually trying to read. That's
> because compressing in larger chunks leads to better compression.

We don't. We just grab dirty pages up to 128KiB, and we can handle
smaller ranges, as small as two blocks.

>
> There's some discrepancy between filesystems whether you need scratch
> space for decompression. Some filesystems read the compressed data into
> the pagecache and decompress in-place, while other filesystems read the
> compressed data into scratch pages and decompress into the page cache.

Btrfs goes the scratch pages way. Decompression in-place looks a little
tricky to me. E.g. what if there is only one compressed page, and it
decompressed to 4 pages.

Won't the plaintext over-write the compressed data halfway?

>
> There also seems to be some discrepancy between filesystems whether the
> decompression involves vmap() of all the memory allocated or whether the
> decompression routines can handle doing kmap_local() on individual pages.

Btrfs is the later case.

All the decompression/compression routines all support swapping
input/output buffer when one of them is full.
So kmap_local() is completely feasible.

Thanks,
Qu

>
> So, my proposal is that filesystems tell the page cache that their minimum
> folio size is the compression block size. That seems to be around 64k,
> so not an unreasonable minimum allocation size. That removes all the
> extra code in filesystems to allocate extra memory in the page cache.
> It means we don't attempt to track dirtiness at a sub-folio granularity
> (there's no point, we have to write back the entire compressed bock
> at once). We also get a single virtually contiguous block ... if you're
> willing to ditch HIGHMEM support. Or there's a proposal to introduce a
> vmap_file() which would give us a virtually contiguous chunk of memory
> (and could be trivially turned into a noop for the case of trying to
> vmap a single large folio).