Re: Sequential read(8K) from compressed files are very slow

[Christoph Hellwig <hch@infradead.org>]

On Tue, Jun 03, 2025 at 06:36:11PM -0700, Boris Burkov wrote:
> However, I do observe the huge delta between bs=8k and bs=128k for
> compressed which is interesting, even if I am doing something dumb and
> failing to reproduce the fast uncompressed reads.
> 
> I also observe that the performance rapidly drops off below bs=32k.
> Using the highly compressible file, I get 1.4GB/s with 128k, 64k, 32k
> and then 200-400MB/s for 4k-16k.
> 
> IN THEORY, add_ra_bio_pages is supposed to be doing our own readahead
> caching for compressed pages that we have read, so I think any overhead
> we incur is not going to be making tons more IO. It will probably be in
> that readahead caching or in some interaction with VFS readahead.

> However, I do see that in the 8k case, we are repeatedly calling
> btrfs_readahead() while in the 128k case, we only call btrfs_readahead
> ~2500 times, and the rest of the time we loop inside btrfs_readahead
> calling btrfs_do_readpage.

Btw, I found the way add_ra_bio_pages in btrfs always a little
odd.  The core readahead code provides a readahead_expand() that should
do something similar, but more efficiently.  The difference is that it
only works for actual readahead calls and not ->read_folio, but the
latter is pretty much a last resort these days.

> I removed all the extent locking as an experiment, as it is not really
> needed for safety in this single threaded test and did see an
> improvement but not full parity between 8k and 128k for the compressed
> file. I'll keep poking at the other sources of overhead in the builtin
> readahead logic and in calling btrfs_readahead more looping inside it.
> 
> I'll keep trying that to see if I can get a full reproduction and try to
> actually explain the difference.
> 
> If you are able to use some kind of tracing to see if these findings
> hold on your system, I think that would be interesting.
> 
> Also, if someone more knowledgeable in how the generic readahead works
> like Christoph could give me a hint to how to hack up the 8k case to
> make fewer calls to btrfs_readahead, I think that could help bolster the
> theory.

I'm not really a readahead expert, but I added who I suspect is (willy).
But my guess is using readahead_expand is the right answer here, but
another thing to look at might if the compressed extent handling in
btrfs somehow messes up the read ahead window calculations.