Re: [PATCH] iomap: add allocation cache for iomap_dio

[guzebing <guzebing1612@gmail.com>]

在 2025/11/21 18:22, Christoph Hellwig 写道:
> On Fri, Nov 21, 2025 at 05:00:52PM +0800, guzebing wrote:
>> From: guzebing <guzebing@bytedance.com>
>>
>> As implemented by the bio structure, we do the same thing on the
>> iomap-dio structure. Add a per-cpu cache for iomap_dio allocations,
>> enabling us to quickly recycle them instead of going through the slab
>> allocator.
>>
>> By making such changes, we can reduce memory allocation on the direct
>> IO path, so that direct IO will not block due to insufficient system
>> memory. In addition, for direct IO, the read performance of io_uring
>> is improved by about 2.6%.
> 
> Have you checked how much of that you'd get by using a dedicated
> slab cache that should also do per-cpu allocations?  Note that even
> if we had a dedicated per-cpu cache we'd probably still want that.
I’m sorry for the long delay in replying to your email due to some other 
matters. I hope you still remember this revision. First, thank you for 
your response.

Yes, I try to use a dedicated kmem cache to allocate cache for iomap-dio 
structure. However, when system memory is sufficient, kmalloc and kmem 
cache deliver identical performance.

For direct I/O reads on the ext4 file system, the test command is:

./t/io_uring -p0 -d128 -b4096 -s32 -c32 -F1 -B1 -R1 -X1 -n1 -P1 /mnt/004.txt

The measured performance is:

kmalloc: 750K IOPS
kmem cache: 750K IOPS
per-CPU cache: 770K IOPS
> 
> Also any chance you could factor this into common code?
> 
For a mempool, we first allocate with kmalloc or kmem cache and finally 
fall back to a reserved cache—this is for reliability. It’s not a great 
fit for our high‑performance scenario.

Additionally, the current need for frequent allocation/free (hundreds of 
thousands to millions of times per second) may be more suitable for the 
bio or dio structures; beyond those, I’m not sure whether similar 
scenarios exist.

If we were to extract a generic implementation solely for this, would it 
yield significant benefits? Do you have any good suggestions?

I’d appreciate your review.