Re: [Patch v7 0/2] io_uring/rsrc: coalescing multi-hugepage registered buffers

[Pavel Begunkov <asml.silence@gmail.com>]

On 7/23/24 06:56, Chenliang Li wrote:
> Registered buffers are stored and processed in the form of bvec array,
> each bvec element typically points to a PAGE_SIZE page but can also work
> with hugepages. Specifically, a buffer consisting of a hugepage is
> coalesced to use only one hugepage bvec entry during registration.
> This coalescing feature helps to save both the space and DMA-mapping time.
> 
> However, currently the coalescing feature doesn't work for multi-hugepage
> buffers. For a buffer with several 2M hugepages, we still split it into
> thousands of 4K page bvec entries while in fact, we can just use a
> handful of hugepage bvecs.
> 
> This patch series enables coalescing registered buffers with more than
> one hugepages. It optimizes the DMA-mapping time and saves memory for
> these kind of buffers.

Doesn't apply, please rebase on top of io_uring-6.11.

iter->nr_segs = bvec->bv_len;

seems your tree is missing the fix you sent before


> 
> Testing:
> 
> The hugepage fixed buffer I/O can be tested using fio without
> modification. The fio command used in the following test is given
> in [1]. There's also a liburing testcase in [2]. Also, the system
> should have enough hugepages available before testing.
> 
> Perf diff of 8M(4 * 2M hugepages) fio randread test:
> 
> Before          After           Symbol
> .....................................................
> 5.88%                           [k] __blk_rq_map_sg
> 3.98%           -3.95%          [k] dma_direct_map_sg
> 2.47%                           [k] dma_pool_alloc
> 1.37%           -1.36%          [k] sg_next
>                  +0.28%          [k] dma_map_page_attrs
> 
> Perf diff of 8M fio randwrite test:
> 
> Before          After           Symbol
> ......................................................
> 2.80%                           [k] __blk_rq_map_sg
> 1.74%                           [k] dma_direct_map_sg
> 1.61%                           [k] dma_pool_alloc
> 0.67%                           [k] sg_next
>                  +0.04%          [k] dma_map_page_attrs
> 
> The first patch prepares for adding the multi-hugepage coalescing
> by storing folio_shift and folio_mask into imu, the 2nd patch
> enables the feature.
> 
> ---
> 
> Changes since v5:
> - Reshuffle the patchset to avoid unused funtion warnings.
> - Store head page of the folio and use folio offset to get rid of
>    branching on bvec setups.
> - Add restrictions for non border-aligned folios.
> - Remove unnecessary folio_size field in io_imu_folio_data struct.
> 
> v5 : https://lore.kernel.org/io-uring/20240708021426.2217-1-cliang01.li@samsung.com/T/#t
> 
> [1]
> fio -iodepth=64 -rw=randread -direct=1 -ioengine=io_uring \
> -bs=8M -numjobs=1 -group_reporting -mem=shmhuge -fixedbufs -hugepage-size=2M \
> -filename=/dev/nvme0n1 -runtime=10s -name=test1
> 
> [2]
> https://lore.kernel.org/io-uring/20240514051343.582556-1-cliang01.li@samsung.com/T/#u
> 
> Chenliang Li (2):
>    io_uring/rsrc: store folio shift and mask into imu
>    io_uring/rsrc: enable multi-hugepage buffer coalescing
> 
>   io_uring/rsrc.c | 149 +++++++++++++++++++++++++++++++++++-------------
>   io_uring/rsrc.h |  10 ++++
>   2 files changed, 118 insertions(+), 41 deletions(-)
> 
> 
> base-commit: ad00e629145b2b9f0d78aa46e204a9df7d628978

-- 
Pavel Begunkov