Re: [PATCH v3 0/5] Add a bio_vec based API to core/rw.c

[Chuck Lever <cel@kernel.org>]

On 1/23/26 1:04 AM, Zhu Yanjun wrote:
> 在 2026/1/22 14:03, Chuck Lever 写道:
>> From: Chuck Lever <chuck.lever@oracle.com>
>>
>> This series introduces a bio_vec based API for RDMA read and write
>> operations in the RDMA core, eliminating unnecessary scatterlist
>> conversions for callers that already work with bvecs.
>>
>> Current users of rdma_rw_ctx_init() must convert their native data
>> structures into scatterlists. For subsystems like svcrdma that
>> maintain data in bvec format, this conversion adds overhead both in
>> CPU cycles and memory footprint. The new API accepts bvec arrays
>> directly.
>>
>> For hardware RDMA devices, the implementation uses the IOVA-based
>> DMA mapping API to reduce IOTLB synchronization overhead from O(n)
>> per-page syncs to a single O(1) sync after all mappings complete.
>> Software RDMA devices (rxe, siw) continue using virtual addressing.
>>
>> The series includes MR registration support for bvec arrays,
>> enabling iWARP devices and the force_mr debug parameter. The MR
>> path reuses existing ib_map_mr_sg() infrastructure by constructing
>> a synthetic scatterlist from the bvec DMA addresses.
> 
> Hi, Chuck Lever
> 
> I’ve read through the patch series. As I understand it, the new bio_vec–
> based RDMA read/write API allows callers that already operate on bvecs
> (for example, svcrdma and potentially NVMe-oF) to avoid converting their
> data into scatterlists, which should reduce CPU overhead and memory
> usage in the data path.
> 
> For hardware RDMA devices, the use of the IOVA-based DMA mapping API
> also seems likely to reduce IOTLB synchronization overhead compared to
> the existing per-page approach, while software devices (rxe, siw) retain
> the current virtual-addressing model.
> 
> Do you happen to have any performance or functional test results you
> could share for this series, in particular:
> 
> Hardware RDMA devices (e.g., latency, bandwidth, or CPU utilization
> changes), and/or

Functional tests with CX-5 Infiniband and NFS/RDMA show no regression.

Performance tests are difficult to evaluate because I don't have a
multi-client set-up here to drive a heavy workload, plus filesystems
bottleneck long before the network transport does. The changes are
designed to improve scalability (eg lower CPU utilization for the same
workload and less interaction between host and RNIC) more than improve
raw throughput. So far I have seen no throughput regression and perhaps
a bit of improvement for tail latencies.

The main purpose of the series, however, is part of an effort to enable
kernel-wide replacement of the use of scatter-gather lists, which are
technical debt. Socket APIs already support struct bio_vec.


> Software RDMA devices such as rxe or siw?

Software providers are not likely to see much change. However, you will
need to test the series with your own preferred configuration and
workload to assess performance and scalability delta.


-- 
Chuck Lever