Re: [PATCH v3 0/5] Add a bio_vec based API to core/rw.c

[Zhu Yanjun <yanjun.zhu@linux.dev>]

在 2026/1/23 6:13, Chuck Lever 写道:
> On 1/23/26 1:04 AM, Zhu Yanjun wrote:
>> 在 2026/1/22 14:03, Chuck Lever 写道:
>>> From: Chuck Lever <chuck.lever@oracle.com>
>>>
>>> This series introduces a bio_vec based API for RDMA read and write
>>> operations in the RDMA core, eliminating unnecessary scatterlist
>>> conversions for callers that already work with bvecs.
>>>
>>> Current users of rdma_rw_ctx_init() must convert their native data
>>> structures into scatterlists. For subsystems like svcrdma that
>>> maintain data in bvec format, this conversion adds overhead both in
>>> CPU cycles and memory footprint. The new API accepts bvec arrays
>>> directly.
>>>
>>> For hardware RDMA devices, the implementation uses the IOVA-based
>>> DMA mapping API to reduce IOTLB synchronization overhead from O(n)
>>> per-page syncs to a single O(1) sync after all mappings complete.
>>> Software RDMA devices (rxe, siw) continue using virtual addressing.
>>>
>>> The series includes MR registration support for bvec arrays,
>>> enabling iWARP devices and the force_mr debug parameter. The MR
>>> path reuses existing ib_map_mr_sg() infrastructure by constructing
>>> a synthetic scatterlist from the bvec DMA addresses.
>>
>> Hi, Chuck Lever
>>
>> I’ve read through the patch series. As I understand it, the new bio_vec–
>> based RDMA read/write API allows callers that already operate on bvecs
>> (for example, svcrdma and potentially NVMe-oF) to avoid converting their
>> data into scatterlists, which should reduce CPU overhead and memory
>> usage in the data path.
>>
>> For hardware RDMA devices, the use of the IOVA-based DMA mapping API
>> also seems likely to reduce IOTLB synchronization overhead compared to
>> the existing per-page approach, while software devices (rxe, siw) retain
>> the current virtual-addressing model.
>>
>> Do you happen to have any performance or functional test results you
>> could share for this series, in particular:
>>
>> Hardware RDMA devices (e.g., latency, bandwidth, or CPU utilization
>> changes), and/or
> 
> Functional tests with CX-5 Infiniband and NFS/RDMA show no regression.
> 
> Performance tests are difficult to evaluate because I don't have a
> multi-client set-up here to drive a heavy workload, plus filesystems
> bottleneck long before the network transport does. The changes are
> designed to improve scalability (eg lower CPU utilization for the same
> workload and less interaction between host and RNIC) more than improve
> raw throughput. So far I have seen no throughput regression and perhaps
> a bit of improvement for tail latencies.

Thanks a lot. Based on the code changes, this patch series should 
improve performance. Unfortunately, due to various limitations, we are 
unable to provide performance test results.

Best Regards,
Zhu Yanjun

> 
> The main purpose of the series, however, is part of an effort to enable
> kernel-wide replacement of the use of scatter-gather lists, which are
> technical debt. Socket APIs already support struct bio_vec.
> 
> 
>> Software RDMA devices such as rxe or siw?
> 
> Software providers are not likely to see much change. However, you will
> need to test the series with your own preferred configuration and
> workload to assess performance and scalability delta.
> 
>