Re: [PATCH net-next] net/smc: introduce autosplit for smc

[Guangguan Wang <guangguan.wang@linux.alibaba.com>]

On 2024/7/11 23:57, Wenjia Zhang wrote:
> 
> 
> On 09.07.24 18:05, Guangguan Wang wrote:
>> When sending large size data in TCP, the data will be split into
>> several segments(packets) to transfer due to MTU config. And in
>> the receive side, application can be woken up to recv data every
>> packet arrived, the data transmission and data recv copy are
>> pipelined.
>>
>> But for SMC-R, it will transmit as many data as possible in one
>> RDMA WRITE and a CDC msg follows the RDMA WRITE, in the receive
>> size, the application only be woken up to recv data when all RDMA
>> WRITE data and the followed CDC msg arrived. The data transmission
>> and data recv copy are sequential.
>>
>> This patch introduce autosplit for SMC, which can automatic split
>> data into several segments and every segment transmitted by one RDMA
>> WRITE when sending large size data in SMC. Because of the split, the
>> data transmission and data send copy can be pipelined in the send side,
>> and the data transmission and data recv copy can be pipelined in the
>> receive side. Thus autosplit helps improving latency performance when
>> sending large size data. The autosplit also works for SMC-D.
>>
>> This patch also introduce a sysctl names autosplit_size for configure
>> the max size of the split segment, whose default value is 128KiB
>> (128KiB perform best in my environment).
>>
>> The sockperf benchmark shows 17%-28% latency improvement when msgsize
>>> = 256KB for SMC-R, 15%-32% latency improvement when msgsize >= 256KB
>> for SMC-D with smc-loopback.
>>
>> Test command:
>> sockperf sr --tcp -m 1048575
>> sockperf pp --tcp -i <server ip> -m <msgsize> -t 20
>>
>> Test config:
>> sysctl -w net.smc.wmem=524288
>> sysctl -w net.smc.rmem=524288
>>
>> Test results:
>> SMC-R
>> msgsize   noautosplit    autosplit
>> 128KB       55.546 us     55.763 us
>> 256KB       83.537 us     69.743 us (17% improve)
>> 512KB      138.306 us    100.313 us (28% improve)
>> 1MB        273.702 us    197.222 us (28% improve)
>>
>> SMC-D with smc-loopback
>> msgsize   noautosplit    autosplit
>> 128KB       14.672 us     14.690 us
>> 256KB       28.277 us     23.958 us (15% improve)
>> 512KB       63.047 us     45.339 us (28% improve)
>> 1MB        129.306 us     87.278 us (32% improve)
>>
>> Signed-off-by: Guangguan Wang <guangguan.wang@linux.alibaba.com>
>> ---
>>   Documentation/networking/smc-sysctl.rst | 11 +++++++++++
>>   include/net/netns/smc.h                 |  1 +
>>   net/smc/smc_sysctl.c                    | 12 ++++++++++++
>>   net/smc/smc_tx.c                        | 19 ++++++++++++++++++-
>>   4 files changed, 42 insertions(+), 1 deletion(-)
>>
> 
> Hi Guangguan,
> 
> If I remember correctly, the intention to use one RDMA-write for a possible large data is to reduce possible many partial stores. Since many year has gone, I'm not that sure if it would still be an issue. I need some time to check on it.
> 

Did you mean too many partial stores will result in some issue? What's the issue?


> BTW, I don't really like the idea to use sysctl to set the autosplit_size in any value at will. That makes no sense to improve the performance.

Although 128KB autosplit_size have a good performance in most scenario, I still found some better autosplit_size for some specific network configurations.
For example, 128KB autosplit_size have a good performance whether the MTU is 1500 or 8500, but for 8500 MTU, 64KB autosplit_size performs better.

Maybe the sysctl is not the best way, but I think it should have a way to set the value of autosplit_size for possible performance tuning.

Thanks,
Guangguan Wang

> 
> Thanks,
> Wenjia