Re: [RFC PATCH net-next] net/smc: Introduce receive queue flow control support

[Karsten Graul <kgraul@linux.ibm.com>]

On 29/01/2022 04:43, Guangguan Wang wrote:
> 
> On 2022/1/25 17:42, Stefan Raspl wrote:
>>
>> That's some truly substantial improvements!
>> But we need to be careful with protocol-level changes: There are other operating systems like z/OS and AIX which have compatible implementations of SMC, too. Changes like a reduction of connections per link group or usage of reserved fields would need to be coordinated, and likely would have unwanted side-effects even when used with older Linux kernel versions.
>> Changing the protocol is "expensive" insofar as it requires time to thoroughly discuss the changes, perform compatibility tests, and so on.
>> So I would like to urge you to investigate alternative ways that do not require protocol-level changes to address this scenario, e.g. by modifying the number of completion queue elements, to see if this could yield similar results.
>>
>> Thx!
>>
> 
> Yes, there are alternative ways, as RNR caused by the missmatch of send rate and receive rate, which means sending too fast
> or receiving too slow. What I have done in this patch is to backpressure the sending side when sending too fast.
> 
> Another solution is to process and refill the receive queue as quickly as posibble, which requires no protocol-level change. 
> The fllowing modifications are needed:
> - Enqueue cdc msgs to backlog queues instead of processing in rx tasklet. llc msgs remain unchanged.
> - A mempool is needed as cdc msgs are processed asynchronously. Allocate new receive buffers from mempool when refill receive queue.
> - Schedule backlog queues to other cpus, which are calculated by 4-tuple or 5-tuple hash of the connections, to process the cdc msgs,
>   in order to reduce the usage of the cpu where rx tasklet runs on.
> 
> the pseudocode shows below:
> rx_tasklet
>     if cdc_msgs
>         enqueue to backlog;
> 	maybe smp_call_function_single_async is needed to wakeup the corresponding cpu to process backlog;
>         allocate new buffer and modify the sge in rq_wr;
>     else
>         process remains unchanged;
>     endif
> 
>     post_recv rq_wr;
> end rx_tasklet
> 
> smp_backlog_process in corresponding cpu, called by smp_call_function_single_async
>     for connections hashed to this cpu
>         for cdc_msgs in backlog
>             process cdc msgs;
>         end cdc_msgs
>     end connections
> end smp_backlog_process
> 
> I‘d like to hear your suggestions of this solution.
> Thank you.

I like this idea, this should improve the RX handling a lot!