Re: [RFC PATCH net-next] net/smc: Introduce receive queue flow control support

Karsten Graul
2022-01-31T12:56:09Z

On 29/01/2022 04:43, Guangguan Wang wrote:
>
> On 2022/1/25 17:42, Stefan Raspl wrote:
>>
>> That's some truly substantial improvements!
>> But we need to be careful with protocol-level changes: There are other operating systems like z/OS and AIX which have compatible implementations of SMC, too. Changes like a reduction of connections per link group or usage of reserved fields would need to be coordinated, and likely would have unwanted side-effects even when used with older Linux kernel versions.
>> Changing the protocol is "expensive" insofar as it requires time to thoroughly discuss the changes, perform compatibility tests, and so on.
>> So I would like to urge you to investigate alternative ways that do not require protocol-level changes to address this scenario, e.g. by modifying the number of completion queue elements, to see if this could yield similar results.
>>
>> Thx!
>>
>
> Yes, there are alternative ways, as RNR caused by the missmatch of send rate and receive rate, which means sending too fast
> or receiving too slow. What I have done in this patch is to backpressure the sending side when sending too fast.
>
> Another solution is to process and refill the receive queue as quickly as posibble, which requires no protocol-level change.
> The fllowing modifications are needed:
> - Enqueue cdc msgs to backlog queues instead of processing in rx tasklet. llc msgs remain unchanged.
> - A mempool is needed as cdc msgs are processed asynchronously. Allocate new receive buffers from mempool when refill receive queue.
> - Schedule backlog queues to other cpus, which are calculated by 4-tuple or 5-tuple hash of the connections, to process the cdc msgs,
> in order to reduce the usage of the cpu where rx tasklet runs on.
>
> the pseudocode shows below:
> rx_tasklet
> if cdc_msgs
> enqueue to backlog;
> maybe smp_call_function_single_async is needed to wakeup the corresponding cpu to process backlog;
> allocate new buffer and modify the sge in rq_wr;
> else
> process remains unchanged;
> endif
>
> post_recv rq_wr;
> end rx_tasklet
>
> smp_backlog_process in corresponding cpu, called by smp_call_function_single_async
> for connections hashed to this cpu
> for cdc_msgs in backlog
> process cdc msgs;
> end cdc_msgs
> end connections
> end smp_backlog_process
>
> I‘d like to hear your suggestions of this solution.
> Thank you.

I like this idea, this should improve the RX handling a lot!