Re: question about the completion tasklet in the rxe driver

[Bob Pearson <rpearsonhpe@gmail.com>]

On 3/16/23 04:40, Bernard Metzler wrote:
> 
> 
>> -----Original Message-----
>> From: Bob Pearson <rpearsonhpe@gmail.com>
>> Sent: Wednesday, 15 March 2023 22:56
>> To: Jason Gunthorpe <jgg@nvidia.com>; Zhu Yanjun <zyjzyj2000@gmail.com>;
>> linux-rdma@vger.kernel.org; Bernard Metzler <BMT@zurich.ibm.com>
>> Subject: [EXTERNAL] question about the completion tasklet in the rxe driver
>>
>> I have a goal of trying to get rid of all the tasklets in the rxe driver
>> and with the replacement of the
>> three QP tasklets by workqueues the only remaining one is the tasklet that
>> defers the CQ completion
>> handler. This has been in there since the driver went upstream so the
>> history of why it is there is lost.
>>
>> I notice that the mlx5 driver does have a deferral mechanism for the
>> completion handler but the siw driver
>> does not. I really do not see what advantage, if any, this has for the rxe
>> driver. Perhaps there is some
>> reason it shouldn't run in hard interrupt context but the CQ tasklet is a
>> soft interrupt so the completion
>> handler can't sleep anyway.
>>
>> As an experiment I removed the CQ tasklet in the rxe driver and it runs
>> fine. In fact the performance is
>> slightly better with the completion handler called inline rather than
>> deferred to another tasklet.
> 
> That is what I would suggest to do. Why would you leave receive
> processing or failing send processing w/o creating the CQE and
> kicking the CQ handler, if you are in a context with
> all information available to build a CQE, signal its availability
> to the consumer and kick a user handler if registered and armed?
> 
> Only exception I see: If you process the SQ in post_send() user context
> and a failure results in immediate CQE creation, direct CQ handler calling
> is not allowed - see Documentation/infiniband/core_locking.rst
> Not sure though, if rxe allows for direct SQ processing out of user
> context.
> 
> Cheers,
> Bernard.

And you did. I am not sure why the mlx5 driver defers the completion handler call
to a tasklet. I could be that it gets called in a hard interrupt and completion
handling is deferred to a soft interrupt context. But for rxe the completion
is always already in a soft interrupt context or a process context.

Bob

> 
>> If we can eliminate this there won't be anymore tasklets in the rxe driver.
>>
>> Does anyone know why the tasklet was put in in the first place?
>>
>> Thanks,
>>
>> Bob