Re: [bug report] blktests srp/002 hang

[Zhu Yanjun <yanjun.zhu@linux.dev>]

在 2023/10/18 1:09, Bob Pearson 写道:
> On 9/25/23 20:17, Daisuke Matsuda (Fujitsu) wrote:
>> On Tue, Sep 26, 2023 12:01 AM Bart Van Assche:
>>> On 9/24/23 21:47, Daisuke Matsuda (Fujitsu) wrote:
>>>> As Bob wrote above, nobody has found any logical failure in rxe
>>>> driver.
>>> That's wrong. In case you would not yet have noticed my latest email in
>>> this thread, please take a look at
>>> https://lore.kernel.org/linux-rdma/e8b76fae-780a-470e-8ec4-c6b650793d10@leemhuis.info/T/#m0fd8ea8a4cbc27b37
>>> b042ae4f8e9b024f1871a73.
>>> I think the report in that email is a 100% proof that there is a
>>> use-after-free issue in the rdma_rxe driver. Use-after-free issues have
>>> security implications and also can cause data corruption. I propose to
>>> revert the commit that introduced the rdma_rxe use-after-free unless
>>> someone comes up with a fix for the rdma_rxe driver.
>>>
>>> Bart.
>> Thank you for the clarification. I see your intention.
>> I hope the hang issue will be resolved by addressing this.
>>
>> Thanks,
>> Daisuke
>>
> I have made some progress in understanding the cause of the srp/002 etc. hang.
>
> The two attached files are traces of activity for two qp's qp#151 and qp#167. In my runs of srp/002
> All the qp's pass before 167 and all fail after 167 which is the first to fail.
>
> It turns out that all the passing qp's call srp_post_send() some number of times and also call
> srp_send_done() the same number of times. Starting at qp#167 the last call to srp_send_done() does
> not take place leaving the srp driver waiting for the final completion and causing the hang I believe.

Thanks, Bob

I will delve into your findings and the source code to find the root cause.

BTW, what linux distribution are you using to find this? Ubuntu, Fedora 
or Debian?

 From the above, sometings this problem is difficult to reproduce on 
Ubuntu. But it can be reproduced in Ubuntu and Debian.

So can you let me know what linux distribution you are using?

Thanks

Zhu Yanjun

>
> There are four cq's involved in each pair of qp's in the srp test. Two in ib_srp and two in ib_srpt
> for the two qp's. Three of them execute completion processing in a soft irq context so the code in
> core/cq.c gathers the completions and calls back to the srp drivers. The send side cq in srp uses
> cq_direct which requires srp to call ib_process_direct() in order to collect the completions. This
> happens in __srp_get_tx_iu() which is called in several places in the srp driver. But only as a side effect
> since the purpose of this routine is to get an iu to start a new command.
>
> In the attached files for qp#151 the final call to srp_post_send is followed by the rxe requester and
> completer work queues processing the send packet and the ack before a final call to __srp_get_rx_iu()
> which gathers the final send side completion and success.
>
> For qp#167 the call to srp_post_send() is followed by the rxe driver processing the send operation and
> generating a work completion which is posted to the send cq but there is never a following call to
> __srp_get_rx_iu() so the cqe is not received by srp and failure.
>
> I don't yet understand the logic of the srp driver to fix this but the problem is not in the rxe driver
> as far as I can tell.
>
> Bob