Re: [Bug report] NULL pointer dereference in frwr_unmap_sync()

[Dai Ngo <dai.ngo@oracle.com>]

On 3/9/25 12:40 PM, Dai Ngo wrote:
> Hi Nan,
>
> Can you try the attached patch with 6.14-rc4?
>
> This patch adds a spinlock to protect the rl_free_mrs and 
> rl_registered list.
> I have seen list corruption in our RDMA testing but the problem is 
> hard to
> reproduce so I have not submitted this patch to upstream.

Any update on testing of this patch?

Thanks,
-Dai

>
> Thanks,
> -Dai
>
> On 3/5/25 6:40 PM, Li Nan wrote:
>>
>>
>> 在 2025/3/5 22:02, Chuck Lever 写道:
>>> On 3/4/25 9:43 PM, Li Nan wrote:
>>>> We found a following problem in kernel 5.10, and the same problem 
>>>> should
>>>> exist in mainline:
>>>>
>>>> During NFS mount using 'soft' option over RoCE network, we observed 
>>>> kernel
>>>> crash with below trace when network issues occur 
>>>> (congestion/disconnect):
>>>>    nfs: server 10.10.253.211 not responding, timed out
>>>>    BUG: kernel NULL pointer dereference, address: 00000000000000a0
>>>>    RIP: 0010:frwr_unmap_sync+0x77/0x200 [rpcrdma]
>>>>    Call Trace:
>>>>     ? __die_body.cold+0x8/0xd
>>>>     ? no_context+0x155/0x230
>>>>     ? __bad_area_nosemaphore+0x52/0x1a0
>>>>     ? exc_page_fault+0x2dc/0x550
>>>>     ? asm_exc_page_fault+0x1e/0x30
>>>>     ? frwr_unmap_sync+0x77/0x200 [rpcrdma]
>>>>     xprt_release+0x9e/0x1a0 [sunrpc]
>>>>     rpc_release_resources_task+0xe/0x50 [sunrpc]
>>>>     rpc_release_task+0x19/0xa0 [sunrpc]
>>>>     rpc_async_schedule+0x29/0x40 [sunrpc]
>>>>     process_one_work+0x1b2/0x350
>>>>     worker_thread+0x49/0x310
>>>>     ? rescuer_thread+0x380/0x380
>>>>     kthread+0xfb/0x140
>>>>
>>>> Problem analysis:
>>>> The crash happens in frwr_unmap_sync() when accessing 
>>>> req->rl_registered
>>>> list, caused by either NULL pointer or accessing freed MR resources.
>>>> There's a race condition between:
>>>> T1
>>>> __ib_process_cq
>>>>   wc->wr_cqe->done (frwr_wc_localinv)
>>>>    rpcrdma_flush_disconnect
>>>>     rpcrdma_force_disconnect
>>>>      xprt_force_disconnect
>>>>       xprt_autoclose
>>>>        xprt_rdma_close
>>>>         rpcrdma_xprt_disconnect
>>>>          rpcrdma_reqs_reset
>>>>           frwr_reset
>>>>            rpcrdma_mr_pop(&req->rl_registered)
>>>> T2
>>>> rpc_async_schedule
>>>>   rpc_release_task
>>>>    rpc_release_resources_task
>>>>     xprt_release
>>>>      xprt_rdma_free
>>>>       frwr_unmap_sync
>>>>        rpcrdma_mr_pop(&req->rl_registered)
>>>>                     This problem also exists in function 
>>>> rpcrdma_mrs_destroy().
>>>>
>>>
>>> Dai, is this the same as the system test problem you've been looking 
>>> at?
>>>
>>
>> Thank you for looking into it. Is there a patch that needs to be 
>> tested? We
>> are happy to help with the testing.
>>