From: Chuck Lever <chuck.lever@oracle.com>
To: Li Nan <linan666@huaweicloud.com>, Dai Ngo <Dai.Ngo@oracle.com>
Cc: "linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
linux-nfs@vger.kernel.org, trondmy@hammerspace.com,
sagi@grimberg.me, cel@kernel.org,
"wanghai (M)" <wanghai38@huawei.com>,
yanhaitao2@huawei.com, chengjike.cheng@huawei.com,
dingming09@huawei.com
Subject: Re: [Bug report] NULL pointer dereference in frwr_unmap_sync()
Date: Wed, 5 Mar 2025 09:02:42 -0500 [thread overview]
Message-ID: <314f60a8-4b0d-45f9-87f4-5a4757d34aea@oracle.com> (raw)
In-Reply-To: <e7c72dfc-ecbc-bd99-16f6-977afa642f18@huaweicloud.com>
On 3/4/25 9:43 PM, Li Nan wrote:
> We found a following problem in kernel 5.10, and the same problem should
> exist in mainline:
>
> During NFS mount using 'soft' option over RoCE network, we observed kernel
> crash with below trace when network issues occur (congestion/disconnect):
> nfs: server 10.10.253.211 not responding, timed out
> BUG: kernel NULL pointer dereference, address: 00000000000000a0
> RIP: 0010:frwr_unmap_sync+0x77/0x200 [rpcrdma]
> Call Trace:
> ? __die_body.cold+0x8/0xd
> ? no_context+0x155/0x230
> ? __bad_area_nosemaphore+0x52/0x1a0
> ? exc_page_fault+0x2dc/0x550
> ? asm_exc_page_fault+0x1e/0x30
> ? frwr_unmap_sync+0x77/0x200 [rpcrdma]
> xprt_release+0x9e/0x1a0 [sunrpc]
> rpc_release_resources_task+0xe/0x50 [sunrpc]
> rpc_release_task+0x19/0xa0 [sunrpc]
> rpc_async_schedule+0x29/0x40 [sunrpc]
> process_one_work+0x1b2/0x350
> worker_thread+0x49/0x310
> ? rescuer_thread+0x380/0x380
> kthread+0xfb/0x140
>
> Problem analysis:
> The crash happens in frwr_unmap_sync() when accessing req->rl_registered
> list, caused by either NULL pointer or accessing freed MR resources.
> There's a race condition between:
> T1
> __ib_process_cq
> wc->wr_cqe->done (frwr_wc_localinv)
> rpcrdma_flush_disconnect
> rpcrdma_force_disconnect
> xprt_force_disconnect
> xprt_autoclose
> xprt_rdma_close
> rpcrdma_xprt_disconnect
> rpcrdma_reqs_reset
> frwr_reset
> rpcrdma_mr_pop(&req->rl_registered)
> T2
> rpc_async_schedule
> rpc_release_task
> rpc_release_resources_task
> xprt_release
> xprt_rdma_free
> frwr_unmap_sync
> rpcrdma_mr_pop(&req->rl_registered)
>
> This problem also exists in function rpcrdma_mrs_destroy().
>
Dai, is this the same as the system test problem you've been looking at?
--
Chuck Lever
next prev parent reply other threads:[~2025-03-05 14:03 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-03-05 2:43 [Bug report] NULL pointer dereference in frwr_unmap_sync() Li Nan
2025-03-05 14:02 ` Chuck Lever [this message]
2025-03-06 2:40 ` Li Nan
2025-03-09 19:40 ` Dai Ngo
2025-04-09 16:26 ` Dai Ngo
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=314f60a8-4b0d-45f9-87f4-5a4757d34aea@oracle.com \
--to=chuck.lever@oracle.com \
--cc=Dai.Ngo@oracle.com \
--cc=cel@kernel.org \
--cc=chengjike.cheng@huawei.com \
--cc=dingming09@huawei.com \
--cc=linan666@huaweicloud.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-nfs@vger.kernel.org \
--cc=sagi@grimberg.me \
--cc=trondmy@hammerspace.com \
--cc=wanghai38@huawei.com \
--cc=yanhaitao2@huawei.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox