* [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id
@ 2026-11-17 11:31 Shin'ichiro Kawasaki
2026-11-17 11:48 ` sashiko-bot
0 siblings, 1 reply; 2+ messages in thread
From: Shin'ichiro Kawasaki @ 2026-11-17 11:31 UTC (permalink / raw)
To: linux-rdma
Cc: linux-nvme, Jason Gunthorpe, Leon Romanovsky, Keith Busch,
Surabhi Gogte, Shin'ichiro Kawasaki
Since commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue allocation
and startup"), the host can send commands on an I/O queue before the
target has seen RDMA_CM_EVENT_ESTABLISHED for that queue. nvmet-rdma
keeps such commands on queue->rsp_wait_list and executes them from
nvmet_rdma_cm_handler() when the ESTABLISHED event arrives, that is,
with the handler_mutex of the queue's cm_id held. Executing a connect
command takes nvmet_config_sem, which creates a handler_mutex ->
nvmet_config_sem dependency. This is the reverse of the order in
nvmet_port_subsys_drop_link(), which holds nvmet_config_sem and calls
rdma_destroy_id() on the port's listening cm_id via
nvmet_disable_port(). Lockdep reports the warning below when blktests
nvme/004 or nvme/031 is run repeatedly with the rdma transport and the
rxe driver:
WARNING: possible circular locking dependency detected
rm/7346 is trying to acquire lock:
(&id_priv->handler_mutex){+.+.}-{4:4}, at: rdma_destroy_id+0x17/0x20 [rdma_cm]
but task is already holding lock:
(nvmet_config_sem){++++}-{4:4}, at: nvmet_port_subsys_drop_link+0x52/0x340 [nvmet]
-> #1 (nvmet_config_sem){++++}-{4:4}:
down_read+0xa6/0x530
nvmet_find_get_subsys+0x1f4/0x630 [nvmet]
nvmet_ctrl_find_get+0xaa/0x320 [nvmet]
nvmet_execute_io_connect+0x1e0/0x480 [nvmet]
nvmet_rdma_execute_command+0x18e/0x480 [nvmet_rdma]
nvmet_rdma_handle_command+0x428/0xb30 [nvmet_rdma]
nvmet_rdma_cm_handler+0x8a8/0x30f0 [nvmet_rdma]
cma_cm_event_handler+0xb0/0x3c0 [rdma_cm]
cma_ib_handler+0x28a/0x9b0 [rdma_cm]
cm_process_work+0x4b/0x3e0 [ib_cm]
cm_work_handler+0x2485/0xac10 [ib_cm]
-> #0 (&id_priv->handler_mutex){+.+.}-{4:4}:
rdma_destroy_id+0x17/0x20 [rdma_cm]
nvmet_rdma_disable_port+0x44/0x300 [nvmet_rdma]
nvmet_rdma_remove_port+0x42/0x60 [nvmet_rdma]
nvmet_disable_port+0x120/0x1f0 [nvmet]
nvmet_port_subsys_drop_link+0x261/0x340 [nvmet]
configfs_unlink+0x36d/0x790
vfs_unlink+0x278/0xb40
filename_unlinkat+0x31d/0x670
__x64_sys_unlinkat+0xae/0x100
other info that might help us debug this:
Possible unsafe locking scenario:
CPU0 CPU1
---- ----
lock(nvmet_config_sem);
lock(&id_priv->handler_mutex);
lock(nvmet_config_sem);
lock(&id_priv->handler_mutex);
*** DEADLOCK ***
However, the two handler_mutex instances belong to different cm_ids. The
one held during command execution belongs to a connected queue's cm_id,
while the one taken by rdma_destroy_id() belongs to the listening cm_id.
nvmet-rdma does not take nvmet_config_sem while handling events of the
listening cm_id, so this is not a real deadlock. Lockdep cannot
distinguish the two because all handler_mutex instances share a single
lock class.
To avoid the false-positive WARN, assign a separate lockdep class to the
handler_mutex of a listening cm_id at the start of rdma_listen().
Link: https://lore.kernel.org/linux-nvme/at36um4zrsfOr3kh@shinmob/
Fixes: 2a8513091d2f ("nvme-rdma: parallelize I/O queue allocation and startup")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
* Changes from RFC v1
- Instead of the new helper function, do the reclassfication in rdma_listen()
- Link to RFC v1: https://lore.kernel.org/linux-rdma/20261108120154.148824-1-shinichiro.kawasaki@wdc.com/
drivers/infiniband/core/cma.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 337a49d1acf7..b3678ddc79a4 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -3949,6 +3949,13 @@ int rdma_listen(struct rdma_cm_id *id, int backlog)
container_of(id, struct rdma_id_private, id);
int ret;
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+ static struct lock_class_key cma_listen_handler_mutex_key;
+
+ lockdep_set_class(&id_priv->handler_mutex,
+ &cma_listen_handler_mutex_key);
+#endif
+
if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_LISTEN)) {
struct sockaddr_in any_in = {
.sin_family = AF_INET,
--
2.56.0
^ permalink raw reply related [flat|nested] 2+ messages in thread
* Re: [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id
2026-11-17 11:31 [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id Shin'ichiro Kawasaki
@ 2026-11-17 11:48 ` sashiko-bot
0 siblings, 0 replies; 2+ messages in thread
From: sashiko-bot @ 2026-11-17 11:48 UTC (permalink / raw)
To: Shin'ichiro Kawasaki, linux-rdma; +Cc: linux-nvme
Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] RDMA/cma: unsafe lockdep_set_class() placement in
rdma_listen()
- [Low] RDMA/cma: unreclassified handler_mutex on
nvmet_rdma_enable_port() error
--
commit d2c299202b2d013f5516cd16c2fdd187fc66e56a
Author: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
RDMA/cma: Use a separate lockdep class for the listening cm_id
This commit assigns a separate lockdep class to id_priv->handler_mutex at the
start of rdma_listen() to prevent a false-positive circular locking dependency
warning between listening and connected cm_ids in nvmet-rdma.
Link: https://lore.kernel.org/linux-nvme/at36um4zrsfOr3kh@shinmob/
> diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
> index 337a49d1acf72..b3678ddc79a4a 100644
> --- a/drivers/infiniband/core/cma.c
> +++ b/drivers/infiniband/core/cma.c
> @@ -3949,6 +3949,13 @@ int rdma_listen(struct rdma_cm_id *id, int backlog)
> container_of(id, struct rdma_id_private, id);
> int ret;
>
> +#ifdef CONFIG_DEBUG_LOCK_ALLOC
> + static struct lock_class_key cma_listen_handler_mutex_key;
> +
> + lockdep_set_class(&id_priv->handler_mutex,
> + &cma_listen_handler_mutex_key);
> +#endif
> +
> if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_LISTEN)) {
> struct sockaddr_in any_in = {
> .sin_family = AF_INET,
[Severity: Medium]
Can calling lockdep_set_class() here race with concurrent users of
id_priv->handler_mutex and misclassify non-listening IDs?
rdma_listen() reinitializes id_priv->handler_mutex.dep_map before validating
id_priv->state and without holding id_priv->handler_mutex. Through
ucma_listen(), userspace only holds ctx->mutex and can call rdma_listen() on a
cm_id in any state while another thread or CM callback acquires or releases
id_priv->handler_mutex:
CPU0 CPU1
ucma_listen() cma_ib_handler()
mutex_lock(&ctx->mutex) mutex_lock(&id_priv->handler_mutex)
rdma_listen() look_up_lock_class()
lockdep_set_class(&handler_mutex) ... <-- data race on dep_map
cma_comp_exch() -> fails
Even when id_priv->state is RDMA_CM_ADDR_BOUND, id_priv is already on
cma_dev->id_list, where cma_send_device_removal_put() or addr_handler() can
hold id_priv->handler_mutex concurrently. Mutating dep_map while the mutex is
being acquired or held can trigger the lockdep WARN_ONCE() in
look_up_lock_class() or corrupt the held-lock stack on unlock and disable
lockdep.
In addition, if rdma_listen() fails the state check, fails rdma_bind_addr(),
or jumps to err: and reverts id_priv->state to RDMA_CM_ADDR_BOUND, a
non-listening cm_id is left with cma_listen_handler_mutex_key, defeating the
lock-class separation if the ID is later used for a connection.
[Severity: Low]
Does reclassifying id_priv->handler_mutex in rdma_listen() still leave the
false-positive lockdep splat reachable when nvmet_rdma_enable_port() fails
before reaching rdma_listen()?
nvmet_port_subsys_allow_link() holds nvmet_config_sem across
nvmet_enable_port() -> nvmet_rdma_add_port() -> nvmet_rdma_enable_port():
drivers/nvme/target/rdma.c:nvmet_rdma_enable_port() {
...
ret = rdma_bind_addr(cm_id, addr);
if (ret) {
pr_err("binding CM ID to %pISpcs failed (%d)\n", addr, ret);
goto out_destroy_id;
}
ret = rdma_listen(cm_id, 128);
if (ret) {
pr_err("listening to %pISpcs failed (%d)\n", addr, ret);
goto out_destroy_id;
}
port->priv = cm_id;
return 0;
out_destroy_id:
rdma_destroy_id(cm_id);
return ret;
}
If rdma_set_afonly() or rdma_bind_addr() fails, rdma_destroy_id(cm_id) locks
id_priv->handler_mutex under nvmet_config_sem while handler_mutex still has
the default lockdep class, recording the same nvmet_config_sem ->
handler_mutex dependency as before.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261117113129.746490-1-shinichiro.kawasaki@wdc.com?part=1
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-11-17 11:48 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-17 11:31 [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id Shin'ichiro Kawasaki
2026-11-17 11:48 ` sashiko-bot
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox