Linux RDMA and InfiniBand development
 help / color / mirror / Atom feed
* [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id
@ 2026-11-17 11:31 Shin'ichiro Kawasaki
  2026-11-17 11:48 ` sashiko-bot
  0 siblings, 1 reply; 2+ messages in thread
From: Shin'ichiro Kawasaki @ 2026-11-17 11:31 UTC (permalink / raw)
  To: linux-rdma
  Cc: linux-nvme, Jason Gunthorpe, Leon Romanovsky, Keith Busch,
	Surabhi Gogte, Shin'ichiro Kawasaki

Since commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue allocation
and startup"), the host can send commands on an I/O queue before the
target has seen RDMA_CM_EVENT_ESTABLISHED for that queue. nvmet-rdma
keeps such commands on queue->rsp_wait_list and executes them from
nvmet_rdma_cm_handler() when the ESTABLISHED event arrives, that is,
with the handler_mutex of the queue's cm_id held. Executing a connect
command takes nvmet_config_sem, which creates a handler_mutex ->
nvmet_config_sem dependency. This is the reverse of the order in
nvmet_port_subsys_drop_link(), which holds nvmet_config_sem and calls
rdma_destroy_id() on the port's listening cm_id via
nvmet_disable_port(). Lockdep reports the warning below when blktests
nvme/004 or nvme/031 is run repeatedly with the rdma transport and the
rxe driver:

    WARNING: possible circular locking dependency detected
    rm/7346 is trying to acquire lock:
     (&id_priv->handler_mutex){+.+.}-{4:4}, at: rdma_destroy_id+0x17/0x20 [rdma_cm]
    but task is already holding lock:
     (nvmet_config_sem){++++}-{4:4}, at: nvmet_port_subsys_drop_link+0x52/0x340 [nvmet]

    -> #1 (nvmet_config_sem){++++}-{4:4}:
           down_read+0xa6/0x530
           nvmet_find_get_subsys+0x1f4/0x630 [nvmet]
           nvmet_ctrl_find_get+0xaa/0x320 [nvmet]
           nvmet_execute_io_connect+0x1e0/0x480 [nvmet]
           nvmet_rdma_execute_command+0x18e/0x480 [nvmet_rdma]
           nvmet_rdma_handle_command+0x428/0xb30 [nvmet_rdma]
           nvmet_rdma_cm_handler+0x8a8/0x30f0 [nvmet_rdma]
           cma_cm_event_handler+0xb0/0x3c0 [rdma_cm]
           cma_ib_handler+0x28a/0x9b0 [rdma_cm]
           cm_process_work+0x4b/0x3e0 [ib_cm]
           cm_work_handler+0x2485/0xac10 [ib_cm]

    -> #0 (&id_priv->handler_mutex){+.+.}-{4:4}:
           rdma_destroy_id+0x17/0x20 [rdma_cm]
           nvmet_rdma_disable_port+0x44/0x300 [nvmet_rdma]
           nvmet_rdma_remove_port+0x42/0x60 [nvmet_rdma]
           nvmet_disable_port+0x120/0x1f0 [nvmet]
           nvmet_port_subsys_drop_link+0x261/0x340 [nvmet]
           configfs_unlink+0x36d/0x790
           vfs_unlink+0x278/0xb40
           filename_unlinkat+0x31d/0x670
           __x64_sys_unlinkat+0xae/0x100

    other info that might help us debug this:

     Possible unsafe locking scenario:

           CPU0                    CPU1
           ----                    ----
      lock(nvmet_config_sem);
                                   lock(&id_priv->handler_mutex);
                                   lock(nvmet_config_sem);
      lock(&id_priv->handler_mutex);

     *** DEADLOCK ***

However, the two handler_mutex instances belong to different cm_ids. The
one held during command execution belongs to a connected queue's cm_id,
while the one taken by rdma_destroy_id() belongs to the listening cm_id.
nvmet-rdma does not take nvmet_config_sem while handling events of the
listening cm_id, so this is not a real deadlock. Lockdep cannot
distinguish the two because all handler_mutex instances share a single
lock class.

To avoid the false-positive WARN, assign a separate lockdep class to the
handler_mutex of a listening cm_id at the start of rdma_listen().

Link: https://lore.kernel.org/linux-nvme/at36um4zrsfOr3kh@shinmob/
Fixes: 2a8513091d2f ("nvme-rdma: parallelize I/O queue allocation and startup")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
* Changes from RFC v1
- Instead of the new helper function, do the reclassfication in rdma_listen()
- Link to RFC v1: https://lore.kernel.org/linux-rdma/20261108120154.148824-1-shinichiro.kawasaki@wdc.com/

 drivers/infiniband/core/cma.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 337a49d1acf7..b3678ddc79a4 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -3949,6 +3949,13 @@ int rdma_listen(struct rdma_cm_id *id, int backlog)
 		container_of(id, struct rdma_id_private, id);
 	int ret;
 
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+	static struct lock_class_key cma_listen_handler_mutex_key;
+
+	lockdep_set_class(&id_priv->handler_mutex,
+			  &cma_listen_handler_mutex_key);
+#endif
+
 	if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_LISTEN)) {
 		struct sockaddr_in any_in = {
 			.sin_family = AF_INET,
-- 
2.56.0


^ permalink raw reply related	[flat|nested] 2+ messages in thread

* Re: [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id
  2026-11-17 11:31 [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id Shin'ichiro Kawasaki
@ 2026-11-17 11:48 ` sashiko-bot
  0 siblings, 0 replies; 2+ messages in thread
From: sashiko-bot @ 2026-11-17 11:48 UTC (permalink / raw)
  To: Shin'ichiro Kawasaki, linux-rdma; +Cc: linux-nvme

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] RDMA/cma: unsafe lockdep_set_class() placement in
  rdma_listen()
- [Low] RDMA/cma: unreclassified handler_mutex on
  nvmet_rdma_enable_port() error
--

commit d2c299202b2d013f5516cd16c2fdd187fc66e56a
Author: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>

RDMA/cma: Use a separate lockdep class for the listening cm_id

This commit assigns a separate lockdep class to id_priv->handler_mutex at the
start of rdma_listen() to prevent a false-positive circular locking dependency
warning between listening and connected cm_ids in nvmet-rdma.

Link: https://lore.kernel.org/linux-nvme/at36um4zrsfOr3kh@shinmob/

> diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
> index 337a49d1acf72..b3678ddc79a4a 100644
> --- a/drivers/infiniband/core/cma.c
> +++ b/drivers/infiniband/core/cma.c
> @@ -3949,6 +3949,13 @@ int rdma_listen(struct rdma_cm_id *id, int backlog)
>  		container_of(id, struct rdma_id_private, id);
>  	int ret;
>  
> +#ifdef CONFIG_DEBUG_LOCK_ALLOC
> +	static struct lock_class_key cma_listen_handler_mutex_key;
> +
> +	lockdep_set_class(&id_priv->handler_mutex,
> +			  &cma_listen_handler_mutex_key);
> +#endif
> +
>  	if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_LISTEN)) {
>  		struct sockaddr_in any_in = {
>  			.sin_family = AF_INET,

[Severity: Medium]
Can calling lockdep_set_class() here race with concurrent users of
id_priv->handler_mutex and misclassify non-listening IDs?

rdma_listen() reinitializes id_priv->handler_mutex.dep_map before validating
id_priv->state and without holding id_priv->handler_mutex.  Through
ucma_listen(), userspace only holds ctx->mutex and can call rdma_listen() on a
cm_id in any state while another thread or CM callback acquires or releases
id_priv->handler_mutex:

CPU0                                  CPU1
ucma_listen()                         cma_ib_handler()
  mutex_lock(&ctx->mutex)               mutex_lock(&id_priv->handler_mutex)
  rdma_listen()                           look_up_lock_class()
    lockdep_set_class(&handler_mutex)     ...  <-- data race on dep_map
    cma_comp_exch() -> fails

Even when id_priv->state is RDMA_CM_ADDR_BOUND, id_priv is already on
cma_dev->id_list, where cma_send_device_removal_put() or addr_handler() can
hold id_priv->handler_mutex concurrently.  Mutating dep_map while the mutex is
being acquired or held can trigger the lockdep WARN_ONCE() in
look_up_lock_class() or corrupt the held-lock stack on unlock and disable
lockdep.

In addition, if rdma_listen() fails the state check, fails rdma_bind_addr(),
or jumps to err: and reverts id_priv->state to RDMA_CM_ADDR_BOUND, a
non-listening cm_id is left with cma_listen_handler_mutex_key, defeating the
lock-class separation if the ID is later used for a connection.

[Severity: Low]
Does reclassifying id_priv->handler_mutex in rdma_listen() still leave the
false-positive lockdep splat reachable when nvmet_rdma_enable_port() fails
before reaching rdma_listen()?

nvmet_port_subsys_allow_link() holds nvmet_config_sem across
nvmet_enable_port() -> nvmet_rdma_add_port() -> nvmet_rdma_enable_port():

drivers/nvme/target/rdma.c:nvmet_rdma_enable_port() {
    ...
	ret = rdma_bind_addr(cm_id, addr);
	if (ret) {
		pr_err("binding CM ID to %pISpcs failed (%d)\n", addr, ret);
		goto out_destroy_id;
	}

	ret = rdma_listen(cm_id, 128);
	if (ret) {
		pr_err("listening to %pISpcs failed (%d)\n", addr, ret);
		goto out_destroy_id;
	}

	port->priv = cm_id;
	return 0;

out_destroy_id:
	rdma_destroy_id(cm_id);
	return ret;
}

If rdma_set_afonly() or rdma_bind_addr() fails, rdma_destroy_id(cm_id) locks
id_priv->handler_mutex under nvmet_config_sem while handler_mutex still has
the default lockdep class, recording the same nvmet_config_sem ->
handler_mutex dependency as before.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261117113129.746490-1-shinichiro.kawasaki@wdc.com?part=1

^ permalink raw reply	[flat|nested] 2+ messages in thread

end of thread, other threads:[~2026-11-17 11:48 UTC | newest]

Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-17 11:31 [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id Shin'ichiro Kawasaki
2026-11-17 11:48 ` sashiko-bot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox