[PATCH RFC 0/2] RDMA/cma, nvmet-rdma: fix a lockdep WARN

[PATCH RFC 0/2] RDMA/cma, nvmet-rdma: fix a lockdep WARN

[Shin'ichiro Kawasaki]

Since the commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue
allocation and startup"), the blktests test cases nvme/004 and nvme/031
often fails with rdma transport due to a lockdep WARN, which is a false-
positive. Please refer to the first patch for the details of the WARN.

This series addresses the failure by reclassifying the handler_mutex of
the port listening cm_id. The first patch introduces
rdma_reclassify_id_for_listen() to RDMA/cma, which sets a dedicated
lockdep class to the handler_mutex of the specified cm_id. The second
patch calls it from nvmet-rdma for the port listening cm_id.

I made this series as RFC, because I was not so sure whether adding a
new function is the best approach. Another idea was to add a new
argument to rdma_create_id() to tell the cm_id created is for listen
of not, but it will touch a number of drivers. Yet another idea was to
add another work in nvmet-rdma to defer the connection establishment
[1], but adding a new work for the false-positive WARN is not desirable.
Then I think this patch series is the best solution at this moment.

Any comments will be welcomed. Thanks.

[1] https://lore.kernel.org/linux-nvme/at36um4zrsfOr3kh@shinmob/


Shin'ichiro Kawasaki (2):
  RDMA/cma: Add rdma_reclassify_id_for_listen()
  nvmet-rdma: use a separate lockdep class for the listening cm_id

 drivers/infiniband/core/cma.c | 13 +++++++++++++
 drivers/nvme/target/rdma.c    |  2 ++
 include/rdma/rdma_cm.h        |  6 ++++++
 3 files changed, 21 insertions(+)

-- 
2.56.0

[PATCH RFC 1/2] RDMA/cma: Add rdma_reclassify_id_for_listen()

[Shin'ichiro Kawasaki]

Since the commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue
allocation and startup"), the host sends commands on an I/O queue before
the target has seen RDMA_CM_EVENT_ESTABLISHED for that queue. nvmet-rdma
keeps such commands on queue->rsp_wait_list and executes them from
nvmet_rdma_cm_handler() when the ESTABLISHED event arrives, that is,
with the handler_mutex of the queue's cm_id held. Executing a connect
command takes nvmet_config_sem, which creates a handler_mutex ->
nvmet_config_sem dependency. This is the reverse of the order in
nvmet_port_subsys_drop_link(), which holds nvmet_config_sem and calls
rdma_destroy_id() on the port's listening cm_id via
nvmet_disable_port(). Lockdep reports the warning below when blktests
nvme/004 or nvme/031 is run repeatedly with the rdma transport and the
rxe driver:

    WARNING: possible circular locking dependency detected
    rm/7346 is trying to acquire lock:
     (&id_priv->handler_mutex){+.+.}-{4:4}, at: rdma_destroy_id+0x17/0x20 [rdma_cm]
    but task is already holding lock:
     (nvmet_config_sem){++++}-{4:4}, at: nvmet_port_subsys_drop_link+0x52/0x340 [nvmet]

    -> #1 (nvmet_config_sem){++++}-{4:4}:
           down_read+0xa6/0x530
           nvmet_find_get_subsys+0x1f4/0x630 [nvmet]
           nvmet_ctrl_find_get+0xaa/0x320 [nvmet]
           nvmet_execute_io_connect+0x1e0/0x480 [nvmet]
           nvmet_rdma_execute_command+0x18e/0x480 [nvmet_rdma]
           nvmet_rdma_handle_command+0x428/0xb30 [nvmet_rdma]
           nvmet_rdma_cm_handler+0x8a8/0x30f0 [nvmet_rdma]
           cma_cm_event_handler+0xb0/0x3c0 [rdma_cm]
           cma_ib_handler+0x28a/0x9b0 [rdma_cm]
           cm_process_work+0x4b/0x3e0 [ib_cm]
           cm_work_handler+0x2485/0xac10 [ib_cm]

    -> #0 (&id_priv->handler_mutex){+.+.}-{4:4}:
           rdma_destroy_id+0x17/0x20 [rdma_cm]
           nvmet_rdma_disable_port+0x44/0x300 [nvmet_rdma]
           nvmet_rdma_remove_port+0x42/0x60 [nvmet_rdma]
           nvmet_disable_port+0x120/0x1f0 [nvmet]
           nvmet_port_subsys_drop_link+0x261/0x340 [nvmet]
           configfs_unlink+0x36d/0x790
           vfs_unlink+0x278/0xb40
           filename_unlinkat+0x31d/0x670
           __x64_sys_unlinkat+0xae/0x100

    other info that might help us debug this:

     Possible unsafe locking scenario:

           CPU0                    CPU1
           ----                    ----
      lock(nvmet_config_sem);
                                   lock(&id_priv->handler_mutex);
                                   lock(nvmet_config_sem);
      lock(&id_priv->handler_mutex);

     *** DEADLOCK ***

However, the two handler_mutex instances belong to different cm_ids. The
one held during command execution belongs to a connected queue's cm_id,
while the one taken by rdma_destroy_id() belongs to the listening cm_id.
nvmet-rdma does not take nvmet_config_sem while handling events of the
listening cm_id, so this is not a real deadlock. Lockdep handles the two
in the same manner because all handler_mutex instances share a single
lock class.

Add rdma_reclassify_id_for_listen(), which assigns a separate lockdep
class to the handler_mutex of a listening cm_id. rdma users can call it
right after creating the cm_id they listen on. A follow-up patch makes
nvmet-rdma call it.

Link: https://lore.kernel.org/linux-nvme/at36um4zrsfOr3kh@shinmob/
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 drivers/infiniband/core/cma.c | 13 +++++++++++++
 include/rdma/rdma_cm.h        |  6 ++++++
 2 files changed, 19 insertions(+)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 337a49d1acf7..436e2467a36f 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -1064,6 +1064,19 @@ __rdma_create_kernel_id(struct net *net, rdma_cm_event_handler event_handler,
 }
 EXPORT_SYMBOL(__rdma_create_kernel_id);
 
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+static struct lock_class_key cma_listen_handler_mutex_key;
+void rdma_reclassify_id_for_listen(struct rdma_cm_id *id)
+{
+	struct rdma_id_private *id_priv;
+
+	id_priv = container_of(id, struct rdma_id_private, id);
+	lockdep_set_class(&id_priv->handler_mutex,
+			  &cma_listen_handler_mutex_key);
+}
+EXPORT_SYMBOL(rdma_reclassify_id_for_listen);
+#endif
+
 struct rdma_cm_id *rdma_create_user_id(rdma_cm_event_handler event_handler,
 				       void *context,
 				       enum rdma_ucm_port_space ps,
diff --git a/include/rdma/rdma_cm.h b/include/rdma/rdma_cm.h
index d639ff889e64..59dda4b0173e 100644
--- a/include/rdma/rdma_cm.h
+++ b/include/rdma/rdma_cm.h
@@ -158,6 +158,12 @@ struct rdma_cm_id *rdma_create_user_id(rdma_cm_event_handler event_handler,
 	__rdma_create_kernel_id(net, event_handler, context, ps, qp_type,      \
 				KBUILD_MODNAME)
 
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+void rdma_reclassify_id_for_listen(struct rdma_cm_id *id);
+#else
+static inline void rdma_reclassify_id_for_listen(struct rdma_cm_id *id) {}
+#endif
+
 /**
   * rdma_destroy_id - Destroys an RDMA identifier.
   *
-- 
2.56.0

Re: [PATCH RFC 1/2] RDMA/cma: Add rdma_reclassify_id_for_listen()

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] RDMA/cma: missing handler_mutex lockdep reclassification in
  cma_listen_on_dev()
- [Low] RDMA/cma: missing kernel-doc for rdma_reclassify_id_for_listen()
  ordering rules
--

commit 1c34d3d5b4de4179de0a92ec2b97020a760e60b6
Author: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>

RDMA/cma: Add rdma_reclassify_id_for_listen()

This commit adds rdma_reclassify_id_for_listen() to assign a separate lockdep
class to the handler_mutex of listening rdma_cm_id instances.

Link: https://lore.kernel.org/linux-nvme/at36um4zrsfOr3kh@shinmob/

> diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
> index 337a49d1acf72..436e2467a36ff 100644
> --- a/drivers/infiniband/core/cma.c
> +++ b/drivers/infiniband/core/cma.c
> @@ -1064,6 +1064,19 @@ __rdma_create_kernel_id(struct net *net, rdma_cm_event_handler event_handler,
>  }
>  EXPORT_SYMBOL(__rdma_create_kernel_id);
>  
> +#ifdef CONFIG_DEBUG_LOCK_ALLOC
> +static struct lock_class_key cma_listen_handler_mutex_key;
> +void rdma_reclassify_id_for_listen(struct rdma_cm_id *id)
> +{
> +	struct rdma_id_private *id_priv;
> +
> +	id_priv = container_of(id, struct rdma_id_private, id);
> +	lockdep_set_class(&id_priv->handler_mutex,
> +			  &cma_listen_handler_mutex_key);
> +}
> +EXPORT_SYMBOL(rdma_reclassify_id_for_listen);
> +#endif

[Severity: Medium]
Does cma_listen_on_dev() also need to reclassify dev_id_priv->handler_mutex
for wildcard listeners?

When a caller binds a listening cm_id to a wildcard address (such as 0.0.0.0
or ::), rdma_listen() -> cma_listen_on_all() -> cma_listen_on_dev() allocates
an internal per-device child listening ID (dev_id_priv) for each RDMA device:

drivers/infiniband/core/cma.c:cma_listen_on_dev() {
	...
	dev_id_priv =
		__rdma_create_id(net, cma_listen_handler, id_priv,
				 id_priv->id.ps, id_priv->id.qp_type, id_priv);
	...
}

Because dev_id_priv->handler_mutex keeps the default handler_mutex lockdep
class, destroying the wildcard listening cm_id under nvmet_config_sem still
triggers the same lockdep circular locking dependency warning when
rdma_destroy_id() -> _destroy_id() -> cma_cancel_operation() ->
cma_cancel_listens() -> _cma_cancel_listens() calls
rdma_destroy_id(&dev_id_priv->id).

> diff --git a/include/rdma/rdma_cm.h b/include/rdma/rdma_cm.h
> index d639ff889e649..59dda4b0173e7 100644
> --- a/include/rdma/rdma_cm.h
> +++ b/include/rdma/rdma_cm.h
> @@ -158,6 +158,12 @@ struct rdma_cm_id *rdma_create_user_id(rdma_cm_event_handler event_handler,
>  	__rdma_create_kernel_id(net, event_handler, context, ps, qp_type,      \
>  				KBUILD_MODNAME)
>  
> +#ifdef CONFIG_DEBUG_LOCK_ALLOC
> +void rdma_reclassify_id_for_listen(struct rdma_cm_id *id);
> +#else
> +static inline void rdma_reclassify_id_for_listen(struct rdma_cm_id *id) {}
> +#endif

[Severity: Low]
Could rdma_reclassify_id_for_listen() have a kernel-doc comment like the
neighboring declarations in include/rdma/rdma_cm.h?  Since it calls
lockdep_set_class() on handler_mutex, documenting that it must be called on a
newly created rdma_cm_id before rdma_bind_addr() or rdma_listen() can acquire
handler_mutex would make the ordering requirement clear to callers.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261108120154.148824-1-shinichiro.kawasaki@wdc.com?part=1

[PATCH RFC 2/2] nvmet-rdma: use a separate lockdep class for the listening cm_id

[Shin'ichiro Kawasaki]

Since the commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue
allocation and startup"), the host sends commands on an I/O queue before
the target has seen RDMA_CM_EVENT_ESTABLISHED for that queue. nvmet-rdma
keeps such commands on queue->rsp_wait_list and executes them from
nvmet_rdma_cm_handler() when the ESTABLISHED event arrives, that is,
with the handler_mutex of the queue's cm_id held. Executing a connect
command takes nvmet_config_sem, which creates a handler_mutex ->
nvmet_config_sem dependency. This is the reverse of the order in
nvmet_port_subsys_drop_link(), which holds nvmet_config_sem and calls
rdma_destroy_id() on the port's listening cm_id via nvmet_disable_port().
Lockdep then reports a possible circular locking dependency:

     Possible unsafe locking scenario:

           CPU0                    CPU1
           ----                    ----
      lock(nvmet_config_sem);
                                   lock(&id_priv->handler_mutex);
                                   lock(nvmet_config_sem);
      lock(&id_priv->handler_mutex);

The warning is a false positive. The handler_mutex on CPU1 belongs to a
connected queue's cm_id, while the one on CPU0 belongs to the listening
cm_id, and nvmet-rdma does not take nvmet_config_sem while handling
events of the listening cm_id.

To silence the warning, call rdma_reclassify_id_for_listen() on the
listening cm_id right after it is created in nvmet_rdma_enable_port(),
so that its handler_mutex gets a lockdep class of its own.

This depends on the previous patch, which adds
rdma_reclassify_id_for_listen() to the RDMA CM.

Fixes: 2a8513091d2f ("nvme-rdma: parallelize I/O queue allocation and startup")
Link: https://lore.kernel.org/linux-nvme/at36um4zrsfOr3kh@shinmob/
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
 drivers/nvme/target/rdma.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index b9e39d1657d2..94f6d8edb517 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -1914,6 +1914,8 @@ static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port)
 		return PTR_ERR(cm_id);
 	}
 
+	rdma_reclassify_id_for_listen(cm_id);
+
 	/*
 	 * Allow both IPv4 and IPv6 sockets to bind a single port
 	 * at the same time.
-- 
2.56.0

Re: [PATCH RFC 2/2] nvmet-rdma: use a separate lockdep class for the listening cm_id

[sashiko-bot]

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Low] nvmet-rdma: wildcard listener child cm_ids keep default lockdep
  class
--

commit a129fce174ee6d77cacc5458875c55e53aa41b7e
Author: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>

nvmet-rdma: use a separate lockdep class for the listening cm_id

This commit calls rdma_reclassify_id_for_listen() on the port's listening
cm_id in nvmet_rdma_enable_port() to give its handler_mutex a separate
lockdep class from connected queue cm_ids.

Link: https://lore.kernel.org/linux-nvme/at36um4zrsfOr3kh@shinmob/

> diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
> index b9e39d1657d29..94f6d8edb517f 100644
> --- a/drivers/nvme/target/rdma.c
> +++ b/drivers/nvme/target/rdma.c
> @@ -1914,6 +1914,8 @@ static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port)
>  		return PTR_ERR(cm_id);
>  	}
>  
> +	rdma_reclassify_id_for_listen(cm_id);
> +
>  	/*
>  	 * Allow both IPv4 and IPv6 sockets to bind a single port
>  	 * at the same time.

[Severity: Low]
Does this still leave the per-device child listening cm_ids in the default
handler_mutex lockdep class when port->addr is a wildcard address?

For a wildcard address, rdma_bind_addr() leaves id_priv->cma_dev NULL, so
rdma_listen() calls cma_listen_on_all() -> cma_listen_on_dev(), which
creates a per-device child dev_id_priv via __rdma_create_id() without
reclassifying its handler_mutex:

drivers/infiniband/core/cma.c:cma_listen_on_dev() {
	...
	dev_id_priv =
		__rdma_create_id(net, cma_listen_handler, id_priv,
				 id_priv->id.ps, id_priv->id.qp_type, id_priv);
	...
}

When nvmet_rdma_disable_port() later calls rdma_destroy_id(cm_id) under
nvmet_config_sem, _cma_cancel_listens() calls
rdma_destroy_id(&dev_id_priv->id) and takes dev_id_priv->handler_mutex,
triggering the same false-positive lockdep warning.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261108120154.148824-1-shinichiro.kawasaki@wdc.com?part=2

Re: [PATCH RFC 2/2] nvmet-rdma: use a separate lockdep class for the listening cm_id

[Jason Gunthorpe]

On Sun, Nov 08, 2026 at 09:01:54PM +0900, Shin'ichiro Kawasaki wrote:
> @@ -1914,6 +1914,8 @@ static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port)
>  		return PTR_ERR(cm_id);
>  	}
>  
> +	rdma_reclassify_id_for_listen(cm_id);
> +

It seems like a flag or new function for the create would make more
sense, and if this is the right fix then all the callers with the
pattern of rdma_create_id() .. rdma_listen() should be fixed too?

I suppose we cannot change the lock class in rdma_listen ?

Jason

Re: [PATCH RFC 2/2] nvmet-rdma: use a separate lockdep class for the listening cm_id

[Shin'ichiro Kawasaki]

Jason, thank you for the comments.

On Nov 09, 2026 / 20:32, Jason Gunthorpe wrote:
> On Sun, Nov 08, 2026 at 09:01:54PM +0900, Shin'ichiro Kawasaki wrote:
> > @@ -1914,6 +1914,8 @@ static int nvmet_rdma_enable_port(struct nvmet_rdma_port *port)
> >  		return PTR_ERR(cm_id);
> >  	}
> >  
> > +	rdma_reclassify_id_for_listen(cm_id);
> > +
> 
> It seems like a flag or new function for the create would make more
> sense,

I see, new function approach sounds better than this patch.

> and if this is the right fix then all the callers with the
> pattern of rdma_create_id() .. rdma_listen() should be fixed too?

I agree that it is ideal to apply the same change to the all the callers.

> 
> I suppose we cannot change the lock class in rdma_listen ?

Oh, this approach sounds the best. It modifies only RDMA driver, and can cover
all the callers. I did a quick trial with the change below, and confirmed it
avoids the WARN. So this approach works well :)  I will wait some more days for
comments. If there is no other voices, I will post this solution as a formal
patch.

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 337a49d1acf7..334f39c2f5e5 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -3949,6 +3949,12 @@ int rdma_listen(struct rdma_cm_id *id, int backlog)
 		container_of(id, struct rdma_id_private, id);
 	int ret;
 
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+	static struct lock_class_key cma_listen_handler_mutex_key;
+	lockdep_set_class(&id_priv->handler_mutex,
+			  &cma_listen_handler_mutex_key);
+#endif
+
 	if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_LISTEN)) {
 		struct sockaddr_in any_in = {
 			.sin_family = AF_INET,

Re: [PATCH RFC 2/2] nvmet-rdma: use a separate lockdep class for the listening cm_id

[Jason Gunthorpe]

On Wed, Nov 11, 2026 at 02:01:58PM +0900, Shin'ichiro Kawasaki wrote:

> diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
> index 337a49d1acf7..334f39c2f5e5 100644
> --- a/drivers/infiniband/core/cma.c
> +++ b/drivers/infiniband/core/cma.c
> @@ -3949,6 +3949,12 @@ int rdma_listen(struct rdma_cm_id *id, int backlog)
>  		container_of(id, struct rdma_id_private, id);
>  	int ret;
>  
> +#ifdef CONFIG_DEBUG_LOCK_ALLOC
> +	static struct lock_class_key cma_listen_handler_mutex_key;
> +	lockdep_set_class(&id_priv->handler_mutex,
> +			  &cma_listen_handler_mutex_key);
> +#endif
> +

This seems OK, I think we already have examples of this in CM IIRC?

Jason

Re: [PATCH RFC 2/2] nvmet-rdma: use a separate lockdep class for the listening cm_id

[Shin'ichiro Kawasaki]

On Nov 11, 2026 / 11:37, Jason Gunthorpe wrote:
> On Wed, Nov 11, 2026 at 02:01:58PM +0900, Shin'ichiro Kawasaki wrote:
> 
> > diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
> > index 337a49d1acf7..334f39c2f5e5 100644
> > --- a/drivers/infiniband/core/cma.c
> > +++ b/drivers/infiniband/core/cma.c
> > @@ -3949,6 +3949,12 @@ int rdma_listen(struct rdma_cm_id *id, int backlog)
> >  		container_of(id, struct rdma_id_private, id);
> >  	int ret;
> >  
> > +#ifdef CONFIG_DEBUG_LOCK_ALLOC
> > +	static struct lock_class_key cma_listen_handler_mutex_key;
> > +	lockdep_set_class(&id_priv->handler_mutex,
> > +			  &cma_listen_handler_mutex_key);
> > +#endif
> > +
> 
> This seems OK, I think we already have examples of this in CM IIRC?

Yes, I think so. siw and rxe drivers use dynamic lockdep classes for their
sockets:

$ git checkout v7.4-rc2
HEAD is now at 7475ea8b5861 Linux 7.4-rc2
$ cd drivers/infiniband/
$ git grep lock_class_key
sw/rxe/rxe_net.c:static struct lock_class_key rxe_recv_sk_key[2];
sw/rxe/rxe_net.c:static struct lock_class_key rxe_recv_slock_key[2];
sw/rxe/rxe_qp.c:static struct lock_class_key rxe_send_sk_key[2];
sw/rxe/rxe_qp.c:static struct lock_class_key rxe_send_slock_key[2];
sw/siw/siw_cm.c:static struct lock_class_key siw_sk_key[2];
sw/siw/siw_cm.c:static struct lock_class_key siw_slock_key[2];