Linux RDMA and InfiniBand development
 help / color / mirror / Atom feed
* [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id
@ 2026-11-17 11:31 Shin'ichiro Kawasaki
  2026-11-17 11:48 ` sashiko-bot
  0 siblings, 1 reply; 5+ messages in thread
From: Shin'ichiro Kawasaki @ 2026-11-17 11:31 UTC (permalink / raw)
  To: linux-rdma
  Cc: linux-nvme, Jason Gunthorpe, Leon Romanovsky, Keith Busch,
	Surabhi Gogte, Shin'ichiro Kawasaki

Since commit 2a8513091d2f ("nvme-rdma: parallelize I/O queue allocation
and startup"), the host can send commands on an I/O queue before the
target has seen RDMA_CM_EVENT_ESTABLISHED for that queue. nvmet-rdma
keeps such commands on queue->rsp_wait_list and executes them from
nvmet_rdma_cm_handler() when the ESTABLISHED event arrives, that is,
with the handler_mutex of the queue's cm_id held. Executing a connect
command takes nvmet_config_sem, which creates a handler_mutex ->
nvmet_config_sem dependency. This is the reverse of the order in
nvmet_port_subsys_drop_link(), which holds nvmet_config_sem and calls
rdma_destroy_id() on the port's listening cm_id via
nvmet_disable_port(). Lockdep reports the warning below when blktests
nvme/004 or nvme/031 is run repeatedly with the rdma transport and the
rxe driver:

    WARNING: possible circular locking dependency detected
    rm/7346 is trying to acquire lock:
     (&id_priv->handler_mutex){+.+.}-{4:4}, at: rdma_destroy_id+0x17/0x20 [rdma_cm]
    but task is already holding lock:
     (nvmet_config_sem){++++}-{4:4}, at: nvmet_port_subsys_drop_link+0x52/0x340 [nvmet]

    -> #1 (nvmet_config_sem){++++}-{4:4}:
           down_read+0xa6/0x530
           nvmet_find_get_subsys+0x1f4/0x630 [nvmet]
           nvmet_ctrl_find_get+0xaa/0x320 [nvmet]
           nvmet_execute_io_connect+0x1e0/0x480 [nvmet]
           nvmet_rdma_execute_command+0x18e/0x480 [nvmet_rdma]
           nvmet_rdma_handle_command+0x428/0xb30 [nvmet_rdma]
           nvmet_rdma_cm_handler+0x8a8/0x30f0 [nvmet_rdma]
           cma_cm_event_handler+0xb0/0x3c0 [rdma_cm]
           cma_ib_handler+0x28a/0x9b0 [rdma_cm]
           cm_process_work+0x4b/0x3e0 [ib_cm]
           cm_work_handler+0x2485/0xac10 [ib_cm]

    -> #0 (&id_priv->handler_mutex){+.+.}-{4:4}:
           rdma_destroy_id+0x17/0x20 [rdma_cm]
           nvmet_rdma_disable_port+0x44/0x300 [nvmet_rdma]
           nvmet_rdma_remove_port+0x42/0x60 [nvmet_rdma]
           nvmet_disable_port+0x120/0x1f0 [nvmet]
           nvmet_port_subsys_drop_link+0x261/0x340 [nvmet]
           configfs_unlink+0x36d/0x790
           vfs_unlink+0x278/0xb40
           filename_unlinkat+0x31d/0x670
           __x64_sys_unlinkat+0xae/0x100

    other info that might help us debug this:

     Possible unsafe locking scenario:

           CPU0                    CPU1
           ----                    ----
      lock(nvmet_config_sem);
                                   lock(&id_priv->handler_mutex);
                                   lock(nvmet_config_sem);
      lock(&id_priv->handler_mutex);

     *** DEADLOCK ***

However, the two handler_mutex instances belong to different cm_ids. The
one held during command execution belongs to a connected queue's cm_id,
while the one taken by rdma_destroy_id() belongs to the listening cm_id.
nvmet-rdma does not take nvmet_config_sem while handling events of the
listening cm_id, so this is not a real deadlock. Lockdep cannot
distinguish the two because all handler_mutex instances share a single
lock class.

To avoid the false-positive WARN, assign a separate lockdep class to the
handler_mutex of a listening cm_id at the start of rdma_listen().

Link: https://lore.kernel.org/linux-nvme/at36um4zrsfOr3kh@shinmob/
Fixes: 2a8513091d2f ("nvme-rdma: parallelize I/O queue allocation and startup")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
---
* Changes from RFC v1
- Instead of the new helper function, do the reclassfication in rdma_listen()
- Link to RFC v1: https://lore.kernel.org/linux-rdma/20261108120154.148824-1-shinichiro.kawasaki@wdc.com/

 drivers/infiniband/core/cma.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
index 337a49d1acf7..b3678ddc79a4 100644
--- a/drivers/infiniband/core/cma.c
+++ b/drivers/infiniband/core/cma.c
@@ -3949,6 +3949,13 @@ int rdma_listen(struct rdma_cm_id *id, int backlog)
 		container_of(id, struct rdma_id_private, id);
 	int ret;
 
+#ifdef CONFIG_DEBUG_LOCK_ALLOC
+	static struct lock_class_key cma_listen_handler_mutex_key;
+
+	lockdep_set_class(&id_priv->handler_mutex,
+			  &cma_listen_handler_mutex_key);
+#endif
+
 	if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_LISTEN)) {
 		struct sockaddr_in any_in = {
 			.sin_family = AF_INET,
-- 
2.56.0


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id
  2026-11-17 11:31 [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id Shin'ichiro Kawasaki
@ 2026-11-17 11:48 ` sashiko-bot
  2026-11-18 11:53   ` Shin'ichiro Kawasaki
  0 siblings, 1 reply; 5+ messages in thread
From: sashiko-bot @ 2026-11-17 11:48 UTC (permalink / raw)
  To: Shin'ichiro Kawasaki, linux-rdma; +Cc: linux-nvme

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [Medium] RDMA/cma: unsafe lockdep_set_class() placement in
  rdma_listen()
- [Low] RDMA/cma: unreclassified handler_mutex on
  nvmet_rdma_enable_port() error
--

commit d2c299202b2d013f5516cd16c2fdd187fc66e56a
Author: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>

RDMA/cma: Use a separate lockdep class for the listening cm_id

This commit assigns a separate lockdep class to id_priv->handler_mutex at the
start of rdma_listen() to prevent a false-positive circular locking dependency
warning between listening and connected cm_ids in nvmet-rdma.

Link: https://lore.kernel.org/linux-nvme/at36um4zrsfOr3kh@shinmob/

> diff --git a/drivers/infiniband/core/cma.c b/drivers/infiniband/core/cma.c
> index 337a49d1acf72..b3678ddc79a4a 100644
> --- a/drivers/infiniband/core/cma.c
> +++ b/drivers/infiniband/core/cma.c
> @@ -3949,6 +3949,13 @@ int rdma_listen(struct rdma_cm_id *id, int backlog)
>  		container_of(id, struct rdma_id_private, id);
>  	int ret;
>  
> +#ifdef CONFIG_DEBUG_LOCK_ALLOC
> +	static struct lock_class_key cma_listen_handler_mutex_key;
> +
> +	lockdep_set_class(&id_priv->handler_mutex,
> +			  &cma_listen_handler_mutex_key);
> +#endif
> +
>  	if (!cma_comp_exch(id_priv, RDMA_CM_ADDR_BOUND, RDMA_CM_LISTEN)) {
>  		struct sockaddr_in any_in = {
>  			.sin_family = AF_INET,

[Severity: Medium]
Can calling lockdep_set_class() here race with concurrent users of
id_priv->handler_mutex and misclassify non-listening IDs?

rdma_listen() reinitializes id_priv->handler_mutex.dep_map before validating
id_priv->state and without holding id_priv->handler_mutex.  Through
ucma_listen(), userspace only holds ctx->mutex and can call rdma_listen() on a
cm_id in any state while another thread or CM callback acquires or releases
id_priv->handler_mutex:

CPU0                                  CPU1
ucma_listen()                         cma_ib_handler()
  mutex_lock(&ctx->mutex)               mutex_lock(&id_priv->handler_mutex)
  rdma_listen()                           look_up_lock_class()
    lockdep_set_class(&handler_mutex)     ...  <-- data race on dep_map
    cma_comp_exch() -> fails

Even when id_priv->state is RDMA_CM_ADDR_BOUND, id_priv is already on
cma_dev->id_list, where cma_send_device_removal_put() or addr_handler() can
hold id_priv->handler_mutex concurrently.  Mutating dep_map while the mutex is
being acquired or held can trigger the lockdep WARN_ONCE() in
look_up_lock_class() or corrupt the held-lock stack on unlock and disable
lockdep.

In addition, if rdma_listen() fails the state check, fails rdma_bind_addr(),
or jumps to err: and reverts id_priv->state to RDMA_CM_ADDR_BOUND, a
non-listening cm_id is left with cma_listen_handler_mutex_key, defeating the
lock-class separation if the ID is later used for a connection.

[Severity: Low]
Does reclassifying id_priv->handler_mutex in rdma_listen() still leave the
false-positive lockdep splat reachable when nvmet_rdma_enable_port() fails
before reaching rdma_listen()?

nvmet_port_subsys_allow_link() holds nvmet_config_sem across
nvmet_enable_port() -> nvmet_rdma_add_port() -> nvmet_rdma_enable_port():

drivers/nvme/target/rdma.c:nvmet_rdma_enable_port() {
    ...
	ret = rdma_bind_addr(cm_id, addr);
	if (ret) {
		pr_err("binding CM ID to %pISpcs failed (%d)\n", addr, ret);
		goto out_destroy_id;
	}

	ret = rdma_listen(cm_id, 128);
	if (ret) {
		pr_err("listening to %pISpcs failed (%d)\n", addr, ret);
		goto out_destroy_id;
	}

	port->priv = cm_id;
	return 0;

out_destroy_id:
	rdma_destroy_id(cm_id);
	return ret;
}

If rdma_set_afonly() or rdma_bind_addr() fails, rdma_destroy_id(cm_id) locks
id_priv->handler_mutex under nvmet_config_sem while handler_mutex still has
the default lockdep class, recording the same nvmet_config_sem ->
handler_mutex dependency as before.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261117113129.746490-1-shinichiro.kawasaki@wdc.com?part=1

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id
  2026-11-17 11:48 ` sashiko-bot
@ 2026-11-18 11:53   ` Shin'ichiro Kawasaki
  2026-11-20  2:35     ` Shin'ichiro Kawasaki
  0 siblings, 1 reply; 5+ messages in thread
From: Shin'ichiro Kawasaki @ 2026-11-18 11:53 UTC (permalink / raw)
  To: sashiko-reviews; +Cc: linux-rdma, linux-nvme

On Nov 17, 2026 / 11:48, sashiko-bot@kernel.org wrote:
> Thank you for your contribution! Sashiko AI review found 2 potential
> issues to consider:
> - [Medium] RDMA/cma: unsafe lockdep_set_class() placement in
>   rdma_listen()
> - [Low] RDMA/cma: unreclassified handler_mutex on
>   nvmet_rdma_enable_port() error

I think the two issues are valid. The first one means that the approach
to reclassify ids in rdma_listen() does not work well. I will think about other
solutions. One idea is to go back to the solutions to add a new function to
create rdma id. The other idea is to reclassify the rdma ids not for
listening. I will try if these approach can address the sashiko issues.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id
  2026-11-18 11:53   ` Shin'ichiro Kawasaki
@ 2026-11-20  2:35     ` Shin'ichiro Kawasaki
  2026-11-20  2:43       ` sashiko-bot
  0 siblings, 1 reply; 5+ messages in thread
From: Shin'ichiro Kawasaki @ 2026-11-20  2:35 UTC (permalink / raw)
  To: linux-rdma, linux-nvme, kbusch; +Cc: jgg

On Nov 18, 2026 / 20:53, Shin'ichiro Kawasaki wrote:
> On Nov 17, 2026 / 11:48, sashiko-bot@kernel.org wrote:
> > Thank you for your contribution! Sashiko AI review found 2 potential
> > issues to consider:
> > - [Medium] RDMA/cma: unsafe lockdep_set_class() placement in
> >   rdma_listen()
> > - [Low] RDMA/cma: unreclassified handler_mutex on
> >   nvmet_rdma_enable_port() error
> 
> I think the two issues are valid. The first one means that the approach
> to reclassify ids in rdma_listen() does not work well. I will think about other
> solutions. One idea is to go back to the solutions to add a new function to
> create rdma id. The other idea is to reclassify the rdma ids not for
> listening. I will try if these approach can address the sashiko issues.

While I worked on the new patch to reclassify the rdma ids not for listening
but for other requests, I ran a local Sashiko review on it before posting.
Sashiko made the comment below, quote:

--- start ---
[Severity: Medium]
Is "this is not a real deadlock" true on iWARP?

All iw_cm_id events run on the single ordered iwcm_wq.  This includes the
connection ID's ESTABLISHED event, which goes through cma_iw_handler().
When the listening ID is destroyed, iw_destroy_cm_id() flushes iwcm_wq if
the ID's refcount is above 1.  A CONNECT_REQUEST still queued for that ID
is enough to raise it:

CPU0                                CPU1 (iw_cm_wq)
----                                ----
nvmet_port_subsys_drop_link()
  down_write(&nvmet_config_sem)
                                    cm_work_handler()
                                      cma_iw_handler()
                                        ...
                                          nvmet_execute_io_connect()
                                            down_read(&nvmet_config_sem)
  nvmet_rdma_disable_port()
    rdma_destroy_id(listen)
      ...
        iw_destroy_cm_id()
          flush_workqueue(iwcm_wq)
          <-- waits for CPU1 forever

The two handler_mutex instances are different.  But the destroy path still
waits for the connection ID's event work, so on siw, cxgb4 or irdma the
hang is real.  With the new class, lockdep no longer reports the
handler_mutex cycle here.  It only sees the iw_cm_wq dependency on runs
where the conditional flush actually happens.
--- end ---

My understanding of the comment is that the lockdep WARN that I'm trying to
suppress is not false-positive. To be precise, it is false-positive with rxe
driver, but not with siw driver or other drivers. I wanted to confirm that the
deadlock really happens, then I asked my AI to create a reproducer patch for
v7.4-rc3 kernel [2] and a reproducer script [3]. With these, I confirmed the
hang is recreated for real [4].

The reproducer patch [2] adds three msleep() calls to siw, nvme-rdma and nvmet-
rdma drivers to widen the reace windows. This is somewhat artificial, but I
think it indicates that the users can face the deadlock in real use cases. So
now I think the WARN should not be hidden and should be fixed properly.

Then I no longer think the cm_id handler mutex reclassification approch is
valid. Instaed, I would like to propose again the nvmet-rdma patch which
executes the commands of established queues in a work item outside the CM
handler [1]. I applied this nvmet-rdma patch on top of the reprodcuer patch, and
confirmed the hang does not happen.

Keith, may I ask your thoughts on this? If you think it's fine, I will revisit
the nvmet-rdma patch and repost [1].


[1] https://lore.kernel.org/linux-nvme/20260922124447.588057-1-shinichiro.kawasaki@opensource.wdc.com/


[2] hang reproducer kernel patch

From df6ffbbebda4bd080703701f0c11c4cfaad5247b Mon Sep 17 00:00:00 2001
From: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Date: Thu, 19 Nov 2026 20:13:24 +0900
Subject: [PATCH] REPRO ONLY: delays to reproduce nvmet-rdma hang on iWARP

---
 drivers/infiniband/sw/siw/siw_cm.c |  1 +
 drivers/nvme/host/rdma.c           |  2 ++
 drivers/nvme/target/rdma.c         |  2 ++
 repro-min.sh                       | 31 ++++++++++++++++++++++++++++++
 4 files changed, 36 insertions(+)
 create mode 100755 repro-min.sh

diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c
index ed49818793dd..bea7c7aacd67 100644
--- a/drivers/infiniband/sw/siw/siw_cm.c
+++ b/drivers/infiniband/sw/siw/siw_cm.c
@@ -1738,6 +1738,7 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params)
 		siw_sk_assign_rtr_upcalls(cep);
 	} else {
 		siw_qp_socket_assoc(cep, qp);
+		msleep(300); /* REPRO: let the host send before ESTABLISHED */
 		rv = siw_cm_upcall(cep, IW_CM_EVENT_ESTABLISHED, 0);
 		if (rv)
 			goto error;
diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
index 32189fdfc0f5..e8c55dfad9d5 100644
--- a/drivers/nvme/host/rdma.c
+++ b/drivers/nvme/host/rdma.c
@@ -583,6 +583,8 @@ static int nvme_rdma_alloc_queue(struct nvme_rdma_queue *queue)
 	struct sockaddr *src_addr = NULL;
 	int ret;
 
+	if (idx == 2)
+		msleep(1000); /* REPRO: connect I/O queue 2 later */
 	mutex_init(&queue->queue_lock);
 	if (idx && ctrl->ctrl.max_integrity_segments)
 		queue->pi_support = true;
diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
index f386c388da18..cde0315ae71e 100644
--- a/drivers/nvme/target/rdma.c
+++ b/drivers/nvme/target/rdma.c
@@ -1673,6 +1673,8 @@ static void nvmet_rdma_queue_established(struct nvmet_rdma_queue *queue)
 {
 	unsigned long flags;
 
+	if (queue->host_qid)
+		msleep(10000); /* REPRO: give time to unlink the port */
 	spin_lock_irqsave(&queue->state_lock, flags);
 	if (queue->state != NVMET_RDMA_Q_CONNECTING) {
 		pr_warn("trying to establish a connected queue\n");


[3] reproducer script (repro_min.sh)

#!/bin/bash
# Reproduce nvmet-rdma hang on iWARP (siw), with the REPRO delay patch applied.
IP=192.168.77.1
NQN=testnqn
CFS=/sys/kernel/config/nvmet

echo 30 > /proc/sys/kernel/hung_task_timeout_secs
ip link add dummy0 type dummy
ip addr add $IP/24 dev dummy0
ip link set dummy0 up
modprobe siw
rdma link add siw0 type siw netdev dummy0
modprobe null_blk
modprobe nvmet-rdma
modprobe nvme-rdma

mkdir $CFS/subsystems/$NQN
echo 1 > $CFS/subsystems/$NQN/attr_allow_any_host
mkdir $CFS/subsystems/$NQN/namespaces/1
echo /dev/nullb0 > $CFS/subsystems/$NQN/namespaces/1/device_path
echo 1 > $CFS/subsystems/$NQN/namespaces/1/enable
mkdir $CFS/ports/1
echo rdma > $CFS/ports/1/addr_trtype
echo ipv4 > $CFS/ports/1/addr_adrfam
echo $IP > $CFS/ports/1/addr_traddr
echo 4420 > $CFS/ports/1/addr_trsvcid
ln -s $CFS/subsystems/$NQN $CFS/ports/1/subsystems/$NQN

nvme connect -t rdma -a $IP -s 4420 -n $NQN --nr-io-queues=2 &
sleep 3
rm $CFS/ports/1/subsystems/$NQN	# hangs


[4] reproducer command log with virtme-ng

$ vng -v --user root --cpus 4 --memory 4G -- ./repro-min.sh .
virtme: waiting for virtiofsd to start
virtme: use 'microvm' QEMU architecture
early console in setup code
Probing EDD (edd=off to disable)... ok
[    0.000000][    T0] Linux version 7.4.0-rc3+ (shin@shinexp) (gcc (GCC) 16.2.1 20260819 (Red Hat 16.2.1-2), GNU ld version 2.46.1-1.fc44) #728 SMP PREEMPT_DYNAMIC Mon Nov 16 16:55:23 JST 2026
[    0.000000][    T0] Command line: virtme_hostname=virtme-ng nr_open=2147483584 virtme_link_mods=/home/shin/Linux/linux_2/.virtme_mods/lib/modules/0.0.0 virtme_rw_overlay0=/etc virtme_rw_overlay1=/lib virtme_rw_overlay2=/home virtme_rw_overlay3=/opt virtme_rw_overlay4=/srv virtme_rw_overlay5=/usr virtme_rw_overlay6=/var virtme_rw_overlay7=/tmp virtme_console=ttyS0 console=ttyS0 earlyprintk=serial,ttyS0,115200 panic=-1 virtme.exec=`Li9yZXByby1taW4uc2ggLg==` virtme_chdir=home/shin/Linux/linux_2 debug init=/usr/lib/python3.14/site-packages/virtme/guest/bin/virtme-ng-init
[    0.000000][    T0] x86/CPU: Model not found in latest microcode list
[    0.000000][    T0] BIOS-provided physical RAM map:
[    0.000000][    T0] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff]  System RAM
(snip)
[    9.479199][   T99] virtme-ng-init: waiting for udev to settle
[   10.680848][  T119] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer
[   11.093015][   T99] virtme-ng-init: udev is done
[   11.097349][    T1] virtme-ng-init: starting script
[   11.517605][  T155] SoftiWARP attached
[   11.549884][  T165] dummy0 speed is unknown, defaulting to 1000
[   11.549944][  T165] dummy0 speed is unknown, defaulting to 1000
[   11.551299][  T165] dummy0 speed is unknown, defaulting to 1000
[   11.654536][  T166] null_blk: disk nullb0 created
[   11.655263][  T166] null_blk: module loaded
[   11.831301][  T168] Key type psk registered
[   12.102256][  T181] nvmet: adding nsid 1 to subsystem testnqn
[   12.156417][  T183] iwpm_register_pid: Unable to send a nlmsg (client = 2)
[   12.160751][  T183] nvmet_rdma: enabling port 1 (192.168.77.1:4420)
[   12.648901][  T187] nvmet: Created nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:26b20028-d00f-6e28-37b2-04d4c447b12c.
[   12.661514][  T184] nvme nvme0: creating 2 I/O queues.
[   12.662218][  T184] nvme nvme0: mapped 2/0/0 default/read/poll queues.
[   21.186230][  T191] nvme nvme0: I/O tag 0 (4000) opcode 0x18 (Admin Cmd) QID 0 timeout
[   21.190379][    C0] nvme nvme0: long keepalive RTT (5063 ms)
[   21.190410][    C0] nvme nvme0: failed nvme_keep_alive_end_io error=10
[   23.294680][  T189]
[   23.296090][  T189] ======================================================
[   23.299754][  T189] WARNING: possible circular locking dependency detected
[   23.300509][  T189] 7.4.0-rc3+ #728 Tainted: G            E
[   23.301157][  T189] ------------------------------------------------------
[   23.301858][  T189] kworker/u16:3/189 is trying to acquire lock:
[   23.302450][  T189] ffffffffc124a568 (nvmet_config_sem){++++}-{4:4}, at: nvmet_find_get_subsys+0x1f4/0x630 [nvmet]
[   23.303464][  T189]
[   23.303464][  T189] but task is already holding lock:
[   23.304162][  T189] ffff8880bddf7420 (&id_priv->handler_mutex){+.+.}-{4:4}, at: cma_iw_handler+0xd7/0x5d0 [rdma_cm]
[   23.305177][  T189]
[   23.305177][  T189] which lock already depends on the new lock.
[   23.305177][  T189]
[   23.306156][  T189]
[   23.306156][  T189] the existing dependency chain (in reverse order) is:
[   23.307219][  T189]
[   23.307219][  T189] -> #1 (&id_priv->handler_mutex){+.+.}-{4:4}:
[   23.308289][  T189]        __mutex_lock+0x1a9/0x2550
[   23.308803][  T189]        rdma_destroy_id+0x17/0x20 [rdma_cm]
[   23.309506][  T189]        nvmet_rdma_disable_port+0x44/0x300 [nvmet_rdma]
[   23.310176][  T189]        nvmet_rdma_remove_port+0x42/0x60 [nvmet_rdma]
[   23.310881][  T189]        nvmet_disable_port+0xdc/0x1c0 [nvmet]
[   23.311546][  T189]        nvmet_port_subsys_drop_link+0x261/0x340 [nvmet]
[   23.312240][  T189]        configfs_unlink+0x36d/0x790
[   23.312773][  T189]        vfs_unlink+0x278/0xbc0
[   23.313256][  T189]        filename_unlinkat+0x31d/0x6b0
[   23.313797][  T189]        __x64_sys_unlinkat+0xae/0x100
[   23.314305][  T189]        do_syscall_64+0xf4/0x6b0
[   23.314790][  T189]        entry_SYSCALL_64_after_hwframe+0x76/0x7e
[   23.315421][  T189]
[   23.315421][  T189] -> #0 (nvmet_config_sem){++++}-{4:4}:
[   23.316172][  T189]        __lock_acquire+0xe68/0x2490
[   23.316700][  T189]        lock_acquire+0x1b1/0x340
[   23.317192][  T189]        down_read+0xa1/0x530
[   23.317656][  T189]        nvmet_find_get_subsys+0x1f4/0x630 [nvmet]
[   23.318303][  T189]        nvmet_ctrl_find_get+0xaa/0x320 [nvmet]
[   23.318930][  T189]        nvmet_execute_io_connect+0x1e0/0x3f0 [nvmet]
[   23.319605][  T189]        nvmet_rdma_execute_command+0x184/0x460 [nvmet_rdma]
[   23.320320][  T189]        nvmet_rdma_handle_command+0x733/0xaf0 [nvmet_rdma]
[   23.321032][  T189]        nvmet_rdma_cm_handler+0x930/0x3140 [nvmet_rdma]
[   23.321716][  T189]        cma_cm_event_handler+0xb0/0x3c0 [rdma_cm]
[   23.322357][  T189]        cma_iw_handler+0x31a/0x5d0 [rdma_cm]
[   23.322934][  T189]        cm_work_handler+0x1100/0x1920 [iw_cm]
[   23.323515][  T189]        process_one_work+0x944/0x1760
[   23.324043][  T189]        worker_thread+0x5fd/0xfe0
[   23.324547][  T189]        kthread+0x36d/0x470
[   23.325004][  T189]        ret_from_fork+0x653/0x9d0
[   23.325514][  T189]        ret_from_fork_asm+0x1a/0x30
[   23.326034][  T189]
[   23.326034][  T189] other info that might help us debug this:
[   23.326034][  T189]
[   23.327096][  T189]  Possible unsafe locking scenario:
[   23.327096][  T189]
[   23.327974][  T189]        CPU0                    CPU1
[   23.328585][  T189]        ----                    ----
[   23.329229][  T189]   lock(&id_priv->handler_mutex);
[   23.329844][  T189]                                lock(nvmet_config_sem);
[   23.330671][  T189]                                lock(&id_priv->handler_mutex);
[   23.331550][  T189]   rlock(nvmet_config_sem);
[   23.332082][  T189]
[   23.332082][  T189]  *** DEADLOCK ***
[   23.332082][  T189]
[   23.332967][  T189] locks held by kworker/u16:3/189: 3, on CPU#1:
[   23.333631][  T189]  #0: ffff8881077ca148 ((wq_completion)iw_cm_wq){+.+.}-{0:0}, at: process_one_work+0x1109/0x1760
[   23.334772][  T189]  #1: ffff888004367d10 ((work_completion)(&work->work)#2){+.+.}-{0:0}, at: process_one_work+0x8c0/0x1760
[   23.336779][  T189]  #2: ffff8880bddf7420 (&id_priv->handler_mutex){+.+.}-{4:4}, at: cma_iw_handler+0xd7/0x5d0 [rdma_cm]
[   23.337954][  T189]
[   23.337954][  T189] stack backtrace:
[   23.338590][  T189] CPU: 1 UID: 0 PID: 189 Comm: kworker/u16:3 Tainted: G            E       7.4.0-rc3+ #728 PREEMPT(full)
[   23.339775][  T189] Tainted: [E]=UNSIGNED_MODULE
[   23.340340][  T189] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011
[   23.341335][  T189] Workqueue: iw_cm_wq cm_work_handler [iw_cm]
[   23.342133][  T189] Call Trace:
[   23.342517][  T189]  <TASK>
[   23.342916][  T189]  dump_stack_lvl+0x6a/0x90
[   23.343405][  T189]  print_circular_bug.cold+0x189/0x1eb
[   23.344037][  T189]  check_noncircular+0x173/0x1a0
[   23.344629][  T189]  __lock_acquire+0xe68/0x2490
[   23.345211][  T189]  lock_acquire+0x1b1/0x340
[   23.345996][  T189]  ? nvmet_find_get_subsys+0x1f4/0x630 [nvmet]
[   23.347189][  T189]  ? __pfx___might_resched+0x10/0x10
[   23.347795][  T189]  down_read+0xa1/0x530
[   23.348240][  T189]  ? nvmet_find_get_subsys+0x1f4/0x630 [nvmet]
[   23.349068][  T189]  ? __pfx_down_read+0x10/0x10
[   23.349602][  T189]  ? _raw_spin_unlock_irqrestore+0x35/0x60
[   23.350210][  T189]  ? stack_depot_save_flags+0x918/0xbd0
[   23.350969][  T189]  ? sg_miter_skip+0x19/0x3d0
[   23.351652][  T189]  nvmet_find_get_subsys+0x1f4/0x630 [nvmet]
[   23.352319][  T189]  ? __pfx_nvmet_find_get_subsys+0x10/0x10 [nvmet]
[   23.353058][  T189]  ? sg_copy_buffer+0x19f/0x220
[   23.353650][  T189]  ? __pfx_sg_copy_buffer+0x10/0x10
[   23.354219][  T189]  nvmet_ctrl_find_get+0xaa/0x320 [nvmet]
[   23.354905][  T189]  ? __pfx_nvmet_ctrl_find_get+0x10/0x10 [nvmet]
[   23.355619][  T189]  ? __kmalloc_cache_noprof+0x2ce/0x6d0
[   23.356156][  T189]  nvmet_execute_io_connect+0x1e0/0x3f0 [nvmet]
[   23.356899][  T189]  nvmet_rdma_execute_command+0x184/0x460 [nvmet_rdma]
[   23.357616][  T189]  nvmet_rdma_handle_command+0x733/0xaf0 [nvmet_rdma]
[   23.358315][  T189]  ? lockdep_hardirqs_on_prepare+0xd5/0x1b0
[   23.358947][  T189]  ? trace_hardirqs_on+0x14/0x190
[   23.359500][  T189]  nvmet_rdma_cm_handler+0x930/0x3140 [nvmet_rdma]
[   23.360246][  T189]  ? cma_iw_handler+0xd7/0x5d0 [rdma_cm]
[   23.360844][  T189]  ? lock_acquire+0x1c1/0x340
[   23.361326][  T189]  ? __pfx___might_resched+0x10/0x10
[   23.361968][  T189]  ? rcu_is_watching+0x12/0xb0
[   23.362467][  T189]  ? __pfx_nvmet_rdma_cm_handler+0x10/0x10 [nvmet_rdma]
[   23.363142][  T189]  ? ret_from_fork_asm+0x1a/0x30
[   23.363711][  T189]  ? cma_iw_handler+0xd7/0x5d0 [rdma_cm]
[   23.364345][  T189]  ? __pfx___mutex_lock+0x10/0x10
[   23.364893][  T189]  ? register_lock_class+0x4a/0x5a0
[   23.365489][  T189]  ? lock_is_held_type+0xf6/0x1b0
[   23.366005][  T189]  cma_cm_event_handler+0xb0/0x3c0 [rdma_cm]
[   23.366683][  T189]  cma_iw_handler+0x31a/0x5d0 [rdma_cm]
[   23.367225][  T189]  ? __pfx_cma_iw_handler+0x10/0x10 [rdma_cm]
[   23.367914][  T189]  ? lockdep_hardirqs_on_prepare+0xd5/0x1b0
[   23.368574][  T189]  ? trace_hardirqs_on+0x14/0x190
[   23.369143][  T189]  cm_work_handler+0x1100/0x1920 [iw_cm]
[   23.369759][  T189]  ? __pfx_cm_work_handler+0x10/0x10 [iw_cm]
[   23.370397][  T189]  ? lock_acquire+0x1c1/0x340
[   23.370864][  T189]  ? process_one_work+0x7e0/0x1760
[   23.371366][  T189]  process_one_work+0x944/0x1760
[   23.371859][  T189]  ? __pfx_process_one_work+0x10/0x10
[   23.372384][  T189]  ? rcuwait_wake_up+0x1e/0x190
[   23.372873][  T189]  worker_thread+0x5fd/0xfe0
[   23.373331][  T189]  ? __pfx_worker_thread+0x10/0x10
[   23.373829][  T189]  kthread+0x36d/0x470
[   23.374231][  T189]  ? __pfx_kthread+0x10/0x10
[   23.374687][  T189]  ret_from_fork+0x653/0x9d0
[   23.375140][  T189]  ? __pfx_ret_from_fork+0x10/0x10
[   23.375644][  T189]  ? __switch_to+0x3a9/0xd90
[   23.376101][  T189]  ? __switch_to_asm+0x39/0x70
[   23.376571][  T189]  ? __switch_to_asm+0x33/0x70
[   23.377039][  T189]  ? __pfx_kthread+0x10/0x10
[   23.377496][  T189]  ret_from_fork_asm+0x1a/0x30
[   23.377969][  T189]  </TASK>
[   43.265793][  T188] nvme nvme0: I/O tag 0 (0000) opcode 0x7f (Fabrics Cmd) QID 1 timeout
[   43.265925][   T50] nvme nvme0: Connect command failed, error wo/DNR bit: 881
[   43.266970][   T50] nvme nvme0: failed to connect queue: 1 ret=881
[   61.182016][   T48] INFO: task kworker/u16:1:14 blocked for more than 30 seconds.
[   61.182047][   T48]       Tainted: G            E       7.4.0-rc3+ #728
[   61.182058][   T48] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   61.182074][   T48] task:kworker/u16:1   state:D stack:0     pid:14    tgid:14    ppid:2      task_flags:0x4208060 flags:0x00080000
[   61.182104][   T48] Workqueue: nvmet-wq nvmet_rdma_release_queue_work [nvmet_rdma]
[   61.182149][   T48] Call Trace:
[   61.182158][   T48]  <TASK>
[   61.182176][   T48]  __schedule+0x1dbe/0x7a90
[   61.182233][   T48]  ? lock_acquire+0x1b1/0x340
[   61.182255][   T48]  ? __pfx___schedule+0x10/0x10
[   61.182279][   T48]  ? schedule+0x24a/0x2e0
[   61.182295][   T48]  ? lock_release+0x1b5/0x340
[   61.182316][   T48]  schedule+0xe3/0x2e0
[   61.182337][   T48]  schedule_timeout+0x1ac/0x290
[   61.182356][   T48]  ? __pfx_schedule_timeout+0x10/0x10
[   61.182392][   T48]  ? lockdep_hardirqs_on_prepare+0xd5/0x1b0
[   61.182406][   T48]  ? trace_hardirqs_on+0x14/0x190
[   61.182428][   T48]  wait_for_completion+0x17a/0x400
[   61.182452][   T48]  ? __pfx_wait_for_completion+0x10/0x10
[   61.182470][   T48]  ? touch_wq_lockdep_map+0x87/0x1c0
[   61.182503][   T48]  __flush_workqueue+0x3a9/0x10c0
[   61.182521][   T48]  ? do_raw_spin_lock+0x12d/0x280
[   61.182554][   T48]  ? __pfx___flush_workqueue+0x10/0x10
[   61.182570][   T48]  ? _raw_spin_unlock_irqrestore+0x4c/0x60
[   61.182591][   T48]  ? siw_qp_put_ref+0x1e/0x60 [siw]
[   61.182763][   T48]  ? _raw_spin_unlock_irqrestore+0x4c/0x60
[   61.182796][   T48]  iw_destroy_cm_id+0x5e/0x80 [iw_cm]
[   61.182836][   T48]  _destroy_id+0x657/0xcf0 [rdma_cm]
[   61.182954][   T48]  nvmet_rdma_free_queue+0x48b/0x9d0 [nvmet_rdma]
[   61.182984][   T48]  ? __lock_acquire+0x1297/0x2490
[   61.183020][   T48]  ? __pfx_nvmet_rdma_free_queue+0x10/0x10 [nvmet_rdma]
[   61.183050][   T48]  ? lock_acquire+0x1c1/0x340
[   61.183076][   T48]  nvmet_rdma_release_queue_work+0x3e/0x90 [nvmet_rdma]
[   61.183106][   T48]  process_one_work+0x944/0x1760
[   61.183148][   T48]  ? __pfx_process_one_work+0x10/0x10
[   61.183168][   T48]  ? lock_acquire+0x1b1/0x340
[   61.183210][   T48]  worker_thread+0x5fd/0xfe0
[   61.183250][   T48]  ? __pfx_worker_thread+0x10/0x10
[   61.183267][   T48]  kthread+0x36d/0x470
[   61.183286][   T48]  ? __pfx_kthread+0x10/0x10
[   61.183304][   T48]  ret_from_fork+0x653/0x9d0
[   61.183329][   T48]  ? __pfx_ret_from_fork+0x10/0x10
[   61.183352][   T48]  ? __switch_to+0x3a9/0xd90
[   61.183367][   T48]  ? __switch_to_asm+0x39/0x70
[   61.183381][   T48]  ? __switch_to_asm+0x33/0x70
[   61.183394][   T48]  ? __pfx_kthread+0x10/0x10
[   61.183413][   T48]  ret_from_fork_asm+0x1a/0x30
[   61.183456][   T48]  </TASK>
[   61.183505][   T48] INFO: task kworker/u16:3:189 blocked for more than 30 seconds.
[   61.183520][   T48]       Tainted: G            E       7.4.0-rc3+ #728
[   61.183529][   T48] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   61.183536][   T48] task:kworker/u16:3   state:D stack:0     pid:189   tgid:189   ppid:2      task_flags:0x4208060 flags:0x00080000
[   61.183560][   T48] Workqueue: iw_cm_wq cm_work_handler [iw_cm]
[   61.183598][   T48] Call Trace:
[   61.183605][   T48]  <TASK>
[   61.183619][   T48]  __schedule+0x1dbe/0x7a90
[   61.183643][   T48]  ? rcu_is_watching+0x12/0xb0
[   61.183721][   T48]  ? nbcon_cpu_emergency_exit+0xd7/0x160
[   61.183752][   T48]  ? __printk_cpu_sync_put+0x31/0x60
[   61.183777][   T48]  ? __pfx_nbcon_cpu_emergency_exit+0x10/0x10
[   61.183795][   T48]  ? __pfx___schedule+0x10/0x10
[   61.183816][   T48]  ? lock_acquire+0x2f5/0x340
[   61.183828][   T48]  ? rcu_is_watching+0x12/0xb0
[   61.183846][   T48]  ? lock_release+0x285/0x340
[   61.183867][   T48]  schedule+0xe3/0x2e0
[   61.183886][   T48]  ? rwsem_down_read_slowpath+0x31e/0x1120
[   61.183934][   T48]  schedule_preempt_disabled+0x14/0x30
[   61.183953][   T48]  rwsem_down_read_slowpath+0x61e/0x1120
[   61.183968][   T48]  ? asm_sysvec_apic_timer_interrupt+0x16/0x20
[   61.183993][   T48]  ? __pfx_rwsem_down_read_slowpath+0x10/0x10
[   61.184028][   T48]  ? __pfx___might_resched+0x10/0x10
[   61.184058][   T48]  down_read+0xf5/0x530
[   61.184077][   T48]  ? __pfx_down_read+0x10/0x10
[   61.184091][   T48]  ? _raw_spin_unlock_irqrestore+0x35/0x60
[   61.184110][   T48]  ? stack_depot_save_flags+0x918/0xbd0
[   61.184131][   T48]  ? sg_miter_skip+0x19/0x3d0
[   61.184154][   T48]  nvmet_find_get_subsys+0x1f4/0x630 [nvmet]
[   61.184265][   T48]  ? __pfx_nvmet_find_get_subsys+0x10/0x10 [nvmet]
[   61.184366][   T48]  ? sg_copy_buffer+0x19f/0x220
[   61.184384][   T48]  ? __pfx_sg_copy_buffer+0x10/0x10
[   61.184409][   T48]  nvmet_ctrl_find_get+0xaa/0x320 [nvmet]
[   61.184523][   T48]  ? __pfx_nvmet_ctrl_find_get+0x10/0x10 [nvmet]
[   61.184741][   T48]  ? __kmalloc_cache_noprof+0x2ce/0x6d0
[   61.184778][   T48]  nvmet_execute_io_connect+0x1e0/0x3f0 [nvmet]
[   61.184919][   T48]  nvmet_rdma_execute_command+0x184/0x460 [nvmet_rdma]
[   61.184958][   T48]  nvmet_rdma_handle_command+0x733/0xaf0 [nvmet_rdma]
[   61.184988][   T48]  ? lockdep_hardirqs_on_prepare+0xd5/0x1b0
[   61.185004][   T48]  ? trace_hardirqs_on+0x14/0x190
[   61.185027][   T48]  nvmet_rdma_cm_handler+0x930/0x3140 [nvmet_rdma]
[   61.185062][   T48]  ? cma_iw_handler+0xd7/0x5d0 [rdma_cm]
[   61.185141][   T48]  ? lock_acquire+0x1c1/0x340
[   61.185157][   T48]  ? __pfx___might_resched+0x10/0x10
[   61.185182][   T48]  ? rcu_is_watching+0x12/0xb0
[   61.185203][   T48]  ? __pfx_nvmet_rdma_cm_handler+0x10/0x10 [nvmet_rdma]
[   61.185232][   T48]  ? ret_from_fork_asm+0x1a/0x30
[   61.185246][   T48]  ? cma_iw_handler+0xd7/0x5d0 [rdma_cm]
[   61.185328][   T48]  ? __pfx___mutex_lock+0x10/0x10
[   61.185347][   T48]  ? register_lock_class+0x4a/0x5a0
[   61.185377][   T48]  ? lock_is_held_type+0xf6/0x1b0
[   61.185405][   T48]  cma_cm_event_handler+0xb0/0x3c0 [rdma_cm]
[   61.185486][   T48]  cma_iw_handler+0x31a/0x5d0 [rdma_cm]
[   61.185566][   T48]  ? __pfx_cma_iw_handler+0x10/0x10 [rdma_cm]
[   61.185659][   T48]  ? lockdep_hardirqs_on_prepare+0xd5/0x1b0
[   61.185710][   T48]  ? trace_hardirqs_on+0x14/0x190
[   61.185735][   T48]  cm_work_handler+0x1100/0x1920 [iw_cm]
[   61.185779][   T48]  ? __pfx_cm_work_handler+0x10/0x10 [iw_cm]
[   61.185861][   T48]  ? lock_acquire+0x1c1/0x340
[   61.185880][   T48]  ? process_one_work+0x7e0/0x1760
[   61.185927][   T48]  process_one_work+0x944/0x1760
[   61.185968][   T48]  ? __pfx_process_one_work+0x10/0x10
[   61.185990][   T48]  ? rcuwait_wake_up+0x1e/0x190
[   61.186032][   T48]  worker_thread+0x5fd/0xfe0
[   61.186072][   T48]  ? __pfx_worker_thread+0x10/0x10
[   61.186089][   T48]  kthread+0x36d/0x470
[   61.186106][   T48]  ? __pfx_kthread+0x10/0x10
[   61.186125][   T48]  ret_from_fork+0x653/0x9d0
[   61.186147][   T48]  ? __pfx_ret_from_fork+0x10/0x10
[   61.186170][   T48]  ? __switch_to+0x3a9/0xd90
[   61.186184][   T48]  ? __switch_to_asm+0x39/0x70
[   61.186197][   T48]  ? __switch_to_asm+0x33/0x70
[   61.186211][   T48]  ? __pfx_kthread+0x10/0x10
[   61.186229][   T48]  ret_from_fork_asm+0x1a/0x30
[   61.186271][   T48]  </TASK>
[   61.186294][   T48] INFO: task kworker/u16:3:189 <reader> blocked on an rw-semaphore likely owned by task rm:194 <writer>
[   61.186311][   T48] INFO: task rm:194 blocked for more than 30 seconds.
[   61.186324][   T48]       Tainted: G            E       7.4.0-rc3+ #728
[   61.186333][   T48] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[   61.186339][   T48] task:rm              state:D stack:0     pid:194   tgid:194   ppid:149    task_flags:0x400000 flags:0x00080000
[   61.186361][   T48] Call Trace:
[   61.186368][   T48]  <TASK>
[   61.186382][   T48]  __schedule+0x1dbe/0x7a90
[   61.186427][   T48]  ? lock_acquire+0x1b1/0x340
[   61.186445][   T48]  ? __pfx___schedule+0x10/0x10
[   61.186461][   T48]  ? lock_acquire+0x1c1/0x340
[   61.186479][   T48]  ? schedule+0x24a/0x2e0
[   61.186494][   T48]  ? lock_release+0x1b5/0x340
[   61.186515][   T48]  schedule+0xe3/0x2e0
[   61.186535][   T48]  schedule_timeout+0x1ac/0x290
[   61.186553][   T48]  ? __pfx_schedule_timeout+0x10/0x10
[   61.186589][   T48]  ? lockdep_hardirqs_on_prepare+0xd5/0x1b0
[   61.186603][   T48]  ? trace_hardirqs_on+0x14/0x190
[   61.186623][   T48]  wait_for_completion+0x17a/0x400
[   61.186647][   T48]  ? __pfx_wait_for_completion+0x10/0x10
[   61.186665][   T48]  ? touch_wq_lockdep_map+0x87/0x1c0
[   61.186744][   T48]  __flush_workqueue+0x3a9/0x10c0
[   61.186763][   T48]  ? do_raw_spin_lock+0x12d/0x280
[   61.186792][   T48]  ? lockdep_hardirqs_on_prepare+0xd5/0x1b0
[   61.186808][   T48]  ? __pfx___flush_workqueue+0x10/0x10
[   61.186823][   T48]  ? lockdep_hardirqs_on+0x91/0x140
[   61.186840][   T48]  ? _raw_spin_unlock_irqrestore+0x4c/0x60
[   61.186857][   T48]  ? _raw_spin_unlock_irqrestore+0x35/0x60
[   61.186914][   T48]  ? mark_held_locks+0x40/0x70
[   61.186942][   T48]  iw_destroy_cm_id+0x5e/0x80 [iw_cm]
[   61.186979][   T48]  _destroy_id+0x657/0xcf0 [rdma_cm]
[   61.187071][   T48]  nvmet_rdma_disable_port+0x44/0x300 [nvmet_rdma]
[   61.187112][   T48]  nvmet_rdma_remove_port+0x42/0x60 [nvmet_rdma]
[   61.187141][   T48]  nvmet_disable_port+0xdc/0x1c0 [nvmet]
[   61.187248][   T48]  nvmet_port_subsys_drop_link+0x261/0x340 [nvmet]
[   61.187355][   T48]  ? simple_unlink+0xae/0x100
[   61.187400][   T48]  ? do_raw_spin_unlock+0x147/0x220
[   61.187403][   T48]  ? __pfx_nvmet_port_subsys_drop_link+0x10/0x10 [nvmet]
[   61.187421][   T48]  configfs_unlink+0x36d/0x790
[   61.187428][   T48]  vfs_unlink+0x278/0xbc0
[   61.187437][   T48]  filename_unlinkat+0x31d/0x6b0
[   61.187441][   T48]  ? __pfx_filename_unlinkat+0x10/0x10
[   61.187444][   T48]  ? __virt_addr_valid+0x228/0x500
[   61.187453][   T48]  ? __check_object_size+0x45a/0x620
[   61.187463][   T48]  __x64_sys_unlinkat+0xae/0x100
[   61.187467][   T48]  do_syscall_64+0xf4/0x6b0
[   61.187471][   T48]  ? exc_page_fault+0x94/0x140
[   61.187474][   T48]  ? lock_release+0x1b5/0x340
[   61.187477][   T48]  ? rcu_is_watching+0x12/0xb0
[   61.187480][   T48]  ? do_syscall_64+0x2d/0x6b0
[   61.187483][   T48]  ? trace_hardirqs_off_finish+0x139/0x180
[   61.187486][   T48]  ? lockdep_hardirqs_off+0xaf/0xf0
[   61.187489][   T48]  ? do_syscall_64+0x60/0x6b0
[   61.187491][   T48]  ? clear_bhb_loop+0x30/0x80
[   61.187495][   T48]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
[   61.187498][   T48] RIP: 0033:0x7f63b6a93e8b
[   61.187501][   T48] RSP: 002b:00007ffdb9dc50d8 EFLAGS: 00000246 ORIG_RAX: 0000000000000107
[   61.187508][   T48] RAX: ffffffffffffffda RBX: 000055768b2ad310 RCX: 00007f63b6a93e8b
[   61.187510][   T48] RDX: 0000000000000000 RSI: 000055768b2ac0f0 RDI: 00000000ffffff9c
[   61.187511][   T48] RBP: 00007ffdb9dc51a0 R08: 0000000000000003 R09: 00007ffdb9dc51ec
[   61.187513][   T48] R10: 0000000000000000 R11: 0000000000000246 R12: 000055768b2ac060
[   61.187515][   T48] R13: 0000000000000000 R14: 00007ffdb9dc51f0 R15: 0000000000000002
[   61.187521][   T48]  </TASK>
[   61.187523][   T48] INFO: lockdep is turned off.
[   91.389938][   T48] INFO: task kworker/u16:1:14 blocked for more than 60 seconds.
[   91.389969][   T48]       Tainted: G            E       7.4.0-rc3+ #728
[   91.389981][   T48] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
(snip)


^ permalink raw reply related	[flat|nested] 5+ messages in thread

* Re: [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id
  2026-11-20  2:35     ` Shin'ichiro Kawasaki
@ 2026-11-20  2:43       ` sashiko-bot
  0 siblings, 0 replies; 5+ messages in thread
From: sashiko-bot @ 2026-11-20  2:43 UTC (permalink / raw)
  To: Shin'ichiro Kawasaki, linux-rdma, linux-nvme

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] RDMA/cma: missing Signed-off-by trailer in commit message
- [High] RDMA: debug msleep() in siw_accept(), nvme_rdma_alloc_queue(),
  and nvmet_rdma
--

commit e60962bf8161cfd3de3b001dc6bdc13944ed9245
Author: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>

> RDMA/cma: Use a separate lockdep class for the listening cm_id
>
> On Nov 18, 2026 / 20:53, Shin'ichiro Kawasaki wrote:
> > On Nov 17, 2026 / 11:48, sashiko-bot@kernel.org wrote:
> > > Thank you for your contribution! Sashiko AI review found 2 potential
> > > issues to consider:
> > > - [Medium] RDMA/cma: unsafe lockdep_set_class() placement in
> > >   rdma_listen()
> > > - [Low] RDMA/cma: unreclassified handler_mutex on
> > >   nvmet_rdma_enable_port() error
> >
> > I think the two issues are valid. The first one means that the approach
> > to reclassify ids in rdma_listen() does not work well. I will think about other
> > solutions. One idea is to go back to the solutions to add a new function to
> > create rdma id. The other idea is to reclassify the rdma ids not for
> > listening. I will try if these approach can address the sashiko issues.
>
> While I worked on the new patch to reclassify the rdma ids not for listening
> but for other requests, I ran a local Sashiko review on it before posting.
> Sashiko made the comment below, quote:

[Severity: High]
The commit message is missing a Signed-off-by: trailer, which is required
under the Developer's Certificate of Origin.

> diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c
> index ed49818793dde..bea7c7aacd678 100644
> --- a/drivers/infiniband/sw/siw/siw_cm.c
> +++ b/drivers/infiniband/sw/siw/siw_cm.c
> @@ -1738,6 +1738,7 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params)
>  		siw_sk_assign_rtr_upcalls(cep);
>  	} else {
>  		siw_qp_socket_assoc(cep, qp);
> +		msleep(300); /* REPRO: let the host send before ESTABLISHED */
>  		rv = siw_cm_upcall(cep, IW_CM_EVENT_ESTABLISHED, 0);
>  		if (rv)
>  			goto error;
> diff --git a/drivers/nvme/host/rdma.c b/drivers/nvme/host/rdma.c
> index 32189fdfc0f51..e8c55dfad9d54 100644
> --- a/drivers/nvme/host/rdma.c
> +++ b/drivers/nvme/host/rdma.c
> @@ -583,6 +583,8 @@ static int nvme_rdma_alloc_queue(struct nvme_rdma_queue *queue)
>  	struct sockaddr *src_addr = NULL;
>  	int ret;
>  
> +	if (idx == 2)
> +		msleep(1000); /* REPRO: connect I/O queue 2 later */
>  	mutex_init(&queue->queue_lock);
>  	if (idx && ctrl->ctrl.max_integrity_segments)
>  		queue->pi_support = true;
> diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c
> index f386c388da188..cde0315ae71e4 100644
> --- a/drivers/nvme/target/rdma.c
> +++ b/drivers/nvme/target/rdma.c
> @@ -1673,6 +1673,8 @@ static void nvmet_rdma_queue_established(struct nvmet_rdma_queue *queue)
>  {
>  	unsigned long flags;
>  
> +	if (queue->host_qid)
> +		msleep(10000); /* REPRO: give time to unlink the port */
>  	spin_lock_irqsave(&queue->state_lock, flags);
>  	if (queue->state != NVMET_RDMA_Q_CONNECTING) {
>  		pr_warn("trying to establish a connected queue\n");

[Severity: High]
Were these debug reproducer msleep() calls committed by mistake instead of
the lockdep class change described in the subject?

In nvmet_rdma_queue_established(), sleeping for 10 seconds on every I/O
queue runs inside the RDMA_CM_EVENT_ESTABLISHED callback under
id_priv->handler_mutex on the single-threaded ordered iwcm_wq workqueue.
This stalls CM event processing and rdma_destroy_id() for 10 seconds per
I/O queue, exceeds the default 5-second NVMe admin keep-alive timeout, and
forces open the nvmet_config_sem deadlock window.  siw_accept() and
nvme_rdma_alloc_queue() likewise add unconditional 300 ms and 1000 ms
delays during connection establishment.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/av-vJXx2CZppcnAW@shinmob?part=1

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2026-11-20  2:43 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-17 11:31 [PATCH v2] RDMA/cma: Use a separate lockdep class for the listening cm_id Shin'ichiro Kawasaki
2026-11-17 11:48 ` sashiko-bot
2026-11-18 11:53   ` Shin'ichiro Kawasaki
2026-11-20  2:35     ` Shin'ichiro Kawasaki
2026-11-20  2:43       ` sashiko-bot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox