blktests failures with v7.3-rc1 kernel

[Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>]

Hi all,

I ran the latest blktests (git hash: ed675852b012) with the v7.3-rc1 kernel. I
observed 2 failure classes listed below. Compared with the previous report for
v7.2 kernel [1], 3 failures are no longer observed (nvme/061, nvme/068 and
nvme/069), while 1 new failure class was observed: Sporadic
cleanup_srcu_struct() WARNs.

[1] https://lore.kernel.org/linux-block/aoRGbVB70YCcxoxF@shinhome/


List of failures
================
#1: nvme/062 (tcp transport)
#2: Sporadic cleanup_srcu_struct() WARNs (rdma transport and siw driver)


Failure description
===================

#1: nvme/062 (tcp transport)

    The test case nvme/062 fails for tcp transport due to the lockdep WARN
    related to the three locks root->kernfs_rwsem, sparse_irq_lock and
    root->kernfs_supers_rwsem [2]. A fix patch from Liu Xixin is queued for
    v7.3-rcX kernel [3].

    [2] https://lore.kernel.org/linux-block/akryKgCjof_7d9na@shinmob/
    [3] https://lore.kernel.org/linux-nvme/c6e2a91f0b62.1787100300.git.liuxixin@kylinos.cn/

#2: Sporadic cleanup_srcu_struct() WARNs
    (observed with rdma transport and siw driver)

    Test cases in the nvme test group fail sporadically with the same WARN in
    cleanup_srcu_struct() [4]. I observed the WARN at nvme/025, nvme/031,
    nvme/043 and nvme/060. A fix patch for the WARN has been posted [5]. With
    this patch applied, I no longer observe the failure.

    [5] https://lore.kernel.org/lkml/20260830095605.3560267-1-shpark061104@gmail.com/


[4] example dmesg of the nvme test group failure with rdma transport

Sep 01 10:13:02 testnode2 unknown: run blktests nvme/025 at 2026-09-01 10:13:02
Sep 01 10:13:02 testnode2 kernel: loop0: detected capacity change from 0 to 2097152
Sep 01 10:13:02 testnode2 kernel: nvmet: adding nsid 1 to subsystem blktests-subsystem-1
Sep 01 10:13:02 testnode2 kernel: nvmet: Created nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
Sep 01 10:13:02 testnode2 kernel: nvme nvme5: creating 4 I/O queues.
Sep 01 10:13:02 testnode2 kernel: nvme nvme5: new ctrl: "blktests-subsystem-1"
Sep 01 10:13:03 testnode2 kernel: nvme nvme5: Removing ctrl: NQN "blktests-subsystem-1"
Sep 01 10:13:10 testnode2 kernel: SoftiWARP attached
Sep 01 10:13:10 testnode2 unknown: run blktests nvme/025 at 2026-09-01 10:13:10
Sep 01 10:13:10 testnode2 kernel: loop0: detected capacity change from 0 to 2097152
Sep 01 10:13:10 testnode2 kernel: nvmet: adding nsid 1 to subsystem blktests-subsystem-1
Sep 01 10:13:10 testnode2 kernel: nvmet_rdma: enabling port 0 (10.0.2.15:4420)
Sep 01 10:13:10 testnode2 kernel: nvmet: Created nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
Sep 01 10:13:10 testnode2 kernel: nvme nvme5: creating 4 I/O queues.
Sep 01 10:13:10 testnode2 kernel: nvme nvme5: mapped 4/0/0 default/read/poll queues.
Sep 01 10:13:10 testnode2 kernel: nvme nvme5: new ctrl: NQN "blktests-subsystem-1", addr 10.0.2.15:4420, hostnqn: nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349
Sep 01 10:13:11 testnode2 kernel: nvme nvme5: Removing ctrl: NQN "blktests-subsystem-1"
Sep 01 10:13:11 testnode2 kernel: ------------[ cut here ]------------
Sep 01 10:13:11 testnode2 kernel: WARNING: kernel/rcu/srcutree.c:706 at cleanup_srcu_struct+0x3e4/0xb00, CPU#1: nvme/55536
Sep 01 10:13:11 testnode2 kernel: Modules linked in: siw nvmet_rdma nvmet nvme_rdma nvme_fabrics rdma_cm loop iw_cm ib_cm ib_core nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr sunrpc 9pnet_virtio pcspkr 9pnet i2c_piix4 netfs i2c_smbus zram vmw_vsock_virtio_transport vmw_vsock_virtio_transport_common vsock bochs drm_client_lib drm_shmem_helper xfs drm_kms_helper nvme sym53c8xx nvme_core drm e1000 nvme_keyring scsi_transport_spi nvme_auth floppy serio_raw ata_generic pata_acpi dm_multipath i2c_dev qemu_fw_cfg virtiofs fuse virtio_console [last unloaded: nvmet]
Sep 01 10:13:11 testnode2 kernel: CPU: 1 UID: 0 PID: 55536 Comm: nvme Not tainted 7.3.0-rc1 #675 PREEMPT(full) 
Sep 01 10:13:11 testnode2 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025
Sep 01 10:13:11 testnode2 kernel: RIP: 0010:cleanup_srcu_struct+0x3e4/0xb00
Sep 01 10:13:11 testnode2 kernel: Code: e8 31 d9 90 00 48 89 e8 48 8b 54 24 18 48 c1 e8 03 42 80 3c 20 00 0f 85 0e 06 00 00 49 83 be e8 00 00 00 00 0f 84 e6 fe ff ff <0f> 0b 49 8d 8e 00 02 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 cf
Sep 01 10:13:11 testnode2 kernel: RSP: 0018:ffff8881aca77a68 EFLAGS: 00010202
Sep 01 10:13:11 testnode2 kernel: RAX: 1ffffd1ffff89c3d RBX: 0000000000000000 RCX: 0000000000000001
Sep 01 10:13:11 testnode2 kernel: RDX: ffffe8ffffc4e280 RSI: 0000000000000008 RDI: ffffe8ffffc4e1e8
Sep 01 10:13:11 testnode2 kernel: RBP: ffffe8ffffc4e1e8 R08: ffffffffa078a1df R09: 1ffffd1ffff89c3d
Sep 01 10:13:11 testnode2 kernel: R10: fffff91ffff89c3e R11: fffff91ffff89c3e R12: dffffc0000000000
Sep 01 10:13:11 testnode2 kernel: R13: fffffbfff4a57cf7 R14: ffffe8ffffc4e100 R15: ffff88818c174118
Sep 01 10:13:11 testnode2 kernel: FS:  00007f5041375780(0000) GS:ffff8883ea987000(0000) knlGS:0000000000000000
Sep 01 10:13:11 testnode2 kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Sep 01 10:13:11 testnode2 kernel: CR2: 0000558b7967fd5c CR3: 0000000127690000 CR4: 00000000000006f0
Sep 01 10:13:11 testnode2 kernel: Call Trace:
Sep 01 10:13:11 testnode2 kernel:  <TASK>
Sep 01 10:13:11 testnode2 kernel:  blk_mq_free_tag_set+0x2ec/0x4d0
Sep 01 10:13:11 testnode2 kernel:  nvme_rdma_shutdown_ctrl+0x169/0x1a0 [nvme_rdma]
Sep 01 10:13:11 testnode2 kernel:  nvme_do_delete_ctrl+0x14a/0x160 [nvme_core]
Sep 01 10:13:11 testnode2 kernel:  nvme_delete_ctrl_sync.cold+0x8/0xd [nvme_core]
Sep 01 10:13:11 testnode2 kernel:  nvme_sysfs_delete+0xb7/0xe0 [nvme_core]
Sep 01 10:13:11 testnode2 kernel:  ? __pfx_sysfs_kf_write+0x10/0x10
Sep 01 10:13:11 testnode2 kernel:  kernfs_fop_write_iter+0x3d6/0x5e0
Sep 01 10:13:11 testnode2 kernel:  ? __pfx_kernfs_fop_write_iter+0x10/0x10
Sep 01 10:13:11 testnode2 kernel:  vfs_write+0x4b3/0xf40
Sep 01 10:13:11 testnode2 kernel:  ? __pfx_vfs_write+0x10/0x10
Sep 01 10:13:11 testnode2 kernel:  ? trace_hardirqs_on+0x14/0x190
Sep 01 10:13:11 testnode2 kernel:  ? do_sys_openat2+0xff/0x170
Sep 01 10:13:11 testnode2 kernel:  ksys_write+0x112/0x250
Sep 01 10:13:11 testnode2 kernel:  ? __pfx_ksys_write+0x10/0x10
Sep 01 10:13:11 testnode2 kernel:  ? rcu_is_watching+0x12/0xb0
Sep 01 10:13:11 testnode2 kernel:  do_syscall_64+0xf2/0x690
Sep 01 10:13:11 testnode2 kernel:  ? __pfx___x64_sys_openat+0x10/0x10
Sep 01 10:13:11 testnode2 kernel:  ? rcu_is_watching+0x12/0xb0
Sep 01 10:13:11 testnode2 kernel:  ? do_syscall_64+0x1ff/0x690
Sep 01 10:13:11 testnode2 kernel:  ? trace_hardirqs_on_prepare+0x139/0x180
Sep 01 10:13:11 testnode2 kernel:  ? lockdep_hardirqs_on+0x91/0x140
Sep 01 10:13:11 testnode2 kernel:  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
Sep 01 10:13:11 testnode2 kernel:  ? do_syscall_64+0x21d/0x690
Sep 01 10:13:11 testnode2 kernel:  ? rcu_is_watching+0x12/0xb0
Sep 01 10:13:11 testnode2 kernel:  ? do_syscall_64+0x1ff/0x690
Sep 01 10:13:11 testnode2 kernel:  ? trace_hardirqs_on_prepare+0x139/0x180
Sep 01 10:13:11 testnode2 kernel:  ? lockdep_hardirqs_on+0x91/0x140
Sep 01 10:13:11 testnode2 kernel:  ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
Sep 01 10:13:11 testnode2 kernel:  ? do_syscall_64+0x21d/0x690
Sep 01 10:13:11 testnode2 kernel:  ? do_syscall_64+0x1ff/0x690
Sep 01 10:13:11 testnode2 kernel:  ? rcu_is_watching+0x12/0xb0
Sep 01 10:13:11 testnode2 kernel:  ? do_syscall_64+0x2d/0x690
Sep 01 10:13:11 testnode2 kernel:  ? trace_hardirqs_off_finish+0x139/0x180
Sep 01 10:13:11 testnode2 kernel:  ? lockdep_hardirqs_off+0xaf/0xf0
Sep 01 10:13:11 testnode2 kernel:  ? do_syscall_64+0x60/0x690
Sep 01 10:13:11 testnode2 kernel:  entry_SYSCALL_64_after_hwframe+0x76/0x7e
Sep 01 10:13:11 testnode2 kernel: RIP: 0033:0x7f504141a08e
Sep 01 10:13:11 testnode2 kernel: Code: 4d 89 d8 e8 94 bd 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 <c9> c3 83 e2 39 83 fa 08 75 e7 e8 03 ff ff ff 0f 1f 00 f3 0f 1e fa
Sep 01 10:13:11 testnode2 kernel: RSP: 002b:00007ffec0901290 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
Sep 01 10:13:11 testnode2 kernel: RAX: ffffffffffffffda RBX: 00007f50415ef006 RCX: 00007f504141a08e
Sep 01 10:13:11 testnode2 kernel: RDX: 0000000000000001 RSI: 00007f50415ef006 RDI: 0000000000000003
Sep 01 10:13:11 testnode2 kernel: RBP: 00007ffec09012a0 R08: 0000000000000000 R09: 0000000000000000
Sep 01 10:13:11 testnode2 kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000024b2a770
Sep 01 10:13:11 testnode2 kernel: R13: 0000000024b28590 R14: 0000000024b2a970 R15: 0000000024b28590
Sep 01 10:13:11 testnode2 kernel:  </TASK>
Sep 01 10:13:11 testnode2 kernel: irq event stamp: 67201
Sep 01 10:13:11 testnode2 kernel: hardirqs last  enabled at (67213): [<ffffffffa072b20e>] __up_console_sem+0x5e/0x70
Sep 01 10:13:11 testnode2 kernel: hardirqs last disabled at (67226): [<ffffffffa072b1f3>] __up_console_sem+0x43/0x70
Sep 01 10:13:11 testnode2 kernel: softirqs last  enabled at (66794): [<ffffffffa0545c4c>] __irq_exit_rcu+0x24c/0x300
Sep 01 10:13:11 testnode2 kernel: softirqs last disabled at (66789): [<ffffffffa0545c4c>] __irq_exit_rcu+0x24c/0x300
Sep 01 10:13:11 testnode2 kernel: ---[ end trace 0000000000000000 ]---
Sep 01 10:13:12 testnode2 kernel: SoftiWARP detached