* blktests failures with v7.2 kernel
@ 2026-08-18 11:50 Shin'ichiro Kawasaki
2026-08-18 12:11 ` Shin'ichiro Kawasaki
0 siblings, 1 reply; 2+ messages in thread
From: Shin'ichiro Kawasaki @ 2026-08-18 11:50 UTC (permalink / raw)
To: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org,
linux-scsi@vger.kernel.org, nbd, linux-rdma
Hi all,
I ran the latest blktests (git hash: ddc0babaccc4) with the v7.2 kernel. I
observed 4 failures listed below. Comparing with the previous report for the
v7.2-rc1 kernel [1], 4 failures are no longer observed (block/005,
nvme/033,036,037, nvme/058 fc transport and nvme/063 tcp transport), and 2
failures are new (nvme/068 and nvme/069).
[1] https://lore.kernel.org/linux-block/akryKgCjof_7d9na@shinmob/
List of failures
================
#1: nvme/061 (rdma transport, siw driver)(kmemleak)
#2: nvme/062 (tcp transport)
#3: nvme/068 (new)
#4: nvme/069 (new)
Failure description
===================
#1: nvme/061 (rdma transport, siw driver)(kmemleak)
When the test case nvme/061 is repeated twice for the rdma transport and the
siw driver on the kernel with CONFIG_DEBUG_KMEMLEAK enabled, it causes
kmemleak that is detected at the beginning of the 2nd run. A fix patch is
queued for v7.3-rc1 kernel [2].
[2] https://lore.kernel.org/linux-nvme/20260629051529.810925-1-shinichiro.kawasaki@wdc.com/
#2: nvme/062 (tcp transport)
The test case nvme/062 fails for tcp transport due to the lockdep WARN
related to the three locks root->kernfs_rwsem, sparse_irq_lock and
root->kernfs_supers_rwsem [1]. Fix discussion is ongoing [3] (Thank goes to
Xixin Liu).
[3] https://lore.kernel.org/linux-nvme/cover.1782436781.git.liuxixin@kylinos.cn/
#3: nvme/068 (new)
When it is repeated several times, the test case nvme/068 fails with the
message below:
nvme/068 (tr=rdma) (NVMe multipath delayed removal test) [failed]
runtime 10.979s ... 10.975s
--- tests/nvme/068.out 2026-05-21 11:44:28.070959369 +0900
+++ /home/shin/Blktests/blktests/results/nodev_tr_rdma/nvme/068.out.bad 2026-08-18 17:51:30.262228567 +0900
@@ -1,3 +1,4 @@
Running nvme/068
pwrite: Input/output error
+module refcount not as original
Test complete
The failure is spoladic. I'm not sure if recent kernel changes triggered
the failure or not.
The failure looks dependent on test nodes. It was observed with loop
transport and rdma transport, but was not observed with tcp transport.
This needs further debug.
#4: nvme/069 (new)
The test case nvme/069 fails with the KASAN slab-use-after-free in
cdev_put() regardless of transport types [4]. This failure is expected since
the test case was added to confirm the kernel side fix [5] for the failure.
The fix is queued for v7.3-rc1 kernel.
[5] https://lore.kernel.org/linux-nvme/20260713104238.3034640-1-john.g.garry@oracle.com/
[4] dmesg during nvme/069
Aug 18 15:18:43 testnode2 unknown: run blktests nvme/069 at 2026-08-18 15:18:43
Aug 18 15:18:43 testnode2 kernel: loop0: detected capacity change from 0 to 2097152
Aug 18 15:18:43 testnode2 kernel: nvmet: adding nsid 1 to subsystem blktests-subsystem-1
Aug 18 15:18:43 testnode2 kernel: nvmet: Created nvm controller 1 for subsystem blktests-subsystem-1 for NQN nqn.2014-08.org.nvmexpress:uuid:0f01fb42-9f7f-4856-b0b3-51e60b8de349.
Aug 18 15:18:43 testnode2 kernel: nvme nvme5: creating 4 I/O queues.
Aug 18 15:18:43 testnode2 kernel: nvme nvme5: new ctrl: "blktests-subsystem-1"
Aug 18 15:18:44 testnode2 kernel: nvme nvme5: Removing ctrl: NQN "blktests-subsystem-1"
Aug 18 15:18:45 testnode2 kernel: ==================================================================
Aug 18 15:18:45 testnode2 kernel: BUG: KASAN: slab-use-after-free in cdev_put+0x54/0x60
Aug 18 15:18:45 testnode2 kernel: Read of size 8 at addr ffff888128968170 by task nvme-delay-ioct/1178
Aug 18 15:18:45 testnode2 kernel:
Aug 18 15:18:45 testnode2 kernel: CPU: 2 UID: 0 PID: 1178 Comm: nvme-delay-ioct Not tainted 7.2.0 #660 PREEMPT(full)
Aug 18 15:18:45 testnode2 kernel: Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025
Aug 18 15:18:45 testnode2 kernel: Call Trace:
Aug 18 15:18:45 testnode2 kernel: <TASK>
Aug 18 15:18:45 testnode2 kernel: dump_stack_lvl+0x6a/0x90
Aug 18 15:18:45 testnode2 kernel: print_report+0x14b/0x4b0
Aug 18 15:18:45 testnode2 kernel: ? __virt_addr_valid+0x150/0x4d0
Aug 18 15:18:45 testnode2 kernel: ? __virt_addr_valid+0x228/0x4d0
Aug 18 15:18:45 testnode2 kernel: kasan_report+0xd5/0x100
Aug 18 15:18:45 testnode2 kernel: ? cdev_put+0x54/0x60
Aug 18 15:18:45 testnode2 kernel: ? cdev_put+0x54/0x60
Aug 18 15:18:45 testnode2 kernel: cdev_put+0x54/0x60
Aug 18 15:18:45 testnode2 kernel: __fput+0x749/0xae0
Aug 18 15:18:45 testnode2 kernel: fput_close_sync+0xda/0x1b0
Aug 18 15:18:45 testnode2 kernel: ? __pfx_fput_close_sync+0x10/0x10
Aug 18 15:18:45 testnode2 kernel: ? do_raw_spin_unlock+0x55/0x230
Aug 18 15:18:45 testnode2 kernel: __x64_sys_close+0x87/0xf0
Aug 18 15:18:45 testnode2 kernel: do_syscall_64+0xdf/0x790
Aug 18 15:18:45 testnode2 kernel: ? lock_acquire+0x199/0x320
Aug 18 15:18:45 testnode2 kernel: ? lock_acquire+0x1a9/0x320
Aug 18 15:18:45 testnode2 kernel: ? find_held_lock+0x2b/0x80
Aug 18 15:18:45 testnode2 kernel: ? nvme_ns_head_chr_ioctl+0x142/0x2e0 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: ? lock_release+0x1b5/0x340
Aug 18 15:18:45 testnode2 kernel: ? nvme_numa_path+0xd5/0x190 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: ? nvme_ns_head_chr_ioctl+0x152/0x2e0 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: ? rcu_is_watching+0x11/0xb0
Aug 18 15:18:45 testnode2 kernel: ? do_syscall_64+0x1ec/0x790
Aug 18 15:18:45 testnode2 kernel: ? trace_hardirqs_on_prepare+0x139/0x180
Aug 18 15:18:45 testnode2 kernel: ? lockdep_hardirqs_on+0x8c/0x130
Aug 18 15:18:45 testnode2 kernel: ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
Aug 18 15:18:45 testnode2 kernel: ? do_syscall_64+0x20a/0x790
Aug 18 15:18:45 testnode2 kernel: ? rcu_read_unlock+0x17/0x60
Aug 18 15:18:45 testnode2 kernel: ? lock_release+0x1b5/0x340
Aug 18 15:18:45 testnode2 kernel: ? find_held_lock+0x2b/0x80
Aug 18 15:18:45 testnode2 kernel: ? exc_page_fault+0x94/0x140
Aug 18 15:18:45 testnode2 kernel: ? lock_release+0x1b5/0x340
Aug 18 15:18:45 testnode2 kernel: ? rcu_is_watching+0x11/0xb0
Aug 18 15:18:45 testnode2 kernel: ? trace_hardirqs_on+0x14/0x190
Aug 18 15:18:45 testnode2 kernel: ? preempt_count_add+0x7f/0x190
Aug 18 15:18:45 testnode2 kernel: ? do_syscall_64+0x5d/0x790
Aug 18 15:18:45 testnode2 kernel: ? do_syscall_64+0x8d/0x790
Aug 18 15:18:45 testnode2 kernel: ? irqentry_exit+0xfc/0x810
Aug 18 15:18:45 testnode2 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Aug 18 15:18:45 testnode2 kernel: RIP: 0033:0x7fa5379ab08e
Aug 18 15:18:45 testnode2 kernel: Code: 4d 89 d8 e8 94 bd 00 00 4c 8b 5d f8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 74 11 c9 c3 0f 1f 80 00 00 00 00 48 8b 45 10 0f 05 <c9> c3 83 e2 39 83 fa 08 75 e7 e8 03 ff ff ff 0f 1f 00 f3 0f 1e fa
Aug 18 15:18:45 testnode2 kernel: RSP: 002b:00007fffd32959b0 EFLAGS: 00000202 ORIG_RAX: 0000000000000003
Aug 18 15:18:45 testnode2 kernel: RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fa5379ab08e
Aug 18 15:18:45 testnode2 kernel: RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000000000003
Aug 18 15:18:45 testnode2 kernel: RBP: 00007fffd32959c0 R08: 0000000000000000 R09: 0000000000000000
Aug 18 15:18:45 testnode2 kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000001
Aug 18 15:18:45 testnode2 kernel: R13: 0000000000000002 R14: 00007fa537b87000 R15: 0000000000402e00
Aug 18 15:18:45 testnode2 kernel: </TASK>
Aug 18 15:18:45 testnode2 kernel:
Aug 18 15:18:45 testnode2 kernel: Allocated by task 88:
Aug 18 15:18:45 testnode2 kernel: kasan_save_stack+0x2b/0x50
Aug 18 15:18:45 testnode2 kernel: kasan_save_track+0x10/0x30
Aug 18 15:18:45 testnode2 kernel: __kasan_kmalloc+0x96/0xb0
Aug 18 15:18:45 testnode2 kernel: __kmalloc_noprof+0x2fa/0x870
Aug 18 15:18:45 testnode2 kernel: nvme_alloc_ns+0x8af/0x2ce0 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: nvme_scan_ns+0x647/0x7b0 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: async_run_entry_fn+0x93/0x550
Aug 18 15:18:45 testnode2 kernel: process_one_work+0x8b2/0x15e0
Aug 18 15:18:45 testnode2 kernel: worker_thread+0x5fd/0xfe0
Aug 18 15:18:45 testnode2 kernel: kthread+0x367/0x460
Aug 18 15:18:45 testnode2 kernel: ret_from_fork+0x655/0x9d0
Aug 18 15:18:45 testnode2 kernel: ret_from_fork_asm+0x1a/0x30
Aug 18 15:18:45 testnode2 kernel:
Aug 18 15:18:45 testnode2 kernel: Freed by task 1178:
Aug 18 15:18:45 testnode2 kernel: kasan_save_stack+0x2b/0x50
Aug 18 15:18:45 testnode2 kernel: kasan_save_track+0x10/0x30
Aug 18 15:18:45 testnode2 kernel: kasan_save_free_info+0x37/0x60
Aug 18 15:18:45 testnode2 kernel: __kasan_slab_free+0x68/0x90
Aug 18 15:18:45 testnode2 kernel: kfree+0x273/0x6e0
Aug 18 15:18:45 testnode2 kernel: nvme_ns_head_chr_release+0x3d/0x60 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: __fput+0x36f/0xae0
Aug 18 15:18:45 testnode2 kernel: fput_close_sync+0xda/0x1b0
Aug 18 15:18:45 testnode2 kernel: __x64_sys_close+0x87/0xf0
Aug 18 15:18:45 testnode2 kernel: do_syscall_64+0xdf/0x790
Aug 18 15:18:45 testnode2 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Aug 18 15:18:45 testnode2 kernel:
Aug 18 15:18:45 testnode2 kernel: Last potentially related work creation:
Aug 18 15:18:45 testnode2 kernel: kasan_save_stack+0x2b/0x50
Aug 18 15:18:45 testnode2 kernel: kasan_record_aux_stack+0xbb/0xf0
Aug 18 15:18:45 testnode2 kernel: insert_work+0x3c/0x2f0
Aug 18 15:18:45 testnode2 kernel: __queue_work+0x9e6/0xf50
Aug 18 15:18:45 testnode2 kernel: queue_work_on+0xc8/0xd0
Aug 18 15:18:45 testnode2 kernel: kblockd_schedule_work+0x19/0x30
Aug 18 15:18:45 testnode2 kernel: nvme_mpath_put_disk+0x46/0xb0 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: nvme_free_ns_head+0x22/0x170 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: nvme_ns_head_chr_release+0x3d/0x60 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: __fput+0x36f/0xae0
Aug 18 15:18:45 testnode2 kernel: fput_close_sync+0xda/0x1b0
Aug 18 15:18:45 testnode2 kernel: __x64_sys_close+0x87/0xf0
Aug 18 15:18:45 testnode2 kernel: do_syscall_64+0xdf/0x790
Aug 18 15:18:45 testnode2 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Aug 18 15:18:45 testnode2 kernel:
Aug 18 15:18:45 testnode2 kernel: Second to last potentially related work creation:
Aug 18 15:18:45 testnode2 kernel: kasan_save_stack+0x2b/0x50
Aug 18 15:18:45 testnode2 kernel: kasan_record_aux_stack+0xbb/0xf0
Aug 18 15:18:45 testnode2 kernel: insert_work+0x3c/0x2f0
Aug 18 15:18:45 testnode2 kernel: __queue_work+0x9e6/0xf50
Aug 18 15:18:45 testnode2 kernel: queue_work_on+0xc8/0xd0
Aug 18 15:18:45 testnode2 kernel: kblockd_schedule_work+0x19/0x30
Aug 18 15:18:45 testnode2 kernel: nvme_remove_head+0x40/0xc0 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: nvme_ns_remove+0x50c/0x9e0 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: nvme_remove_namespaces+0x289/0x3f0 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: nvme_do_delete_ctrl+0xf6/0x160 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: nvme_delete_ctrl_sync.cold+0x8/0xd [nvme_core]
Aug 18 15:18:45 testnode2 kernel: nvme_sysfs_delete+0xb7/0xe0 [nvme_core]
Aug 18 15:18:45 testnode2 kernel: kernfs_fop_write_iter+0x3d6/0x5e0
Aug 18 15:18:45 testnode2 kernel: vfs_write+0x4b3/0xf40
Aug 18 15:18:45 testnode2 kernel: ksys_write+0x112/0x250
Aug 18 15:18:45 testnode2 kernel: do_syscall_64+0xdf/0x790
Aug 18 15:18:45 testnode2 kernel: entry_SYSCALL_64_after_hwframe+0x76/0x7e
Aug 18 15:18:45 testnode2 kernel:
Aug 18 15:18:45 testnode2 kernel: The buggy address belongs to the object at ffff888128968000
which belongs to the cache kmalloc-4k of size 4096
Aug 18 15:18:45 testnode2 kernel: The buggy address is located 368 bytes inside of
freed 4096-byte region [ffff888128968000, ffff888128969000)
Aug 18 15:18:45 testnode2 kernel:
Aug 18 15:18:45 testnode2 kernel: The buggy address belongs to the physical page:
Aug 18 15:18:45 testnode2 kernel: page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x128968
Aug 18 15:18:45 testnode2 kernel: head: order:3 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0
Aug 18 15:18:45 testnode2 kernel: flags: 0x17ffffc0000040(head|node=0|zone=2|lastcpupid=0x1fffff)
Aug 18 15:18:45 testnode2 kernel: page_type: f5(slab)
Aug 18 15:18:45 testnode2 kernel: raw: 0017ffffc0000040 ffff888100043040 dead000000000100 dead000000000122
Aug 18 15:18:45 testnode2 kernel: raw: 0000000000000000 0000000800040004 00000000f5000000 0000000000000000
Aug 18 15:18:45 testnode2 kernel: head: 0017ffffc0000040 ffff888100043040 dead000000000100 dead000000000122
Aug 18 15:18:45 testnode2 kernel: head: 0000000000000000 0000000800040004 00000000f5000000 0000000000000000
Aug 18 15:18:45 testnode2 kernel: head: 0017ffffc0000003 fffffffffffffe01 00000000ffffffff 00000000ffffffff
Aug 18 15:18:45 testnode2 kernel: head: ffffffffffffffff 0000000000000000 00000000ffffffff 0000000000000008
Aug 18 15:18:45 testnode2 kernel: page dumped because: kasan: bad access detected
Aug 18 15:18:45 testnode2 kernel:
Aug 18 15:18:45 testnode2 kernel: Memory state around the buggy address:
Aug 18 15:18:45 testnode2 kernel: ffff888128968000: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
Aug 18 15:18:45 testnode2 kernel: ffff888128968080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
Aug 18 15:18:45 testnode2 kernel: >ffff888128968100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
Aug 18 15:18:45 testnode2 kernel: ^
Aug 18 15:18:45 testnode2 kernel: ffff888128968180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
Aug 18 15:18:45 testnode2 kernel: ffff888128968200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
Aug 18 15:18:45 testnode2 kernel: ==================================================================
Aug 18 15:18:45 testnode2 kernel: Disabling lock debugging due to kernel taint
^ permalink raw reply [flat|nested] 2+ messages in thread* Re: blktests failures with v7.2 kernel
2026-08-18 11:50 blktests failures with v7.2 kernel Shin'ichiro Kawasaki
@ 2026-08-18 12:11 ` Shin'ichiro Kawasaki
0 siblings, 0 replies; 2+ messages in thread
From: Shin'ichiro Kawasaki @ 2026-08-18 12:11 UTC (permalink / raw)
To: linux-block@vger.kernel.org, linux-nvme@lists.infradead.org,
linux-scsi@vger.kernel.org, nbd, linux-rdma
On Aug 18, 2026 / 20:50, Shin'ichiro Kawasaki wrote:
[...]
> #3: nvme/068 (new)
>
> When it is repeated several times, the test case nvme/068 fails with the
> message below:
>
> nvme/068 (tr=rdma) (NVMe multipath delayed removal test) [failed]
> runtime 10.979s ... 10.975s
> --- tests/nvme/068.out 2026-05-21 11:44:28.070959369 +0900
> +++ /home/shin/Blktests/blktests/results/nodev_tr_rdma/nvme/068.out.bad 2026-08-18 17:51:30.262228567 +0900
> @@ -1,3 +1,4 @@
> Running nvme/068
> pwrite: Input/output error
> +module refcount not as original
> Test complete
>
> The failure is spoladic. I'm not sure if recent kernel changes triggered
> the failure or not.
>
> The failure looks dependent on test nodes. It was observed with loop
> transport and rdma transport, but was not observed with tcp transport.
> This needs further debug.
Let me amend this part. I totally forgot that I myself chased down this failure
and posted the fix patch for blktests [*]. No need to debug further.
[*] https://lore.kernel.org/linux-nvme/20260806033146.2067433-1-shinichiro.kawasaki@wdc.com/
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2026-08-18 12:11 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-18 11:50 blktests failures with v7.2 kernel Shin'ichiro Kawasaki
2026-08-18 12:11 ` Shin'ichiro Kawasaki
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox