* [bug report] fstests generic/085 btrfs hang with use-after-free at bdev_super_lock
@ 2025-11-05 1:01 Shinichiro Kawasaki
2025-11-05 6:54 ` Nilay Shroff
0 siblings, 1 reply; 2+ messages in thread
From: Shinichiro Kawasaki @ 2025-11-05 1:01 UTC (permalink / raw)
To: linux-block@vger.kernel.org; +Cc: linux-fsdevel@vger.kernel.org, Nilay Shroff
When I run fstests for btrfs on regular null_blk devices, I observe KASAN
slab-use-after-free in bdev_super_lock() followed by kernel hang. I observed it
for the kernel v6.17-rc4 for the first time. And I still observe it for the
latest kernel v6.18-rc4.
The hang is recreated when I prepare eight of 5Gib size null_blk devices, assign
one for TEST_DEV, and assign the other seven for SCRATCH_DEV_POOl. The hang
happens at generic/085. It is sporadic. When I repeat the test case g085 only,
the hang is not recreated. But when I repeat the whole fstests a few times, the
hang is recreated in stable manner. It takes several hours to recreate the hang.
I spent some weeks to bisect, and found the trigger commit is this:
370ac285f23a ("block: avoid cpu_hotplug_lock depedency on freeze_lock")
The commit was included in the kernel tag v6.17-rc3. When I reverted the commit
from v6.17-rc3, the hang disappeared (I repeated the whole fstests 5 times on
two test nodes, and did not observe the hang). I'm not sure if the commit
created the problem cause or revealed the hidden problem.
Any actions or advice for fix will be appreciated. If test runs in my
environment helps, please let me know.
[1]
run fstests generic/085 at 2025-11-04 21:23:17
BTRFS: device fsid 1b24d69c-9ed9-4d19-844e-1ae84715e4a3 devid 1 transid 519 /dev/nullb0 (250:0) scanned by mount (859368)
BTRFS info (device nullb0): first mount of filesystem 1b24d69c-9ed9-4d19-844e-1ae84715e4a3
BTRFS info (device nullb0): using crc32c (crc32c-lib) checksum algorithm
BTRFS info (device nullb0): enabling ssd optimizations
BTRFS info (device nullb0): enabling free space tree
BTRFS: device fsid 48ad7e3a-aa92-4748-a2be-0098bbc1a4f8 devid 1 transid 8 /dev/mapper/085-test (252:0) scanned by mount (859456)
BTRFS info (device dm-0): first mount of filesystem 48ad7e3a-aa92-4748-a2be-0098bbc1a4f8
BTRFS info (device dm-0): using crc32c (crc32c-lib) checksum algorithm
BTRFS info (device dm-0): checking UUID tree
BTRFS info (device dm-0): enabling ssd optimizations
BTRFS info (device dm-0): enabling free space tree
BTRFS info (device dm-0): last unmount of filesystem 48ad7e3a-aa92-4748-a2be-0098bbc1a4f8
BTRFS: device fsid 48ad7e3a-aa92-4748-a2be-0098bbc1a4f8 devid 1 transid 9 /dev/mapper/085-test (252:0) scanned by mount (859487)
BTRFS info (device dm-0): first mount of filesystem 48ad7e3a-aa92-4748-a2be-0098bbc1a4f8
BTRFS info (device dm-0): using crc32c (crc32c-lib) checksum algorithm
BTRFS info (device dm-0): enabling ssd optimizations
BTRFS info (device dm-0): enabling free space tree
BTRFS info (device dm-0): last unmount of filesystem 48ad7e3a-aa92-4748-a2be-0098bbc1a4f8
BTRFS: device fsid 48ad7e3a-aa92-4748-a2be-0098bbc1a4f8 devid 1 transid 9 /dev/mapper/085-test (252:0) scanned by mount (859516)
BTRFS info (device dm-0): first mount of filesystem 48ad7e3a-aa92-4748-a2be-0098bbc1a4f8
BTRFS info (device dm-0): using crc32c (crc32c-lib) checksum algorithm
BTRFS info (device dm-0): enabling ssd optimizations
BTRFS info (device dm-0): enabling free space tree
BTRFS info (device dm-0): last unmount of filesystem 48ad7e3a-aa92-4748-a2be-0098bbc1a4f8
BTRFS info (device dm-0): first mount of filesystem 48ad7e3a-aa92-4748-a2be-0098bbc1a4f8
BTRFS info (device dm-0): using crc32c (crc32c-lib) checksum algorithm
BTRFS info (device dm-0): enabling ssd optimizations
BTRFS info (device dm-0): enabling free space tree
BTRFS info (device dm-0): last unmount of filesystem 48ad7e3a-aa92-4748-a2be-0098bbc1a4f8
==================================================================
BUG: KASAN: slab-use-after-free in bdev_super_lock+0x2c7/0x320
Read of size 4 at addr ffff888395682108 by task dmsetup/859561
CPU: 7 UID: 0 PID: 859561 Comm: dmsetup Not tainted 6.18.0-rc4-kts-btrfs #6 PREEMPT(lazy)
Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015
Call Trace:
<TASK>
? bdev_super_lock+0x2c7/0x320
dump_stack_lvl+0x6e/0xa0
print_address_description.constprop.0+0x88/0x320
? bdev_super_lock+0x2c7/0x320
print_report+0xfc/0x1ff
? __virt_addr_valid+0x25a/0x4e0
? bdev_super_lock+0x2c7/0x320
kasan_report+0xe1/0x1a0
? bdev_super_lock+0x2c7/0x320
bdev_super_lock+0x2c7/0x320
get_bdev_super+0x11/0xa0
fs_bdev_freeze+0x54/0x180
bdev_freeze+0xbc/0x1f0
__dm_suspend+0x115/0x490
? lock_is_held_type+0x9a/0x110
dm_suspend+0x16d/0x230
dev_suspend+0x128/0x170
ctl_ioctl+0x397/0x760
? __pfx_ctl_ioctl+0x10/0x10
? __pfx_handle_pte_fault+0x10/0x10
dm_ctl_ioctl+0xe/0x20
__x64_sys_ioctl+0x13c/0x1c0
do_syscall_64+0x94/0x7f0
? __lock_acquire+0x55d/0xbf0
? __pfx_css_rstat_updated+0x10/0x10
? lock_acquire.part.0+0xb8/0x230
? handle_mm_fault+0x485/0xa30
? find_held_lock+0x2b/0x80
? __lock_release.isra.0+0x59/0x170
? lock_release.part.0+0x1c/0x50
? find_held_lock+0x2b/0x80
? __lock_release.isra.0+0x59/0x170
? do_user_addr_fault+0x4cb/0xa40
? trace_hardirqs_on_prepare+0x101/0x150
? lockdep_hardirqs_on_prepare.part.0+0x9b/0x150
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7f3dea2d20ed
Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
RSP: 002b:00007ffd151d6960 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f3dea3d6780 RCX: 00007f3dea2d20ed
RDX: 000055d6f1c61b10 RSI: 00000000c138fd06 RDI: 0000000000000003
RBP: 00007ffd151d69b0 R08: 00007f3dea422c38 R09: 00007ffd151d6810
R10: 00007f3dea41adec R11: 0000000000000246 R12: 000055d6f1c61bc0
R13: 0000000000000080 R14: 00007f3dea41adec R15: 00007f3dea41adec
</TASK>
Allocated by task 859516:
kasan_save_stack+0x30/0x50
kasan_save_track+0x14/0x30
__kasan_kmalloc+0x9a/0xb0
alloc_super+0x9a/0xb40
sget_fc+0xe8/0xb40
btrfs_get_tree_super+0x45a/0xc70 [btrfs]
btrfs_get_tree_subvol+0x238/0x640 [btrfs]
vfs_get_tree+0x8b/0x2f0
vfs_cmd_create+0xbd/0x280
__do_sys_fsconfig+0x659/0xa40
do_syscall_64+0x94/0x7f0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
Freed by task 852689:
kasan_save_stack+0x30/0x50
kasan_save_track+0x14/0x30
__kasan_save_free_info+0x3b/0x70
__kasan_slab_free+0x6b/0x90
kfree+0x14a/0x650
process_one_work+0x86b/0x14c0
worker_thread+0x5f2/0xfd0
kthread+0x3a4/0x760
ret_from_fork+0x2d6/0x3e0
ret_from_fork_asm+0x1a/0x30
Last potentially related work creation:
kasan_save_stack+0x30/0x50
kasan_record_aux_stack+0xb0/0xc0
__queue_work+0x8c2/0x1250
queue_work_on+0xc1/0xd0
rcu_do_batch+0x34a/0x1900
rcu_core+0x62f/0x9f0
handle_softirqs+0x1de/0x7e0
__irq_exit_rcu+0x181/0x1d0
irq_exit_rcu+0xe/0x20
sysvec_apic_timer_interrupt+0x71/0x90
asm_sysvec_apic_timer_interrupt+0x1a/0x20
Second to last potentially related work creation:
kasan_save_stack+0x30/0x50
kasan_record_aux_stack+0xb0/0xc0
__call_rcu_common.constprop.0+0xc4/0x840
deactivate_locked_super+0x12a/0x160
fs_bdev_thaw+0xc2/0x150
bdev_thaw+0x10a/0x1d0
unlock_fs+0xa2/0xf0
__dm_resume+0x92/0xf0
dm_resume+0x159/0x1f0
do_resume+0x421/0x5f0
ctl_ioctl+0x397/0x760
dm_ctl_ioctl+0xe/0x20
__x64_sys_ioctl+0x13c/0x1c0
do_syscall_64+0x94/0x7f0
entry_SYSCALL_64_after_hwframe+0x76/0x7e
The buggy address belongs to the object at ffff888395682000
which belongs to the cache kmalloc-rnd-12-4k of size 4096
The buggy address is located 264 bytes inside of
freed 4096-byte region [ffff888395682000, ffff888395683000)
The buggy address belongs to the physical page:
page: refcount:0 mapcount:0 mapping:0000000000000000 index:0x0 pfn:0x395680
head: order:3 mapcount:0 entire_mapcount:0 nr_pages_mapped:0 pincount:0
ksm flags: 0x17ffffc0000040(head|node=0|zone=2|lastcpupid=0x1fffff)
page_type: f5(slab)
raw: 0017ffffc0000040 ffff8881000597c0 ffffea00053e8800 dead000000000003
raw: 0000000000000000 0000000080040004 00000000f5000000 0000000000000000
head: 0017ffffc0000040 ffff8881000597c0 ffffea00053e8800 dead000000000003
head: 0000000000000000 0000000080040004 00000000f5000000 0000000000000000
head: 0017ffffc0000003 ffffea000e55a001 00000000ffffffff 00000000ffffffff
head: ffffffffffffffff 0000000000000000 00000000ffffffff 0000000000000008
page dumped because: kasan: bad access detected
Memory state around the buggy address:
ffff888395682000: fa fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff888395682080: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
>ffff888395682100: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
^
ffff888395682180: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
ffff888395682200: fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb fb
==================================================================
Disabling lock debugging due to kernel taint
Oops: general protection fault, probably for non-canonical address 0xe092bc00e001a05a: 0000 [#1] SMP KASAN PTI
KASAN: maybe wild-memory-access in range [0x04960007000d02d0-0x04960007000d02d7]
CPU: 3 UID: 0 PID: 859561 Comm: dmsetup Tainted: G B 6.18.0-rc4-kts-btrfs #6 PREEMPT(lazy)
Tainted: [B]=BAD_PAGE
Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015
RIP: 0010:__list_del_entry_valid_or_report+0x91/0x280
Code: de 48 39 c1 74 6e 48 b8 22 01 00 00 00 00 ad de 48 39 c2 0f 84 83 00 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 d7 48 c1 ef 03 <80> 3c 07 00 0f 85 84 01 00 00 48 39 32 0f 85 87 00 00 00 48 ba 00
RSP: 0018:ffff88838ee77900 EFLAGS: 00010206
RAX: dffffc0000000000 RBX: ffff888395682000 RCX: ffff88827f8e5b00
RDX: 04960007000d02d1 RSI: ffff888395682000 RDI: 0092c000e001a05a
RBP: 0000000000000000 R08: ffffffff8c75dc08 R09: ffffed1071dcef1c
R10: 0000000000000003 R11: ffffffff90a75358 R12: ffff888395682108
R13: 0000000000000001 R14: ffff888115996000 R15: 0000000000000001
FS: 00007f3de9f91840(0000) GS:ffff88876b558000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f068a2d2c58 CR3: 000000029b4a8001 CR4: 00000000001726f0
Call Trace:
<TASK>
__put_super.part.0+0x12/0x240
bdev_super_lock+0x275/0x320
get_bdev_super+0x11/0xa0
fs_bdev_freeze+0x54/0x180
bdev_freeze+0xbc/0x1f0
__dm_suspend+0x115/0x490
? lock_is_held_type+0x9a/0x110
dm_suspend+0x16d/0x230
dev_suspend+0x128/0x170
ctl_ioctl+0x397/0x760
? __pfx_ctl_ioctl+0x10/0x10
? __pfx_handle_pte_fault+0x10/0x10
dm_ctl_ioctl+0xe/0x20
__x64_sys_ioctl+0x13c/0x1c0
do_syscall_64+0x94/0x7f0
? __lock_acquire+0x55d/0xbf0
? __pfx_css_rstat_updated+0x10/0x10
? lock_acquire.part.0+0xb8/0x230
? handle_mm_fault+0x485/0xa30
? find_held_lock+0x2b/0x80
? __lock_release.isra.0+0x59/0x170
? lock_release.part.0+0x1c/0x50
? find_held_lock+0x2b/0x80
? __lock_release.isra.0+0x59/0x170
? do_user_addr_fault+0x4cb/0xa40
? trace_hardirqs_on_prepare+0x101/0x150
? lockdep_hardirqs_on_prepare.part.0+0x9b/0x150
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7f3dea2d20ed
Code: 04 25 28 00 00 00 48 89 45 c8 31 c0 48 8d 45 10 c7 45 b0 10 00 00 00 48 89 45 b8 48 8d 45 d0 48 89 45 c0 b8 10 00 00 00 0f 05 <89> c2 3d 00 f0 ff ff 77 1a 48 8b 45 c8 64 48 2b 04 25 28 00 00 00
RSP: 002b:00007ffd151d6960 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
RAX: ffffffffffffffda RBX: 00007f3dea3d6780 RCX: 00007f3dea2d20ed
RDX: 000055d6f1c61b10 RSI: 00000000c138fd06 RDI: 0000000000000003
RBP: 00007ffd151d69b0 R08: 00007f3dea422c38 R09: 00007ffd151d6810
R10: 00007f3dea41adec R11: 0000000000000246 R12: 000055d6f1c61bc0
R13: 0000000000000080 R14: 00007f3dea41adec R15: 00007f3dea41adec
</TASK>
Modules linked in: dm_flakey null_blk target_core_user target_core_mod rfkill nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr sunrpc binfmt_misc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel jc42 iTCO_wdt kvm intel_pmc_bxt at24 iTCO_vendor_support irqbypass rapl intel_cstate btrfs intel_uncore i2c_i801 pcspkr i2c_smbus intel_pch_thermal igb ses xor enclosure dca lpc_ich raid6_pq e1000e joydev video ie31200_edac wmi loop dm_multipath nfnetlink zram lz4hc_compress lz4_compress zstd_compress ast drm_client_lib i2c_algo_bit drm_shmem_helper drm_kms_helper drm polyval_clmulni ghash_clmulni_intel mpi3mr scsi_transport_sas scsi_dh_rdac scsi_dh_emc scsi_dh_alua fuse
---[ end trace 0000000000000000 ]---
RIP: 0010:__list_del_entry_valid_or_report+0x91/0x280
Code: de 48 39 c1 74 6e 48 b8 22 01 00 00 00 00 ad de 48 39 c2 0f 84 83 00 00 00 48 b8 00 00 00 00 00 fc ff df 48 89 d7 48 c1 ef 03 <80> 3c 07 00 0f 85 84 01 00 00 48 39 32 0f 85 87 00 00 00 48 ba 00
RSP: 0018:ffff88838ee77900 EFLAGS: 00010206
RAX: dffffc0000000000 RBX: ffff888395682000 RCX: ffff88827f8e5b00
RDX: 04960007000d02d1 RSI: ffff888395682000 RDI: 0092c000e001a05a
RBP: 0000000000000000 R08: ffffffff8c75dc08 R09: ffffed1071dcef1c
R10: 0000000000000003 R11: ffffffff90a75358 R12: ffff888395682108
R13: 0000000000000001 R14: ffff888115996000 R15: 0000000000000001
FS: 00007f3de9f91840(0000) GS:ffff88876b558000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f068a2d2c58 CR3: 000000029b4a8001 CR4: 00000000001726f0
note: dmsetup[859561] exited with preempt_count 1
watchdog: BUG: soft lockup - CPU#4 stuck for 22s! [umount:859559]
CPU#4 Utilization every 4000ms during lockup:
#1: 100% system, 0% softirq, 1% hardirq, 0% idle
#2: 100% system, 0% softirq, 1% hardirq, 0% idle
#3: 100% system, 0% softirq, 1% hardirq, 0% idle
#4: 100% system, 0% softirq, 1% hardirq, 0% idle
#5: 100% system, 0% softirq, 1% hardirq, 0% idle
Modules linked in: dm_flakey null_blk target_core_user target_core_mod rfkill nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr sunrpc binfmt_misc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel jc42 iTCO_wdt kvm intel_pmc_bxt at24 iTCO_vendor_support irqbypass rapl intel_cstate btrfs intel_uncore i2c_i801 pcspkr i2c_smbus intel_pch_thermal igb ses xor enclosure dca lpc_ich raid6_pq e1000e joydev video ie31200_edac wmi loop dm_multipath nfnetlink zram lz4hc_compress lz4_compress zstd_compress ast drm_client_lib i2c_algo_bit drm_shmem_helper drm_kms_helper drm polyval_clmulni ghash_clmulni_intel mpi3mr scsi_transport_sas scsi_dh_rdac scsi_dh_emc scsi_dh_alua fuse
irq event stamp: 4676
hardirqs last enabled at (4675): [<ffffffff8f3bcd44>] _raw_spin_unlock_irqrestore+0x44/0x60
hardirqs last disabled at (4676): [<ffffffff8f39cabb>] __schedule+0xd1b/0x1ab0
softirqs last enabled at (4672): [<ffffffff8ce75d91>] bdi_unregister+0x161/0x5b0
softirqs last disabled at (4670): [<ffffffff8ce75cc0>] bdi_unregister+0x90/0x5b0
CPU: 4 UID: 0 PID: 859559 Comm: umount Tainted: G B D 6.18.0-rc4-kts-btrfs #6 PREEMPT(lazy)
Tainted: [B]=BAD_PAGE, [D]=DIE
Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015
RIP: 0010:native_queued_spin_lock_slowpath+0x398/0xbe0
Code: 3d 0f b6 03 84 c0 74 36 48 b8 00 00 00 00 00 fc ff df 49 89 dc 49 89 dd 49 c1 ec 03 41 83 e5 07 49 01 c4 f3 90 41 0f b6 04 24 <44> 38 e8 7f 08 84 c0 0f 85 9b 06 00 00 0f b6 03 84 c0 75 e5 48 b8
RSP: 0018:ffff888461417b00 EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffff90a75340 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffffffff90a75340
RBP: 1ffff1108c282f62 R08: ffffffff8f3bd7a3 R09: fffffbfff214ea68
R10: fffffbfff214ea69 R11: ffffffff90a75358 R12: fffffbfff214ea68
R13: 0000000000000000 R14: ffffed109bb8c84f R15: 0000000000000000
FS: 00007fe027c4b380(0000) GS:ffff88876b5d8000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd3d77b3d90 CR3: 00000004c5318006 CR4: 00000000001726f0
Call Trace:
<TASK>
? __pfx_native_queued_spin_lock_slowpath+0x10/0x10
? trace_hardirqs_on+0x18/0x150
do_raw_spin_lock+0x1d9/0x270
? kfree+0x14a/0x650
? __pfx_do_raw_spin_lock+0x10/0x10
? generic_shutdown_super+0x225/0x320
? lock_acquire+0xf6/0x140
kill_super_notify+0x86/0x230
kill_anon_super+0x42/0x60
btrfs_kill_super+0x3e/0x60 [btrfs]
deactivate_locked_super+0xa8/0x160
cleanup_mnt+0x1da/0x420
task_work_run+0x116/0x200
? __pfx_task_work_run+0x10/0x10
? __x64_sys_umount+0x10c/0x140
? find_held_lock+0x2b/0x80
? __pfx___x64_sys_umount+0x10/0x10
exit_to_user_mode_loop+0x133/0x170
do_syscall_64+0x201/0x7f0
? lock_release.part.0+0x1c/0x50
? do_faccessat+0x1ed/0x9a0
? __pfx_do_faccessat+0x10/0x10
? __pfx_from_kgid_munged+0x10/0x10
? trace_hardirqs_on_prepare+0x101/0x150
? do_syscall_64+0x137/0x7f0
? do_syscall_64+0x137/0x7f0
? trace_hardirqs_on_prepare+0x101/0x150
? lockdep_hardirqs_on_prepare.part.0+0x9b/0x150
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7fe027d41e4b
Code: c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 81 1f 0f 00 f7 d8
RSP: 002b:00007fff6e76f488 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007fe027d41e4b
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001c2f8c0
RBP: 00007fe027f29ffc R08: 0000000001c2fcd0 R09: 00007fe027e34ac0
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000001c29528
R13: 0000000001c2f8c0 R14: 0000000001c29420 R15: 0000000001c29860
</TASK>
watchdog: BUG: soft lockup - CPU#4 stuck for 48s! [umount:859559]
CPU#4 Utilization every 4000ms during lockup:
#1: 100% system, 0% softirq, 1% hardirq, 0% idle
#2: 100% system, 0% softirq, 1% hardirq, 0% idle
#3: 100% system, 0% softirq, 1% hardirq, 0% idle
#4: 100% system, 0% softirq, 1% hardirq, 0% idle
#5: 100% system, 0% softirq, 1% hardirq, 0% idle
Modules linked in: dm_flakey null_blk target_core_user target_core_mod rfkill nf_conntrack_netbios_ns nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 nf_tables qrtr sunrpc binfmt_misc intel_rapl_msr intel_rapl_common x86_pkg_temp_thermal intel_powerclamp coretemp kvm_intel jc42 iTCO_wdt kvm intel_pmc_bxt at24 iTCO_vendor_support irqbypass rapl intel_cstate btrfs intel_uncore i2c_i801 pcspkr i2c_smbus intel_pch_thermal igb ses xor enclosure dca lpc_ich raid6_pq e1000e joydev video ie31200_edac wmi loop dm_multipath nfnetlink zram lz4hc_compress lz4_compress zstd_compress ast drm_client_lib i2c_algo_bit drm_shmem_helper drm_kms_helper drm polyval_clmulni ghash_clmulni_intel mpi3mr scsi_transport_sas scsi_dh_rdac scsi_dh_emc scsi_dh_alua fuse
irq event stamp: 4676
hardirqs last enabled at (4675): [<ffffffff8f3bcd44>] _raw_spin_unlock_irqrestore+0x44/0x60
hardirqs last disabled at (4676): [<ffffffff8f39cabb>] __schedule+0xd1b/0x1ab0
softirqs last enabled at (4672): [<ffffffff8ce75d91>] bdi_unregister+0x161/0x5b0
softirqs last disabled at (4670): [<ffffffff8ce75cc0>] bdi_unregister+0x90/0x5b0
CPU: 4 UID: 0 PID: 859559 Comm: umount Tainted: G B D L 6.18.0-rc4-kts-btrfs #6 PREEMPT(lazy)
Tainted: [B]=BAD_PAGE, [D]=DIE, [L]=SOFTLOCKUP
Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015
RIP: 0010:native_queued_spin_lock_slowpath+0x391/0xbe0
Code: f8 05 00 00 85 c0 74 3d 0f b6 03 84 c0 74 36 48 b8 00 00 00 00 00 fc ff df 49 89 dc 49 89 dd 49 c1 ec 03 41 83 e5 07 49 01 c4 <f3> 90 41 0f b6 04 24 44 38 e8 7f 08 84 c0 0f 85 9b 06 00 00 0f b6
RSP: 0018:ffff888461417b00 EFLAGS: 00000202
RAX: 0000000000000001 RBX: ffffffff90a75340 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffffffff90a75340
RBP: 1ffff1108c282f62 R08: ffffffff8f3bd7a3 R09: fffffbfff214ea68
R10: fffffbfff214ea69 R11: ffffffff90a75358 R12: fffffbfff214ea68
R13: 0000000000000000 R14: ffffed109bb8c84f R15: 0000000000000000
FS: 00007fe027c4b380(0000) GS:ffff88876b5d8000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd3d77b3d90 CR3: 00000004c5318006 CR4: 00000000001726f0
Call Trace:
<TASK>
? __pfx_native_queued_spin_lock_slowpath+0x10/0x10
? trace_hardirqs_on+0x18/0x150
do_raw_spin_lock+0x1d9/0x270
? kfree+0x14a/0x650
? __pfx_do_raw_spin_lock+0x10/0x10
? generic_shutdown_super+0x225/0x320
? lock_acquire+0xf6/0x140
kill_super_notify+0x86/0x230
kill_anon_super+0x42/0x60
btrfs_kill_super+0x3e/0x60 [btrfs]
deactivate_locked_super+0xa8/0x160
cleanup_mnt+0x1da/0x420
task_work_run+0x116/0x200
? __pfx_task_work_run+0x10/0x10
? __x64_sys_umount+0x10c/0x140
? find_held_lock+0x2b/0x80
? __pfx___x64_sys_umount+0x10/0x10
exit_to_user_mode_loop+0x133/0x170
do_syscall_64+0x201/0x7f0
? lock_release.part.0+0x1c/0x50
? do_faccessat+0x1ed/0x9a0
? __pfx_do_faccessat+0x10/0x10
? __pfx_from_kgid_munged+0x10/0x10
? trace_hardirqs_on_prepare+0x101/0x150
? do_syscall_64+0x137/0x7f0
? do_syscall_64+0x137/0x7f0
? trace_hardirqs_on_prepare+0x101/0x150
? lockdep_hardirqs_on_prepare.part.0+0x9b/0x150
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7fe027d41e4b
Code: c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 81 1f 0f 00 f7 d8
RSP: 002b:00007fff6e76f488 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007fe027d41e4b
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001c2f8c0
RBP: 00007fe027f29ffc R08: 0000000001c2fcd0 R09: 00007fe027e34ac0
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000001c29528
R13: 0000000001c2f8c0 R14: 0000000001c29420 R15: 0000000001c29860
</TASK>
rcu: INFO: rcu_preempt self-detected stall on CPU
rcu: 4-....: (63898 ticks this GP) idle=bd74/1/0x4000000000000000 softirq=1419797/1419808 fqs=16248
rcu: (t=65019 jiffies g=2934629 q=15440 ncpus=8)
CPU: 4 UID: 0 PID: 859559 Comm: umount Tainted: G B D L 6.18.0-rc4-kts-btrfs #6 PREEMPT(lazy)
Tainted: [B]=BAD_PAGE, [D]=DIE, [L]=SOFTLOCKUP
Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015
RIP: 0010:native_queued_spin_lock_slowpath+0x398/0xbe0
Code: 3d 0f b6 03 84 c0 74 36 48 b8 00 00 00 00 00 fc ff df 49 89 dc 49 89 dd 49 c1 ec 03 41 83 e5 07 49 01 c4 f3 90 41 0f b6 04 24 <44> 38 e8 7f 08 84 c0 0f 85 9b 06 00 00 0f b6 03 84 c0 75 e5 48 b8
RSP: 0018:ffff888461417b00 EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffff90a75340 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffffffff90a75340
RBP: 1ffff1108c282f62 R08: ffffffff8f3bd7a3 R09: fffffbfff214ea68
R10: fffffbfff214ea69 R11: ffffffff90a75358 R12: fffffbfff214ea68
R13: 0000000000000000 R14: ffffed109bb8c84f R15: 0000000000000000
FS: 00007fe027c4b380(0000) GS:ffff88876b5d8000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd3d77b3d90 CR3: 00000004c5318006 CR4: 00000000001726f0
Call Trace:
<TASK>
? __pfx_native_queued_spin_lock_slowpath+0x10/0x10
? trace_hardirqs_on+0x18/0x150
do_raw_spin_lock+0x1d9/0x270
? kfree+0x14a/0x650
? __pfx_do_raw_spin_lock+0x10/0x10
? generic_shutdown_super+0x225/0x320
? lock_acquire+0xf6/0x140
kill_super_notify+0x86/0x230
kill_anon_super+0x42/0x60
btrfs_kill_super+0x3e/0x60 [btrfs]
deactivate_locked_super+0xa8/0x160
cleanup_mnt+0x1da/0x420
task_work_run+0x116/0x200
? __pfx_task_work_run+0x10/0x10
? __x64_sys_umount+0x10c/0x140
? find_held_lock+0x2b/0x80
? __pfx___x64_sys_umount+0x10/0x10
exit_to_user_mode_loop+0x133/0x170
do_syscall_64+0x201/0x7f0
? lock_release.part.0+0x1c/0x50
? do_faccessat+0x1ed/0x9a0
? __pfx_do_faccessat+0x10/0x10
? __pfx_from_kgid_munged+0x10/0x10
? trace_hardirqs_on_prepare+0x101/0x150
? do_syscall_64+0x137/0x7f0
? do_syscall_64+0x137/0x7f0
? trace_hardirqs_on_prepare+0x101/0x150
? lockdep_hardirqs_on_prepare.part.0+0x9b/0x150
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7fe027d41e4b
Code: c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 81 1f 0f 00 f7 d8
RSP: 002b:00007fff6e76f488 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007fe027d41e4b
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001c2f8c0
RBP: 00007fe027f29ffc R08: 0000000001c2fcd0 R09: 00007fe027e34ac0
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000001c29528
R13: 0000000001c2f8c0 R14: 0000000001c29420 R15: 0000000001c29860
</TASK>
rcu: INFO: rcu_preempt detected expedited stalls on CPUs/tasks: { 4-.... } 84119 jiffies s: 105653 root: 0x10/.
rcu: blocking rcu_node structures (internal RCU debug):
Sending NMI from CPU 6 to CPUs 4:
NMI backtrace for cpu 4
CPU: 4 UID: 0 PID: 859559 Comm: umount Tainted: G B D L 6.18.0-rc4-kts-btrfs #6 PREEMPT(lazy)
Tainted: [B]=BAD_PAGE, [D]=DIE, [L]=SOFTLOCKUP
Hardware name: Supermicro X10SLL-F/X10SLL-F, BIOS 3.0 04/24/2015
RIP: 0010:native_queued_spin_lock_slowpath+0x398/0xbe0
Code: 3d 0f b6 03 84 c0 74 36 48 b8 00 00 00 00 00 fc ff df 49 89 dc 49 89 dd 49 c1 ec 03 41 83 e5 07 49 01 c4 f3 90 41 0f b6 04 24 <44> 38 e8 7f 08 84 c0 0f 85 9b 06 00 00 0f b6 03 84 c0 75 e5 48 b8
RSP: 0018:ffff888461417b00 EFLAGS: 00000202
RAX: 0000000000000000 RBX: ffffffff90a75340 RCX: 0000000000000001
RDX: 0000000000000000 RSI: 0000000000000004 RDI: ffffffff90a75340
RBP: 1ffff1108c282f62 R08: ffffffff8f3bd7a3 R09: fffffbfff214ea68
R10: fffffbfff214ea69 R11: ffffffff90a75358 R12: fffffbfff214ea68
R13: 0000000000000000 R14: ffffed109bb8c84f R15: 0000000000000000
FS: 00007fe027c4b380(0000) GS:ffff88876b5d8000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fd3d77b3d90 CR3: 00000004c5318006 CR4: 00000000001726f0
Call Trace:
<TASK>
? __pfx_native_queued_spin_lock_slowpath+0x10/0x10
? trace_hardirqs_on+0x18/0x150
do_raw_spin_lock+0x1d9/0x270
? kfree+0x14a/0x650
? __pfx_do_raw_spin_lock+0x10/0x10
? generic_shutdown_super+0x225/0x320
? lock_acquire+0xf6/0x140
kill_super_notify+0x86/0x230
kill_anon_super+0x42/0x60
btrfs_kill_super+0x3e/0x60 [btrfs]
deactivate_locked_super+0xa8/0x160
cleanup_mnt+0x1da/0x420
task_work_run+0x116/0x200
? __pfx_task_work_run+0x10/0x10
? __x64_sys_umount+0x10c/0x140
? find_held_lock+0x2b/0x80
? __pfx___x64_sys_umount+0x10/0x10
exit_to_user_mode_loop+0x133/0x170
do_syscall_64+0x201/0x7f0
? lock_release.part.0+0x1c/0x50
? do_faccessat+0x1ed/0x9a0
? __pfx_do_faccessat+0x10/0x10
? __pfx_from_kgid_munged+0x10/0x10
? trace_hardirqs_on_prepare+0x101/0x150
? do_syscall_64+0x137/0x7f0
? do_syscall_64+0x137/0x7f0
? trace_hardirqs_on_prepare+0x101/0x150
? lockdep_hardirqs_on_prepare.part.0+0x9b/0x150
entry_SYSCALL_64_after_hwframe+0x76/0x7e
RIP: 0033:0x7fe027d41e4b
Code: c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 40 00 f3 0f 1e fa 31 f6 e9 05 00 00 00 0f 1f 44 00 00 f3 0f 1e fa b8 a6 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 05 c3 0f 1f 40 00 48 8b 15 81 1f 0f 00 f7 d8
RSP: 002b:00007fff6e76f488 EFLAGS: 00000246 ORIG_RAX: 00000000000000a6
RAX: 0000000000000000 RBX: 0000000000000000 RCX: 00007fe027d41e4b
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000001c2f8c0
RBP: 00007fe027f29ffc R08: 0000000001c2fcd0 R09: 00007fe027e34ac0
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000001c29528
R13: 0000000001c2f8c0 R14: 0000000001c29420 R15: 0000000001c29860
</TASK>
watchdog: BUG: soft lockup - CPU#4 stuck for 104s! [umount:859559]
...
^ permalink raw reply [flat|nested] 2+ messages in thread* Re: [bug report] fstests generic/085 btrfs hang with use-after-free at bdev_super_lock
2025-11-05 1:01 [bug report] fstests generic/085 btrfs hang with use-after-free at bdev_super_lock Shinichiro Kawasaki
@ 2025-11-05 6:54 ` Nilay Shroff
0 siblings, 0 replies; 2+ messages in thread
From: Nilay Shroff @ 2025-11-05 6:54 UTC (permalink / raw)
To: Shinichiro Kawasaki, linux-block@vger.kernel.org
Cc: linux-fsdevel@vger.kernel.org
Hi Shinichiro,
On 11/5/25 6:31 AM, Shinichiro Kawasaki wrote:
> When I run fstests for btrfs on regular null_blk devices, I observe KASAN
> slab-use-after-free in bdev_super_lock() followed by kernel hang. I observed it
> for the kernel v6.17-rc4 for the first time. And I still observe it for the
> latest kernel v6.18-rc4.
>
> The hang is recreated when I prepare eight of 5Gib size null_blk devices, assign
> one for TEST_DEV, and assign the other seven for SCRATCH_DEV_POOl. The hang
> happens at generic/085. It is sporadic. When I repeat the test case g085 only,
> the hang is not recreated. But when I repeat the whole fstests a few times, the
> hang is recreated in stable manner. It takes several hours to recreate the hang.
>
> I spent some weeks to bisect, and found the trigger commit is this:
>
> 370ac285f23a ("block: avoid cpu_hotplug_lock depedency on freeze_lock")
>
> The commit was included in the kernel tag v6.17-rc3. When I reverted the commit
> from v6.17-rc3, the hang disappeared (I repeated the whole fstests 5 times on
> two test nodes, and did not observe the hang). I'm not sure if the commit
> created the problem cause or revealed the hidden problem.
Thanks for the report!
This doesn't seem to be caused due to 370ac285f23a ("block: avoid cpu_hotplug_lock
depedency on freeze_lock"). However it appears that we're hitting a race while
freezing/unfreezing filesystem. It'd better if someone from fs team can take a
look at it.
Thanks,
--Nilay
^ permalink raw reply [flat|nested] 2+ messages in thread
end of thread, other threads:[~2025-11-05 6:54 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-11-05 1:01 [bug report] fstests generic/085 btrfs hang with use-after-free at bdev_super_lock Shinichiro Kawasaki
2025-11-05 6:54 ` Nilay Shroff
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).