* 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456
@ 2025-07-21 21:54 Mikhail Gavrilov
2025-07-22 0:51 ` Hillf Danton
2025-07-22 7:39 ` Thomas Hellström
0 siblings, 2 replies; 9+ messages in thread
From: Mikhail Gavrilov @ 2025-07-21 21:54 UTC (permalink / raw)
To: thomas.hellstrom, axboe, linux-block, Linux List Kernel Mailing,
Ming Lei, Linux regressions mailing list
[-- Attachment #1: Type: text/plain, Size: 28999 bytes --]
Hi Thomas,
After commit
ffa1e7ada456 ("block: Make request_queue lockdep splats show up earlier"),
I started seeing lockdep warnings about circular locking dependencies
in the kernel log every time I plug in a USB flash drive.
The warning looks like this:
[ 247.453773] sd 6:0:0:0: [sda] Attached SCSI removable disk
[ 247.486193] ======================================================
[ 247.486195] WARNING: possible circular locking dependency detected
[ 247.486197] 6.16.0-rc7 #36 Tainted: G L
[ 247.486199] ------------------------------------------------------
[ 247.486200] (udev-worker)/4257 is trying to acquire lock:
[ 247.486202] ffff88816b9c0650 (&q->elevator_lock){+.+.}-{4:4}, at:
elevator_change+0xb6/0x380
[ 247.486213]
but task is already holding lock:
[ 247.486214] ffff88816b9c00b0
(&q->q_usage_counter(io)#6){++++}-{0:0}, at:
blk_mq_freeze_queue_nomemsave+0x16/0x30
[ 247.486221]
which lock already depends on the new lock.
[ 247.486222]
the existing dependency chain (in reverse order) is:
[ 247.486224]
-> #3 (&q->q_usage_counter(io)#6){++++}-{0:0}:
[ 247.486228] __lock_acquire+0x56a/0xbe0
[ 247.486233] lock_acquire.part.0+0xc8/0x270
[ 247.486235] blk_alloc_queue+0x5cd/0x720
[ 247.486237] blk_mq_alloc_queue+0x14d/0x260
[ 247.486239] scsi_alloc_sdev+0x862/0xc90
[ 247.486242] scsi_probe_and_add_lun+0x4be/0xc10
[ 247.486245] __scsi_scan_target+0x18b/0x3b0
[ 247.486247] scsi_scan_channel+0xee/0x180
[ 247.486250] scsi_scan_host_selected+0x1fd/0x2c0
[ 247.486252] do_scan_async+0x42/0x450
[ 247.486254] async_run_entry_fn+0x94/0x540
[ 247.486258] process_one_work+0x87a/0x14d0
[ 247.486260] worker_thread+0x5f2/0xfd0
[ 247.486262] kthread+0x3b0/0x770
[ 247.486266] ret_from_fork+0x3ef/0x510
[ 247.486269] ret_from_fork_asm+0x1a/0x30
[ 247.486271]
-> #2 (fs_reclaim){+.+.}-{0:0}:
[ 247.486275] __lock_acquire+0x56a/0xbe0
[ 247.486277] lock_acquire.part.0+0xc8/0x270
[ 247.486279] fs_reclaim_acquire+0xd9/0x130
[ 247.486282] prepare_alloc_pages+0x153/0x5a0
[ 247.486284] __alloc_frozen_pages_noprof+0x142/0x490
[ 247.486286] __alloc_pages_noprof+0x12/0x210
[ 247.486288] pcpu_alloc_pages.isra.0+0xfa/0x4d0
[ 247.486291] pcpu_populate_chunk+0x39/0x80
[ 247.486293] pcpu_alloc_noprof+0x759/0xeb0
[ 247.486296] iommu_dma_init_fq+0x19c/0x7c0
[ 247.486299] iommu_dma_init_domain+0x53f/0x7f0
[ 247.486301] iommu_setup_dma_ops+0xd3/0x200
[ 247.486303] bus_iommu_probe+0x1f0/0x4b0
[ 247.486306] iommu_device_register+0x186/0x280
[ 247.486308] iommu_init_pci+0xc8c/0xd00
[ 247.486312] amd_iommu_init_pci+0x83/0x4e0
[ 247.486314] state_next+0x28f/0x5c0
[ 247.486317] iommu_go_to_state+0x2b/0x60
[ 247.486319] amd_iommu_init+0x21/0x60
[ 247.486321] pci_iommu_init+0x38/0x60
[ 247.486325] do_one_initcall+0xd2/0x450
[ 247.486327] do_initcalls+0x216/0x240
[ 247.486330] kernel_init_freeable+0x299/0x2d0
[ 247.486332] kernel_init+0x1c/0x150
[ 247.486335] ret_from_fork+0x3ef/0x510
[ 247.486337] ret_from_fork_asm+0x1a/0x30
[ 247.486338]
-> #1 (pcpu_alloc_mutex){+.+.}-{4:4}:
[ 247.486342] __lock_acquire+0x56a/0xbe0
[ 247.486344] lock_acquire.part.0+0xc8/0x270
[ 247.486346] __mutex_lock+0x1b2/0x1b70
[ 247.486348] pcpu_alloc_noprof+0x884/0xeb0
[ 247.486351] sbitmap_init_node+0x252/0x6a0
[ 247.486354] sbitmap_queue_init_node+0x2d/0x420
[ 247.486356] blk_mq_init_tags+0x154/0x2a0
[ 247.486359] blk_mq_alloc_map_and_rqs+0xa6/0x310
[ 247.486361] blk_mq_init_sched+0x2a4/0x580
[ 247.486363] elevator_switch+0x18b/0x630
[ 247.486365] elevator_change+0x209/0x380
[ 247.486368] elevator_set_default+0x22d/0x2a0
[ 247.486370] blk_register_queue+0x33a/0x490
[ 247.486372] __add_disk+0x5fd/0xd50
[ 247.486374] add_disk_fwnode+0x113/0x590
[ 247.486377] sd_probe+0x873/0xe10
[ 247.486380] really_probe+0x1de/0x890
[ 247.486383] __driver_probe_device+0x18c/0x390
[ 247.486385] driver_probe_device+0x4a/0x120
[ 247.486388] __device_attach_driver+0x156/0x280
[ 247.486389] bus_for_each_drv+0x111/0x1a0
[ 247.486392] __device_attach_async_helper+0x19c/0x240
[ 247.486394] async_run_entry_fn+0x94/0x540
[ 247.486396] process_one_work+0x87a/0x14d0
[ 247.486398] worker_thread+0x5f2/0xfd0
[ 247.486400] kthread+0x3b0/0x770
[ 247.486402] ret_from_fork+0x3ef/0x510
[ 247.486404] ret_from_fork_asm+0x1a/0x30
[ 247.486406]
-> #0 (&q->elevator_lock){+.+.}-{4:4}:
[ 247.486409] check_prev_add+0xe1/0xcf0
[ 247.486411] validate_chain+0x4cf/0x740
[ 247.486413] __lock_acquire+0x56a/0xbe0
[ 247.486414] lock_acquire.part.0+0xc8/0x270
[ 247.486416] __mutex_lock+0x1b2/0x1b70
[ 247.486418] elevator_change+0xb6/0x380
[ 247.486420] elv_iosched_store+0x24a/0x2c0
[ 247.486422] queue_attr_store+0x238/0x340
[ 247.486425] kernfs_fop_write_iter+0x39b/0x5a0
[ 247.486428] vfs_write+0x524/0xe70
[ 247.486430] ksys_write+0xff/0x200
[ 247.486432] do_syscall_64+0x98/0x3c0
[ 247.486435] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 247.486438]
other info that might help us debug this:
[ 247.486439] Chain exists of:
&q->elevator_lock --> fs_reclaim --> &q->q_usage_counter(io)#6
[ 247.486444] Possible unsafe locking scenario:
[ 247.486446] CPU0 CPU1
[ 247.486447] ---- ----
[ 247.486448] lock(&q->q_usage_counter(io)#6);
[ 247.486451] lock(fs_reclaim);
[ 247.486453] lock(&q->q_usage_counter(io)#6);
[ 247.486456] lock(&q->elevator_lock);
[ 247.486458]
*** DEADLOCK ***
[ 247.486459] 6 locks held by (udev-worker)/4257:
[ 247.486461] #0: ffff88817b49c458 (sb_writers#4){.+.+}-{0:0}, at:
ksys_write+0xff/0x200
[ 247.486467] #1: ffff88819e767490 (&of->mutex#2){+.+.}-{4:4}, at:
kernfs_fop_write_iter+0x25b/0x5a0
[ 247.486473] #2: ffff8883b352c8c0 (kn->active#204){.+.+}-{0:0}, at:
kernfs_fop_write_iter+0x27e/0x5a0
[ 247.486479] #3: ffff888169ecc3c0
(&set->update_nr_hwq_lock){.+.+}-{4:4}, at:
elv_iosched_store+0x1ba/0x2c0
[ 247.486484] #4: ffff88816b9c00b0
(&q->q_usage_counter(io)#6){++++}-{0:0}, at:
blk_mq_freeze_queue_nomemsave+0x16/0x30
[ 247.486490] #5: ffff88816b9c00f0
(&q->q_usage_counter(queue)#5){++++}-{0:0}, at:
blk_mq_freeze_queue_nomemsave+0x16/0x30
[ 247.486495]
stack backtrace:
[ 247.486498] CPU: 3 UID: 0 PID: 4257 Comm: (udev-worker) Tainted: G
L 6.16.0-rc7 #36 PREEMPT(lazy)
[ 247.486501] Tainted: [L]=SOFTLOCKUP
[ 247.486501] Hardware name: ASRock B650I Lightning WiFi/B650I
Lightning WiFi, BIOS 3.30 06/16/2025
[ 247.486503] Call Trace:
[ 247.486503] <TASK>
[ 247.486505] dump_stack_lvl+0x84/0xd0
[ 247.486509] print_circular_bug.cold+0x38/0x46
[ 247.486512] check_noncircular+0x14a/0x170
[ 247.486515] check_prev_add+0xe1/0xcf0
[ 247.486517] ? lock_acquire.part.0+0xc8/0x270
[ 247.486518] validate_chain+0x4cf/0x740
[ 247.486520] __lock_acquire+0x56a/0xbe0
[ 247.486522] lock_acquire.part.0+0xc8/0x270
[ 247.486524] ? elevator_change+0xb6/0x380
[ 247.486526] ? __lock_release.isra.0+0x1cb/0x340
[ 247.486527] ? rcu_is_watching+0x15/0xe0
[ 247.486530] ? __pfx___might_resched+0x10/0x10
[ 247.486532] ? elevator_change+0xb6/0x380
[ 247.486534] ? lock_acquire+0xf7/0x140
[ 247.486535] __mutex_lock+0x1b2/0x1b70
[ 247.486537] ? elevator_change+0xb6/0x380
[ 247.486539] ? elevator_change+0xb6/0x380
[ 247.486541] ? __pfx_xa_find_after+0x10/0x10
[ 247.486543] ? __pfx___mutex_lock+0x10/0x10
[ 247.486545] ? __pfx___might_resched+0x10/0x10
[ 247.486547] ? blk_mq_cancel_work_sync+0xc0/0x100
[ 247.486549] ? __pfx_blk_mq_cancel_work_sync+0x10/0x10
[ 247.486551] ? elevator_change+0xb6/0x380
[ 247.486553] elevator_change+0xb6/0x380
[ 247.486556] elv_iosched_store+0x24a/0x2c0
[ 247.486558] ? __pfx_elv_iosched_store+0x10/0x10
[ 247.486560] ? __pfx___might_resched+0x10/0x10
[ 247.486562] ? __pfx_sysfs_kf_write+0x10/0x10
[ 247.486564] queue_attr_store+0x238/0x340
[ 247.486566] ? __pfx_queue_attr_store+0x10/0x10
[ 247.486567] ? __lock_acquire+0x56a/0xbe0
[ 247.486569] ? lock_acquire.part.0+0xc8/0x270
[ 247.486570] ? find_held_lock+0x2b/0x80
[ 247.486572] ? __lock_release.isra.0+0x1cb/0x340
[ 247.486574] ? sysfs_file_kobj+0xb3/0x1c0
[ 247.486576] ? sysfs_file_kobj+0xbd/0x1c0
[ 247.486577] ? sysfs_kf_write+0x65/0x170
[ 247.486579] ? __pfx_sysfs_kf_write+0x10/0x10
[ 247.486580] kernfs_fop_write_iter+0x39b/0x5a0
[ 247.486582] ? __pfx_kernfs_fop_write_iter+0x10/0x10
[ 247.486584] vfs_write+0x524/0xe70
[ 247.486586] ? __pfx_vfs_write+0x10/0x10
[ 247.486588] ? __pfx___seccomp_filter+0x10/0x10
[ 247.486591] ksys_write+0xff/0x200
[ 247.486593] ? __pfx_ksys_write+0x10/0x10
[ 247.486595] ? syscall_trace_enter+0x8e/0x2e0
[ 247.486598] do_syscall_64+0x98/0x3c0
[ 247.486600] ? __x64_sys_openat+0x10e/0x210
[ 247.486602] ? do_syscall_64+0x161/0x3c0
[ 247.486604] ? do_sys_openat2+0x109/0x180
[ 247.486605] ? __pfx___x64_sys_openat+0x10/0x10
[ 247.486607] ? __pfx_do_sys_openat2+0x10/0x10
[ 247.486609] ? lockdep_hardirqs_on+0x8c/0x130
[ 247.486611] ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 247.486613] ? do_syscall_64+0x161/0x3c0
[ 247.486615] ? __x64_sys_openat+0x10e/0x210
[ 247.486616] ? lockdep_hardirqs_on+0x8c/0x130
[ 247.486618] ? __pfx___x64_sys_openat+0x10/0x10
[ 247.486621] ? lockdep_hardirqs_on+0x8c/0x130
[ 247.486623] ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 247.486624] ? do_syscall_64+0x161/0x3c0
[ 247.486626] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 247.486627] RIP: 0033:0x7f9afd67b5c6
[ 247.486642] Code: 5d e8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 75
19 83 e2 39 83 fa 08 75 11 e8 26 ff ff ff 66 0f 1f 44 00 00 48 8b 45
10 0f 05 <48> 8b 5d f8 c9 c3 0f 1f 40 00 f3 0f 1e fa 55 48 89 e5 48 83
ec 08
[ 247.486643] RSP: 002b:00007fff244eb210 EFLAGS: 00000202 ORIG_RAX:
0000000000000001
[ 247.486646] RAX: ffffffffffffffda RBX: 000055e41796e060 RCX: 00007f9afd67b5c6
[ 247.486647] RDX: 0000000000000003 RSI: 00007fff244eb560 RDI: 0000000000000014
[ 247.486647] RBP: 00007fff244eb230 R08: 0000000000000000 R09: 0000000000000000
[ 247.486648] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000003
[ 247.486649] R13: 0000000000000003 R14: 00007fff244eb560 R15: 00007fff244eb560
[ 247.486652] </TASK>
I bisected the issue to this commit:
ffa1e7ada456087c2402b37cd6b2863ced29aff0 is the first bad commit
commit ffa1e7ada456087c2402b37cd6b2863ced29aff0
Author: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Date: Tue Mar 18 10:55:48 2025 +0100
block: Make request_queue lockdep splats show up earlier
In recent kernels, there are lockdep splats around the
struct request_queue::io_lockdep_map, similar to [1], but they
typically don't show up until reclaim with writeback happens.
Having multiple kernel versions released with a known risc of kernel
deadlock during reclaim writeback should IMHO be addressed and
backported to -stable with the highest priority.
In order to have these lockdep splats show up earlier,
preferrably during system initialization, prime the
struct request_queue::io_lockdep_map as GFP_KERNEL reclaim-
tainted. This will instead lead to lockdep splats looking similar
to [2], but without the need for reclaim + writeback
happening.
[1]:
[ 189.762244] ======================================================
[ 189.762432] WARNING: possible circular locking dependency detected
[ 189.762441] 6.14.0-rc6-xe+ #6 Tainted: G U
[ 189.762450] ------------------------------------------------------
[ 189.762459] kswapd0/119 is trying to acquire lock:
[ 189.762467] ffff888110ceb710
(&q->q_usage_counter(io)#26){++++}-{0:0}, at: __submit_bio+0x76/0x230
[ 189.762485]
but task is already holding lock:
[ 189.762494] ffffffff834c97c0 (fs_reclaim){+.+.}-{0:0}, at:
balance_pgdat+0xbe/0xb00
[ 189.762507]
which lock already depends on the new lock.
[ 189.762519]
the existing dependency chain (in reverse order) is:
[ 189.762529]
-> #2 (fs_reclaim){+.+.}-{0:0}:
[ 189.762540] fs_reclaim_acquire+0xc5/0x100
[ 189.762548] kmem_cache_alloc_lru_noprof+0x4a/0x480
[ 189.762558] alloc_inode+0xaa/0xe0
[ 189.762566] iget_locked+0x157/0x330
[ 189.762573] kernfs_get_inode+0x1b/0x110
[ 189.762582] kernfs_get_tree+0x1b0/0x2e0
[ 189.762590] sysfs_get_tree+0x1f/0x60
[ 189.762597] vfs_get_tree+0x2a/0xf0
[ 189.762605] path_mount+0x4cd/0xc00
[ 189.762613] __x64_sys_mount+0x119/0x150
[ 189.762621] x64_sys_call+0x14f2/0x2310
[ 189.762630] do_syscall_64+0x91/0x180
[ 189.762637] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 189.762647]
-> #1 (&root->kernfs_rwsem){++++}-{3:3}:
[ 189.762659] down_write+0x3e/0xf0
[ 189.762667] kernfs_remove+0x32/0x60
[ 189.762676] sysfs_remove_dir+0x4f/0x60
[ 189.762685] __kobject_del+0x33/0xa0
[ 189.762709] kobject_del+0x13/0x30
[ 189.762716] elv_unregister_queue+0x52/0x80
[ 189.762725] elevator_switch+0x68/0x360
[ 189.762733] elv_iosched_store+0x14b/0x1b0
[ 189.762756] queue_attr_store+0x181/0x1e0
[ 189.762765] sysfs_kf_write+0x49/0x80
[ 189.762773] kernfs_fop_write_iter+0x17d/0x250
[ 189.762781] vfs_write+0x281/0x540
[ 189.762790] ksys_write+0x72/0xf0
[ 189.762798] __x64_sys_write+0x19/0x30
[ 189.762807] x64_sys_call+0x2a3/0x2310
[ 189.762815] do_syscall_64+0x91/0x180
[ 189.762823] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 189.762833]
-> #0 (&q->q_usage_counter(io)#26){++++}-{0:0}:
[ 189.762845] __lock_acquire+0x1525/0x2760
[ 189.762854] lock_acquire+0xca/0x310
[ 189.762861] blk_mq_submit_bio+0x8a2/0xba0
[ 189.762870] __submit_bio+0x76/0x230
[ 189.762878] submit_bio_noacct_nocheck+0x323/0x430
[ 189.762888] submit_bio_noacct+0x2cc/0x620
[ 189.762896] submit_bio+0x38/0x110
[ 189.762904] __swap_writepage+0xf5/0x380
[ 189.762912] swap_writepage+0x3c7/0x600
[ 189.762920] shmem_writepage+0x3da/0x4f0
[ 189.762929] pageout+0x13f/0x310
[ 189.762937] shrink_folio_list+0x61c/0xf60
[ 189.763261] evict_folios+0x378/0xcd0
[ 189.763584] try_to_shrink_lruvec+0x1b0/0x360
[ 189.763946] shrink_one+0x10e/0x200
[ 189.764266] shrink_node+0xc02/0x1490
[ 189.764586] balance_pgdat+0x563/0xb00
[ 189.764934] kswapd+0x1e8/0x430
[ 189.765249] kthread+0x10b/0x260
[ 189.765559] ret_from_fork+0x44/0x70
[ 189.765889] ret_from_fork_asm+0x1a/0x30
[ 189.766198]
other info that might help us debug this:
[ 189.767089] Chain exists of:
&q->q_usage_counter(io)#26 -->
&root->kernfs_rwsem --> fs_reclaim
[ 189.767971] Possible unsafe locking scenario:
[ 189.768555] CPU0 CPU1
[ 189.768849] ---- ----
[ 189.769136] lock(fs_reclaim);
[ 189.769421] lock(&root->kernfs_rwsem);
[ 189.769714] lock(fs_reclaim);
[ 189.770016] rlock(&q->q_usage_counter(io)#26);
[ 189.770305]
*** DEADLOCK ***
[ 189.771167] 1 lock held by kswapd0/119:
[ 189.771453] #0: ffffffff834c97c0 (fs_reclaim){+.+.}-{0:0}, at:
balance_pgdat+0xbe/0xb00
[ 189.771770]
stack backtrace:
[ 189.772351] CPU: 4 UID: 0 PID: 119 Comm: kswapd0 Tainted: G
U 6.14.0-rc6-xe+ #6
[ 189.772353] Tainted: [U]=USER
[ 189.772354] Hardware name: ASUS System Product Name/PRIME
B560M-A AC, BIOS 2001 02/01/2023
[ 189.772354] Call Trace:
[ 189.772355] <TASK>
[ 189.772356] dump_stack_lvl+0x6e/0xa0
[ 189.772359] dump_stack+0x10/0x18
[ 189.772360] print_circular_bug.cold+0x17a/0x1b7
[ 189.772363] check_noncircular+0x13a/0x150
[ 189.772365] ? __pfx_stack_trace_consume_entry+0x10/0x10
[ 189.772368] __lock_acquire+0x1525/0x2760
[ 189.772368] ? ret_from_fork_asm+0x1a/0x30
[ 189.772371] lock_acquire+0xca/0x310
[ 189.772372] ? __submit_bio+0x76/0x230
[ 189.772375] ? lock_release+0xd5/0x2c0
[ 189.772376] blk_mq_submit_bio+0x8a2/0xba0
[ 189.772378] ? __submit_bio+0x76/0x230
[ 189.772380] __submit_bio+0x76/0x230
[ 189.772382] ? trace_hardirqs_on+0x1e/0xe0
[ 189.772384] submit_bio_noacct_nocheck+0x323/0x430
[ 189.772386] ? submit_bio_noacct_nocheck+0x323/0x430
[ 189.772387] ? __might_sleep+0x58/0xa0
[ 189.772390] submit_bio_noacct+0x2cc/0x620
[ 189.772391] ? count_memcg_events+0x68/0x90
[ 189.772393] submit_bio+0x38/0x110
[ 189.772395] __swap_writepage+0xf5/0x380
[ 189.772396] swap_writepage+0x3c7/0x600
[ 189.772397] shmem_writepage+0x3da/0x4f0
[ 189.772401] pageout+0x13f/0x310
[ 189.772406] shrink_folio_list+0x61c/0xf60
[ 189.772409] ? isolate_folios+0xe80/0x16b0
[ 189.772410] ? mark_held_locks+0x46/0x90
[ 189.772412] evict_folios+0x378/0xcd0
[ 189.772414] ? evict_folios+0x34a/0xcd0
[ 189.772415] ? lock_is_held_type+0xa3/0x130
[ 189.772417] try_to_shrink_lruvec+0x1b0/0x360
[ 189.772420] shrink_one+0x10e/0x200
[ 189.772421] shrink_node+0xc02/0x1490
[ 189.772423] ? shrink_node+0xa08/0x1490
[ 189.772424] ? shrink_node+0xbd8/0x1490
[ 189.772425] ? mem_cgroup_iter+0x366/0x480
[ 189.772427] balance_pgdat+0x563/0xb00
[ 189.772428] ? balance_pgdat+0x563/0xb00
[ 189.772430] ? trace_hardirqs_on+0x1e/0xe0
[ 189.772431] ? finish_task_switch.isra.0+0xcb/0x330
[ 189.772433] ? __switch_to_asm+0x33/0x70
[ 189.772437] kswapd+0x1e8/0x430
[ 189.772438] ? __pfx_autoremove_wake_function+0x10/0x10
[ 189.772440] ? __pfx_kswapd+0x10/0x10
[ 189.772441] kthread+0x10b/0x260
[ 189.772443] ? __pfx_kthread+0x10/0x10
[ 189.772444] ret_from_fork+0x44/0x70
[ 189.772446] ? __pfx_kthread+0x10/0x10
[ 189.772447] ret_from_fork_asm+0x1a/0x30
[ 189.772450] </TASK>
[2]:
[ 8.760253] ======================================================
[ 8.760254] WARNING: possible circular locking dependency detected
[ 8.760255] 6.14.0-rc6-xe+ #7 Tainted: G U
[ 8.760256] ------------------------------------------------------
[ 8.760257] (udev-worker)/674 is trying to acquire lock:
[ 8.760259] ffff888100e39148 (&root->kernfs_rwsem){++++}-{3:3},
at: kernfs_remove+0x32/0x60
[ 8.760265]
but task is already holding lock:
[ 8.760266] ffff888110dc7680
(&q->q_usage_counter(io)#27){++++}-{0:0}, at:
blk_mq_freeze_queue_nomemsave+0x12/0x30
[ 8.760272]
which lock already depends on the new lock.
[ 8.760272]
the existing dependency chain (in reverse order) is:
[ 8.760273]
-> #2 (&q->q_usage_counter(io)#27){++++}-{0:0}:
[ 8.760276] blk_alloc_queue+0x30a/0x350
[ 8.760279] blk_mq_alloc_queue+0x6b/0xe0
[ 8.760281] scsi_alloc_sdev+0x276/0x3c0
[ 8.760284] scsi_probe_and_add_lun+0x22a/0x440
[ 8.760286] __scsi_scan_target+0x109/0x230
[ 8.760288] scsi_scan_channel+0x65/0xc0
[ 8.760290] scsi_scan_host_selected+0xff/0x140
[ 8.760292] do_scsi_scan_host+0xa7/0xc0
[ 8.760293] do_scan_async+0x1c/0x160
[ 8.760295] async_run_entry_fn+0x32/0x150
[ 8.760299] process_one_work+0x224/0x5f0
[ 8.760302] worker_thread+0x1d4/0x3e0
[ 8.760304] kthread+0x10b/0x260
[ 8.760306] ret_from_fork+0x44/0x70
[ 8.760309] ret_from_fork_asm+0x1a/0x30
[ 8.760312]
-> #1 (fs_reclaim){+.+.}-{0:0}:
[ 8.760315] fs_reclaim_acquire+0xc5/0x100
[ 8.760317] kmem_cache_alloc_lru_noprof+0x4a/0x480
[ 8.760319] alloc_inode+0xaa/0xe0
[ 8.760322] iget_locked+0x157/0x330
[ 8.760323] kernfs_get_inode+0x1b/0x110
[ 8.760325] kernfs_get_tree+0x1b0/0x2e0
[ 8.760327] sysfs_get_tree+0x1f/0x60
[ 8.760329] vfs_get_tree+0x2a/0xf0
[ 8.760332] path_mount+0x4cd/0xc00
[ 8.760334] __x64_sys_mount+0x119/0x150
[ 8.760336] x64_sys_call+0x14f2/0x2310
[ 8.760338] do_syscall_64+0x91/0x180
[ 8.760340] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 8.760342]
-> #0 (&root->kernfs_rwsem){++++}-{3:3}:
[ 8.760345] __lock_acquire+0x1525/0x2760
[ 8.760347] lock_acquire+0xca/0x310
[ 8.760348] down_write+0x3e/0xf0
[ 8.760350] kernfs_remove+0x32/0x60
[ 8.760351] sysfs_remove_dir+0x4f/0x60
[ 8.760353] __kobject_del+0x33/0xa0
[ 8.760355] kobject_del+0x13/0x30
[ 8.760356] elv_unregister_queue+0x52/0x80
[ 8.760358] elevator_switch+0x68/0x360
[ 8.760360] elv_iosched_store+0x14b/0x1b0
[ 8.760362] queue_attr_store+0x181/0x1e0
[ 8.760364] sysfs_kf_write+0x49/0x80
[ 8.760366] kernfs_fop_write_iter+0x17d/0x250
[ 8.760367] vfs_write+0x281/0x540
[ 8.760370] ksys_write+0x72/0xf0
[ 8.760372] __x64_sys_write+0x19/0x30
[ 8.760374] x64_sys_call+0x2a3/0x2310
[ 8.760376] do_syscall_64+0x91/0x180
[ 8.760377] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 8.760380]
other info that might help us debug this:
[ 8.760380] Chain exists of:
&root->kernfs_rwsem --> fs_reclaim -->
&q->q_usage_counter(io)#27
[ 8.760384] Possible unsafe locking scenario:
[ 8.760384] CPU0 CPU1
[ 8.760385] ---- ----
[ 8.760385] lock(&q->q_usage_counter(io)#27);
[ 8.760387] lock(fs_reclaim);
[ 8.760388]
lock(&q->q_usage_counter(io)#27);
[ 8.760390] lock(&root->kernfs_rwsem);
[ 8.760391]
*** DEADLOCK ***
[ 8.760391] 6 locks held by (udev-worker)/674:
[ 8.760392] #0: ffff8881209ac420 (sb_writers#4){.+.+}-{0:0},
at: ksys_write+0x72/0xf0
[ 8.760398] #1: ffff88810c80f488 (&of->mutex#2){+.+.}-{3:3},
at: kernfs_fop_write_iter+0x136/0x250
[ 8.760402] #2: ffff888125d1d330 (kn->active#101){.+.+}-{0:0},
at: kernfs_fop_write_iter+0x13f/0x250
[ 8.760406] #3: ffff888110dc7bb0 (&q->sysfs_lock){+.+.}-{3:3},
at: queue_attr_store+0x148/0x1e0
[ 8.760411] #4: ffff888110dc7680
(&q->q_usage_counter(io)#27){++++}-{0:0}, at:
blk_mq_freeze_queue_nomemsave+0x12/0x30
[ 8.760416] #5: ffff888110dc76b8
(&q->q_usage_counter(queue)#27){++++}-{0:0}, at:
blk_mq_freeze_queue_nomemsave+0x12/0x30
[ 8.760421]
stack backtrace:
[ 8.760422] CPU: 7 UID: 0 PID: 674 Comm: (udev-worker) Tainted:
G U 6.14.0-rc6-xe+ #7
[ 8.760424] Tainted: [U]=USER
[ 8.760425] Hardware name: ASUS System Product Name/PRIME
B560M-A AC, BIOS 2001 02/01/2023
[ 8.760426] Call Trace:
[ 8.760427] <TASK>
[ 8.760428] dump_stack_lvl+0x6e/0xa0
[ 8.760431] dump_stack+0x10/0x18
[ 8.760433] print_circular_bug.cold+0x17a/0x1b7
[ 8.760437] check_noncircular+0x13a/0x150
[ 8.760441] ? save_trace+0x54/0x360
[ 8.760445] __lock_acquire+0x1525/0x2760
[ 8.760446] ? irqentry_exit+0x3a/0xb0
[ 8.760448] ? sysvec_apic_timer_interrupt+0x57/0xc0
[ 8.760452] lock_acquire+0xca/0x310
[ 8.760453] ? kernfs_remove+0x32/0x60
[ 8.760457] down_write+0x3e/0xf0
[ 8.760459] ? kernfs_remove+0x32/0x60
[ 8.760460] kernfs_remove+0x32/0x60
[ 8.760462] sysfs_remove_dir+0x4f/0x60
[ 8.760464] __kobject_del+0x33/0xa0
[ 8.760466] kobject_del+0x13/0x30
[ 8.760467] elv_unregister_queue+0x52/0x80
[ 8.760470] elevator_switch+0x68/0x360
[ 8.760472] elv_iosched_store+0x14b/0x1b0
[ 8.760475] queue_attr_store+0x181/0x1e0
[ 8.760479] ? lock_acquire+0xca/0x310
[ 8.760480] ? kernfs_fop_write_iter+0x13f/0x250
[ 8.760482] ? lock_is_held_type+0xa3/0x130
[ 8.760485] sysfs_kf_write+0x49/0x80
[ 8.760487] kernfs_fop_write_iter+0x17d/0x250
[ 8.760489] vfs_write+0x281/0x540
[ 8.760494] ksys_write+0x72/0xf0
[ 8.760497] __x64_sys_write+0x19/0x30
[ 8.760499] x64_sys_call+0x2a3/0x2310
[ 8.760502] do_syscall_64+0x91/0x180
[ 8.760504] ? trace_hardirqs_off+0x5d/0xe0
[ 8.760506] ? handle_softirqs+0x479/0x4d0
[ 8.760508] ? hrtimer_interrupt+0x13f/0x280
[ 8.760511] ? irqentry_exit_to_user_mode+0x8b/0x260
[ 8.760513] ? clear_bhb_loop+0x15/0x70
[ 8.760515] ? clear_bhb_loop+0x15/0x70
[ 8.760516] ? clear_bhb_loop+0x15/0x70
[ 8.760518] entry_SYSCALL_64_after_hwframe+0x76/0x7e
[ 8.760520] RIP: 0033:0x7aa3bf2f5504
[ 8.760522] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f
1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d c5 8b 10 00 00 74 13 b8 01 00
00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83 ec
20 48 89
[ 8.760523] RSP: 002b:00007ffc1e3697d8 EFLAGS: 00000202
ORIG_RAX: 0000000000000001
[ 8.760526] RAX: ffffffffffffffda RBX: 0000000000000003 RCX:
00007aa3bf2f5504
[ 8.760527] RDX: 0000000000000003 RSI: 00007ffc1e369ae0 RDI:
000000000000001c
[ 8.760528] RBP: 00007ffc1e369800 R08: 00007aa3bf3f51c8 R09:
00007ffc1e3698b0
[ 8.760528] R10: 0000000000000000 R11: 0000000000000202 R12:
0000000000000003
[ 8.760529] R13: 00007ffc1e369ae0 R14: 0000613ccf21f2f0 R15:
00007aa3bf3f4e80
[ 8.760533] </TASK>
v2:
- Update a code comment to increase readability (Ming Lei).
Cc: Jens Axboe <axboe@kernel.dk>
Cc: linux-block@vger.kernel.org
Cc: linux-kernel@vger.kernel.org
Cc: Ming Lei <ming.lei@redhat.com>
Signed-off-by: Thomas Hellström <thomas.hellstrom@linux.intel.com>
Reviewed-by: Ming Lei <ming.lei@redhat.com>
Link: https://lore.kernel.org/r/20250318095548.5187-1-thomas.hellstrom@linux.intel.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
block/blk-core.c | 6 ++++++
1 file changed, 6 insertions(+)
Reverting commit ffa1e7ada456 on top of 6.16-rc7 completely eliminates
the warning. No lockdep splats occur when plugging in USB devices
after the revert.
The full dmesg trace, kernel config, and hardware probe are attached below:
Hardware: https://linux-hardware.org/?probe=347fa4c2c6
Kernel config: (attached)
Full kernel log: (attached)
This looks like a false-positive introduced by priming io_lockdep_map
earlier, as explained in the commit message. However, it creates noise
for regular users and may obscure more critical lockdep reports.
Thanks for looking into this.
--
Best Regards,
Mike Gavrilov.
[-- Attachment #2: .config.zip --]
[-- Type: application/zip, Size: 69582 bytes --]
[-- Attachment #3: dmesg-6.16.0-rc7.zip --]
[-- Type: application/zip, Size: 47603 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456
2025-07-21 21:54 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456 Mikhail Gavrilov
@ 2025-07-22 0:51 ` Hillf Danton
2025-07-22 7:11 ` Mikhail Gavrilov
2025-07-22 12:14 ` Jens Axboe
2025-07-22 7:39 ` Thomas Hellström
1 sibling, 2 replies; 9+ messages in thread
From: Hillf Danton @ 2025-07-22 0:51 UTC (permalink / raw)
To: Mikhail Gavrilov
Cc: thomas.hellstrom, axboe, linux-block, linux-kernel, Ming Lei,
regressions
On Tue, 22 Jul 2025 02:54:46 +0500 Mikhail Gavrilov wrote:
> Hi Thomas,
>
> After commit
> ffa1e7ada456 ("block: Make request_queue lockdep splats show up earlier"),
> I started seeing lockdep warnings about circular locking dependencies
> in the kernel log every time I plug in a USB flash drive.
>
> The warning looks like this:
> [ 247.453773] sd 6:0:0:0: [sda] Attached SCSI removable disk
>
> [ 247.486193] ======================================================
> [ 247.486195] WARNING: possible circular locking dependency detected
> [ 247.486197] 6.16.0-rc7 #36 Tainted: G L
> [ 247.486199] ------------------------------------------------------
> [ 247.486200] (udev-worker)/4257 is trying to acquire lock:
> [ 247.486202] ffff88816b9c0650 (&q->elevator_lock){+.+.}-{4:4}, at:
> elevator_change+0xb6/0x380
> [ 247.486213]
> but task is already holding lock:
> [ 247.486214] ffff88816b9c00b0
> (&q->q_usage_counter(io)#6){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x16/0x30
> [ 247.486221]
> which lock already depends on the new lock.
>
> [ 247.486222]
> the existing dependency chain (in reverse order) is:
> [ 247.486224]
> -> #3 (&q->q_usage_counter(io)#6){++++}-{0:0}:
> [ 247.486228] __lock_acquire+0x56a/0xbe0
> [ 247.486233] lock_acquire.part.0+0xc8/0x270
> [ 247.486235] blk_alloc_queue+0x5cd/0x720
> [ 247.486237] blk_mq_alloc_queue+0x14d/0x260
> [ 247.486239] scsi_alloc_sdev+0x862/0xc90
> [ 247.486242] scsi_probe_and_add_lun+0x4be/0xc10
> [ 247.486245] __scsi_scan_target+0x18b/0x3b0
> [ 247.486247] scsi_scan_channel+0xee/0x180
> [ 247.486250] scsi_scan_host_selected+0x1fd/0x2c0
> [ 247.486252] do_scan_async+0x42/0x450
> [ 247.486254] async_run_entry_fn+0x94/0x540
> [ 247.486258] process_one_work+0x87a/0x14d0
> [ 247.486260] worker_thread+0x5f2/0xfd0
> [ 247.486262] kthread+0x3b0/0x770
> [ 247.486266] ret_from_fork+0x3ef/0x510
> [ 247.486269] ret_from_fork_asm+0x1a/0x30
> [ 247.486271]
> -> #2 (fs_reclaim){+.+.}-{0:0}:
> [ 247.486275] __lock_acquire+0x56a/0xbe0
> [ 247.486277] lock_acquire.part.0+0xc8/0x270
> [ 247.486279] fs_reclaim_acquire+0xd9/0x130
> [ 247.486282] prepare_alloc_pages+0x153/0x5a0
> [ 247.486284] __alloc_frozen_pages_noprof+0x142/0x490
> [ 247.486286] __alloc_pages_noprof+0x12/0x210
> [ 247.486288] pcpu_alloc_pages.isra.0+0xfa/0x4d0
> [ 247.486291] pcpu_populate_chunk+0x39/0x80
> [ 247.486293] pcpu_alloc_noprof+0x759/0xeb0
> [ 247.486296] iommu_dma_init_fq+0x19c/0x7c0
> [ 247.486299] iommu_dma_init_domain+0x53f/0x7f0
> [ 247.486301] iommu_setup_dma_ops+0xd3/0x200
> [ 247.486303] bus_iommu_probe+0x1f0/0x4b0
> [ 247.486306] iommu_device_register+0x186/0x280
> [ 247.486308] iommu_init_pci+0xc8c/0xd00
> [ 247.486312] amd_iommu_init_pci+0x83/0x4e0
> [ 247.486314] state_next+0x28f/0x5c0
> [ 247.486317] iommu_go_to_state+0x2b/0x60
> [ 247.486319] amd_iommu_init+0x21/0x60
> [ 247.486321] pci_iommu_init+0x38/0x60
> [ 247.486325] do_one_initcall+0xd2/0x450
> [ 247.486327] do_initcalls+0x216/0x240
> [ 247.486330] kernel_init_freeable+0x299/0x2d0
> [ 247.486332] kernel_init+0x1c/0x150
> [ 247.486335] ret_from_fork+0x3ef/0x510
> [ 247.486337] ret_from_fork_asm+0x1a/0x30
> [ 247.486338]
> -> #1 (pcpu_alloc_mutex){+.+.}-{4:4}:
> [ 247.486342] __lock_acquire+0x56a/0xbe0
> [ 247.486344] lock_acquire.part.0+0xc8/0x270
> [ 247.486346] __mutex_lock+0x1b2/0x1b70
> [ 247.486348] pcpu_alloc_noprof+0x884/0xeb0
> [ 247.486351] sbitmap_init_node+0x252/0x6a0
> [ 247.486354] sbitmap_queue_init_node+0x2d/0x420
> [ 247.486356] blk_mq_init_tags+0x154/0x2a0
> [ 247.486359] blk_mq_alloc_map_and_rqs+0xa6/0x310
> [ 247.486361] blk_mq_init_sched+0x2a4/0x580
> [ 247.486363] elevator_switch+0x18b/0x630
> [ 247.486365] elevator_change+0x209/0x380
> [ 247.486368] elevator_set_default+0x22d/0x2a0
> [ 247.486370] blk_register_queue+0x33a/0x490
> [ 247.486372] __add_disk+0x5fd/0xd50
> [ 247.486374] add_disk_fwnode+0x113/0x590
> [ 247.486377] sd_probe+0x873/0xe10
> [ 247.486380] really_probe+0x1de/0x890
> [ 247.486383] __driver_probe_device+0x18c/0x390
> [ 247.486385] driver_probe_device+0x4a/0x120
> [ 247.486388] __device_attach_driver+0x156/0x280
> [ 247.486389] bus_for_each_drv+0x111/0x1a0
> [ 247.486392] __device_attach_async_helper+0x19c/0x240
> [ 247.486394] async_run_entry_fn+0x94/0x540
> [ 247.486396] process_one_work+0x87a/0x14d0
> [ 247.486398] worker_thread+0x5f2/0xfd0
> [ 247.486400] kthread+0x3b0/0x770
> [ 247.486402] ret_from_fork+0x3ef/0x510
> [ 247.486404] ret_from_fork_asm+0x1a/0x30
> [ 247.486406]
> -> #0 (&q->elevator_lock){+.+.}-{4:4}:
> [ 247.486409] check_prev_add+0xe1/0xcf0
> [ 247.486411] validate_chain+0x4cf/0x740
> [ 247.486413] __lock_acquire+0x56a/0xbe0
> [ 247.486414] lock_acquire.part.0+0xc8/0x270
> [ 247.486416] __mutex_lock+0x1b2/0x1b70
> [ 247.486418] elevator_change+0xb6/0x380
> [ 247.486420] elv_iosched_store+0x24a/0x2c0
> [ 247.486422] queue_attr_store+0x238/0x340
> [ 247.486425] kernfs_fop_write_iter+0x39b/0x5a0
> [ 247.486428] vfs_write+0x524/0xe70
> [ 247.486430] ksys_write+0xff/0x200
> [ 247.486432] do_syscall_64+0x98/0x3c0
> [ 247.486435] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 247.486438]
> other info that might help us debug this:
>
> [ 247.486439] Chain exists of:
> &q->elevator_lock --> fs_reclaim --> &q->q_usage_counter(io)#6
>
> [ 247.486444] Possible unsafe locking scenario:
>
> [ 247.486446] CPU0 CPU1
> [ 247.486447] ---- ----
> [ 247.486448] lock(&q->q_usage_counter(io)#6);
> [ 247.486451] lock(fs_reclaim);
> [ 247.486453] lock(&q->q_usage_counter(io)#6);
> [ 247.486456] lock(&q->elevator_lock);
> [ 247.486458]
> *** DEADLOCK ***
Try the diff that serializes elevator_change() with q->elevator_lock if
reproducer is available.
--- x/block/elevator.c
+++ y/block/elevator.c
@@ -661,6 +661,7 @@ static int elevator_change(struct reques
unsigned int memflags;
int ret = 0;
+ /* updaters should be serialized */
lockdep_assert_held(&q->tag_set->update_nr_hwq_lock);
memflags = blk_mq_freeze_queue(q);
@@ -674,11 +675,11 @@ static int elevator_change(struct reques
* Disk isn't added yet, so verifying queue lock only manually.
*/
blk_mq_cancel_work_sync(q);
+ blk_mq_unfreeze_queue(q, memflags);
mutex_lock(&q->elevator_lock);
if (!(q->elevator && elevator_match(q->elevator->type, ctx->name)))
ret = elevator_switch(q, ctx);
mutex_unlock(&q->elevator_lock);
- blk_mq_unfreeze_queue(q, memflags);
if (!ret)
ret = elevator_change_done(q, ctx);
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456
2025-07-22 0:51 ` Hillf Danton
@ 2025-07-22 7:11 ` Mikhail Gavrilov
2025-07-23 1:03 ` Hillf Danton
2025-07-22 12:14 ` Jens Axboe
1 sibling, 1 reply; 9+ messages in thread
From: Mikhail Gavrilov @ 2025-07-22 7:11 UTC (permalink / raw)
To: Hillf Danton
Cc: thomas.hellstrom, axboe, linux-block, linux-kernel, Ming Lei,
regressions
[-- Attachment #1: Type: text/plain, Size: 7202 bytes --]
On Tue, Jul 22, 2025 at 5:51 AM Hillf Danton <hdanton@sina.com> wrote:
>
> Try the diff that serializes elevator_change() with q->elevator_lock if
> reproducer is available.
>
> --- x/block/elevator.c
> +++ y/block/elevator.c
> @@ -661,6 +661,7 @@ static int elevator_change(struct reques
> unsigned int memflags;
> int ret = 0;
>
> + /* updaters should be serialized */
> lockdep_assert_held(&q->tag_set->update_nr_hwq_lock);
>
> memflags = blk_mq_freeze_queue(q);
> @@ -674,11 +675,11 @@ static int elevator_change(struct reques
> * Disk isn't added yet, so verifying queue lock only manually.
> */
> blk_mq_cancel_work_sync(q);
> + blk_mq_unfreeze_queue(q, memflags);
> mutex_lock(&q->elevator_lock);
> if (!(q->elevator && elevator_match(q->elevator->type, ctx->name)))
> ret = elevator_switch(q, ctx);
> mutex_unlock(&q->elevator_lock);
> - blk_mq_unfreeze_queue(q, memflags);
> if (!ret)
> ret = elevator_change_done(q, ctx);
>
Hi Hillf,
Thanks for the patch.
I tested your proposed diff that serializes elevator_change() with
q->elevator_lock. Unfortunately, instead of the previous lockdep
warning, I'm now seeing a soft lockup warning.
Here is the relevant excerpt from the kernel log:
[ 78.573292] sd 6:0:0:0: [sda] Assuming drive cache: write through
[ 78.581496] ------------[ cut here ]------------
[ 78.581507] WARNING: CPU: 7 PID: 300 at block/elevator.c:578
elevator_switch+0x512/0x630
[ 78.581515] Modules linked in: uas usb_storage uinput snd_seq_dummy
snd_hrtimer nft_queue nfnetlink_queue nf_conntrack_netbios_ns
nf_conntrack_broadcast nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib
nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct
nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4
nf_tables qrtr bnep binfmt_misc intel_rapl_msr amd_atl
intel_rapl_common mt7921e mt7921_common mt792x_lib mt76_connac_lib
mt76 snd_hda_codec_realtek snd_hda_codec_generic edac_mce_amd
snd_hda_scodec_component snd_hda_codec_hdmi btusb mac80211 btrtl
kvm_amd btintel btbcm snd_hda_intel btmtk snd_intel_dspcfg bluetooth
snd_intel_sdw_acpi kvm snd_hda_codec snd_hda_core vfat snd_hwdep fat
snd_seq spd5118 snd_seq_device libarc4 irqbypass snd_pcm cfg80211
wmi_bmof rapl snd_timer pcspkr k10temp i2c_piix4 snd joydev i2c_smbus
rfkill soundcore gpio_amdpt gpio_generic loop zram lz4hc_compress
lz4_compress amdgpu amdxcp i2c_algo_bit drm_ttm_helper ttm drm_exec
gpu_sched drm_suballoc_helper nvme
[ 78.581629] drm_panel_backlight_quirks drm_buddy polyval_clmulni
r8169 drm_display_helper ghash_clmulni_intel nvme_core sha512_ssse3
sha1_ssse3 cec sp5100_tco nvme_keyring realtek nvme_auth video wmi
sunrpc iscsi_tcp libiscsi_tcp libiscsi scsi_transport_iscsi i2c_dev
fuse nfnetlink
[ 78.581663] CPU: 7 UID: 0 PID: 300 Comm: kworker/u129:6 Tainted: G
L 6.16.0-rc7-with-fix+ #38 PREEMPT(lazy)
[ 78.581668] Tainted: [L]=SOFTLOCKUP
[ 78.581669] Hardware name: ASRock B650I Lightning WiFi/B650I
Lightning WiFi, BIOS 3.30 06/16/2025
[ 78.581672] Workqueue: async async_run_entry_fn
[ 78.581677] RIP: 0010:elevator_switch+0x512/0x630
[ 78.581681] Code: 0f 85 20 fd ff ff 48 c7 c2 a0 08 27 a6 be 4a 03
00 00 48 c7 c7 00 09 27 a6 c6 05 1e e1 d7 04 01 e8 03 31 b0 fe e9 fc
fc ff ff <0f> 0b e9 34 fb ff ff 48 b8 00 00 00 00 00 fc ff df 4c 89 e2
48 c1
[ 78.581684] RSP: 0018:ffffc9000326f7d0 EFLAGS: 00010246
[ 78.581687] RAX: 0000000000000000 RBX: ffff8883d97334d0 RCX: 1ffff9200064dedc
[ 78.581689] RDX: 1ffff1107b2e679d RSI: 0000000000000000 RDI: ffff8883d9733ce8
[ 78.581691] RBP: 0000000000000000 R08: ffffffffa5c33e41 R09: fffffbfff515a0fc
[ 78.581693] R10: ffffc9000326f808 R11: 0000000000000001 R12: ffffc9000326f860
[ 78.581694] R13: ffff8883d9733ab0 R14: 00000000ffffffff R15: ffff8881eff5c098
[ 78.581696] FS: 0000000000000000(0000) GS:ffff889026d50000(0000)
knlGS:0000000000000000
[ 78.581698] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 78.581700] CR2: 000056377a0b42d0 CR3: 00000002206a6000 CR4: 0000000000f50ef0
[ 78.581702] PKRU: 55555554
[ 78.581704] Call Trace:
[ 78.581706] <TASK>
[ 78.581710] elevator_change+0x1f1/0x2d0
[ 78.581715] elevator_set_default+0x22d/0x2a0
[ 78.581719] ? __pfx_elevator_set_default+0x10/0x10
[ 78.581724] ? disk_register_independent_access_ranges+0x24f/0x310
[ 78.581730] blk_register_queue+0x33a/0x490
[ 78.581736] __add_disk+0x5fd/0xd50
[ 78.581742] add_disk_fwnode+0x113/0x590
[ 78.581748] sd_probe+0x873/0xe10
[ 78.581755] really_probe+0x1de/0x890
[ 78.581761] ? __pfx___device_attach_driver+0x10/0x10
[ 78.581763] __driver_probe_device+0x18c/0x390
[ 78.581766] ? lockdep_hardirqs_on+0x8c/0x130
[ 78.581771] driver_probe_device+0x4a/0x120
[ 78.581775] __device_attach_driver+0x156/0x280
[ 78.581778] bus_for_each_drv+0x111/0x1a0
[ 78.581782] ? __pfx_bus_for_each_drv+0x10/0x10
[ 78.581786] ? trace_hardirqs_on+0x18/0x150
[ 78.581790] ? lockdep_hardirqs_on+0x8c/0x130
[ 78.581792] ? __raw_spin_unlock_irqrestore+0x5d/0x80
[ 78.581796] ? __raw_spin_unlock_irqrestore+0x46/0x80
[ 78.581800] __device_attach_async_helper+0x19c/0x240
[ 78.581804] ? __pfx___device_attach_async_helper+0x10/0x10
[ 78.581808] ? seqcount_lockdep_reader_access.constprop.0+0xa5/0xb0
[ 78.581812] ? ktime_get+0x63/0x180
[ 78.581817] async_run_entry_fn+0x94/0x540
[ 78.581821] process_one_work+0x87a/0x14d0
[ 78.581830] ? __pfx_process_one_work+0x10/0x10
[ 78.581833] ? find_held_lock+0x2b/0x80
[ 78.581840] ? assign_work+0x156/0x390
[ 78.581845] worker_thread+0x5f2/0xfd0
[ 78.581853] ? __pfx_worker_thread+0x10/0x10
[ 78.581855] kthread+0x3b0/0x770
[ 78.581859] ? local_clock_noinstr+0x68/0x130
[ 78.581862] ? __pfx_kthread+0x10/0x10
[ 78.581867] ? rcu_is_watching+0x15/0xe0
[ 78.581871] ? __pfx_kthread+0x10/0x10
[ 78.581875] ret_from_fork+0x3ef/0x510
[ 78.581879] ? __pfx_kthread+0x10/0x10
[ 78.581881] ? __pfx_kthread+0x10/0x10
[ 78.581885] ret_from_fork_asm+0x1a/0x30
[ 78.581894] </TASK>
[ 78.581896] irq event stamp: 1002669
[ 78.581898] hardirqs last enabled at (1002675):
[<ffffffffa282630e>] __up_console_sem+0x7e/0x90
[ 78.581902] hardirqs last disabled at (1002680):
[<ffffffffa28262f3>] __up_console_sem+0x63/0x90
[ 78.581904] softirqs last enabled at (999948):
[<ffffffffa25f8c49>] handle_softirqs+0x579/0x840
[ 78.581908] softirqs last disabled at (999941):
[<ffffffffa25f9046>] __irq_exit_rcu+0x126/0x240
[ 78.581910] ---[ end trace 0000000000000000 ]---
[ 78.637462] GPT:Primary header thinks Alt. header is not at the end
of the disk.
[ 78.637472] GPT:2097151 != 3907583
This happens after plugging in a USB flash stick (sd 6:0:0:0) with the
patched kernel.
Full dmesg trace is attached below.
Let me know if you'd like me to try additional debugging or patches.
Thanks for looking into this!
--
Best Regards,
Mike Gavrilov.
[-- Attachment #2: dmesg-6.16.0-rc7-with-fix.zip --]
[-- Type: application/zip, Size: 46529 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456
2025-07-21 21:54 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456 Mikhail Gavrilov
2025-07-22 0:51 ` Hillf Danton
@ 2025-07-22 7:39 ` Thomas Hellström
1 sibling, 0 replies; 9+ messages in thread
From: Thomas Hellström @ 2025-07-22 7:39 UTC (permalink / raw)
To: Mikhail Gavrilov, axboe, linux-block, Linux List Kernel Mailing,
Ming Lei, Linux regressions mailing list
Hi Mikhail,
On Tue, 2025-07-22 at 02:54 +0500, Mikhail Gavrilov wrote:
> Hi Thomas,
>
> After commit
> ffa1e7ada456 ("block: Make request_queue lockdep splats show up
> earlier"),
> I started seeing lockdep warnings about circular locking dependencies
> in the kernel log every time I plug in a USB flash drive.
This means that there is a real circular locking dependency problem.
However the commit you are pointing to only makes it show up early.
Without the commit a similar (but not identical) lockdep splat would
have shown up under memory pressure with swapping activated.
Since I'm not the right person to fix the underlying locking order
violation, this will likely get picked up by someone on the linux-block
list.
Thanks for reporting,
Thomas.
>
> The warning looks like this:
> [ 247.453773] sd 6:0:0:0: [sda] Attached SCSI removable disk
>
> [ 247.486193] ======================================================
> [ 247.486195] WARNING: possible circular locking dependency detected
> [ 247.486197] 6.16.0-rc7 #36 Tainted: G L
> [ 247.486199] ------------------------------------------------------
> [ 247.486200] (udev-worker)/4257 is trying to acquire lock:
> [ 247.486202] ffff88816b9c0650 (&q->elevator_lock){+.+.}-{4:4}, at:
> elevator_change+0xb6/0x380
> [ 247.486213]
> but task is already holding lock:
> [ 247.486214] ffff88816b9c00b0
> (&q->q_usage_counter(io)#6){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x16/0x30
> [ 247.486221]
> which lock already depends on the new lock.
>
> [ 247.486222]
> the existing dependency chain (in reverse order) is:
> [ 247.486224]
> -> #3 (&q->q_usage_counter(io)#6){++++}-{0:0}:
> [ 247.486228] __lock_acquire+0x56a/0xbe0
> [ 247.486233] lock_acquire.part.0+0xc8/0x270
> [ 247.486235] blk_alloc_queue+0x5cd/0x720
> [ 247.486237] blk_mq_alloc_queue+0x14d/0x260
> [ 247.486239] scsi_alloc_sdev+0x862/0xc90
> [ 247.486242] scsi_probe_and_add_lun+0x4be/0xc10
> [ 247.486245] __scsi_scan_target+0x18b/0x3b0
> [ 247.486247] scsi_scan_channel+0xee/0x180
> [ 247.486250] scsi_scan_host_selected+0x1fd/0x2c0
> [ 247.486252] do_scan_async+0x42/0x450
> [ 247.486254] async_run_entry_fn+0x94/0x540
> [ 247.486258] process_one_work+0x87a/0x14d0
> [ 247.486260] worker_thread+0x5f2/0xfd0
> [ 247.486262] kthread+0x3b0/0x770
> [ 247.486266] ret_from_fork+0x3ef/0x510
> [ 247.486269] ret_from_fork_asm+0x1a/0x30
> [ 247.486271]
> -> #2 (fs_reclaim){+.+.}-{0:0}:
> [ 247.486275] __lock_acquire+0x56a/0xbe0
> [ 247.486277] lock_acquire.part.0+0xc8/0x270
> [ 247.486279] fs_reclaim_acquire+0xd9/0x130
> [ 247.486282] prepare_alloc_pages+0x153/0x5a0
> [ 247.486284] __alloc_frozen_pages_noprof+0x142/0x490
> [ 247.486286] __alloc_pages_noprof+0x12/0x210
> [ 247.486288] pcpu_alloc_pages.isra.0+0xfa/0x4d0
> [ 247.486291] pcpu_populate_chunk+0x39/0x80
> [ 247.486293] pcpu_alloc_noprof+0x759/0xeb0
> [ 247.486296] iommu_dma_init_fq+0x19c/0x7c0
> [ 247.486299] iommu_dma_init_domain+0x53f/0x7f0
> [ 247.486301] iommu_setup_dma_ops+0xd3/0x200
> [ 247.486303] bus_iommu_probe+0x1f0/0x4b0
> [ 247.486306] iommu_device_register+0x186/0x280
> [ 247.486308] iommu_init_pci+0xc8c/0xd00
> [ 247.486312] amd_iommu_init_pci+0x83/0x4e0
> [ 247.486314] state_next+0x28f/0x5c0
> [ 247.486317] iommu_go_to_state+0x2b/0x60
> [ 247.486319] amd_iommu_init+0x21/0x60
> [ 247.486321] pci_iommu_init+0x38/0x60
> [ 247.486325] do_one_initcall+0xd2/0x450
> [ 247.486327] do_initcalls+0x216/0x240
> [ 247.486330] kernel_init_freeable+0x299/0x2d0
> [ 247.486332] kernel_init+0x1c/0x150
> [ 247.486335] ret_from_fork+0x3ef/0x510
> [ 247.486337] ret_from_fork_asm+0x1a/0x30
> [ 247.486338]
> -> #1 (pcpu_alloc_mutex){+.+.}-{4:4}:
> [ 247.486342] __lock_acquire+0x56a/0xbe0
> [ 247.486344] lock_acquire.part.0+0xc8/0x270
> [ 247.486346] __mutex_lock+0x1b2/0x1b70
> [ 247.486348] pcpu_alloc_noprof+0x884/0xeb0
> [ 247.486351] sbitmap_init_node+0x252/0x6a0
> [ 247.486354] sbitmap_queue_init_node+0x2d/0x420
> [ 247.486356] blk_mq_init_tags+0x154/0x2a0
> [ 247.486359] blk_mq_alloc_map_and_rqs+0xa6/0x310
> [ 247.486361] blk_mq_init_sched+0x2a4/0x580
> [ 247.486363] elevator_switch+0x18b/0x630
> [ 247.486365] elevator_change+0x209/0x380
> [ 247.486368] elevator_set_default+0x22d/0x2a0
> [ 247.486370] blk_register_queue+0x33a/0x490
> [ 247.486372] __add_disk+0x5fd/0xd50
> [ 247.486374] add_disk_fwnode+0x113/0x590
> [ 247.486377] sd_probe+0x873/0xe10
> [ 247.486380] really_probe+0x1de/0x890
> [ 247.486383] __driver_probe_device+0x18c/0x390
> [ 247.486385] driver_probe_device+0x4a/0x120
> [ 247.486388] __device_attach_driver+0x156/0x280
> [ 247.486389] bus_for_each_drv+0x111/0x1a0
> [ 247.486392] __device_attach_async_helper+0x19c/0x240
> [ 247.486394] async_run_entry_fn+0x94/0x540
> [ 247.486396] process_one_work+0x87a/0x14d0
> [ 247.486398] worker_thread+0x5f2/0xfd0
> [ 247.486400] kthread+0x3b0/0x770
> [ 247.486402] ret_from_fork+0x3ef/0x510
> [ 247.486404] ret_from_fork_asm+0x1a/0x30
> [ 247.486406]
> -> #0 (&q->elevator_lock){+.+.}-{4:4}:
> [ 247.486409] check_prev_add+0xe1/0xcf0
> [ 247.486411] validate_chain+0x4cf/0x740
> [ 247.486413] __lock_acquire+0x56a/0xbe0
> [ 247.486414] lock_acquire.part.0+0xc8/0x270
> [ 247.486416] __mutex_lock+0x1b2/0x1b70
> [ 247.486418] elevator_change+0xb6/0x380
> [ 247.486420] elv_iosched_store+0x24a/0x2c0
> [ 247.486422] queue_attr_store+0x238/0x340
> [ 247.486425] kernfs_fop_write_iter+0x39b/0x5a0
> [ 247.486428] vfs_write+0x524/0xe70
> [ 247.486430] ksys_write+0xff/0x200
> [ 247.486432] do_syscall_64+0x98/0x3c0
> [ 247.486435] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 247.486438]
> other info that might help us debug this:
>
> [ 247.486439] Chain exists of:
> &q->elevator_lock --> fs_reclaim --> &q-
> >q_usage_counter(io)#6
>
> [ 247.486444] Possible unsafe locking scenario:
>
> [ 247.486446] CPU0 CPU1
> [ 247.486447] ---- ----
> [ 247.486448] lock(&q->q_usage_counter(io)#6);
> [ 247.486451] lock(fs_reclaim);
> [ 247.486453] lock(&q-
> >q_usage_counter(io)#6);
> [ 247.486456] lock(&q->elevator_lock);
> [ 247.486458]
> *** DEADLOCK ***
>
> [ 247.486459] 6 locks held by (udev-worker)/4257:
> [ 247.486461] #0: ffff88817b49c458 (sb_writers#4){.+.+}-{0:0}, at:
> ksys_write+0xff/0x200
> [ 247.486467] #1: ffff88819e767490 (&of->mutex#2){+.+.}-{4:4}, at:
> kernfs_fop_write_iter+0x25b/0x5a0
> [ 247.486473] #2: ffff8883b352c8c0 (kn->active#204){.+.+}-{0:0},
> at:
> kernfs_fop_write_iter+0x27e/0x5a0
> [ 247.486479] #3: ffff888169ecc3c0
> (&set->update_nr_hwq_lock){.+.+}-{4:4}, at:
> elv_iosched_store+0x1ba/0x2c0
> [ 247.486484] #4: ffff88816b9c00b0
> (&q->q_usage_counter(io)#6){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x16/0x30
> [ 247.486490] #5: ffff88816b9c00f0
> (&q->q_usage_counter(queue)#5){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x16/0x30
> [ 247.486495]
> stack backtrace:
> [ 247.486498] CPU: 3 UID: 0 PID: 4257 Comm: (udev-worker) Tainted: G
> L 6.16.0-rc7 #36 PREEMPT(lazy)
> [ 247.486501] Tainted: [L]=SOFTLOCKUP
> [ 247.486501] Hardware name: ASRock B650I Lightning WiFi/B650I
> Lightning WiFi, BIOS 3.30 06/16/2025
> [ 247.486503] Call Trace:
> [ 247.486503] <TASK>
> [ 247.486505] dump_stack_lvl+0x84/0xd0
> [ 247.486509] print_circular_bug.cold+0x38/0x46
> [ 247.486512] check_noncircular+0x14a/0x170
> [ 247.486515] check_prev_add+0xe1/0xcf0
> [ 247.486517] ? lock_acquire.part.0+0xc8/0x270
> [ 247.486518] validate_chain+0x4cf/0x740
> [ 247.486520] __lock_acquire+0x56a/0xbe0
> [ 247.486522] lock_acquire.part.0+0xc8/0x270
> [ 247.486524] ? elevator_change+0xb6/0x380
> [ 247.486526] ? __lock_release.isra.0+0x1cb/0x340
> [ 247.486527] ? rcu_is_watching+0x15/0xe0
> [ 247.486530] ? __pfx___might_resched+0x10/0x10
> [ 247.486532] ? elevator_change+0xb6/0x380
> [ 247.486534] ? lock_acquire+0xf7/0x140
> [ 247.486535] __mutex_lock+0x1b2/0x1b70
> [ 247.486537] ? elevator_change+0xb6/0x380
> [ 247.486539] ? elevator_change+0xb6/0x380
> [ 247.486541] ? __pfx_xa_find_after+0x10/0x10
> [ 247.486543] ? __pfx___mutex_lock+0x10/0x10
> [ 247.486545] ? __pfx___might_resched+0x10/0x10
> [ 247.486547] ? blk_mq_cancel_work_sync+0xc0/0x100
> [ 247.486549] ? __pfx_blk_mq_cancel_work_sync+0x10/0x10
> [ 247.486551] ? elevator_change+0xb6/0x380
> [ 247.486553] elevator_change+0xb6/0x380
> [ 247.486556] elv_iosched_store+0x24a/0x2c0
> [ 247.486558] ? __pfx_elv_iosched_store+0x10/0x10
> [ 247.486560] ? __pfx___might_resched+0x10/0x10
> [ 247.486562] ? __pfx_sysfs_kf_write+0x10/0x10
> [ 247.486564] queue_attr_store+0x238/0x340
> [ 247.486566] ? __pfx_queue_attr_store+0x10/0x10
> [ 247.486567] ? __lock_acquire+0x56a/0xbe0
> [ 247.486569] ? lock_acquire.part.0+0xc8/0x270
> [ 247.486570] ? find_held_lock+0x2b/0x80
> [ 247.486572] ? __lock_release.isra.0+0x1cb/0x340
> [ 247.486574] ? sysfs_file_kobj+0xb3/0x1c0
> [ 247.486576] ? sysfs_file_kobj+0xbd/0x1c0
> [ 247.486577] ? sysfs_kf_write+0x65/0x170
> [ 247.486579] ? __pfx_sysfs_kf_write+0x10/0x10
> [ 247.486580] kernfs_fop_write_iter+0x39b/0x5a0
> [ 247.486582] ? __pfx_kernfs_fop_write_iter+0x10/0x10
> [ 247.486584] vfs_write+0x524/0xe70
> [ 247.486586] ? __pfx_vfs_write+0x10/0x10
> [ 247.486588] ? __pfx___seccomp_filter+0x10/0x10
> [ 247.486591] ksys_write+0xff/0x200
> [ 247.486593] ? __pfx_ksys_write+0x10/0x10
> [ 247.486595] ? syscall_trace_enter+0x8e/0x2e0
> [ 247.486598] do_syscall_64+0x98/0x3c0
> [ 247.486600] ? __x64_sys_openat+0x10e/0x210
> [ 247.486602] ? do_syscall_64+0x161/0x3c0
> [ 247.486604] ? do_sys_openat2+0x109/0x180
> [ 247.486605] ? __pfx___x64_sys_openat+0x10/0x10
> [ 247.486607] ? __pfx_do_sys_openat2+0x10/0x10
> [ 247.486609] ? lockdep_hardirqs_on+0x8c/0x130
> [ 247.486611] ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 247.486613] ? do_syscall_64+0x161/0x3c0
> [ 247.486615] ? __x64_sys_openat+0x10e/0x210
> [ 247.486616] ? lockdep_hardirqs_on+0x8c/0x130
> [ 247.486618] ? __pfx___x64_sys_openat+0x10/0x10
> [ 247.486621] ? lockdep_hardirqs_on+0x8c/0x130
> [ 247.486623] ? entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 247.486624] ? do_syscall_64+0x161/0x3c0
> [ 247.486626] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 247.486627] RIP: 0033:0x7f9afd67b5c6
> [ 247.486642] Code: 5d e8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 75
> 19 83 e2 39 83 fa 08 75 11 e8 26 ff ff ff 66 0f 1f 44 00 00 48 8b 45
> 10 0f 05 <48> 8b 5d f8 c9 c3 0f 1f 40 00 f3 0f 1e fa 55 48 89 e5 48
> 83
> ec 08
> [ 247.486643] RSP: 002b:00007fff244eb210 EFLAGS: 00000202 ORIG_RAX:
> 0000000000000001
> [ 247.486646] RAX: ffffffffffffffda RBX: 000055e41796e060 RCX:
> 00007f9afd67b5c6
> [ 247.486647] RDX: 0000000000000003 RSI: 00007fff244eb560 RDI:
> 0000000000000014
> [ 247.486647] RBP: 00007fff244eb230 R08: 0000000000000000 R09:
> 0000000000000000
> [ 247.486648] R10: 0000000000000000 R11: 0000000000000202 R12:
> 0000000000000003
> [ 247.486649] R13: 0000000000000003 R14: 00007fff244eb560 R15:
> 00007fff244eb560
> [ 247.486652] </TASK>
>
> I bisected the issue to this commit:
> ffa1e7ada456087c2402b37cd6b2863ced29aff0 is the first bad commit
> commit ffa1e7ada456087c2402b37cd6b2863ced29aff0
> Author: Thomas Hellström <thomas.hellstrom@linux.intel.com>
> Date: Tue Mar 18 10:55:48 2025 +0100
>
> block: Make request_queue lockdep splats show up earlier
>
> In recent kernels, there are lockdep splats around the
> struct request_queue::io_lockdep_map, similar to [1], but they
> typically don't show up until reclaim with writeback happens.
>
> Having multiple kernel versions released with a known risc of
> kernel
> deadlock during reclaim writeback should IMHO be addressed and
> backported to -stable with the highest priority.
>
> In order to have these lockdep splats show up earlier,
> preferrably during system initialization, prime the
> struct request_queue::io_lockdep_map as GFP_KERNEL reclaim-
> tainted. This will instead lead to lockdep splats looking similar
> to [2], but without the need for reclaim + writeback
> happening.
>
> [1]:
> [ 189.762244]
> ======================================================
> [ 189.762432] WARNING: possible circular locking dependency
> detected
> [ 189.762441] 6.14.0-rc6-xe+ #6 Tainted: G U
> [ 189.762450] --------------------------------------------------
> ----
> [ 189.762459] kswapd0/119 is trying to acquire lock:
> [ 189.762467] ffff888110ceb710
> (&q->q_usage_counter(io)#26){++++}-{0:0}, at: __submit_bio+0x76/0x230
> [ 189.762485]
> but task is already holding lock:
> [ 189.762494] ffffffff834c97c0 (fs_reclaim){+.+.}-{0:0}, at:
> balance_pgdat+0xbe/0xb00
> [ 189.762507]
> which lock already depends on the new lock.
>
> [ 189.762519]
> the existing dependency chain (in reverse order)
> is:
> [ 189.762529]
> -> #2 (fs_reclaim){+.+.}-{0:0}:
> [ 189.762540] fs_reclaim_acquire+0xc5/0x100
> [ 189.762548] kmem_cache_alloc_lru_noprof+0x4a/0x480
> [ 189.762558] alloc_inode+0xaa/0xe0
> [ 189.762566] iget_locked+0x157/0x330
> [ 189.762573] kernfs_get_inode+0x1b/0x110
> [ 189.762582] kernfs_get_tree+0x1b0/0x2e0
> [ 189.762590] sysfs_get_tree+0x1f/0x60
> [ 189.762597] vfs_get_tree+0x2a/0xf0
> [ 189.762605] path_mount+0x4cd/0xc00
> [ 189.762613] __x64_sys_mount+0x119/0x150
> [ 189.762621] x64_sys_call+0x14f2/0x2310
> [ 189.762630] do_syscall_64+0x91/0x180
> [ 189.762637] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 189.762647]
> -> #1 (&root->kernfs_rwsem){++++}-{3:3}:
> [ 189.762659] down_write+0x3e/0xf0
> [ 189.762667] kernfs_remove+0x32/0x60
> [ 189.762676] sysfs_remove_dir+0x4f/0x60
> [ 189.762685] __kobject_del+0x33/0xa0
> [ 189.762709] kobject_del+0x13/0x30
> [ 189.762716] elv_unregister_queue+0x52/0x80
> [ 189.762725] elevator_switch+0x68/0x360
> [ 189.762733] elv_iosched_store+0x14b/0x1b0
> [ 189.762756] queue_attr_store+0x181/0x1e0
> [ 189.762765] sysfs_kf_write+0x49/0x80
> [ 189.762773] kernfs_fop_write_iter+0x17d/0x250
> [ 189.762781] vfs_write+0x281/0x540
> [ 189.762790] ksys_write+0x72/0xf0
> [ 189.762798] __x64_sys_write+0x19/0x30
> [ 189.762807] x64_sys_call+0x2a3/0x2310
> [ 189.762815] do_syscall_64+0x91/0x180
> [ 189.762823] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 189.762833]
> -> #0 (&q->q_usage_counter(io)#26){++++}-{0:0}:
> [ 189.762845] __lock_acquire+0x1525/0x2760
> [ 189.762854] lock_acquire+0xca/0x310
> [ 189.762861] blk_mq_submit_bio+0x8a2/0xba0
> [ 189.762870] __submit_bio+0x76/0x230
> [ 189.762878] submit_bio_noacct_nocheck+0x323/0x430
> [ 189.762888] submit_bio_noacct+0x2cc/0x620
> [ 189.762896] submit_bio+0x38/0x110
> [ 189.762904] __swap_writepage+0xf5/0x380
> [ 189.762912] swap_writepage+0x3c7/0x600
> [ 189.762920] shmem_writepage+0x3da/0x4f0
> [ 189.762929] pageout+0x13f/0x310
> [ 189.762937] shrink_folio_list+0x61c/0xf60
> [ 189.763261] evict_folios+0x378/0xcd0
> [ 189.763584] try_to_shrink_lruvec+0x1b0/0x360
> [ 189.763946] shrink_one+0x10e/0x200
> [ 189.764266] shrink_node+0xc02/0x1490
> [ 189.764586] balance_pgdat+0x563/0xb00
> [ 189.764934] kswapd+0x1e8/0x430
> [ 189.765249] kthread+0x10b/0x260
> [ 189.765559] ret_from_fork+0x44/0x70
> [ 189.765889] ret_from_fork_asm+0x1a/0x30
> [ 189.766198]
> other info that might help us debug this:
>
> [ 189.767089] Chain exists of:
> &q->q_usage_counter(io)#26 -->
> &root->kernfs_rwsem --> fs_reclaim
>
> [ 189.767971] Possible unsafe locking scenario:
>
> [ 189.768555] CPU0 CPU1
> [ 189.768849] ---- ----
> [ 189.769136] lock(fs_reclaim);
> [ 189.769421] lock(&root-
> >kernfs_rwsem);
> [ 189.769714] lock(fs_reclaim);
> [ 189.770016] rlock(&q->q_usage_counter(io)#26);
> [ 189.770305]
> *** DEADLOCK ***
>
> [ 189.771167] 1 lock held by kswapd0/119:
> [ 189.771453] #0: ffffffff834c97c0 (fs_reclaim){+.+.}-{0:0},
> at:
> balance_pgdat+0xbe/0xb00
> [ 189.771770]
> stack backtrace:
> [ 189.772351] CPU: 4 UID: 0 PID: 119 Comm: kswapd0 Tainted: G
> U 6.14.0-rc6-xe+ #6
> [ 189.772353] Tainted: [U]=USER
> [ 189.772354] Hardware name: ASUS System Product Name/PRIME
> B560M-A AC, BIOS 2001 02/01/2023
> [ 189.772354] Call Trace:
> [ 189.772355] <TASK>
> [ 189.772356] dump_stack_lvl+0x6e/0xa0
> [ 189.772359] dump_stack+0x10/0x18
> [ 189.772360] print_circular_bug.cold+0x17a/0x1b7
> [ 189.772363] check_noncircular+0x13a/0x150
> [ 189.772365] ? __pfx_stack_trace_consume_entry+0x10/0x10
> [ 189.772368] __lock_acquire+0x1525/0x2760
> [ 189.772368] ? ret_from_fork_asm+0x1a/0x30
> [ 189.772371] lock_acquire+0xca/0x310
> [ 189.772372] ? __submit_bio+0x76/0x230
> [ 189.772375] ? lock_release+0xd5/0x2c0
> [ 189.772376] blk_mq_submit_bio+0x8a2/0xba0
> [ 189.772378] ? __submit_bio+0x76/0x230
> [ 189.772380] __submit_bio+0x76/0x230
> [ 189.772382] ? trace_hardirqs_on+0x1e/0xe0
> [ 189.772384] submit_bio_noacct_nocheck+0x323/0x430
> [ 189.772386] ? submit_bio_noacct_nocheck+0x323/0x430
> [ 189.772387] ? __might_sleep+0x58/0xa0
> [ 189.772390] submit_bio_noacct+0x2cc/0x620
> [ 189.772391] ? count_memcg_events+0x68/0x90
> [ 189.772393] submit_bio+0x38/0x110
> [ 189.772395] __swap_writepage+0xf5/0x380
> [ 189.772396] swap_writepage+0x3c7/0x600
> [ 189.772397] shmem_writepage+0x3da/0x4f0
> [ 189.772401] pageout+0x13f/0x310
> [ 189.772406] shrink_folio_list+0x61c/0xf60
> [ 189.772409] ? isolate_folios+0xe80/0x16b0
> [ 189.772410] ? mark_held_locks+0x46/0x90
> [ 189.772412] evict_folios+0x378/0xcd0
> [ 189.772414] ? evict_folios+0x34a/0xcd0
> [ 189.772415] ? lock_is_held_type+0xa3/0x130
> [ 189.772417] try_to_shrink_lruvec+0x1b0/0x360
> [ 189.772420] shrink_one+0x10e/0x200
> [ 189.772421] shrink_node+0xc02/0x1490
> [ 189.772423] ? shrink_node+0xa08/0x1490
> [ 189.772424] ? shrink_node+0xbd8/0x1490
> [ 189.772425] ? mem_cgroup_iter+0x366/0x480
> [ 189.772427] balance_pgdat+0x563/0xb00
> [ 189.772428] ? balance_pgdat+0x563/0xb00
> [ 189.772430] ? trace_hardirqs_on+0x1e/0xe0
> [ 189.772431] ? finish_task_switch.isra.0+0xcb/0x330
> [ 189.772433] ? __switch_to_asm+0x33/0x70
> [ 189.772437] kswapd+0x1e8/0x430
> [ 189.772438] ? __pfx_autoremove_wake_function+0x10/0x10
> [ 189.772440] ? __pfx_kswapd+0x10/0x10
> [ 189.772441] kthread+0x10b/0x260
> [ 189.772443] ? __pfx_kthread+0x10/0x10
> [ 189.772444] ret_from_fork+0x44/0x70
> [ 189.772446] ? __pfx_kthread+0x10/0x10
> [ 189.772447] ret_from_fork_asm+0x1a/0x30
> [ 189.772450] </TASK>
>
> [2]:
> [ 8.760253]
> ======================================================
> [ 8.760254] WARNING: possible circular locking dependency
> detected
> [ 8.760255] 6.14.0-rc6-xe+ #7 Tainted: G U
> [ 8.760256] --------------------------------------------------
> ----
> [ 8.760257] (udev-worker)/674 is trying to acquire lock:
> [ 8.760259] ffff888100e39148 (&root->kernfs_rwsem){++++}-
> {3:3},
> at: kernfs_remove+0x32/0x60
> [ 8.760265]
> but task is already holding lock:
> [ 8.760266] ffff888110dc7680
> (&q->q_usage_counter(io)#27){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x12/0x30
> [ 8.760272]
> which lock already depends on the new lock.
>
> [ 8.760272]
> the existing dependency chain (in reverse order)
> is:
> [ 8.760273]
> -> #2 (&q->q_usage_counter(io)#27){++++}-{0:0}:
> [ 8.760276] blk_alloc_queue+0x30a/0x350
> [ 8.760279] blk_mq_alloc_queue+0x6b/0xe0
> [ 8.760281] scsi_alloc_sdev+0x276/0x3c0
> [ 8.760284] scsi_probe_and_add_lun+0x22a/0x440
> [ 8.760286] __scsi_scan_target+0x109/0x230
> [ 8.760288] scsi_scan_channel+0x65/0xc0
> [ 8.760290] scsi_scan_host_selected+0xff/0x140
> [ 8.760292] do_scsi_scan_host+0xa7/0xc0
> [ 8.760293] do_scan_async+0x1c/0x160
> [ 8.760295] async_run_entry_fn+0x32/0x150
> [ 8.760299] process_one_work+0x224/0x5f0
> [ 8.760302] worker_thread+0x1d4/0x3e0
> [ 8.760304] kthread+0x10b/0x260
> [ 8.760306] ret_from_fork+0x44/0x70
> [ 8.760309] ret_from_fork_asm+0x1a/0x30
> [ 8.760312]
> -> #1 (fs_reclaim){+.+.}-{0:0}:
> [ 8.760315] fs_reclaim_acquire+0xc5/0x100
> [ 8.760317] kmem_cache_alloc_lru_noprof+0x4a/0x480
> [ 8.760319] alloc_inode+0xaa/0xe0
> [ 8.760322] iget_locked+0x157/0x330
> [ 8.760323] kernfs_get_inode+0x1b/0x110
> [ 8.760325] kernfs_get_tree+0x1b0/0x2e0
> [ 8.760327] sysfs_get_tree+0x1f/0x60
> [ 8.760329] vfs_get_tree+0x2a/0xf0
> [ 8.760332] path_mount+0x4cd/0xc00
> [ 8.760334] __x64_sys_mount+0x119/0x150
> [ 8.760336] x64_sys_call+0x14f2/0x2310
> [ 8.760338] do_syscall_64+0x91/0x180
> [ 8.760340] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 8.760342]
> -> #0 (&root->kernfs_rwsem){++++}-{3:3}:
> [ 8.760345] __lock_acquire+0x1525/0x2760
> [ 8.760347] lock_acquire+0xca/0x310
> [ 8.760348] down_write+0x3e/0xf0
> [ 8.760350] kernfs_remove+0x32/0x60
> [ 8.760351] sysfs_remove_dir+0x4f/0x60
> [ 8.760353] __kobject_del+0x33/0xa0
> [ 8.760355] kobject_del+0x13/0x30
> [ 8.760356] elv_unregister_queue+0x52/0x80
> [ 8.760358] elevator_switch+0x68/0x360
> [ 8.760360] elv_iosched_store+0x14b/0x1b0
> [ 8.760362] queue_attr_store+0x181/0x1e0
> [ 8.760364] sysfs_kf_write+0x49/0x80
> [ 8.760366] kernfs_fop_write_iter+0x17d/0x250
> [ 8.760367] vfs_write+0x281/0x540
> [ 8.760370] ksys_write+0x72/0xf0
> [ 8.760372] __x64_sys_write+0x19/0x30
> [ 8.760374] x64_sys_call+0x2a3/0x2310
> [ 8.760376] do_syscall_64+0x91/0x180
> [ 8.760377] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 8.760380]
> other info that might help us debug this:
>
> [ 8.760380] Chain exists of:
> &root->kernfs_rwsem --> fs_reclaim -->
> &q->q_usage_counter(io)#27
>
> [ 8.760384] Possible unsafe locking scenario:
>
> [ 8.760384] CPU0 CPU1
> [ 8.760385] ---- ----
> [ 8.760385] lock(&q->q_usage_counter(io)#27);
> [ 8.760387] lock(fs_reclaim);
> [ 8.760388]
> lock(&q->q_usage_counter(io)#27);
> [ 8.760390] lock(&root->kernfs_rwsem);
> [ 8.760391]
> *** DEADLOCK ***
>
> [ 8.760391] 6 locks held by (udev-worker)/674:
> [ 8.760392] #0: ffff8881209ac420 (sb_writers#4){.+.+}-{0:0},
> at: ksys_write+0x72/0xf0
> [ 8.760398] #1: ffff88810c80f488 (&of->mutex#2){+.+.}-{3:3},
> at: kernfs_fop_write_iter+0x136/0x250
> [ 8.760402] #2: ffff888125d1d330 (kn->active#101){.+.+}-
> {0:0},
> at: kernfs_fop_write_iter+0x13f/0x250
> [ 8.760406] #3: ffff888110dc7bb0 (&q->sysfs_lock){+.+.}-
> {3:3},
> at: queue_attr_store+0x148/0x1e0
> [ 8.760411] #4: ffff888110dc7680
> (&q->q_usage_counter(io)#27){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x12/0x30
> [ 8.760416] #5: ffff888110dc76b8
> (&q->q_usage_counter(queue)#27){++++}-{0:0}, at:
> blk_mq_freeze_queue_nomemsave+0x12/0x30
> [ 8.760421]
> stack backtrace:
> [ 8.760422] CPU: 7 UID: 0 PID: 674 Comm: (udev-worker)
> Tainted:
> G U 6.14.0-rc6-xe+ #7
> [ 8.760424] Tainted: [U]=USER
> [ 8.760425] Hardware name: ASUS System Product Name/PRIME
> B560M-A AC, BIOS 2001 02/01/2023
> [ 8.760426] Call Trace:
> [ 8.760427] <TASK>
> [ 8.760428] dump_stack_lvl+0x6e/0xa0
> [ 8.760431] dump_stack+0x10/0x18
> [ 8.760433] print_circular_bug.cold+0x17a/0x1b7
> [ 8.760437] check_noncircular+0x13a/0x150
> [ 8.760441] ? save_trace+0x54/0x360
> [ 8.760445] __lock_acquire+0x1525/0x2760
> [ 8.760446] ? irqentry_exit+0x3a/0xb0
> [ 8.760448] ? sysvec_apic_timer_interrupt+0x57/0xc0
> [ 8.760452] lock_acquire+0xca/0x310
> [ 8.760453] ? kernfs_remove+0x32/0x60
> [ 8.760457] down_write+0x3e/0xf0
> [ 8.760459] ? kernfs_remove+0x32/0x60
> [ 8.760460] kernfs_remove+0x32/0x60
> [ 8.760462] sysfs_remove_dir+0x4f/0x60
> [ 8.760464] __kobject_del+0x33/0xa0
> [ 8.760466] kobject_del+0x13/0x30
> [ 8.760467] elv_unregister_queue+0x52/0x80
> [ 8.760470] elevator_switch+0x68/0x360
> [ 8.760472] elv_iosched_store+0x14b/0x1b0
> [ 8.760475] queue_attr_store+0x181/0x1e0
> [ 8.760479] ? lock_acquire+0xca/0x310
> [ 8.760480] ? kernfs_fop_write_iter+0x13f/0x250
> [ 8.760482] ? lock_is_held_type+0xa3/0x130
> [ 8.760485] sysfs_kf_write+0x49/0x80
> [ 8.760487] kernfs_fop_write_iter+0x17d/0x250
> [ 8.760489] vfs_write+0x281/0x540
> [ 8.760494] ksys_write+0x72/0xf0
> [ 8.760497] __x64_sys_write+0x19/0x30
> [ 8.760499] x64_sys_call+0x2a3/0x2310
> [ 8.760502] do_syscall_64+0x91/0x180
> [ 8.760504] ? trace_hardirqs_off+0x5d/0xe0
> [ 8.760506] ? handle_softirqs+0x479/0x4d0
> [ 8.760508] ? hrtimer_interrupt+0x13f/0x280
> [ 8.760511] ? irqentry_exit_to_user_mode+0x8b/0x260
> [ 8.760513] ? clear_bhb_loop+0x15/0x70
> [ 8.760515] ? clear_bhb_loop+0x15/0x70
> [ 8.760516] ? clear_bhb_loop+0x15/0x70
> [ 8.760518] entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [ 8.760520] RIP: 0033:0x7aa3bf2f5504
> [ 8.760522] Code: c7 00 16 00 00 00 b8 ff ff ff ff c3 66 2e 0f
> 1f 84 00 00 00 00 00 f3 0f 1e fa 80 3d c5 8b 10 00 00 74 13 b8 01 00
> 00 00 0f 05 <48> 3d 00 f0 ff ff 77 54 c3 0f 1f 00 55 48 89 e5 48 83
> ec
> 20 48 89
> [ 8.760523] RSP: 002b:00007ffc1e3697d8 EFLAGS: 00000202
> ORIG_RAX: 0000000000000001
> [ 8.760526] RAX: ffffffffffffffda RBX: 0000000000000003 RCX:
> 00007aa3bf2f5504
> [ 8.760527] RDX: 0000000000000003 RSI: 00007ffc1e369ae0 RDI:
> 000000000000001c
> [ 8.760528] RBP: 00007ffc1e369800 R08: 00007aa3bf3f51c8 R09:
> 00007ffc1e3698b0
> [ 8.760528] R10: 0000000000000000 R11: 0000000000000202 R12:
> 0000000000000003
> [ 8.760529] R13: 00007ffc1e369ae0 R14: 0000613ccf21f2f0 R15:
> 00007aa3bf3f4e80
> [ 8.760533] </TASK>
>
> v2:
> - Update a code comment to increase readability (Ming Lei).
>
> Cc: Jens Axboe <axboe@kernel.dk>
> Cc: linux-block@vger.kernel.org
> Cc: linux-kernel@vger.kernel.org
> Cc: Ming Lei <ming.lei@redhat.com>
> Signed-off-by: Thomas Hellström
> <thomas.hellstrom@linux.intel.com>
> Reviewed-by: Ming Lei <ming.lei@redhat.com>
> Link:
> https://lore.kernel.org/r/20250318095548.5187-1-thomas.hellstrom@linux.intel.com
> Signed-off-by: Jens Axboe <axboe@kernel.dk>
>
> block/blk-core.c | 6 ++++++
> 1 file changed, 6 insertions(+)
>
> Reverting commit ffa1e7ada456 on top of 6.16-rc7 completely
> eliminates
> the warning. No lockdep splats occur when plugging in USB devices
> after the revert.
>
> The full dmesg trace, kernel config, and hardware probe are attached
> below:
>
> Hardware: https://linux-hardware.org/?probe=347fa4c2c6
> Kernel config: (attached)
> Full kernel log: (attached)
>
> This looks like a false-positive introduced by priming io_lockdep_map
> earlier, as explained in the commit message. However, it creates
> noise
> for regular users and may obscure more critical lockdep reports.
>
> Thanks for looking into this.
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456
2025-07-22 0:51 ` Hillf Danton
2025-07-22 7:11 ` Mikhail Gavrilov
@ 2025-07-22 12:14 ` Jens Axboe
2025-07-23 1:16 ` Hillf Danton
1 sibling, 1 reply; 9+ messages in thread
From: Jens Axboe @ 2025-07-22 12:14 UTC (permalink / raw)
To: Hillf Danton, Mikhail Gavrilov
Cc: thomas.hellstrom, linux-block, linux-kernel, Ming Lei,
regressions
On 7/21/25 6:51 PM, Hillf Danton wrote:
> Try the diff that serializes elevator_change() with q->elevator_lock if
> reproducer is available.
Hillf, these seemingly random and not tested or thought through patches
flung out in response to reports is not useful. It just wastes peoples
time and resources.
--
Jens Axboe
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456
2025-07-22 7:11 ` Mikhail Gavrilov
@ 2025-07-23 1:03 ` Hillf Danton
2025-07-23 20:36 ` Mikhail Gavrilov
0 siblings, 1 reply; 9+ messages in thread
From: Hillf Danton @ 2025-07-23 1:03 UTC (permalink / raw)
To: Mikhail Gavrilov
Cc: thomas.hellstrom, axboe, linux-block, linux-kernel, Ming Lei,
regressions
On Tue, 22 Jul 2025 12:11:36 +0500 Mikhail Gavrilov wrote:
> On Tue, Jul 22, 2025 at 5:51 AM Hillf Danton <hdanton@sina.com> wrote:
> >
> > Try the diff that serializes elevator_change() with q->elevator_lock if
> > reproducer is available.
> >
> > --- x/block/elevator.c
> > +++ y/block/elevator.c
> > @@ -661,6 +661,7 @@ static int elevator_change(struct reques
> > unsigned int memflags;
> > int ret = 0;
> >
> > + /* updaters should be serialized */
> > lockdep_assert_held(&q->tag_set->update_nr_hwq_lock);
> >
> > memflags = blk_mq_freeze_queue(q);
> > @@ -674,11 +675,11 @@ static int elevator_change(struct reques
> > * Disk isn't added yet, so verifying queue lock only manually.
> > */
> > blk_mq_cancel_work_sync(q);
> > + blk_mq_unfreeze_queue(q, memflags);
> > mutex_lock(&q->elevator_lock);
> > if (!(q->elevator && elevator_match(q->elevator->type, ctx->name)))
> > ret = elevator_switch(q, ctx);
> > mutex_unlock(&q->elevator_lock);
> > - blk_mq_unfreeze_queue(q, memflags);
> > if (!ret)
> > ret = elevator_change_done(q, ctx);
> >
>
> Hi Hillf,
>
> Thanks for the patch.
>
> I tested your proposed diff that serializes elevator_change() with
> q->elevator_lock. Unfortunately, instead of the previous lockdep
> warning, I'm now seeing a soft lockup warning.
>
> Here is the relevant excerpt from the kernel log:
>
> [ 78.573292] sd 6:0:0:0: [sda] Assuming drive cache: write through
> [ 78.581496] ------------[ cut here ]------------
> [ 78.581507] WARNING: CPU: 7 PID: 300 at block/elevator.c:578
> elevator_switch+0x512/0x630
>
> This happens after plugging in a USB flash stick (sd 6:0:0:0) with the
> patched kernel.
>
> Full dmesg trace is attached below.
>
> Let me know if you'd like me to try additional debugging or patches.
>
> Thanks for looking into this!
In order to cure the deadlock, queue is thawed before switching elevator,
so lets see what comes out with that warning ignored.
--- x/block/elevator.c
+++ y/block/elevator.c
@@ -575,7 +575,6 @@ static int elevator_switch(struct reques
struct elevator_type *new_e = NULL;
int ret = 0;
- WARN_ON_ONCE(q->mq_freeze_depth == 0);
lockdep_assert_held(&q->elevator_lock);
if (strncmp(ctx->name, "none", 4)) {
@@ -661,6 +660,7 @@ static int elevator_change(struct reques
unsigned int memflags;
int ret = 0;
+ /* updaters should be serialized */
lockdep_assert_held(&q->tag_set->update_nr_hwq_lock);
memflags = blk_mq_freeze_queue(q);
@@ -674,11 +674,11 @@ static int elevator_change(struct reques
* Disk isn't added yet, so verifying queue lock only manually.
*/
blk_mq_cancel_work_sync(q);
+ blk_mq_unfreeze_queue(q, memflags);
mutex_lock(&q->elevator_lock);
if (!(q->elevator && elevator_match(q->elevator->type, ctx->name)))
ret = elevator_switch(q, ctx);
mutex_unlock(&q->elevator_lock);
- blk_mq_unfreeze_queue(q, memflags);
if (!ret)
ret = elevator_change_done(q, ctx);
--
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456
2025-07-22 12:14 ` Jens Axboe
@ 2025-07-23 1:16 ` Hillf Danton
0 siblings, 0 replies; 9+ messages in thread
From: Hillf Danton @ 2025-07-23 1:16 UTC (permalink / raw)
To: Jens Axboe
Cc: Mikhail Gavrilov, thomas.hellstrom, linux-block, linux-kernel,
Ming Lei, regressions
On Tue, 22 Jul 2025 06:14:18 -0600 Jens Axboe wrote:
> On 7/21/25 6:51 PM, Hillf Danton wrote:
> > Try the diff that serializes elevator_change() with q->elevator_lock if
> > reproducer is available.
>
> Hillf, these seemingly random and not tested or thought through patches
> flung out in response to reports is not useful. It just wastes peoples
> time and resources.
>
Interesting, if you are right, so does the guy who created the deadlock, no?
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456
2025-07-23 1:03 ` Hillf Danton
@ 2025-07-23 20:36 ` Mikhail Gavrilov
2025-07-24 1:19 ` Hillf Danton
0 siblings, 1 reply; 9+ messages in thread
From: Mikhail Gavrilov @ 2025-07-23 20:36 UTC (permalink / raw)
To: Hillf Danton
Cc: thomas.hellstrom, axboe, linux-block, linux-kernel, Ming Lei,
regressions
[-- Attachment #1: Type: text/plain, Size: 1753 bytes --]
On Wed, Jul 23, 2025 at 6:03 AM Hillf Danton <hdanton@sina.com> wrote:
>
> In order to cure the deadlock, queue is thawed before switching elevator,
> so lets see what comes out with that warning ignored.
>
> --- x/block/elevator.c
> +++ y/block/elevator.c
> @@ -575,7 +575,6 @@ static int elevator_switch(struct reques
> struct elevator_type *new_e = NULL;
> int ret = 0;
>
> - WARN_ON_ONCE(q->mq_freeze_depth == 0);
> lockdep_assert_held(&q->elevator_lock);
>
> if (strncmp(ctx->name, "none", 4)) {
> @@ -661,6 +660,7 @@ static int elevator_change(struct reques
> unsigned int memflags;
> int ret = 0;
>
> + /* updaters should be serialized */
> lockdep_assert_held(&q->tag_set->update_nr_hwq_lock);
>
> memflags = blk_mq_freeze_queue(q);
> @@ -674,11 +674,11 @@ static int elevator_change(struct reques
> * Disk isn't added yet, so verifying queue lock only manually.
> */
> blk_mq_cancel_work_sync(q);
> + blk_mq_unfreeze_queue(q, memflags);
> mutex_lock(&q->elevator_lock);
> if (!(q->elevator && elevator_match(q->elevator->type, ctx->name)))
> ret = elevator_switch(q, ctx);
> mutex_unlock(&q->elevator_lock);
> - blk_mq_unfreeze_queue(q, memflags);
> if (!ret)
> ret = elevator_change_done(q, ctx);
>
> --
Hi Hillf,
Thanks for the patch.
With this patch applied, I haven't seen either the lockdep warning or
a soft lockup within 13 hours of runtime. Not sure if that's
sufficient yet for a final verdict, but it's definitely promising.
The full dmesg trace is attached below.
--
Best Regards,
Mike Gavrilov.
[-- Attachment #2: dmesg-6.16.0-rc7-with-fix-2.zip --]
[-- Type: application/zip, Size: 46067 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456
2025-07-23 20:36 ` Mikhail Gavrilov
@ 2025-07-24 1:19 ` Hillf Danton
0 siblings, 0 replies; 9+ messages in thread
From: Hillf Danton @ 2025-07-24 1:19 UTC (permalink / raw)
To: Mikhail Gavrilov
Cc: thomas.hellstrom, axboe, linux-block, linux-kernel, Ming Lei,
regressions
On Thu, 24 Jul 2025 01:36:37 +0500 Mikhail Gavrilov wrote:
> On Wed, Jul 23, 2025 at 6:03 AM Hillf Danton <hdanton@sina.com> wrote:
> >
> > In order to cure the deadlock, queue is thawed before switching elevator,
> > so lets see what comes out with that warning ignored.
> >
> > --- x/block/elevator.c
> > +++ y/block/elevator.c
> > @@ -575,7 +575,6 @@ static int elevator_switch(struct reques
> > struct elevator_type *new_e = NULL;
> > int ret = 0;
> >
> > - WARN_ON_ONCE(q->mq_freeze_depth == 0);
> > lockdep_assert_held(&q->elevator_lock);
> >
> > if (strncmp(ctx->name, "none", 4)) {
> > @@ -661,6 +660,7 @@ static int elevator_change(struct reques
> > unsigned int memflags;
> > int ret = 0;
> >
> > + /* updaters should be serialized */
> > lockdep_assert_held(&q->tag_set->update_nr_hwq_lock);
> >
> > memflags = blk_mq_freeze_queue(q);
> > @@ -674,11 +674,11 @@ static int elevator_change(struct reques
> > * Disk isn't added yet, so verifying queue lock only manually.
> > */
> > blk_mq_cancel_work_sync(q);
> > + blk_mq_unfreeze_queue(q, memflags);
> > mutex_lock(&q->elevator_lock);
> > if (!(q->elevator && elevator_match(q->elevator->type, ctx->name)))
> > ret = elevator_switch(q, ctx);
> > mutex_unlock(&q->elevator_lock);
> > - blk_mq_unfreeze_queue(q, memflags);
> > if (!ret)
> > ret = elevator_change_done(q, ctx);
> >
> > --
>
> Hi Hillf,
>
> Thanks for the patch.
>
> With this patch applied, I haven't seen either the lockdep warning or
> a soft lockup within 13 hours of runtime. Not sure if that's
> sufficient yet for a final verdict, but it's definitely promising.
Thank you for testing it.
It works for you so far, but given the "correct" locking order enforced
in ffa1e7ada456, I know the chance for reversing that order is not zero
yet either in Jens or upstream tree. Nor simple to fix every single case.
Hillf Danton
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2025-07-24 1:20 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-07-21 21:54 6.15/regression/bisected - lockdep warning: circular locking dependency detected when plugging USB stick after ffa1e7ada456 Mikhail Gavrilov
2025-07-22 0:51 ` Hillf Danton
2025-07-22 7:11 ` Mikhail Gavrilov
2025-07-23 1:03 ` Hillf Danton
2025-07-23 20:36 ` Mikhail Gavrilov
2025-07-24 1:19 ` Hillf Danton
2025-07-22 12:14 ` Jens Axboe
2025-07-23 1:16 ` Hillf Danton
2025-07-22 7:39 ` Thomas Hellström
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.