From: "Valdis Klētnieks" <valdis.kletnieks@vt.edu>
To: Ming Lei <ming.lei@redhat.com>
Cc: Stephen Rothwell <sfr@canb.auug.org.au>,
Jens Axboe <axboe@kernel.dk>,
linux-next@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-block@vger.kernel.org
Subject: Re: next-20250327 - lockdep whine and USB issues at boot
Date: Sat, 29 Mar 2025 02:02:10 -0400 [thread overview]
Message-ID: <7755.1743228130@turing-police> (raw)
In-Reply-To: <Z-dVr6cIyrOID0J0@fedora>
[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=us-ascii, Size: 10548 bytes --]
On Sat, 29 Mar 2025 10:06:39 +0800, Ming Lei said:
> On Fri, Mar 28, 2025 at 02:10:53PM -0400, Valdis Klētnieks wrote:
> > Saw this during boot on a Dell Inspiron 5559 laptop.
> >
> > In addition, the external USB ports all gave up, rendering a USB mouse and a
> > USB external drive totally dead in the water. May or may not be related, I didn't
> > dig too far into it.
>
> It shouldn't be related to the warning.
> For this lockdep warning, feel free to try patch in the following link:
>
> https://lore.kernel.org/linux-block/Z-dUCLvf06SfTOHy@fedora/
After applying that patch, USB *didn't* die during boot. So apparently
*something* changed.
Also, the patch merely caused a *different* lockdep warning.
Rather than &q->q_usage_counter(io) and &q->elevator_lock, the
new one is &q->elevator_lock versus pcpu_alloc_mutex...
Looks like it's a bit more convoluted than first looked?
[ 52.140990] [ T954] io scheduler bfq registered
[ 52.355286] [ T848] ======================================================
[ 52.436367] [ T848] WARNING: possible circular locking dependency detected
[ 52.517414] [ T848] 6.14.0-next-20250327-dirty #111 Tainted: G I T
[ 52.598492] [ T848] ------------------------------------------------------
[ 52.679571] [ T848] (udev-worker)/848 is trying to acquire lock:
[ 52.760647] [ T848] ffffffff99bb4470 (pcpu_alloc_mutex){+.+.}-{4:4}, at: pcpu_alloc_noprof+0x524/0x840
[ 53.162776] [ T848]
but task is already holding lock:
[ 53.244616] [ T848] ffff8c22c3392428 (&q->elevator_lock){+.+.}-{4:4}, at: elv_iosched_store+0xe1/0x260
[ 53.646741] [ T848]
which lock already depends on the new lock.
[ 53.729459] [ T848]
the existing dependency chain (in reverse order) is:
[ 53.811348] [ T848]
-> #3 (&q->elevator_lock){+.+.}-{4:4}:
[ 54.214266] [ T848] lock_acquire.part.0+0xbe/0x240
[ 54.295342] [ T848] __mutex_lock+0xcb/0xfc0
[ 54.376409] [ T848] elv_iosched_store+0xe1/0x260
[ 54.457476] [ T848] kernfs_fop_write_iter+0x160/0x240
[ 54.538548] [ T848] vfs_write+0x2ec/0x5c0
[ 54.619619] [ T848] ksys_write+0x7a/0xf0
[ 54.700687] [ T848] do_syscall_64+0x68/0x140
[ 54.781751] [ T848] entry_SYSCALL_64_after_hwframe+0x71/0x79
[ 54.862814] [ T848]
-> #2 (&q->q_usage_counter(io)#10){++++}-{0:0}:
[ 55.345959] [ T848] lock_acquire.part.0+0xbe/0x240
[ 55.427018] [ T848] blk_alloc_queue+0x30b/0x350
[ 55.508082] [ T848] blk_mq_alloc_queue+0x62/0xd0
[ 55.589134] [ T848] scsi_alloc_sdev+0x29c/0x3d0
[ 55.670185] [ T848] scsi_probe_and_add_lun+0x1d8/0x2b0
[ 55.751232] [ T848] __scsi_add_device+0x114/0x130
[ 55.832276] [ T848] ata_scsi_scan_host+0x7a/0x190
[ 55.913321] [ T848] async_run_entry_fn+0x24/0xc0
[ 55.994358] [ T848] process_one_work+0x21e/0x5a0
[ 56.075404] [ T848] worker_thread+0x1d5/0x3c0
[ 56.156444] [ T848] kthread+0x114/0x230
[ 56.237474] [ T848] ret_from_fork+0x2c/0x50
[ 56.318492] [ T848] ret_from_fork_asm+0x1a/0x30
[ 56.399515] [ T848]
-> #1 (fs_reclaim){+.+.}-{0:0}:
[ 56.802275] [ T848] lock_acquire.part.0+0xbe/0x240
[ 56.883298] [ T848] fs_reclaim_acquire+0xa8/0xe0
[ 56.964317] [ T848] __alloc_frozen_pages_noprof+0xd2/0x340
[ 57.045338] [ T848] __alloc_pages_noprof+0x9/0x20
[ 57.126360] [ T848] pcpu_alloc_pages.isra.0+0x97/0x180
[ 57.207375] [ T848] pcpu_populate_chunk+0x34/0x70
[ 57.288388] [ T848] pcpu_alloc_noprof+0x2d1/0x840
[ 57.369366] [ T848] iommu_dma_init_fq+0x91/0x260
[ 57.450373] [ T848] iommu_dma_init_domain+0x1de/0x210
[ 57.531381] [ T848] iommu_setup_dma_ops+0x47/0x80
[ 57.612355] [ T848] iommu_device_register+0x172/0x250
[ 57.693348] [ T848] intel_iommu_init+0x5ef/0xbb0
[ 57.774344] [ T848] pci_iommu_init+0x29/0x90
[ 57.855333] [ T848] do_one_initcall+0x83/0x3c0
[ 57.936316] [ T848] do_initcalls+0x130/0x1b0
[ 58.017300] [ T848] kernel_init_freeable+0x292/0x300
[ 58.098281] [ T848] kernel_init+0x15/0x130
[ 58.179254] [ T848] ret_from_fork+0x2c/0x50
[ 58.260224] [ T848] ret_from_fork_asm+0x1a/0x30
[ 58.341195] [ T848]
-> #0 (pcpu_alloc_mutex){+.+.}-{4:4}:
[ 58.743832] [ T848] check_prev_add+0xe1/0xcf0
[ 58.824752] [ T848] __lock_acquire+0x1031/0x13b0
[ 58.905697] [ T848] lock_acquire.part.0+0xbe/0x240
[ 58.986633] [ T848] __mutex_lock+0xcb/0xfc0
[ 59.067565] [ T848] pcpu_alloc_noprof+0x524/0x840
[ 59.148463] [ T848] sbitmap_init_node+0xf0/0x230
[ 59.229392] [ T848] sbitmap_queue_init_node+0x29/0x140
[ 59.310317] [ T848] blk_mq_init_tags+0x95/0x110
[ 59.391240] [ T848] blk_mq_alloc_map_and_rqs+0x3c/0x110
[ 59.472131] [ T848] blk_mq_init_sched+0x100/0x260
[ 59.553017] [ T848] elevator_switch+0xb1/0x190
[ 59.633926] [ T848] elv_iosched_store+0x1db/0x260
[ 59.714841] [ T848] kernfs_fop_write_iter+0x160/0x240
[ 59.795728] [ T848] vfs_write+0x2ec/0x5c0
[ 59.876599] [ T848] ksys_write+0x7a/0xf0
[ 59.957500] [ T848] do_syscall_64+0x68/0x140
[ 60.038396] [ T848] entry_SYSCALL_64_after_hwframe+0x71/0x79
[ 60.119269] [ T848]
other info that might help us debug this:
[ 60.201395] [ T848] Chain exists of:
pcpu_alloc_mutex --> &q->q_usage_counter(io)#10 --> &q->elevator_lock
[ 60.845339] [ T848] Possible unsafe locking scenario:
[ 60.926872] [ T848] CPU0 CPU1
[ 61.007776] [ T848] ---- ----
[ 61.088676] [ T848] lock(&q->elevator_lock);
[ 61.330053] [ T848] lock(&q->q_usage_counter(io)#10);
[ 61.651731] [ T848] lock(&q->elevator_lock);
[ 61.893151] [ T848] lock(pcpu_alloc_mutex);
[ 62.134602] [ T848]
*** DEADLOCK ***
[ 62.216729] [ T848] 6 locks held by (udev-worker)/848:
[ 62.297630] [ T848] #0: ffff8c22c8575440 (sb_writers#4){.+.+}-{0:0}, at: ksys_write+0x7a/0xf0
[ 62.860055] [ T848] #1: ffff8c22c9fb4090 (&of->mutex#2){+.+.}-{4:4}, at: kernfs_fop_write_iter+0x119/0x240
[ 63.422482] [ T848] #2: ffff8c22c821fad0 (kn->active#89){.+.+}-{0:0}, at: kernfs_fop_write_iter+0x122/0x240
[ 63.984915] [ T848] #3: ffff8c22c3391e98 (&q->q_usage_counter(io)#10){++++}-{0:0}, at: blk_mq_freeze_queue_nomemsave+0x11/0x20
[ 64.547390] [ T848] #4: ffff8c22c3391ed8 (&q->q_usage_counter(queue)){++++}-{0:0}, at: blk_mq_freeze_queue_nomemsave+0x11/0x20
[ 65.029578] [ T848] #5: ffff8c22c3392428 (&q->elevator_lock){+.+.}-{4:4}, at: elv_iosched_store+0xe1/0x260
[ 65.511794] [ T848]
stack backtrace:
[ 65.593348] [ T848] CPU: 1 UID: 0 PID: 848 Comm: (udev-worker) Tainted: G I T 6.14.0-next-20250327-dirty #111 PREEMPT(full) 7b54f670bb22238ec039f82f30e0fd3bcc8dbc83
[ 65.673611] [ T848] Tainte\0\0\0\0\0\0\0\0d: [I]=FIRMWARE_WORKAROUND, [T]=RANDSTRUCT
[ 65.753868] [ T848] Hardware name: Dell Inc. Unidentified System/052K07, BIOS 1.9.0 09/07/2020
[ 65.834126] [ T848] Call Trace:
[ 65.914385] [ T848] <TASK>
[ 65.994655] [ T848] dump_stack_lvl+0x65/0x90
[ 66.074916] [ T848] print_circular_bug.cold+0x38/0x46
[ 66.155178] [ T848] check_noncircular+0x169/0x190
[ 66.235441] [ T848] check_prev_add+0xe1/0xcf0
[ 66.315699] [ T848] ? add_chain_cache+0x115/0x4c0
[ 66.395961] [ T848] __lock_acquire+0x1031/0x13b0
[ 66.476225] [ T848] lock_acquire.part.0+0xbe/0x240
[ 66.556484] [ T848] ? pcpu_alloc_noprof+0x524/0x840
[ 66.636745] [ T848] ? lock_acquire+0xf8/0x140
[ 66.717006] [ T848] __mutex_lock+0xcb/0xfc0
[ 66.797266] [ T848] ? pcpu_alloc_noprof+0x524/0x840
[ 66.877526] [ T848] ? lockdep_hardirqs_on+0x84/0x130
[ 66.957786] [ T848] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 67.038044] [ T848] ? asm_sysvec_apic_timer_interrupt+0x1a/0x20
[ 67.118303] [ T848] ? pcpu_alloc_noprof+0x524/0x840
[ 67.198566] [ T848] ? pcpu_alloc_noprof+0x524/0x840
[ 67.278825] [ T848] ? find_held_lock+0x2b/0x80
[ 67.359084] [ T848] pcpu_alloc_noprof+0x524/0x840
[ 67.439344] [ T848] ? local_clock_noinstr+0xf/0x100
[ 67.519608] [ T848] sbitmap_init_node+0xf0/0x230
[ 67.599870] [ T848] sbitmap_queue_init_node+0x29/0x140
[ 67.680131] [ T848] blk_mq_init_tags+0x95/0x110
[ 67.760392] [ T848] blk_mq_alloc_map_and_rqs+0x3c/0x110
[ 67.840654] [ T848] blk_mq_init_sched+0x100/0x260
[ 67.920916] [ T848] elevator_switch+0xb1/0x190
[ 68.001176] [ T848] ? elv_iosched_store+0x1d0/0x260
[ 68.081437] [ T848] elv_iosched_store+0x1db/0x260
[ 68.161700] [ T848] kernfs_fop_write_iter+0x160/0x240
[ 68.241960] [ T848] vfs_write+0x2ec/0x5c0
[ 68.322224] [ T848] ksys_write+0x7a/0xf0
[ 68.402485] [ T848] do_syscall_64+0x68/0x140
[ 68.482745] [ T848] entry_SYSCALL_64_after_hwframe+0x71/0x79
[ 68.563004] [ T848] RIP: 0033:0x7f2a1747ca06
[ 68.643263] [ T848] Code: 5d e8 41 8b 93 08 03 00 00 59 5e 48 83 f8 fc 75 19 83 e2 39 83 fa 08 75 11 e8 26 ff ff ff 66 0f 1f 44 00 00 48 8b 45 10 0f 05 <48> 8b 5d f8 c9 c3 0f 1f 40 00 f3 0f 1e fa 55 48 89 e5 48 83 ec 08
[ 68.723521] [ T848] RSP: 002b:00007ffe48933eb0 EFLAGS: 00000202 ORIG_RAX: 0000000000000001
[ 68.884037] [ T848] RAX: ffffffffffffffda RBX: 000056119a4eaff0 RCX: 00007f2a1747ca06
[ 68.964295] [ T848] RDX: 0000000000000003 RSI: 00007ffe48934200 RDI: 0000000000000027
[ 69.044553] [ T848] RBP: 00007ffe48933ed0 R08: 0000000000000000 R09: 0000000000000000
[ 69.124811] [ T848] R10: 0000000000000000 R11: 0000000000000202 R12: 0000000000000003
[ 69.205068] [ T848] R13: 0000000000000003 R14: 00007ffe48934200 R15: 00007ffe48934200
[ 69.285332] [ T848] </TASK>
next prev parent reply other threads:[~2025-03-29 6:02 UTC|newest]
Thread overview: 5+ messages / expand[flat|nested] mbox.gz Atom feed top
2025-03-28 18:10 next-20250327 - lockdep whine and USB issues at boot Valdis Klētnieks
2025-03-28 20:32 ` Luck, Tony
2025-03-29 2:06 ` Ming Lei
2025-03-29 6:02 ` Valdis Klētnieks [this message]
2025-03-31 9:21 ` Ming Lei
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=7755.1743228130@turing-police \
--to=valdis.kletnieks@vt.edu \
--cc=axboe@kernel.dk \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-next@vger.kernel.org \
--cc=ming.lei@redhat.com \
--cc=sfr@canb.auug.org.au \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.