From: Benjamin Eikel <debian@eikel.org>
To: Thorsten Leemhuis <regressions@leemhuis.info>
Cc: linux-kernel@vger.kernel.org, Jens Axboe <axboe@kernel.dk>,
linux-block@vger.kernel.org,
Linux kernel regressions list <regressions@lists.linux.dev>
Subject: Re: [BUG] RCU stall in blk_mq_timeout_work (potentially a regression in 6.19.7 or 6.19.8)
Date: Thu, 02 Apr 2026 09:33:42 +0200 [thread overview]
Message-ID: <5090885.31r3eYUQgx@thinkpad-benjamin> (raw)
In-Reply-To: <b9262818-14e7-4902-af6c-6c7d3560ac4c@leemhuis.info>
Hi Thorsten and others,
Am Dienstag, 31. März 2026, 10:32:30 Mitteleuropäische Sommerzeit schrieb Thorsten Leemhuis:
> [CCing the regressions list and Jens, the Block layer maintainer, who
> might have an idea what's wrong here and/or how to best debug it]
>
> On 3/27/26 20:28, Benjamin Eikel wrote:
> > Dear Linux kernel developers,
> >
> > I experience repeated RCU stalls in blk_mq_timeout_work causing system
> > freezes. The NVMe drive (more info at the end) shows no errors or
> > controller resets in the kernel log: the hardware appears healthy. No I/O
> > timeout messages precede the stalls.
> >
> > The stall cascades: khugepaged blocks on __lru_add_drain_all waiting for a
> > workqueue flush that cannot complete, and additional kworkers block on a
> > mutex held by the kblockd rescuer thread. The system becomes noticeably
> > unresponsive and then I notice it in the logs.
> >
> > In case the logs are too condensed, please tell me and I can provide them
> > fully. If you have pointers on how this could be reproduced and a commit
> > range,>
> > I could try to bisect the problem.
>
> A bisection with mainline would help a lot. Maybe doing it between
> 6.19.6 and 6.19.8 (which would be quicker) might work out, too (but
> normally it's best to focus on mainline).
I have not yet done this because I do not have a reliable way to reproduce it.
>
> And this is not my area of expertise, but I noticed that some RCU fixes
> went into -rc6, so might be wise to try that first.
I encountered this issue with -rc6 (see log below, same config). I suspect that
this only happens after the notebook has previously been suspended.
Kind regards
Benjamin
== Trace from Linux version 7.0.0-rc6 (1–2 April) ==
Apr 01 20:36:59 thinkpad-benjamin kernel: Linux version 7.0.0-rc6-70-rc6 (benjamin@thinkpad-benjamin) (gcc (Debian 15.2.0-16) 15.2.0, GNU ld (GNU Binutils for Debian) 2.46) #24 SMP PREEMPT_DYNAMIC Tue Mar 31 14:36:43 CEST 2026
Apr 01 20:36:59 thinkpad-benjamin kernel: Command line: BOOT_IMAGE=/vmlinuz-7.0.0-rc6-70-rc6 root=/dev/mapper/thinkpad--benjamin--vg-root ro quiet
[…]
Apr 01 21:06:54 thinkpad-benjamin kernel: PM: suspend entry (s2idle)
[…]
Apr 02 08:44:55 thinkpad-benjamin kernel: PM: suspend exit
[…]
Apr 02 08:50:59 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 120 seconds.
Apr 02 08:50:59 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:50:59 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:50:59 thinkpad-benjamin kernel: task:khugepaged state:D stack:0 pid:138 tgid:138 ppid:2 task_flags:0x200040 flags:0x00080000
Apr 02 08:50:59 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:50:59 thinkpad-benjamin kernel: <TASK>
Apr 02 08:50:59 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:50:59 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:50:59 thinkpad-benjamin kernel: schedule_timeout+0xbd/0x100
Apr 02 08:50:59 thinkpad-benjamin kernel: wait_for_completion+0x8a/0x140
Apr 02 08:50:59 thinkpad-benjamin kernel: __flush_work+0x276/0x3b0
Apr 02 08:50:59 thinkpad-benjamin kernel: ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel: __lru_add_drain_all+0x198/0x2b0
Apr 02 08:50:59 thinkpad-benjamin kernel: khugepaged+0x65/0x950
Apr 02 08:50:59 thinkpad-benjamin kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel: ? __pfx_khugepaged+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:50:59 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:50:59 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:50:59 thinkpad-benjamin kernel: </TASK>
Apr 02 08:53:00 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 241 seconds.
Apr 02 08:53:00 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:53:00 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:53:00 thinkpad-benjamin kernel: task:khugepaged state:D stack:0 pid:138 tgid:138 ppid:2 task_flags:0x200040 flags:0x00080000
Apr 02 08:53:00 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:53:00 thinkpad-benjamin kernel: <TASK>
Apr 02 08:53:00 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:53:00 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:53:00 thinkpad-benjamin kernel: schedule_timeout+0xbd/0x100
Apr 02 08:53:00 thinkpad-benjamin kernel: wait_for_completion+0x8a/0x140
Apr 02 08:53:00 thinkpad-benjamin kernel: __flush_work+0x276/0x3b0
Apr 02 08:53:00 thinkpad-benjamin kernel: ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel: __lru_add_drain_all+0x198/0x2b0
Apr 02 08:53:00 thinkpad-benjamin kernel: khugepaged+0x65/0x950
Apr 02 08:53:00 thinkpad-benjamin kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel: ? __pfx_khugepaged+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:53:00 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:53:00 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:53:00 thinkpad-benjamin kernel: </TASK>
Apr 02 08:55:00 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 362 seconds.
Apr 02 08:55:00 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:55:00 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:55:00 thinkpad-benjamin kernel: task:khugepaged state:D stack:0 pid:138 tgid:138 ppid:2 task_flags:0x200040 flags:0x00080000
Apr 02 08:55:00 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:55:00 thinkpad-benjamin kernel: <TASK>
Apr 02 08:55:00 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:55:00 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:55:00 thinkpad-benjamin kernel: schedule_timeout+0xbd/0x100
Apr 02 08:55:00 thinkpad-benjamin kernel: wait_for_completion+0x8a/0x140
Apr 02 08:55:00 thinkpad-benjamin kernel: __flush_work+0x276/0x3b0
Apr 02 08:55:00 thinkpad-benjamin kernel: ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel: __lru_add_drain_all+0x198/0x2b0
Apr 02 08:55:00 thinkpad-benjamin kernel: khugepaged+0x65/0x950
Apr 02 08:55:00 thinkpad-benjamin kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel: ? __pfx_khugepaged+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:55:00 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:55:00 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:55:00 thinkpad-benjamin kernel: </TASK>
Apr 02 08:57:01 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 483 seconds.
Apr 02 08:57:01 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:57:01 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:57:01 thinkpad-benjamin kernel: task:khugepaged state:D stack:0 pid:138 tgid:138 ppid:2 task_flags:0x200040 flags:0x00080000
Apr 02 08:57:01 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:57:01 thinkpad-benjamin kernel: <TASK>
Apr 02 08:57:01 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:57:01 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:57:01 thinkpad-benjamin kernel: schedule_timeout+0xbd/0x100
Apr 02 08:57:01 thinkpad-benjamin kernel: wait_for_completion+0x8a/0x140
Apr 02 08:57:01 thinkpad-benjamin kernel: __flush_work+0x276/0x3b0
Apr 02 08:57:01 thinkpad-benjamin kernel: ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel: __lru_add_drain_all+0x198/0x2b0
Apr 02 08:57:01 thinkpad-benjamin kernel: khugepaged+0x65/0x950
Apr 02 08:57:01 thinkpad-benjamin kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel: ? __pfx_khugepaged+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:57:01 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:57:01 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:57:01 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:14:126 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:14 state:D stack:0 pid:126 tgid:126 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ttwu_queue_wakelist+0x13c/0x230
Apr 02 08:59:02 thinkpad-benjamin kernel: idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel: process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:14:126 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R running task stack:0 pid:139 tgid:139 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: RIP: 0010:affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: Code: 00 49 8b 36 48 8d bb ec 0d 00 00 e8 22 8a d9 00 45 85 ff 44 8b 04 24 0f 84 21 02 00 00 65 ff 0d 4e 73 8b 02 0f 84 6f 03 00 00 <41> 83 e0 04 49 8d 7c 24 08 75 3e e8 18 09 d9 00 b8 ff ff ff ff f0
Apr 02 08:59:02 thinkpad-benjamin kernel: RSP: 0018:ffffcc5f4063fcd8 EFLAGS: 00000246
Apr 02 08:59:02 thinkpad-benjamin kernel: RAX: 0000000000000001 RBX: ffff8af641ab2180 RCX: 0000000000000004
Apr 02 08:59:02 thinkpad-benjamin kernel: RDX: 0000000000000000 RSI: 0000000000000283 RDI: ffff8af640c08dec
Apr 02 08:59:02 thinkpad-benjamin kernel: RBP: ffff8b04a1eb2100 R08: 0000000000000000 R09: 0000000000000283
Apr 02 08:59:02 thinkpad-benjamin kernel: R10: 0000000000000002 R11: 0000000000000000 R12: ffffcc5f4063fd28
Apr 02 08:59:02 thinkpad-benjamin kernel: R13: ffffcc5f4063fd28 R14: ffffcc5f4063fde0 R15: 0000000000000000
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel: set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 604 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:khugepaged state:D stack:0 pid:138 tgid:138 ppid:2 task_flags:0x200040 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_timeout+0xbd/0x100
Apr 02 08:59:02 thinkpad-benjamin kernel: wait_for_completion+0x8a/0x140
Apr 02 08:59:02 thinkpad-benjamin kernel: __flush_work+0x276/0x3b0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: __lru_add_drain_all+0x198/0x2b0
Apr 02 08:59:02 thinkpad-benjamin kernel: khugepaged+0x65/0x950
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_khugepaged+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:16:5369 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:16 state:D stack:0 pid:5369 tgid:5369 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? finish_task_switch.isra.0+0x96/0x2c0
Apr 02 08:59:02 thinkpad-benjamin kernel: idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel: process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:16:5369 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R running task stack:0 pid:139 tgid:139 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __schedule+0x484/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: ? preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: ? asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: ? worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:12:9088 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:12 state:D stack:0 pid:9088 tgid:9088 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel: idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel: process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:12:9088 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R running task stack:0 pid:139 tgid:139 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: RIP: 0010:affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: Code: 00 49 8b 36 48 8d bb ec 0d 00 00 e8 22 8a d9 00 45 85 ff 44 8b 04 24 0f 84 21 02 00 00 65 ff 0d 4e 73 8b 02 0f 84 6f 03 00 00 <41> 83 e0 04 49 8d 7c 24 08 75 3e e8 18 09 d9 00 b8 ff ff ff ff f0
Apr 02 08:59:02 thinkpad-benjamin kernel: RSP: 0018:ffffcc5f4063fcd8 EFLAGS: 00000246
Apr 02 08:59:02 thinkpad-benjamin kernel: RAX: 0000000000000001 RBX: ffff8af641ab2180 RCX: 0000000000000004
Apr 02 08:59:02 thinkpad-benjamin kernel: RDX: 0000000000000000 RSI: 0000000000000283 RDI: ffff8af640c08dec
Apr 02 08:59:02 thinkpad-benjamin kernel: RBP: ffff8b04a1eb2100 R08: 0000000000000000 R09: 0000000000000283
Apr 02 08:59:02 thinkpad-benjamin kernel: R10: 0000000000000002 R11: 0000000000000000 R12: ffffcc5f4063fd28
Apr 02 08:59:02 thinkpad-benjamin kernel: R13: ffffcc5f4063fd28 R14: ffffcc5f4063fde0 R15: 0000000000000000
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel: set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:22:10249 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:22 state:D stack:0 pid:10249 tgid:10249 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? finish_task_switch.isra.0+0x96/0x2c0
Apr 02 08:59:02 thinkpad-benjamin kernel: idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel: process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:22:10249 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R running task stack:0 pid:139 tgid:139 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: RIP: 0010:affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: Code: 00 49 8b 36 48 8d bb ec 0d 00 00 e8 22 8a d9 00 45 85 ff 44 8b 04 24 0f 84 21 02 00 00 65 ff 0d 4e 73 8b 02 0f 84 6f 03 00 00 <41> 83 e0 04 49 8d 7c 24 08 75 3e e8 18 09 d9 00 b8 ff ff ff ff f0
Apr 02 08:59:02 thinkpad-benjamin kernel: RSP: 0018:ffffcc5f4063fcd8 EFLAGS: 00000246
Apr 02 08:59:02 thinkpad-benjamin kernel: RAX: 0000000000000001 RBX: ffff8af641ab2180 RCX: 0000000000000004
Apr 02 08:59:02 thinkpad-benjamin kernel: RDX: 0000000000000000 RSI: 0000000000000283 RDI: ffff8af640c08dec
Apr 02 08:59:02 thinkpad-benjamin kernel: RBP: ffff8b04a1eb2100 R08: 0000000000000000 R09: 0000000000000283
Apr 02 08:59:02 thinkpad-benjamin kernel: R10: 0000000000000002 R11: 0000000000000000 R12: ffffcc5f4063fd28
Apr 02 08:59:02 thinkpad-benjamin kernel: R13: ffffcc5f4063fd28 R14: ffffcc5f4063fde0 R15: 0000000000000000
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel: set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:31:10258 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:31 state:D stack:0 pid:10258 tgid:10258 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ttwu_queue_wakelist+0x13c/0x230
Apr 02 08:59:02 thinkpad-benjamin kernel: idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel: process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:31:10258 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R running task stack:0 pid:139 tgid:139 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __schedule+0x484/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: ? preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: ? asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: ? worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[…]
Apr 02 09:05:10 thinkpad-benjamin kernel: rcu_tasks_wait_gp: rcu_tasks grace period number 9 (since boot) is 2583 jiffies old.
Apr 02 09:05:40 thinkpad-benjamin kernel: rcu_tasks_wait_gp: rcu_tasks grace period number 9 (since boot) is 10136 jiffies old.
[…]
Apr 02 09:07:11 thinkpad-benjamin kernel: rcu_tasks_wait_gp: rcu_tasks grace period number 9 (since boot) is 32832 jiffies old.
[…]
Apr 02 09:08:09 thinkpad-benjamin kernel: sysrq: Emergency Sync
Apr 02 09:08:09 thinkpad-benjamin kernel: Emergency Sync complete
Apr 02 09:08:09 thinkpad-benjamin kernel: sysrq: Emergency Remount R/O
prev parent reply other threads:[~2026-04-02 7:35 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-03-27 19:28 [BUG] RCU stall in blk_mq_timeout_work (potentially a regression in 6.19.7 or 6.19.8) Benjamin Eikel
2026-03-31 8:32 ` Thorsten Leemhuis
2026-04-02 7:33 ` Benjamin Eikel [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5090885.31r3eYUQgx@thinkpad-benjamin \
--to=debian@eikel.org \
--cc=axboe@kernel.dk \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=regressions@leemhuis.info \
--cc=regressions@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox