From: Benjamin Eikel <debian@eikel.org>
To: Thorsten Leemhuis <regressions@leemhuis.info>
Cc: linux-kernel@vger.kernel.org, Jens Axboe <axboe@kernel.dk>,
linux-block@vger.kernel.org,
Linux kernel regressions list <regressions@lists.linux.dev>
Subject: Re: [BUG] RCU stall in blk_mq_timeout_work (potentially a regression in 6.19.7 or 6.19.8)
Date: Thu, 02 Apr 2026 09:33:42 +0200 [thread overview]
Message-ID: <5090885.31r3eYUQgx@thinkpad-benjamin> (raw)
In-Reply-To: <b9262818-14e7-4902-af6c-6c7d3560ac4c@leemhuis.info>
Hi Thorsten and others,
Am Dienstag, 31. März 2026, 10:32:30 Mitteleuropäische Sommerzeit schrieb Thorsten Leemhuis:
> [CCing the regressions list and Jens, the Block layer maintainer, who
> might have an idea what's wrong here and/or how to best debug it]
>
> On 3/27/26 20:28, Benjamin Eikel wrote:
> > Dear Linux kernel developers,
> >
> > I experience repeated RCU stalls in blk_mq_timeout_work causing system
> > freezes. The NVMe drive (more info at the end) shows no errors or
> > controller resets in the kernel log: the hardware appears healthy. No I/O
> > timeout messages precede the stalls.
> >
> > The stall cascades: khugepaged blocks on __lru_add_drain_all waiting for a
> > workqueue flush that cannot complete, and additional kworkers block on a
> > mutex held by the kblockd rescuer thread. The system becomes noticeably
> > unresponsive and then I notice it in the logs.
> >
> > In case the logs are too condensed, please tell me and I can provide them
> > fully. If you have pointers on how this could be reproduced and a commit
> > range,>
> > I could try to bisect the problem.
>
> A bisection with mainline would help a lot. Maybe doing it between
> 6.19.6 and 6.19.8 (which would be quicker) might work out, too (but
> normally it's best to focus on mainline).
I have not yet done this because I do not have a reliable way to reproduce it.
>
> And this is not my area of expertise, but I noticed that some RCU fixes
> went into -rc6, so might be wise to try that first.
I encountered this issue with -rc6 (see log below, same config). I suspect that
this only happens after the notebook has previously been suspended.
Kind regards
Benjamin
== Trace from Linux version 7.0.0-rc6 (1–2 April) ==
Apr 01 20:36:59 thinkpad-benjamin kernel: Linux version 7.0.0-rc6-70-rc6 (benjamin@thinkpad-benjamin) (gcc (Debian 15.2.0-16) 15.2.0, GNU ld (GNU Binutils for Debian) 2.46) #24 SMP PREEMPT_DYNAMIC Tue Mar 31 14:36:43 CEST 2026
Apr 01 20:36:59 thinkpad-benjamin kernel: Command line: BOOT_IMAGE=/vmlinuz-7.0.0-rc6-70-rc6 root=/dev/mapper/thinkpad--benjamin--vg-root ro quiet
[…]
Apr 01 21:06:54 thinkpad-benjamin kernel: PM: suspend entry (s2idle)
[…]
Apr 02 08:44:55 thinkpad-benjamin kernel: PM: suspend exit
[…]
Apr 02 08:50:59 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 120 seconds.
Apr 02 08:50:59 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:50:59 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:50:59 thinkpad-benjamin kernel: task:khugepaged state:D stack:0 pid:138 tgid:138 ppid:2 task_flags:0x200040 flags:0x00080000
Apr 02 08:50:59 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:50:59 thinkpad-benjamin kernel: <TASK>
Apr 02 08:50:59 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:50:59 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:50:59 thinkpad-benjamin kernel: schedule_timeout+0xbd/0x100
Apr 02 08:50:59 thinkpad-benjamin kernel: wait_for_completion+0x8a/0x140
Apr 02 08:50:59 thinkpad-benjamin kernel: __flush_work+0x276/0x3b0
Apr 02 08:50:59 thinkpad-benjamin kernel: ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel: __lru_add_drain_all+0x198/0x2b0
Apr 02 08:50:59 thinkpad-benjamin kernel: khugepaged+0x65/0x950
Apr 02 08:50:59 thinkpad-benjamin kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel: ? __pfx_khugepaged+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:50:59 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:50:59 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:50:59 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:50:59 thinkpad-benjamin kernel: </TASK>
Apr 02 08:53:00 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 241 seconds.
Apr 02 08:53:00 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:53:00 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:53:00 thinkpad-benjamin kernel: task:khugepaged state:D stack:0 pid:138 tgid:138 ppid:2 task_flags:0x200040 flags:0x00080000
Apr 02 08:53:00 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:53:00 thinkpad-benjamin kernel: <TASK>
Apr 02 08:53:00 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:53:00 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:53:00 thinkpad-benjamin kernel: schedule_timeout+0xbd/0x100
Apr 02 08:53:00 thinkpad-benjamin kernel: wait_for_completion+0x8a/0x140
Apr 02 08:53:00 thinkpad-benjamin kernel: __flush_work+0x276/0x3b0
Apr 02 08:53:00 thinkpad-benjamin kernel: ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel: __lru_add_drain_all+0x198/0x2b0
Apr 02 08:53:00 thinkpad-benjamin kernel: khugepaged+0x65/0x950
Apr 02 08:53:00 thinkpad-benjamin kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel: ? __pfx_khugepaged+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:53:00 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:53:00 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:53:00 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:53:00 thinkpad-benjamin kernel: </TASK>
Apr 02 08:55:00 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 362 seconds.
Apr 02 08:55:00 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:55:00 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:55:00 thinkpad-benjamin kernel: task:khugepaged state:D stack:0 pid:138 tgid:138 ppid:2 task_flags:0x200040 flags:0x00080000
Apr 02 08:55:00 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:55:00 thinkpad-benjamin kernel: <TASK>
Apr 02 08:55:00 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:55:00 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:55:00 thinkpad-benjamin kernel: schedule_timeout+0xbd/0x100
Apr 02 08:55:00 thinkpad-benjamin kernel: wait_for_completion+0x8a/0x140
Apr 02 08:55:00 thinkpad-benjamin kernel: __flush_work+0x276/0x3b0
Apr 02 08:55:00 thinkpad-benjamin kernel: ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel: __lru_add_drain_all+0x198/0x2b0
Apr 02 08:55:00 thinkpad-benjamin kernel: khugepaged+0x65/0x950
Apr 02 08:55:00 thinkpad-benjamin kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel: ? __pfx_khugepaged+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:55:00 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:55:00 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:55:00 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:55:00 thinkpad-benjamin kernel: </TASK>
Apr 02 08:57:01 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 483 seconds.
Apr 02 08:57:01 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:57:01 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:57:01 thinkpad-benjamin kernel: task:khugepaged state:D stack:0 pid:138 tgid:138 ppid:2 task_flags:0x200040 flags:0x00080000
Apr 02 08:57:01 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:57:01 thinkpad-benjamin kernel: <TASK>
Apr 02 08:57:01 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:57:01 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:57:01 thinkpad-benjamin kernel: schedule_timeout+0xbd/0x100
Apr 02 08:57:01 thinkpad-benjamin kernel: wait_for_completion+0x8a/0x140
Apr 02 08:57:01 thinkpad-benjamin kernel: __flush_work+0x276/0x3b0
Apr 02 08:57:01 thinkpad-benjamin kernel: ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel: __lru_add_drain_all+0x198/0x2b0
Apr 02 08:57:01 thinkpad-benjamin kernel: khugepaged+0x65/0x950
Apr 02 08:57:01 thinkpad-benjamin kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel: ? __pfx_khugepaged+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:57:01 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:57:01 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:57:01 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:57:01 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:14:126 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:14 state:D stack:0 pid:126 tgid:126 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ttwu_queue_wakelist+0x13c/0x230
Apr 02 08:59:02 thinkpad-benjamin kernel: idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel: process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:14:126 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R running task stack:0 pid:139 tgid:139 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: RIP: 0010:affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: Code: 00 49 8b 36 48 8d bb ec 0d 00 00 e8 22 8a d9 00 45 85 ff 44 8b 04 24 0f 84 21 02 00 00 65 ff 0d 4e 73 8b 02 0f 84 6f 03 00 00 <41> 83 e0 04 49 8d 7c 24 08 75 3e e8 18 09 d9 00 b8 ff ff ff ff f0
Apr 02 08:59:02 thinkpad-benjamin kernel: RSP: 0018:ffffcc5f4063fcd8 EFLAGS: 00000246
Apr 02 08:59:02 thinkpad-benjamin kernel: RAX: 0000000000000001 RBX: ffff8af641ab2180 RCX: 0000000000000004
Apr 02 08:59:02 thinkpad-benjamin kernel: RDX: 0000000000000000 RSI: 0000000000000283 RDI: ffff8af640c08dec
Apr 02 08:59:02 thinkpad-benjamin kernel: RBP: ffff8b04a1eb2100 R08: 0000000000000000 R09: 0000000000000283
Apr 02 08:59:02 thinkpad-benjamin kernel: R10: 0000000000000002 R11: 0000000000000000 R12: ffffcc5f4063fd28
Apr 02 08:59:02 thinkpad-benjamin kernel: R13: ffffcc5f4063fd28 R14: ffffcc5f4063fde0 R15: 0000000000000000
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel: set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task khugepaged:138 blocked for more than 604 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:khugepaged state:D stack:0 pid:138 tgid:138 ppid:2 task_flags:0x200040 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_timeout+0xbd/0x100
Apr 02 08:59:02 thinkpad-benjamin kernel: wait_for_completion+0x8a/0x140
Apr 02 08:59:02 thinkpad-benjamin kernel: __flush_work+0x276/0x3b0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_wq_barrier_func+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: __lru_add_drain_all+0x198/0x2b0
Apr 02 08:59:02 thinkpad-benjamin kernel: khugepaged+0x65/0x950
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_autoremove_wake_function+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_khugepaged+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:16:5369 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:16 state:D stack:0 pid:5369 tgid:5369 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? finish_task_switch.isra.0+0x96/0x2c0
Apr 02 08:59:02 thinkpad-benjamin kernel: idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel: process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:16:5369 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R running task stack:0 pid:139 tgid:139 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __schedule+0x484/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: ? preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: ? asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: ? worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:12:9088 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:12 state:D stack:0 pid:9088 tgid:9088 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel: idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel: process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:12:9088 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R running task stack:0 pid:139 tgid:139 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: RIP: 0010:affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: Code: 00 49 8b 36 48 8d bb ec 0d 00 00 e8 22 8a d9 00 45 85 ff 44 8b 04 24 0f 84 21 02 00 00 65 ff 0d 4e 73 8b 02 0f 84 6f 03 00 00 <41> 83 e0 04 49 8d 7c 24 08 75 3e e8 18 09 d9 00 b8 ff ff ff ff f0
Apr 02 08:59:02 thinkpad-benjamin kernel: RSP: 0018:ffffcc5f4063fcd8 EFLAGS: 00000246
Apr 02 08:59:02 thinkpad-benjamin kernel: RAX: 0000000000000001 RBX: ffff8af641ab2180 RCX: 0000000000000004
Apr 02 08:59:02 thinkpad-benjamin kernel: RDX: 0000000000000000 RSI: 0000000000000283 RDI: ffff8af640c08dec
Apr 02 08:59:02 thinkpad-benjamin kernel: RBP: ffff8b04a1eb2100 R08: 0000000000000000 R09: 0000000000000283
Apr 02 08:59:02 thinkpad-benjamin kernel: R10: 0000000000000002 R11: 0000000000000000 R12: ffffcc5f4063fd28
Apr 02 08:59:02 thinkpad-benjamin kernel: R13: ffffcc5f4063fd28 R14: ffffcc5f4063fde0 R15: 0000000000000000
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel: set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:22:10249 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:22 state:D stack:0 pid:10249 tgid:10249 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? finish_task_switch.isra.0+0x96/0x2c0
Apr 02 08:59:02 thinkpad-benjamin kernel: idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel: process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:22:10249 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R running task stack:0 pid:139 tgid:139 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: RIP: 0010:affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: Code: 00 49 8b 36 48 8d bb ec 0d 00 00 e8 22 8a d9 00 45 85 ff 44 8b 04 24 0f 84 21 02 00 00 65 ff 0d 4e 73 8b 02 0f 84 6f 03 00 00 <41> 83 e0 04 49 8d 7c 24 08 75 3e e8 18 09 d9 00 b8 ff ff ff ff f0
Apr 02 08:59:02 thinkpad-benjamin kernel: RSP: 0018:ffffcc5f4063fcd8 EFLAGS: 00000246
Apr 02 08:59:02 thinkpad-benjamin kernel: RAX: 0000000000000001 RBX: ffff8af641ab2180 RCX: 0000000000000004
Apr 02 08:59:02 thinkpad-benjamin kernel: RDX: 0000000000000000 RSI: 0000000000000283 RDI: ffff8af640c08dec
Apr 02 08:59:02 thinkpad-benjamin kernel: RBP: ffff8b04a1eb2100 R08: 0000000000000000 R09: 0000000000000283
Apr 02 08:59:02 thinkpad-benjamin kernel: R10: 0000000000000002 R11: 0000000000000000 R12: ffffcc5f4063fd28
Apr 02 08:59:02 thinkpad-benjamin kernel: R13: ffffcc5f4063fd28 R14: ffffcc5f4063fde0 R15: 0000000000000000
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel: set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:31:10258 blocked for more than 120 seconds.
Apr 02 08:59:02 thinkpad-benjamin kernel: Not tainted 7.0.0-rc6-70-rc6 #24
Apr 02 08:59:02 thinkpad-benjamin kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/u64:31 state:D stack:0 pid:10258 tgid:10258 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Workqueue: events_unbound idle_cull_fn
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: __schedule+0x47c/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule+0x27/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: schedule_preempt_disabled+0x15/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: __mutex_lock.constprop.0+0x54b/0xac0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ttwu_queue_wakelist+0x13c/0x230
Apr 02 08:59:02 thinkpad-benjamin kernel: idle_cull_fn+0x3e/0x170
Apr 02 08:59:02 thinkpad-benjamin kernel: process_one_work+0x19d/0x3a0
Apr 02 08:59:02 thinkpad-benjamin kernel: worker_thread+0x1af/0x320
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_worker_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: INFO: task kworker/u64:31:10258 is blocked on a mutex likely owned by task kworker/R-kbloc:139.
Apr 02 08:59:02 thinkpad-benjamin kernel: task:kworker/R-kbloc state:R running task stack:0 pid:139 tgid:139 ppid:2 task_flags:0x4208060 flags:0x00080000
Apr 02 08:59:02 thinkpad-benjamin kernel: Call Trace:
Apr 02 08:59:02 thinkpad-benjamin kernel: <TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __schedule+0x484/0x1000
Apr 02 08:59:02 thinkpad-benjamin kernel: ? preempt_schedule_irq+0x38/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: ? asm_common_interrupt+0x26/0x40
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x198/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x50c/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? srso_alias_return_thunk+0x5/0xfbef5
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_next_entity+0xe7/0x270
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_migration_cpu_stop+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? affine_move_task+0x3c8/0x580
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __set_cpus_allowed_ptr+0x66/0xc0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? set_cpus_allowed_ptr+0x3a/0x60
Apr 02 08:59:02 thinkpad-benjamin kernel: ? worker_attach_to_pool+0x5f/0xd0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? rescuer_thread+0x12d/0x5d0
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_rescuer_thread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? kthread+0xe3/0x120
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ret_from_fork+0x2c9/0x360
Apr 02 08:59:02 thinkpad-benjamin kernel: ? __pfx_kthread+0x10/0x10
Apr 02 08:59:02 thinkpad-benjamin kernel: ? ret_from_fork_asm+0x1a/0x30
Apr 02 08:59:02 thinkpad-benjamin kernel: </TASK>
Apr 02 08:59:02 thinkpad-benjamin kernel: Future hung task reports are suppressed, see sysctl kernel.hung_task_warnings
[…]
Apr 02 09:05:10 thinkpad-benjamin kernel: rcu_tasks_wait_gp: rcu_tasks grace period number 9 (since boot) is 2583 jiffies old.
Apr 02 09:05:40 thinkpad-benjamin kernel: rcu_tasks_wait_gp: rcu_tasks grace period number 9 (since boot) is 10136 jiffies old.
[…]
Apr 02 09:07:11 thinkpad-benjamin kernel: rcu_tasks_wait_gp: rcu_tasks grace period number 9 (since boot) is 32832 jiffies old.
[…]
Apr 02 09:08:09 thinkpad-benjamin kernel: sysrq: Emergency Sync
Apr 02 09:08:09 thinkpad-benjamin kernel: Emergency Sync complete
Apr 02 09:08:09 thinkpad-benjamin kernel: sysrq: Emergency Remount R/O
prev parent reply other threads:[~2026-04-02 7:35 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-03-27 19:28 [BUG] RCU stall in blk_mq_timeout_work (potentially a regression in 6.19.7 or 6.19.8) Benjamin Eikel
2026-03-31 8:32 ` Thorsten Leemhuis
2026-04-02 7:33 ` Benjamin Eikel [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5090885.31r3eYUQgx@thinkpad-benjamin \
--to=debian@eikel.org \
--cc=axboe@kernel.dk \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=regressions@leemhuis.info \
--cc=regressions@lists.linux.dev \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.