[Kernel Bug] INFO: task hung in cgroup_drain_dying

[Kernel Bug] INFO: task hung in cgroup_drain_dying

[Longxing Li]

Dear Linux kernel developers and maintainers,

We would like to report a new kernel bug found by our tool. INFO: task
hung in cgroup_drain_dying. Details are as follows.

Kernel commit: v7.0.6
Kernel config: see attachment
report: see attachment

We are currently analyzing the root cause and  working on a
reproducible PoC. We will provide further updates in this thread as
soon as we have more information.

Best regards,
Longxing Li

==================================================================
https://drive.google.com/file/d/1riFUIPWojkYVZu0B5BW8uVPocUWwibqN/view?usp=drive_link

Re: [Kernel Bug] INFO: task hung in cgroup_drain_dying

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

Hello Longxing.

On Tue, Jun 09, 2026 at 07:42:06PM +0800, Longxing Li <coregee2000@gmail.com> wrote:
> We would like to report a new kernel bug found by our tool. INFO: task
> hung in cgroup_drain_dying. Details are as follows.

Thanks but I see no attachment.

(Greater if you could add description as plaintext [1])

> Kernel commit: v7.0.6
> Kernel config: see attachment

Do you have lockdep enabled (CONFIG_PROVE_LOCKING)? That may help
debugging here.

Thanks,
Michal

[1] https://docs.kernel.org/process/email-clients.html#general-preferences


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]

Re: [Kernel Bug] INFO: task hung in cgroup_drain_dying

[Longxing Li]

sorry for not containing full information in last email. the config[1]
and report[2] are as follows. CONFIG_PROVE_LOCKING is not enabled in
our config.

[1] https://drive.google.com/file/d/1Bx2unEf-QntjVi8g6Zw7QNO6OP4cjGO_/view?usp=drive_link

[2] https://drive.google.com/file/d/1riFUIPWojkYVZu0B5BW8uVPocUWwibqN/view?usp=sharing

and report plain text is as follows:

INFO: task systemd:1 blocked for more than 143 seconds.
      Not tainted 7.0.6 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd         state:D stack:20616 pid:1     tgid:1     ppid:0
   task_flags:0x400100 flags:0x00080001
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5298 [inline]
 __schedule+0x1006/0x5f00 kernel/sched/core.c:6911
 __schedule_loop kernel/sched/core.c:6993 [inline]
 schedule+0xe7/0x3a0 kernel/sched/core.c:7008
 cgroup_drain_dying+0x1ed/0x360 kernel/cgroup/cgroup.c:6294
 cgroup_rmdir+0x38/0x300 kernel/cgroup/cgroup.c:6309
 kernfs_iop_rmdir+0x10a/0x180 fs/kernfs/dir.c:1311
 vfs_rmdir fs/namei.c:5344 [inline]
 vfs_rmdir+0x340/0x860 fs/namei.c:5317
 filename_rmdir+0x3be/0x510 fs/namei.c:5399
 __do_sys_rmdir fs/namei.c:5422 [inline]
 __se_sys_rmdir fs/namei.c:5419 [inline]
 __x64_sys_rmdir+0x47/0x90 fs/namei.c:5419
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0x11b/0xf80 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fb6c32a61c7
RSP: 002b:00007fff90d2bc98 EFLAGS: 00000202 ORIG_RAX: 0000000000000054
RAX: ffffffffffffffda RBX: 000055c177d80fb0 RCX: 00007fb6c32a61c7
RDX: 00007fb6c3387be0 RSI: 0000000000000000 RDI: 000055c177eb1300
RBP: 00007fb6c35eb2da R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000100 R11: 0000000000000202 R12: 0000000000000000
R13: 00007fb6c2ddb6c8 R14: 0000000000000001 R15: 0000000000000000
 </TASK>

Showing all locks held in the system:
3 locks held by systemd/1:
 #0: ffff8880294f8420 (sb_writers#10){.+.+}-{0:0}, at:
filename_rmdir+0x2cc/0x510 fs/namei.c:5388
 #1: ffff888034d16e98 (&type->i_mutex_dir_key#6/1){+.+.}-{4:4}, at:
inode_lock_nested include/linux/fs.h:1073 [inline]
 #1: ffff888034d16e98 (&type->i_mutex_dir_key#6/1){+.+.}-{4:4}, at:
__start_dirop fs/namei.c:2929 [inline]
 #1: ffff888034d16e98 (&type->i_mutex_dir_key#6/1){+.+.}-{4:4}, at:
start_dirop fs/namei.c:2940 [inline]
 #1: ffff888034d16e98 (&type->i_mutex_dir_key#6/1){+.+.}-{4:4}, at:
filename_rmdir+0x318/0x510 fs/namei.c:5392
 #2: ffff8880386d7888 (&type->i_mutex_dir_key#6){++++}-{4:4}, at:
inode_lock include/linux/fs.h:1028 [inline]
 #2: ffff8880386d7888 (&type->i_mutex_dir_key#6){++++}-{4:4}, at:
vfs_rmdir fs/namei.c:5329 [inline]
 #2: ffff8880386d7888 (&type->i_mutex_dir_key#6){++++}-{4:4}, at:
vfs_rmdir+0xef/0x860 fs/namei.c:5317
6 locks held by kworker/u4:0/12:
3 locks held by kworker/u4:1/13:
1 lock held by khungtaskd/25:
 #0: ffffffff8e5e6ce0 (rcu_read_lock){....}-{1:3}, at:
rcu_lock_acquire include/linux/rcupdate.h:312 [inline]
 #0: ffffffff8e5e6ce0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock
include/linux/rcupdate.h:850 [inline]
 #0: ffffffff8e5e6ce0 (rcu_read_lock){....}-{1:3}, at:
debug_show_all_locks+0x36/0x1c0 kernel/locking/lockdep.c:6775
1 lock held by kcompactd0/28:
3 locks held by kworker/u4:3/45:
2 locks held by kworker/0:2/49:
3 locks held by kworker/u4:6/597:
3 locks held by kworker/u4:8/3491:
2 locks held by systemd-journal/5166:
2 locks held by systemd-udevd/5178:
1 lock held by in:imklog/9177:
4 locks held by sshd/9696:
2 locks held by syz-fuzzer/32911:
2 locks held by syz-executor.6/9754:
2 locks held by syz-executor.7/9774:
1 lock held by syz-executor.2/9812:
1 lock held by syz-executor.1/9902:
2 locks held by syz-executor.14/10080:
2 locks held by syz-executor.9/10842:
1 lock held by syz-executor.15/11893:
 #0: ffffffff8e5f25f8 (rcu_state.exp_mutex){+.+.}-{4:4}, at:
exp_funnel_lock+0x1a3/0x3b0 kernel/rcu/tree_exp.h:343
3 locks held by kworker/0:8/13140:
 #0: ffff88801b8a6948 ((wq_completion)events){+.+.}-{0:0}, at:
process_one_work+0x139e/0x1c60 kernel/workqueue.c:3263
 #1: ffffc9000cd37d08 (free_ipc_work){+.+.}-{0:0}, at:
process_one_work+0x938/0x1c60 kernel/workqueue.c:3264
 #2: ffffffff8e5f25f8 (rcu_state.exp_mutex){+.+.}-{4:4}, at:
exp_funnel_lock+0x1a3/0x3b0 kernel/rcu/tree_exp.h:343
2 locks held by kworker/0:10/13232:
3 locks held by kworker/u4:10/13343:
3 locks held by kworker/u4:12/14656:
1 lock held by syz-executor.13/24672:
3 locks held by kworker/u4:5/45131:
3 locks held by kworker/u4:9/46406:
3 locks held by kworker/u4:13/46990:
3 locks held by kworker/u4:16/46993:
2 locks held by syz-executor.8/48198:
3 locks held by kworker/u4:17/53143:
4 locks held by kworker/u4:18/53144:
2 locks held by systemd-rfkill/53174:
2 locks held by syz-executor.7/53471:
2 locks held by kworker/u4:20/53472:
3 locks held by kworker/u4:21/53476:
3 locks held by kworker/u4:22/53479:
3 locks held by kworker/u4:24/53484:
3 locks held by kworker/u4:25/53488:
2 locks held by kworker/0:19/53491:
2 locks held by systemd-udevd/53495:

=============================================

NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 25 Comm: khungtaskd Not tainted 7.0.6 #1 PREEMPT(full)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x116/0x1b0 lib/dump_stack.c:120
 nmi_cpu_backtrace+0x2a0/0x350 lib/nmi_backtrace.c:113
 nmi_trigger_cpumask_backtrace+0x29c/0x300 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:161 [inline]
 __sys_info lib/sys_info.c:157 [inline]
 sys_info+0x133/0x180 lib/sys_info.c:165
 check_hung_uninterruptible_tasks kernel/hung_task.c:346 [inline]
 watchdog+0xeac/0x11e0 kernel/hung_task.c:515
 kthread+0x38d/0x4a0 kernel/kthread.c:436
 ret_from_fork+0x942/0xe50 arch/x86/kernel/process.c:158
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
 </TASK>

Michal Koutný <mkoutny@suse.com> 于2026年6月9日周二 20:58写道：
>
> Hello Longxing.
>
> On Tue, Jun 09, 2026 at 07:42:06PM +0800, Longxing Li <coregee2000@gmail.com> wrote:
> > We would like to report a new kernel bug found by our tool. INFO: task
> > hung in cgroup_drain_dying. Details are as follows.
>
> Thanks but I see no attachment.
>
> (Greater if you could add description as plaintext [1])
>
> > Kernel commit: v7.0.6
> > Kernel config: see attachment
>
> Do you have lockdep enabled (CONFIG_PROVE_LOCKING)? That may help
> debugging here.
>
> Thanks,
> Michal
>
> [1] https://docs.kernel.org/process/email-clients.html#general-preferences
>

Re: [Kernel Bug] INFO: task hung in cgroup_drain_dying

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 1588 bytes --]

On Wed, Jun 10, 2026 at 03:11:41PM +0800, Longxing Li <coregee2000@gmail.com> wrote:
> sorry for not containing full information in last email. the config[1]
> and report[2] are as follows. CONFIG_PROVE_LOCKING is not enabled in
> our config.

Thanks.

> INFO: task systemd:1 blocked for more than 143 seconds.
>       Not tainted 7.0.6 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> task:systemd         state:D stack:20616 pid:1     tgid:1     ppid:0
>    task_flags:0x400100 flags:0x00080001
> Call Trace:
>  <TASK>
>  context_switch kernel/sched/core.c:5298 [inline]
>  __schedule+0x1006/0x5f00 kernel/sched/core.c:6911
>  __schedule_loop kernel/sched/core.c:6993 [inline]
>  schedule+0xe7/0x3a0 kernel/sched/core.c:7008
>  cgroup_drain_dying+0x1ed/0x360 kernel/cgroup/cgroup.c:6294
>  cgroup_rmdir+0x38/0x300 kernel/cgroup/cgroup.c:6309
>  kernfs_iop_rmdir+0x10a/0x180 fs/kernfs/dir.c:1311
>  vfs_rmdir fs/namei.c:5344 [inline]
>  vfs_rmdir+0x340/0x860 fs/namei.c:5317
>  filename_rmdir+0x3be/0x510 fs/namei.c:5399
>  __do_sys_rmdir fs/namei.c:5422 [inline]
>  __se_sys_rmdir fs/namei.c:5419 [inline]
>  __x64_sys_rmdir+0x47/0x90 fs/namei.c:5419
>  do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
>  do_syscall_64+0x11b/0xf80 arch/x86/entry/syscall_64.c:94
>  entry_SYSCALL_64_after_hwframe+0x77/0x7f

Hm, hm, this kinds fits 93618edf75383 ("cgroup: Defer css percpu_ref
kill on rmdir until cgroup is depopulated") 
which got into stable 7.0.9.
Can you reproduce even with that (or newer) kernel?

Michal

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]