[Kernel Bug] INFO: task hung in cgroup_drain_dying

[Kernel Bug] INFO: task hung in cgroup_drain_dying

[Longxing Li]

Dear Linux kernel developers and maintainers,

We would like to report a new kernel bug found by our tool. INFO: task
hung in cgroup_drain_dying. Details are as follows.

Kernel commit: v7.0.6
Kernel config: see attachment
report: see attachment

We are currently analyzing the root cause and  working on a
reproducible PoC. We will provide further updates in this thread as
soon as we have more information.

Best regards,
Longxing Li

==================================================================
https://drive.google.com/file/d/1riFUIPWojkYVZu0B5BW8uVPocUWwibqN/view?usp=drive_link

Re: [Kernel Bug] INFO: task hung in cgroup_drain_dying

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 552 bytes --]

Hello Longxing.

On Tue, Jun 09, 2026 at 07:42:06PM +0800, Longxing Li <coregee2000@gmail.com> wrote:
> We would like to report a new kernel bug found by our tool. INFO: task
> hung in cgroup_drain_dying. Details are as follows.

Thanks but I see no attachment.

(Greater if you could add description as plaintext [1])

> Kernel commit: v7.0.6
> Kernel config: see attachment

Do you have lockdep enabled (CONFIG_PROVE_LOCKING)? That may help
debugging here.

Thanks,
Michal

[1] https://docs.kernel.org/process/email-clients.html#general-preferences


[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]

Re: [Kernel Bug] INFO: task hung in cgroup_drain_dying

[Longxing Li]

sorry for not containing full information in last email. the config[1]
and report[2] are as follows. CONFIG_PROVE_LOCKING is not enabled in
our config.

[1] https://drive.google.com/file/d/1Bx2unEf-QntjVi8g6Zw7QNO6OP4cjGO_/view?usp=drive_link

[2] https://drive.google.com/file/d/1riFUIPWojkYVZu0B5BW8uVPocUWwibqN/view?usp=sharing

and report plain text is as follows:

INFO: task systemd:1 blocked for more than 143 seconds.
      Not tainted 7.0.6 #1
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
task:systemd         state:D stack:20616 pid:1     tgid:1     ppid:0
   task_flags:0x400100 flags:0x00080001
Call Trace:
 <TASK>
 context_switch kernel/sched/core.c:5298 [inline]
 __schedule+0x1006/0x5f00 kernel/sched/core.c:6911
 __schedule_loop kernel/sched/core.c:6993 [inline]
 schedule+0xe7/0x3a0 kernel/sched/core.c:7008
 cgroup_drain_dying+0x1ed/0x360 kernel/cgroup/cgroup.c:6294
 cgroup_rmdir+0x38/0x300 kernel/cgroup/cgroup.c:6309
 kernfs_iop_rmdir+0x10a/0x180 fs/kernfs/dir.c:1311
 vfs_rmdir fs/namei.c:5344 [inline]
 vfs_rmdir+0x340/0x860 fs/namei.c:5317
 filename_rmdir+0x3be/0x510 fs/namei.c:5399
 __do_sys_rmdir fs/namei.c:5422 [inline]
 __se_sys_rmdir fs/namei.c:5419 [inline]
 __x64_sys_rmdir+0x47/0x90 fs/namei.c:5419
 do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
 do_syscall_64+0x11b/0xf80 arch/x86/entry/syscall_64.c:94
 entry_SYSCALL_64_after_hwframe+0x77/0x7f
RIP: 0033:0x7fb6c32a61c7
RSP: 002b:00007fff90d2bc98 EFLAGS: 00000202 ORIG_RAX: 0000000000000054
RAX: ffffffffffffffda RBX: 000055c177d80fb0 RCX: 00007fb6c32a61c7
RDX: 00007fb6c3387be0 RSI: 0000000000000000 RDI: 000055c177eb1300
RBP: 00007fb6c35eb2da R08: 0000000000000000 R09: 0000000000000001
R10: 0000000000000100 R11: 0000000000000202 R12: 0000000000000000
R13: 00007fb6c2ddb6c8 R14: 0000000000000001 R15: 0000000000000000
 </TASK>

Showing all locks held in the system:
3 locks held by systemd/1:
 #0: ffff8880294f8420 (sb_writers#10){.+.+}-{0:0}, at:
filename_rmdir+0x2cc/0x510 fs/namei.c:5388
 #1: ffff888034d16e98 (&type->i_mutex_dir_key#6/1){+.+.}-{4:4}, at:
inode_lock_nested include/linux/fs.h:1073 [inline]
 #1: ffff888034d16e98 (&type->i_mutex_dir_key#6/1){+.+.}-{4:4}, at:
__start_dirop fs/namei.c:2929 [inline]
 #1: ffff888034d16e98 (&type->i_mutex_dir_key#6/1){+.+.}-{4:4}, at:
start_dirop fs/namei.c:2940 [inline]
 #1: ffff888034d16e98 (&type->i_mutex_dir_key#6/1){+.+.}-{4:4}, at:
filename_rmdir+0x318/0x510 fs/namei.c:5392
 #2: ffff8880386d7888 (&type->i_mutex_dir_key#6){++++}-{4:4}, at:
inode_lock include/linux/fs.h:1028 [inline]
 #2: ffff8880386d7888 (&type->i_mutex_dir_key#6){++++}-{4:4}, at:
vfs_rmdir fs/namei.c:5329 [inline]
 #2: ffff8880386d7888 (&type->i_mutex_dir_key#6){++++}-{4:4}, at:
vfs_rmdir+0xef/0x860 fs/namei.c:5317
6 locks held by kworker/u4:0/12:
3 locks held by kworker/u4:1/13:
1 lock held by khungtaskd/25:
 #0: ffffffff8e5e6ce0 (rcu_read_lock){....}-{1:3}, at:
rcu_lock_acquire include/linux/rcupdate.h:312 [inline]
 #0: ffffffff8e5e6ce0 (rcu_read_lock){....}-{1:3}, at: rcu_read_lock
include/linux/rcupdate.h:850 [inline]
 #0: ffffffff8e5e6ce0 (rcu_read_lock){....}-{1:3}, at:
debug_show_all_locks+0x36/0x1c0 kernel/locking/lockdep.c:6775
1 lock held by kcompactd0/28:
3 locks held by kworker/u4:3/45:
2 locks held by kworker/0:2/49:
3 locks held by kworker/u4:6/597:
3 locks held by kworker/u4:8/3491:
2 locks held by systemd-journal/5166:
2 locks held by systemd-udevd/5178:
1 lock held by in:imklog/9177:
4 locks held by sshd/9696:
2 locks held by syz-fuzzer/32911:
2 locks held by syz-executor.6/9754:
2 locks held by syz-executor.7/9774:
1 lock held by syz-executor.2/9812:
1 lock held by syz-executor.1/9902:
2 locks held by syz-executor.14/10080:
2 locks held by syz-executor.9/10842:
1 lock held by syz-executor.15/11893:
 #0: ffffffff8e5f25f8 (rcu_state.exp_mutex){+.+.}-{4:4}, at:
exp_funnel_lock+0x1a3/0x3b0 kernel/rcu/tree_exp.h:343
3 locks held by kworker/0:8/13140:
 #0: ffff88801b8a6948 ((wq_completion)events){+.+.}-{0:0}, at:
process_one_work+0x139e/0x1c60 kernel/workqueue.c:3263
 #1: ffffc9000cd37d08 (free_ipc_work){+.+.}-{0:0}, at:
process_one_work+0x938/0x1c60 kernel/workqueue.c:3264
 #2: ffffffff8e5f25f8 (rcu_state.exp_mutex){+.+.}-{4:4}, at:
exp_funnel_lock+0x1a3/0x3b0 kernel/rcu/tree_exp.h:343
2 locks held by kworker/0:10/13232:
3 locks held by kworker/u4:10/13343:
3 locks held by kworker/u4:12/14656:
1 lock held by syz-executor.13/24672:
3 locks held by kworker/u4:5/45131:
3 locks held by kworker/u4:9/46406:
3 locks held by kworker/u4:13/46990:
3 locks held by kworker/u4:16/46993:
2 locks held by syz-executor.8/48198:
3 locks held by kworker/u4:17/53143:
4 locks held by kworker/u4:18/53144:
2 locks held by systemd-rfkill/53174:
2 locks held by syz-executor.7/53471:
2 locks held by kworker/u4:20/53472:
3 locks held by kworker/u4:21/53476:
3 locks held by kworker/u4:22/53479:
3 locks held by kworker/u4:24/53484:
3 locks held by kworker/u4:25/53488:
2 locks held by kworker/0:19/53491:
2 locks held by systemd-udevd/53495:

=============================================

NMI backtrace for cpu 0
CPU: 0 UID: 0 PID: 25 Comm: khungtaskd Not tainted 7.0.6 #1 PREEMPT(full)
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.15.0-1 04/01/2014
Call Trace:
 <TASK>
 __dump_stack lib/dump_stack.c:94 [inline]
 dump_stack_lvl+0x116/0x1b0 lib/dump_stack.c:120
 nmi_cpu_backtrace+0x2a0/0x350 lib/nmi_backtrace.c:113
 nmi_trigger_cpumask_backtrace+0x29c/0x300 lib/nmi_backtrace.c:62
 trigger_all_cpu_backtrace include/linux/nmi.h:161 [inline]
 __sys_info lib/sys_info.c:157 [inline]
 sys_info+0x133/0x180 lib/sys_info.c:165
 check_hung_uninterruptible_tasks kernel/hung_task.c:346 [inline]
 watchdog+0xeac/0x11e0 kernel/hung_task.c:515
 kthread+0x38d/0x4a0 kernel/kthread.c:436
 ret_from_fork+0x942/0xe50 arch/x86/kernel/process.c:158
 ret_from_fork_asm+0x1a/0x30 arch/x86/entry/entry_64.S:245
 </TASK>

Michal Koutný <mkoutny@suse.com> 于2026年6月9日周二 20:58写道：
>
> Hello Longxing.
>
> On Tue, Jun 09, 2026 at 07:42:06PM +0800, Longxing Li <coregee2000@gmail.com> wrote:
> > We would like to report a new kernel bug found by our tool. INFO: task
> > hung in cgroup_drain_dying. Details are as follows.
>
> Thanks but I see no attachment.
>
> (Greater if you could add description as plaintext [1])
>
> > Kernel commit: v7.0.6
> > Kernel config: see attachment
>
> Do you have lockdep enabled (CONFIG_PROVE_LOCKING)? That may help
> debugging here.
>
> Thanks,
> Michal
>
> [1] https://docs.kernel.org/process/email-clients.html#general-preferences
>

Re: [Kernel Bug] INFO: task hung in cgroup_drain_dying

[Michal Koutný]

[-- Attachment #1: Type: text/plain, Size: 1588 bytes --]

On Wed, Jun 10, 2026 at 03:11:41PM +0800, Longxing Li <coregee2000@gmail.com> wrote:
> sorry for not containing full information in last email. the config[1]
> and report[2] are as follows. CONFIG_PROVE_LOCKING is not enabled in
> our config.

Thanks.

> INFO: task systemd:1 blocked for more than 143 seconds.
>       Not tainted 7.0.6 #1
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> task:systemd         state:D stack:20616 pid:1     tgid:1     ppid:0
>    task_flags:0x400100 flags:0x00080001
> Call Trace:
>  <TASK>
>  context_switch kernel/sched/core.c:5298 [inline]
>  __schedule+0x1006/0x5f00 kernel/sched/core.c:6911
>  __schedule_loop kernel/sched/core.c:6993 [inline]
>  schedule+0xe7/0x3a0 kernel/sched/core.c:7008
>  cgroup_drain_dying+0x1ed/0x360 kernel/cgroup/cgroup.c:6294
>  cgroup_rmdir+0x38/0x300 kernel/cgroup/cgroup.c:6309
>  kernfs_iop_rmdir+0x10a/0x180 fs/kernfs/dir.c:1311
>  vfs_rmdir fs/namei.c:5344 [inline]
>  vfs_rmdir+0x340/0x860 fs/namei.c:5317
>  filename_rmdir+0x3be/0x510 fs/namei.c:5399
>  __do_sys_rmdir fs/namei.c:5422 [inline]
>  __se_sys_rmdir fs/namei.c:5419 [inline]
>  __x64_sys_rmdir+0x47/0x90 fs/namei.c:5419
>  do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
>  do_syscall_64+0x11b/0xf80 arch/x86/entry/syscall_64.c:94
>  entry_SYSCALL_64_after_hwframe+0x77/0x7f

Hm, hm, this kinds fits 93618edf75383 ("cgroup: Defer css percpu_ref
kill on rmdir until cgroup is depopulated") 
which got into stable 7.0.9.
Can you reproduce even with that (or newer) kernel?

Michal

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]

Re: [Kernel Bug] INFO: task hung in cgroup_drain_dying

[Longxing Li]

Hi Michal,
  Thank you for the reply.
  We will check this problem on the latest kernel to see if it's been
fixed, and will report back if we find any more valuable things to
discuss.

  Best regards,
  Longxing Li

Michal Koutný <mkoutny@suse.com> 于2026年6月10日周三 21:28写道：
>
> On Wed, Jun 10, 2026 at 03:11:41PM +0800, Longxing Li <coregee2000@gmail.com> wrote:
> > sorry for not containing full information in last email. the config[1]
> > and report[2] are as follows. CONFIG_PROVE_LOCKING is not enabled in
> > our config.
>
> Thanks.
>
> > INFO: task systemd:1 blocked for more than 143 seconds.
> >       Not tainted 7.0.6 #1
> > "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> > task:systemd         state:D stack:20616 pid:1     tgid:1     ppid:0
> >    task_flags:0x400100 flags:0x00080001
> > Call Trace:
> >  <TASK>
> >  context_switch kernel/sched/core.c:5298 [inline]
> >  __schedule+0x1006/0x5f00 kernel/sched/core.c:6911
> >  __schedule_loop kernel/sched/core.c:6993 [inline]
> >  schedule+0xe7/0x3a0 kernel/sched/core.c:7008
> >  cgroup_drain_dying+0x1ed/0x360 kernel/cgroup/cgroup.c:6294
> >  cgroup_rmdir+0x38/0x300 kernel/cgroup/cgroup.c:6309
> >  kernfs_iop_rmdir+0x10a/0x180 fs/kernfs/dir.c:1311
> >  vfs_rmdir fs/namei.c:5344 [inline]
> >  vfs_rmdir+0x340/0x860 fs/namei.c:5317
> >  filename_rmdir+0x3be/0x510 fs/namei.c:5399
> >  __do_sys_rmdir fs/namei.c:5422 [inline]
> >  __se_sys_rmdir fs/namei.c:5419 [inline]
> >  __x64_sys_rmdir+0x47/0x90 fs/namei.c:5419
> >  do_syscall_x64 arch/x86/entry/syscall_64.c:63 [inline]
> >  do_syscall_64+0x11b/0xf80 arch/x86/entry/syscall_64.c:94
> >  entry_SYSCALL_64_after_hwframe+0x77/0x7f
>
> Hm, hm, this kinds fits 93618edf75383 ("cgroup: Defer css percpu_ref
> kill on rmdir until cgroup is depopulated")
> which got into stable 7.0.9.
> Can you reproduce even with that (or newer) kernel?
>
> Michal