Re: [REGRESSION] 6.9.11: systemd hangs in cgroup_drain_dying during cleanup after podman operations

[Martin Pitt <martin@piware.de>]

Hello Tejun,

(Dropping lizefan.x@bytedance.com from CC:, it doesn't exist any more)

Tejun Heo [2026-04-29  6:21 -1000]:
> Thanks for the report. The dmesg you attached has only a partial sysrq-t
> - the dying-task stacks I need were pushed out of the ring buffer. Could
> you increase log_buf_len, reproduce, trigger sysrq-t, and send the
> resulting dmesg?

Increased to 4M, which was enough. I added it to the bottom of the debug notes
comment [1], direct link: [2]. I suppose its' not necessary any more, but just
for the records..

[1] https://github.com/cockpit-project/bots/pull/8970#issuecomment-4342147158
[2] https://github.com/user-attachments/files/27231725/dmesg-task-dump.txt

Tejun Heo [2026-04-29 11:15 -1000]:
> I think I have the mechanism. The deadlock chains three things together.

You are a genius!

> 3. The container's PID 1 (whatever the entrypoint runs) is in
>    do_exit() but parked in zap_pid_ns_processes' second wait loop:

FTR, the container is pretty dumb, just 

  podman run quay.io/prometheus/busybox sh -c 'echo 123; sleep infinity'

we are not actually interested in the container workload for this tests, but
testing cockpit-podman for managing containers on the host.

However, I just confirmed that busybox'es sh, like "proper" bash, does reap
child processes (unlike for example running `sleep` directly as pid 1, then you
do get zombies)

> ----- min-repro.c -----

On Fedora 44 with 6.9.13, this hangs at

    A: rmdir(/sys/fs/cgroup/drain-min/inner) — wedges if bug present (deliberately NOT wait4-ing C)

root        1501  0.0  0.1   2460  1764 pts/0    D+   06:10   0:00 /tmp/repr
root        1502  0.0  0.0      0     0 pts/0    S+   06:10   0:00 [repr]
root        1503  0.0  0.0      0     0 pts/0    Z+   06:10   0:00 [repr] <defunct>

as expected. It does not wedge up the system in the same way as breaking all
"ls /proc" and such.

On Fedora 44 with older 6.9.10 kernel the reproducer finishes (no hang), with
EBUSY:

: B host pid=1444, C host pid=1445
  pid=1444 NSpid:	1444	1
  pid=1445 NSpid:	1445	2
A: rmdir(/sys/fs/cgroup/drain-min/inner) — wedges if bug present (deliberately NOT wait4-ing C)
A: rmdir returned -1 (errno=16 Device or resource busy)

I suppose you know all that, but just in case confirming on my setup helps in
any way.

Thanks!

Martin