* [PATCH] cgroup: avoid flushing global workqueue in cgroup1_pidlist_destroy_all
@ 2026-08-14 8:40 Junnan Zhang
2026-08-14 9:45 ` [PATCH v2] " Junnan Zhang
2026-08-14 10:20 ` [PATCH v3] " Junnan Zhang
0 siblings, 2 replies; 6+ messages in thread
From: Junnan Zhang @ 2026-08-14 8:40 UTC (permalink / raw)
To: tj, hannes, mkoutny
Cc: cgroups, linux-kernel, zhangjn_dev, Junnan Zhang, Shouxin Sun
From: Junnan Zhang <zhangjn11@chinatelecom.cn>
cgroup1_pidlist_destroy_all() flushes the global
cgroup_pidlist_destroy_wq while destroying a cgroup. Because all cgroup
v1 pidlist destruction work items are queued on the same shared workqueue,
a single slow or stuck work item (e.g. waiting for pidlist_mutex held by a
user-space reader) blocks every concurrent cgroup destruction path.
This can lead to kworker tasks stuck in flush_workqueue() for over
hung_task_timeout seconds, as observed on busy systems running Docker or
Kubernetes workloads.
INFO: task kworker/0:1:1438499 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/0:1 D 0 1438499 2 0x80000080
Workqueue: cgroup_destroy css_free_rwork_fn
? __schedule+0x296/0x900
schedule+0x28/0x80
schedule_timeout+0x1ee/0x3a0
? kvm_sched_clock_read+0xd/0x20
wait_for_completion+0x12c/0x190
? wake_up_q+0x70/0x70
flush_workqueue+0x132/0x430
? cgroup1_pidlist_destroy_all+0x7c/0xa0
cgroup1_pidlist_destroy_all+0x7c/0xa0
css_free_rwork_fn+0xb5/0x390
process_one_work+0x195/0x3e0
worker_thread+0x30/0x390
? process_one_work+0x3e0/0x3e0
kthread+0x113/0x130
? kthread_create_worker_on_cpu+0x70/0x70
ret_from_fork+0x1f/0x40
Fix it by moving the cgroup's pidlists to a local orphan list under
pidlist_mutex, clearing their ->owner pointer, and then cancelling each
pidlist's delayed work outside the lock. The destroy work function now
checks ->owner and skips freeing orphaned pidlists, so
cgroup1_pidlist_destroy_all() can free them safely without flushing the
whole shared workqueue.
Signed-off-by: Junnan Zhang <zhangjn11@chinatelecom.cn>
Signed-off-by: Shouxin Sun <sunshx@chinatelecom.cn>
---
kernel/cgroup/cgroup-v1.c | 53 ++++++++++++++++++++++++++++-----------
1 file changed, 38 insertions(+), 15 deletions(-)
diff --git a/kernel/cgroup/cgroup-v1.c b/kernel/cgroup/cgroup-v1.c
index a4337c9b5287..874fe4dc6ffd 100644
--- a/kernel/cgroup/cgroup-v1.c
+++ b/kernel/cgroup/cgroup-v1.c
@@ -206,13 +206,31 @@ struct cgroup_pidlist {
void cgroup1_pidlist_destroy_all(struct cgroup *cgrp)
{
struct cgroup_pidlist *l, *tmp_l;
+ LIST_HEAD(orphan);
+
+ /*
+ * Move pidlists to a local orphan list and mark them as owner-less.
+ * The destroy work function will see ->owner == NULL and skip freeing.
+ * We then cancel and free them outside pidlist_mutex to avoid
+ * flush_workqueue() blocking on the shared workqueue.
+ */
mutex_lock(&cgrp->pidlist_mutex);
- list_for_each_entry_safe(l, tmp_l, &cgrp->pidlists, links)
- mod_delayed_work(cgroup_pidlist_destroy_wq, &l->destroy_dwork, 0);
+ list_for_each_entry_safe(l, tmp_l, &cgrp->pidlists, links) {
+ list_del(&l->links);
+ l->owner = NULL;
+ list_add(&l->links, &orphan);
+ }
mutex_unlock(&cgrp->pidlist_mutex);
- flush_workqueue(cgroup_pidlist_destroy_wq);
+ list_for_each_entry_safe(l, tmp_l, &orphan, links) {
+ list_del(&l->links);
+ cancel_delayed_work_sync(&l->destroy_dwork);
+ kvfree(l->list);
+ put_pid_ns(l->key.ns);
+ kfree(l);
+ }
+
BUG_ON(!list_empty(&cgrp->pidlists));
}
@@ -222,21 +240,26 @@ static void cgroup_pidlist_destroy_work_fn(struct work_struct *work)
struct cgroup_pidlist *l = container_of(dwork, struct cgroup_pidlist,
destroy_dwork);
struct cgroup_pidlist *tofree = NULL;
+ struct cgroup *owner;
- mutex_lock(&l->owner->pidlist_mutex);
+ owner = l->owner;
+ if (owner) {
+ mutex_lock(&owner->pidlist_mutex);
- /*
- * Destroy iff we didn't get queued again. The state won't change
- * as destroy_dwork can only be queued while locked.
- */
- if (!delayed_work_pending(dwork)) {
- list_del(&l->links);
- kvfree(l->list);
- put_pid_ns(l->key.ns);
- tofree = l;
- }
+ /*
+ * Destroy iff we didn't get queued again and we're still
+ * owned by the cgroup. If ->owner was cleared by
+ * cgroup1_pidlist_destroy_all(), it will free us.
+ */
+ if (l->owner == owner && !delayed_work_pending(dwork)) {
+ list_del(&l->links);
+ kvfree(l->list);
+ put_pid_ns(l->key.ns);
+ tofree = l;
+ }
- mutex_unlock(&l->owner->pidlist_mutex);
+ mutex_unlock(&l->owner->pidlist_mutex);
+ }
kfree(tofree);
}
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v2] cgroup: avoid flushing global workqueue in cgroup1_pidlist_destroy_all
2026-08-14 8:40 [PATCH] cgroup: avoid flushing global workqueue in cgroup1_pidlist_destroy_all Junnan Zhang
@ 2026-08-14 9:45 ` Junnan Zhang
2026-08-14 10:20 ` [PATCH v3] " Junnan Zhang
1 sibling, 0 replies; 6+ messages in thread
From: Junnan Zhang @ 2026-08-14 9:45 UTC (permalink / raw)
To: tj, hannes, mkoutny
Cc: cgroups, linux-kernel, zhangjn_dev, Junnan Zhang, Shouxin Sun
From: Junnan Zhang <zhangjn11@chinatelecom.cn>
cgroup1_pidlist_destroy_all() flushes the global
cgroup_pidlist_destroy_wq while destroying a cgroup. Because all cgroup
v1 pidlist destruction work items are queued on the same shared workqueue,
a single slow or stuck work item (e.g. waiting for pidlist_mutex held by a
user-space reader) blocks every concurrent cgroup destruction path.
This can lead to kworker tasks stuck in flush_workqueue() for over
hung_task_timeout seconds, as observed on busy systems running Docker or
Kubernetes workloads.
INFO: task kworker/0:1:1438499 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/0:1 D 0 1438499 2 0x80000080
Workqueue: cgroup_destroy css_free_rwork_fn
? __schedule+0x296/0x900
schedule+0x28/0x80
schedule_timeout+0x1ee/0x3a0
? kvm_sched_clock_read+0xd/0x20
wait_for_completion+0x12c/0x190
? wake_up_q+0x70/0x70
flush_workqueue+0x132/0x430
? cgroup1_pidlist_destroy_all+0x7c/0xa0
cgroup1_pidlist_destroy_all+0x7c/0xa0
css_free_rwork_fn+0xb5/0x390
process_one_work+0x195/0x3e0
worker_thread+0x30/0x390
? process_one_work+0x3e0/0x3e0
kthread+0x113/0x130
? kthread_create_worker_on_cpu+0x70/0x70
ret_from_fork+0x1f/0x40
Fix it by moving the cgroup's pidlists to a local orphan list under
pidlist_mutex, clearing their ->owner pointer, and then cancelling each
pidlist's delayed work outside the lock. The destroy work function now
checks ->owner and skips freeing orphaned pidlists, so
cgroup1_pidlist_destroy_all() can free them safely without flushing the
whole shared workqueue.
Signed-off-by: Junnan Zhang <zhangjn11@chinatelecom.cn>
Signed-off-by: Shouxin Sun <sunshx@chinatelecom.cn>
---
v1 -> v2:
- Fix a NULL pointer dereference in cgroup_pidlist_destroy_work_fn():
use the cached owner pointer for mutex_unlock(), as ->owner may have
been cleared concurrently by cgroup1_pidlist_destroy_all(). Spotted
by Sashiko AI review.
---
kernel/cgroup/cgroup-v1.c | 53 ++++++++++++++++++++++++++++-----------
1 file changed, 38 insertions(+), 15 deletions(-)
diff --git a/kernel/cgroup/cgroup-v1.c b/kernel/cgroup/cgroup-v1.c
index a4337c9b5287..88b10cd59c7b 100644
--- a/kernel/cgroup/cgroup-v1.c
+++ b/kernel/cgroup/cgroup-v1.c
@@ -206,13 +206,31 @@ struct cgroup_pidlist {
void cgroup1_pidlist_destroy_all(struct cgroup *cgrp)
{
struct cgroup_pidlist *l, *tmp_l;
+ LIST_HEAD(orphan);
+
+ /*
+ * Move pidlists to a local orphan list and mark them as owner-less.
+ * The destroy work function will see ->owner == NULL and skip freeing.
+ * We then cancel and free them outside pidlist_mutex to avoid
+ * flush_workqueue() blocking on the shared workqueue.
+ */
mutex_lock(&cgrp->pidlist_mutex);
- list_for_each_entry_safe(l, tmp_l, &cgrp->pidlists, links)
- mod_delayed_work(cgroup_pidlist_destroy_wq, &l->destroy_dwork, 0);
+ list_for_each_entry_safe(l, tmp_l, &cgrp->pidlists, links) {
+ list_del(&l->links);
+ l->owner = NULL;
+ list_add(&l->links, &orphan);
+ }
mutex_unlock(&cgrp->pidlist_mutex);
- flush_workqueue(cgroup_pidlist_destroy_wq);
+ list_for_each_entry_safe(l, tmp_l, &orphan, links) {
+ list_del(&l->links);
+ cancel_delayed_work_sync(&l->destroy_dwork);
+ kvfree(l->list);
+ put_pid_ns(l->key.ns);
+ kfree(l);
+ }
+
BUG_ON(!list_empty(&cgrp->pidlists));
}
@@ -222,21 +240,26 @@ static void cgroup_pidlist_destroy_work_fn(struct work_struct *work)
struct cgroup_pidlist *l = container_of(dwork, struct cgroup_pidlist,
destroy_dwork);
struct cgroup_pidlist *tofree = NULL;
+ struct cgroup *owner;
- mutex_lock(&l->owner->pidlist_mutex);
+ owner = l->owner;
+ if (owner) {
+ mutex_lock(&owner->pidlist_mutex);
- /*
- * Destroy iff we didn't get queued again. The state won't change
- * as destroy_dwork can only be queued while locked.
- */
- if (!delayed_work_pending(dwork)) {
- list_del(&l->links);
- kvfree(l->list);
- put_pid_ns(l->key.ns);
- tofree = l;
- }
+ /*
+ * Destroy iff we didn't get queued again and we're still
+ * owned by the cgroup. If ->owner was cleared by
+ * cgroup1_pidlist_destroy_all(), it will free us.
+ */
+ if (l->owner == owner && !delayed_work_pending(dwork)) {
+ list_del(&l->links);
+ kvfree(l->list);
+ put_pid_ns(l->key.ns);
+ tofree = l;
+ }
- mutex_unlock(&l->owner->pidlist_mutex);
+ mutex_unlock(&owner->pidlist_mutex);
+ }
kfree(tofree);
}
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* [PATCH v3] cgroup: avoid flushing global workqueue in cgroup1_pidlist_destroy_all
2026-08-14 8:40 [PATCH] cgroup: avoid flushing global workqueue in cgroup1_pidlist_destroy_all Junnan Zhang
2026-08-14 9:45 ` [PATCH v2] " Junnan Zhang
@ 2026-08-14 10:20 ` Junnan Zhang
2026-08-31 7:39 ` Junnan Zhang
2026-08-31 8:16 ` Michal Koutný
1 sibling, 2 replies; 6+ messages in thread
From: Junnan Zhang @ 2026-08-14 10:20 UTC (permalink / raw)
To: tj, hannes, mkoutny
Cc: cgroups, linux-kernel, zhangjn_dev, Junnan Zhang, Shouxin Sun
From: Junnan Zhang <zhangjn11@chinatelecom.cn>
cgroup1_pidlist_destroy_all() flushes the global
cgroup_pidlist_destroy_wq while destroying a cgroup. Because all cgroup
v1 pidlist destruction work items are queued on the same shared workqueue,
a single slow or stuck work item (e.g. waiting for pidlist_mutex held by a
user-space reader) blocks every concurrent cgroup destruction path.
This can lead to kworker tasks stuck in flush_workqueue() for over
hung_task_timeout seconds, as observed on busy systems running Docker or
Kubernetes workloads.
INFO: task kworker/0:1:1438499 blocked for more than 120 seconds.
"echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kworker/0:1 D 0 1438499 2 0x80000080
Workqueue: cgroup_destroy css_free_rwork_fn
? __schedule+0x296/0x900
schedule+0x28/0x80
schedule_timeout+0x1ee/0x3a0
? kvm_sched_clock_read+0xd/0x20
wait_for_completion+0x12c/0x190
? wake_up_q+0x70/0x70
flush_workqueue+0x132/0x430
? cgroup1_pidlist_destroy_all+0x7c/0xa0
cgroup1_pidlist_destroy_all+0x7c/0xa0
css_free_rwork_fn+0xb5/0x390
process_one_work+0x195/0x3e0
worker_thread+0x30/0x390
? process_one_work+0x3e0/0x3e0
kthread+0x113/0x130
? kthread_create_worker_on_cpu+0x70/0x70
ret_from_fork+0x1f/0x40
Fix it by moving the cgroup's pidlists to a local orphan list under
pidlist_mutex, clearing their ->owner pointer, and then cancelling each
pidlist's delayed work outside the lock. The destroy work function now
checks ->owner and skips freeing orphaned pidlists, so
cgroup1_pidlist_destroy_all() can free them safely without flushing the
whole shared workqueue.
Signed-off-by: Junnan Zhang <zhangjn11@chinatelecom.cn>
Signed-off-by: Shouxin Sun <sunshx@chinatelecom.cn>
---
v1 -> v2:
- Fix a NULL pointer dereference in cgroup_pidlist_destroy_work_fn():
use the cached owner pointer for mutex_unlock(), as ->owner may have
been cleared concurrently by cgroup1_pidlist_destroy_all(). Spotted
by Sashiko AI review.
v2 -> v3:
- Use READ_ONCE()/WRITE_ONCE() for the ->owner field shared between
cgroup1_pidlist_destroy_all() and cgroup_pidlist_destroy_work_fn() to
prevent data races. Spotted by Sashiko AI review.
---
kernel/cgroup/cgroup-v1.c | 53 ++++++++++++++++++++++++++++-----------
1 file changed, 38 insertions(+), 15 deletions(-)
diff --git a/kernel/cgroup/cgroup-v1.c b/kernel/cgroup/cgroup-v1.c
index a4337c9b5287..9abe8b791bb1 100644
--- a/kernel/cgroup/cgroup-v1.c
+++ b/kernel/cgroup/cgroup-v1.c
@@ -206,13 +206,31 @@ struct cgroup_pidlist {
void cgroup1_pidlist_destroy_all(struct cgroup *cgrp)
{
struct cgroup_pidlist *l, *tmp_l;
+ LIST_HEAD(orphan);
+
+ /*
+ * Move pidlists to a local orphan list and mark them as owner-less.
+ * The destroy work function will see ->owner == NULL and skip freeing.
+ * We then cancel and free them outside pidlist_mutex to avoid
+ * flush_workqueue() blocking on the shared workqueue.
+ */
mutex_lock(&cgrp->pidlist_mutex);
- list_for_each_entry_safe(l, tmp_l, &cgrp->pidlists, links)
- mod_delayed_work(cgroup_pidlist_destroy_wq, &l->destroy_dwork, 0);
+ list_for_each_entry_safe(l, tmp_l, &cgrp->pidlists, links) {
+ list_del(&l->links);
+ WRITE_ONCE(l->owner, NULL);
+ list_add(&l->links, &orphan);
+ }
mutex_unlock(&cgrp->pidlist_mutex);
- flush_workqueue(cgroup_pidlist_destroy_wq);
+ list_for_each_entry_safe(l, tmp_l, &orphan, links) {
+ list_del(&l->links);
+ cancel_delayed_work_sync(&l->destroy_dwork);
+ kvfree(l->list);
+ put_pid_ns(l->key.ns);
+ kfree(l);
+ }
+
BUG_ON(!list_empty(&cgrp->pidlists));
}
@@ -222,21 +240,26 @@ static void cgroup_pidlist_destroy_work_fn(struct work_struct *work)
struct cgroup_pidlist *l = container_of(dwork, struct cgroup_pidlist,
destroy_dwork);
struct cgroup_pidlist *tofree = NULL;
+ struct cgroup *owner;
- mutex_lock(&l->owner->pidlist_mutex);
+ owner = READ_ONCE(l->owner);
+ if (owner) {
+ mutex_lock(&owner->pidlist_mutex);
- /*
- * Destroy iff we didn't get queued again. The state won't change
- * as destroy_dwork can only be queued while locked.
- */
- if (!delayed_work_pending(dwork)) {
- list_del(&l->links);
- kvfree(l->list);
- put_pid_ns(l->key.ns);
- tofree = l;
- }
+ /*
+ * Destroy iff we didn't get queued again and we're still
+ * owned by the cgroup. If ->owner was cleared by
+ * cgroup1_pidlist_destroy_all(), it will free us.
+ */
+ if (READ_ONCE(l->owner) == owner && !delayed_work_pending(dwork)) {
+ list_del(&l->links);
+ kvfree(l->list);
+ put_pid_ns(l->key.ns);
+ tofree = l;
+ }
- mutex_unlock(&l->owner->pidlist_mutex);
+ mutex_unlock(&owner->pidlist_mutex);
+ }
kfree(tofree);
}
--
2.43.0
^ permalink raw reply related [flat|nested] 6+ messages in thread* Re: [PATCH v3] cgroup: avoid flushing global workqueue in cgroup1_pidlist_destroy_all
2026-08-14 10:20 ` [PATCH v3] " Junnan Zhang
@ 2026-08-31 7:39 ` Junnan Zhang
2026-08-31 8:16 ` Michal Koutný
1 sibling, 0 replies; 6+ messages in thread
From: Junnan Zhang @ 2026-08-31 7:39 UTC (permalink / raw)
To: zhangjn_dev; +Cc: cgroups, hannes, linux-kernel, mkoutny, sunshx, tj, zhangjn11
Gentle ping.
It's been a couple of weeks since v3, and there hasn't been any
maintainer feedback yet. Could you take a look when you have a
chance? Happy to address any comments or rebase if needed.
Thanks,
Junnan
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3] cgroup: avoid flushing global workqueue in cgroup1_pidlist_destroy_all
2026-08-14 10:20 ` [PATCH v3] " Junnan Zhang
2026-08-31 7:39 ` Junnan Zhang
@ 2026-08-31 8:16 ` Michal Koutný
2026-09-01 1:53 ` Ridong Chen
1 sibling, 1 reply; 6+ messages in thread
From: Michal Koutný @ 2026-08-31 8:16 UTC (permalink / raw)
To: Junnan Zhang; +Cc: tj, hannes, cgroups, linux-kernel, Junnan Zhang, Shouxin Sun
[-- Attachment #1: Type: text/plain, Size: 3099 bytes --]
Hi Junnan.
(Sorry for late response, I sketched some notes and then didn't get down
to sent them. Now they're below.)
On Fri, Aug 14, 2026 at 06:20:52PM +0800, Junnan Zhang <zhangjn_dev@163.com> wrote:
> From: Junnan Zhang <zhangjn11@chinatelecom.cn>
>
> cgroup1_pidlist_destroy_all() flushes the global
> cgroup_pidlist_destroy_wq while destroying a cgroup. Because all cgroup
> v1 pidlist destruction work items are queued on the same shared workqueue,
> a single slow or stuck work item (e.g. waiting for pidlist_mutex held by a
> user-space reader) blocks every concurrent cgroup destruction path.
>
> This can lead to kworker tasks stuck in flush_workqueue() for over
> hung_task_timeout seconds, as observed on busy systems running Docker or
> Kubernetes workloads.
Since the cgroup_pidlist_destroy_wq is already a dedicated workqueue (no
other conteders), the pursuit of pidlist_mutex holder is a feasible
theory. However, that would also mean:
a) a single reader taking more than hung_task_timeout_secs (that'd be
a softlockup earlier),
b) starvation of cgroup1_pidlist_destroy_all() by many (queued)
cgroup_pidlist_start() callers which goes against the second-long
caching of pidlists,
c) there is large number of nr_cgroups * nr_pidnses which makes the
caching ineffective
>
> INFO: task kworker/0:1:1438499 blocked for more than 120 seconds.
> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
> kworker/0:1 D 0 1438499 2 0x80000080
> Workqueue: cgroup_destroy css_free_rwork_fn
> ? __schedule+0x296/0x900
> schedule+0x28/0x80
> schedule_timeout+0x1ee/0x3a0
> ? kvm_sched_clock_read+0xd/0x20
> wait_for_completion+0x12c/0x190
> ? wake_up_q+0x70/0x70
> flush_workqueue+0x132/0x430
> ? cgroup1_pidlist_destroy_all+0x7c/0xa0
> cgroup1_pidlist_destroy_all+0x7c/0xa0
> css_free_rwork_fn+0xb5/0x390
> process_one_work+0x195/0x3e0
> worker_thread+0x30/0x390
> ? process_one_work+0x3e0/0x3e0
> kthread+0x113/0x130
> ? kthread_create_worker_on_cpu+0x70/0x70
> ret_from_fork+0x1f/0x40
>
> Fix it by moving the cgroup's pidlists to a local orphan list under
> pidlist_mutex, clearing their ->owner pointer, and then cancelling each
> pidlist's delayed work outside the lock. The destroy work function now
> checks ->owner and skips freeing orphaned pidlists, so
> cgroup1_pidlist_destroy_all() can free them safely without flushing the
> whole shared workqueue.
What's the point of the workqueue after this change? (Mainly the
expiration + having process context for the handler.)
The flushing isn't necessary if there's a way how to ensure pidlists
head won't be used after cgrp removal, which the fix should achieve.
So I'd say, the narrow-focused cancellation may work, no need to wait
for other cgroups. OTOH, I'm surprised this v1-issue popped up only now
and whether such a long contention can happen over pidlist_mutex as your
commit message implies. What nr_cgroups, nr_pidnses could cause this in
your theory?
Thanks,
Michal
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 265 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [PATCH v3] cgroup: avoid flushing global workqueue in cgroup1_pidlist_destroy_all
2026-08-31 8:16 ` Michal Koutný
@ 2026-09-01 1:53 ` Ridong Chen
0 siblings, 0 replies; 6+ messages in thread
From: Ridong Chen @ 2026-09-01 1:53 UTC (permalink / raw)
To: Michal Koutný, Junnan Zhang
Cc: tj, hannes, cgroups, linux-kernel, Junnan Zhang, Shouxin Sun
On 8/31/2026 4:16 PM, Michal Koutný wrote:
> Hi Junnan.
>
> (Sorry for late response, I sketched some notes and then didn't get down
> to sent them. Now they're below.)
>
> On Fri, Aug 14, 2026 at 06:20:52PM +0800, Junnan Zhang <zhangjn_dev@163.com> wrote:
>> From: Junnan Zhang <zhangjn11@chinatelecom.cn>
>>
>> cgroup1_pidlist_destroy_all() flushes the global
>> cgroup_pidlist_destroy_wq while destroying a cgroup. Because all cgroup
>> v1 pidlist destruction work items are queued on the same shared workqueue,
>> a single slow or stuck work item (e.g. waiting for pidlist_mutex held by a
>> user-space reader) blocks every concurrent cgroup destruction path.
>>
>> This can lead to kworker tasks stuck in flush_workqueue() for over
>> hung_task_timeout seconds, as observed on busy systems running Docker or
>> Kubernetes workloads.
>
> Since the cgroup_pidlist_destroy_wq is already a dedicated workqueue (no
> other conteders), the pursuit of pidlist_mutex holder is a feasible
> theory. However, that would also mean:
> a) a single reader taking more than hung_task_timeout_secs (that'd be
> a softlockup earlier),
> b) starvation of cgroup1_pidlist_destroy_all() by many (queued)
> cgroup_pidlist_start() callers which goes against the second-long
> caching of pidlists,
> c) there is large number of nr_cgroups * nr_pidnses which makes the
> caching ineffective
>
>>
>> INFO: task kworker/0:1:1438499 blocked for more than 120 seconds.
>> "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
>> kworker/0:1 D 0 1438499 2 0x80000080
>> Workqueue: cgroup_destroy css_free_rwork_fn
>> ? __schedule+0x296/0x900
>> schedule+0x28/0x80
>> schedule_timeout+0x1ee/0x3a0
>> ? kvm_sched_clock_read+0xd/0x20
>> wait_for_completion+0x12c/0x190
>> ? wake_up_q+0x70/0x70
>> flush_workqueue+0x132/0x430
>> ? cgroup1_pidlist_destroy_all+0x7c/0xa0
>> cgroup1_pidlist_destroy_all+0x7c/0xa0
>> css_free_rwork_fn+0xb5/0x390
>> process_one_work+0x195/0x3e0
>> worker_thread+0x30/0x390
>> ? process_one_work+0x3e0/0x3e0
>> kthread+0x113/0x130
>> ? kthread_create_worker_on_cpu+0x70/0x70
>> ret_from_fork+0x1f/0x40
>>
I'm not sure we've found the real root cause yet, and the current analysis
doesn't convince me.
Shouldn't we first figure out why flush_workqueue() waited 120s? Was it because
there were too many pids, or because someone held pidlist_mutex for too long?
>> Fix it by moving the cgroup's pidlists to a local orphan list under
>> pidlist_mutex, clearing their ->owner pointer, and then cancelling each
>> pidlist's delayed work outside the lock. The destroy work function now
>> checks ->owner and skips freeing orphaned pidlists, so
>> cgroup1_pidlist_destroy_all() can free them safely without flushing the
>> whole shared workqueue.
>
> What's the point of the workqueue after this change? (Mainly the
> expiration + having process context for the handler.)
>
> The flushing isn't necessary if there's a way how to ensure pidlists
> head won't be used after cgrp removal, which the fix should achieve.
>
>
> So I'd say, the narrow-focused cancellation may work, no need to wait
> for other cgroups. OTOH, I'm surprised this v1-issue popped up only now
> and whether such a long contention can happen over pidlist_mutex as your
> commit message implies. What nr_cgroups, nr_pidnses could cause this in
> your theory?
>
> Thanks,
> Michal
--
Best regards
Ridong
^ permalink raw reply [flat|nested] 6+ messages in thread
end of thread, other threads:[~2026-09-01 1:53 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-14 8:40 [PATCH] cgroup: avoid flushing global workqueue in cgroup1_pidlist_destroy_all Junnan Zhang
2026-08-14 9:45 ` [PATCH v2] " Junnan Zhang
2026-08-14 10:20 ` [PATCH v3] " Junnan Zhang
2026-08-31 7:39 ` Junnan Zhang
2026-08-31 8:16 ` Michal Koutný
2026-09-01 1:53 ` Ridong Chen
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox