* [PATCH v4 0/2] cgroup/cpuset: Keep the top scheduling domain populated @ 2026-11-11 13:42 Guopeng Zhang 2026-11-11 13:42 ` [PATCH v4 1/2] cgroup/cpuset: Keep an active housekeeping CPU in the top sched domain Guopeng Zhang 2026-11-11 13:42 ` [PATCH v4 2/2] selftests/cgroup: Add tests for top sched domain housekeeping checks Guopeng Zhang 0 siblings, 2 replies; 5+ messages in thread From: Guopeng Zhang @ 2026-11-11 13:42 UTC (permalink / raw) To: longman, ridong.chen Cc: cgroups, tj, hannes, mkoutny, shuah, mathieu.desnoyers, linux-kernel, linux-kselftest, Guopeng Zhang From: Guopeng Zhang <zhangguopeng@kylinos.cn> Boot-isolated CPUs stay in top_cpuset.effective_cpus, but are excluded from its scheduling domain. Allocating all boot domain housekeeping CPUs to partitions can therefore leave the top scheduling domain empty, leading to warnings and an OOPS in build_perf_domains(). CPU hotplug can reach the same state. Following Ridong's review, use one tasks_nocpu_error() check for all parents, with an explicit mask of CPUs returned by the update. This also allows CPU replacements under populated local parents. Hotplug passes no returned CPUs and preserves partitions unable to restore an active boot domain housekeeping CPU. Patch 1 contains the complete kernel fix, including all caller updates. Patch 2 contains the selftests. Changes since v3: Unify the CPU availability checks following Ridong's suggestion. Restrict returned CPUs to the parent's current exclusive grant, and retain the active housekeeping CPU guard during hotplug. Collect the kernel changes in patch 1 and selftests in patch 2, adding a regression for CPU replacement under a populated local parent. Update Ridong's email address as requested. v3: https://lore.kernel.org/all/20261110050324.173461-1-guopeng.zhang@linux.dev/ Changes since v2: Keep patch 1's kernel changes unchanged. Update its selftests following Longman's and Sashiko's comments and cover recovery with a single top-level HK CPU. Add patch 2 to fix the CPU replacement rejection reported by Sashiko, with its own regression test. Update the hotplug fix, now patch 3, to avoid invalidating unrelated partitions, including partitions whose boot housekeeping CPUs are all offline. Drop v2's isolated CPU cleanup patch in favor of Ridong's fix: https://lore.kernel.org/all/20261107120100.3587595-4-ridong.chen@linux.dev/ v2: https://lore.kernel.org/all/20261105130344.664336-1-guopeng.zhang@linux.dev/ Changes since v1: Rework the fix following Ridong's suggestion and add test cases. Add patch 2 for CPU hotplug and patch 3 for isolated CPU cleanup when the parent partition becomes invalid. v1: https://lore.kernel.org/all/20261104071600.474587-1-guopeng.zhang@linux.dev/ Guopeng Zhang (2): cgroup/cpuset: Keep an active housekeeping CPU in the top sched domain selftests/cgroup: Add tests for top sched domain housekeeping checks kernel/cgroup/cpuset.c | 85 +++++++++++++++---- .../selftests/cgroup/test_cpuset_prs.sh | 71 ++++++++++++++++ 2 files changed, 141 insertions(+), 15 deletions(-) base-commit: 630059a5a632377b9b7b487baccde92b7ae1dd47 -- 2.43.0 ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v4 1/2] cgroup/cpuset: Keep an active housekeeping CPU in the top sched domain 2026-11-11 13:42 [PATCH v4 0/2] cgroup/cpuset: Keep the top scheduling domain populated Guopeng Zhang @ 2026-11-11 13:42 ` Guopeng Zhang 2026-11-12 3:05 ` Ridong Chen 2026-11-11 13:42 ` [PATCH v4 2/2] selftests/cgroup: Add tests for top sched domain housekeeping checks Guopeng Zhang 1 sibling, 1 reply; 5+ messages in thread From: Guopeng Zhang @ 2026-11-11 13:42 UTC (permalink / raw) To: longman, ridong.chen Cc: cgroups, tj, hannes, mkoutny, shuah, mathieu.desnoyers, linux-kernel, linux-kselftest, Guopeng Zhang, Ridong Chen From: Guopeng Zhang <zhangguopeng@kylinos.cn> Boot-isolated CPUs remain in top_cpuset.effective_cpus, but are excluded from its scheduling domain. A partition can therefore take all boot domain housekeeping CPUs while leaving effective_cpus nonempty. For example, on a 32-CPU system with all CPUs online and isolcpus=domain,10-31: echo 0-9 > A/cpuset.cpus echo root > A/cpuset.cpus.partition Oops: general protection fault RIP: 0010:build_perf_domains+... tasks_nocpu_error() also rejects valid CPU replacements because it does not account for CPUs returned by the update. With a fresh cgroup A under the same boot configuration: echo 2-31 > A/cpuset.cpus echo isolated > A/cpuset.cpus.partition echo 0-4 > A/cpuset.cpus cat A/cpuset.cpus.partition isolated invalid (Parent unable to distribute cpu downstream) The update should return CPUs 5-31 to top_cpuset, including housekeeping CPUs 5-9, and leave A valid. This replacement failure also occurs without boot isolation and under populated local parents. Extend tasks_nocpu_error() to take an explicit mask of CPUs being returned and use it for local and remote updates. Require an active HK_TYPE_DOMAIN_BOOT CPU to remain in top_cpuset. For other populated parents, check within their current exclusive grant so an ancestor update cannot credit CPUs that the parent no longer owns. CPU hotplug can also empty the top scheduling domain. Pass no returned CPUs during its availability check, and only invalidate a partition to restore the top domain if it can return an active boot domain housekeeping CPU. Still invalidate a populated partition that has no active CPU of its own. Fixes: 9b496a8bbed9 ("cgroup/cpuset: Prevent leakage of isolated CPUs into sched domains") Fixes: 0c7f293efc87 ("cgroup/cpuset: Add cpuset.cpus.exclusive.effective for v2") Co-developed-by: Ridong Chen <chenridong@xiaomi.com> Signed-off-by: Ridong Chen <chenridong@xiaomi.com> Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn> --- kernel/cgroup/cpuset.c | 85 ++++++++++++++++++++++++++++++++++-------- 1 file changed, 70 insertions(+), 15 deletions(-) diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c index 3052413e41e7..85f92ae0c6f4 100644 --- a/kernel/cgroup/cpuset.c +++ b/kernel/cgroup/cpuset.c @@ -1181,18 +1181,54 @@ static void update_partition_sd_lb(struct cpuset *cs, int old_prs) } /* - * tasks_nocpu_error - Return true if tasks will have no effective_cpus + * tasks_nocpu_error - Return true if tasks or the top domain lack active CPUs + * @parent: parent partition, NULL for a remote partition + * @cs: the partition + * @xcpus: exclusive CPUs that @cs will hold after the update + * @del_cpus: exclusive CPUs to be returned to @parent, can be NULL + * + * @del_cpus may include CPUs that are also in @xcpus and will stay in + * @cs. They are skipped by the @xcpus check, so callers can pass the + * current effective_xcpus of @cs without masking it first. + * + * top_cpuset always has tasks. Boot-isolated CPUs remain in its + * effective_cpus but are excluded from its sched domain, which must + * retain an active boot domain housekeeping CPU. */ static bool tasks_nocpu_error(struct cpuset *parent, struct cpuset *cs, - struct cpumask *xcpus) + const struct cpumask *xcpus, + const struct cpumask *del_cpus) { + const struct cpumask *mask; + int cpu; + + /* A populated @cs must have an active CPU in @xcpus. */ + if (!cpumask_intersects(xcpus, cpu_active_mask) && + partition_is_populated(cs, NULL)) + return true; + /* - * A populated partition (cs or parent) can't have empty effective_cpus + * A populated @parent must keep an active CPU in @mask, either from + * its effective_cpus or from @del_cpus, that isn't taken by @cs. */ - return (cpumask_subset(parent->effective_cpus, xcpus) && - partition_is_populated(parent, cs)) || - (!cpumask_intersects(xcpus, cpu_active_mask) && - partition_is_populated(cs, NULL)); + if (!parent) + parent = &top_cpuset; + + /* CPUs outside the parent's current grant cannot be returned. */ + mask = parent->effective_xcpus; + if (parent == &top_cpuset) + mask = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT); + else if (!partition_is_populated(parent, cs)) + return false; + + for_each_cpu_and(cpu, mask, cpu_active_mask) { + if (cpumask_test_cpu(cpu, xcpus)) + continue; + if (cpumask_test_cpu(cpu, parent->effective_cpus) || + (del_cpus && cpumask_test_cpu(cpu, del_cpus))) + return false; + } + return true; } static void reset_partition_data(struct cpuset *cs) @@ -1724,7 +1760,7 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs, */ compute_excpus(cs, tmp->new_cpus); if (!cpumask_intersects(tmp->new_cpus, cpu_active_mask) || - cpumask_subset(top_cpuset.effective_cpus, tmp->new_cpus)) + tasks_nocpu_error(NULL, cs, tmp->new_cpus, NULL)) return PERR_INVCPUS; if (cpumask_intersects(tmp->new_cpus, subpartitions_cpus)) return PERR_NOCPUS; @@ -1828,7 +1864,7 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus, if (!capable(CAP_SYS_ADMIN)) WRITE_ONCE(cs->prs_err, PERR_ACCESS); else if (cpumask_intersects(tmp->addmask, subpartitions_cpus) || - cpumask_subset(top_cpuset.effective_cpus, tmp->addmask)) + tasks_nocpu_error(NULL, cs, excpus, cs->effective_xcpus)) WRITE_ONCE(cs->prs_err, PERR_NOCPUS); else if (prstate_housekeeping_conflict(prs, PRS_ROOT, excpus, tmp->delmask)) @@ -1961,7 +1997,8 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd, */ if ((cmd != partcmd_disable) && !newmask) { compute_excpus(cs, tmp->delmask); - nocpu = tasks_nocpu_error(parent, cs, tmp->delmask); + nocpu = tasks_nocpu_error(parent, cs, tmp->delmask, + is_partition_valid(cs) ? cs->effective_xcpus : NULL); } if ((cmd == partcmd_enable) || (cmd == partcmd_enablei)) { @@ -2497,7 +2534,9 @@ static enum prs_errcode validate_partition(struct cpuset *cs, struct cpuset *tri if (cpumask_empty(trialcs->effective_xcpus)) return PERR_INVCPUS; - if (tasks_nocpu_error(parent, cs, trialcs->effective_xcpus)) + if (tasks_nocpu_error(is_remote_partition(cs) ? NULL : parent, cs, + trialcs->effective_xcpus, + is_partition_valid(cs) ? cs->effective_xcpus : NULL)) return PERR_NOCPUS; return PERR_NONE; @@ -3997,6 +4036,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) static nodemask_t new_mems; bool cpus_updated; bool mems_updated; + bool nocpu; int partcmd = -1; struct cpuset *parent; @@ -4008,9 +4048,23 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) if (!tmp || !cs->partition_root_state) goto update_tasks; - if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) || - (cpumask_empty(&new_cpus) && - partition_is_populated(cs, NULL)))) { + /* Do not count any CPUs as returned during this hotplug check. */ + nocpu = tasks_nocpu_error(is_remote_partition(cs) ? NULL : parent, cs, + &new_cpus, NULL); + if (nocpu && (parent == &top_cpuset || is_remote_partition(cs))) { + /* + * Only invalidate a partition for an empty top sched domain if + * it can return an active boot domain housekeeping CPU. A + * populated partition with no active CPU must still be invalidated. + */ + cpumask_and(tmp->addmask, cs->effective_xcpus, + housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT)); + if (!cpumask_intersects(tmp->addmask, cpu_active_mask)) + nocpu = cpumask_empty(&new_cpus) && + partition_is_populated(cs, NULL); + } + + if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) || nocpu)) { WRITE_ONCE(cs->prs_err, PERR_HOTPLUG); remote_partition_disable(cs, tmp); compute_effective_cpumask(&new_cpus, cs, parent); @@ -4023,10 +4077,11 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) * 2) parent is invalid or doesn't grant any cpus to child * partitions. * 3) subpartitions_cpus is empty. + * 4) top_cpuset needs an active boot domain housekeeping CPU from cs. */ if (is_local_partition(cs) && (!is_partition_valid(parent) || - tasks_nocpu_error(parent, cs, &new_cpus) || + nocpu || cpumask_empty(subpartitions_cpus))) partcmd = partcmd_invalidate; /* -- 2.43.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
* Re: [PATCH v4 1/2] cgroup/cpuset: Keep an active housekeeping CPU in the top sched domain 2026-11-11 13:42 ` [PATCH v4 1/2] cgroup/cpuset: Keep an active housekeeping CPU in the top sched domain Guopeng Zhang @ 2026-11-12 3:05 ` Ridong Chen 2026-11-12 9:43 ` Guopeng Zhang 0 siblings, 1 reply; 5+ messages in thread From: Ridong Chen @ 2026-11-12 3:05 UTC (permalink / raw) To: Guopeng Zhang, longman Cc: cgroups, tj, hannes, mkoutny, shuah, mathieu.desnoyers, linux-kernel, linux-kselftest, Guopeng Zhang, Ridong Chen On 11/11/2026 9:42 PM, Guopeng Zhang wrote: > From: Guopeng Zhang <zhangguopeng@kylinos.cn> > > Boot-isolated CPUs remain in top_cpuset.effective_cpus, but are excluded > from its scheduling domain. A partition can therefore take all boot > domain housekeeping CPUs while leaving effective_cpus nonempty. For > example, on a 32-CPU system with all CPUs online and > isolcpus=domain,10-31: > > echo 0-9 > A/cpuset.cpus > echo root > A/cpuset.cpus.partition > Oops: general protection fault > RIP: 0010:build_perf_domains+... > > tasks_nocpu_error() also rejects valid CPU replacements because it does > not account for CPUs returned by the update. With a fresh cgroup A under > the same boot configuration: > > echo 2-31 > A/cpuset.cpus > echo isolated > A/cpuset.cpus.partition > echo 0-4 > A/cpuset.cpus > cat A/cpuset.cpus.partition > isolated invalid (Parent unable to distribute cpu downstream) > > The update should return CPUs 5-31 to top_cpuset, including housekeeping > CPUs 5-9, and leave A valid. This replacement failure also occurs without > boot isolation and under populated local parents. > > Extend tasks_nocpu_error() to take an explicit mask of CPUs being > returned and use it for local and remote updates. Require an active > HK_TYPE_DOMAIN_BOOT CPU to remain in top_cpuset. For other populated > parents, check within their current exclusive grant so an ancestor > update cannot credit CPUs that the parent no longer owns. > > CPU hotplug can also empty the top scheduling domain. Pass no returned > CPUs during its availability check, and only invalidate a partition to > restore the top domain if it can return an active boot domain > housekeeping CPU. Still invalidate a populated partition that has no > active CPU of its own. > > Fixes: 9b496a8bbed9 ("cgroup/cpuset: Prevent leakage of isolated CPUs into sched domains") > Fixes: 0c7f293efc87 ("cgroup/cpuset: Add cpuset.cpus.exclusive.effective for v2") > Co-developed-by: Ridong Chen <chenridong@xiaomi.com> > Signed-off-by: Ridong Chen <chenridong@xiaomi.com> > Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn> > --- > kernel/cgroup/cpuset.c | 85 ++++++++++++++++++++++++++++++++++-------- > 1 file changed, 70 insertions(+), 15 deletions(-) > > diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c > index 3052413e41e7..85f92ae0c6f4 100644 > --- a/kernel/cgroup/cpuset.c > +++ b/kernel/cgroup/cpuset.c > @@ -1181,18 +1181,54 @@ static void update_partition_sd_lb(struct cpuset *cs, int old_prs) > } > > /* > - * tasks_nocpu_error - Return true if tasks will have no effective_cpus > + * tasks_nocpu_error - Return true if tasks or the top domain lack active CPUs > + * @parent: parent partition, NULL for a remote partition > + * @cs: the partition > + * @xcpus: exclusive CPUs that @cs will hold after the update > + * @del_cpus: exclusive CPUs to be returned to @parent, can be NULL > + * > + * @del_cpus may include CPUs that are also in @xcpus and will stay in > + * @cs. They are skipped by the @xcpus check, so callers can pass the > + * current effective_xcpus of @cs without masking it first. > + * > + * top_cpuset always has tasks. Boot-isolated CPUs remain in its > + * effective_cpus but are excluded from its sched domain, which must > + * retain an active boot domain housekeeping CPU. > */ > static bool tasks_nocpu_error(struct cpuset *parent, struct cpuset *cs, > - struct cpumask *xcpus) > + const struct cpumask *xcpus, > + const struct cpumask *del_cpus) > { > + const struct cpumask *mask; > + int cpu; > + > + /* A populated @cs must have an active CPU in @xcpus. */ > + if (!cpumask_intersects(xcpus, cpu_active_mask) && > + partition_is_populated(cs, NULL)) > + return true; > + > /* > - * A populated partition (cs or parent) can't have empty effective_cpus > + * A populated @parent must keep an active CPU in @mask, either from > + * its effective_cpus or from @del_cpus, that isn't taken by @cs. > */ > - return (cpumask_subset(parent->effective_cpus, xcpus) && > - partition_is_populated(parent, cs)) || > - (!cpumask_intersects(xcpus, cpu_active_mask) && > - partition_is_populated(cs, NULL)); > + if (!parent) > + parent = &top_cpuset; > + > + /* CPUs outside the parent's current grant cannot be returned. */ > + mask = parent->effective_xcpus; > + if (parent == &top_cpuset) > + mask = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT); > + else if (!partition_is_populated(parent, cs)) > + return false; > + The mask selection looks right to me. HK_TYPE_DOMAIN_BOOT is the right one for top_cpuset, since CPUs returned by an isolated child are not in HK_TYPE_DOMAIN yet. Limiting a local parent to its effective_xcpus also correctly drops stale CPUs in del_cpus after the parent shrinks. One corner case: validate_partition() also calls this for an invalid cs whose parent is not a valid partition. If the parent is an invalid partition without cpuset.cpus.exclusive, reset_partition_data() has cleared its effective_xcpus, the loop does nothing, and we return true when the parent is populated. The child's prs_err then changes from PERR_INVPARENT to PERR_NOCPUS, which reports the wrong reason. The old code did not do that. How about: else if (!is_partition_valid(parent) || !partition_is_populated(parent, cs)) return false; The invalid parent case is already reported by PERR_INVPARENT/PERR_NOTPART. Nit: the comment "Return true if tasks or the top domain lack active CPUs" doesn't cover local parents. Maybe "Return true if @cs or @parent would lack active CPUs; for top_cpuset, only active boot domain housekeeping CPUs count". > + for_each_cpu_and(cpu, mask, cpu_active_mask) { > + if (cpumask_test_cpu(cpu, xcpus)) > + continue; > + if (cpumask_test_cpu(cpu, parent->effective_cpus) || > + (del_cpus && cpumask_test_cpu(cpu, del_cpus))) > + return false; > + } > + return true; > } > > static void reset_partition_data(struct cpuset *cs) > @@ -1724,7 +1760,7 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs, > */ > compute_excpus(cs, tmp->new_cpus); > if (!cpumask_intersects(tmp->new_cpus, cpu_active_mask) || > - cpumask_subset(top_cpuset.effective_cpus, tmp->new_cpus)) > + tasks_nocpu_error(NULL, cs, tmp->new_cpus, NULL)) > return PERR_INVCPUS; > if (cpumask_intersects(tmp->new_cpus, subpartitions_cpus)) > return PERR_NOCPUS; > @@ -1828,7 +1864,7 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus, > if (!capable(CAP_SYS_ADMIN)) > WRITE_ONCE(cs->prs_err, PERR_ACCESS); > else if (cpumask_intersects(tmp->addmask, subpartitions_cpus) || > - cpumask_subset(top_cpuset.effective_cpus, tmp->addmask)) > + tasks_nocpu_error(NULL, cs, excpus, cs->effective_xcpus)) > WRITE_ONCE(cs->prs_err, PERR_NOCPUS); > else if (prstate_housekeeping_conflict(prs, PRS_ROOT, > excpus, tmp->delmask)) > @@ -1961,7 +1997,8 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd, > */ > if ((cmd != partcmd_disable) && !newmask) { > compute_excpus(cs, tmp->delmask); > - nocpu = tasks_nocpu_error(parent, cs, tmp->delmask); > + nocpu = tasks_nocpu_error(parent, cs, tmp->delmask, > + is_partition_valid(cs) ? cs->effective_xcpus : NULL); > } > > if ((cmd == partcmd_enable) || (cmd == partcmd_enablei)) { > @@ -2497,7 +2534,9 @@ static enum prs_errcode validate_partition(struct cpuset *cs, struct cpuset *tri > if (cpumask_empty(trialcs->effective_xcpus)) > return PERR_INVCPUS; > > - if (tasks_nocpu_error(parent, cs, trialcs->effective_xcpus)) > + if (tasks_nocpu_error(is_remote_partition(cs) ? NULL : parent, cs, > + trialcs->effective_xcpus, > + is_partition_valid(cs) ? cs->effective_xcpus : NULL)) > return PERR_NOCPUS; > > return PERR_NONE; > @@ -3997,6 +4036,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) > static nodemask_t new_mems; > bool cpus_updated; > bool mems_updated; > + bool nocpu; > int partcmd = -1; > struct cpuset *parent; > > @@ -4008,9 +4048,23 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) > if (!tmp || !cs->partition_root_state) > goto update_tasks; > > - if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) || > - (cpumask_empty(&new_cpus) && > - partition_is_populated(cs, NULL)))) { > + /* Do not count any CPUs as returned during this hotplug check. */ > + nocpu = tasks_nocpu_error(is_remote_partition(cs) ? NULL : parent, cs, > + &new_cpus, NULL); > + if (nocpu && (parent == &top_cpuset || is_remote_partition(cs))) { > + /* > + * Only invalidate a partition for an empty top sched domain if > + * it can return an active boot domain housekeeping CPU. A > + * populated partition with no active CPU must still be invalidated. > + */ > + cpumask_and(tmp->addmask, cs->effective_xcpus, > + housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT)); > + if (!cpumask_intersects(tmp->addmask, cpu_active_mask)) > + nocpu = cpumask_empty(&new_cpus) && > + partition_is_populated(cs, NULL); > + } > + This is hard to follow. nocpu is computed first and then partly undone, so that only a partition owning an active housekeeping CPU gets invalidated. Do we really need this? When hotplug leaves top_cpuset with no active CPU, cpuset_handle_hotplug() already clears subpartitions_cpus and invalidates all the partitions under it. If we handle an empty top sched domain the same way, this block can go away. Longman, what do you think? > + if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) || nocpu)) { > WRITE_ONCE(cs->prs_err, PERR_HOTPLUG); > remote_partition_disable(cs, tmp); > compute_effective_cpumask(&new_cpus, cs, parent); > @@ -4023,10 +4077,11 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) > * 2) parent is invalid or doesn't grant any cpus to child > * partitions. > * 3) subpartitions_cpus is empty. > + * 4) top_cpuset needs an active boot domain housekeeping CPU from cs. > */ > if (is_local_partition(cs) && > (!is_partition_valid(parent) || > - tasks_nocpu_error(parent, cs, &new_cpus) || > + nocpu || > cpumask_empty(subpartitions_cpus))) > partcmd = partcmd_invalidate; > /* -- Best regards Ridong ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH v4 1/2] cgroup/cpuset: Keep an active housekeeping CPU in the top sched domain 2026-11-12 3:05 ` Ridong Chen @ 2026-11-12 9:43 ` Guopeng Zhang 0 siblings, 0 replies; 5+ messages in thread From: Guopeng Zhang @ 2026-11-12 9:43 UTC (permalink / raw) To: Ridong Chen, longman Cc: cgroups, tj, hannes, mkoutny, shuah, mathieu.desnoyers, linux-kernel, linux-kselftest, Guopeng Zhang, Ridong Chen 在 2026/11/12 11:05, Ridong Chen 写道: > > > On 11/11/2026 9:42 PM, Guopeng Zhang wrote: >> From: Guopeng Zhang <zhangguopeng@kylinos.cn> >> >> Boot-isolated CPUs remain in top_cpuset.effective_cpus, but are excluded >> from its scheduling domain. A partition can therefore take all boot >> domain housekeeping CPUs while leaving effective_cpus nonempty. For >> example, on a 32-CPU system with all CPUs online and >> isolcpus=domain,10-31: >> >> echo 0-9 > A/cpuset.cpus >> echo root > A/cpuset.cpus.partition >> Oops: general protection fault >> RIP: 0010:build_perf_domains+... >> >> tasks_nocpu_error() also rejects valid CPU replacements because it does >> not account for CPUs returned by the update. With a fresh cgroup A under >> the same boot configuration: >> >> echo 2-31 > A/cpuset.cpus >> echo isolated > A/cpuset.cpus.partition >> echo 0-4 > A/cpuset.cpus >> cat A/cpuset.cpus.partition >> isolated invalid (Parent unable to distribute cpu downstream) >> >> The update should return CPUs 5-31 to top_cpuset, including housekeeping >> CPUs 5-9, and leave A valid. This replacement failure also occurs without >> boot isolation and under populated local parents. >> >> Extend tasks_nocpu_error() to take an explicit mask of CPUs being >> returned and use it for local and remote updates. Require an active >> HK_TYPE_DOMAIN_BOOT CPU to remain in top_cpuset. For other populated >> parents, check within their current exclusive grant so an ancestor >> update cannot credit CPUs that the parent no longer owns. >> >> CPU hotplug can also empty the top scheduling domain. Pass no returned >> CPUs during its availability check, and only invalidate a partition to >> restore the top domain if it can return an active boot domain >> housekeeping CPU. Still invalidate a populated partition that has no >> active CPU of its own. >> >> Fixes: 9b496a8bbed9 ("cgroup/cpuset: Prevent leakage of isolated CPUs into sched domains") >> Fixes: 0c7f293efc87 ("cgroup/cpuset: Add cpuset.cpus.exclusive.effective for v2") >> Co-developed-by: Ridong Chen <chenridong@xiaomi.com> >> Signed-off-by: Ridong Chen <chenridong@xiaomi.com> >> Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn> >> --- >> kernel/cgroup/cpuset.c | 85 ++++++++++++++++++++++++++++++++++-------- >> 1 file changed, 70 insertions(+), 15 deletions(-) >> >> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c >> index 3052413e41e7..85f92ae0c6f4 100644 >> --- a/kernel/cgroup/cpuset.c >> +++ b/kernel/cgroup/cpuset.c >> @@ -1181,18 +1181,54 @@ static void update_partition_sd_lb(struct cpuset *cs, int old_prs) >> } >> /* >> - * tasks_nocpu_error - Return true if tasks will have no effective_cpus >> + * tasks_nocpu_error - Return true if tasks or the top domain lack active CPUs >> + * @parent: parent partition, NULL for a remote partition >> + * @cs: the partition >> + * @xcpus: exclusive CPUs that @cs will hold after the update >> + * @del_cpus: exclusive CPUs to be returned to @parent, can be NULL >> + * >> + * @del_cpus may include CPUs that are also in @xcpus and will stay in >> + * @cs. They are skipped by the @xcpus check, so callers can pass the >> + * current effective_xcpus of @cs without masking it first. >> + * >> + * top_cpuset always has tasks. Boot-isolated CPUs remain in its >> + * effective_cpus but are excluded from its sched domain, which must >> + * retain an active boot domain housekeeping CPU. >> */ >> static bool tasks_nocpu_error(struct cpuset *parent, struct cpuset *cs, >> - struct cpumask *xcpus) >> + const struct cpumask *xcpus, >> + const struct cpumask *del_cpus) >> { >> + const struct cpumask *mask; >> + int cpu; >> + >> + /* A populated @cs must have an active CPU in @xcpus. */ >> + if (!cpumask_intersects(xcpus, cpu_active_mask) && >> + partition_is_populated(cs, NULL)) >> + return true; >> + >> /* >> - * A populated partition (cs or parent) can't have empty effective_cpus >> + * A populated @parent must keep an active CPU in @mask, either from >> + * its effective_cpus or from @del_cpus, that isn't taken by @cs. >> */ >> - return (cpumask_subset(parent->effective_cpus, xcpus) && >> - partition_is_populated(parent, cs)) || >> - (!cpumask_intersects(xcpus, cpu_active_mask) && >> - partition_is_populated(cs, NULL)); >> + if (!parent) >> + parent = &top_cpuset; >> + >> + /* CPUs outside the parent's current grant cannot be returned. */ >> + mask = parent->effective_xcpus; >> + if (parent == &top_cpuset) >> + mask = housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT); >> + else if (!partition_is_populated(parent, cs)) >> + return false; >> + > > The mask selection looks right to me. HK_TYPE_DOMAIN_BOOT is the right > one for top_cpuset, since CPUs returned by an isolated child are not in > HK_TYPE_DOMAIN yet. Limiting a local parent to its effective_xcpus also > correctly drops stale CPUs in del_cpus after the parent shrinks. > > One corner case: validate_partition() also calls this for an invalid > cs whose parent is not a valid partition. If the parent is an invalid > partition without cpuset.cpus.exclusive, reset_partition_data() has > cleared its effective_xcpus, the loop does nothing, and we return true > when the parent is populated. The child's prs_err then changes from > PERR_INVPARENT to PERR_NOCPUS, which reports the wrong reason. The old > code did not do that. How about: > > else if (!is_partition_valid(parent) || > !partition_is_populated(parent, cs)) > return false; > > The invalid parent case is already reported by PERR_INVPARENT/PERR_NOTPART. > Hi Ridong, Thanks for the review. I tested your guard and will add it. However, I reproduced the error change under slightly different conditions from those you described: the populated invalid parent still had an explicit exclusive mask. In that case, v4 changes PERR_INVPARENT to PERR_NOCPUS. Your check preserves the original reason, and also preserves PERR_NOTPART in the corresponding case where the parent is a member. In the case you described, with no explicit exclusive mask and the parent's effective_xcpus already cleared, I get PERR_INVCPUS in both the base kernel and v4. compute_trialcs_excpus() also produces an empty child trial mask, so validate_partition() returns PERR_INVCPUS before calling tasks_nocpu_error(). > Nit: the comment "Return true if tasks or the top domain lack active > CPUs" doesn't cover local parents. Maybe "Return true if @cs or @parent > would lack active CPUs; for top_cpuset, only active boot domain > housekeeping CPUs count". > Sure, I'll update the comment. > >> + for_each_cpu_and(cpu, mask, cpu_active_mask) { >> + if (cpumask_test_cpu(cpu, xcpus)) >> + continue; >> + if (cpumask_test_cpu(cpu, parent->effective_cpus) || >> + (del_cpus && cpumask_test_cpu(cpu, del_cpus))) >> + return false; >> + } >> + return true; >> } >> static void reset_partition_data(struct cpuset *cs) >> @@ -1724,7 +1760,7 @@ static int remote_partition_enable(struct cpuset *cs, int new_prs, >> */ >> compute_excpus(cs, tmp->new_cpus); >> if (!cpumask_intersects(tmp->new_cpus, cpu_active_mask) || >> - cpumask_subset(top_cpuset.effective_cpus, tmp->new_cpus)) >> + tasks_nocpu_error(NULL, cs, tmp->new_cpus, NULL)) >> return PERR_INVCPUS; >> if (cpumask_intersects(tmp->new_cpus, subpartitions_cpus)) >> return PERR_NOCPUS; >> @@ -1828,7 +1864,7 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus, >> if (!capable(CAP_SYS_ADMIN)) >> WRITE_ONCE(cs->prs_err, PERR_ACCESS); >> else if (cpumask_intersects(tmp->addmask, subpartitions_cpus) || >> - cpumask_subset(top_cpuset.effective_cpus, tmp->addmask)) >> + tasks_nocpu_error(NULL, cs, excpus, cs->effective_xcpus)) >> WRITE_ONCE(cs->prs_err, PERR_NOCPUS); >> else if (prstate_housekeeping_conflict(prs, PRS_ROOT, >> excpus, tmp->delmask)) >> @@ -1961,7 +1997,8 @@ static int update_parent_effective_cpumask(struct cpuset *cs, int cmd, >> */ >> if ((cmd != partcmd_disable) && !newmask) { >> compute_excpus(cs, tmp->delmask); >> - nocpu = tasks_nocpu_error(parent, cs, tmp->delmask); >> + nocpu = tasks_nocpu_error(parent, cs, tmp->delmask, >> + is_partition_valid(cs) ? cs->effective_xcpus : NULL); >> } >> if ((cmd == partcmd_enable) || (cmd == partcmd_enablei)) { >> @@ -2497,7 +2534,9 @@ static enum prs_errcode validate_partition(struct cpuset *cs, struct cpuset *tri >> if (cpumask_empty(trialcs->effective_xcpus)) >> return PERR_INVCPUS; >> - if (tasks_nocpu_error(parent, cs, trialcs->effective_xcpus)) >> + if (tasks_nocpu_error(is_remote_partition(cs) ? NULL : parent, cs, >> + trialcs->effective_xcpus, >> + is_partition_valid(cs) ? cs->effective_xcpus : NULL)) >> return PERR_NOCPUS; >> return PERR_NONE; >> @@ -3997,6 +4036,7 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) >> static nodemask_t new_mems; >> bool cpus_updated; >> bool mems_updated; >> + bool nocpu; >> int partcmd = -1; >> struct cpuset *parent; >> @@ -4008,9 +4048,23 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) >> if (!tmp || !cs->partition_root_state) >> goto update_tasks; >> - if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) || >> - (cpumask_empty(&new_cpus) && >> - partition_is_populated(cs, NULL)))) { >> + /* Do not count any CPUs as returned during this hotplug check. */ >> + nocpu = tasks_nocpu_error(is_remote_partition(cs) ? NULL : parent, cs, >> + &new_cpus, NULL); >> + if (nocpu && (parent == &top_cpuset || is_remote_partition(cs))) { >> + /* >> + * Only invalidate a partition for an empty top sched domain if >> + * it can return an active boot domain housekeeping CPU. A >> + * populated partition with no active CPU must still be invalidated. >> + */ >> + cpumask_and(tmp->addmask, cs->effective_xcpus, >> + housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT)); >> + if (!cpumask_intersects(tmp->addmask, cpu_active_mask)) >> + nocpu = cpumask_empty(&new_cpus) && >> + partition_is_populated(cs, NULL); >> + } >> + > > This is hard to follow. nocpu is computed first and then partly undone, > so that only a partition owning an active housekeeping CPU gets > invalidated. > v3 used separate branches for these cases. Adapting that structure to the unified helper, would something like the following be easier to follow? if (parent == &top_cpuset || is_remote_partition(cs)) { /* * Do not count child partition CPUs as already returned. * Only invalidate a partition for an empty top domain if * it can return an active boot domain housekeeping CPU. * A populated partition with no active CPU of its own * must still be invalidated. */ cpumask_and(tmp->addmask, cs->effective_xcpus, housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT)); nocpu = cpumask_intersects(tmp->addmask, cpu_active_mask) && tasks_nocpu_error(NULL, cs, cs->effective_xcpus, NULL); nocpu |= cpumask_empty(&new_cpus) && partition_is_populated(cs, NULL); } else { nocpu = tasks_nocpu_error(parent, cs, &new_cpus, NULL); } > Do we really need this? When hotplug leaves top_cpuset with no active > CPU, cpuset_handle_hotplug() already clears subpartitions_cpus and > invalidates all the partitions under it. If we handle an empty top > sched domain the same way, this block can go away. > The check keeps partitions that cannot return an active HK CPU from being invalidated just because the top domain is empty. For example, on a 32-CPU system with isolcpus=domain,10-31, A is an isolated partition holding CPU 10 and its sibling B is a root partition holding CPUs 0,2-9. When CPU 1 goes offline, v4 invalidates B and keeps A valid. As I understand it, if we use the reset approach, it can restore the top scheduling domain, but it should invalidate both A and B. If we eventually want the same reset policy for an empty top scheduling domain, I can make that change and update the existing test expectations. Thanks, Guopeng > Longman, what do you think? > >> + if (is_remote_partition(cs) && (cpumask_empty(subpartitions_cpus) || nocpu)) { >> WRITE_ONCE(cs->prs_err, PERR_HOTPLUG); >> remote_partition_disable(cs, tmp); >> compute_effective_cpumask(&new_cpus, cs, parent); >> @@ -4023,10 +4077,11 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp) >> * 2) parent is invalid or doesn't grant any cpus to child >> * partitions. >> * 3) subpartitions_cpus is empty. >> + * 4) top_cpuset needs an active boot domain housekeeping CPU from cs. >> */ >> if (is_local_partition(cs) && >> (!is_partition_valid(parent) || >> - tasks_nocpu_error(parent, cs, &new_cpus) || >> + nocpu || >> cpumask_empty(subpartitions_cpus))) >> partcmd = partcmd_invalidate; >> /* > ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH v4 2/2] selftests/cgroup: Add tests for top sched domain housekeeping checks 2026-11-11 13:42 [PATCH v4 0/2] cgroup/cpuset: Keep the top scheduling domain populated Guopeng Zhang 2026-11-11 13:42 ` [PATCH v4 1/2] cgroup/cpuset: Keep an active housekeeping CPU in the top sched domain Guopeng Zhang @ 2026-11-11 13:42 ` Guopeng Zhang 1 sibling, 0 replies; 5+ messages in thread From: Guopeng Zhang @ 2026-11-11 13:42 UTC (permalink / raw) To: longman, ridong.chen Cc: cgroups, tj, hannes, mkoutny, shuah, mathieu.desnoyers, linux-kernel, linux-kselftest, Guopeng Zhang From: Guopeng Zhang <zhangguopeng@kylinos.cn> A partition update or CPU hotplug must leave at least one active boot domain housekeeping CPU in the top cpuset scheduling domain. Add tests to test_cpuset_prs.sh for local and remote partition updates, recovery and CPU hotplug. Run the new matrix when boot domain isolation leaves CPUs 0-9 for housekeeping. Include hotplug cases that keep unrelated partitions valid. Also add CPU replacement tests to TEST_MATRIX for the top cpuset and populated local parents. Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn> --- .../selftests/cgroup/test_cpuset_prs.sh | 71 +++++++++++++++++++ 1 file changed, 71 insertions(+) diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh index bdca6c471f87..c35d4fd50eeb 100755 --- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh +++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh @@ -168,6 +168,13 @@ TEST_ISOLCPUS= skip_test "Current isolated CPUs ($CURRENT_ISOLCPUS) don't match boot isolated CPUs!" } +# These tests require boot domain housekeeping CPUs to be exactly 0-9. +# Match the boot mask for isolcpus= with or without explicit domain flags. +DOMAIN_ISOLCPUS="10-$LAST_PRESENT_CPU" +[[ "$LAST_PRESENT_CPU" == 10 ]] && DOMAIN_ISOLCPUS=10 +RUN_ISOLCPUS_DOMAIN_TESTS= +[[ "$BOOT_ISOLCPUS" == "$DOMAIN_ISOLCPUS" ]] && RUN_ISOLCPUS_DOMAIN_TESTS=1 + # # This test assumes that all the relevant CPUs being tested are online. # If some of them are offline, some of the tests may fail. So we are going to @@ -547,6 +554,12 @@ TEST_MATRIX=( " C2-6:X2-5:P1 CX3-6:P2 \ . . T:X4-6 . . . 0 A1:4-6|A2:4-6 A1:P1|A2:P-2 ." + # A populated local parent must also count CPUs returned by its child. + " C2-7:P1 C4-7:P2 . . T C2-5 . . 0 A1:6-7|A2:2-5 A1:P1|A2:P2 2-5" + + # A CPU replacement must account for CPUs returned to the top cpuset. + " C2-${CPULIST#0-}:P2 . . . C0-4 . . . 0 A1:0-4 A1:P2 0-4" + # # old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pstate ISOLCPUS # ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ------ -------- @@ -602,6 +615,63 @@ ISOLCPUS_TEST_MATRIX=( " CX1-2,I:X1,I:P1 . . . X1,2,I . . . 0 A1:1-2,I|XA1:1-2,I A1:P-1 ." ) +# Test matrix for isolcpus=domain,10-<last> or isolcpus=10-<last> boots. +# The top cpuset must retain an active boot domain housekeeping CPU. +ISOLCPUS_DOMAIN_TEST_MATRIX=( + # old-A1 old-A2 old-A3 old-B1 new-A1 new-A2 new-A3 new-B1 fail ECPUs Pstate ISOLCPUS + # ------ ------ ------ ------ ------ ------ ------ ------ ---- ----- ------ -------- + # Neither a root nor an isolated partition may take all boot HK CPUs. + " C0-9 . . . P1 . . . 0 A1:0-9 A1:P-1 ." + " C0-9 . . . P2 . . . 0 A1:0-9 A1:P-2 ." + + # Growing cpuset.cpus must not take the last top-level HK CPU. + " C0-8:P1 . . . C0-9 . . . 0 A1:0-9 A1:P-1 ." + + # CPUs returned by the same update can keep the top sched domain alive. + " C2-9:P2 . . . C0-4 . . . 0 A1:0-4 A1:P2 0-4" + + # Root and isolated partitions together must leave a top-level HK CPU. + " C0-4:P1 . . C5-9 . . . P2 0 A1:0-4|B1:5-9 A1:P1|B1:P-2 ." + + # Remote partition creation must enforce the same constraint. + " X0-9 CX0-9 . . . P1 . . 0 A2:0-9 A2:P-1 ." + + # Expanding a remote partition must leave a top-level HK CPU. + " X0-9 C0-9:X0-8:P1 . . . X0-9 . . 0 A2:0-9 A2:P-1 ." + + # Remote CPU replacements must account for the CPUs being returned. + " X0-9 C0-9:X2-9:P2 . . . X0-4 . . 0 A2:0-4 A2:P2 0-4" + + # Local exclusive CPU updates must not consume the last top-level HK CPU. + # Keep the initial partition valid with nohz_full enabled. + " C0-9:X2-9:P2 . . . X0-9 . . . 0 A1:0-9 A1:P-2 ." + + # Ancestor updates must not grow a remote partition over the last HK CPU. + " X0-8 CX0-9:P1 . . X0-9 . . . 0 A2:0-9 A2:P-1 ." + + # An invalid partition can recover once it leaves a top-level HK CPU. + " C0-9:P1 . . . C0-6,8-9 . . . 0 A1:0-6,8-9 A1:P1 7|" + + # Offlining the last top-level HK CPU invalidates a local root and its child. + " C0,2-9:P1 C5-9:P1 . . O1=0 . . . 0 A1:0,2-9|A2:5-9 A1:P-1|A2:P-1 1|" + + # A remote root must return CPUs when the top domain loses its last HK CPU. + " X0,2-9 CX0,2-9:P1 . . O1=0 . . . 0 A2:0,2-9 A2:P-1 1|" +) + +# Keep an online boot-isolated CPU at the top so hotplug does not +# clear subpartitions_cpus before checking the individual partitions. +if [[ "$ONLINE_LAST" -gt 10 ]]; then + ISOLCPUS_DOMAIN_TEST_MATRIX+=( + # An offline HK CPU cannot restore the top domain. Keep this local + # partition valid while its sibling returns active HK CPUs. + " O2=0:C2,10:P2 . . C0,3-9:P1 O1=0 . . . 0 A1:10|B1:0,3-9 A1:P2|B1:P-1 1-2|2" + + # Check the same active-CPU rule for a remote partition. + " O2=0:X2,10 CX2,10:P2 . C0,3-9:P1 O1=0 . . . 0 A2:10|B1:0,3-9 A2:P2|B1:P-1 1-2|2" + ) +fi + # Test matrix to be run when boot time nohz_full CPUs cover all CPUs except CPUs 0-7 # This also means that CPU 8 is available and is a nohz_full CPU. NOHZ_FULL_TEST_MATRIX=( @@ -1472,6 +1542,7 @@ test_inotify() run_state_test TEST_MATRIX run_remote_state_test REMOTE_TEST_MATRIX TEST_ISOLCPUS=1 run_state_test ISOLCPUS_TEST_MATRIX +[[ -n "$RUN_ISOLCPUS_DOMAIN_TESTS" ]] && run_state_test ISOLCPUS_DOMAIN_TEST_MATRIX [[ -n "$RUN_NOHZ_FULL_TESTS" ]] && run_state_test NOHZ_FULL_TEST_MATRIX test_isolated test_boot_isolated -- 2.43.0 ^ permalink raw reply related [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-11-12 9:43 UTC | newest] Thread overview: 5+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-11 13:42 [PATCH v4 0/2] cgroup/cpuset: Keep the top scheduling domain populated Guopeng Zhang 2026-11-11 13:42 ` [PATCH v4 1/2] cgroup/cpuset: Keep an active housekeeping CPU in the top sched domain Guopeng Zhang 2026-11-12 3:05 ` Ridong Chen 2026-11-12 9:43 ` Guopeng Zhang 2026-11-11 13:42 ` [PATCH v4 2/2] selftests/cgroup: Add tests for top sched domain housekeeping checks Guopeng Zhang
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox