From: Guopeng Zhang <guopeng.zhang@linux.dev>
To: "Waiman Long" <longman@redhat.com>,
"Ridong Chen" <ridong.chen@linux.dev>,
"Tejun Heo" <tj@kernel.org>,
"Johannes Weiner" <hannes@cmpxchg.org>,
"Michal Koutný" <mkoutny@suse.com>,
"Shuah Khan" <shuah@kernel.org>
Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org,
linux-kselftest@vger.kernel.org, Sashiko <sashiko-bot@kernel.org>
Subject: Re: [PATCH-next v4 5/6] cgroup/cpuset: Handle remote partitions before local partitions in cpuset_handle_hotplug()
Date: Mon, 16 Nov 2026 18:33:25 +0800 [thread overview]
Message-ID: <70eb1395-ebcf-4ece-ac03-f0796cd02c81@linux.dev> (raw)
In-Reply-To: <20261116012330.2271010-6-longman@redhat.com>
在 2026/11/16 09:23, Waiman Long 写道:
> When there is a valid remote partition underneath an invalid local
> partition with subpartitions_cpus cleared because of missing active
> CPUs in the top_cpuset partition, the current iteration logic in
> cpuset_handle_hotplug() will handle the invalid local partition first
> and potentially make the local partition valid. The remote partition
> will be handled later with the potential that both the local and remote
> partitions will be valid with overlapping exclusive CPUs violating the
> cpu exclusivity rule.
>
> Fix this problem by adding a new remote_partition_scan() helper to
> iterate the existing remote partitions first before iterating down
> the hierarchy to handle the local partitions. That also means that
> when subpartitions_cpus is cleared, remote partitions have precedence
> to acquire the exclusive CPUs first before local partitions. It also
> gives remote partition the ability to recover from the clearing of
> subpartitions_cpus as all the remote partitions will be disabled before
> this commit.
>
> A remote partition can only be created if there is at least one active
> CPU in its effective_xcpus. So remote_partition_scan() will also
> invalidate the remote partition if there is no active CPU left in its
> effective_xcpus. This requires a minor change in a test_cpuset_prs.sh
> test case where a remote partition with no active CPU left will now
> be invalidated.
>
> With the addition of the remote_partition_scan() call in
> cpuset_handle_hotplug(), there is no longer a need to do a remote
> partition check in cpuset_hotplug_update_tasks() and the corresponding
> code can be removed.
>
> While at it, the subpartition_cpus clearing logic is enhanced to
> clear it when no domain housekeeping CPUs are left in the top_cpuset
> partition. When the remote partition is disabled, the effective_cpus
> is modified in a such a way that cpus_updated will be set true later
> in cpuset_hotplug_update_tasks() so that the CPU affinity of the
> tasks within the partition will be updated. This will also fix a
> bug reported by sashiko that CPU affinity will not be updated when
> remote_partition_disable() is called in such circumstance.
>
> When efffective_cpus is read during this interim period, the content
> will be invalid before and after this commit.
>
> Fixes: 181c8e091aae ("cgroup/cpuset: Introduce remote partition")
> Reported-by: Guopeng Zhang <guopeng.zhang@linux.dev>
> Link: https://lore.kernel.org/lkml/fe7aa630-d323-4981-9f89-0958cb6204a2@linux.dev
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Link: https://sashiko.dev/#/bug/linux-ab90f7e8-0d60-40ca-83e9-e405075b24d2
> Signed-off-by: Waiman Long <longman@redhat.com>
Hi,Longman
Tested all six patches on next-20261113 (32-CPU VM, cgroup v2, no boot
isolation, all CPUs initially online). I can still reproduce an invalid
local partition becoming valid while its remote descendant remains
valid:
cd /sys/fs/cgroup
echo +cpuset > cgroup.subtree_control
echo 0 > /sys/devices/system/cpu/cpu31/online
mkdir A
echo 0-30 > A/cpuset.cpus
echo 0-30 > A/cpuset.cpus.exclusive
echo root > A/cpuset.cpus.partition # invalid local root
echo +cpuset > A/cgroup.subtree_control
mkdir A/B
echo 20-25 > A/B/cpuset.cpus
echo 20-25 > A/B/cpuset.cpus.exclusive
echo root > A/B/cpuset.cpus.partition # valid remote partition
echo 1 > /sys/devices/system/cpu/cpu31/online
After CPU31 comes online, A is a valid root (exclusive.effective 0-30),
A/B is still a valid remote (exclusive.effective 20-25), and the top
cpuset's effective CPUs are 31. The kernel reports:
WARNING: cpuset.c:1654 compute_partition_effective_cpumask()
WARN_ON_ONCE(is_remote_partition(child))
Two more results:
1. Local partitions break too, no remote involved. test_cpuset_prs.sh
fails at TEST_MATRIX[19] (C0-3:P1 C2-3:P1 + O2=0): with cpu2
offlined, both partitions become "root invalid" instead of remaining
valid. A1's effective_cpus then falls back to 0-1,3 instead of 0-1.
2. Booted with isolcpus=31, the same kernel passes the whole suite.
Thanks,
Guopeng> ---
> kernel/cgroup/cpuset.c | 97 ++++++++++++++++---
> .../selftests/cgroup/test_cpuset_prs.sh | 2 +-
> 2 files changed, 84 insertions(+), 15 deletions(-)
>
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index 0fe9f2731fee..64cbcbdbf6da 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -153,6 +153,12 @@ static cpumask_var_t isolated_cpus; /* CSCB */
> */
> static struct list_head remote_children; /* RWCS */
>
> +/*
> + * Temporary cpumask for local cpumask computation
> + * The cpumask may be changed when a cpuset helper is being called.
> + */
> +static cpumask_var_t tmp_cpus; /* RWCS */
> +
> /*
> * Set if housekeeping cpumasks are to be updated.
> */
> @@ -1873,6 +1879,67 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
> remote_partition_disable(cs, tmp);
> }
>
> +/**
> + * remote_partition_scan: scan remote partitions to check exclusive CPUs
> + * @tmpmask: ptr to tmpmasks structure
> + *
> + * This function is called in cpuset_handle_hotplug() after the update on
> + * top_cpuset.effective_cpus when there is cpu online/offline operation.
> + * As subpartitions_cpus may have been cleared, we need to check if the remote
> + * partition should acquire the exclusive CPUs again without impacting
> + * effective_cpus or it needs to be disabled when one or more of the following
> + * conditions is true.
> + * 1) All the domain housekeeping CPUs in the top_cpuset will be taken away
> + * 2) A populated partition with no active CPU or no active CPU in
> + * effective_xcpus
> + */
> +static void remote_partition_scan(struct tmpmasks *tmp)
> +{
> + struct cpuset *cs, *next;
> +
> + rcu_read_lock();
> + list_for_each_entry_safe(cs, next, &remote_children, remote_sibling) {
> + bool noxcpus = !cpumask_subset(cs->effective_xcpus, subpartitions_cpus);
> + bool nocpus = !cpumask_intersects(cs->effective_xcpus, cpu_active_mask) ||
> + (!cpumask_intersects(cs->effective_cpus, cpu_active_mask) &&
> + partition_is_populated(cs, NULL));
> + int cpu;
> +
> + cpumask_and(tmp_cpus, top_cpuset.effective_cpus,
> + housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
> + if (noxcpus && !nocpus && !cpumask_subset(tmp_cpus, cs->effective_xcpus)) {
> + /*
> + * Reacquire exclusive CPUs & update top_cpuset.effective_cpus
> + */
> + spin_lock_irq(&callback_lock);
> + cpumask_andnot(subpartitions_cpus, subpartitions_cpus,
> + cs->effective_xcpus);
> + cpumask_andnot(top_cpuset.effective_cpus, top_cpuset.effective_xcpus,
> + cs->effective_xcpus);
> + spin_unlock_irq(&callback_lock);
> + continue;
> + }
> +
> + if (noxcpus || nocpus || cpumask_empty(tmp_cpus)) {
> + /*
> + * Disable remote partition & modifiy effective_cpus
> + * so that cpus_updated will be set true in
> + * cpuset_hotplug_update_tasks().
> + */
> + WRITE_ONCE(cs->prs_err, PERR_HOTPLUG);
> + remote_partition_disable(cs, tmp);
> + cpu = cpumask_first(cs->effective_cpus);
> + spin_lock_irq(&callback_lock);
> + if (cpu < nr_cpu_ids)
> + cpumask_clear_cpu(cpu, cs->effective_cpus);
> + else
> + cpumask_set_cpu(0, cs->effective_cpus);
> + spin_unlock_irq(&callback_lock);
> + }
> + }
> + rcu_read_unlock();
> +}
> +
> /**
> * update_parent_effective_cpumask - update effective_cpus mask of parent cpuset
> * @cs: The cpuset that requests change in partition root state
> @@ -3919,6 +3986,7 @@ int __init cpuset_init(void)
> BUG_ON(!zalloc_cpumask_var(&subpartitions_cpus, GFP_KERNEL));
> BUG_ON(!zalloc_cpumask_var(&isolated_cpus, GFP_KERNEL));
> BUG_ON(!zalloc_cpumask_var(&isolated_hk_cpus, GFP_KERNEL));
> + BUG_ON(!alloc_cpumask_var(&tmp_cpus, GFP_KERNEL));
>
> cpumask_setall(top_cpuset.cpus_allowed);
> nodes_setall(top_cpuset.mems_allowed);
> @@ -3993,14 +4061,6 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
> if (!tmp || !cs->partition_root_state)
> goto update_tasks;
>
> - if (is_remote_partition(cs) &&
> - (!cpumask_subset(cs->effective_xcpus, subpartitions_cpus) ||
> - (cpumask_empty(&new_cpus) && partition_is_populated(cs, NULL)))) {
> - WRITE_ONCE(cs->prs_err, PERR_HOTPLUG);
> - remote_partition_disable(cs, tmp);
> - compute_effective_cpumask(&new_cpus, cs, parent);
> - }
> -
> /*
> * Force the partition to become invalid if either one of
> * the following conditions hold:
> @@ -4104,22 +4164,31 @@ static void cpuset_handle_hotplug(void)
> if (!on_dfl)
> cpumask_copy(top_cpuset.cpus_allowed, &new_cpus);
> /*
> - * Make sure that CPUs allocated to child partitions
> - * do not show up in effective_cpus. If no CPU is left,
> - * we clear the subpartitions_cpus & let the child partitions
> - * fight for the CPUs again.
> + * Make sure that CPUs allocated to child partitions do not
> + * show up in effective_cpus. If no housekeeping CPU is left,
> + * we clear the subpartitions_cpus, scan the remote partitions
> + * to determine if they can remain valid & let the descendant
> + * local partitions fight for the CPUs again.
> */
> if (!cpumask_empty(subpartitions_cpus)) {
> + cpumask_andnot(&new_cpus, cpu_active_mask,
> + housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
> if (cpumask_subset(&new_cpus, subpartitions_cpus)) {
> cpumask_clear(subpartitions_cpus);
> + cpumask_copy(&new_cpus, cpu_active_mask);
> } else {
> - cpumask_andnot(&new_cpus, &new_cpus,
> + cpumask_andnot(&new_cpus, cpu_active_mask,
> subpartitions_cpus);
> }
> }
> cpumask_copy(top_cpuset.effective_cpus, &new_cpus);
> spin_unlock_irq(&callback_lock);
> - /* we don't mess with cpumasks of tasks in top_cpuset */
> + /*
> + * We don't mess with cpumasks of tasks in top_cpuset, but we
> + * have to examine impact on remote partition roots if present.
> + */
> + if (cpuset_v2() && ptmp)
> + remote_partition_scan(ptmp);
> }
>
> /* synchronize mems_allowed to N_MEMORY */
> diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> index 81f0eecaca80..5c7b583cba18 100755
> --- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> +++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> @@ -378,7 +378,7 @@ TEST_MATRIX=(
> # Remote partition offline tests
> " C0-3 C1-3 C2-3 . X2-3 X2-3 X2-3:P2:O2=0 . 0 A1:0-1|A2:1|A3:3 A1:P0|A3:P2 2-3"
> " C0-3 C1-3 C2-3 . X2-3 X2-3 X2-3:P2:O2=0 O2=1 0 A1:0-1|A2:1|A3:2-3 A1:P0|A3:P2 2-3"
> - " C0-3 C1-3 C3 . X2-3 X2-3 P2:O3=0 . 0 A1:0-2|A2:1-2|A3: A1:P0|A3:P2 3"
> + " C0-3 C1-3 C3 . X2-3 X2-3 P2:O3=0 . 0 A1:0-2|A2:1-2|A3:1-2 A1:P0|A3:P-2 3|"
> " C0-3 C1-3 C3 . X2-3 X2-3 T:P2:O3=0 . 0 A1:0-2|A2:1-2|A3:1-2 A1:P0|A3:P-2 3|"
>
> # An invalidated remote partition cannot self-recover from hotplug
next prev parent reply other threads:[~2026-11-16 10:33 UTC|newest]
Thread overview: 18+ messages / expand[flat|nested] mbox.gz Atom feed top
2026-11-16 1:23 [PATCH-next v4 0/6] cgroup/cpuset: More fix patches Waiman Long
2026-11-16 1:23 ` [PATCH-next v4 1/6] cgroup/cpuset: Fix incorrect partition invalidation logic in cpuset_hotplug_update_tasks() Waiman Long
2026-11-16 10:19 ` Guopeng Zhang
2026-11-16 1:23 ` [PATCH-next v4 2/6] cgroup/cpuset: excpus parameter of remote_cpus_update() must have at least 1 active CPU Waiman Long
2026-11-16 3:07 ` Ridong Chen
2026-11-16 3:34 ` Waiman Long
2026-11-16 7:37 ` Ridong Chen
2026-11-16 1:23 ` [PATCH-next v4 3/6] cgroup/cpuset: Check remote partition underneath before making an invalid partition valid Waiman Long
2026-11-16 3:47 ` Ridong Chen
2026-11-16 4:21 ` Waiman Long
2026-11-16 1:23 ` [PATCH-next v4 4/6] cgroup/cpuset: Revert "remove global remote_children list" Waiman Long
2026-11-16 1:23 ` [PATCH-next v4 5/6] cgroup/cpuset: Handle remote partitions before local partitions in cpuset_handle_hotplug() Waiman Long
2026-11-16 7:08 ` Ridong Chen
2026-11-16 8:56 ` kernel test robot
2026-11-16 10:33 ` Guopeng Zhang [this message]
2026-11-16 1:23 ` [PATCH-next v4 6/6] cgroup/cpuset: Fix lockdep_is_cpuset_held() Waiman Long
2026-11-16 7:24 ` Ridong Chen
2026-11-16 10:34 ` Guopeng Zhang
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=70eb1395-ebcf-4ece-ac03-f0796cd02c81@linux.dev \
--to=guopeng.zhang@linux.dev \
--cc=cgroups@vger.kernel.org \
--cc=hannes@cmpxchg.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-kselftest@vger.kernel.org \
--cc=longman@redhat.com \
--cc=mkoutny@suse.com \
--cc=ridong.chen@linux.dev \
--cc=sashiko-bot@kernel.org \
--cc=shuah@kernel.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox