cgroups.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Guopeng Zhang <guopeng.zhang@linux.dev>
To: "Waiman Long" <longman@redhat.com>,
	"Ridong Chen" <ridong.chen@linux.dev>,
	"Tejun Heo" <tj@kernel.org>,
	"Johannes Weiner" <hannes@cmpxchg.org>,
	"Michal Koutný" <mkoutny@suse.com>,
	"Shuah Khan" <shuah@kernel.org>
Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org,
	linux-kselftest@vger.kernel.org, Sashiko <sashiko-bot@kernel.org>
Subject: Re: [PATCH-next v4 5/6] cgroup/cpuset: Handle remote partitions before local partitions in cpuset_handle_hotplug()
Date: Mon, 16 Nov 2026 18:33:25 +0800	[thread overview]
Message-ID: <70eb1395-ebcf-4ece-ac03-f0796cd02c81@linux.dev> (raw)
In-Reply-To: <20261116012330.2271010-6-longman@redhat.com>



在 2026/11/16 09:23, Waiman Long 写道:
> When there is a valid remote partition underneath an invalid local
> partition with subpartitions_cpus cleared because of missing active
> CPUs in the top_cpuset partition, the current iteration logic in
> cpuset_handle_hotplug() will handle the invalid local partition first
> and potentially make the local partition valid. The remote partition
> will be handled later with the potential that both the local and remote
> partitions will be valid with overlapping exclusive CPUs violating the
> cpu exclusivity rule.
> 
> Fix this problem by adding a new remote_partition_scan() helper to
> iterate the existing remote partitions first before iterating down
> the hierarchy to handle the local partitions. That also means that
> when subpartitions_cpus is cleared, remote partitions have precedence
> to acquire the exclusive CPUs first before local partitions. It also
> gives remote partition the ability to recover from the clearing of
> subpartitions_cpus as all the remote partitions will be disabled before
> this commit.
> 
> A remote partition can only be created if there is at least one active
> CPU in its effective_xcpus. So remote_partition_scan() will also
> invalidate the remote partition if there is no active CPU left in its
> effective_xcpus. This requires a minor change in a test_cpuset_prs.sh
> test case where a remote partition with no active CPU left will now
> be invalidated.
> 
> With the addition of the remote_partition_scan() call in
> cpuset_handle_hotplug(), there is no longer a need to do a remote
> partition check in cpuset_hotplug_update_tasks() and the corresponding
> code can be removed.
> 
> While at it, the subpartition_cpus clearing logic is enhanced to
> clear it when no domain housekeeping CPUs are left in the top_cpuset
> partition. When the remote partition is disabled, the effective_cpus
> is modified in a such a way that cpus_updated will be set true later
> in cpuset_hotplug_update_tasks() so that the CPU affinity of the
> tasks within the partition will be updated. This will also fix a
> bug reported by sashiko that CPU affinity will not be updated when
> remote_partition_disable() is called in such circumstance.
> 
> When efffective_cpus is read during this interim period, the content
> will be invalid before and after this commit.
> 
> Fixes: 181c8e091aae ("cgroup/cpuset: Introduce remote partition")
> Reported-by: Guopeng Zhang <guopeng.zhang@linux.dev>
> Link: https://lore.kernel.org/lkml/fe7aa630-d323-4981-9f89-0958cb6204a2@linux.dev
> Reported-by: Sashiko <sashiko-bot@kernel.org>
> Link: https://sashiko.dev/#/bug/linux-ab90f7e8-0d60-40ca-83e9-e405075b24d2
> Signed-off-by: Waiman Long <longman@redhat.com>

Hi,Longman

Tested all six patches on next-20261113 (32-CPU VM, cgroup v2, no boot
isolation, all CPUs initially online). I can still reproduce an invalid
local partition becoming valid while its remote descendant remains
valid:

  cd /sys/fs/cgroup
  echo +cpuset > cgroup.subtree_control
  echo 0 > /sys/devices/system/cpu/cpu31/online
  mkdir A
  echo 0-30 > A/cpuset.cpus
  echo 0-30 > A/cpuset.cpus.exclusive
  echo root > A/cpuset.cpus.partition          # invalid local root
  echo +cpuset > A/cgroup.subtree_control
  mkdir A/B
  echo 20-25 > A/B/cpuset.cpus
  echo 20-25 > A/B/cpuset.cpus.exclusive
  echo root > A/B/cpuset.cpus.partition        # valid remote partition
  echo 1 > /sys/devices/system/cpu/cpu31/online

After CPU31 comes online, A is a valid root (exclusive.effective 0-30),
A/B is still a valid remote (exclusive.effective 20-25), and the top
cpuset's effective CPUs are 31. The kernel reports:

  WARNING: cpuset.c:1654 compute_partition_effective_cpumask()
           WARN_ON_ONCE(is_remote_partition(child))

Two more results:
1. Local partitions break too, no remote involved.  test_cpuset_prs.sh
   fails at TEST_MATRIX[19] (C0-3:P1 C2-3:P1 + O2=0): with cpu2
   offlined, both partitions become "root invalid" instead of remaining
   valid. A1's effective_cpus then falls back to 0-1,3 instead of 0-1.
2. Booted with isolcpus=31, the same kernel passes the whole suite.

Thanks,
Guopeng> ---
>  kernel/cgroup/cpuset.c                        | 97 ++++++++++++++++---
>  .../selftests/cgroup/test_cpuset_prs.sh       |  2 +-
>  2 files changed, 84 insertions(+), 15 deletions(-)
> 
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index 0fe9f2731fee..64cbcbdbf6da 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -153,6 +153,12 @@ static cpumask_var_t	isolated_cpus;		/* CSCB */
>   */
>  static struct list_head remote_children;	/* RWCS */
>  
> +/*
> + * Temporary cpumask for local cpumask computation
> + * The cpumask may be changed when a cpuset helper is being called.
> + */
> +static cpumask_var_t	tmp_cpus;		/* RWCS */
> +
>  /*
>   * Set if housekeeping cpumasks are to be updated.
>   */
> @@ -1873,6 +1879,67 @@ static void remote_cpus_update(struct cpuset *cs, struct cpumask *xcpus,
>  	remote_partition_disable(cs, tmp);
>  }
>  
> +/**
> + * remote_partition_scan: scan remote partitions to check exclusive CPUs
> + * @tmpmask: ptr to tmpmasks structure
> + *
> + * This function is called in cpuset_handle_hotplug() after the update on
> + * top_cpuset.effective_cpus when there is cpu online/offline operation.
> + * As subpartitions_cpus may have been cleared, we need to check if the remote
> + * partition should acquire the exclusive CPUs again without impacting
> + * effective_cpus or it needs to be disabled when one or more of the following
> + * conditions is true.
> + * 1) All the domain housekeeping CPUs in the top_cpuset will be taken away
> + * 2) A populated partition with no active CPU or no active CPU in
> + *    effective_xcpus
> + */
> +static void remote_partition_scan(struct tmpmasks *tmp)
> +{
> +	struct cpuset *cs, *next;
> +
> +	rcu_read_lock();
> +	list_for_each_entry_safe(cs, next, &remote_children, remote_sibling) {
> +		bool noxcpus = !cpumask_subset(cs->effective_xcpus, subpartitions_cpus);
> +		bool nocpus = !cpumask_intersects(cs->effective_xcpus, cpu_active_mask) ||
> +			      (!cpumask_intersects(cs->effective_cpus, cpu_active_mask) &&
> +			       partition_is_populated(cs, NULL));
> +		int cpu;
> +
> +		cpumask_and(tmp_cpus, top_cpuset.effective_cpus,
> +			    housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
> +		if (noxcpus && !nocpus && !cpumask_subset(tmp_cpus, cs->effective_xcpus)) {
> +			/*
> +			 * Reacquire exclusive CPUs & update top_cpuset.effective_cpus
> +			 */
> +			spin_lock_irq(&callback_lock);
> +			cpumask_andnot(subpartitions_cpus, subpartitions_cpus,
> +				       cs->effective_xcpus);
> +			cpumask_andnot(top_cpuset.effective_cpus, top_cpuset.effective_xcpus,
> +				       cs->effective_xcpus);
> +			spin_unlock_irq(&callback_lock);
> +			continue;
> +		}
> +
> +		if (noxcpus || nocpus || cpumask_empty(tmp_cpus)) {
> +			/*
> +			 * Disable remote partition & modifiy effective_cpus
> +			 * so that cpus_updated will be set true in
> +			 * cpuset_hotplug_update_tasks().
> +			 */
> +			WRITE_ONCE(cs->prs_err, PERR_HOTPLUG);
> +			remote_partition_disable(cs, tmp);
> +			cpu = cpumask_first(cs->effective_cpus);
> +			spin_lock_irq(&callback_lock);
> +			if (cpu < nr_cpu_ids)
> +				cpumask_clear_cpu(cpu, cs->effective_cpus);
> +			else
> +				cpumask_set_cpu(0, cs->effective_cpus);
> +			spin_unlock_irq(&callback_lock);
> +		}
> +	}
> +	rcu_read_unlock();
> +}
> +
>  /**
>   * update_parent_effective_cpumask - update effective_cpus mask of parent cpuset
>   * @cs:      The cpuset that requests change in partition root state
> @@ -3919,6 +3986,7 @@ int __init cpuset_init(void)
>  	BUG_ON(!zalloc_cpumask_var(&subpartitions_cpus, GFP_KERNEL));
>  	BUG_ON(!zalloc_cpumask_var(&isolated_cpus, GFP_KERNEL));
>  	BUG_ON(!zalloc_cpumask_var(&isolated_hk_cpus, GFP_KERNEL));
> +	BUG_ON(!alloc_cpumask_var(&tmp_cpus, GFP_KERNEL));
>  
>  	cpumask_setall(top_cpuset.cpus_allowed);
>  	nodes_setall(top_cpuset.mems_allowed);
> @@ -3993,14 +4061,6 @@ static void cpuset_hotplug_update_tasks(struct cpuset *cs, struct tmpmasks *tmp)
>  	if (!tmp || !cs->partition_root_state)
>  		goto update_tasks;
>  
> -	if (is_remote_partition(cs) &&
> -	    (!cpumask_subset(cs->effective_xcpus, subpartitions_cpus) ||
> -	     (cpumask_empty(&new_cpus) && partition_is_populated(cs, NULL)))) {
> -		WRITE_ONCE(cs->prs_err, PERR_HOTPLUG);
> -		remote_partition_disable(cs, tmp);
> -		compute_effective_cpumask(&new_cpus, cs, parent);
> -	}
> -
>  	/*
>  	 * Force the partition to become invalid if either one of
>  	 * the following conditions hold:
> @@ -4104,22 +4164,31 @@ static void cpuset_handle_hotplug(void)
>  		if (!on_dfl)
>  			cpumask_copy(top_cpuset.cpus_allowed, &new_cpus);
>  		/*
> -		 * Make sure that CPUs allocated to child partitions
> -		 * do not show up in effective_cpus. If no CPU is left,
> -		 * we clear the subpartitions_cpus & let the child partitions
> -		 * fight for the CPUs again.
> +		 * Make sure that CPUs allocated to child partitions do not
> +		 * show up in effective_cpus. If no housekeeping CPU is left,
> +		 * we clear the subpartitions_cpus, scan the remote partitions
> +		 * to determine if they can remain valid & let the descendant
> +		 * local partitions fight for the CPUs again.
>  		 */
>  		if (!cpumask_empty(subpartitions_cpus)) {
> +			cpumask_andnot(&new_cpus, cpu_active_mask,
> +				       housekeeping_cpumask(HK_TYPE_DOMAIN_BOOT));
>  			if (cpumask_subset(&new_cpus, subpartitions_cpus)) {
>  				cpumask_clear(subpartitions_cpus);
> +				cpumask_copy(&new_cpus, cpu_active_mask);
>  			} else {
> -				cpumask_andnot(&new_cpus, &new_cpus,
> +				cpumask_andnot(&new_cpus, cpu_active_mask,
>  					       subpartitions_cpus);
>  			}
>  		}
>  		cpumask_copy(top_cpuset.effective_cpus, &new_cpus);
>  		spin_unlock_irq(&callback_lock);
> -		/* we don't mess with cpumasks of tasks in top_cpuset */
> +		/*
> +		 * We don't mess with cpumasks of tasks in top_cpuset, but we
> +		 * have to examine impact on remote partition roots if present.
> +		 */
> +		if (cpuset_v2() && ptmp)
> +			remote_partition_scan(ptmp);
>  	}
>  
>  	/* synchronize mems_allowed to N_MEMORY */
> diff --git a/tools/testing/selftests/cgroup/test_cpuset_prs.sh b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> index 81f0eecaca80..5c7b583cba18 100755
> --- a/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> +++ b/tools/testing/selftests/cgroup/test_cpuset_prs.sh
> @@ -378,7 +378,7 @@ TEST_MATRIX=(
>  	# Remote partition offline tests
>  	"   C0-3    C1-3  C2-3     .    X2-3   X2-3 X2-3:P2:O2=0 .   0 A1:0-1|A2:1|A3:3 A1:P0|A3:P2 2-3"
>  	"   C0-3    C1-3  C2-3     .    X2-3   X2-3 X2-3:P2:O2=0 O2=1 0 A1:0-1|A2:1|A3:2-3 A1:P0|A3:P2 2-3"
> -	"   C0-3    C1-3   C3      .    X2-3   X2-3    P2:O3=0   .   0 A1:0-2|A2:1-2|A3: A1:P0|A3:P2 3"
> +	"   C0-3    C1-3   C3      .    X2-3   X2-3    P2:O3=0   .   0 A1:0-2|A2:1-2|A3:1-2 A1:P0|A3:P-2 3|"
>  	"   C0-3    C1-3   C3      .    X2-3   X2-3   T:P2:O3=0  .   0 A1:0-2|A2:1-2|A3:1-2 A1:P0|A3:P-2 3|"
>  
>  	# An invalidated remote partition cannot self-recover from hotplug


  parent reply	other threads:[~2026-11-16 10:33 UTC|newest]

Thread overview: 18+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2026-11-16  1:23 [PATCH-next v4 0/6] cgroup/cpuset: More fix patches Waiman Long
2026-11-16  1:23 ` [PATCH-next v4 1/6] cgroup/cpuset: Fix incorrect partition invalidation logic in cpuset_hotplug_update_tasks() Waiman Long
2026-11-16 10:19   ` Guopeng Zhang
2026-11-16  1:23 ` [PATCH-next v4 2/6] cgroup/cpuset: excpus parameter of remote_cpus_update() must have at least 1 active CPU Waiman Long
2026-11-16  3:07   ` Ridong Chen
2026-11-16  3:34     ` Waiman Long
2026-11-16  7:37       ` Ridong Chen
2026-11-16  1:23 ` [PATCH-next v4 3/6] cgroup/cpuset: Check remote partition underneath before making an invalid partition valid Waiman Long
2026-11-16  3:47   ` Ridong Chen
2026-11-16  4:21     ` Waiman Long
2026-11-16  1:23 ` [PATCH-next v4 4/6] cgroup/cpuset: Revert "remove global remote_children list" Waiman Long
2026-11-16  1:23 ` [PATCH-next v4 5/6] cgroup/cpuset: Handle remote partitions before local partitions in cpuset_handle_hotplug() Waiman Long
2026-11-16  7:08   ` Ridong Chen
2026-11-16  8:56   ` kernel test robot
2026-11-16 10:33   ` Guopeng Zhang [this message]
2026-11-16  1:23 ` [PATCH-next v4 6/6] cgroup/cpuset: Fix lockdep_is_cpuset_held() Waiman Long
2026-11-16  7:24   ` Ridong Chen
2026-11-16 10:34   ` Guopeng Zhang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=70eb1395-ebcf-4ece-ac03-f0796cd02c81@linux.dev \
    --to=guopeng.zhang@linux.dev \
    --cc=cgroups@vger.kernel.org \
    --cc=hannes@cmpxchg.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-kselftest@vger.kernel.org \
    --cc=longman@redhat.com \
    --cc=mkoutny@suse.com \
    --cc=ridong.chen@linux.dev \
    --cc=sashiko-bot@kernel.org \
    --cc=shuah@kernel.org \
    --cc=tj@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).