From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 6DDA33451CD for ; Thu, 4 Jun 2026 15:03:14 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=170.10.129.124 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780585396; cv=none; b=g9QyHXtlfT9neSM6w01GtnfWoPhgOqPljeyAeUrSG7EBMpLB75ThAoGNhawNys7pAiFu0/VEP8wr4Ro2vR+3c9VkP1nIwOnlbN9xhQxLNr2YZCtJWMDOPrz7HQFi/LNjhxRWw+gQSC44OJf5FpxmwuBcvu+Xp0219SHgJ/+jtFc= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1780585396; c=relaxed/simple; bh=znMM3647Sh828ZACjredD0kc198ohO48pkMjQGu5wTU=; h=From:To:Cc:Subject:Date:Message-ID:In-Reply-To:References: MIME-Version; b=FqHTxtWbu4TSa7yGHKVkCd7fy8+1cG7e92xrz0tnCHlTaI7rN/rmFvKbDJViDQ1+jRkHJcD8XWUrVBEVVTbeltHt67E8wMongoRxT6iHQwcLNt9cU3NkzE/dILVTg44AEDYd1RbcccSBCwBBjcAc7Zx04tBtL/MYcrt8xdl0Tjc= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com; spf=pass smtp.mailfrom=redhat.com; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b=ecB2RweG; arc=none smtp.client-ip=170.10.129.124 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=redhat.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=redhat.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=redhat.com header.i=@redhat.com header.b="ecB2RweG" DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1780585393; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=NLkvco/hYViqdTfggj3+WAooj52GCaGs2AJxcM/3BHw=; b=ecB2RweGa9/voWfgrxEetyYkZO2ghiUxsYw30QoaDdVvA41bAQIe3g+DUx0QhaEWvJJJQ1 qDzJCFPckyso9efGDj4EI1ILmbIVWNFDBHQKVmbBC6TpXfpGp0qvD0FoX+Ab7PM2fDyxmA WqNk+OvDXdgQnXPZyxjLeJDBvfi0UkY= Received: from mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-391-FMUuwMKeOIuHGpEkMziPEw-1; Thu, 04 Jun 2026 11:03:08 -0400 X-MC-Unique: FMUuwMKeOIuHGpEkMziPEw-1 X-Mimecast-MFC-AGG-ID: FMUuwMKeOIuHGpEkMziPEw_1780585386 Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-01.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id A88C91956089; Thu, 4 Jun 2026 15:03:06 +0000 (UTC) Received: from llong-thinkpadp16vgen1.westford.csb (unknown [10.22.88.175]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id F37EA195608E; Thu, 4 Jun 2026 15:03:04 +0000 (UTC) From: Waiman Long To: Ridong Chen , Tejun Heo , Johannes Weiner , =?UTF-8?q?Michal=20Koutn=C3=BD?= , Peter Zijlstra Cc: cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Aaron Tomlin , Guopeng Zhang , Waiman Long Subject: [PATCH-next v6 5/6] cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside cpuset_attach_task() Date: Thu, 4 Jun 2026 11:02:28 -0400 Message-ID: <20260604150229.414135-6-longman@redhat.com> In-Reply-To: <20260604150229.414135-1-longman@redhat.com> References: <20260604150229.414135-1-longman@redhat.com> Precedence: bulk X-Mailing-List: cgroups@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 The cpuset_attach_task() was introduced in commit 42a11bf5c543 ("cgroup/cpuset: Make cpuset_fork() handle CLONE_INTO_CGROUP properly") to enable the CLONE_INTO_CGROUP flag of clone(2) to behave more like moving a task from one cpuset into another one. That commits didn't move the mpol_rebind_mm() and cpuset_migrate_mm() calls for group leader into cpuset_attach_task(). When the CLONE_INTO_CGROUP flag is used without CLONE_THREAD, the new task is its own group leader. So it is still not equivalent to moving task between cpusets in this case. Make CLONE_INTO_CGROUP behaves more close to cpuset_attach() by moving the mpol_rebind_mm() and cpuset_migrate_mm() calls inside cpuset_attach_task(). As a result, the following static variables will have to be updated in cpuset_fork(). - cpuset_attach_old_cs - attach_cpus_updated - attach_mems_updated - queue_task_work Signed-off-by: Waiman Long --- kernel/cgroup/cpuset.c | 103 ++++++++++++++++++++++++----------------- 1 file changed, 60 insertions(+), 43 deletions(-) diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c index e29129467c98..1142d5eba58d 100644 --- a/kernel/cgroup/cpuset.c +++ b/kernel/cgroup/cpuset.c @@ -2981,8 +2981,13 @@ static int update_prstate(struct cpuset *cs, int new_prs) /* * cpuset_can_attach() and cpuset_attach() specific internal data * Protected by cpuset_mutex + * + * The attach_cpus_updated/attach_mems_updated flags are set in either + * cpuset_attach() or cpuset_fork() and used in cpuset_attach_task(). */ static struct cpuset *cpuset_attach_old_cs; +static bool attach_cpus_updated; +static bool attach_mems_updated; /* * Check to see if a cpuset can accept a new task @@ -3160,9 +3165,12 @@ static void cpuset_cancel_attach(struct cgroup_taskset *tset) */ static cpumask_var_t cpus_attach; static nodemask_t cpuset_attach_nodemask_to; +static bool queue_task_work; static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task) { + struct mm_struct *mm; + lockdep_assert_cpuset_lock_held(); if (cs != &top_cpuset) @@ -3176,28 +3184,60 @@ static void cpuset_attach_task(struct cpuset *cs, struct task_struct *task) */ WARN_ON_ONCE(set_cpus_allowed_ptr(task, cpus_attach)); + if (cpuset_v2() && !attach_mems_updated) + return; + cpuset_change_task_nodemask(task, &cpuset_attach_nodemask_to); cpuset1_update_task_spread_flags(cs, task); + + if ((task != task->group_leader) || + (!is_memory_migrate(cs) && !attach_mems_updated)) + return; + + /* + * Change mm for threadgroup leader. This is expensive and may + * sleep and should be moved outside migration path proper. + */ + mm = get_task_mm(task); + if (mm) { + struct cpuset *oldcs = cpuset_attach_old_cs; + + mpol_rebind_mm(mm, &cs->effective_mems); + + /* + * old_mems_allowed is the same with mems_allowed + * here, except if this task is being moved + * automatically due to hotplug. In that case + * @mems_allowed has been updated and is empty, so + * @old_mems_allowed is the right nodesets that we + * migrate mm from. + */ + if (is_memory_migrate(cs)) { + cpuset_migrate_mm(mm, &oldcs->old_mems_allowed, + &cpuset_attach_nodemask_to); + queue_task_work = true; + } else { + mmput(mm); + } + } } static void cpuset_attach(struct cgroup_taskset *tset) { struct task_struct *task; - struct task_struct *leader; struct cgroup_subsys_state *css; struct cpuset *cs; struct cpuset *oldcs = cpuset_attach_old_cs; - bool cpus_updated, mems_updated; - bool queue_task_work = false; cgroup_taskset_first(tset, &css); cs = css_cs(css); lockdep_assert_cpus_held(); /* see cgroup_attach_lock() */ mutex_lock(&cpuset_mutex); - cpus_updated = !cpumask_equal(cs->effective_cpus, - oldcs->effective_cpus); - mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems); + queue_task_work = false; + + attach_cpus_updated = !cpumask_equal(cs->effective_cpus, oldcs->effective_cpus); + attach_mems_updated = !nodes_equal(cs->effective_mems, oldcs->effective_mems); /* * In the default hierarchy, enabling cpuset in the child cgroups @@ -3207,7 +3247,7 @@ static void cpuset_attach(struct cgroup_taskset *tset) */ if (cpuset_v2()) { cpuset_attach_nodemask_to = cs->effective_mems; - if (!cpus_updated && !mems_updated) + if (!attach_cpus_updated && !attach_mems_updated) goto out; } else { guarantee_online_mems(cs, &cpuset_attach_nodemask_to); @@ -3216,38 +3256,6 @@ static void cpuset_attach(struct cgroup_taskset *tset) cgroup_taskset_for_each(task, css, tset) cpuset_attach_task(cs, task); - /* - * Change mm for all threadgroup leaders. This is expensive and may - * sleep and should be moved outside migration path proper. Skip it - * if there is no change in effective_mems and CS_MEMORY_MIGRATE is - * not set. - */ - if (!is_memory_migrate(cs) && !mems_updated) - goto out; - - cgroup_taskset_for_each_leader(leader, css, tset) { - struct mm_struct *mm = get_task_mm(leader); - - if (mm) { - mpol_rebind_mm(mm, &cs->effective_mems); - - /* - * old_mems_allowed is the same with mems_allowed - * here, except if this task is being moved - * automatically due to hotplug. In that case - * @mems_allowed has been updated and is empty, so - * @old_mems_allowed is the right nodesets that we - * migrate mm from. - */ - if (is_memory_migrate(cs)) { - cpuset_migrate_mm(mm, &oldcs->old_mems_allowed, - &cpuset_attach_nodemask_to); - queue_task_work = true; - } else - mmput(mm); - } - } - out: if (queue_task_work) schedule_flush_migrate_mm(); @@ -3681,15 +3689,14 @@ static void cpuset_cancel_fork(struct task_struct *task, struct css_set *cset) */ static void cpuset_fork(struct task_struct *task) { - struct cpuset *cs; - bool same_cs; + struct cpuset *cs, *oldcs; rcu_read_lock(); cs = task_cs(task); - same_cs = (cs == task_cs(current)); + oldcs = task_cs(current); rcu_read_unlock(); - if (same_cs) { + if (cs == oldcs) { if (cs == &top_cpuset) return; @@ -3701,7 +3708,17 @@ static void cpuset_fork(struct task_struct *task) /* CLONE_INTO_CGROUP */ mutex_lock(&cpuset_mutex); guarantee_online_mems(cs, &cpuset_attach_nodemask_to); + /* + * Assume CPUs and memory nodes are updated + * A CLONE_INTO_CGROUP operation should have taken the cgroup mutex + * and so there shouldn't be a competing cpuset_attach() operation. + */ + attach_cpus_updated = attach_mems_updated = true; + queue_task_work = false; + cpuset_attach_old_cs = oldcs; cpuset_attach_task(cs, task); + if (queue_task_work) + schedule_flush_migrate_mm(); dec_attach_in_progress_locked(cs); mutex_unlock(&cpuset_mutex); -- 2.54.0