* Re: [PATCH 2/5] bfq: protect q->blkg_list iteration in bfq_end_wr_async() with blkcg_mutex
From: Nilay Shroff @ 2026-06-04 17:31 UTC (permalink / raw)
To: Yu Kuai, Jens Axboe
Cc: Tejun Heo, Josef Bacik, Ming Lei, Bart Van Assche, linux-block,
cgroups, linux-kernel
In-Reply-To: <89f9448c5d703e6123e1be6c8e0550c803e9c057.1780492756.git.yukuai@fygo.io>
On 6/3/26 6:57 PM, Yu Kuai wrote:
> bfq_end_wr_async() iterates q->blkg_list while only holding bfqd->lock,
> but not blkcg_mutex. This can race with blkg_free_workfn() that removes
> blkgs from the list while holding blkcg_mutex.
>
> Add blkcg_mutex protection in bfq_end_wr() before taking bfqd->lock to
> ensure proper synchronization when iterating q->blkg_list.
>
> Signed-off-by: Yu Kuai <yukuai@fygo.io>
> ---
> block/bfq-cgroup.c | 3 ++-
> block/bfq-iosched.c | 6 ++++++
> 2 files changed, 8 insertions(+), 1 deletion(-)
>
> diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c
> index 37ab70930c8d..f765e767d36a 100644
> --- a/block/bfq-cgroup.c
> +++ b/block/bfq-cgroup.c
> @@ -939,11 +939,12 @@ void bfq_end_wr_async(struct bfq_data *bfqd)
> struct blkcg_gq *blkg;
>
> list_for_each_entry(blkg, &bfqd->queue->blkg_list, q_node) {
> struct bfq_group *bfqg = blkg_to_bfqg(blkg);
>
> - bfq_end_wr_async_queues(bfqd, bfqg);
> + if (bfqg)
> + bfq_end_wr_async_queues(bfqd, bfqg);
> }
> bfq_end_wr_async_queues(bfqd, bfqd->root_group);
> }
>
> static int bfq_io_show_weight_legacy(struct seq_file *sf, void *v)
> diff --git a/block/bfq-iosched.c b/block/bfq-iosched.c
> index 141c602d5e85..42ccfd0c6140 100644
> --- a/block/bfq-iosched.c
> +++ b/block/bfq-iosched.c
> @@ -2643,10 +2643,13 @@ void bfq_end_wr_async_queues(struct bfq_data *bfqd,
> static void bfq_end_wr(struct bfq_data *bfqd)
> {
> struct bfq_queue *bfqq;
> int i;
>
> +#ifdef CONFIG_BFQ_GROUP_IOSCHED
> + mutex_lock(&bfqd->queue->blkcg_mutex);
> +#endif
> spin_lock_irq(&bfqd->lock);
>
> for (i = 0; i < bfqd->num_actuators; i++) {
> list_for_each_entry(bfqq, &bfqd->active_list[i], bfqq_list)
> bfq_bfqq_end_wr(bfqq);
> @@ -2654,10 +2657,13 @@ static void bfq_end_wr(struct bfq_data *bfqd)
> list_for_each_entry(bfqq, &bfqd->idle_list, bfqq_list)
> bfq_bfqq_end_wr(bfqq);
> bfq_end_wr_async(bfqd);
>
> spin_unlock_irq(&bfqd->lock);
> +#ifdef CONFIG_BFQ_GROUP_IOSCHED
> + mutex_unlock(&bfqd->queue->blkcg_mutex);
> +#endif
> }
The above change protects the q->blkg_list iteration in bfq_end_wr_async()
against list removal in blkg_free_workfn(). However the blkg insertion in
blkg_create() still doesn't use q->blkcg_mutex and so list traversal in
bfq_end_wr_async() may still race with blkg_create().
So I think we may also need to protect blkg insert in blkg_create() using
q->blkcg_mutex.
Thanks,
--Nilay
^ permalink raw reply
* Re: [GIT PULL] cgroup: Fixes for v7.1-rc6
From: pr-tracker-bot @ 2026-06-04 19:31 UTC (permalink / raw)
To: Tejun Heo
Cc: Linus Torvalds, Johannes Weiner, Michal Koutny, Waiman Long,
cgroups, linux-kernel
In-Reply-To: <547622bc7e85cf6c2c3a2f95bf146f5e@kernel.org>
The pull request you sent on Tue, 02 Jun 2026 12:47:53 -1000:
> https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git tags/cgroup-for-7.1-rc6-fixes
has been merged into torvalds/linux.git:
https://git.kernel.org/torvalds/c/e7524845cda3c0713c0e61681dcd5263f0270fbe
Thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/prtracker.html
^ permalink raw reply
* [PATCH-next v6 7/6] cgroup/cpuset: Set old_mems_allowed from guarantee_online_mems() consistently
From: Waiman Long @ 2026-06-05 0:02 UTC (permalink / raw)
To: Ridong Chen, Tejun Heo, Johannes Weiner, Michal Koutný,
Peter Zijlstra
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang, Waiman Long
In-Reply-To: <20260604150229.414135-1-longman@redhat.com>
An earlier patch has added an optimization in guarantee_online_mems()
to just return effective_mems for v2. However there is a short window
during memory hotunplug operation that it can return a nodemask with
no online node leading to possible memory OOM. To avoid this scenario,
though highly unlikely, the optimization is dropped.
Also set old_mems_allowed of the cpuset structure consistently with
the output of guarantee_online_mems() whenever an attach or a related
operation is in progress.
Signed-off-by: Waiman Long <longman@redhat.com>
---
kernel/cgroup/cpuset.c | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
index d624cd0a1e04..5dabe9d040e9 100644
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -502,10 +502,6 @@ static void guarantee_active_cpus(struct task_struct *tsk,
*/
static void guarantee_online_mems(struct cpuset *cs, nodemask_t *pmask)
{
- if (cpuset_v2()) {
- *pmask = cs->effective_mems;
- return;
- }
while (!nodes_and(*pmask, cs->effective_mems, node_states[N_MEMORY]))
cs = parent_cs(cs);
}
@@ -3350,22 +3346,24 @@ static void cpuset_attach(struct cgroup_taskset *tset)
}
cs = css_cs(css);
+ guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+
/*
* In the default hierarchy, enabling cpuset in the child cgroups
* will trigger a cpuset_attach() call with no change in effective cpus
* and mems. In that case, we can optimize out by skipping the task
* iteration and update, but the destination cpuset list is iterated to
- * set old_mems_sllowed.
+ * set old_mems_allowed.
*/
- if (cpuset_v2()) {
- cpuset_attach_nodemask_to = cs->effective_mems;
- if (!attach_cpus_updated && !attach_mems_updated) {
- llist_for_each_entry(cs, dst_cs_head.first, attach_node)
- cs->old_mems_allowed = cs->effective_mems;
- goto out;
+ if (cpuset_v2() && !attach_cpus_updated && !attach_mems_updated) {
+ struct cpuset *tcs;
+
+ llist_for_each_entry(tcs, dst_cs_head.first, attach_node) {
+ if (tcs == cs)
+ continue;
+ guarantee_online_mems(tcs, &tcs->old_mems_allowed);
}
- } else {
- guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+ goto out;
}
cgroup_taskset_for_each(task, css, tset) {
@@ -3381,8 +3379,8 @@ static void cpuset_attach(struct cgroup_taskset *tset)
if (queue_task_work)
schedule_flush_migrate_mm();
- cs->old_mems_allowed = cpuset_attach_nodemask_to;
out:
+ cs->old_mems_allowed = cpuset_attach_nodemask_to;
reset_attach_in_progress();
clear_attach_data(false);
mutex_unlock(&cpuset_mutex);
@@ -3824,6 +3822,8 @@ static void cpuset_fork(struct task_struct *task)
/* CLONE_INTO_CGROUP */
mutex_lock(&cpuset_mutex);
guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
+ cs->old_mems_allowed = cpuset_attach_nodemask_to;
+
/*
* Assume CPUs and memory nodes are updated
* A CLONE_INTO_CGROUP operation should have taken the cgroup mutex
--
2.54.0
^ permalink raw reply related
* Re: [PATCH] cgroup/cpuset: Support multiple source/destination cpusets using pids pattern
From: Ridong Chen @ 2026-06-05 7:35 UTC (permalink / raw)
To: Waiman Long; +Cc: cgroups, Tejun Heo, Johannes Weiner, linux-kernel
In-Reply-To: <07bfe9cc-b8ab-4c4c-bfe0-b974abd3ff08@redhat.com>
On 6/4/2026 2:47 AM, Waiman Long wrote:
> On 6/3/26 6:26 AM, Ridong Chen wrote:
>> The current cpuset_can_attach() and cpuset_attach() functions assume task
>> migration is from one source cpuset to one destination cpuset. This
>> can be
>> wrong in several scenarios:
>> - Moving a multi-threaded process with threads in different cpusets
>> - Disabling the cpuset controller (many children to one parent)
>> - Enabling the cpuset controller (one parent to many children)
>>
>> Fix this by adopting the pids subsystem's per-task accounting pattern.
>> In cpuset_can_attach(), use task_cs(task) to get the correct source
>> cpuset
>> for each task (like pids_can_attach uses task_css), adjust
>> nr_deadline_tasks
>> and reserve DL bandwidth per-task, and increment attach_in_progress
>> per-task
>> on the destination cpuset. In cpuset_attach(), handle destination cpuset
>> changes within the task iteration loop.
>>
>> A shared helper cpuset_undo_attach() reverses the per-task operations for
>> both partial rollback in cpuset_can_attach() and full reversal in
>> cpuset_cancel_attach().
>>
>> When multiple source cpusets are detected in can_attach(), set
>> attach_many_sources so that cpuset_attach() forces cpus_updated and
>> mems_updated to true, ensuring all tasks get properly updated regardless
>> of which source cpuset cpuset_attach_old_cs points to.
>>
>> This eliminates the need for nr_migrate_dl_tasks, sum_migrate_dl_bw, and
>> dl_bw_cpu fields in struct cpuset.
>>
>> Fixes: 4ec22e9c5a90 ("cpuset: Enable cpuset controller in default
>> hierarchy")
>> Signed-off-by: Ridong Chen <ridong.chen@linux.dev>
>
> It is not a problem doing per-task DL BW allocation and eliminating the
> *dl_bw* fields. However, updating nr_deadline_tasks before it is
> committed can be problematic.
>
Good to hear that.
> nr_deadline_tasks is used in dl_rebuild_rd_accounting() which is called
> by partition_sched_domains_locked(). After the release of cpuset_mutex
> at the end of cpuset_can_attach() and before cpuset_attach() or
> cpuset_cancel_attach() is called, it is possible
> that partition_sched_domains_locked() can be called
> and dl_rebuild_rd_accounting() is not getting the right DL BW accounting
> information. So unless there is a way to confirm that this situation
> cannot happen, we can't change nr_deadline_tasks before the attach is
> commited.
>
We can keep the nr_migrate_dl_tasks field and update nr_deadline_tasks
once migration is complete. I think this will be much simpler than
fixing the issue using lists.
--
Best regards,
Ridong
^ permalink raw reply
* Re: [PATCH-next v6 1/6] cgroup/cpuset: Fix node inconsistencies between cpuset_update_tasks_nodemask() and cpuset_attach()
From: Ridong Chen @ 2026-06-05 7:48 UTC (permalink / raw)
To: Waiman Long, Tejun Heo, Johannes Weiner, Michal Koutný,
Peter Zijlstra
Cc: cgroups, linux-kernel, Aaron Tomlin, Guopeng Zhang
In-Reply-To: <20260604150229.414135-2-longman@redhat.com>
On 6/4/2026 11:02 PM, Waiman Long wrote:
> Whenever memory node mask is changed, there are 4 places where the node
> mask has to be updated or used.
> 1) task's node mask via cpuset_change_task_nodemask()
> 2) memory policy binding via mpol_rebind_mm()
> 3) if memory migration is enabled, migrate from old_mems_allowed to
> the new node mask via cpuset_migrate_mm().
> 4) setting old_mems_allowed
>
> These memory actions are done in cpuset_update_tasks_nodemask() and
> cpuset_attach(). However there are inconsistencies in what node masks
> are being used in these 2 functions.
>
> In cpuset_update_tasks_nodemask(),
> - cpuset_change_task_nodemask(): guarantee_online_mems()
> - mpol_rebind_mm(): mems_allowed
> - cpuset_migrate_mm(): guarantee_online_mems()
> - old_mems_allowed: guarantee_online_mems()
>
> In cpuset_attach(),
> - cpuset_change_task_nodemask(): guarantee_online_mems()
> - mpol_rebind_mm(): effective_mems
> - cpuset_migrate_mm(): effective_mems
> - old_mems_allowed: effective_mems
>
> These inconsistencies dates back to quite a long time ago and it is
> hard to say what should be the correct values.
>
> The guarantee_online_mems() function returns a node mask from current or
> an ancestor cpuset that is a subset of node_states[N_MEMORY]. Nodes in
> node_states[N_MEMORY] are all online, i.e. in node_states[N_ONLINE].
> However, node in node_states[N_ONLINE] may not have memory. So
> node_states[N_MEMORY] should be a subset of node_states[N_ONLINE].
>
> The guarantee_online_mems() function should only be useful for v1 where
> mems_allowed is the same as effective_mems. With v2, the memory nodes
> in effective_mems should always be a subset of node_states[N_MEMORY].
> The only time that may not be true is when a memory hot-unplug operation
> is in progress and a memory node is removed from node_states[N_MEMORY]
> but not yet reflected in effective_mems as cpuset_handle_hotplug()
> has not yet been called from cpuset_track_online_nodes(). When
> cpuset_handle_hotplug() is called later, the memory node setting
> of the relevant cpusets and tasks will be updated. So replacing the
> guarantee_online_mems() call by just using cs->effective_mems should
> be fine.
>
The message should be updated.
> Let use the following setup for both of them and make them consistent.
> - cpuset_change_task_nodemask(): guarantee_online_mems()
> - mpol_rebind_mm(): effective_mems
> - cpuset_migrate_mm(): guarantee_online_mems()
> - old_mems_allowed: guarantee_online_mems()
>
> So for v2, it is effectively all effective_mems. For v1, mpol_rebind_mm()
> uses mems_allowed which may differ from what guarantee_online_mems()
> returns.
>
> Signed-off-by: Waiman Long <longman@redhat.com>
> ---
> kernel/cgroup/cpuset.c | 37 +++++++++++++++++++++++++------------
> 1 file changed, 25 insertions(+), 12 deletions(-)
>
> diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c
> index 6bdb68689c24..8305b5830c3c 100644
> --- a/kernel/cgroup/cpuset.c
> +++ b/kernel/cgroup/cpuset.c
> @@ -489,7 +489,10 @@ static void guarantee_active_cpus(struct task_struct *tsk,
> * Return in *pmask the portion of a cpusets's mems_allowed that
> * are online, with memory. If none are online with memory, walk
> * up the cpuset hierarchy until we find one that does have some
> - * online mems. The top cpuset always has some mems online.
> + * online mems. The top cpuset always has some mems online. With v2,
> + * effective_mems should always contain online memory nodes except
> + * during the transition period where a memory node hotunplug operation
> + * is in progress.
> *
> * One way or another, we guarantee to return some non-empty subset
> * of node_states[N_MEMORY].
> @@ -498,6 +501,10 @@ static void guarantee_active_cpus(struct task_struct *tsk,
> */
> static void guarantee_online_mems(struct cpuset *cs, nodemask_t *pmask)
> {
> + if (cpuset_v2()) {
> + *pmask = cs->effective_mems;
> + return;
> + }
> while (!nodes_and(*pmask, cs->effective_mems, node_states[N_MEMORY]))
> cs = parent_cs(cs);
> }
> @@ -2616,6 +2623,13 @@ static void *cpuset_being_rebound;
> * Iterate through each task of @cs updating its mems_allowed to the
> * effective cpuset's. As this function is called with cpuset_mutex held,
> * cpuset membership stays stable.
> + *
> + * - cpuset_change_task_nodemask(): guarantee_online_mems()
> + * - mpol_rebind_mm(): effective_mems
> + * - cpuset_migrate_mm(): guarantee_online_mems()
> + * - old_mems_allowed: guarantee_online_mems()
> + *
> + * For v2, guarantee_online_mems() should just return effective_mems.
> */
> void cpuset_update_tasks_nodemask(struct cpuset *cs)
> {
> @@ -2624,7 +2638,6 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)
> struct task_struct *task;
>
> cpuset_being_rebound = cs; /* causes mpol_dup() rebind */
> -
> guarantee_online_mems(cs, &newmems);
>
> /*
> @@ -2650,7 +2663,7 @@ void cpuset_update_tasks_nodemask(struct cpuset *cs)
>
> migrate = is_memory_migrate(cs);
>
> - mpol_rebind_mm(mm, &cs->mems_allowed);
> + mpol_rebind_mm(mm, &cs->effective_mems);
> if (migrate)
> cpuset_migrate_mm(mm, &cs->old_mems_allowed, &newmems);
> else
> @@ -3148,17 +3161,18 @@ static void cpuset_attach(struct cgroup_taskset *tset)
>
> /*
> * In the default hierarchy, enabling cpuset in the child cgroups
> - * will trigger a number of cpuset_attach() calls with no change
> - * in effective cpus and mems. In that case, we can optimize out
> - * by skipping the task iteration and update.
> + * will trigger a cpuset_attach() call with no change in effective cpus
> + * and mems. In that case, we can optimize out by skipping the task
> + * iteration and update.
> */
> - if (cpuset_v2() && !cpus_updated && !mems_updated) {
> + if (cpuset_v2()) {
> cpuset_attach_nodemask_to = cs->effective_mems;
> - goto out;
> + if (!cpus_updated && !mems_updated)
> + goto out;
> + } else {
> + guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
> }
>
> - guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
> -
Nit.
I prefer:
guarantee_online_mems(cs, &cpuset_attach_nodemask_to);
if (cpuset_v2() && !cpus_updated && !mems_updated)
goto out;
> cgroup_taskset_for_each(task, css, tset)
> cpuset_attach_task(cs, task);
>
> @@ -3168,7 +3182,6 @@ static void cpuset_attach(struct cgroup_taskset *tset)
> * if there is no change in effective_mems and CS_MEMORY_MIGRATE is
> * not set.
> */
> - cpuset_attach_nodemask_to = cs->effective_mems;
> if (!is_memory_migrate(cs) && !mems_updated)
> goto out;
>
> @@ -3176,7 +3189,7 @@ static void cpuset_attach(struct cgroup_taskset *tset)
> struct mm_struct *mm = get_task_mm(leader);
>
> if (mm) {
> - mpol_rebind_mm(mm, &cpuset_attach_nodemask_to);
> + mpol_rebind_mm(mm, &cs->effective_mems);
>
> /*
> * old_mems_allowed is the same with mems_allowed
Other than that, looks good to me.
Reviewed-by: Ridong Chen <ridong.chen@linux.dev>
--
Best regards,
Ridong
^ permalink raw reply
* Re: [PATCH v3] cgroup/dmem: introduce a peak file
From: Maarten Lankhorst @ 2026-06-05 11:13 UTC (permalink / raw)
To: Michal Koutný
Cc: Thadeu Lima de Souza Cascardo, Tejun Heo, Johannes Weiner,
Michal Hocko, Roman Gushchin, Shakeel Butt, Muchun Song,
Andrew Morton, Jonathan Corbet, Shuah Khan, Maxime Ripard,
Natalie Vock, Tvrtko Ursulin, cgroups, linux-kernel, linux-mm,
linux-doc, dri-devel, kernel-dev
In-Reply-To: <ahmOBo02TA8u8RW2@localhost.localdomain>
Hey,
On 5/29/26 15:01, Michal Koutný wrote:
> On Fri, May 29, 2026 at 09:34:28AM +0200, Maarten Lankhorst <dev@lankhorst.se> wrote:
>>> Reviewed-by: Michal Koutný <mkoutny@suse.com>
>> Reviewed-by: Maarten Lankhorst <dev@lankhorst.se>
>>
>> With your r-b it's ok to push it to the dmemcg tree?
>
> Please go for it.
>
> Michal
Thanks, pushed and queued for v7.3!
^ permalink raw reply
* Re: [PATCH v2 2/2] cgroup/dmem: add dmem.memcg control file for double-charging to memcg
From: Maarten Lankhorst @ 2026-06-05 11:27 UTC (permalink / raw)
To: Eric Chanudet, Michal Koutný
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, Maxime Ripard, Natalie Vock,
Tejun Heo, Jonathan Corbet, Shuah Khan, cgroups, linux-mm,
linux-kernel, dri-devel, T.J. Mercier, Christian König,
Maxime Ripard, Albert Esteve, Dave Airlie, linux-doc
In-Reply-To: <ahXKFYBdCMDBvc_N@x1nano>
Hey,
On 5/26/26 18:59, Eric Chanudet wrote:
> On Fri, May 22, 2026 at 05:26:16PM +0200, Michal Koutný wrote:
>> Hello Eric.
>>
>> On Tue, May 19, 2026 at 11:59:02AM -0400, Eric Chanudet <echanude@redhat.com> wrote:
>>> Add a root-only cgroupfs file "dmem.memcg" that lets an administrator
>>> configure whether allocations in a dmem region should also be charged to
>>> the memory controller.
>>
>> This kinda makes sense as it is not unlike io.cost.* device
>> configurators.
>>
>> Just for my better understanding -- will there be a space for userspace
>> to switch this? (No charged dmem allocations happen before responsible
>> userspace runs, so that the attribute remains unlocked.)
>>
>> (I'm rather indifferent about the actual double charging/non-charging
>> matter.)
>
> Yes, this is intended to be configured before the user space stack that
> would start allocating things is started. Once it has started (and tried
> to charge something), the configuration is locked
>
>>
>>>
>>> To handle inheritance, dmem adds a depends_on the memory controller,
>>> unless MEMCG isn't configured in.
>>>
>>> Double-charging is disabled by default. Once a charge is attempted, the
>>> setting is locked to prevent inconsistent accounting by a small 4-state
>>> machine (off, on, locked off, locked on).
>>>
>>> The memcg to charge is derived from the pool's cgroup, since the pool
>>> holds a reference to the dmem cgroup state that keeps the cgroup alive
>>> until it gets uncharged.
>>>
>>> Signed-off-by: Eric Chanudet <echanude@redhat.com>
>>> ---
>>> Documentation/admin-guide/cgroup-v2.rst | 23 +++++
>>> kernel/cgroup/dmem.c | 158 +++++++++++++++++++++++++++++++-
>>> 2 files changed, 178 insertions(+), 3 deletions(-)
>>>
>>> diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
>>> index 6efd0095ed995b1550317662bc1b56c7a7f3db23..1d2fa55ddf0faa17baa916a8914d3033e8e42359 100644
>>> --- a/Documentation/admin-guide/cgroup-v2.rst
>>> +++ b/Documentation/admin-guide/cgroup-v2.rst
>>> @@ -2828,6 +2828,29 @@ DMEM Interface Files
>>> drm/0000:03:00.0/vram0 12550144
>>> drm/0000:03:00.0/stolen 8650752
>>>
>>> + dmem.memcg
>>> + A readwrite nested-keyed file that exists only on the root
>>> + cgroup.
>>
>> Strictly speaking this is not nested-keyed but flat keyed [1],
>
> Indeed,
>
>> which leads me to realization that this is the first instance of a boolean.
>> All in call, such a composition comes to my mind (latter is RO):
>>
>> drm/0000:03:00.0/vram0 enable=0|1 locked=0|1
>>
>
> So per[1] 1 key, 2 sub-keys (enable RW, locked RO), that looks better
> and match the documentation, thanks!
>
>>
>>
>>> +static ssize_t dmem_cgroup_memcg_write(struct kernfs_open_file *of, char *buf,
>>> + size_t nbytes, loff_t off)
>>> +{
>>> + while (buf) {
>>> + struct dmem_cgroup_region *region;
>>> + char *options, *name;
>>> + bool flag;
>>> +
>>> + options = buf;
>>> + buf = strchr(buf, '\n');
>>> + if (buf)
>>> + *buf++ = '\0';
>>
>> I recall there was a discussion about accepting only a single device per
>> write(2) (at the same time I see this idiom is still present in other
>> dmem.* files, so this is nothing to change in _this_ patch).
>
> I would second that. When setting say dmem.max for 2 regions, with a
> typo on the second, the first one is set, but write still get EINVAL.
>
> Also, I just notice dmemcg_limit_write() returns EINVAL if the region is
> not found (this patch returns ENODEV).
>
>>
>> Thanks,
>> Michal
>>
>> [1] https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#format
>
>
>
Perhaps a bit late, but before we start adding this UAPI we should enforce a
single region per write?
Kind regards,
~Maarten Lankhorst
^ permalink raw reply
* [PATCH v3 1/7] sched/fair: Add cgroup_mode switch
From: Peter Zijlstra @ 2026-06-05 12:40 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260605105513.354837583@infradead.org>
The effective task weight (W_t') for a task in cgroup g on CPU n is given by:
W_t
W_t' = W_g * F_g_n * ----------
\Sum W_t_n
Where W_g is the group's weight (cpu.weight), F_g_n is the fraction of the
group weight for CPU n and W_t/W is the relative weight of this task against
all other tasks in the same group on the same CPU.
Furthermore, this makes:
\Sum W_t_n
F_g_n = ----------
\Sum W_t
The fraction of weight inside the group of CPU n against the whole group.
The problem is with F_g_n, the primary goal of this fraction is to make sure
that the relative weight of tasks, when distributed over CPUs is maintained.
For example, consider 4 (equal weight) tasks and 2 CPUs with a 1:3
distribution, then if F_g_n would simply be 1 (no weight re-distribution) the
effective relative weights (W_t') of the tasks in our group would be:
CPU0 CPU1
W_g W_g/3
W_g/3
W_g/3
IOW, the lucky task on CPU0 would get an equal amount of weight as all 3 tasks
on CPU1 combined. However, with the weight redistribution, this becomes:
CPU0 CPU1
W_g/4 W_g/4
W_g/4
W_g/4
All tasks are equal weight (as intended). However, as is already evident from
this example, the more CPUs you add, the smaller F_g_n becomes, which creates a
disparity against tasks not in our group.
Specifically:
avg(F_g_n) ~ 1/N
This leads to a weight mismatch in the hierarchy. IOW tasks cannot compete
fairly across hierarchy levels.
*Notably*, what is meant by avg(F_g_n) being proportional to 1/N is that when
there are at least N runnable tasks, the average of this fraction tends to 1/N.
For a hierarchy of depth d, this gets even worse, since that gets terms on the
order of:
avg(F_g_n)^d ~ 1/(N^d)
Given fixed point arithmetic, this also leads to numerical trouble.
However, the meaning of "cpu.weight" is simple and intiutive: the total weight
of the cgroup. But as explored above, there is deception in this simplicity.
Prepare to add a few alternative methods for distributing weight.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/debug.c | 74 +++++++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 74 insertions(+)
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -633,6 +633,76 @@ static void debugfs_fair_server_init(voi
}
}
+#ifdef CONFIG_FAIR_GROUP_SCHED
+static int cgroup_mode = 0;
+
+static const char *cgroup_mode_str[] = {
+ "smp",
+};
+
+static int sched_cgroup_mode(const char *str)
+{
+ for (int i = 0; i < ARRAY_SIZE(cgroup_mode_str); i++) {
+ if (!strcmp(str, cgroup_mode_str[i]))
+ return i;
+ }
+ return -EINVAL;
+}
+
+static ssize_t sched_cgroup_write(struct file *filp, const char __user *ubuf,
+ size_t cnt, loff_t *ppos)
+{
+ char buf[16];
+ int mode;
+
+ if (cnt > 15)
+ cnt = 15;
+
+ if (copy_from_user(buf, ubuf, cnt))
+ return -EFAULT;
+
+ buf[cnt] = 0;
+ mode = sched_cgroup_mode(strstrip(buf));
+ if (mode < 0)
+ return mode;
+
+ WRITE_ONCE(cgroup_mode, mode);
+
+ *ppos += cnt;
+ return cnt;
+}
+
+static int sched_cgroup_show(struct seq_file *m, void *v)
+{
+ int mode = READ_ONCE(cgroup_mode);
+
+ for (int i = 0; i < ARRAY_SIZE(cgroup_mode_str); i++) {
+ if (mode == i)
+ seq_puts(m, "(");
+ seq_puts(m, cgroup_mode_str[i]);
+ if (mode == i)
+ seq_puts(m, ")");
+
+ seq_puts(m, " ");
+ }
+ seq_puts(m, "\n");
+ return 0;
+}
+
+static int sched_cgroup_open(struct inode *inode, struct file *filp)
+{
+ return single_open(filp, sched_cgroup_show, NULL);
+}
+
+static const struct file_operations sched_cgroup_fops = {
+ .open = sched_cgroup_open,
+ .write = sched_cgroup_write,
+ .read = seq_read,
+ .llseek = seq_lseek,
+ .release = single_release,
+};
+#endif
+
static __init int sched_init_debug(void)
{
struct dentry __maybe_unused *numa, *llc;
@@ -686,6 +756,10 @@ static __init int sched_init_debug(void)
debugfs_create_file("debug", 0444, debugfs_sched, NULL, &sched_debug_fops);
+#ifdef CONFIG_FAIR_GROUP_SCHED
+ debugfs_create_file("cgroup_mode", 0644, debugfs_sched, NULL, &sched_cgroup_fops);
+#endif
+
debugfs_fair_server_init();
#ifdef CONFIG_SCHED_CLASS_EXT
debugfs_ext_server_init();
^ permalink raw reply
* [PATCH v3 0/7] sched: Flatten the pick
From: Peter Zijlstra @ 2026-06-05 12:40 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
Hi!
New version, same story [1]. TL;DR:
- Adds new cgroup_mode knob and implements new policies to address the
hierarchy level weight mismatch.
- Builds upon that base to create a flat / single runqueue scheduler where the
cgroup hierarchy is expressed through dynamic weight management.
I'm hoping to be able to merge these patches early in the next cycle (after
7.2-rc1).
Random benchmark:
Game vs 'for ((i=0; i<8; i++)) do nice ./spin.sh; done':
Lutris / GE-Proton10-34 / Steam Runtime 3 (sniper)
Intel Core i7-2600K
AMD Radeon RX 580
Shadows Awakening (GOG)
default slice(*)
FPS min 4.0 29.0
avg 47.5 59.2
max 83.7 83.7
FT min 9.3 10.2
avg 34.0 17.0
max 121.2 30.0
FPS (Frames Per Second)
FT (FrameTime)
[*] Command prefix: 'chrt -o --sched-runtime 100000 0'
Changes since v2:
- merged debug and prep patches
- fixed update_entity_lag() on dequeue (Vincent)
- fixed throttle vs tick (Prateek)
- fixed wakeup_preempt_fair()
- rebased on tip/sched/core
- rewritten cgroup_mode changelogs
- reworked cgroup_mode concur
- added cgroup_mode tasks
- changed default cgroup_mode
[1] - https://lore.kernel.org/r/20260511113104.563854162@infradead.org
Can also be had:
git://git.kernel.org/pub/scm/linux/kernel/git/peterz/queue.git sched/flat
include/linux/cpuset.h | 6
include/linux/sched.h | 1
kernel/cgroup/cpuset.c | 15
kernel/sched/core.c | 5
kernel/sched/debug.c | 89 ++++
kernel/sched/fair.c | 943 ++++++++++++++++++++++++-------------------------
kernel/sched/pelt.c | 6
kernel/sched/sched.h | 30 -
8 files changed, 607 insertions(+), 488 deletions(-)
^ permalink raw reply
* [PATCH v3 5/7] sched/fair: Add cgroup_mode: tasks
From: Peter Zijlstra @ 2026-06-05 12:40 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260605105513.354837583@infradead.org>
Since we are exploring this space; include a scheme that scales by total number
of runnable tasks. This results in:
F_g_n' = M * F_g_n
This will obviously have: avg(F_g_n') > 1, (it will be ~M/N in fact).
And while that sounds odd, it actually has a fairly straight foward meaning for
"cpu.weight": average weight per member task.
This is an entirely valid and workable option, it is however wildly different
from the traditional meaning.
Included for completeness (and curiosity).
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/debug.c | 1 +
kernel/sched/fair.c | 16 ++++++++++++++++
2 files changed, 17 insertions(+)
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -643,6 +643,7 @@ static const char *cgroup_mode_str[] = {
"smp",
"concur",
"max",
+ "tasks",
};
static int sched_cgroup_mode(const char *str)
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4852,6 +4852,19 @@ static inline int tg_tasks(struct task_g
}
/*
+ * Func: fraction(nr_tasks * tg->shares)
+ *
+ * Scale tg->shares by the number of tasks.
+ */
+static long calc_tasks_shares(struct cfs_rq *cfs_rq)
+{
+ struct task_group *tg = cfs_rq->tg;
+ int nr = tg_tasks(tg);
+ long tg_shares = READ_ONCE(tg->shares);
+ return __calc_smp_shares(cfs_rq, nr * tg_shares, nr * tg_shares);
+}
+
+/*
* Func: min(fraction(nr_cpus * tg->shares), nice -20)
*
* Scale tg->shares by the maximal number of CPUs; but clip the max shares at
@@ -4921,6 +4934,9 @@ void __sched_cgroup_mode_update(int mode
case 3:
func = &calc_max_shares;
break;
+ case 4:
+ func = &calc_tasks_shares;
+ break;
}
static_call_update(calc_group_shares, func);
}
^ permalink raw reply
* [PATCH v3 3/7] sched/fair: Add cgroup_mode: max
From: Peter Zijlstra @ 2026-06-05 12:40 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260605105513.354837583@infradead.org>
In order to avoid the average CPU fraction avg(F_g_n) becoming tiny '1/N',
assume each cgroup is maximally concurrent and distrubute 'N*weight', such
that:
F_g_n' = N * F_g_n
Giving:
avg(F_g_n') = N*avg(F_g_n) ~ N * 1/N = 1
And while this sounds like it solves things, remember what that ~ meant. There
is the corner case when a cgroup is minimally loaded, eg a single runnable
task, therefore limit the CPU fraction to that of a nice -20 task to avoid
getting too much load.
This last bit is what makes it different from a previous proposal to allow
raising cpu.weight to '100 * N', that would not limit the mininal concurrency
case and results in a very large F_g_n. And just like F_g_n << 1 is
problematic, so is F_g_n >> 1 for the exact same reasons (it would drown the
kthreads, but it also risks overflowing the load values).
So while this might appear to be a better scheme than the current default
scheme, it doesn't really handle less than maximal concurrency nicely -- it
clips and introduces artificially large weights. So where the traditional SMP
mode works well when nr_tasks << nr_cpus, MAX doesn't work well in that regime
and vice-versa.
The meaning of "cpu.weight" would be: weight per allowed CPU.
Included for completeness (and infrastructure).
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
include/linux/cpuset.h | 6 +++++
kernel/cgroup/cpuset.c | 15 ++++++++++++++
kernel/sched/debug.c | 1
kernel/sched/fair.c | 52 ++++++++++++++++++++++++++++++++++++++++++++-----
4 files changed, 69 insertions(+), 5 deletions(-)
--- a/include/linux/cpuset.h
+++ b/include/linux/cpuset.h
@@ -80,6 +80,7 @@ extern void lockdep_assert_cpuset_lock_h
extern void cpuset_cpus_allowed_locked(struct task_struct *p, struct cpumask *mask);
extern void cpuset_cpus_allowed(struct task_struct *p, struct cpumask *mask);
extern bool cpuset_cpus_allowed_fallback(struct task_struct *p);
+extern int cpuset_num_cpus(struct cgroup *cgroup);
extern nodemask_t cpuset_mems_allowed(struct task_struct *p);
#define cpuset_current_mems_allowed (current->mems_allowed)
void cpuset_init_current_mems_allowed(void);
@@ -216,6 +217,11 @@ static inline bool cpuset_cpus_allowed_f
return false;
}
+static inline int cpuset_num_cpus(struct cgroup *cgroup)
+{
+ return num_online_cpus();
+}
+
static inline nodemask_t cpuset_mems_allowed(struct task_struct *p)
{
return node_possible_map;
--- a/kernel/cgroup/cpuset.c
+++ b/kernel/cgroup/cpuset.c
@@ -4116,6 +4116,21 @@ bool cpuset_cpus_allowed_fallback(struct
return changed;
}
+int cpuset_num_cpus(struct cgroup *cgrp)
+{
+ int nr = num_online_cpus();
+ struct cpuset *cs;
+
+ if (is_in_v2_mode()) {
+ guard(rcu)();
+ cs = css_cs(cgroup_e_css(cgrp, &cpuset_cgrp_subsys));
+ if (cs)
+ nr = cpumask_weight(cs->effective_cpus);
+ }
+
+ return nr;
+}
+
void __init cpuset_init_current_mems_allowed(void)
{
nodes_setall(current->mems_allowed);
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -641,6 +641,7 @@ static int cgroup_mode = 1;
static const char *cgroup_mode_str[] = {
"up",
"smp",
+ "max",
};
static int sched_cgroup_mode(const char *str)
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4801,12 +4801,10 @@ static inline int throttled_hierarchy(st
*
* hence icky!
*/
-static long calc_smp(struct cfs_rq *cfs_rq)
+static long __calc_smp_shares(struct cfs_rq *cfs_rq, long tg_shares, long shares_max)
{
- long tg_weight, tg_shares, load, shares;
struct task_group *tg = cfs_rq->tg;
-
- tg_shares = READ_ONCE(tg->shares);
+ long tg_weight, load, shares;
load = max(scale_load_down(cfs_rq->load.weight), cfs_rq->avg.load_avg);
@@ -4832,7 +4830,48 @@ static long calc_smp(struct cfs_rq *cfs_
* case no task is runnable on a CPU MIN_SHARES=2 should be returned
* instead of 0.
*/
- return clamp_t(long, shares, MIN_SHARES, tg_shares);
+ return clamp_t(long, shares, MIN_SHARES, shares_max);
+}
+
+static int tg_cpus(struct task_group *tg)
+{
+ int nr = num_online_cpus();
+
+ if (cpusets_enabled()) {
+ struct cgroup *cgrp = tg->css.cgroup;
+ if (cgrp)
+ nr = cpuset_num_cpus(cgrp);
+ }
+
+ return nr;
+}
+
+/*
+ * Func: min(fraction(nr_cpus * tg->shares), nice -20)
+ *
+ * Scale tg->shares by the maximal number of CPUs; but clip the max shares at
+ * nice -20, otherwise a single spinner on a 512 CPU machine would result in
+ * 512*NICE_0_LOAD, which is also crazy.
+ */
+static long calc_max_shares(struct cfs_rq *cfs_rq)
+{
+ struct task_group *tg = cfs_rq->tg;
+ int nr = tg_cpus(tg);
+ long tg_shares = READ_ONCE(tg->shares);
+ long max_shares = scale_load(sched_prio_to_weight[0]);
+ return __calc_smp_shares(cfs_rq, tg_shares * nr, max_shares);
+}
+
+/*
+ * Func: fraction(tg->shares)
+ *
+ * This infamously results in tiny shares when you have many CPUs.
+ */
+static long calc_smp_shares(struct cfs_rq *cfs_rq)
+{
+ struct task_group *tg = cfs_rq->tg;
+ long tg_shares = READ_ONCE(tg->shares);
+ return __calc_smp_shares(cfs_rq, tg_shares, tg_shares);
}
/*
@@ -4857,6 +4896,9 @@ void __sched_cgroup_mode_update(int mode
default:
func = &calc_smp_shares;
break;
+ case 2:
+ func = &calc_max_shares;
+ break;
}
static_call_update(calc_group_shares, func);
}
^ permalink raw reply
* [PATCH v3 7/7] sched/eevdf: Move to a single runqueue
From: Peter Zijlstra @ 2026-06-05 12:40 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260605105513.354837583@infradead.org>
Change fair/cgroup to a single runqueue.
Infamously fair/cgroup isn't working for a number of people; typically
the complaint is latencies and/or overhead. The latency issue is due
to the intermediate entries that represent a combination of tasks and
thereby obfuscate the runnability of tasks.
The approach here is to leave the cgroup hierarchy as is; including
the intermediate enqueue/dequeue but move the actual EEVDF runqueue
outside. This means things like the shares_weight approximation are
fully preserved.
That is, given a hierarchy like:
R
|
se--G1
/ \
G2--se se--G3
/ \ |
T1--se se--T2 se--T3
This is fully maintained for load tracking, however the EEVDF parts of
cfs_rq/se go unused for the intermediates and are instead connected
like:
_R_
/ | \
T1 T2 T3
Since the effective weight of the entities is determined by the
hierarchy, this gets recomputed on enqueue,set_next_task and tick.
Notably, the effective weight (se->h_load) is computed from the
hierarchical fraction: se->load / cfs_rq->load.
Since EEVDF is now exclusively operating on rq->cfs, it needs to
consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly,
only tasks can get delayed, simplifying some of the cgroup cleanup.
One place where additional information was required was
set_next_task() / put_prev_task(), where we need to track 'current'
both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense
(cfs_rq->curr).
As a result of only having a single level to pick from, much of the
complications in pick_next_task() and preemption go away.
Since many of the hierarchical operations are still there, this won't
immediately fix the performance issues, but hopefully it will fix some
of the latency issues.
TODO: split struct cfs_rq / struct sched_entity
TODO: try and get rid of h_curr
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
include/linux/sched.h | 1
kernel/sched/core.c | 5
kernel/sched/debug.c | 9
kernel/sched/fair.c | 803 +++++++++++++++++++++-----------------------------
kernel/sched/pelt.c | 6
kernel/sched/sched.h | 26 -
6 files changed, 375 insertions(+), 475 deletions(-)
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -574,6 +574,7 @@ struct sched_statistics {
struct sched_entity {
/* For load-balancing: */
struct load_weight load;
+ struct load_weight h_load;
struct rb_node run_node;
u64 deadline;
u64 min_vruntime;
--- a/kernel/sched/core.c
+++ b/kernel/sched/core.c
@@ -5652,11 +5652,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);
*/
static inline void prefetch_curr_exec_start(struct task_struct *p)
{
-#ifdef CONFIG_FAIR_GROUP_SCHED
- struct sched_entity *curr = p->se.cfs_rq->curr;
-#else
struct sched_entity *curr = task_rq(p)->cfs.curr;
-#endif
+
prefetch(curr);
prefetch(&curr->exec_start);
}
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -975,10 +975,11 @@ print_task(struct seq_file *m, struct rq
else
SEQ_printf(m, " %c", task_state_to_char(p));
- SEQ_printf(m, " %15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ",
+ SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ",
p->comm, task_pid_nr(p),
+ p->se.h_load.weight,
SPLIT_NS(p->se.vruntime),
- entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N',
+ entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N',
SPLIT_NS(p->se.deadline),
p->se.custom_slice ? 'S' : ' ',
SPLIT_NS(p->se.slice),
@@ -1007,7 +1008,7 @@ static void print_rq(struct seq_file *m,
SEQ_printf(m, "\n");
SEQ_printf(m, "runnable tasks:\n");
- SEQ_printf(m, " S task PID vruntime eligible "
+ SEQ_printf(m, " S task PID weight vruntime eligible "
"deadline slice sum-exec switches "
"prio wait-time sum-sleep sum-block"
#ifdef CONFIG_NUMA_BALANCING
@@ -1115,6 +1116,8 @@ void print_cfs_rq(struct seq_file *m, in
cfs_rq->tg_load_avg_contrib);
SEQ_printf(m, " .%-30s: %ld\n", "tg_load_avg",
atomic_long_read(&cfs_rq->tg->load_avg));
+ SEQ_printf(m, " .%-30s: %lu\n", "h_load",
+ cfs_rq->h_load);
#endif /* CONFIG_FAIR_GROUP_SCHED */
#ifdef CONFIG_CFS_BANDWIDTH
SEQ_printf(m, " .%-30s: %d\n", "throttled",
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -297,8 +297,8 @@ static u64 __calc_delta(u64 delta_exec,
*/
static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)
{
- if (unlikely(se->load.weight != NICE_0_LOAD))
- delta = __calc_delta(delta, NICE_0_LOAD, &se->load);
+ if (se->h_load.weight != NICE_0_LOAD)
+ delta = __calc_delta(delta, NICE_0_LOAD, &se->h_load);
return delta;
}
@@ -428,38 +428,6 @@ static inline struct sched_entity *paren
return se->parent;
}
-static void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
- int se_depth, pse_depth;
-
- /*
- * preemption test can be made between sibling entities who are in the
- * same cfs_rq i.e who have a common parent. Walk up the hierarchy of
- * both tasks until we find their ancestors who are siblings of common
- * parent.
- */
-
- /* First walk up until both entities are at same depth */
- se_depth = (*se)->depth;
- pse_depth = (*pse)->depth;
-
- while (se_depth > pse_depth) {
- se_depth--;
- *se = parent_entity(*se);
- }
-
- while (pse_depth > se_depth) {
- pse_depth--;
- *pse = parent_entity(*pse);
- }
-
- while (!is_same_group(*se, *pse)) {
- *se = parent_entity(*se);
- *pse = parent_entity(*pse);
- }
-}
-
static int tg_is_idle(struct task_group *tg)
{
return tg->idle > 0;
@@ -503,11 +471,6 @@ static inline struct sched_entity *paren
return NULL;
}
-static inline void
-find_matching_se(struct sched_entity **se, struct sched_entity **pse)
-{
-}
-
static inline int tg_is_idle(struct task_group *tg)
{
return 0;
@@ -686,7 +649,7 @@ static inline unsigned long avg_vruntime
static inline void
__sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
- unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+ unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
s64 w_vruntime, key = entity_key(cfs_rq, se);
w_vruntime = key * weight;
@@ -703,7 +666,7 @@ sum_w_vruntime_add_paranoid(struct cfs_r
s64 key, tmp;
again:
- weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+ weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
key = entity_key(cfs_rq, se);
if (check_mul_overflow(key, weight, &key))
@@ -749,7 +712,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rq
static void
sum_w_vruntime_sub(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
- unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+ unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
s64 key = entity_key(cfs_rq, se);
cfs_rq->sum_w_vruntime -= key * weight;
@@ -791,7 +754,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq)
s64 runtime = cfs_rq->sum_w_vruntime;
if (curr) {
- unsigned long w = avg_vruntime_weight(cfs_rq, curr->load.weight);
+ unsigned long w = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
runtime += entity_key(cfs_rq, curr) * w;
weight += w;
@@ -862,8 +825,6 @@ bool update_entity_lag(struct cfs_rq *cf
u64 avruntime = avg_vruntime(cfs_rq);
s64 vlag = entity_lag(cfs_rq, se, avruntime);
- WARN_ON_ONCE(!se->on_rq);
-
if (se->sched_delayed) {
/* previous vlag < 0 otherwise se would not be delayed */
vlag = max(vlag, se->vlag);
@@ -899,7 +860,7 @@ static int vruntime_eligible(struct cfs_
long load = cfs_rq->sum_weight;
if (curr && curr->on_rq) {
- unsigned long weight = avg_vruntime_weight(cfs_rq, curr->load.weight);
+ unsigned long weight = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
avg += entity_key(cfs_rq, curr) * weight;
load += weight;
@@ -1040,6 +1001,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntim
*/
static void __enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
+ WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+ WARN_ON_ONCE(!entity_is_task(se));
+
sum_w_vruntime_add(cfs_rq, se);
se->min_vruntime = se->vruntime;
se->min_slice = se->slice;
@@ -1049,6 +1013,9 @@ static void __enqueue_entity(struct cfs_
static void __dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
+ WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
+ WARN_ON_ONCE(!entity_is_task(se));
+
rb_erase_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline,
&min_vruntime_cb);
sum_w_vruntime_sub(cfs_rq, se);
@@ -1145,7 +1112,7 @@ static struct sched_entity *pick_eevdf(s
* We can safely skip eligibility check if there is only one entity
* in this cfs_rq, saving some cycles.
*/
- if (cfs_rq->nr_queued == 1)
+ if (cfs_rq->h_nr_queued == 1)
return curr && curr->on_rq ? curr : se;
/*
@@ -1395,8 +1362,6 @@ static s64 update_se(struct rq *rq, stru
return delta_exec;
}
-static void set_next_buddy(struct sched_entity *se);
-
#ifdef CONFIG_SCHED_CACHE
/*
@@ -1991,7 +1956,7 @@ static void update_curr(struct cfs_rq *c
* not necessarily be the actual task running
* (rq->curr.se). This is easy to confuse!
*/
- struct sched_entity *curr = cfs_rq->curr;
+ struct sched_entity *curr = cfs_rq->h_curr;
struct rq *rq = rq_of(cfs_rq);
s64 delta_exec;
bool resched;
@@ -2003,26 +1968,29 @@ static void update_curr(struct cfs_rq *c
if (unlikely(delta_exec <= 0))
return;
+ account_cfs_rq_runtime(cfs_rq, delta_exec);
+
+ if (!entity_is_task(curr))
+ return;
+
+ cfs_rq = &rq->cfs;
+
curr->vruntime += calc_delta_fair(delta_exec, curr);
resched = update_deadline(cfs_rq, curr);
- if (entity_is_task(curr)) {
- /*
- * If the fair_server is active, we need to account for the
- * fair_server time whether or not the task is running on
- * behalf of fair_server or not:
- * - If the task is running on behalf of fair_server, we need
- * to limit its time based on the assigned runtime.
- * - Fair task that runs outside of fair_server should account
- * against fair_server such that it can account for this time
- * and possibly avoid running this period.
- */
- dl_server_update(&rq->fair_server, delta_exec);
- }
-
- account_cfs_rq_runtime(cfs_rq, delta_exec);
+ /*
+ * If the fair_server is active, we need to account for the
+ * fair_server time whether or not the task is running on
+ * behalf of fair_server or not:
+ * - If the task is running on behalf of fair_server, we need
+ * to limit its time based on the assigned runtime.
+ * - Fair task that runs outside of fair_server should account
+ * against fair_server such that it can account for this time
+ * and possibly avoid running this period.
+ */
+ dl_server_update(&rq->fair_server, delta_exec);
- if (cfs_rq->nr_queued == 1)
+ if (cfs_rq->h_nr_queued == 1)
return;
if (resched || !protect_slice(curr)) {
@@ -2033,7 +2001,10 @@ static void update_curr(struct cfs_rq *c
static void update_curr_fair(struct rq *rq)
{
- update_curr(cfs_rq_of(&rq->donor->se));
+ struct sched_entity *se = &rq->donor->se;
+
+ for_each_sched_entity(se)
+ update_curr(cfs_rq_of(se));
}
static inline void
@@ -2109,7 +2080,7 @@ update_stats_enqueue_fair(struct cfs_rq
* Are we enqueueing a waiting task? (for current tasks
* a dequeue/enqueue event is a NOP)
*/
- if (se != cfs_rq->curr)
+ if (se != cfs_rq->h_curr)
update_stats_wait_start_fair(cfs_rq, se);
if (flags & ENQUEUE_WAKEUP)
@@ -2127,7 +2098,7 @@ update_stats_dequeue_fair(struct cfs_rq
* Mark the end of the wait period if dequeueing a
* waiting task:
*/
- if (se != cfs_rq->curr)
+ if (se != cfs_rq->h_curr)
update_stats_wait_end_fair(cfs_rq, se);
if ((flags & DEQUEUE_SLEEP) && entity_is_task(se)) {
@@ -4468,6 +4439,7 @@ static inline void update_scan_period(st
static void
account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
+ WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
update_load_add(&cfs_rq->load, se->load.weight);
if (entity_is_task(se)) {
struct task_struct *p = task_of(se);
@@ -4483,6 +4455,7 @@ account_entity_enqueue(struct cfs_rq *cf
static void
account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
+ WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
update_load_sub(&cfs_rq->load, se->load.weight);
if (entity_is_task(se)) {
struct task_struct *p = task_of(se);
@@ -4564,7 +4537,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq,
static void
rescale_entity(struct sched_entity *se, unsigned long weight, bool rel_vprot)
{
- unsigned long old_weight = se->load.weight;
+ long old_weight = se->h_load.weight;
/*
* VRUNTIME
@@ -4664,16 +4637,17 @@ rescale_entity(struct sched_entity *se,
se->vprot = div64_long(se->vprot * old_weight, weight);
}
-static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
- unsigned long weight)
+static void reweight_eevdf(struct cfs_rq *cfs_rq, struct sched_entity *se,
+ unsigned long weight, bool on_rq)
{
bool curr = cfs_rq->curr == se;
bool rel_vprot = false;
u64 avruntime = 0;
- if (se->on_rq) {
- /* commit outstanding execution time */
- update_curr(cfs_rq);
+ if (se->h_load.weight == weight)
+ return;
+
+ if (on_rq) {
avruntime = avg_vruntime(cfs_rq);
se->vlag = entity_lag(cfs_rq, se, avruntime);
se->deadline -= avruntime;
@@ -4683,46 +4657,90 @@ static void reweight_entity(struct cfs_r
rel_vprot = true;
}
- cfs_rq->nr_queued--;
+ cfs_rq->h_nr_queued--;
if (!curr)
__dequeue_entity(cfs_rq, se);
- update_load_sub(&cfs_rq->load, se->load.weight);
}
- dequeue_load_avg(cfs_rq, se);
rescale_entity(se, weight, rel_vprot);
- update_load_set(&se->load, weight);
+ update_load_set(&se->h_load, weight);
- do {
- u32 divider = get_pelt_divider(&se->avg);
- se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
- } while (0);
-
- enqueue_load_avg(cfs_rq, se);
- if (se->on_rq) {
+ if (on_rq) {
if (rel_vprot)
se->vprot += avruntime;
se->deadline += avruntime;
se->rel_deadline = 0;
se->vruntime = avruntime - se->vlag;
- update_load_add(&cfs_rq->load, se->load.weight);
if (!curr)
__enqueue_entity(cfs_rq, se);
- cfs_rq->nr_queued++;
+ cfs_rq->h_nr_queued++;
}
}
+static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
+ unsigned long weight)
+{
+ if (se->load.weight == weight)
+ return;
+
+ if (se->on_rq) {
+ WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
+ update_load_sub(&cfs_rq->load, se->load.weight);
+ }
+ dequeue_load_avg(cfs_rq, se);
+
+ update_load_set(&se->load, weight);
+
+ do {
+ u32 divider = get_pelt_divider(&se->avg);
+ se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
+ } while (0);
+
+ enqueue_load_avg(cfs_rq, se);
+
+ if (se->on_rq)
+ update_load_add(&cfs_rq->load, se->load.weight);
+}
+
+/*
+ * weight = NICE_0_LOAD;
+ * for_each_entity_se(se)
+ * weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
+ */
+static __always_inline
+unsigned long __calc_prop_weight(struct cfs_rq *cfs_rq, struct sched_entity *se,
+ unsigned long weight)
+{
+ weight *= se->load.weight;
+ if (parent_entity(se))
+ weight /= cfs_rq->load.weight;
+ else
+ weight /= NICE_0_LOAD;
+
+ return max(weight, MIN_SHARES);
+}
+
static void reweight_task_fair(struct rq *rq, struct task_struct *p,
const struct load_weight *lw)
{
struct sched_entity *se = &p->se;
- struct cfs_rq *cfs_rq = cfs_rq_of(se);
- struct load_weight *load = &se->load;
+ unsigned long weight = NICE_0_LOAD;
- reweight_entity(cfs_rq, se, lw->weight);
- load->inv_weight = lw->inv_weight;
+ if (se->on_rq)
+ update_curr_fair(rq);
+
+ reweight_entity(cfs_rq_of(se), se, lw->weight);
+ se->load.inv_weight = lw->inv_weight;
+
+ if (!se->on_rq)
+ return;
+
+ for_each_sched_entity(se)
+ weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
+
+ reweight_eevdf(&rq->cfs, &p->se, weight, p->se.on_rq);
}
static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);
@@ -4958,8 +4976,7 @@ static void update_cfs_group(struct sche
return;
shares = static_call(calc_group_shares)(gcfs_rq);
- if (unlikely(se->load.weight != shares))
- reweight_entity(cfs_rq_of(se), se, shares);
+ reweight_entity(cfs_rq_of(se), se, shares);
}
#else /* !CONFIG_FAIR_GROUP_SCHED: */
@@ -5077,7 +5094,7 @@ static inline bool cfs_rq_is_decayed(str
* differential update where we store the last value we propagated. This in
* turn allows skipping updates if the differential is 'small'.
*
- * Updating tg's load_avg is necessary before update_cfs_share().
+ * Updating tg's load_avg is necessary before update_cfs_group().
*/
static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
{
@@ -5544,7 +5561,7 @@ static void migrate_se_pelt_lag(struct s
* The cfs_rq avg is the direct sum of all its entities (blocked and runnable)
* avg. The immediate corollary is that all (fair) tasks must be attached.
*
- * cfs_rq->avg is used for task_h_load() and update_cfs_share() for example.
+ * cfs_rq->avg is used for task_h_load() and update_cfs_group() for example.
*
* Return: true if the load decayed or we removed load.
*
@@ -6082,6 +6099,7 @@ static void
place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
{
u64 vslice, vruntime = avg_vruntime(cfs_rq);
+ unsigned int nr_queued = cfs_rq->h_nr_queued;
bool update_zero = false;
s64 lag = 0;
@@ -6089,6 +6107,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
se->slice = sysctl_sched_base_slice;
vslice = calc_delta_fair(se->slice, se);
+ if (flags & ENQUEUE_QUEUED)
+ nr_queued -= 1;
+
/*
* Due to how V is constructed as the weighted average of entities,
* adding tasks with positive lag, or removing tasks with negative lag
@@ -6097,7 +6118,7 @@ place_entity(struct cfs_rq *cfs_rq, stru
*
* EEVDF: placement strategy #1 / #2
*/
- if (sched_feat(PLACE_LAG) && cfs_rq->nr_queued && se->vlag) {
+ if (sched_feat(PLACE_LAG) && nr_queued && se->vlag) {
struct sched_entity *curr = cfs_rq->curr;
long load, weight;
@@ -6157,9 +6178,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
*/
load = cfs_rq->sum_weight;
if (curr && curr->on_rq)
- load += avg_vruntime_weight(cfs_rq, curr->load.weight);
+ load += avg_vruntime_weight(cfs_rq, curr->h_load.weight);
- weight = avg_vruntime_weight(cfs_rq, se->load.weight);
+ weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
lag *= load + weight;
if (WARN_ON_ONCE(!load))
load = 1;
@@ -6218,22 +6239,8 @@ static void check_enqueue_throttle(struc
static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
static void
-requeue_delayed_entity(struct sched_entity *se);
-
-static void
enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
{
- bool curr = cfs_rq->curr == se;
-
- /*
- * If we're the current task, we must renormalise before calling
- * update_curr().
- */
- if (curr)
- place_entity(cfs_rq, se, flags);
-
- update_curr(cfs_rq);
-
/*
* When enqueuing a sched_entity, we must:
* - Update loads to have both entity and cfs_rq synced with now.
@@ -6252,13 +6259,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
*/
update_cfs_group(se);
- /*
- * XXX now that the entity has been re-weighted, and it's lag adjusted,
- * we can place the entity.
- */
- if (!curr)
- place_entity(cfs_rq, se, flags);
-
account_entity_enqueue(cfs_rq, se);
/* Entity has migrated, no longer consider this task hot */
@@ -6267,8 +6267,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
check_schedstat_required();
update_stats_enqueue_fair(cfs_rq, se, flags);
- if (!curr)
- __enqueue_entity(cfs_rq, se);
se->on_rq = 1;
if (cfs_rq->nr_queued == 1) {
@@ -6286,21 +6284,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
}
}
-static void __clear_buddies_next(struct sched_entity *se)
+static void set_next_buddy(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
- for_each_sched_entity(se) {
- struct cfs_rq *cfs_rq = cfs_rq_of(se);
- if (cfs_rq->next != se)
- break;
-
- cfs_rq->next = NULL;
- }
+ if (WARN_ON_ONCE(!se->on_rq || se->sched_delayed))
+ return;
+ if (se_is_idle(se))
+ return;
+ cfs_rq->next = se;
}
static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
if (cfs_rq->next == se)
- __clear_buddies_next(se);
+ cfs_rq->next = NULL;
}
static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
@@ -6311,7 +6307,7 @@ static void set_delayed(struct sched_ent
/*
* Delayed se of cfs_rq have no tasks queued on them.
- * Do not adjust h_nr_runnable since dequeue_entities()
+ * Do not adjust h_nr_runnable since __dequeue_task()
* will account it for blocked tasks.
*/
if (!entity_is_task(se))
@@ -6344,45 +6340,16 @@ static void clear_delayed(struct sched_e
}
}
-static bool
+static void
dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
{
- bool sleep = flags & DEQUEUE_SLEEP;
- int action = 0;
-
- update_curr(cfs_rq);
- clear_buddies(cfs_rq, se);
-
- if (flags & DEQUEUE_DELAYED) {
- WARN_ON_ONCE(!se->sched_delayed);
- } else {
- bool delay = sleep;
- /*
- * DELAY_DEQUEUE relies on spurious wakeups, special task
- * states must not suffer spurious wakeups, excempt them.
- */
- if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
- delay = false;
-
- WARN_ON_ONCE(delay && se->sched_delayed);
+ int action = UPDATE_TG;
- if (sched_feat(DELAY_DEQUEUE) && delay &&
- !entity_eligible(cfs_rq, se)) {
- if (entity_is_task(se))
- action |= UPDATE_UTIL_EST;
- update_load_avg(cfs_rq, se, action);
- update_entity_lag(cfs_rq, se);
- set_delayed(se);
- return false;
- }
- }
-
- action = UPDATE_TG;
if (entity_is_task(se)) {
if (task_on_rq_migrating(task_of(se)))
action |= DO_DETACH;
- if (sleep && !(flags & DEQUEUE_DELAYED))
+ if ((flags & DEQUEUE_SLEEP) && !(flags & DEQUEUE_DELAYED))
action |= UPDATE_UTIL_EST;
}
@@ -6400,14 +6367,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
update_stats_dequeue_fair(cfs_rq, se, flags);
- update_entity_lag(cfs_rq, se);
- if (sched_feat(PLACE_REL_DEADLINE) && !sleep) {
- se->deadline -= se->vruntime;
- se->rel_deadline = 1;
- }
-
- if (se != cfs_rq->curr)
- __dequeue_entity(cfs_rq, se);
se->on_rq = 0;
account_entity_dequeue(cfs_rq, se);
@@ -6416,9 +6375,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
update_cfs_group(se);
- if (flags & DEQUEUE_DELAYED)
- clear_delayed(se);
-
if (cfs_rq->nr_queued == 0) {
update_idle_cfs_rq_clock_pelt(cfs_rq);
#ifdef CONFIG_CFS_BANDWIDTH
@@ -6431,15 +6387,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
}
#endif
}
-
- return true;
}
static void
-set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool first)
+set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
- clear_buddies(cfs_rq, se);
-
/* 'current' is not kept within the tree. */
if (se->on_rq) {
/*
@@ -6448,16 +6400,12 @@ set_next_entity(struct cfs_rq *cfs_rq, s
* runqueue.
*/
update_stats_wait_end_fair(cfs_rq, se);
- __dequeue_entity(cfs_rq, se);
update_load_avg(cfs_rq, se, UPDATE_TG);
-
- if (first)
- set_protect_slice(cfs_rq, se);
}
update_stats_curr_start(cfs_rq, se);
- WARN_ON_ONCE(cfs_rq->curr);
- cfs_rq->curr = se;
+ WARN_ON_ONCE(cfs_rq->h_curr);
+ cfs_rq->h_curr = se;
/*
* Track our maximum slice length, if the CPU's load is at
@@ -6477,23 +6425,17 @@ set_next_entity(struct cfs_rq *cfs_rq, s
se->prev_sum_exec_runtime = se->sum_exec_runtime;
}
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags);
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags);
-/*
- * Pick the next process, keeping these things in mind, in this order:
- * 1) keep things fair between processes/task groups
- * 2) pick the "next" process, since someone really wants that to run
- * 3) pick the "last" process, for cache locality
- * 4) do not run the "skip" process, if something else is available
- */
static struct sched_entity *
-pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect)
+pick_next_entity(struct rq *rq, bool protect)
{
+ struct cfs_rq *cfs_rq = &rq->cfs;
struct sched_entity *se;
se = pick_eevdf(cfs_rq, protect);
if (se->sched_delayed) {
- dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
+ __dequeue_task(rq, task_of(se), DEQUEUE_SLEEP | DEQUEUE_DELAYED);
/*
* Must not reference @se again, see __block_task().
*/
@@ -6513,13 +6455,11 @@ static void put_prev_entity(struct cfs_r
if (prev->on_rq) {
update_stats_wait_start_fair(cfs_rq, prev);
- /* Put 'current' back into the tree. */
- __enqueue_entity(cfs_rq, prev);
/* in !on_rq case, update occurred at dequeue */
update_load_avg(cfs_rq, prev, 0);
}
- WARN_ON_ONCE(cfs_rq->curr != prev);
- cfs_rq->curr = NULL;
+ WARN_ON_ONCE(cfs_rq->h_curr != prev);
+ cfs_rq->h_curr = NULL;
}
static void
@@ -7074,7 +7014,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cf
assert_list_leaf_cfs_rq(rq);
/* Determine whether we need to wake up potentially idle CPU: */
- if (rq->curr == rq->idle && rq->cfs.nr_queued)
+ if (rq->curr == rq->idle && rq->cfs.h_nr_queued)
resched_curr(rq);
}
@@ -7409,7 +7349,7 @@ static void check_enqueue_throttle(struc
return;
/* an active group must be handled by the update_curr() path */
- if (!cfs_rq->runtime_enabled || cfs_rq->curr)
+ if (!cfs_rq->runtime_enabled || cfs_rq->h_curr)
return;
/* ensure the group is not already throttled */
@@ -7781,7 +7721,7 @@ static void hrtick_start_fair(struct rq
resched_curr(rq);
return;
}
- delta = (se->load.weight * vdelta) / NICE_0_LOAD;
+ delta = (se->h_load.weight * vdelta) / NICE_0_LOAD;
/*
* Correct for instantaneous load of other classes.
@@ -7881,10 +7821,8 @@ static int choose_idle_cpu(int cpu, stru
}
static void
-requeue_delayed_entity(struct sched_entity *se)
+requeue_delayed_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
{
- struct cfs_rq *cfs_rq = cfs_rq_of(se);
-
/*
* se->sched_delayed should imply: se->on_rq == 1.
* Because a delayed entity is one that is still on
@@ -7894,19 +7832,58 @@ requeue_delayed_entity(struct sched_enti
WARN_ON_ONCE(!se->on_rq);
if (update_entity_lag(cfs_rq, se)) {
- cfs_rq->nr_queued--;
+ cfs_rq->h_nr_queued--;
if (se != cfs_rq->curr)
__dequeue_entity(cfs_rq, se);
place_entity(cfs_rq, se, 0);
if (se != cfs_rq->curr)
__enqueue_entity(cfs_rq, se);
- cfs_rq->nr_queued++;
+ cfs_rq->h_nr_queued++;
}
update_load_avg(cfs_rq, se, 0);
clear_delayed(se);
}
+static unsigned long enqueue_hierarchy(struct task_struct *p, int flags)
+{
+ unsigned long weight = NICE_0_LOAD;
+ int task_new = !(flags & ENQUEUE_WAKEUP);
+ struct sched_entity *se = &p->se;
+ int h_nr_idle = task_has_idle_policy(p);
+ int h_nr_runnable = 1;
+
+ if (task_new && se->sched_delayed)
+ h_nr_runnable = 0;
+
+ for_each_sched_entity(se) {
+ struct cfs_rq *cfs_rq = cfs_rq_of(se);
+
+ update_curr(cfs_rq);
+
+ if (!se->on_rq) {
+ enqueue_entity(cfs_rq, se, flags);
+ } else {
+ update_load_avg(cfs_rq, se, UPDATE_TG);
+ se_update_runnable(se);
+ update_cfs_group(se);
+ }
+
+ cfs_rq->h_nr_runnable += h_nr_runnable;
+ cfs_rq->h_nr_queued++;
+ cfs_rq->h_nr_idle += h_nr_idle;
+
+ if (cfs_rq_is_idle(cfs_rq))
+ h_nr_idle = 1;
+
+ weight = __calc_prop_weight(cfs_rq, se, weight);
+
+ flags = ENQUEUE_WAKEUP;
+ }
+
+ return weight;
+}
+
/*
* The enqueue_task method is called before nr_running is
* increased. Here we update the fair scheduling stats and
@@ -7915,13 +7892,12 @@ requeue_delayed_entity(struct sched_enti
static void
enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
{
- struct cfs_rq *cfs_rq;
- struct sched_entity *se = &p->se;
- int h_nr_idle = task_has_idle_policy(p);
- int h_nr_runnable = 1;
- int task_new = !(flags & ENQUEUE_WAKEUP);
int rq_h_nr_queued = rq->cfs.h_nr_queued;
- u64 slice = 0;
+ int task_new = !(flags & ENQUEUE_WAKEUP);
+ struct sched_entity *se = &p->se;
+ struct cfs_rq *cfs_rq = &rq->cfs;
+ unsigned long weight;
+ bool curr;
if (task_is_throttled(p) && enqueue_throttled_task(p))
return;
@@ -7933,10 +7909,10 @@ enqueue_task_fair(struct rq *rq, struct
* estimated utilization, before we update schedutil.
*/
if (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED))
- util_est_enqueue(&rq->cfs, p);
+ util_est_enqueue(cfs_rq, p);
if (flags & ENQUEUE_DELAYED) {
- requeue_delayed_entity(se);
+ requeue_delayed_entity(cfs_rq, se);
return;
}
@@ -7948,57 +7924,22 @@ enqueue_task_fair(struct rq *rq, struct
if (p->in_iowait)
cpufreq_update_util(rq, SCHED_CPUFREQ_IOWAIT);
- if (task_new && se->sched_delayed)
- h_nr_runnable = 0;
-
- for_each_sched_entity(se) {
- if (se->on_rq) {
- if (se->sched_delayed)
- requeue_delayed_entity(se);
- break;
- }
- cfs_rq = cfs_rq_of(se);
-
- /*
- * Basically set the slice of group entries to the min_slice of
- * their respective cfs_rq. This ensures the group can service
- * its entities in the desired time-frame.
- */
- if (slice) {
- se->slice = slice;
- se->custom_slice = 1;
- }
- enqueue_entity(cfs_rq, se, flags);
- slice = cfs_rq_min_slice(cfs_rq);
-
- cfs_rq->h_nr_runnable += h_nr_runnable;
- cfs_rq->h_nr_queued++;
- cfs_rq->h_nr_idle += h_nr_idle;
-
- if (cfs_rq_is_idle(cfs_rq))
- h_nr_idle = 1;
-
- flags = ENQUEUE_WAKEUP;
- }
-
- for_each_sched_entity(se) {
- cfs_rq = cfs_rq_of(se);
+ /*
+ * XXX comment on the curr thing
+ */
+ curr = (cfs_rq->curr == se);
+ if (curr)
+ place_entity(cfs_rq, se, flags);
- update_load_avg(cfs_rq, se, UPDATE_TG);
- se_update_runnable(se);
- update_cfs_group(se);
+ if (se->on_rq && se->sched_delayed)
+ requeue_delayed_entity(cfs_rq, se);
- se->slice = slice;
- if (se != cfs_rq->curr)
- min_vruntime_cb_propagate(&se->run_node, NULL);
- slice = cfs_rq_min_slice(cfs_rq);
+ weight = enqueue_hierarchy(p, flags);
- cfs_rq->h_nr_runnable += h_nr_runnable;
- cfs_rq->h_nr_queued++;
- cfs_rq->h_nr_idle += h_nr_idle;
-
- if (cfs_rq_is_idle(cfs_rq))
- h_nr_idle = 1;
+ if (!curr) {
+ reweight_eevdf(cfs_rq, se, weight, false);
+ place_entity(cfs_rq, se, flags | ENQUEUE_QUEUED);
+ __enqueue_entity(cfs_rq, se);
}
if (!rq_h_nr_queued && rq->cfs.h_nr_queued)
@@ -8029,105 +7970,109 @@ enqueue_task_fair(struct rq *rq, struct
hrtick_update(rq);
}
-/*
- * Basically dequeue_task_fair(), except it can deal with dequeue_entity()
- * failing half-way through and resume the dequeue later.
- *
- * Returns:
- * -1 - dequeue delayed
- * 0 - dequeue throttled
- * 1 - dequeue complete
- */
-static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
+static void dequeue_hierarchy(struct task_struct *p, int flags)
{
- bool was_sched_idle = sched_idle_rq(rq);
+ struct sched_entity *se = &p->se;
bool task_sleep = flags & DEQUEUE_SLEEP;
bool task_delayed = flags & DEQUEUE_DELAYED;
bool task_throttled = flags & DEQUEUE_THROTTLE;
- struct task_struct *p = NULL;
- int h_nr_idle = 0;
- int h_nr_queued = 0;
int h_nr_runnable = 0;
- struct cfs_rq *cfs_rq;
- u64 slice = 0;
+ int h_nr_idle = task_has_idle_policy(p);
+ bool dequeue = true;
- if (entity_is_task(se)) {
- p = task_of(se);
- h_nr_queued = 1;
- h_nr_idle = task_has_idle_policy(p);
- if (task_sleep || task_delayed || !se->sched_delayed)
- h_nr_runnable = 1;
- }
+ if (task_sleep || task_delayed || !se->sched_delayed)
+ h_nr_runnable = 1;
for_each_sched_entity(se) {
- cfs_rq = cfs_rq_of(se);
+ struct cfs_rq *cfs_rq = cfs_rq_of(se);
- if (!dequeue_entity(cfs_rq, se, flags)) {
- if (p && &p->se == se)
- return -1;
+ update_curr(cfs_rq);
- slice = cfs_rq_min_slice(cfs_rq);
- break;
+ if (dequeue) {
+ dequeue_entity(cfs_rq, se, flags);
+ /* Don't dequeue parent if it has other entities besides us */
+ if (cfs_rq->load.weight)
+ dequeue = false;
+ } else {
+ update_load_avg(cfs_rq, se, UPDATE_TG);
+ se_update_runnable(se);
+ update_cfs_group(se);
}
cfs_rq->h_nr_runnable -= h_nr_runnable;
- cfs_rq->h_nr_queued -= h_nr_queued;
+ cfs_rq->h_nr_queued--;
cfs_rq->h_nr_idle -= h_nr_idle;
if (cfs_rq_is_idle(cfs_rq))
- h_nr_idle = h_nr_queued;
+ h_nr_idle = 1;
if (throttled_hierarchy(cfs_rq) && task_throttled)
record_throttle_clock(cfs_rq);
- /* Don't dequeue parent if it has other entities besides us */
- if (cfs_rq->load.weight) {
- slice = cfs_rq_min_slice(cfs_rq);
-
- /* Avoid re-evaluating load for this entity: */
- se = parent_entity(se);
- /*
- * Bias pick_next to pick a task from this cfs_rq, as
- * p is sleeping when it is within its sched_slice.
- */
- if (task_sleep && se)
- set_next_buddy(se);
- break;
- }
flags |= DEQUEUE_SLEEP;
flags &= ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL);
}
+}
- for_each_sched_entity(se) {
- cfs_rq = cfs_rq_of(se);
+/*
+ * The part of dequeue_task_fair() that is needed to dequeue delayed tasks.
+ *
+ * Returns:
+ * true - dequeued
+ * false - delayed
+ */
+static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
+{
+ struct sched_entity *se = &p->se;
+ struct cfs_rq *cfs_rq = &rq->cfs;
+ bool was_sched_idle = sched_idle_rq(rq);
+ bool task_sleep = flags & DEQUEUE_SLEEP;
+ bool task_delayed = flags & DEQUEUE_DELAYED;
- update_load_avg(cfs_rq, se, UPDATE_TG);
- se_update_runnable(se);
- update_cfs_group(se);
+ clear_buddies(cfs_rq, se);
- se->slice = slice;
- if (se != cfs_rq->curr)
- min_vruntime_cb_propagate(&se->run_node, NULL);
- slice = cfs_rq_min_slice(cfs_rq);
+ update_curr(cfs_rq_of(se));
+ update_entity_lag(cfs_rq, se);
- cfs_rq->h_nr_runnable -= h_nr_runnable;
- cfs_rq->h_nr_queued -= h_nr_queued;
- cfs_rq->h_nr_idle -= h_nr_idle;
+ if (flags & DEQUEUE_DELAYED) {
+ WARN_ON_ONCE(!se->sched_delayed);
+ } else {
+ bool delay = task_sleep;
+ /*
+ * DELAY_DEQUEUE relies on spurious wakeups, special task
+ * states must not suffer spurious wakeups, excempt them.
+ */
+ if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
+ delay = false;
- if (cfs_rq_is_idle(cfs_rq))
- h_nr_idle = h_nr_queued;
+ WARN_ON_ONCE(delay && se->sched_delayed);
- if (throttled_hierarchy(cfs_rq) && task_throttled)
- record_throttle_clock(cfs_rq);
+ if (sched_feat(DELAY_DEQUEUE) && delay &&
+ !entity_eligible(cfs_rq, se)) {
+ update_load_avg(cfs_rq_of(se), se, UPDATE_UTIL_EST);
+ set_delayed(se);
+ return false;
+ }
}
- sub_nr_running(rq, h_nr_queued);
+ dequeue_hierarchy(p, flags);
+
+ if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
+ se->deadline -= se->vruntime;
+ se->rel_deadline = 1;
+ }
+ if (se != cfs_rq->curr)
+ __dequeue_entity(cfs_rq, se);
+
+ sub_nr_running(rq, 1);
/* balance early to pull high priority tasks */
if (unlikely(!was_sched_idle && sched_idle_rq(rq)))
rq->next_balance = jiffies;
- if (p && task_delayed) {
+ if (task_delayed) {
+ clear_delayed(se);
+
WARN_ON_ONCE(!task_sleep);
WARN_ON_ONCE(p->on_rq != 1);
@@ -8139,7 +8084,7 @@ static int dequeue_entities(struct rq *r
__block_task(rq, p);
}
- return 1;
+ return true;
}
/*
@@ -8157,11 +8102,11 @@ static bool dequeue_task_fair(struct rq
if (!p->se.sched_delayed)
util_est_dequeue(&rq->cfs, p);
- if (dequeue_entities(rq, &p->se, flags) < 0)
+ if (!__dequeue_task(rq, p, flags))
return false;
/*
- * Must not reference @p after dequeue_entities(DEQUEUE_DELAYED).
+ * Must not reference @p after __dequeue_task(DEQUEUE_DELAYED).
*/
return true;
}
@@ -9749,19 +9694,6 @@ static void migrate_task_rq_fair(struct
static void task_dead_fair(struct task_struct *p)
{
struct sched_entity *se = &p->se;
-
- if (se->sched_delayed) {
- struct rq_flags rf;
- struct rq *rq;
-
- rq = task_rq_lock(p, &rf);
- if (se->sched_delayed) {
- update_rq_clock(rq);
- dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
- }
- task_rq_unlock(rq, p, &rf);
- }
-
remove_entity_load_avg(se);
}
@@ -9795,21 +9727,10 @@ static void set_cpus_allowed_fair(struct
set_task_max_allowed_capacity(p);
}
-static void set_next_buddy(struct sched_entity *se)
-{
- for_each_sched_entity(se) {
- if (WARN_ON_ONCE(!se->on_rq))
- return;
- if (se_is_idle(se))
- return;
- cfs_rq_of(se)->next = se;
- }
-}
-
enum preempt_wakeup_action {
PREEMPT_WAKEUP_NONE, /* No preemption. */
PREEMPT_WAKEUP_SHORT, /* Ignore slice protection. */
- PREEMPT_WAKEUP_PICK, /* Let __pick_eevdf() decide. */
+ PREEMPT_WAKEUP_PICK, /* Let pick_eevdf() decide. */
PREEMPT_WAKEUP_RESCHED, /* Force reschedule. */
};
@@ -9826,7 +9747,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq,
if (cfs_rq->next && entity_before(cfs_rq->next, pse))
return false;
- set_next_buddy(pse);
+ set_next_buddy(cfs_rq, pse);
return true;
}
@@ -9879,7 +9800,7 @@ static void wakeup_preempt_fair(struct r
enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
struct task_struct *donor = rq->donor;
struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
- struct cfs_rq *cfs_rq = task_cfs_rq(donor);
+ struct cfs_rq *cfs_rq = &rq->cfs;
int cse_is_idle, pse_is_idle;
/*
@@ -9916,7 +9837,6 @@ static void wakeup_preempt_fair(struct r
if (!sched_feat(WAKEUP_PREEMPTION))
return;
- find_matching_se(&se, &pse);
WARN_ON_ONCE(!pse);
cse_is_idle = se_is_idle(se);
@@ -9944,8 +9864,7 @@ static void wakeup_preempt_fair(struct r
if (unlikely(!normal_policy(p->policy)))
return;
- cfs_rq = cfs_rq_of(se);
- update_curr(cfs_rq);
+ update_curr_fair(rq);
/*
* If @p has a shorter slice than current and @p is eligible, override
* current's slice protection in order to allow preemption.
@@ -9989,18 +9908,15 @@ static void wakeup_preempt_fair(struct r
}
pick:
- nse = pick_next_entity(rq, cfs_rq, preempt_action != PREEMPT_WAKEUP_SHORT);
- /* If @p has become the most eligible task, force preemption */
- if (nse == pse)
- goto preempt;
-
- /*
- * Because p is enqueued, nse being null can only mean that we
- * dequeued a delayed task. If there are still entities queued in
- * cfs, check if the next one will be p.
- */
- if (!nse && cfs_rq->nr_queued)
- goto pick;
+ if (cfs_rq->h_nr_queued) {
+ nse = pick_next_entity(rq, preempt_action != PREEMPT_WAKEUP_SHORT);
+ if (unlikely(!nse))
+ goto pick;
+
+ /* If @p has become the most eligible task, force preemption */
+ if (nse == pse)
+ goto preempt;
+ }
if (sched_feat(RUN_TO_PARITY))
update_protect_slice(cfs_rq, se);
@@ -10019,33 +9935,24 @@ static void wakeup_preempt_fair(struct r
struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
__must_hold(__rq_lockp(rq))
{
+ struct cfs_rq *cfs_rq = &rq->cfs;
struct sched_entity *se;
- struct cfs_rq *cfs_rq;
struct task_struct *p;
- bool throttled;
int new_tasks;
again:
- cfs_rq = &rq->cfs;
- if (!cfs_rq->nr_queued)
+ if (!cfs_rq->h_nr_queued)
goto idle;
- throttled = false;
-
- do {
- /* Might not have done put_prev_entity() */
- if (cfs_rq->curr && cfs_rq->curr->on_rq)
- update_curr(cfs_rq);
+ /* Might not have done put_prev_entity() */
+ if (cfs_rq->curr && cfs_rq->curr->on_rq)
+ update_curr(cfs_rq);
- se = pick_next_entity(rq, cfs_rq, true);
- if (!se)
- goto again;
- cfs_rq = group_cfs_rq(se);
- } while (cfs_rq);
+ se = pick_next_entity(rq, true);
+ if (!se)
+ goto again;
p = task_of(se);
- if (unlikely(throttled))
- task_throttle_setup_work(p);
return p;
idle:
@@ -10079,7 +9986,7 @@ void fair_server_init(struct rq *rq)
static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct task_struct *next)
{
struct sched_entity *se = &prev->se;
- struct cfs_rq *cfs_rq;
+ struct cfs_rq *cfs_rq = &rq->cfs;
struct sched_entity *nse = NULL;
#ifdef CONFIG_FAIR_GROUP_SCHED
@@ -10089,7 +9996,7 @@ static void put_prev_task_fair(struct rq
while (se) {
cfs_rq = cfs_rq_of(se);
- if (!nse || cfs_rq->curr)
+ if (!nse || cfs_rq->h_curr)
put_prev_entity(cfs_rq, se);
#ifdef CONFIG_FAIR_GROUP_SCHED
if (nse) {
@@ -10108,6 +10015,14 @@ static void put_prev_task_fair(struct rq
#endif
se = parent_entity(se);
}
+
+ /* Put 'current' back into the tree. */
+ cfs_rq = &rq->cfs;
+ se = &prev->se;
+ WARN_ON_ONCE(cfs_rq->curr != se);
+ cfs_rq->curr = NULL;
+ if (se->on_rq)
+ __enqueue_entity(cfs_rq, se);
}
/*
@@ -10116,8 +10031,8 @@ static void put_prev_task_fair(struct rq
static void yield_task_fair(struct rq *rq)
{
struct task_struct *curr = rq->donor;
- struct cfs_rq *cfs_rq = task_cfs_rq(curr);
struct sched_entity *se = &curr->se;
+ struct cfs_rq *cfs_rq = &rq->cfs;
/*
* Are we the only task in the tree?
@@ -10158,11 +10073,11 @@ static bool yield_to_task_fair(struct rq
struct sched_entity *se = &p->se;
/* !se->on_rq also covers throttled task */
- if (!se->on_rq)
+ if (!se->on_rq || se->sched_delayed)
return false;
/* Tell the scheduler that we'd really like se to run next. */
- set_next_buddy(se);
+ set_next_buddy(&task_rq(p)->cfs, se);
yield_task_fair(rq);
@@ -10501,15 +10416,10 @@ static inline long migrate_degrades_loca
*/
static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_cpu)
{
- struct cfs_rq *dst_cfs_rq;
+ struct cfs_rq *dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
-#ifdef CONFIG_FAIR_GROUP_SCHED
- dst_cfs_rq = tg_cfs_rq(task_group(p), dest_cpu);
-#else
- dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
-#endif
- if (sched_feat(PLACE_LAG) && dst_cfs_rq->nr_queued &&
- !entity_eligible(task_cfs_rq(p), &p->se))
+ if (sched_feat(PLACE_LAG) && dst_cfs_rq->h_nr_queued &&
+ !entity_eligible(&task_rq(p)->cfs, &p->se))
return 1;
return 0;
@@ -11292,7 +11202,7 @@ static void update_cfs_rq_h_load(struct
while ((se = READ_ONCE(cfs_rq->h_load_next)) != NULL) {
load = cfs_rq->h_load;
load = div64_ul(load * se->avg.load_avg,
- cfs_rq_load_avg(cfs_rq) + 1);
+ cfs_rq_load_avg(cfs_rq) + 1);
cfs_rq = group_cfs_rq(se);
cfs_rq->h_load = load;
cfs_rq->last_h_load_update = now;
@@ -14684,7 +14594,7 @@ static inline void task_tick_core(struct
* MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check
* if we need to give up the CPU.
*/
- if (rq->core->core_forceidle_count && rq->cfs.nr_queued == 1 &&
+ if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued == 1 &&
__entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE))
resched_curr(rq);
}
@@ -14893,30 +14803,8 @@ bool cfs_prio_less(const struct task_str
WARN_ON_ONCE(task_rq(b)->core != rq->core);
-#ifdef CONFIG_FAIR_GROUP_SCHED
- /*
- * Find an se in the hierarchy for tasks a and b, such that the se's
- * are immediate siblings.
- */
- while (sea->cfs_rq->tg != seb->cfs_rq->tg) {
- int sea_depth = sea->depth;
- int seb_depth = seb->depth;
-
- if (sea_depth >= seb_depth)
- sea = parent_entity(sea);
- if (sea_depth <= seb_depth)
- seb = parent_entity(seb);
- }
-
- se_fi_update(sea, rq->core->core_forceidle_seq, in_fi);
- se_fi_update(seb, rq->core->core_forceidle_seq, in_fi);
-
- cfs_rqa = sea->cfs_rq;
- cfs_rqb = seb->cfs_rq;
-#else /* !CONFIG_FAIR_GROUP_SCHED: */
cfs_rqa = &task_rq(a)->cfs;
cfs_rqb = &task_rq(b)->cfs;
-#endif /* !CONFIG_FAIR_GROUP_SCHED */
/*
* Find delta after normalizing se's vruntime with its cfs_rq's
@@ -14955,11 +14843,20 @@ static inline void task_tick_core(struct
static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
{
struct sched_entity *se = &curr->se;
- struct cfs_rq *cfs_rq;
- for_each_sched_entity(se) {
- cfs_rq = cfs_rq_of(se);
- entity_tick(cfs_rq, se, queued);
+ if (se->on_rq) {
+ unsigned long weight = NICE_0_LOAD;
+ struct cfs_rq *cfs_rq;
+
+ for_each_sched_entity(se) {
+ cfs_rq = cfs_rq_of(se);
+ entity_tick(cfs_rq, se, queued);
+
+ weight = __calc_prop_weight(cfs_rq, se, weight);
+ }
+
+ se = &curr->se;
+ reweight_eevdf(cfs_rq, se, weight, se->on_rq);
}
if (queued)
@@ -14999,7 +14896,7 @@ prio_changed_fair(struct rq *rq, struct
if (p->prio == oldprio)
return;
- if (rq->cfs.nr_queued == 1)
+ if (rq->cfs.h_nr_queued == 1)
return;
/*
@@ -15128,33 +15025,44 @@ static void switched_to_fair(struct rq *
}
}
-/*
- * Account for a task changing its policy or group.
- *
- * This routine is mostly called to set cfs_rq->curr field when a task
- * migrates between groups/classes.
- */
static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
{
struct sched_entity *se = &p->se;
bool throttled = false;
+ struct cfs_rq *cfs_rq = &rq->cfs;
+ unsigned long weight = NICE_0_LOAD;
+ bool on_rq = se->on_rq;
+
+ clear_buddies(cfs_rq, se);
+
+ if (on_rq)
+ __dequeue_entity(cfs_rq, se);
for_each_sched_entity(se) {
- struct cfs_rq *cfs_rq = cfs_rq_of(se);
+ cfs_rq = cfs_rq_of(se);
- if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) &&
- first && cfs_rq->curr)
- break;
+ if (!IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) ||
+ !first || !cfs_rq->h_curr)
+ set_next_entity(cfs_rq, se);
- set_next_entity(cfs_rq, se, first);
/* ensure bandwidth has been allocated on our new cfs_rq */
throttled |= account_cfs_rq_runtime(cfs_rq, 0);
+
+ if (on_rq)
+ weight = __calc_prop_weight(cfs_rq, se, weight);
}
if (throttled)
task_throttle_setup_work(p);
se = &p->se;
+ cfs_rq->curr = se;
+
+ if (on_rq) {
+ reweight_eevdf(cfs_rq, se, weight, se->on_rq);
+ if (first)
+ set_protect_slice(cfs_rq, se);
+ }
if (task_on_rq_queued(p)) {
/*
@@ -15267,17 +15175,8 @@ void unregister_fair_sched_group(struct
struct sched_entity *se = tg_se(tg, cpu);
struct rq *rq = cpu_rq(cpu);
- if (se) {
- if (se->sched_delayed) {
- guard(rq_lock_irqsave)(rq);
- if (se->sched_delayed) {
- update_rq_clock(rq);
- dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
- }
- list_del_leaf_cfs_rq(cfs_rq);
- }
+ if (se)
remove_entity_load_avg(se);
- }
/*
* Only empty task groups can be destroyed; so we can speculatively
--- a/kernel/sched/pelt.c
+++ b/kernel/sched/pelt.c
@@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched
/*
* running is a subset of runnable (weight) so running can't be set if
* runnable is clear. But there are some corner cases where the current
- * se has been already dequeued but cfs_rq->curr still points to it.
+ * se has been already dequeued but cfs_rq->h_curr still points to it.
* This means that weight will be 0 but not running for a sched_entity
* but also for a cfs_rq if the latter becomes idle. As an example,
* this happens during sched_balance_newidle() which calls
@@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 now
int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se)
{
if (___update_load_sum(now, &se->avg, !!se->on_rq, se_runnable(se),
- cfs_rq->curr == se)) {
+ cfs_rq->h_curr == se)) {
___update_load_avg(&se->avg, se_weight(se));
cfs_se_util_change(&se->avg);
@@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, st
if (___update_load_sum(now, &cfs_rq->avg,
scale_load_down(cfs_rq->load.weight),
cfs_rq->h_nr_runnable,
- cfs_rq->curr != NULL)) {
+ cfs_rq->h_curr != NULL)) {
___update_load_avg(&cfs_rq->avg, 1);
trace_pelt_cfs_tp(cfs_rq);
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -530,21 +530,8 @@ struct task_group {
};
-#ifdef CONFIG_GROUP_SCHED_WEIGHT
#define ROOT_TASK_GROUP_LOAD NICE_0_LOAD
-/*
- * A weight of 0 or 1 can cause arithmetics problems.
- * A weight of a cfs_rq is the sum of weights of which entities
- * are queued on this cfs_rq, so a weight of a entity should not be
- * too large, so as the shares value of a task group.
- * (The default weight is 1024 - so there's no practical
- * limitation from this.)
- */
-#define MIN_SHARES (1UL << 1)
-#define MAX_SHARES (1UL << 18)
-#endif
-
typedef int (*tg_visitor)(struct task_group *, void *);
extern int walk_tg_tree_from(struct task_group *from,
@@ -631,6 +618,17 @@ static inline bool cfs_task_bw_constrain
#endif /* !CONFIG_CGROUP_SCHED */
+/*
+ * A weight of 0 or 1 can cause arithmetics problems.
+ * A weight of a cfs_rq is the sum of weights of which entities
+ * are queued on this cfs_rq, so a weight of a entity should not be
+ * too large, so as the shares value of a task group.
+ * (The default weight is 1024 - so there's no practical
+ * limitation from this.)
+ */
+#define MIN_SHARES (1UL << 1)
+#define MAX_SHARES (1UL << 18)
+
extern void unregister_rt_sched_group(struct task_group *tg);
extern void free_rt_sched_group(struct task_group *tg);
extern int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent);
@@ -709,6 +707,7 @@ struct cfs_rq {
/*
* CFS load tracking
*/
+ struct sched_entity *h_curr;
struct sched_avg avg;
#ifndef CONFIG_64BIT
u64 last_update_time_copy;
@@ -2575,6 +2574,7 @@ extern const u32 sched_prio_to_wmult[40
#define ENQUEUE_MIGRATED 0x00040000
#define ENQUEUE_INITIAL 0x00080000
#define ENQUEUE_RQ_SELECTED 0x00100000
+#define ENQUEUE_QUEUED 0x00200000
#define RETRY_TASK ((void *)-1UL)
^ permalink raw reply
* [PATCH v3 6/7] sched/fair: Change the default cgroup_mode to concur
From: Peter Zijlstra @ 2026-06-05 12:40 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260605105513.354837583@infradead.org>
For all the reasons described in the preceding patches, the way cgroup
weight is computed is problematic. However, changing it is bound to
also lead to trouble. Esp. since people might have taken to inflating
the weight value where they can.
Since things are configurable, change the default and hope this serves
more people than it hurts, esp. in the longer run.
Specifically, this prepares for a flattened runqueue, where the hierarchical
weight becomes far more important (F_g^d terms), so getting rid of small F_g is
imperative.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/debug.c | 2 +-
kernel/sched/fair.c | 4 ++--
2 files changed, 3 insertions(+), 3 deletions(-)
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -635,7 +635,7 @@ static void debugfs_fair_server_init(voi
}
#ifdef CONFIG_FAIR_GROUP_SCHED
-static int cgroup_mode = 1;
+static int cgroup_mode = 2;
/* See __sched_cgroup_mode_update(). */
static const char *cgroup_mode_str[] = {
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4921,7 +4921,7 @@ static long calc_up_shares(struct cfs_rq
return READ_ONCE(tg->shares);
}
-DEFINE_STATIC_CALL(calc_group_shares, calc_smp_shares);
+DEFINE_STATIC_CALL(calc_group_shares, calc_concur_shares);
void __sched_cgroup_mode_update(int mode)
{
@@ -4931,10 +4931,10 @@ void __sched_cgroup_mode_update(int mode
func = &calc_up_shares;
break;
case 1:
- default:
func = &calc_smp_shares;
break;
case 2:
+ default:
func = &calc_concur_shares;
break;
case 3:
^ permalink raw reply
* [PATCH v3 2/7] sched/fair: Add cgroup_mode: up
From: Peter Zijlstra @ 2026-06-05 12:40 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260605105513.354837583@infradead.org>
Instead of calculating the proportional fraction of the group weight for each
CPU, just give each CPU the full measure, ignoring these pesky SMP problems.
This makes the SMP cgroup fraction (F_g_n) equal to 1, and ensures a single
task in a cgroup competes on equal footing to a task in a level above.
However, as already explored, this is not a very good policy because it gets
the SMP weight distribution wrong. Included for completeness.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/debug.c | 5 ++++-
kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++--
kernel/sched/sched.h | 1 +
3 files changed, 34 insertions(+), 3 deletions(-)
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -271,6 +271,7 @@ static ssize_t sched_dynamic_write(struc
if (mode < 0)
return mode;
+ __sched_cgroup_mode_update(mode);
sched_dynamic_update(mode);
*ppos += cnt;
@@ -634,9 +635,11 @@ static void debugfs_fair_server_init(voi
}
#ifdef CONFIG_FAIR_GROUP_SCHED
-static int cgroup_mode = 0;
+static int cgroup_mode = 1;
+/* See __sched_cgroup_mode_update(). */
static const char *cgroup_mode_str[] = {
+ "up",
"smp",
};
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -38,6 +38,7 @@
#include <linux/sched/isolation.h>
#include <linux/sched/nohz.h>
#include <linux/sched/prio.h>
+#include <linux/static_call.h>
#include <linux/cpuidle.h>
#include <linux/interrupt.h>
@@ -4800,7 +4801,7 @@ static inline int throttled_hierarchy(st
*
* hence icky!
*/
-static long calc_group_shares(struct cfs_rq *cfs_rq)
+static long calc_smp(struct cfs_rq *cfs_rq)
{
long tg_weight, tg_shares, load, shares;
struct task_group *tg = cfs_rq->tg;
@@ -4835,6 +4836,32 @@ static long calc_group_shares(struct cfs
}
/*
+ * Ignore this pesky SMP stuff, use (4).
+ */
+static long calc_up_shares(struct cfs_rq *cfs_rq)
+{
+ struct task_group *tg = cfs_rq->tg;
+ return READ_ONCE(tg->shares);
+}
+
+DEFINE_STATIC_CALL(calc_group_shares, calc_smp_shares);
+
+void __sched_cgroup_mode_update(int mode)
+{
+ long (*func)(struct cfs_rq *);
+ switch (mode) {
+ case 0:
+ func = &calc_up_shares;
+ break;
+ case 1:
+ default:
+ func = &calc_smp_shares;
+ break;
+ }
+ static_call_update(calc_group_shares, func);
+}
+
+/*
* Recomputes the group entity based on the current state of its group
* runqueue.
*/
@@ -4850,7 +4877,7 @@ static void update_cfs_group(struct sche
if (!gcfs_rq || !gcfs_rq->load.weight)
return;
- shares = calc_group_shares(gcfs_rq);
+ shares = static_call(calc_group_shares)(gcfs_rq);
if (unlikely(se->load.weight != shares))
reweight_entity(cfs_rq_of(se), se, shares);
}
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -571,6 +571,7 @@ extern void free_fair_sched_group(struct
extern int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent);
extern void online_fair_sched_group(struct task_group *tg);
extern void unregister_fair_sched_group(struct task_group *tg);
+extern void __sched_cgroup_mode_update(int mode);
#else /* !CONFIG_FAIR_GROUP_SCHED: */
static inline void free_fair_sched_group(struct task_group *tg) { }
static inline int alloc_fair_sched_group(struct task_group *tg, struct task_group *parent)
^ permalink raw reply
* [PATCH v3 4/7] sched/fair: Add cgroup_mode: concur
From: Peter Zijlstra @ 2026-06-05 12:40 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, peterz, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260605105513.354837583@infradead.org>
Improve upon the previous scheme ("max") by no longer assuming maximal
concurrency. Instead scale by: 'min(nr_tasks, nr_cpus)'. This handles
the low concurrency cases more gracefully:
F_g_n' = min(M, N) * F_g_n
Notably this is the first mode where:
avg(F_g_n) = 1
In the single task case it reduces to ("smp") and then it nicely scales up
until it hits N, where it behaves like ("max").
This is no longer clipped at nice -20. Strictly speaking it isn't different
from the normal SMP scenario where all tasks are extremely unbalanced. There
are no unnatural inflations in this scheme.
The meaning of "cpu.weight" would be: weight per active CPU.
NOTE: Compute the group wide number of tasks by extending the tg->load_avg
computation with tg->runnable_avg, since cfs_rq->runnable_avg is based on
cfs_rq->h_nr_running.
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
---
kernel/sched/debug.c | 1 +
kernel/sched/fair.c | 43 ++++++++++++++++++++++++++++++++++++-------
kernel/sched/sched.h | 3 +++
3 files changed, 40 insertions(+), 7 deletions(-)
--- a/kernel/sched/debug.c
+++ b/kernel/sched/debug.c
@@ -641,6 +641,7 @@ static int cgroup_mode = 1;
static const char *cgroup_mode_str[] = {
"up",
"smp",
+ "concur",
"max",
};
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -4846,6 +4846,11 @@ static int tg_cpus(struct task_group *tg
return nr;
}
+static inline int tg_tasks(struct task_group *tg)
+{
+ return max(1, atomic_long_read(&tg->runnable_avg) >> SCHED_CAPACITY_SHIFT);
+}
+
/*
* Func: min(fraction(nr_cpus * tg->shares), nice -20)
*
@@ -4863,6 +4868,20 @@ static long calc_max_shares(struct cfs_r
}
/*
+ * Func: fraction(nr * tg->shares); nr = min(nr_tasks, nr_cpus)
+ *
+ * Scales between "smp" and "max" in a natural way. No longer needs clipping
+ * since there are no unnatural inflations like with "max".
+ */
+static long calc_concur_shares(struct cfs_rq *cfs_rq)
+{
+ struct task_group *tg = cfs_rq->tg;
+ int nr = min(tg_tasks(tg), tg_cpus(tg));
+ long tg_shares = READ_ONCE(tg->shares);
+ return __calc_smp_shares(cfs_rq, nr * tg_shares, nr * tg_shares);
+}
+
+/*
* Func: fraction(tg->shares)
*
* This infamously results in tiny shares when you have many CPUs.
@@ -4897,6 +4916,9 @@ void __sched_cgroup_mode_update(int mode
func = &calc_smp_shares;
break;
case 2:
+ func = &calc_concur_shares;
+ break;
+ case 3:
func = &calc_max_shares;
break;
}
@@ -5043,7 +5065,7 @@ static inline bool cfs_rq_is_decayed(str
*/
static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
{
- long delta;
+ long dl, dr;
u64 now;
/*
@@ -5064,17 +5086,21 @@ static inline void update_tg_load_avg(st
if (now - cfs_rq->last_update_tg_load_avg < NSEC_PER_MSEC)
return;
- delta = cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib;
- if (abs(delta) > cfs_rq->tg_load_avg_contrib / 64) {
- atomic_long_add(delta, &cfs_rq->tg->load_avg);
+ dl = cfs_rq->avg.load_avg - cfs_rq->tg_load_avg_contrib;
+ dr = cfs_rq->avg.runnable_avg - cfs_rq->tg_runnable_avg_contrib;
+ if (abs(dl) > cfs_rq->tg_load_avg_contrib / 64 ||
+ abs(dr) > cfs_rq->tg_runnable_avg_contrib / 64) {
+ atomic_long_add(dl, &cfs_rq->tg->load_avg);
+ atomic_long_add(dr, &cfs_rq->tg->runnable_avg);
cfs_rq->tg_load_avg_contrib = cfs_rq->avg.load_avg;
+ cfs_rq->tg_runnable_avg_contrib = cfs_rq->avg.runnable_avg;
cfs_rq->last_update_tg_load_avg = now;
}
}
static inline void clear_tg_load_avg(struct cfs_rq *cfs_rq)
{
- long delta;
+ long dl, dr;
u64 now;
/*
@@ -5084,9 +5110,12 @@ static inline void clear_tg_load_avg(str
return;
now = rq_clock(rq_of(cfs_rq));
- delta = 0 - cfs_rq->tg_load_avg_contrib;
- atomic_long_add(delta, &cfs_rq->tg->load_avg);
+ dl = 0 - cfs_rq->tg_load_avg_contrib;
+ dr = 0 - cfs_rq->tg_runnable_avg_contrib;
+ atomic_long_add(dl, &cfs_rq->tg->load_avg);
+ atomic_long_add(dr, &cfs_rq->tg->runnable_avg);
cfs_rq->tg_load_avg_contrib = 0;
+ cfs_rq->tg_runnable_avg_contrib = 0;
cfs_rq->last_update_tg_load_avg = now;
}
--- a/kernel/sched/sched.h
+++ b/kernel/sched/sched.h
@@ -493,6 +493,8 @@ struct task_group {
* will also be accessed at each tick.
*/
atomic_long_t load_avg ____cacheline_aligned;
+ atomic_long_t runnable_avg;
+
#endif /* CONFIG_FAIR_GROUP_SCHED */
#ifdef CONFIG_RT_GROUP_SCHED
@@ -722,6 +724,7 @@ struct cfs_rq {
#ifdef CONFIG_FAIR_GROUP_SCHED
u64 last_update_tg_load_avg;
unsigned long tg_load_avg_contrib;
+ unsigned long tg_runnable_avg_contrib;
long propagate;
long prop_runnable_sum;
^ permalink raw reply
* Re: [PATCH v3 2/7] sched/fair: Add cgroup_mode: up
From: Peter Zijlstra @ 2026-06-05 15:07 UTC (permalink / raw)
To: mingo
Cc: longman, chenridong, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260605124051.450303977@infradead.org>
On Fri, Jun 05, 2026 at 02:40:15PM +0200, Peter Zijlstra wrote:
> Instead of calculating the proportional fraction of the group weight for each
> CPU, just give each CPU the full measure, ignoring these pesky SMP problems.
>
> This makes the SMP cgroup fraction (F_g_n) equal to 1, and ensures a single
> task in a cgroup competes on equal footing to a task in a level above.
>
> However, as already explored, this is not a very good policy because it gets
> the SMP weight distribution wrong. Included for completeness.
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
> kernel/sched/debug.c | 5 ++++-
> kernel/sched/fair.c | 31 +++++++++++++++++++++++++++++--
> kernel/sched/sched.h | 1 +
> 3 files changed, 34 insertions(+), 3 deletions(-)
>
> --- a/kernel/sched/debug.c
> +++ b/kernel/sched/debug.c
> @@ -271,6 +271,7 @@ static ssize_t sched_dynamic_write(struc
> if (mode < 0)
> return mode;
>
> + __sched_cgroup_mode_update(mode);
> sched_dynamic_update(mode);
>
> *ppos += cnt;
Yeez, I'm not sute WTF happened here. Let me go fix that up.
^ permalink raw reply
* Re: [PATCH] mm: don't allow empty relative nodemask in mpol_relative_nodemask()
From: David Hildenbrand (Arm) @ 2026-06-05 15:18 UTC (permalink / raw)
To: Farhad Alemi, Gregory Price
Cc: falemi, Yury Norov, Joshua Hahn, Andrew Morton, Zi Yan,
Matthew Brost, Rakie Kim, Byungchul Park, Ying Huang,
Alistair Popple, linux-mm, linux-kernel, Waiman Long,
Rasmus Villemoes, cgroups
In-Reply-To: <CA+0ovCiEz6SP_sn3kN4Tb+_oC=eHMXy_Ffj=usV3wREdQrUtww@mail.gmail.com>
On 6/2/26 17:01, Farhad Alemi wrote:
> Confirmed, with a standalone reproducer (attached); it panics linus/master
> at e8c2f9fdadee. cs->mems_allowed can legitimately be empty
> on v2 -- a freshly created cpuset child that never had cpuset.mems
> written keeps mems_allowed empty (never initialized) while effective_mems
> is inherited non-empty in cpuset_css_online(), and v2 allows attaching
> tasks to it (the empty-mems guard in cpuset_can_attach_check() is gated
> on !is_in_v2_mode()). So the non-empty guarantee holds for effective_mems,
> not for the configured cs->mems_allowed; forbidding empty cpuset.mems
> would break v2's inherit-from-parent semantics.
>
> The reproducer enables +cpuset, mkdirs a child without writing
> cpuset.mems, moves a task in, mbind()s a VMA with
> MPOL_BIND | MPOL_F_RELATIVE_NODES, and offlines a CPU; the hotplug walk
> then calls mpol_rebind_mm(mm, &cs->mems_allowed) with the empty mask and
> folds modulo nodes_weight(*rel) == 0 (console logs attached).
>
> The newmems instinct looks right: it's the effective, online mask the
> task is actually allowed to use, guarantee_online_mems() keeps it
> non-empty, and it matches cpuset_attach(), which already rebinds against
> cs->effective_mems. The fix this implies:
>
> - mpol_rebind_mm(mm, &cs->mems_allowed);
> + mpol_rebind_mm(mm, &newmems);
>
> I built the current base (e8c2f9fdadee) with and without this one-liner:
> the unpatched kernel panics on the first cpu1 offline, while the patched
> kernel runs the reproducer's 8 offline/online cycles cleanly, with no
> divide error.
>
> This regressed in ae1c802382f7 ("cpuset: apply cs->effective_{cpus,mems}",
> v3.17), which moved cpuset_attach() to the effective mask but left this
> rebind on cs->mems_allowed.
>
> Happy to send this as a proper patch (Fixes: ae1c802382f7, Cc: stable,
> reproducer) if you agree the cpuset side is right, or to test a
> mempolicy-side fix if not.
Yes, please send a patch, including a high-level explanation of what you
analyzed above!
--
Cheers,
David
^ permalink raw reply
* [PATCH 0/6 v3] mm/memcontrol, page_counter: move stock from mem_cgroup to page_counter
From: Joshua Hahn @ 2026-06-05 15:35 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R . Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, cgroups,
linux-mm, linux-kernel, kernel-team
Memcg currently keeps a "stock" of 64 pages per-cpu to cache pre-charged
allocations, allowing small allocations to avoid walking the expensive
mem_cgroup hierarchy traversal and atomic operations on each charge.
This design introduces a fastpath, but there is room for improvement:
1. Currently, each CPU tracks up to 7 (NR_MEMCG_STOCK) mem_cgroups. When
more than 7 mem_cgroups are actively charging on a single CPU, a
random victim is evicted and its associated stock is drained.
2. Stock management is tightly coupled to struct mem_cgroup, which makes
it difficult to add a new page_counter to mem_cgroup and have
multiple sources of stock management, which is required when trying
to introduce fastpaths to multiple hard limit checks.
This series moves the per-cpu stock down into the page_counter which
consolidates stock limit checking and page_counter limit checking into
page_counter_try_charge. This eliminates the 7-memcg-per-cpu slot limit,
the random evictions (drain & refill), and slot traversal.
In turn, we can add independent stock management for additional
page_counters in each memcg, which is used in my tiered memory limits
series to add a new page_counter to track toptier usage [1].
The resulting code in memcg is also easier to follow, as the caching
becomes transparent from memcg's perspective and managed entirely within
page_counter.
There are, however, a few tradeoffs.
First, the bound on how much memory can be overcharged (and remain stale
as stock) is raised. Previously, it was fixed to nr_cpus x 7 x 64 pages.
Now, it becomes nr_leaf_cgroups x nr_cpus x 64 pages. On large machines
with many cgroups, this could be significant.
There are four qualifying points:
1. Larger machines should be able to tolerate the additional overhead,
2. Stock should not remain stale as long as the cgroups are actively
charging memory,
3. Getting close to this overhead is rare, as it would require a process
to migrate across all CPUs and leave stock there.
4. These charges are not "real" allocated memory, but rather accounting
done in memcg; they are easily returned on pressure.
Secondly, we introduce some additional memory footprint. The new struct
page_counter_stock adds 2 words of extra overhead per-(cpu x memcg).
A small change is that for cgroupv1, reported memsw usage can be lower
than reported memory usage, if the memsw page_counter overcharges to its
stock whereas the memory page_counter does not.
Finally, to keep the above memory footprint limited, I opted to not
embed a work_struct into page_counter_stock, but rather decided to
trigger synchronous stock draining, since the drain operation is rarer
now, and only happens under memory pressure and on cgroup death.
One side effect of doing synchronous work is that drain_all_stock holds
the percpu_charge_mutex longer while it performs the work, which means
chargers may be more likely to be unable to grab the mutex lock and
exhaust MAX_RECLAIM_RETRIES and OOM, in theory. In practice, I have not
been able to replicate this behavior in my experiments.
Performance testing across single-cgroup, as well as 4-cgroup (under the
7 memcg limit) and 32-cgroup scenarios on a 40CPU, 50G memory system
shows moderate performance gains (~1%). In the tests, I repeatedly
fault and release anonymous pages using madvise(MADV_DONTNEED) to
stress the charge/uncharge path, across 30 trials of 50 iterations.
Metric here is time it took across each iteration (ms).
+----------+--------+-------+-----------+
| #cgroups | before | after | delta (%) |
+----------+--------+-------+-----------+
| 1 | 357 | 350 | -1.960 |
| 4 | 1221 | 1204 | -1.392 |
| 32 | 9184 | 9032 | -1.682 |
+----------+--------+-------+-----------+
Further testing on other stress-ng microbenchmarks also agreed with
these results.
v2 --> v3:
- Dropped the cgroup v2 optimization, since it could indeed lead to too
much time held with the cgroup_mutex. Instead we let the stock
accumulate in the parent cgroups, which is not so bad; charges can
still land on these cgroups, and if we ever reach the mem_cgroup
limit, we can easily return those charges.
- page_counter_disable_stock no longer drains, just prevents
accumulating stock. The actual draining is done in the free_stock
variant, where we know for sure there are no in-flight charges.
- Reordering the page_counter_disable_stock path to disable before
draining as to prevent accumulating stock first.
- Skip isolated CPUs when draining synchronously
- Rebase on newest mm-new
- Wordsmithing
v1 --> v2:
- Dropped stock returning on uncharge to preserve same behavior as memcg
stock. This resolves some race conditions present in v1.
- Fixed many race conditions between disabling page_counter_stock and
in-flight charges
- Restructured drain_all_stock to iterate over all CPUs first before
memcgs, to reduce the number of synchronous CPU work scheduling
- Optimized cgroup v2 further to drain only on the first child and skip
the root mem_cgroup
- Dropped RFC
- Wordsmithing cover letter
[1] https://lore.kernel.org/all/20260423203445.2914963-1-joshua.hahnjy@gmail.com/
Joshua Hahn (6):
mm/page_counter: introduce per-page_counter stock
mm/page_counter: use page_counter_stock in page_counter_try_charge
mm/page_counter: introduce stock drain APIs
mm/memcontrol: convert memcg to use page_counter_stock
mm/memcontrol: optimize memsw stock for cgroup v1
mm/memcontrol: remove unused memcg_stock code
include/linux/page_counter.h | 16 ++
mm/memcontrol.c | 276 ++++++-----------------------------
mm/page_counter.c | 129 +++++++++++++++-
3 files changed, 188 insertions(+), 233 deletions(-)
--
2.53.0-Meta
^ permalink raw reply
* [PATCH 1/6 v3] mm/page_counter: introduce per-page_counter stock
From: Joshua Hahn @ 2026-06-05 15:35 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R . Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260605153603.234296-1-joshua.hahnjy@gmail.com>
In order to avoid expensive hierarchy walks on every memcg charge and
limit check, memcontrol uses per-cpu stocks (memcg_stock_pcp) to cache
pre-charged pages and introduce a fast path to try_charge_memcg.
However, there are a few quirks with the current implementation that
could be improved upon.
First, each memcg_stock_pcp can only cache the charges of 7 memcgs
(defined as NR_MEMCG_STOCK), which means that once a CPU starts handling
the charging of more than 7 memcgs, it randomly selects a victim memcg
to evict and drain from the cpu, which can cause unnecessarily increased
latencies and thrashing as memcgs continually evict each others' stock.
Second, stock is tightly coupled with memcg, which means that all page
counters in a memcg share the same resource. This may simplify some of
the charging logic, but it prevents new page counters from being added
and using a separate stock.
We can address these concerns by pushing the concept of stock down to
the page_counter level, which addresses the random eviction problem by
getting rid of the 7 slot limit, and makes enabling separate stock
caches for other page_counters simpler.
Introduce a generic per-cpu stock directly in struct page_counter.
Stock can optionally be enabled per-page_counter, limiting the overhead
increase for page_counters who do not benefit greatly from caching
charges.
This patch introduces the page_counter_stock struct and its
alloc/disable/free functions, but does not use these yet.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
include/linux/page_counter.h | 13 +++++++++
mm/page_counter.c | 53 ++++++++++++++++++++++++++++++++++++
2 files changed, 66 insertions(+)
diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h
index d649b6bbbc871..c92bb2ee2a581 100644
--- a/include/linux/page_counter.h
+++ b/include/linux/page_counter.h
@@ -5,8 +5,15 @@
#include <linux/atomic.h>
#include <linux/cache.h>
#include <linux/limits.h>
+#include <linux/local_lock.h>
+#include <linux/percpu.h>
#include <asm/page.h>
+struct page_counter_stock {
+ local_trylock_t lock;
+ unsigned long nr_pages;
+};
+
struct page_counter {
/*
* Make sure 'usage' does not share cacheline with any other field in
@@ -41,6 +48,8 @@ struct page_counter {
unsigned long high;
unsigned long max;
struct page_counter *parent;
+ struct page_counter_stock __percpu *stock;
+ unsigned int batch;
} ____cacheline_internodealigned_in_smp;
#if BITS_PER_LONG == 32
@@ -99,6 +108,10 @@ static inline void page_counter_reset_watermark(struct page_counter *counter)
counter->watermark = usage;
}
+int page_counter_alloc_stock(struct page_counter *counter, unsigned int batch);
+void page_counter_disable_stock(struct page_counter *counter);
+void page_counter_free_stock(struct page_counter *counter);
+
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
void page_counter_calculate_protection(struct page_counter *root,
struct page_counter *counter,
diff --git a/mm/page_counter.c b/mm/page_counter.c
index 661e0f2a5127a..9f3e3f8d896c4 100644
--- a/mm/page_counter.c
+++ b/mm/page_counter.c
@@ -8,6 +8,7 @@
#include <linux/page_counter.h>
#include <linux/atomic.h>
#include <linux/kernel.h>
+#include <linux/percpu.h>
#include <linux/string.h>
#include <linux/sched.h>
#include <linux/bug.h>
@@ -289,6 +290,58 @@ int page_counter_memparse(const char *buf, const char *max,
return 0;
}
+int page_counter_alloc_stock(struct page_counter *counter, unsigned int batch)
+{
+ struct page_counter_stock __percpu *stock;
+ int cpu;
+
+ stock = alloc_percpu(struct page_counter_stock);
+ if (!stock)
+ return -ENOMEM;
+
+ for_each_possible_cpu(cpu) {
+ struct page_counter_stock *s = per_cpu_ptr(stock, cpu);
+
+ local_trylock_init(&s->lock);
+ }
+ counter->stock = stock;
+ counter->batch = batch;
+
+ return 0;
+}
+
+void page_counter_disable_stock(struct page_counter *counter)
+{
+ if (!counter->stock)
+ return;
+
+ /* This prevents future charges from trying to deposit pages */
+ WRITE_ONCE(counter->batch, 0);
+}
+
+void page_counter_free_stock(struct page_counter *counter)
+{
+ unsigned long stock_to_drain = 0;
+ int cpu;
+
+ if (!counter->stock)
+ return;
+
+ for_each_possible_cpu(cpu) {
+ struct page_counter_stock *stock;
+
+ stock = per_cpu_ptr(counter->stock, cpu);
+ stock_to_drain += stock->nr_pages;
+ stock->nr_pages = 0;
+ }
+
+ if (stock_to_drain)
+ page_counter_uncharge(counter, stock_to_drain);
+
+ free_percpu(counter->stock);
+ counter->stock = NULL;
+}
+
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
/*
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 2/6 v3] mm/page_counter: use page_counter_stock in page_counter_try_charge
From: Joshua Hahn @ 2026-06-05 15:35 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260605153603.234296-1-joshua.hahnjy@gmail.com>
Make page_counter_try_charge() stock-aware. We preserve the same
semantics as the existing stock handling logic in try_charge_memcg:
1. Limit-check against the stock. If there is enough, charge to the
stock (non-hierarchical) and return immediately.
2. Greedily attempt to fulfill the charge request and fill the stock up
at the same time via a hierarchical charge.
3. If we fail with this charge, retry again (once) with the exact number
of pages requested.
4. If we succeed with the greedy attempt, then try to add those extra
pages to the stock. If that fails (trylock), then uncharge those
surplus pages hierarchically.
As of this patch, the page_counter_stock is unused, as it has not been
enabled on any memcg yet. No functional changes intended.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/page_counter.c | 42 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 39 insertions(+), 3 deletions(-)
diff --git a/mm/page_counter.c b/mm/page_counter.c
index 9f3e3f8d896c4..1a71de4f43fd0 100644
--- a/mm/page_counter.c
+++ b/mm/page_counter.c
@@ -121,9 +121,25 @@ bool page_counter_try_charge(struct page_counter *counter,
struct page_counter **fail)
{
struct page_counter *c;
+ unsigned long charge = nr_pages;
+ unsigned long batch = READ_ONCE(counter->batch);
bool protection = track_protection(counter);
bool track_failcnt = counter->track_failcnt;
+ if (counter->stock && local_trylock(&counter->stock->lock)) {
+ struct page_counter_stock *stock = this_cpu_ptr(counter->stock);
+
+ if (stock->nr_pages >= charge) {
+ stock->nr_pages -= charge;
+ local_unlock(&counter->stock->lock);
+ return true;
+ }
+ local_unlock(&counter->stock->lock);
+ }
+
+ charge = max_t(unsigned long, batch, nr_pages);
+
+retry:
for (c = counter; c; c = c->parent) {
long new;
/*
@@ -140,9 +156,9 @@ bool page_counter_try_charge(struct page_counter *counter,
* we either see the new limit or the setter sees the
* counter has changed and retries.
*/
- new = atomic_long_add_return(nr_pages, &c->usage);
+ new = atomic_long_add_return(charge, &c->usage);
if (new > c->max) {
- atomic_long_sub(nr_pages, &c->usage);
+ atomic_long_sub(charge, &c->usage);
/*
* This is racy, but we can live with some
* inaccuracy in the failcnt which is only used
@@ -163,11 +179,31 @@ bool page_counter_try_charge(struct page_counter *counter,
WRITE_ONCE(c->watermark, new);
}
}
+
+ /* charge > nr_pages implies this page_counter has stock enabled */
+ if (charge > nr_pages) {
+ if (local_trylock(&counter->stock->lock)) {
+ struct page_counter_stock *stock;
+
+ stock = this_cpu_ptr(counter->stock);
+ stock->nr_pages += charge - nr_pages;
+ local_unlock(&counter->stock->lock);
+ } else {
+ page_counter_uncharge(counter, charge - nr_pages);
+ }
+ }
+
return true;
failed:
for (c = counter; c != *fail; c = c->parent)
- page_counter_cancel(c, nr_pages);
+ page_counter_cancel(c, charge);
+
+ if (charge > nr_pages) {
+ /* Retry without trying to grab extra pages to refill stock */
+ charge = nr_pages;
+ goto retry;
+ }
return false;
}
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 3/6 v3] mm/page_counter: introduce stock drain APIs
From: Joshua Hahn @ 2026-06-05 15:35 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
David Hildenbrand, Lorenzo Stoakes, Liam R . Howlett,
Vlastimil Babka, Mike Rapoport, Suren Baghdasaryan, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260605153603.234296-1-joshua.hahnjy@gmail.com>
Introduce page_counter variants to replace memcg stock draining
functions.
page_counter_drain_stock_local() drains the stock of the local CPU,
taking a local stock lock to serialize against concurrent charges.
page_counter_drain_stock_cpu() does the same, but without taking a local
lock. This is possible because it will only be called from the CPU
hotplug path, where the CPU is dead and there cannot be any more charges.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
include/linux/page_counter.h | 3 +++
mm/page_counter.c | 34 ++++++++++++++++++++++++++++++++++
2 files changed, 37 insertions(+)
diff --git a/include/linux/page_counter.h b/include/linux/page_counter.h
index c92bb2ee2a581..4a88874019af0 100644
--- a/include/linux/page_counter.h
+++ b/include/linux/page_counter.h
@@ -111,6 +111,9 @@ static inline void page_counter_reset_watermark(struct page_counter *counter)
int page_counter_alloc_stock(struct page_counter *counter, unsigned int batch);
void page_counter_disable_stock(struct page_counter *counter);
void page_counter_free_stock(struct page_counter *counter);
+void page_counter_drain_stock_local(struct page_counter *counter);
+void page_counter_drain_stock_cpu(struct page_counter *counter,
+ unsigned int cpu);
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
void page_counter_calculate_protection(struct page_counter *root,
diff --git a/mm/page_counter.c b/mm/page_counter.c
index 1a71de4f43fd0..7e7eb683472d9 100644
--- a/mm/page_counter.c
+++ b/mm/page_counter.c
@@ -378,6 +378,40 @@ void page_counter_free_stock(struct page_counter *counter)
counter->stock = NULL;
}
+void page_counter_drain_stock_local(struct page_counter *counter)
+{
+ struct page_counter_stock *stock;
+ unsigned long nr_pages;
+
+ if (!counter->stock)
+ return;
+
+ local_lock(&counter->stock->lock);
+ stock = this_cpu_ptr(counter->stock);
+ nr_pages = stock->nr_pages;
+ stock->nr_pages = 0;
+ local_unlock(&counter->stock->lock);
+
+ if (nr_pages)
+ page_counter_uncharge(counter, nr_pages);
+}
+
+void page_counter_drain_stock_cpu(struct page_counter *counter,
+ unsigned int cpu)
+{
+ struct page_counter_stock *stock;
+ unsigned long nr_pages;
+
+ if (!counter->stock)
+ return;
+
+ stock = per_cpu_ptr(counter->stock, cpu);
+ nr_pages = stock->nr_pages;
+ if (nr_pages) {
+ stock->nr_pages = 0;
+ page_counter_uncharge(counter, nr_pages);
+ }
+}
#if IS_ENABLED(CONFIG_MEMCG) || IS_ENABLED(CONFIG_CGROUP_DMEM)
/*
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 4/6 v3] mm/memcontrol: convert memcg to use page_counter_stock
From: Joshua Hahn @ 2026-06-05 15:36 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260605153603.234296-1-joshua.hahnjy@gmail.com>
Now with all of the memcg_stock handling logic replicated in
page_counter_stock, switch memcg to use the page_counter_stock.
There are a few details that have changed:
First, the old special-casing for the !allow_spinning check to avoid
refilling and flushing of the old stock is removed. This special casing
was important previously, because refilling the stock could do a lot of
extra work by evicting one of 7 random victim memcgs in the percpu
memcg_stock slots. In the new per-counter design, refilling stock just
adds pages to the counter's own local cache without affecting other memcgs,
so the original reason for the special case no longer applies.
Also, we can now fail during page_counter_alloc_stock(), if there is
not enough memory to allocate a percpu page_counter_stock. This failure
is rare and nonfatal; the system can continue to operate, with the page
counter working without stock and falling back to walking the hierarchy.
Finally, drain_all_stock is restructured to iterate CPUs in the outer
loop (rather than memcgs) to be able to schedule draining all memcgs
via a single work_on_cpu call. It reduces the number of synchronous
per-CPU work calls from O(memcgs * CPUs) to just O(CPUs).
We also skip isolated CPUs, as schedule_drain_work() did before. We
don't need its guard(rcu) here though; that rcu section existed to
order async work scheduling against cpumask updates and workqueue
flushes, which would lead to drain work pending.
Since all work here is synchronous, we don't leave any work behind.
Note that obj_stock remains untouched by these changes, and that memsw
stock will be handled in the next patch.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/memcontrol.c | 80 ++++++++++++++++++++++---------------------------
1 file changed, 36 insertions(+), 44 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 56cd4af082326..562ed9301f5a4 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2269,6 +2269,17 @@ static void schedule_drain_work(int cpu, struct work_struct *work)
queue_work_on(cpu, memcg_wq, work);
}
+static long drain_stock_on_cpu(void *arg)
+{
+ struct mem_cgroup *root_memcg = arg;
+ struct mem_cgroup *memcg;
+
+ for_each_mem_cgroup_tree(memcg, root_memcg)
+ page_counter_drain_stock_local(&memcg->memory);
+
+ return 0;
+}
+
/*
* Drains all per-CPU charge caches for given root_memcg resp. subtree
* of the hierarchy under it.
@@ -2280,28 +2291,18 @@ void drain_all_stock(struct mem_cgroup *root_memcg)
/* If someone's already draining, avoid adding running more workers. */
if (!mutex_trylock(&percpu_charge_mutex))
return;
- /*
- * Notify other cpus that system-wide "drain" is running
- * We do not care about races with the cpu hotplug because cpu down
- * as well as workers from this path always operate on the local
- * per-cpu data. CPU up doesn't touch memcg_stock at all.
- */
+
+ for_each_online_cpu(cpu) {
+ if (!cpu_is_isolated(cpu))
+ work_on_cpu(cpu, drain_stock_on_cpu, root_memcg);
+ }
+
+ /* Drain obj_stock on all online CPUs */
migrate_disable();
curcpu = smp_processor_id();
for_each_online_cpu(cpu) {
- struct memcg_stock_pcp *memcg_st = &per_cpu(memcg_stock, cpu);
struct obj_stock_pcp *obj_st = &per_cpu(obj_stock, cpu);
- if (!test_bit(FLUSHING_CACHED_CHARGE, &memcg_st->flags) &&
- is_memcg_drain_needed(memcg_st, root_memcg) &&
- !test_and_set_bit(FLUSHING_CACHED_CHARGE,
- &memcg_st->flags)) {
- if (cpu == curcpu)
- drain_local_memcg_stock(&memcg_st->work);
- else
- schedule_drain_work(cpu, &memcg_st->work);
- }
-
if (!test_bit(FLUSHING_CACHED_CHARGE, &obj_st->flags) &&
obj_stock_flush_required(obj_st, root_memcg) &&
!test_and_set_bit(FLUSHING_CACHED_CHARGE,
@@ -2318,9 +2319,13 @@ void drain_all_stock(struct mem_cgroup *root_memcg)
static int memcg_hotplug_cpu_dead(unsigned int cpu)
{
+ struct mem_cgroup *memcg;
+
/* no need for the local lock */
drain_obj_stock(&per_cpu(obj_stock, cpu));
- drain_stock_fully(&per_cpu(memcg_stock, cpu));
+
+ for_each_mem_cgroup(memcg)
+ page_counter_drain_stock_cpu(&memcg->memory, cpu);
return 0;
}
@@ -2595,7 +2600,6 @@ void __mem_cgroup_handle_over_high(gfp_t gfp_mask)
static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
unsigned int nr_pages)
{
- unsigned int batch = max(MEMCG_CHARGE_BATCH, nr_pages);
int nr_retries = MAX_RECLAIM_RETRIES;
struct mem_cgroup *mem_over_limit;
struct page_counter *counter;
@@ -2608,31 +2612,19 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
bool allow_spinning = gfpflags_allow_spinning(gfp_mask);
retry:
- if (consume_stock(memcg, nr_pages))
- return 0;
-
- if (!allow_spinning)
- /* Avoid the refill and flush of the older stock */
- batch = nr_pages;
-
reclaim_options = MEMCG_RECLAIM_MAY_SWAP;
if (!do_memsw_account() ||
- page_counter_try_charge(&memcg->memsw, batch, &counter)) {
- if (page_counter_try_charge(&memcg->memory, batch, &counter))
+ page_counter_try_charge(&memcg->memsw, nr_pages, &counter)) {
+ if (page_counter_try_charge(&memcg->memory, nr_pages, &counter))
goto done_restock;
if (do_memsw_account())
- page_counter_uncharge(&memcg->memsw, batch);
+ page_counter_uncharge(&memcg->memsw, nr_pages);
mem_over_limit = mem_cgroup_from_counter(counter, memory);
} else {
mem_over_limit = mem_cgroup_from_counter(counter, memsw);
reclaim_options &= ~MEMCG_RECLAIM_MAY_SWAP;
}
- if (batch > nr_pages) {
- batch = nr_pages;
- goto retry;
- }
-
/*
* Prevent unbounded recursion when reclaim operations need to
* allocate memory. This might exceed the limits temporarily,
@@ -2729,9 +2721,6 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
return 0;
done_restock:
- if (batch > nr_pages)
- refill_stock(memcg, batch - nr_pages);
-
/*
* If the hierarchy is above the normal consumption range, schedule
* reclaim on returning to userland. We can perform reclaim here
@@ -2768,7 +2757,7 @@ static int try_charge_memcg(struct mem_cgroup *memcg, gfp_t gfp_mask,
* and distribute reclaim work and delay penalties
* based on how much each task is actually allocating.
*/
- current->memcg_nr_pages_over_high += batch;
+ current->memcg_nr_pages_over_high += nr_pages;
set_notify_resume(current);
break;
}
@@ -3073,7 +3062,7 @@ static void obj_cgroup_uncharge_pages(struct obj_cgroup *objcg,
account_kmem_nmi_safe(memcg, -nr_pages);
memcg1_account_kmem(memcg, -nr_pages);
if (!mem_cgroup_is_root(memcg))
- refill_stock(memcg, nr_pages);
+ memcg_uncharge(memcg, nr_pages);
css_put(&memcg->css);
}
@@ -4077,6 +4066,8 @@ static void __mem_cgroup_free(struct mem_cgroup *memcg)
static void mem_cgroup_free(struct mem_cgroup *memcg)
{
+ page_counter_free_stock(&memcg->memory);
+ page_counter_free_stock(&memcg->memsw);
lru_gen_exit_memcg(memcg);
memcg_wb_domain_exit(memcg);
__mem_cgroup_free(memcg);
@@ -4244,6 +4235,9 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
refcount_set(&memcg->id.ref, 1);
css_get(css);
+ /* failure is nonfatal, charges fall back to direct hierarchy */
+ page_counter_alloc_stock(&memcg->memory, MEMCG_CHARGE_BATCH);
+
/*
* Ensure mem_cgroup_from_private_id() works once we're fully online.
*
@@ -4304,6 +4298,7 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
wb_memcg_offline(memcg);
lru_gen_offline_memcg(memcg);
+ page_counter_disable_stock(&memcg->memory);
drain_all_stock(memcg);
mem_cgroup_private_id_put(memcg, 1);
@@ -5499,7 +5494,7 @@ void mem_cgroup_sk_uncharge(const struct sock *sk, unsigned int nr_pages)
mod_memcg_state(memcg, MEMCG_SOCK, -nr_pages);
- refill_stock(memcg, nr_pages);
+ page_counter_uncharge(&memcg->memory, nr_pages);
}
void mem_cgroup_flush_workqueue(void)
@@ -5552,12 +5547,9 @@ int __init mem_cgroup_init(void)
memcg_wq = alloc_workqueue("memcg", WQ_PERCPU, 0);
WARN_ON(!memcg_wq);
- for_each_possible_cpu(cpu) {
- INIT_WORK(&per_cpu_ptr(&memcg_stock, cpu)->work,
- drain_local_memcg_stock);
+ for_each_possible_cpu(cpu)
INIT_WORK(&per_cpu_ptr(&obj_stock, cpu)->work,
drain_local_obj_stock);
- }
memcg_size = struct_size_t(struct mem_cgroup, nodeinfo, nr_node_ids);
memcg_cachep = kmem_cache_create("mem_cgroup", memcg_size, 0,
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 5/6 v3] mm/memcontrol: optimize memsw stock for cgroup v1
From: Joshua Hahn @ 2026-06-05 15:36 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260605153603.234296-1-joshua.hahnjy@gmail.com>
Previously, each memcg had its own stock, which was shared by all page
counters within it. Specifically in try_charge_memcg, the stock limit
check would occur before the memsw and memory page_counters were
charged hierarchically. Now that the memcg stock was folded into the
page_counter level, and we have replaced try_charge_memcg's stock check
against the memory page_counter's stock, this leaves no fast path available
for cgroup v1's memsw check.
Introduce a new stock for the memsw page_counter, charged independently
from the memory page_counter. This provides better caching on cgroup v1:
The best case scenario is when both the memsw and memory page_counters
can use their cached stock charge; this is the old behavior.
The halfway scenario is when either the memsw or memory page_counter
is within the stock size, but the other isn't. This requires one
hierarchical charge.
The worst case scenario is when both memsw and memory page_counters
are over their limit, and must walk two page_counter hierarchies. This
is the same as the old behavior.
By introducing an independent stock for memsw, we can avoid the worst
case scenario more often and can fail or succeed separately from the
memory page counter.
One user-visible change is that reported memsw usage may transiently
be lower than memory usage. This happens because each counter
independently batches the stock charges, so the visible values can
differ by up to the stock batch size (MEMCG_CHARGE_BATCH) pages.
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/memcontrol.c | 14 ++++++++++++--
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 562ed9301f5a4..d0da2f842e2d4 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2274,8 +2274,11 @@ static long drain_stock_on_cpu(void *arg)
struct mem_cgroup *root_memcg = arg;
struct mem_cgroup *memcg;
- for_each_mem_cgroup_tree(memcg, root_memcg)
+ for_each_mem_cgroup_tree(memcg, root_memcg) {
page_counter_drain_stock_local(&memcg->memory);
+ if (do_memsw_account())
+ page_counter_drain_stock_local(&memcg->memsw);
+ }
return 0;
}
@@ -2324,8 +2327,11 @@ static int memcg_hotplug_cpu_dead(unsigned int cpu)
/* no need for the local lock */
drain_obj_stock(&per_cpu(obj_stock, cpu));
- for_each_mem_cgroup(memcg)
+ for_each_mem_cgroup(memcg) {
page_counter_drain_stock_cpu(&memcg->memory, cpu);
+ if (do_memsw_account())
+ page_counter_drain_stock_cpu(&memcg->memsw, cpu);
+ }
return 0;
}
@@ -4237,6 +4243,8 @@ static int mem_cgroup_css_online(struct cgroup_subsys_state *css)
/* failure is nonfatal, charges fall back to direct hierarchy */
page_counter_alloc_stock(&memcg->memory, MEMCG_CHARGE_BATCH);
+ if (do_memsw_account())
+ page_counter_alloc_stock(&memcg->memsw, MEMCG_CHARGE_BATCH);
/*
* Ensure mem_cgroup_from_private_id() works once we're fully online.
@@ -4299,6 +4307,8 @@ static void mem_cgroup_css_offline(struct cgroup_subsys_state *css)
lru_gen_offline_memcg(memcg);
page_counter_disable_stock(&memcg->memory);
+ if (do_memsw_account())
+ page_counter_disable_stock(&memcg->memsw);
drain_all_stock(memcg);
mem_cgroup_private_id_put(memcg, 1);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 6/6 v3] mm/memcontrol: remove unused memcg_stock code
From: Joshua Hahn @ 2026-06-05 15:36 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko
Cc: Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton, cgroups,
linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260605153603.234296-1-joshua.hahnjy@gmail.com>
Now that all memcg_stock logic has been moved to page_counter_stock, we
can remove all code related to handling memcg_stock. Note that obj_stock
is untouched and is still needed. FLUSHING_CACHED_CHARGE is preserved
so that it can be used by obj_stock as well.
Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Joshua Hahn <joshua.hahnjy@gmail.com>
---
mm/memcontrol.c | 186 ------------------------------------------------
1 file changed, 186 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index d0da2f842e2d4..3e3f8fbd19a48 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1998,25 +1998,7 @@ void mem_cgroup_print_oom_group(struct mem_cgroup *memcg)
pr_cont(" are going to be killed due to memory.oom.group set\n");
}
-/*
- * The value of NR_MEMCG_STOCK is selected to keep the cached memcgs and their
- * nr_pages in a single cacheline. This may change in future.
- */
-#define NR_MEMCG_STOCK 7
#define FLUSHING_CACHED_CHARGE 0
-struct memcg_stock_pcp {
- local_trylock_t lock;
- uint8_t nr_pages[NR_MEMCG_STOCK];
- struct mem_cgroup *cached[NR_MEMCG_STOCK];
-
- struct work_struct work;
- unsigned long flags;
- uint8_t drain_idx;
-};
-
-static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
- .lock = INIT_LOCAL_TRYLOCK(lock),
-};
/*
* NR_OBJ_STOCK is sized so the entire hot path of obj_stock_pcp
@@ -2065,47 +2047,6 @@ static void drain_obj_stock(struct obj_stock_pcp *stock);
static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
struct mem_cgroup *root_memcg);
-/**
- * consume_stock: Try to consume stocked charge on this cpu.
- * @memcg: memcg to consume from.
- * @nr_pages: how many pages to charge.
- *
- * Consume the cached charge if enough nr_pages are present otherwise return
- * failure. Also return failure for charge request larger than
- * MEMCG_CHARGE_BATCH or if the local lock is already taken.
- *
- * returns true if successful, false otherwise.
- */
-static bool consume_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
-{
- struct memcg_stock_pcp *stock;
- uint8_t stock_pages;
- bool ret = false;
- int i;
-
- if (nr_pages > MEMCG_CHARGE_BATCH ||
- !local_trylock(&memcg_stock.lock))
- return ret;
-
- stock = this_cpu_ptr(&memcg_stock);
-
- for (i = 0; i < NR_MEMCG_STOCK; ++i) {
- if (memcg != READ_ONCE(stock->cached[i]))
- continue;
-
- stock_pages = READ_ONCE(stock->nr_pages[i]);
- if (stock_pages >= nr_pages) {
- WRITE_ONCE(stock->nr_pages[i], stock_pages - nr_pages);
- ret = true;
- }
- break;
- }
-
- local_unlock(&memcg_stock.lock);
-
- return ret;
-}
-
static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
{
page_counter_uncharge(&memcg->memory, nr_pages);
@@ -2113,51 +2054,6 @@ static void memcg_uncharge(struct mem_cgroup *memcg, unsigned int nr_pages)
page_counter_uncharge(&memcg->memsw, nr_pages);
}
-/*
- * Returns stocks cached in percpu and reset cached information.
- */
-static void drain_stock(struct memcg_stock_pcp *stock, int i)
-{
- struct mem_cgroup *old = READ_ONCE(stock->cached[i]);
- uint8_t stock_pages;
-
- if (!old)
- return;
-
- stock_pages = READ_ONCE(stock->nr_pages[i]);
- if (stock_pages) {
- memcg_uncharge(old, stock_pages);
- WRITE_ONCE(stock->nr_pages[i], 0);
- }
-
- css_put(&old->css);
- WRITE_ONCE(stock->cached[i], NULL);
-}
-
-static void drain_stock_fully(struct memcg_stock_pcp *stock)
-{
- int i;
-
- for (i = 0; i < NR_MEMCG_STOCK; ++i)
- drain_stock(stock, i);
-}
-
-static void drain_local_memcg_stock(struct work_struct *dummy)
-{
- struct memcg_stock_pcp *stock;
-
- if (WARN_ONCE(!in_task(), "drain in non-task context"))
- return;
-
- local_lock(&memcg_stock.lock);
-
- stock = this_cpu_ptr(&memcg_stock);
- drain_stock_fully(stock);
- clear_bit(FLUSHING_CACHED_CHARGE, &stock->flags);
-
- local_unlock(&memcg_stock.lock);
-}
-
static void drain_local_obj_stock(struct work_struct *dummy)
{
struct obj_stock_pcp *stock;
@@ -2174,88 +2070,6 @@ static void drain_local_obj_stock(struct work_struct *dummy)
local_unlock(&obj_stock.lock);
}
-static void refill_stock(struct mem_cgroup *memcg, unsigned int nr_pages)
-{
- struct memcg_stock_pcp *stock;
- struct mem_cgroup *cached;
- uint8_t stock_pages;
- bool success = false;
- int empty_slot = -1;
- int i;
-
- /*
- * For now limit MEMCG_CHARGE_BATCH to 127 and less. In future if we
- * decide to increase it more than 127 then we will need more careful
- * handling of nr_pages[] in struct memcg_stock_pcp.
- */
- BUILD_BUG_ON(MEMCG_CHARGE_BATCH > S8_MAX);
-
- VM_WARN_ON_ONCE(mem_cgroup_is_root(memcg));
-
- if (nr_pages > MEMCG_CHARGE_BATCH ||
- !local_trylock(&memcg_stock.lock)) {
- /*
- * In case of larger than batch refill or unlikely failure to
- * lock the percpu memcg_stock.lock, uncharge memcg directly.
- */
- memcg_uncharge(memcg, nr_pages);
- return;
- }
-
- stock = this_cpu_ptr(&memcg_stock);
- for (i = 0; i < NR_MEMCG_STOCK; ++i) {
- cached = READ_ONCE(stock->cached[i]);
- if (!cached && empty_slot == -1)
- empty_slot = i;
- if (memcg == READ_ONCE(stock->cached[i])) {
- stock_pages = READ_ONCE(stock->nr_pages[i]) + nr_pages;
- WRITE_ONCE(stock->nr_pages[i], stock_pages);
- if (stock_pages > MEMCG_CHARGE_BATCH)
- drain_stock(stock, i);
- success = true;
- break;
- }
- }
-
- if (!success) {
- i = empty_slot;
- if (i == -1) {
- i = stock->drain_idx++;
- if (stock->drain_idx == NR_MEMCG_STOCK)
- stock->drain_idx = 0;
- drain_stock(stock, i);
- }
- css_get(&memcg->css);
- WRITE_ONCE(stock->cached[i], memcg);
- WRITE_ONCE(stock->nr_pages[i], nr_pages);
- }
-
- local_unlock(&memcg_stock.lock);
-}
-
-static bool is_memcg_drain_needed(struct memcg_stock_pcp *stock,
- struct mem_cgroup *root_memcg)
-{
- struct mem_cgroup *memcg;
- bool flush = false;
- int i;
-
- rcu_read_lock();
- for (i = 0; i < NR_MEMCG_STOCK; ++i) {
- memcg = READ_ONCE(stock->cached[i]);
- if (!memcg)
- continue;
-
- if (READ_ONCE(stock->nr_pages[i]) &&
- mem_cgroup_is_descendant(memcg, root_memcg)) {
- flush = true;
- break;
- }
- }
- rcu_read_unlock();
- return flush;
-}
-
static void schedule_drain_work(int cpu, struct work_struct *work)
{
/*
--
2.53.0-Meta
^ permalink raw reply related
* Re: [PATCH 0/6 v3] mm/memcontrol, page_counter: move stock from mem_cgroup to page_counter
From: Joshua Hahn @ 2026-06-05 15:41 UTC (permalink / raw)
To: Joshua Hahn
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
Liam R . Howlett, Vlastimil Babka, Mike Rapoport,
Suren Baghdasaryan, cgroups, linux-mm, linux-kernel, kernel-team
In-Reply-To: <20260605153603.234296-1-joshua.hahnjy@gmail.com>
On Fri, 5 Jun 2026 08:35:56 -0700 Joshua Hahn <joshua.hahnjy@gmail.com> wrote:
> Memcg currently keeps a "stock" of 64 pages per-cpu to cache pre-charged
> allocations, allowing small allocations to avoid walking the expensive
> mem_cgroup hierarchy traversal and atomic operations on each charge.
> This design introduces a fastpath, but there is room for improvement:
>
> 1. Currently, each CPU tracks up to 7 (NR_MEMCG_STOCK) mem_cgroups. When
> more than 7 mem_cgroups are actively charging on a single CPU, a
> random victim is evicted and its associated stock is drained.
>
> 2. Stock management is tightly coupled to struct mem_cgroup, which makes
> it difficult to add a new page_counter to mem_cgroup and have
> multiple sources of stock management, which is required when trying
> to introduce fastpaths to multiple hard limit checks.
>
> This series moves the per-cpu stock down into the page_counter which
> consolidates stock limit checking and page_counter limit checking into
> page_counter_try_charge. This eliminates the 7-memcg-per-cpu slot limit,
> the random evictions (drain & refill), and slot traversal.
Sorry, two things that I forgot to add as reviewers' notes:
- There was a previous v3, but that was just a rebase, so I wasn't sure
how to name that / this. I decided to name this one v3, since the
last one didn't have any changes at all. I apologize in case there are
any confusions.
- I think it is quite late in the merge cycle, this is intended for the
next cycle, not this one.
Thank you!
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox