* Re: [PATCH v3] memcg: cache obj_stock by memcg, not by objcg pointer
From: Shakeel Butt @ 2026-05-19 20:11 UTC (permalink / raw)
To: Harry Yoo
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Roman Gushchin,
Muchun Song, Qi Zheng, Alexandre Ghiti, Joshua Hahn,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <ca8e655d-5fe7-4957-8a36-6667616be8b6@kernel.org>
On Wed, May 20, 2026 at 12:00:16AM +0900, Harry Yoo wrote:
>
>
[...]
> >
> > The full clean solution might take one more cycle and I think we can not just
> > ignore 67% regression on 7.1.
>
> That is valid point, unfortunately.
>
> One more thing I have to ask... for v7.1, wouldn't it be a safer option to
> revert the per-node object change and re-introduce it once we have a cleaner
> solution?
The issue with that revert is that we reintroduce all node lru locking in the
objcg reparenting path.
>
> This change was introduced in v5, but the implementation before v4 had been
> exposed in -next for a while, and I think we don't have enough justification
> to keep the per-node objcgs change, at least for v7.1, given that we have an
> unexpected last-minute regression and
> correctness concerns (albeit slight).
I am waiting for Oliver to test the multi-objcg patch I sent. If that also
resolves the regression then we have one more option i.e. backport that to 7.1
to fix the regression.
So to summarize, for future kernels we will be having multi-objcg in some form.
For 7.1, we have to decide between (1) do nothing (2) this patch or (3) backport
the multi-objcg path to 7.1.
Andrew, please don't send this patch to Linus until we decide on the option.
^ permalink raw reply
* Re: [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: T.J. Mercier @ 2026-05-19 18:07 UTC (permalink / raw)
To: Christian König
Cc: Albert Esteve, Christian Brauner, Tejun Heo, Johannes Weiner,
Michal Koutný, Jonathan Corbet, Shuah Khan, Sumit Semwal,
Michal Hocko, Roman Gushchin, Shakeel Butt, Muchun Song,
Andrew Morton, Benjamin Gaignard, Brian Starkey, John Stultz,
Paul Moore, James Morris, Serge E. Hallyn, Stephen Smalley,
Ondrej Mosnacek, Shuah Khan, cgroups, linux-doc, linux-kernel,
linux-media, dri-devel, linaro-mm-sig, linux-mm,
linux-security-module, selinux, linux-kselftest, mripard,
echanude
In-Reply-To: <01b6eefc-c107-4f8c-9d7c-3b86f54cabaa@amd.com>
On Tue, May 19, 2026 at 12:19 AM Christian König
<christian.koenig@amd.com> wrote:
>
> On 5/19/26 01:39, T.J. Mercier wrote:
> > On Mon, May 18, 2026 at 7:07 AM Christian König
> > <christian.koenig@amd.com> wrote:
> >>
> >> On 5/18/26 14:50, Albert Esteve wrote:
> >>> On Mon, May 18, 2026 at 9:20 AM Christian König
> >>> <christian.koenig@amd.com> wrote:
> >>>>
> >>>> On 5/15/26 19:06, T.J. Mercier wrote:
> >>>>> On Fri, May 15, 2026 at 6:53 AM Christian Brauner <brauner@kernel.org> wrote:
> >>>>>>
> >>>>>> On Tue, May 12, 2026 at 11:10:44AM +0200, Albert Esteve wrote:
> >>>>>>> On embedded platforms a central process often allocates dma-buf
> >>>>>>> memory on behalf of client applications. Without a way to
> >>>>>>> attribute the charge to the requesting client's cgroup, the
> >>>>>>> cost lands on the allocator, making per-cgroup memory limits
> >>>>>>> ineffective for the actual consumers.
> >>>>>>>
> >>>>>>> Add charge_pid_fd to struct dma_heap_allocation_data. When set to
> >>>>>>
> >>>>>> Please be aware that pidfds come in two flavors:
> >>>>>>
> >>>>>> thread-group pidfds and thread-specific pidfds. Make sure that your API
> >>>>>> doesn't implicitly depend on this distinction not existing.
> >>>>>
> >>>>> Hi Christian,
> >>>>>
> >>>>> Memcg is not a controller that supports "thread mode" so all threads
> >>>>> in a group should belong to the same memcg.
> >>>>
> >>>> BTW: Exactly that is the requirement automotive has with their native context use case.
> >>>>
> >>>> The use case is that you have a deamon which has multiple threads were each one is acting on behalve of some other process.
> >>>>
> >>>> At the moment we basically say they are simply not using cgroups for that use case, but it would be really nice if we could handle that as well.
> >>>>
> >>>> Summarizing the requirement of that use case: You need a different cgroup for each thread of a process.
> >>>
> >>> Hi Christian,
> >>>
> >>> Thanks for sharing this atuomotive usecase. If I understand correctly,
> >>> the actual requirement is attributing dma-buf charges to the right
> >>> client, not putting each daemon thread in a different cgroup?
> >>
> >> Nope, exactly that's the difference.
> >>
> >> The thread acts as a filtering agent for both memory allocation and command submission for somebody else, the process on which behalve the daemon does things can even be in a client VM, completely remote over some network or even something like a microcontroller.
> >>
> >> Everything the thread does regarding CPU time, GPU driver memory allocation as well as resources like GPU processing and I/O time etc.. needs to be accounted to one client which can be different for each thread of the process.
> >>
> >> The only thing which is shared with the main process thread is CPU memory resources, e.g. malloc() because that is basically just needed for housekeeping and pretty much irrelevant for this kind of use case.
> >>
> >> The problem is now you can't do that with cgroups at the moment but unfortunately only the kernel has the information you need to know to do this.
> >>
> >> So what you end up with is to define tons of interfaces just to get the necessary information from the kernel into userspace and then essentially duplicate the same infrastructure cgroup provides in the kernel in userspace again.
> >>
> >>> If so,
> >>> the `charge_pid_fd` approach achieves this directly by passing the
> >>> client's `pid_fd`, without needing to add per-thread cgroup
> >>> infrastructure.
> >>
> >> Well it's already a massive improvemt, we could basically stop doing the whole duplication part for the GPU driver stack and just use cgroups for this part.
> >>
> >> Doing that automatically for CPU and I/O time would just be nice to have additionally.
> >>
> >> Regards,
> >> Christian.
> >
> > Hopefully I'm following correctly here.... So you are duplicating the
> > GPU driver stack to achieve remote accounting on a per-thread basis?
>
> Not quite, we are duplicating the handling cgroup provides in the kernel in userspace.
>
> For this memory usage information as well as execution times of the GPU kernel driver is exposed in fdinfo for example.
Oh I see, thanks.
> > Does this mean for GPU allocations you currently have some GFP_ACCOUNT
> > magic in your driver to attribute GPU memory to the correct remote
> > client?
>
> No, we just expose what the kernel driver has allocated for itself. E.g. page tables, buffers etc...
>
> When userspace allocates something using memfd_create() for example we just ignore that.
>
> > So this series would close the gap for dma-buf allocations,
> > but what about private GPU driver memory allocated on behalf of a
> > client?
>
> Well we would need a cgroup which isn't associated with any process were we could charge the GPU driver allocations against.
>
> But good point, charging against a pid wouldn't work in this use case.
It would be pretty low overhead to put a process doing while(1)
pause(); in a separate cgroup for this purpose, but I guess a fd for
the actual cgroup would be a little cleaner in this case.
> Regards,
> Christian.
^ permalink raw reply
* Re: [Linaro-mm-sig] Re: [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: T.J. Mercier @ 2026-05-19 18:06 UTC (permalink / raw)
To: Christian König
Cc: Barry Song, Albert Esteve, Tejun Heo, Johannes Weiner,
Michal Koutný, Jonathan Corbet, Shuah Khan, Sumit Semwal,
Michal Hocko, Roman Gushchin, Shakeel Butt, Muchun Song,
Andrew Morton, Benjamin Gaignard, Brian Starkey, John Stultz,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan, cgroups, linux-doc,
linux-kernel, linux-media, dri-, linaro-mm-sig, linux-mm,
linux-security-module, selinux, linux-kselftest, mripard,
echanude
In-Reply-To: <8a13b1ad-f1be-4ef4-905e-0d9828ae8cb5@amd.com>
On Tue, May 19, 2026 at 12:10 AM Christian König
<christian.koenig@amd.com> wrote:
>
> On 5/19/26 01:00, Barry Song wrote:
> > On Mon, May 18, 2026 at 3:34 PM Christian König
> > <christian.koenig@amd.com> wrote:
> >>
> >> On 5/16/26 11:19, Barry Song wrote:
> >>> On Thu, May 14, 2026 at 12:35 AM T.J. Mercier <tjmercier@google.com> wrote:
> >>> [...]
> >>>>>> I have a question about this part. Albert I guess you are interested
> >>>>>> only in accounting dmabuf-heap allocations, or do you expect to add
> >>>>>> __GFP_ACCOUNT or mem_cgroup_charge_dmabuf calls to other
> >>>>>> non-dmabuf-heap exporters?
> >>>>>
> >>>>> We're scoping this to dma-buf heaps for now. CMA heaps and the dmem
> >>>>> controller are on the radar for follow-up/parallel work (there will be
> >>>>> dragons and will surely need discussion). For DRM and V4L2 the
> >>>>> long-term intent is migration to heaps, which would make direct
> >>>>> accounting on those paths unnecessary.
> >>>>
> >>>> Ah I see. GEM buffers exported to dmabufs are what I had in mind. I
> >>>> guess this would only leave the odd non-DRM driver with the need to
> >>>> add their own accounting calls, which I don't expect would be a big
> >>>> problem.
> >>>>
> >>>
> >>> sounds like we still have a long way to go to correctly account for
> >>> various v4l2, drm, GEM, CMA, etc. In patch 1, the charging is done in
> >>> dma_buf_export(), so I guess it covers all dma-buf types except
> >>> dma_heap, but the problem is that it has no remote charging support at
> >>> all?
> >>
> >> No, just the other way around
> >>
> >> DMA-buf heaps can be handled here because we know that it is pure system memory and nothing special so memcg always applies.
> >>
> >> dma_buf_export() on the other hand handles tons of different use cases, ranging from buffer accounted to dmem, over special resources which aren't even memory all the way to buffers which can migrate from dmem to memcg and back during their lifetime.
> >>
> >
> > Hi Christian,
> >
> > Thanks very much for your explanation. So basically it seems that
> > dma_buf_export() is not the proper place to charge, since it may end up
> > mixing in non-system-memory accounting?
>
> Yes, exactly that.
>
> > My question is also about the global view for both heap and non-heap cases.
> > After reading the discussion, I’ve tried to summarize it—please let me know
> > if my understanding is correct.
> >
> > for dma_heap, we have the ioctl DMA_HEAP_IOCTL_ALLOC, where users can pass a
> > remote pidfd or similar information to indicate where the dma-buf should be
> > charged, as in Albert's patchset.
>
> Well that's the current proposal, but I think we need to come up with something more general.
>
> > For non-dma_heap dma-bufs, we don’t have an obvious userspace entry point that
> > triggers the allocation. So we likely need other approaches. We could either
> > move more drivers over to dma-heap, or introduce something like
> > DMA_BUF_IOCTL_XFER_CHARGE, as you are discussing, to let userspace explicitly
> > declare a charge.
>
> Yeah but that's not only for DMA-buf, we need that for file descriptors returned by memfd_create() as well.
memfds get charged on fault, so an allocator shouldn't currently be
charged just for creating the fd. Unlike system/CMA heap buffers, the
shmem backing a memfd / udmabuf is LRU memory, and swapping the memcg
owner of those pages is a more-involved process which is not supported
by memcg v2. There used to be some support in memcg v1, but it was
removed. Commit e548ad4a7cbf ("mm: memcg: move charge migration code
to memcontrol-v1.c ") said, "It's a fairly large and complicated code
which created a number of problems in the past." So I'm not sure how
much appetite there would be to support it in v2 for this.
^ permalink raw reply
* Re: [RFC PATCH] cgroup/rstat: convert rstat lock from spinlock to rwlock
From: Falcon, Thomas @ 2026-05-19 18:04 UTC (permalink / raw)
To: tj@kernel.org
Cc: mkoutny@suse.com, hannes@cmpxchg.org,
linux-kernel@vger.kernel.org, cgroups@vger.kernel.org
In-Reply-To: <agyisB5nfUdDBCk5@slm.duckdns.org>
On Tue, 2026-05-19 at 07:49 -1000, Tejun Heo wrote:
> On Tue, May 19, 2026 at 12:31:34PM -0500, Thomas Falcon wrote:
> > Implement rstat_ss_lock and rstat_base_lock as read-write locks
> > instead of spinlocks. In addition, update tracing to reflect new
> > locking implementation.
> >
> > The benchmark below, meant to simulate a workload performing many
> > concurrent cgroup cpu.stat reads, completes in 134 seconds on an
> > Intel
> > Xeon Platinum 8568Y with 144 cpus compared to 241 seconds when
> > using spinlocks.
>
> Can you try using seqlock for readers? That'd be decouple readers a
> lot
> better than rwlocks.
>
> Thanks.
>
Hi, thanks for the quick response. I will try that.
Thanks,
Tom
^ permalink raw reply
* Re: [RFC PATCH] cgroup/rstat: convert rstat lock from spinlock to rwlock
From: Tejun Heo @ 2026-05-19 17:49 UTC (permalink / raw)
To: Thomas Falcon; +Cc: Johannes Weiner, Michal Koutný, cgroups, linux-kernel
In-Reply-To: <20260519173134.1486365-1-thomas.falcon@intel.com>
On Tue, May 19, 2026 at 12:31:34PM -0500, Thomas Falcon wrote:
> Implement rstat_ss_lock and rstat_base_lock as read-write locks
> instead of spinlocks. In addition, update tracing to reflect new
> locking implementation.
>
> The benchmark below, meant to simulate a workload performing many
> concurrent cgroup cpu.stat reads, completes in 134 seconds on an Intel
> Xeon Platinum 8568Y with 144 cpus compared to 241 seconds when
> using spinlocks.
Can you try using seqlock for readers? That'd be decouple readers a lot
better than rwlocks.
Thanks.
--
tejun
^ permalink raw reply
* [RFC PATCH] cgroup/rstat: convert rstat lock from spinlock to rwlock
From: Thomas Falcon @ 2026-05-19 17:31 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný; +Cc: cgroups, linux-kernel
Implement rstat_ss_lock and rstat_base_lock as read-write locks
instead of spinlocks. In addition, update tracing to reflect new
locking implementation.
The benchmark below, meant to simulate a workload performing many
concurrent cgroup cpu.stat reads, completes in 134 seconds on an Intel
Xeon Platinum 8568Y with 144 cpus compared to 241 seconds when
using spinlocks.
Signed-off-by: Thomas Falcon <thomas.falcon@intel.com>
---
Initially discussed here: https://lore.kernel.org/cgroups/20250617102644.752201-2-bertrand.wlodarczyk@intel.com/
Benchmark: https://gist.github.com/bwlodarcz/c955b36b5667f0167dffcff23953d1da
musl-gcc -o benchmark -static -g3 -DNUM_THREADS=10 -DNUM_ITER=10000 -O2 -Wall benchmark.c -pthread
---
include/linux/cgroup-defs.h | 2 +-
include/trace/events/cgroup.h | 22 ++++++++--------
kernel/cgroup/rstat.c | 47 ++++++++++++++++++++++-------------
3 files changed, 43 insertions(+), 28 deletions(-)
diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h
index 50a784da7a81..8609d6f6b29f 100644
--- a/include/linux/cgroup-defs.h
+++ b/include/linux/cgroup-defs.h
@@ -861,7 +861,7 @@ struct cgroup_subsys {
*/
unsigned int depends_on;
- spinlock_t rstat_ss_lock;
+ rwlock_t rstat_ss_lock;
struct llist_head __percpu *lhead; /* lockless update list head */
};
diff --git a/include/trace/events/cgroup.h b/include/trace/events/cgroup.h
index b736da06340a..1bd80a08c116 100644
--- a/include/trace/events/cgroup.h
+++ b/include/trace/events/cgroup.h
@@ -206,15 +206,16 @@ DEFINE_EVENT(cgroup_event, cgroup_notify_frozen,
DECLARE_EVENT_CLASS(cgroup_rstat,
- TP_PROTO(struct cgroup *cgrp, int cpu, bool contended),
+ TP_PROTO(struct cgroup *cgrp, int cpu, const char *type, bool contended),
- TP_ARGS(cgrp, cpu, contended),
+ TP_ARGS(cgrp, cpu, type, contended),
TP_STRUCT__entry(
__field( int, root )
__field( int, level )
__field( u64, id )
__field( int, cpu )
+ __string( type, type )
__field( bool, contended )
),
@@ -223,12 +224,13 @@ DECLARE_EVENT_CLASS(cgroup_rstat,
__entry->id = cgroup_id(cgrp);
__entry->level = cgrp->level;
__entry->cpu = cpu;
+ __assign_str(type);
__entry->contended = contended;
),
- TP_printk("root=%d id=%llu level=%d cpu=%d lock contended:%d",
+ TP_printk("root=%d id=%llu level=%d cpu=%d lock-type=%s lock contended:%d",
__entry->root, __entry->id, __entry->level,
- __entry->cpu, __entry->contended)
+ __entry->cpu, __get_str(type), __entry->contended)
);
/*
@@ -238,23 +240,23 @@ DECLARE_EVENT_CLASS(cgroup_rstat,
*/
DEFINE_EVENT(cgroup_rstat, cgroup_rstat_lock_contended,
- TP_PROTO(struct cgroup *cgrp, int cpu, bool contended),
+ TP_PROTO(struct cgroup *cgrp, int cpu, const char *type, bool contended),
- TP_ARGS(cgrp, cpu, contended)
+ TP_ARGS(cgrp, cpu, type, contended)
);
DEFINE_EVENT(cgroup_rstat, cgroup_rstat_locked,
- TP_PROTO(struct cgroup *cgrp, int cpu, bool contended),
+ TP_PROTO(struct cgroup *cgrp, int cpu, const char *type, bool contended),
- TP_ARGS(cgrp, cpu, contended)
+ TP_ARGS(cgrp, cpu, type, contended)
);
DEFINE_EVENT(cgroup_rstat, cgroup_rstat_unlock,
- TP_PROTO(struct cgroup *cgrp, int cpu, bool contended),
+ TP_PROTO(struct cgroup *cgrp, int cpu, const char *type, bool contended),
- TP_ARGS(cgrp, cpu, contended)
+ TP_ARGS(cgrp, cpu, type, contended)
);
#endif /* _TRACE_CGROUP_H */
diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c
index 150e5871e66f..e33f31914b7d 100644
--- a/kernel/cgroup/rstat.c
+++ b/kernel/cgroup/rstat.c
@@ -9,7 +9,7 @@
#include <trace/events/cgroup.h>
-static DEFINE_SPINLOCK(rstat_base_lock);
+static DEFINE_RWLOCK(rstat_base_lock);
static DEFINE_PER_CPU(struct llist_head, rstat_backlog_list);
static void cgroup_base_stat_flush(struct cgroup *cgrp, int cpu);
@@ -37,7 +37,7 @@ static struct cgroup_rstat_base_cpu *cgroup_rstat_base_cpu(
return per_cpu_ptr(cgrp->rstat_base_cpu, cpu);
}
-static spinlock_t *ss_rstat_lock(struct cgroup_subsys *ss)
+static rwlock_t *ss_rstat_lock(struct cgroup_subsys *ss)
{
if (ss)
return &ss->rstat_ss_lock;
@@ -356,32 +356,45 @@ __bpf_hook_end();
* number processed last.
*/
static inline void __css_rstat_lock(struct cgroup_subsys_state *css,
- int cpu_in_loop)
+ int cpu_in_loop, bool write)
__acquires(ss_rstat_lock(css->ss))
{
+ const char *type = write ? "write" : "read";
struct cgroup *cgrp = css->cgroup;
- spinlock_t *lock;
+ rwlock_t *lock;
bool contended;
lock = ss_rstat_lock(css->ss);
- contended = !spin_trylock_irq(lock);
+
+ local_irq_disable();
+ contended = write ? !write_trylock(lock) : !read_trylock(lock);
if (contended) {
- trace_cgroup_rstat_lock_contended(cgrp, cpu_in_loop, contended);
- spin_lock_irq(lock);
+ local_irq_enable();
+ trace_cgroup_rstat_lock_contended(cgrp, cpu_in_loop, type, contended);
+ if (write)
+ write_lock_irq(lock);
+ else
+ read_lock_irq(lock);
}
- trace_cgroup_rstat_locked(cgrp, cpu_in_loop, contended);
+
+ trace_cgroup_rstat_locked(cgrp, cpu_in_loop, type, contended);
}
static inline void __css_rstat_unlock(struct cgroup_subsys_state *css,
- int cpu_in_loop)
+ int cpu_in_loop, bool write)
__releases(ss_rstat_lock(css->ss))
{
+ const char *type = write ? "write" : "read";
struct cgroup *cgrp = css->cgroup;
- spinlock_t *lock;
+ rwlock_t *lock;
lock = ss_rstat_lock(css->ss);
- trace_cgroup_rstat_unlock(cgrp, cpu_in_loop, false);
- spin_unlock_irq(lock);
+ trace_cgroup_rstat_unlock(cgrp, cpu_in_loop, type, false);
+
+ if (write)
+ write_unlock_irq(lock);
+ else
+ read_unlock_irq(lock);
}
/**
@@ -414,7 +427,7 @@ __bpf_kfunc void css_rstat_flush(struct cgroup_subsys_state *css)
struct cgroup_subsys_state *pos;
/* Reacquire for each CPU to avoid disabling IRQs too long */
- __css_rstat_lock(css, cpu);
+ __css_rstat_lock(css, cpu, true);
pos = css_rstat_updated_list(css, cpu);
for (; pos; pos = pos->rstat_flush_next) {
if (is_self) {
@@ -424,7 +437,7 @@ __bpf_kfunc void css_rstat_flush(struct cgroup_subsys_state *css)
} else
pos->ss->css_rstat_flush(pos, cpu);
}
- __css_rstat_unlock(css, cpu);
+ __css_rstat_unlock(css, cpu, true);
if (!cond_resched())
cpu_relax();
}
@@ -525,7 +538,7 @@ int __init ss_rstat_init(struct cgroup_subsys *ss)
return -ENOMEM;
}
- spin_lock_init(ss_rstat_lock(ss));
+ rwlock_init(ss_rstat_lock(ss));
for_each_possible_cpu(cpu)
init_llist_head(ss_lhead_cpu(ss, cpu));
@@ -717,11 +730,11 @@ void cgroup_base_stat_cputime_show(struct seq_file *seq)
if (cgroup_parent(cgrp)) {
css_rstat_flush(&cgrp->self);
- __css_rstat_lock(&cgrp->self, -1);
+ __css_rstat_lock(&cgrp->self, -1, false);
bstat = cgrp->bstat;
cputime_adjust(&cgrp->bstat.cputime, &cgrp->prev_cputime,
&bstat.cputime.utime, &bstat.cputime.stime);
- __css_rstat_unlock(&cgrp->self, -1);
+ __css_rstat_unlock(&cgrp->self, -1, false);
} else {
root_cgroup_cputime(&bstat);
}
--
2.43.0
^ permalink raw reply related
* Re: [Linaro-mm-sig] Re: [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: Maxime Ripard @ 2026-05-19 16:32 UTC (permalink / raw)
To: Christian König
Cc: Albert Esteve, Barry Song, T.J. Mercier, Tejun Heo,
Johannes Weiner, Michal Koutný, Jonathan Corbet, Shuah Khan,
Sumit Semwal, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, Benjamin Gaignard, Brian Starkey,
John Stultz, Christian Brauner, Paul Moore, James Morris,
Serge E. Hallyn, Stephen Smalley, Ondrej Mosnacek, Shuah Khan,
cgroups, linux-doc, linux-kernel, linux-media, dri-,
linaro-mm-sig, linux-mm, linux-security-module, selinux,
linux-kselftest, echanude
In-Reply-To: <9cc79977-9a42-40eb-bfa7-460881c1e10f@amd.com>
[-- Attachment #1: Type: text/plain, Size: 6159 bytes --]
Hi Chritian,
On Tue, May 19, 2026 at 09:53:19AM +0200, Christian König wrote:
> On 5/18/26 14:06, Albert Esteve wrote:
> >>>>> udmabufs are already
> >>>>> memcg-charged, so adding a separate MEMCG_DMABUF would double count.
> >>>>> Are there any other exporters you had in mind that would benefit from
> >>>>> this approach?
> >>
> >> Well apart from DMA-buf memfd_create() is one of the things which as broken our neck in the past a couple of times.
> >>
> >> But thinking more about it what if instead of making this DMA-buf heaps specific what if we have a general cgroups function which allows to change accounting of a buffer referenced by a file descriptor to a different process?
> >>
> >> That would cover not only the DMA-buf heaps use case, but also all other DMA-buf with dmem and whatever we come up in the future as well.
> >
> > I removed a draft adding an ioctl for charge transfer from the series
> > before sending because I wanted to focus on the charge_pid_fd approach
> > and keep things simple, deferring the recharge path to a follow-up
> > depending on feedback.
> >
> > The main difference between my removed draft and what you're
> > describing, iiuc, is scope and layer: my draft was an explicit ioctl
> > on the dma-buf fd that the consumer calls to claim the charge (see
> > below), while you seem to be suggesting a more general kernel-internal
> > function that could work across buffer types and cgroup controllers,
> > so not necessarily userspace-initiated? A kernel-internal function
> > will need a way to identify the target process, which sounds similar
> > to the binder-backed approach from TJ [1]. For everything else, the
> > receiver still needs to declare itself, which the ioctl accomplishes.
> >
> > ```
> > # When an app imports a daemon-allocated buffer, it can transfer the
> > charge to itself:
> > int buf_fd = receive_dmabuf_from_daemon();
> > ioctl(buf_fd, DMA_BUF_IOCTL_XFER_CHARGE); /* charge now attributed to
> > apps's cgroup */
>
> Well that thinking goes into the right direction, but the requirements are still not completely
> covered as far as I can see.
>
> Let me explain below a bit more.
>
> >
> > [1] https://lore.kernel.org/cgroups/20230109213809.418135-1-tjmercier@google.com/
> >
> >>
> >> The only drawback I can see is that DMA-buf heap allocations would be temporarily accounted to the memory allocation daemon, but I don't think that this would be a problem.
> >
> > The main reasons we moved away from TJ's transfer-based approach
> > toward `charge_pid_fd` are: avoid the transient charge window on the
> > daemon's cgroup; and to decouple from Binder, allowing any allocator
> > to use it.
>
> Yeah those concerns are completely correct.
>
> The application should not volunteering says 'Charge that buffer to
> me.', but rather that the daemon says force charge that buffer to this
> application and tell me when the application is over its limit.
I would agree, but with a caveat: how do we want to deal with malicious
applications here? The application should have expressed that it's okay
for it to be charged by a different process, otherwise it becomes
trivial for a malicious app to create arbitrary charges against another
application in the system and DoS it.
But then, that means that an application could arbitrarily charge the
daemon as well if it doesn't opt-in but asks for allocations.
So maybe we should have an opt-in for the caller, and a way for the
daemon to check if the caller has indeed opted in before performing the
allocation (and the charge transfer)?
> > Technically, both approaches could coexist, though. Of the three
> > scenarios TJ described:
> > - Scenario 2 is directly addressed by charge_pid_fd approach without
> > any transient charge on the daemon at the cost of one extra field in
> > the heap ioctl uAPI struct.
>
> Yeah extending the uAPI to pass in the pid on allocation time is not
> much of a problem, but you also need to modify the whole stack above
> it and that is a bit more trickier.
>
> > - Scenario 3 can be handled by the charge transfer function without
> > changes to SurfaceFlinger. The app or dequeueBuffer claims the charge
> > for itself or the app, respectively (depending on whether we include a
> > pid_fd field in the transfer ioctl). It also covers non-heap
> > exporters. The con in both variants is the transient charge window on
> > the daemon.
>
> It should be trivial for the deamon to charge the buffer to an
> application before handing it out.
>
> > Both approaches shift the responsibility for correct charging
> > attribution to userspace: first, 'charge_pid_fd` on the allocator's
> > side, and the transfer charge on the consumer's side.
>
> Yeah that's why I said it would be better if we do that without any
> uAPI change, but with all the uAPI we have to transfer file
> descriptors (dup(), fork(), passing FDs over sockets etc...) it could
> be really tricky to implement that.
>
> > Deciding on one, the other or both depends on how much we value
> > avoiding transient attribution, and how much we need a non-heap
> > generic solution. With the XFER_CHARGE we can cover both. Thus, the
> > `charge_pid_fd` approach in this RFC can be seen as a
> > performance/strictness optimisation, eliminating transient charges to
> > the daemon at the cost of a permanent uAPI addition to the heap ioctl
> > struct, but not strictly required for correctness.
>
> Well all we need is a uAPI which says charge this buffer (file
> descriptor) to that cgroup (pidfd).
>
> With this at hand we should be able to handle all use cases at the
> same time.
>
> > On the other hand, if we agree on the end goal of migrating other
> > exporters to use dma-buf heaps
>
> That won't work. DMA-buf heaps is actually only a rather small and
> Anroid specific use case.
I don't think that's true anymore. heaps are used in lots of different
use cases now in the embedded space, including in regular, generic,
components not specifically used for embedded systems.
Maxime
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 273 bytes --]
^ permalink raw reply
* [PATCH v2 1/2] mm/memcontrol: add dmem charge/uncharge functions
From: Eric Chanudet @ 2026-05-19 15:59 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, Maarten Lankhorst, Maxime Ripard,
Natalie Vock, Tejun Heo, Michal Koutný, Jonathan Corbet,
Shuah Khan
Cc: cgroups, linux-mm, linux-kernel, dri-devel, T.J. Mercier,
Christian König, Maxime Ripard, Albert Esteve, Dave Airlie,
linux-doc, Eric Chanudet
In-Reply-To: <20260519-cgroup-dmem-memcg-double-charge-v2-0-db4d1407062b@redhat.com>
Add mem_cgroup_dmem_charge() and mem_cgroup_dmem_uncharge() to allow
dmem pool allocations to optionally be double-charged against the memory
controller. Take the struct cgroup from the dmem pool's css as there is
no convenient object exported to represent these allocations. These will
resolve the effective memory css from that cgroup and perform the
charge.
Introduce a MEMCG_DMEM stat counter to memory.stat to make the cgroup's
dmem charge visible.
Signed-off-by: Eric Chanudet <echanude@redhat.com>
---
include/linux/memcontrol.h | 16 ++++++++++++
mm/memcontrol.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 81 insertions(+)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index dc3fa687759b45748b2acee6d7f43da325eb50c1..8e1d49b87fb64e6114f3eb920293e14920290fe7 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -39,6 +39,7 @@ enum memcg_stat_item {
MEMCG_ZSWAP_B,
MEMCG_ZSWAPPED,
MEMCG_ZSWAP_INCOMP,
+ MEMCG_DMEM,
MEMCG_NR_STAT,
};
@@ -1872,6 +1873,21 @@ static inline bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg)
}
#endif
+#if defined(CONFIG_MEMCG) && defined(CONFIG_CGROUP_DMEM)
+bool mem_cgroup_dmem_charge(struct cgroup *cgrp, unsigned int nr_pages,
+ gfp_t gfp_mask);
+void mem_cgroup_dmem_uncharge(struct cgroup *cgrp, unsigned int nr_pages);
+#else
+static inline bool mem_cgroup_dmem_charge(struct cgroup *cgrp,
+ unsigned int nr_pages, gfp_t gfp_mask)
+{
+ return true;
+}
+static inline void mem_cgroup_dmem_uncharge(struct cgroup *cgrp,
+ unsigned int nr_pages)
+{
+}
+#endif
/* Cgroup v1-related declarations */
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index c03d4787d466803db49cdaa90e6d6ba426b7afe2..91a7ac16b6eac2d6c3700b6885a068bf8b640706 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -433,6 +433,7 @@ static const unsigned int memcg_stat_items[] = {
MEMCG_ZSWAP_B,
MEMCG_ZSWAPPED,
MEMCG_ZSWAP_INCOMP,
+ MEMCG_DMEM,
};
#define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items)
@@ -1606,6 +1607,9 @@ static const struct memory_stat memory_stats[] = {
#ifdef CONFIG_NUMA_BALANCING
{ "pgpromote_success", PGPROMOTE_SUCCESS },
#endif
+#ifdef CONFIG_CGROUP_DMEM
+ { "dmem", MEMCG_DMEM },
+#endif
};
/* The actual unit of the state item, not the same as the output unit */
@@ -5909,6 +5913,67 @@ static struct cftype zswap_files[] = {
};
#endif /* CONFIG_ZSWAP */
+#ifdef CONFIG_CGROUP_DMEM
+/**
+ * mem_cgroup_dmem_charge - charge memcg for a dmem pool allocation
+ * @cgrp: cgroup of the dmem pool
+ * @nr_pages: number of pages to charge
+ * @gfp_mask: reclaim mode
+ *
+ * Charges @nr_pages to @memcg. Returns %true if the charge fit within
+ * @memcg's configured limit, %false if it doesn't.
+ */
+bool mem_cgroup_dmem_charge(struct cgroup *cgrp, unsigned int nr_pages,
+ gfp_t gfp_mask)
+{
+ struct cgroup_subsys_state *mem_css;
+ struct mem_cgroup *memcg;
+
+ /* CGROUP_DMEM and MEMCG guarantees this cannot be NULL. */
+ mem_css = cgroup_get_e_css(cgrp, &memory_cgrp_subsys);
+
+ /* Use the memcg, if any, of the dmem cgroup. */
+ memcg = mem_cgroup_from_css(mem_css);
+ if (!memcg || mem_cgroup_is_root(memcg)) {
+ css_put(mem_css);
+ return false;
+ }
+
+ if (try_charge_memcg(memcg, gfp_mask, nr_pages)) {
+ css_put(mem_css);
+ return false;
+ }
+
+ mod_memcg_state(memcg, MEMCG_DMEM, nr_pages);
+ css_put(mem_css);
+ return true;
+}
+
+/**
+ * mem_cgroup_dmem_uncharge - uncharge memcg from a dmem pool allocation
+ * @cgrp: cgroup of the dmem pool
+ * @nr_pages: number of pages to uncharge
+ */
+void mem_cgroup_dmem_uncharge(struct cgroup *cgrp, unsigned int nr_pages)
+{
+ struct cgroup_subsys_state *mem_css;
+ struct mem_cgroup *memcg;
+
+ /* CGROUP_DMEM and MEMCG guarantees this cannot be NULL. */
+ mem_css = cgroup_get_e_css(cgrp, &memory_cgrp_subsys);
+
+ memcg = mem_cgroup_from_css(mem_css);
+ if (!memcg || mem_cgroup_is_root(memcg)) {
+ css_put(mem_css);
+ return;
+ }
+
+ mod_memcg_state(memcg, MEMCG_DMEM, -nr_pages);
+ refill_stock(memcg, nr_pages);
+ css_put(mem_css);
+}
+#endif /* CONFIG_CGROUP_DMEM */
+
static int __init mem_cgroup_swap_init(void)
{
if (mem_cgroup_disabled())
--
2.52.0
^ permalink raw reply related
* [PATCH v2 0/2] cgroup/dmem: allow double-charging dmem allocations to memcg
From: Eric Chanudet @ 2026-05-19 15:59 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, Maarten Lankhorst, Maxime Ripard,
Natalie Vock, Tejun Heo, Michal Koutný, Jonathan Corbet,
Shuah Khan
Cc: cgroups, linux-mm, linux-kernel, dri-devel, T.J. Mercier,
Christian König, Maxime Ripard, Albert Esteve, Dave Airlie,
linux-doc, Eric Chanudet
Following suggestion[1], offer a cgroupfs entry to allow an
administrator to request that a dmem controlled region also charges to
the memory controller.
Add mem_cgroup_dmem_charge/uncharge helpers to resolve the effective
cgroup from a dmem pool's cgroup, perform the charge and update a
MEMCG_DMEM stat counter.
Add a "dmem.memcg" control file at the root level to configure memcg
charging per region. The setting is disabled by default and locked on
first charge attempt.
[1] https://lore.kernel.org/all/a446b598-5041-450b-aaa9-3c39a09ff6a0@amd.com/
Signed-off-by: Eric Chanudet <echanude@redhat.com>
---
Changes in v2:
- Use mem_cgroup_dmem_{,un}charge to account for memcg pages instead of
exposing raw nr_pages functions. Use it to centralize where to find
the effective cgroup from the pool's cgroup (Johannes)
- Set depends_on for cgrp_memory if CONFIG_MEMCG by having a memory
controller in children cgroup (Michal)
- Move dmem.memcg to the root level as it applies by region for all
cgroups
- Add a dmem memory.stats entry for reporting memcg charges for dmem
allocations.
- Wrap the memcg enable/disable/lock configuration under a single state
to avoid toctou races and simplify transitions.
- Link to v1: https://lore.kernel.org/r/20260403-cgroup-dmem-memcg-double-charge-v1-0-c371d155de2a@redhat.com
---
Eric Chanudet (2):
mm/memcontrol: add dmem charge/uncharge functions
cgroup/dmem: add dmem.memcg control file for double-charging to memcg
Documentation/admin-guide/cgroup-v2.rst | 23 +++++
include/linux/memcontrol.h | 16 ++++
kernel/cgroup/dmem.c | 158 +++++++++++++++++++++++++++++++-
mm/memcontrol.c | 65 +++++++++++++
4 files changed, 259 insertions(+), 3 deletions(-)
---
base-commit: d989f135f71699294bb2ffd4726b526456e2db68
change-id: 20260327-cgroup-dmem-memcg-double-charge-0f100a9ffbf2
Best regards,
--
Eric Chanudet <echanude@redhat.com>
^ permalink raw reply
* Re: [PATCH v2 00/10] sched: Flatten the pick
From: Vincent Guittot @ 2026-05-19 16:00 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <CAKfTPtBF5UX_0_zvOpBwz9ZDZWcgGzAZC+ansyA8LVP58v6SAQ@mail.gmail.com>
On Tue, 19 May 2026 at 12:13, Vincent Guittot
<vincent.guittot@linaro.org> wrote:
>
> On Mon, 18 May 2026 at 23:12, Peter Zijlstra <peterz@infradead.org> wrote:
> >
> > On Mon, May 18, 2026 at 03:34:51PM +0200, Vincent Guittot wrote:
> > > On Wed, 13 May 2026 at 13:35, Peter Zijlstra <peterz@infradead.org> wrote:
> > > >
> > > > On Tue, May 12, 2026 at 10:42:33AM +0200, Vincent Guittot wrote:
> > > >
> > > > > I haven't reviewed the patches yet but I ran some tests with it while
> > > > > testing sched latency related changes for short slice wakeup
> > > > > preemption. I have some large hackbench regressions with this series
> > > > > on HMP system with and without EAS. those figures are unexpected
> > > > > because the benchs run on root cfs
> > > > >
> > > > > One example with hackbench 8 groups thread pipe
> > > > > tip/sched/core tip/sched/core +this patchset +this patchset
> > > > > slice 2.8ms 16ms 2.8ms 16ms
> > > > > dragonboard rb5 with EAS
> > > > > 0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
> > > > > 0,689(+/- 9.1%) +8%
> > > > >
> > > > > radxa orion6 HMP without EAS
> > > > > 0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
> > > > > 1,071(+/-5.9%) -82%
> > > > >
> > > > > Increasing the slice partly removes regressions but tis is surprising
> > > > > because the bench runs at root cfs and I thought that results will not
> > > > > change in such a case
> > > >
> > > > D'oh :/
> > > >
> > > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > > index e54da4c6c945..77d0e1937f2c 100644
> > > > --- a/kernel/sched/fair.c
> > > > +++ b/kernel/sched/fair.c
> > > > @@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
> > > > enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
> > > > struct task_struct *donor = rq->donor;
> > > > struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
> > > > - struct cfs_rq *cfs_rq = task_cfs_rq(donor);
> > > > + struct cfs_rq *cfs_rq = &rq->cfs;
> > >
> > > I tested this patch on top of the series but it doesn't fix the perf
> > > regression on rb5
> > >
> > > hackbench 8 groups thread pipe is still at 1.907(+/-7.6%) with default
> > > slice duration
> >
> > Weird, I can't reproduce anymore with this fixed :/
> >
> > I'll try more hackbench variants tomorrow I suppose.
>
> I tried several conf :
> - HMP with EAS enabled
> - HMP without EAS enabled (perf cpufreq gov)
> - SMP (only the 4 little cores)
>
> All of them show large regressions with hackbench which are almost
> recovered when increasing the slice from 2.8 to 16ms
With patch 10 the vlag value is very often set to the max 3.8ms (the
clamp value of 2.8ms slice + 1ms tick) whereas it is usually less than
a 1ms without patch 10
^ permalink raw reply
* [PATCH v2 2/2] cgroup/dmem: add dmem.memcg control file for double-charging to memcg
From: Eric Chanudet @ 2026-05-19 15:59 UTC (permalink / raw)
To: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, Maarten Lankhorst, Maxime Ripard,
Natalie Vock, Tejun Heo, Michal Koutný, Jonathan Corbet,
Shuah Khan
Cc: cgroups, linux-mm, linux-kernel, dri-devel, T.J. Mercier,
Christian König, Maxime Ripard, Albert Esteve, Dave Airlie,
linux-doc, Eric Chanudet
In-Reply-To: <20260519-cgroup-dmem-memcg-double-charge-v2-0-db4d1407062b@redhat.com>
Add a root-only cgroupfs file "dmem.memcg" that lets an administrator
configure whether allocations in a dmem region should also be charged to
the memory controller.
To handle inheritance, dmem adds a depends_on the memory controller,
unless MEMCG isn't configured in.
Double-charging is disabled by default. Once a charge is attempted, the
setting is locked to prevent inconsistent accounting by a small 4-state
machine (off, on, locked off, locked on).
The memcg to charge is derived from the pool's cgroup, since the pool
holds a reference to the dmem cgroup state that keeps the cgroup alive
until it gets uncharged.
Signed-off-by: Eric Chanudet <echanude@redhat.com>
---
Documentation/admin-guide/cgroup-v2.rst | 23 +++++
kernel/cgroup/dmem.c | 158 +++++++++++++++++++++++++++++++-
2 files changed, 178 insertions(+), 3 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 6efd0095ed995b1550317662bc1b56c7a7f3db23..1d2fa55ddf0faa17baa916a8914d3033e8e42359 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -2828,6 +2828,29 @@ DMEM Interface Files
drm/0000:03:00.0/vram0 12550144
drm/0000:03:00.0/stolen 8650752
+ dmem.memcg
+ A readwrite nested-keyed file that exists only on the root
+ cgroup. It configures whether allocations in a dmem region
+ should also be charged to the memory controller.
+
+ Upon the first charge to a region, its setting can no longer be changed
+ and is reported as "[true|false] (locked)".
+
+ Charges to the memory controller are visible in ``memory.stat`` as the
+ ``dmem`` entry, reported in bytes.
+
+ An example read output follows::
+
+ drm/0000:03:00.0/vram0 false
+ drm/0000:03:00.0/stolen false (locked)
+
+ Writing uses the same nested-keyed format::
+
+ echo "drm/0000:03:00.0/vram0 true" > dmem.memcg
+
+ This file is only available when the kernel is built with
+ ``CONFIG_MEMCG``.
+
HugeTLB
-------
diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c
index 1ab1fb47f2711ecc60dd13e611a8a4920b48f3e9..e07b20b8025c528f190f84c76b088cb8a32a7f5e 100644
--- a/kernel/cgroup/dmem.c
+++ b/kernel/cgroup/dmem.c
@@ -17,6 +17,14 @@
#include <linux/refcount.h>
#include <linux/rculist.h>
#include <linux/slab.h>
+#include <linux/memcontrol.h>
+
+enum dmem_memcg_status {
+ DMEM_MEMCG_OFF,
+ DMEM_MEMCG_ON,
+ DMEM_MEMCG_LOCKED_OFF,
+ DMEM_MEMCG_LOCKED_ON,
+};
struct dmem_cgroup_region {
/**
@@ -51,6 +59,14 @@ struct dmem_cgroup_region {
* No new pools should be added to the region afterwards.
*/
bool unregistered;
+
+ /**
+ * @memcg_status: Whether allocation in this region should charge memcg.
+ * DMEM_MEMCG_OFF/DMEM_MEMCG_ON or
+ * DMEM_MEMCG_LOCKED_OFF/DMEM_MEMCG_LOCKED_ON, frozen after first allocation.
+ * Transitions to a locked state are one-way.
+ */
+ atomic_t memcg_status;
};
struct dmemcg_state {
@@ -609,6 +625,34 @@ get_cg_pool_unlocked(struct dmemcg_state *cg, struct dmem_cgroup_region *region)
return pool;
}
+static bool apply_memcg_charge(atomic_t *status)
+{
+ int state = atomic_read(status);
+
+ for (;;) {
+ switch (state) {
+ case DMEM_MEMCG_OFF:
+ state = atomic_cmpxchg(status, DMEM_MEMCG_OFF,
+ DMEM_MEMCG_LOCKED_OFF);
+ if (state != DMEM_MEMCG_OFF)
+ continue;
+ return false;
+ case DMEM_MEMCG_LOCKED_OFF:
+ return false;
+ case DMEM_MEMCG_ON:
+ state = atomic_cmpxchg(status, DMEM_MEMCG_ON,
+ DMEM_MEMCG_LOCKED_ON);
+ if (state != DMEM_MEMCG_ON)
+ continue;
+ return true;
+ case DMEM_MEMCG_LOCKED_ON:
+ return true;
+ }
+ WARN_ONCE(1, "Invalid memcg_status (%#x).\n", state);
+ return false;
+ }
+}
+
/**
* dmem_cgroup_uncharge() - Uncharge a pool.
* @pool: Pool to uncharge.
@@ -624,6 +668,12 @@ void dmem_cgroup_uncharge(struct dmem_cgroup_pool_state *pool, u64 size)
return;
page_counter_uncharge(&pool->cnt, size);
+
+ if (atomic_read(&pool->region->memcg_status) == DMEM_MEMCG_LOCKED_ON &&
+ !WARN_ON_ONCE(size > (u64)UINT_MAX << PAGE_SHIFT))
+ mem_cgroup_dmem_uncharge(pool->cs->css.cgroup,
+ PAGE_ALIGN(size) >> PAGE_SHIFT);
+
css_put(&pool->cs->css);
dmemcg_pool_put(pool);
}
@@ -655,6 +705,8 @@ int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size,
struct dmemcg_state *cg;
struct dmem_cgroup_pool_state *pool;
struct page_counter *fail;
+ unsigned long nr_pages = PAGE_ALIGN(size) >> PAGE_SHIFT;
+ bool charge_memcg;
int ret;
*ret_pool = NULL;
@@ -670,7 +722,28 @@ int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size,
pool = get_cg_pool_unlocked(cg, region);
if (IS_ERR(pool)) {
ret = PTR_ERR(pool);
- goto err;
+ goto err_css_put;
+ }
+
+ charge_memcg = apply_memcg_charge(®ion->memcg_status);
+ if (charge_memcg) {
+ /* mem_cgroup_dmem_charge limitation from try_charge_memcg */
+ if (size > (u64)UINT_MAX << PAGE_SHIFT) {
+ ret = -EINVAL;
+ dmemcg_pool_put(pool);
+ goto err_css_put;
+ }
+
+ if (!mem_cgroup_dmem_charge(pool->cs->css.cgroup, nr_pages,
+ GFP_KERNEL)) {
+ /*
+ * No dmem_cgroup_state_evict_valuable() could help,
+ * there's no ret_limit_pool to return.
+ */
+ ret = -ENOMEM;
+ dmemcg_pool_put(pool);
+ goto err_css_put;
+ }
}
if (!page_counter_try_charge(&pool->cnt, size, &fail)) {
@@ -681,14 +754,17 @@ int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size,
}
dmemcg_pool_put(pool);
ret = -EAGAIN;
- goto err;
+ goto err_uncharge_memcg;
}
/* On success, reference from get_current_dmemcs is transferred to *ret_pool */
*ret_pool = pool;
return 0;
-err:
+err_uncharge_memcg:
+ if (charge_memcg)
+ mem_cgroup_dmem_uncharge(pool->cs->css.cgroup, nr_pages);
+err_css_put:
css_put(&cg->css);
return ret;
}
@@ -845,6 +921,71 @@ static ssize_t dmem_cgroup_region_max_write(struct kernfs_open_file *of,
return dmemcg_limit_write(of, buf, nbytes, off, set_resource_max);
}
+#ifdef CONFIG_MEMCG
+static int dmem_cgroup_memcg_show(struct seq_file *sf, void *v)
+{
+ struct dmem_cgroup_region *region;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(region, &dmem_cgroup_regions, region_node) {
+ int state = atomic_read(®ion->memcg_status);
+
+ seq_printf(sf, "%s %s\n", region->name,
+ state == DMEM_MEMCG_ON ? "true" :
+ state == DMEM_MEMCG_OFF ? "false" :
+ state == DMEM_MEMCG_LOCKED_ON ? "true (locked)" :
+ state == DMEM_MEMCG_LOCKED_OFF ? "false (locked)" :
+ "(invalid)");
+ }
+ rcu_read_unlock();
+ return 0;
+}
+
+static ssize_t dmem_cgroup_memcg_write(struct kernfs_open_file *of, char *buf,
+ size_t nbytes, loff_t off)
+{
+ while (buf) {
+ struct dmem_cgroup_region *region;
+ char *options, *name;
+ bool flag;
+
+ options = buf;
+ buf = strchr(buf, '\n');
+ if (buf)
+ *buf++ = '\0';
+
+ options = strstrip(options);
+ if (!options[0])
+ continue;
+
+ name = strsep(&options, " \t");
+ if (!name[0])
+ continue;
+
+ if (!options || !options[0])
+ return -EINVAL;
+
+ if (kstrtobool(options, &flag))
+ return -EINVAL;
+
+ rcu_read_lock();
+ region = dmemcg_get_region_by_name(name);
+ rcu_read_unlock();
+ if (!region)
+ return -ENODEV;
+
+ atomic_cmpxchg(®ion->memcg_status,
+ flag ? DMEM_MEMCG_OFF : DMEM_MEMCG_ON,
+ flag ? DMEM_MEMCG_ON : DMEM_MEMCG_OFF);
+ /* Continue if a region is already locked. */
+
+ kref_put(®ion->ref, dmemcg_free_region);
+ }
+
+ return nbytes;
+}
+#endif
+
static struct cftype files[] = {
{
.name = "capacity",
@@ -873,6 +1014,14 @@ static struct cftype files[] = {
.seq_show = dmem_cgroup_region_max_show,
.flags = CFTYPE_NOT_ON_ROOT,
},
+#ifdef CONFIG_MEMCG
+ {
+ .name = "memcg",
+ .write = dmem_cgroup_memcg_write,
+ .seq_show = dmem_cgroup_memcg_show,
+ .flags = CFTYPE_ONLY_ON_ROOT,
+ },
+#endif
{ } /* Zero entry terminates. */
};
@@ -882,4 +1031,7 @@ struct cgroup_subsys dmem_cgrp_subsys = {
.css_offline = dmemcs_offline,
.legacy_cftypes = files,
.dfl_cftypes = files,
+#ifdef CONFIG_MEMCG
+ .depends_on = 1 << memory_cgrp_id,
+#endif
};
--
2.52.0
^ permalink raw reply related
* Re: [PATCH v2 09/10] sched: Remove sched_class::pick_next_task()
From: Vincent Guittot @ 2026-05-19 15:14 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511120628.057634261@infradead.org>
On Mon, 11 May 2026 at 14:07, Peter Zijlstra <peterz@infradead.org> wrote:
>
> The reason for pick_next_task_fair() is the put/set optimization that
> avoids touching the common ancestors. However, it is possible to
> implement this in the put_prev_task() and set_next_task() calls as
> used in put_prev_set_next_task().
>
> Notably, put_prev_set_next_task() is the only site that:
>
> - calls put_prev_task() with a .next argument;
> - calls set_next_task() with .first = true.
>
> This means that put_prev_task() can determine the common hierarchy and
> stop there, and then set_next_task() can terminate where put_prev_task
> stopped.
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Hackbench results on my Arm64 dev machine stay similars with patch 8
and 9 (unlike patch 10)
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
> ---
> kernel/sched/core.c | 27 +++------
> kernel/sched/fair.c | 139 +++++++++++++++++----------------------------------
> kernel/sched/sched.h | 14 -----
> 3 files changed, 57 insertions(+), 123 deletions(-)
>
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -5980,16 +5980,15 @@ __pick_next_task(struct rq *rq, struct t
> if (likely(!sched_class_above(prev->sched_class, &fair_sched_class) &&
> rq->nr_running == rq->cfs.h_nr_queued)) {
>
> - p = pick_next_task_fair(rq, prev, rf);
> + p = pick_task_fair(rq, rf);
> if (unlikely(p == RETRY_TASK))
> goto restart;
>
> /* Assume the next prioritized class is idle_sched_class */
> - if (!p) {
> + if (!p)
> p = pick_task_idle(rq, rf);
> - put_prev_set_next_task(rq, prev, p);
> - }
>
> + put_prev_set_next_task(rq, prev, p);
> return p;
> }
>
> @@ -5997,20 +5996,12 @@ __pick_next_task(struct rq *rq, struct t
> prev_balance(rq, prev, rf);
>
> for_each_active_class(class) {
> - if (class->pick_next_task) {
> - p = class->pick_next_task(rq, prev, rf);
> - if (unlikely(p == RETRY_TASK))
> - goto restart;
> - if (p)
> - return p;
> - } else {
> - p = class->pick_task(rq, rf);
> - if (unlikely(p == RETRY_TASK))
> - goto restart;
> - if (p) {
> - put_prev_set_next_task(rq, prev, p);
> - return p;
> - }
> + p = class->pick_task(rq, rf);
> + if (unlikely(p == RETRY_TASK))
> + goto restart;
> + if (p) {
> + put_prev_set_next_task(rq, prev, p);
> + return p;
> }
> }
>
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9214,7 +9214,7 @@ static void wakeup_preempt_fair(struct r
> resched_curr_lazy(rq);
> }
>
> -static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
> +struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
> __must_hold(__rq_lockp(rq))
> {
> struct sched_entity *se;
> @@ -9257,72 +9257,6 @@ static struct task_struct *pick_task_fai
> return NULL;
> }
>
> -static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool first);
> -static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first);
> -
> -struct task_struct *
> -pick_next_task_fair(struct rq *rq, struct task_struct *prev, struct rq_flags *rf)
> - __must_hold(__rq_lockp(rq))
> -{
> - struct sched_entity *se;
> - struct task_struct *p;
> -
> - p = pick_task_fair(rq, rf);
> - if (unlikely(p == RETRY_TASK))
> - return p;
> - if (!p)
> - return p;
> - se = &p->se;
> -
> -#ifdef CONFIG_FAIR_GROUP_SCHED
> - if (prev->sched_class != &fair_sched_class)
> - goto simple;
> -
> - __put_prev_set_next_dl_server(rq, prev, p);
> -
> - /*
> - * Because of the set_next_buddy() in dequeue_task_fair() it is rather
> - * likely that a next task is from the same cgroup as the current.
> - *
> - * Therefore attempt to avoid putting and setting the entire cgroup
> - * hierarchy, only change the part that actually changes.
> - *
> - * Since we haven't yet done put_prev_entity and if the selected task
> - * is a different task than we started out with, try and touch the
> - * least amount of cfs_rqs.
> - */
> - if (prev != p) {
> - struct sched_entity *pse = &prev->se;
> - struct cfs_rq *cfs_rq;
> -
> - while (!(cfs_rq = is_same_group(se, pse))) {
> - int se_depth = se->depth;
> - int pse_depth = pse->depth;
> -
> - if (se_depth <= pse_depth) {
> - put_prev_entity(cfs_rq_of(pse), pse);
> - pse = parent_entity(pse);
> - }
> - if (se_depth >= pse_depth) {
> - set_next_entity(cfs_rq_of(se), se, true);
> - se = parent_entity(se);
> - }
> - }
> -
> - put_prev_entity(cfs_rq, pse);
> - set_next_entity(cfs_rq, se, true);
> -
> - __set_next_task_fair(rq, p, true);
> - }
> -
> - return p;
> -
> -simple:
> -#endif /* CONFIG_FAIR_GROUP_SCHED */
> - put_prev_set_next_task(rq, prev, p);
> - return p;
> -}
> -
> static struct task_struct *
> fair_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf)
> __must_hold(__rq_lockp(dl_se->rq))
> @@ -9346,10 +9280,33 @@ static void put_prev_task_fair(struct rq
> {
> struct sched_entity *se = &prev->se;
> struct cfs_rq *cfs_rq;
> + struct sched_entity *nse = NULL;
>
> - for_each_sched_entity(se) {
> +#ifdef CONFIG_FAIR_GROUP_SCHED
> + if (next && next->sched_class == &fair_sched_class)
> + nse = &next->se;
> +#endif
> +
> + while (se) {
> cfs_rq = cfs_rq_of(se);
> - put_prev_entity(cfs_rq, se);
> + if (!nse || cfs_rq->curr)
> + put_prev_entity(cfs_rq, se);
> +#ifdef CONFIG_FAIR_GROUP_SCHED
> + if (nse) {
> + if (is_same_group(se, nse))
> + break;
> +
> + int d = nse->depth - se->depth;
> + if (d >= 0) {
> + /* nse has equal or greater depth, ascend */
> + nse = parent_entity(nse);
> + /* if nse is the deeper, do not ascend se */
> + if (d > 0)
> + continue;
> + }
> + }
> +#endif
> + se = parent_entity(se);
> }
> }
>
> @@ -13896,10 +13853,30 @@ static void switched_to_fair(struct rq *
> }
> }
>
> -static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> +/*
> + * Account for a task changing its policy or group.
> + *
> + * This routine is mostly called to set cfs_rq->curr field when a task
> + * migrates between groups/classes.
> + */
> +static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> {
> struct sched_entity *se = &p->se;
>
> + for_each_sched_entity(se) {
> + struct cfs_rq *cfs_rq = cfs_rq_of(se);
> +
> + if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) &&
> + first && cfs_rq->curr)
> + break;
> +
> + set_next_entity(cfs_rq, se, first);
> + /* ensure bandwidth has been allocated on our new cfs_rq */
> + account_cfs_rq_runtime(cfs_rq, 0);
> + }
> +
> + se = &p->se;
> +
> if (task_on_rq_queued(p)) {
> /*
> * Move the next running task to the front of the list, so our
> @@ -13919,27 +13896,6 @@ static void __set_next_task_fair(struct
> sched_fair_update_stop_tick(rq, p);
> }
>
> -/*
> - * Account for a task changing its policy or group.
> - *
> - * This routine is mostly called to set cfs_rq->curr field when a task
> - * migrates between groups/classes.
> - */
> -static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> -{
> - struct sched_entity *se = &p->se;
> -
> - for_each_sched_entity(se) {
> - struct cfs_rq *cfs_rq = cfs_rq_of(se);
> -
> - set_next_entity(cfs_rq, se, first);
> - /* ensure bandwidth has been allocated on our new cfs_rq */
> - account_cfs_rq_runtime(cfs_rq, 0);
> - }
> -
> - __set_next_task_fair(rq, p, first);
> -}
> -
> void init_cfs_rq(struct cfs_rq *cfs_rq)
> {
> cfs_rq->tasks_timeline = RB_ROOT_CACHED;
> @@ -14251,7 +14207,6 @@ DEFINE_SCHED_CLASS(fair) = {
> .wakeup_preempt = wakeup_preempt_fair,
>
> .pick_task = pick_task_fair,
> - .pick_next_task = pick_next_task_fair,
> .put_prev_task = put_prev_task_fair,
> .set_next_task = set_next_task_fair,
>
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -2555,17 +2555,6 @@ struct sched_class {
> * schedule/pick_next_task: rq->lock
> */
> struct task_struct *(*pick_task)(struct rq *rq, struct rq_flags *rf);
> - /*
> - * Optional! When implemented pick_next_task() should be equivalent to:
> - *
> - * next = pick_task();
> - * if (next) {
> - * put_prev_task(prev);
> - * set_next_task_first(next);
> - * }
> - */
> - struct task_struct *(*pick_next_task)(struct rq *rq, struct task_struct *prev,
> - struct rq_flags *rf);
>
> /*
> * sched_change:
> @@ -2789,8 +2778,7 @@ static inline bool sched_fair_runnable(s
> return rq->cfs.nr_queued > 0;
> }
>
> -extern struct task_struct *pick_next_task_fair(struct rq *rq, struct task_struct *prev,
> - struct rq_flags *rf);
> +extern struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf);
> extern struct task_struct *pick_task_idle(struct rq *rq, struct rq_flags *rf);
>
> #define SCA_CHECK 0x01
>
>
^ permalink raw reply
* Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()
From: Vincent Guittot @ 2026-05-19 15:13 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511120627.944705718@infradead.org>
On Mon, 11 May 2026 at 14:07, Peter Zijlstra <peterz@infradead.org> wrote:
>
> With commit 50653216e4ff ("sched: Add support to pick functions to
> take rf") removing the balance callback, the pick_task() callback is
> in charge of newidle balancing.
>
> This means pick_task_fair() should do so too. This hasn't been a
> problem in practise because pick_next_task_fair() is used. However,
> since we'll be removing that one shortly, make sure pick_next_task()
> is up to scratch.
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Vincent Guittot <vincent.guittot@linaro.org>
> ---
> kernel/sched/fair.c | 38 +++++++++++++++-----------------------
> 1 file changed, 15 insertions(+), 23 deletions(-)
>
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9215,16 +9215,18 @@ static void wakeup_preempt_fair(struct r
> }
>
> static struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
> + __must_hold(__rq_lockp(rq))
> {
> struct sched_entity *se;
> struct cfs_rq *cfs_rq;
> struct task_struct *p;
> bool throttled;
> + int new_tasks;
>
> again:
> cfs_rq = &rq->cfs;
> if (!cfs_rq->nr_queued)
> - return NULL;
> + goto idle;
>
> throttled = false;
>
> @@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_fai
> if (unlikely(throttled))
> task_throttle_setup_work(p);
> return p;
> +
> +idle:
> + new_tasks = sched_balance_newidle(rq, rf);
> + if (new_tasks < 0)
> + return RETRY_TASK;
> + if (new_tasks > 0)
> + goto again;
> + return NULL;
> }
>
> static void __set_next_task_fair(struct rq *rq, struct task_struct *p, bool first);
> @@ -9256,12 +9266,12 @@ pick_next_task_fair(struct rq *rq, struc
> {
> struct sched_entity *se;
> struct task_struct *p;
> - int new_tasks;
>
> -again:
> p = pick_task_fair(rq, rf);
> + if (unlikely(p == RETRY_TASK))
> + return p;
> if (!p)
> - goto idle;
> + return p;
> se = &p->se;
>
> #ifdef CONFIG_FAIR_GROUP_SCHED
> @@ -9311,29 +9321,11 @@ pick_next_task_fair(struct rq *rq, struc
> #endif /* CONFIG_FAIR_GROUP_SCHED */
> put_prev_set_next_task(rq, prev, p);
> return p;
> -
> -idle:
> - if (rf) {
> - new_tasks = sched_balance_newidle(rq, rf);
> -
> - /*
> - * Because sched_balance_newidle() releases (and re-acquires)
> - * rq->lock, it is possible for any higher priority task to
> - * appear. In that case we must re-start the pick_next_entity()
> - * loop.
> - */
> - if (new_tasks < 0)
> - return RETRY_TASK;
> -
> - if (new_tasks > 0)
> - goto again;
> - }
> -
> - return NULL;
> }
>
> static struct task_struct *
> fair_server_pick_task(struct sched_dl_entity *dl_se, struct rq_flags *rf)
> + __must_hold(__rq_lockp(dl_se->rq))
> {
> return pick_task_fair(dl_se->rq, rf);
> }
>
>
^ permalink raw reply
* Re: [PATCH v3] memcg: cache obj_stock by memcg, not by objcg pointer
From: Harry Yoo @ 2026-05-19 15:00 UTC (permalink / raw)
To: Shakeel Butt
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Roman Gushchin,
Muchun Song, Qi Zheng, Alexandre Ghiti, Joshua Hahn,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <agxszIIN6FtK0fEb@linux.dev>
On 5/19/26 11:02 PM, Shakeel Butt wrote:
> On Tue, May 19, 2026 at 03:46:51PM +0900, Harry Yoo wrote:
>>
>>
>> On 5/19/26 8:41 AM, Shakeel Butt wrote:
>>> On Mon, May 18, 2026 at 03:28:27PM -0700, Shakeel Butt wrote:
>>>> Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
>>>> per-node type") split a memcg's single obj_cgroup into one per NUMA
>>>> node, but the per-CPU obj_stock_pcp still keys cached_objcg by
>>>> pointer. Cross-NUMA workloads now see a drain on every refill and a
>>>> miss on every consume that targets a sibling per-node objcg of the
>>>> same memcg, producing the 67.7% stress-ng switch-mq regression
>>>> reported by LKP.
>>>>
>>>> stock->nr_bytes are fungible across per-node objcgs of one memcg.
>>>> Treat the cache as keyed by memcg in __consume_obj_stock() and
>>>> __refill_obj_stock() so siblings share the reserve. Compare via
>>>> READ_ONCE(objcg->memcg) directly: pointer-compare only, no deref, so
>>>> the rcu_read_lock contract on obj_cgroup_memcg() does not apply.
>>>>
>>>> Sharing the reserve without re-caching means bytes funded by one
>>>> per-node objcg's slow path can be consumed/freed under a different
>>>> sibling, leaving sub-page residue on whichever sibling was cached at
>>>> drain time. The pre-existing obj_cgroup_release() path would WARN and
>>>> silently drop that residue, leaking up to nr_node_ids * (PAGE_SIZE - 1)
>>>> bytes per memcg lifecycle from the page_counter. Forward the residue
>>>> into a per-node objcg of the same (post-reparent) memcg at release time
>>>> instead, so it can be reconciled later via a refill atomic_xchg or
>>>> another release; the chain terminates at root_mem_cgroup, whose
>>>> page_counter has no enforced limit.
>>>>
>>>> Please note that this is temporary fix and will be reverted when
>>>> per-node kmem accounting is introduced.
>>
>> ... because once per-node kmem accounting is introduced,
>> "stock->nr_bytes are fungible across per-node objcgs of one memcg"
>> no longer holds?
>
> Yes
>
>>
>> And the follow-up plain is to revert this and address it with a multi-objcg
>> percpu stock [1], similar to a multi-memcg percpu charge cache we have now,
>> right? (regardless of per-node kmem accounting's progress)
>>
>
> Yes
Thanks for confirming!
>> If this temporary fix imposes other potential correctness issues, would it
>> make sense to land [1] in mainline before the next LTS release and skip this
>> temporary fix?
>>
>> [1] https://lore.kernel.org/oe-lkp/agtPMpQK2jXdQAY4@linux.dev
>>
>
> The full clean solution might take one more cycle and I think we can not just
> ignore 67% regression on 7.1.
That is valid point, unfortunately.
One more thing I have to ask... for v7.1, wouldn't it be a safer option
to revert the per-node object change and re-introduce it once we have a
cleaner solution?
This change was introduced in v5, but the implementation before v4 had
been exposed in -next for a while, and I think we don't have enough
justification to keep the per-node objcgs change, at least for v7.1,
given that we have an unexpected last-minute regression and
correctness concerns (albeit slight).
--
Cheers,
Harry / Hyeonggon
^ permalink raw reply
* Re: [linus:master] [mm] 01b9da291c: stress-ng.switch.ops_per_sec 67.7% regression
From: Shakeel Butt @ 2026-05-19 14:22 UTC (permalink / raw)
To: Oliver Sang
Cc: Qi Zheng, oe-lkp, lkp, linux-kernel, Andrew Morton, David Carlier,
Allen Pais, Axel Rasmussen, Baoquan He, Chengming Zhou,
Chen Ridong, David Hildenbrand, Hamza Mahfooz, Harry Yoo,
Hugh Dickins, Imran Khan, Johannes Weiner, Kamalesh Babulal,
Lance Yang, Liam Howlett, Lorenzo Stoakes, Michal Hocko,
Michal Koutný, Mike Rapoport, Muchun Song, Muchun Song,
Nhat Pham, Roman Gushchin, Suren Baghdasaryan, Usama Arif,
Vlastimil Babka, Wei Xu, Yosry Ahmed, Yuanchu Xie, Zi Yan,
Usama Arif, cgroups, linux-mm
In-Reply-To: <agvvRNJTAtNkCVZc@xsang-OptiPlex-9020>
Hi Oliver,
On Tue, May 19, 2026 at 01:04:04PM +0800, Oliver Sang wrote:
> hi, Shakeel,
>
> On Mon, May 18, 2026 at 10:54:20AM -0700, Shakeel Butt wrote:
> > On Mon, May 18, 2026 at 09:39:00AM -0700, Shakeel Butt wrote:
> > > On Sun, May 17, 2026 at 12:38:48PM -0700, Shakeel Butt wrote:
> > > > On Sun, May 17, 2026 at 08:55:50PM +0800, Oliver Sang wrote:
> > > > > hi, Shakeel, hi, Qi,
> > > > >
> > > > > #2: when we test above patch, we found the server easy to crash while running
> > > > > tests. we try to run up to 20 times, only 2 of them run successfully (above
> > > > > 37739220 is just the average data from these 2 runs, since the data is stable,
> > > > > we think maybe it's ok to report to you with this data).
> > > > > we also noticed for [1] there is a [syzbot ci] report in [2]. since we don't
> > > > > have serial output for our test server in this report which is for performance
> > > > > tests, we cannot say if other 18 runs failed due to similar reason. just FYI.
> > > > >
> > > >
> > > > The syzbot report is simply a rcu warning which will be fixed in v2. Do you
> > > > have more details on the crash you are seeing? Is it page counter underflow
> > > > warning?
> > > >
> > > > Thanks again for the help.
> > >
> > > Hi Oliver, it seems like sashiko found another issue with v2, so, if you have
> > > not yet started the test, you can skip it.
>
> firstly, let me still give you an update about v2. I applied it directly on top
> of 01b9da291c, found it can recover the performance.
>
> =========================================================================================
> compiler/cpufreq_governor/kconfig/method/nr_threads/rootfs/tbox_group/test/testcase/testtime:
> gcc-14/performance/x86_64-rhel-9.4/mq/100%/debian-13-x86_64-20250902.cgz/lkp-spr-r02/switch/stress-ng/60s
>
> commit:
> 8285917d6f ("mm: memcontrol: prepare for reparenting non-hierarchical stats")
> 01b9da291c ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> 8da1b1ea43 ("memcg: cache obj_stock by memcg, not by objcg pointer") <---- v2
>
> 8285917d6f383aef 01b9da291c4969354807b52956f 8da1b1ea4344c152a3892cbb132
> ---------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev
> \ | \ | \
> 5849 +210.2% 18145 ± 3% +0.8% 5896 stress-ng.switch.nanosecs_per_context_switch_mq_method
> 2.296e+09 -67.7% 7.408e+08 ± 3% -0.8% 2.278e+09 stress-ng.switch.ops
> 38288993 -67.7% 12355813 ± 3% -0.8% 37987427 stress-ng.switch.ops_per_sec
>
> but since this version is out-of-date now, I won't give out the full
> comparison. if you still want it, please let me know.
>
Thanks a lot and this is good enough.
> > >
> > > Also I am rethinking the approach, so I will send a prototype in response on
> > > this email for which I will need your help in testing.
> >
> > Hi Oliver, can you please test the following patch?
>
> got it. will change to test following patch. and this looks quite different
> with v2 or v3, so if you still want us to test v3, please let me know. thanks!
>
No need to test v3 as it is similar to v2. Please test the following patch as it
is a direction I want to pursue and wanted an early signal if this is the right
direction.
> >
> > From: Shakeel Butt <shakeel.butt@linux.dev>
> > Subject: [PATCH] memcg: shrink obj_stock_pcp and cache multiple objcgs
> >
> >
> > Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> > ---
> > mm/memcontrol.c | 213 +++++++++++++++++++++++++++++++++++-------------
> > 1 file changed, 156 insertions(+), 57 deletions(-)
> >
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index d978e18b9b2d..2a9e5136a956 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -150,14 +150,14 @@ static void obj_cgroup_release(struct percpu_ref *ref)
> > * However, it can be PAGE_SIZE or (x * PAGE_SIZE).
> > *
> > * The following sequence can lead to it:
> > - * 1) CPU0: objcg == stock->cached_objcg
> > + * 1) CPU0: objcg cached in one of stock->cached[i]
> > * 2) CPU1: we do a small allocation (e.g. 92 bytes),
> > * PAGE_SIZE bytes are charged
> > * 3) CPU1: a process from another memcg is allocating something,
> > * the stock if flushed,
> > * objcg->nr_charged_bytes = PAGE_SIZE - 92
> > * 5) CPU0: we do release this object,
> > - * 92 bytes are added to stock->nr_bytes
> > + * 92 bytes are added to stock->nr_bytes[i]
> > * 6) CPU0: stock is flushed,
> > * 92 bytes are added to objcg->nr_charged_bytes
> > *
> > @@ -2017,13 +2017,25 @@ static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
> > .lock = INIT_LOCAL_TRYLOCK(lock),
> > };
> >
> > +/*
> > + * NR_OBJ_STOCK is sized so the entire hot path of obj_stock_pcp
> > + * (lock, accounting metadata, nr_bytes[] and cached[]) fits within a
> > + * single 64-byte cache line on non-debug 64-bit builds. With 5 slots:
> > + * lock(1) + index(1) + node_id(2) + slab stats(4) + nr_bytes(10)
> > + * + pad(6) + cached(40) == 64 bytes.
> > + * A CPU can thus consume/refill/account against five different objcgs
> > + * (typically per-node variants of the same memcg) while incurring at
> > + * most one cache miss on the stock.
> > + */
> > +#define NR_OBJ_STOCK 5
> > struct obj_stock_pcp {
> > local_trylock_t lock;
> > - unsigned int nr_bytes;
> > - struct obj_cgroup *cached_objcg;
> > - struct pglist_data *cached_pgdat;
> > - int nr_slab_reclaimable_b;
> > - int nr_slab_unreclaimable_b;
> > + int8_t index;
> > + int16_t node_id;
> > + int16_t nr_slab_reclaimable_b;
> > + int16_t nr_slab_unreclaimable_b;
> > + uint16_t nr_bytes[NR_OBJ_STOCK];
> > + struct obj_cgroup *cached[NR_OBJ_STOCK];
> >
> > struct work_struct work;
> > unsigned long flags;
> > @@ -2031,10 +2043,13 @@ struct obj_stock_pcp {
> >
> > static DEFINE_PER_CPU_ALIGNED(struct obj_stock_pcp, obj_stock) = {
> > .lock = INIT_LOCAL_TRYLOCK(lock),
> > + .index = -1,
> > + .node_id = NUMA_NO_NODE,
> > };
> >
> > static DEFINE_MUTEX(percpu_charge_mutex);
> >
> > +static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i);
> > static void drain_obj_stock(struct obj_stock_pcp *stock);
> > static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
> > struct mem_cgroup *root_memcg);
> > @@ -3152,39 +3167,68 @@ static void unlock_stock(struct obj_stock_pcp *stock)
> > local_unlock(&obj_stock.lock);
> > }
> >
> > -/* Call after __refill_obj_stock() to ensure stock->cached_objg == objcg */
> > +/* Call after __refill_obj_stock() so a slot for objcg exists in the stock */
> > static void __account_obj_stock(struct obj_cgroup *objcg,
> > struct obj_stock_pcp *stock, int nr,
> > struct pglist_data *pgdat, enum node_stat_item idx)
> > {
> > - int *bytes;
> > + int16_t *bytes;
> > + int i;
> >
> > - if (!stock || READ_ONCE(stock->cached_objcg) != objcg)
> > + /*
> > + * node_id is stored as int16_t and -1 is used as the "no pgdat
> > + * cached" sentinel, so MAX_NUMNODES must fit in a positive int16_t.
> > + */
> > + BUILD_BUG_ON(MAX_NUMNODES >= S16_MAX);
> > +
> > + if (!stock)
> > + goto direct;
> > +
> > + for (i = 0; i < NR_OBJ_STOCK; ++i) {
> > + if (READ_ONCE(stock->cached[i]) == objcg)
> > + break;
> > + }
> > + if (i == NR_OBJ_STOCK)
> > goto direct;
> >
> > /*
> > * Save vmstat data in stock and skip vmstat array update unless
> > - * accumulating over a page of vmstat data or when pgdat changes.
> > + * accumulating over a page of vmstat data or when the objcg slot or
> > + * pgdat the stats belong to changes.
> > */
> > - if (stock->cached_pgdat != pgdat) {
> > - /* Flush the existing cached vmstat data */
> > - struct pglist_data *oldpg = stock->cached_pgdat;
> > + if (stock->index < 0) {
> > + stock->index = i;
> > + stock->node_id = pgdat->node_id;
> > + } else if (stock->index != i || stock->node_id != pgdat->node_id) {
> > + struct obj_cgroup *old = READ_ONCE(stock->cached[stock->index]);
> > + struct pglist_data *oldpg = NODE_DATA(stock->node_id);
> >
> > if (stock->nr_slab_reclaimable_b) {
> > - mod_objcg_mlstate(objcg, oldpg, NR_SLAB_RECLAIMABLE_B,
> > + mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
> > stock->nr_slab_reclaimable_b);
> > stock->nr_slab_reclaimable_b = 0;
> > }
> > if (stock->nr_slab_unreclaimable_b) {
> > - mod_objcg_mlstate(objcg, oldpg, NR_SLAB_UNRECLAIMABLE_B,
> > + mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
> > stock->nr_slab_unreclaimable_b);
> > stock->nr_slab_unreclaimable_b = 0;
> > }
> > - stock->cached_pgdat = pgdat;
> > + stock->index = i;
> > + stock->node_id = pgdat->node_id;
> > }
> >
> > bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b
> > : &stock->nr_slab_unreclaimable_b;
> > + /*
> > + * Cached stats are int16_t; flush directly if accumulating @nr would
> > + * overflow or underflow the cache.
> > + */
> > + if (abs(nr + *bytes) >= S16_MAX) {
> > + nr += *bytes;
> > + *bytes = 0;
> > + goto direct;
> > + }
> > +
> > /*
> > * Even for large object >= PAGE_SIZE, the vmstat data will still be
> > * cached locally at least once before pushing it out.
> > @@ -3210,10 +3254,16 @@ static bool __consume_obj_stock(struct obj_cgroup *objcg,
> > struct obj_stock_pcp *stock,
> > unsigned int nr_bytes)
> > {
> > - if (objcg == READ_ONCE(stock->cached_objcg) &&
> > - stock->nr_bytes >= nr_bytes) {
> > - stock->nr_bytes -= nr_bytes;
> > - return true;
> > + int i;
> > +
> > + for (i = 0; i < NR_OBJ_STOCK; ++i) {
> > + if (READ_ONCE(stock->cached[i]) != objcg)
> > + continue;
> > + if (stock->nr_bytes[i] >= nr_bytes) {
> > + stock->nr_bytes[i] -= nr_bytes;
> > + return true;
> > + }
> > + return false;
> > }
> >
> > return false;
> > @@ -3234,16 +3284,42 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes)
> > return ret;
> > }
> >
> > -static void drain_obj_stock(struct obj_stock_pcp *stock)
> > +/* Flush the cached slab stats (if any) back to their owning objcg/pgdat. */
> > +static void drain_obj_stock_stats(struct obj_stock_pcp *stock)
> > +{
> > + struct obj_cgroup *old;
> > + struct pglist_data *oldpg;
> > +
> > + if (stock->index < 0)
> > + return;
> > +
> > + old = READ_ONCE(stock->cached[stock->index]);
> > + oldpg = NODE_DATA(stock->node_id);
> > +
> > + if (stock->nr_slab_reclaimable_b) {
> > + mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
> > + stock->nr_slab_reclaimable_b);
> > + stock->nr_slab_reclaimable_b = 0;
> > + }
> > + if (stock->nr_slab_unreclaimable_b) {
> > + mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
> > + stock->nr_slab_unreclaimable_b);
> > + stock->nr_slab_unreclaimable_b = 0;
> > + }
> > + stock->index = -1;
> > + stock->node_id = NUMA_NO_NODE;
> > +}
> > +
> > +static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i)
> > {
> > - struct obj_cgroup *old = READ_ONCE(stock->cached_objcg);
> > + struct obj_cgroup *old = READ_ONCE(stock->cached[i]);
> >
> > if (!old)
> > return;
> >
> > - if (stock->nr_bytes) {
> > - unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT;
> > - unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1);
> > + if (stock->nr_bytes[i]) {
> > + unsigned int nr_pages = stock->nr_bytes[i] >> PAGE_SHIFT;
> > + unsigned int nr_bytes = stock->nr_bytes[i] & (PAGE_SIZE - 1);
> >
> > if (nr_pages) {
> > struct mem_cgroup *memcg;
> > @@ -3269,44 +3345,43 @@ static void drain_obj_stock(struct obj_stock_pcp *stock)
> > * so it might be changed in the future.
> > */
> > atomic_add(nr_bytes, &old->nr_charged_bytes);
> > - stock->nr_bytes = 0;
> > + stock->nr_bytes[i] = 0;
> > }
> >
> > - /*
> > - * Flush the vmstat data in current stock
> > - */
> > - if (stock->nr_slab_reclaimable_b || stock->nr_slab_unreclaimable_b) {
> > - if (stock->nr_slab_reclaimable_b) {
> > - mod_objcg_mlstate(old, stock->cached_pgdat,
> > - NR_SLAB_RECLAIMABLE_B,
> > - stock->nr_slab_reclaimable_b);
> > - stock->nr_slab_reclaimable_b = 0;
> > - }
> > - if (stock->nr_slab_unreclaimable_b) {
> > - mod_objcg_mlstate(old, stock->cached_pgdat,
> > - NR_SLAB_UNRECLAIMABLE_B,
> > - stock->nr_slab_unreclaimable_b);
> > - stock->nr_slab_unreclaimable_b = 0;
> > - }
> > - stock->cached_pgdat = NULL;
> > - }
> > + /* Flush vmstat data when its owning slot is being drained. */
> > + if (stock->index == i)
> > + drain_obj_stock_stats(stock);
> >
> > - WRITE_ONCE(stock->cached_objcg, NULL);
> > + WRITE_ONCE(stock->cached[i], NULL);
> > obj_cgroup_put(old);
> > }
> >
> > +static void drain_obj_stock(struct obj_stock_pcp *stock)
> > +{
> > + int i;
> > +
> > + for (i = 0; i < NR_OBJ_STOCK; ++i)
> > + drain_obj_stock_slot(stock, i);
> > +}
> > +
> > static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
> > struct mem_cgroup *root_memcg)
> > {
> > - struct obj_cgroup *objcg = READ_ONCE(stock->cached_objcg);
> > + struct obj_cgroup *objcg;
> > struct mem_cgroup *memcg;
> > bool flush = false;
> > + int i;
> >
> > rcu_read_lock();
> > - if (objcg) {
> > + for (i = 0; i < NR_OBJ_STOCK; ++i) {
> > + objcg = READ_ONCE(stock->cached[i]);
> > + if (!objcg)
> > + continue;
> > memcg = obj_cgroup_memcg(objcg);
> > - if (memcg && mem_cgroup_is_descendant(memcg, root_memcg))
> > + if (memcg && mem_cgroup_is_descendant(memcg, root_memcg)) {
> > flush = true;
> > + break;
> > + }
> > }
> > rcu_read_unlock();
> >
> > @@ -3319,6 +3394,8 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > bool allow_uncharge)
> > {
> > unsigned int nr_pages = 0;
> > + unsigned int stock_nr_bytes;
> > + int i, slot = -1, empty_slot = -1;
> >
> > if (!stock) {
> > nr_pages = nr_bytes >> PAGE_SHIFT;
> > @@ -3327,21 +3404,43 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > goto out;
> > }
> >
> > - if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
> > - drain_obj_stock(stock);
> > + for (i = 0; i < NR_OBJ_STOCK; ++i) {
> > + struct obj_cgroup *cached = READ_ONCE(stock->cached[i]);
> > +
> > + if (!cached) {
> > + if (empty_slot == -1)
> > + empty_slot = i;
> > + continue;
> > + }
> > + if (cached == objcg) {
> > + slot = i;
> > + break;
> > + }
> > + }
> > +
> > + if (slot == -1) {
> > + slot = empty_slot;
> > + if (slot == -1) {
> > + slot = get_random_u32_below(NR_OBJ_STOCK);
> > + drain_obj_stock_slot(stock, slot);
> > + }
> > obj_cgroup_get(objcg);
> > - stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
> > + stock->nr_bytes[slot] = atomic_read(&objcg->nr_charged_bytes)
> > ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
> > - WRITE_ONCE(stock->cached_objcg, objcg);
> > + WRITE_ONCE(stock->cached[slot], objcg);
> >
> > allow_uncharge = true; /* Allow uncharge when objcg changes */
> > }
> > - stock->nr_bytes += nr_bytes;
> >
> > - if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) {
> > - nr_pages = stock->nr_bytes >> PAGE_SHIFT;
> > - stock->nr_bytes &= (PAGE_SIZE - 1);
> > + stock_nr_bytes = (unsigned int)stock->nr_bytes[slot] + nr_bytes;
> > +
> > + /* nr_bytes[] is uint16_t; flush if we would refill >= U16_MAX. */
> > + if ((allow_uncharge && (stock_nr_bytes > PAGE_SIZE)) ||
> > + stock_nr_bytes >= U16_MAX) {
> > + nr_pages = stock_nr_bytes >> PAGE_SHIFT;
> > + stock_nr_bytes &= (PAGE_SIZE - 1);
> > }
> > + stock->nr_bytes[slot] = stock_nr_bytes;
> >
> > out:
> > if (nr_pages)
> > --
> > 2.53.0-Meta
> >
^ permalink raw reply
* Re: [PATCH v3] memcg: cache obj_stock by memcg, not by objcg pointer
From: Shakeel Butt @ 2026-05-19 14:02 UTC (permalink / raw)
To: Harry Yoo
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Roman Gushchin,
Muchun Song, Qi Zheng, Alexandre Ghiti, Joshua Hahn,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <4e296262-fbbf-4ac7-aecc-3ef831583704@kernel.org>
On Tue, May 19, 2026 at 03:46:51PM +0900, Harry Yoo wrote:
>
>
> On 5/19/26 8:41 AM, Shakeel Butt wrote:
> > On Mon, May 18, 2026 at 03:28:27PM -0700, Shakeel Butt wrote:
> > > Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
> > > per-node type") split a memcg's single obj_cgroup into one per NUMA
> > > node, but the per-CPU obj_stock_pcp still keys cached_objcg by
> > > pointer. Cross-NUMA workloads now see a drain on every refill and a
> > > miss on every consume that targets a sibling per-node objcg of the
> > > same memcg, producing the 67.7% stress-ng switch-mq regression
> > > reported by LKP.
> > >
> > > stock->nr_bytes are fungible across per-node objcgs of one memcg.
> > > Treat the cache as keyed by memcg in __consume_obj_stock() and
> > > __refill_obj_stock() so siblings share the reserve. Compare via
> > > READ_ONCE(objcg->memcg) directly: pointer-compare only, no deref, so
> > > the rcu_read_lock contract on obj_cgroup_memcg() does not apply.
> > >
> > > Sharing the reserve without re-caching means bytes funded by one
> > > per-node objcg's slow path can be consumed/freed under a different
> > > sibling, leaving sub-page residue on whichever sibling was cached at
> > > drain time. The pre-existing obj_cgroup_release() path would WARN and
> > > silently drop that residue, leaking up to nr_node_ids * (PAGE_SIZE - 1)
> > > bytes per memcg lifecycle from the page_counter. Forward the residue
> > > into a per-node objcg of the same (post-reparent) memcg at release time
> > > instead, so it can be reconciled later via a refill atomic_xchg or
> > > another release; the chain terminates at root_mem_cgroup, whose
> > > page_counter has no enforced limit.
> > >
> > > Please note that this is temporary fix and will be reverted when
> > > per-node kmem accounting is introduced.
>
> ... because once per-node kmem accounting is introduced,
> "stock->nr_bytes are fungible across per-node objcgs of one memcg"
> no longer holds?
Yes
>
> And the follow-up plain is to revert this and address it with a multi-objcg
> percpu stock [1], similar to a multi-memcg percpu charge cache we have now,
> right? (regardless of per-node kmem accounting's progress)
>
Yes
> If this temporary fix imposes other potential correctness issues, would it
> make sense to land [1] in mainline before the next LTS release and skip this
> temporary fix?
>
> [1] https://lore.kernel.org/oe-lkp/agtPMpQK2jXdQAY4@linux.dev
>
The full clean solution might take one more cycle and I think we can not just
ignore 67% regression on 7.1.
^ permalink raw reply
* Re: [PATCH] mm: memcontrol: propagate NMI slab stats to memcg vmstats
From: Harry Yoo @ 2026-05-19 11:22 UTC (permalink / raw)
To: Alexandre Ghiti, Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Vlastimil Babka, stable, linux-mm, linux-kernel,
cgroups
In-Reply-To: <20260518082830.599102-1-alex@ghiti.fr>
On 5/18/26 5:28 PM, Alexandre Ghiti wrote:
> flush_nmi_stats() drains per-node NMI slab atomics into the per-node
> lruvec_stats, but does not propagate them to the memcg-level vmstats.
>
> For non NMI case, account_slab_nmi_safe() calls mod_memcg_lruvec_state()
> which updates both per-node lruvec_stats and memcg-level vmstats, so
> flush_nmi_stats() needs to flush to per-node lruvec_stats as well as
> memcg-level vmstats.
>
> So fix this by flushing to the memcg-level vmstats for NMI too.
>
> Fixes: 940b01fc8dc1 ("memcg: nmi safe memcg stats for specific archs")
> Cc: stable@vger.kernel.org
> Acked-by: Shakeel Butt <shakeel.butt@linux.dev>
> Signed-off-by: Alexandre Ghiti <alex@ghiti.fr>
> ---
Looks good to me,
Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
--
Cheers,
Harry / Hyeonggon
^ permalink raw reply
* [PATCH v4 3/4] selftests: cgroup: Add vmtest-dmem runner script
From: Albert Esteve @ 2026-05-19 11:03 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Shuah Khan
Cc: linux-kernel, cgroups, linux-kselftest, Albert Esteve, mripard,
echanude
In-Reply-To: <20260519-kunit_cgroups-v4-0-f6c2f498fae4@redhat.com>
Currently, test_dmem relies on the dmem_selftest helper module
and a VM setup that may not have the helper preinstalled.
This makes automated coverage of dmem charge paths harder in
virtme-based runs.
Add tools/testing/selftests/cgroup/vmtest-dmem.sh to provide a
repeatable VM workflow for dmem tests. The script uses vng --exec
to run the test directly inside a virtme-ng guest with minimal
setup.
The script boots a virtme-ng guest, validates dmem controller
availability, ensures the dmem helper path is present, and runs
tools/testing/selftests/cgroup/test_dmem. If the helper is not
available as a loaded module, it attempts module build/load for
the running guest kernel before executing the test binary.
The runner also supports interactive shell mode (-s) and reuses
the verbosity and KTAP exit-code conventions used by other vmtest
scripts, so it integrates with existing kselftest workflows.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
tools/testing/selftests/cgroup/Makefile | 2 +-
tools/testing/selftests/cgroup/vmtest-dmem.sh | 156 ++++++++++++++++++++++++++
2 files changed, 157 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/cgroup/Makefile b/tools/testing/selftests/cgroup/Makefile
index e1a5e9316620e..2c407710c6e3b 100644
--- a/tools/testing/selftests/cgroup/Makefile
+++ b/tools/testing/selftests/cgroup/Makefile
@@ -3,7 +3,7 @@ CFLAGS += -Wall -pthread
all: ${HELPER_PROGS}
-TEST_FILES := with_stress.sh
+TEST_FILES := with_stress.sh vmtest-dmem.sh
TEST_PROGS := test_stress.sh test_cpuset_prs.sh test_cpuset_v1_hp.sh
TEST_GEN_FILES := wait_inotify
# Keep the lists lexicographically sorted
diff --git a/tools/testing/selftests/cgroup/vmtest-dmem.sh b/tools/testing/selftests/cgroup/vmtest-dmem.sh
new file mode 100755
index 0000000000000..b395b7153f635
--- /dev/null
+++ b/tools/testing/selftests/cgroup/vmtest-dmem.sh
@@ -0,0 +1,156 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (c) 2026 Red Hat, Inc.
+#
+# Run cgroup test_dmem inside a virtme-ng VM.
+# Dependencies:
+# * virtme-ng
+# * qemu (used by virtme-ng)
+
+set -euo pipefail
+
+readonly SCRIPT_DIR="$(cd -P -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"
+readonly KERNEL_CHECKOUT="$(realpath "${SCRIPT_DIR}"/../../../../)"
+
+source "${SCRIPT_DIR}"/../kselftest/ktap_helpers.sh
+
+QEMU="qemu-system-$(uname -m)"
+VERBOSE=0
+SHELL_MODE=0
+GUEST_TREE="${GUEST_TREE:-$KERNEL_CHECKOUT}"
+
+VM_SCRIPT=""
+
+function usage() {
+ cat <<EOF
+$0 [OPTIONS]
+Options:
+ -q QEMU binary/path (default: ${QEMU})
+ -s Start interactive shell in VM instead of running tests
+ -v Verbose output (vng boot logs on stdout)
+ -h Display this help
+EOF
+}
+
+function cleanup() {
+ rm -f "${VM_SCRIPT}"
+}
+trap cleanup EXIT
+
+function skip() {
+ local msg=${1:-""}
+
+ ktap_test_skip "${msg}"
+ exit "${KSFT_SKIP}"
+}
+
+function fail() {
+ local msg=${1:-""}
+
+ ktap_test_fail "${msg}"
+ exit "${KSFT_FAIL}"
+}
+
+function check_deps() {
+ for dep in vng "${QEMU}"; do
+ if ! command -v "${dep}" >/dev/null 2>&1; then
+ skip "dependency ${dep} not found"
+ fi
+ done
+}
+
+# Run vng with common flags. Extra arguments are appended by the caller:
+# --exec <script> for automated test runs
+# (nothing) for interactive shell mode
+function run_vm() {
+ local verbose_opt=""
+
+ [[ "${VERBOSE}" -eq 1 ]] && verbose_opt="--verbose"
+
+ vng \
+ --run \
+ ${verbose_opt:+"${verbose_opt}"} \
+ --qemu="$(command -v "${QEMU}")" \
+ --user root \
+ --rw \
+ "$@"
+}
+
+function main() {
+ while getopts ':hvq:s' opt; do
+ case "${opt}" in
+ v) VERBOSE=1 ;;
+ q) QEMU="${OPTARG}" ;;
+ s) SHELL_MODE=1 ;;
+ h) usage; exit 0 ;;
+ *) usage; exit 1 ;;
+ esac
+ done
+
+ check_deps
+
+ if [[ "${SHELL_MODE}" -eq 1 ]]; then
+ echo "Starting interactive shell in VM. Exit to stop VM."
+ run_vm
+ exit 0
+ fi
+
+ ktap_print_header
+ ktap_set_plan 1
+
+ # Write the VM-side script to a tempfile. Because vng mounts the host
+ # filesystem read-write via --rw, the guest can read it at the same path.
+ VM_SCRIPT="$(mktemp --suffix=.sh /tmp/dmem_vmtest_XXXX)"
+
+ cat > "${VM_SCRIPT}" << EOF
+#!/bin/bash
+set -euo pipefail
+
+mountpoint -q /sys/kernel/debug || mount -t debugfs none /sys/kernel/debug
+
+# Verify cgroup controllers are available.
+if ! grep -q dmem /sys/fs/cgroup/cgroup.controllers || \
+ ! grep -q memory /sys/fs/cgroup/cgroup.controllers; then
+ echo "guest kernel missing CONFIG_CGROUP_DMEM or CONFIG_MEMCG" >&2
+ exit 1
+fi
+
+# Load dmem_selftest: try built-in, then modprobe, then build + insmod.
+if [[ -e /sys/kernel/debug/dmem_selftest/charge ]]; then
+ echo "dmem_selftest ready (built-in or already loaded)"
+elif modprobe -q dmem_selftest 2>/dev/null && \
+ [[ -e /sys/kernel/debug/dmem_selftest/charge ]]; then
+ echo "dmem_selftest ready (modprobe)"
+else
+ kdir="/lib/modules/\$(uname -r)/build"
+ if [[ -d "\$kdir" ]]; then
+ echo "Building dmem_selftest.ko against running guest kernel..."
+ if make -C "\$kdir" M="${GUEST_TREE}/kernel/cgroup" \
+ CONFIG_DMEM_SELFTEST=m modules; then
+ insmod "${GUEST_TREE}/kernel/cgroup/dmem_selftest.ko" \
+ 2>/dev/null || modprobe -q dmem_selftest 2>/dev/null || true
+ fi
+ fi
+ if [[ ! -e /sys/kernel/debug/dmem_selftest/charge ]]; then
+ echo "dmem_selftest unavailable (modprobe/build+insmod failed)" >&2
+ exit 1
+ fi
+ echo "dmem_selftest ready (built + insmod)"
+fi
+
+echo "Running cgroup/test_dmem in VM..."
+cd "${GUEST_TREE}"
+make -C tools/testing/selftests TARGETS=cgroup
+./tools/testing/selftests/cgroup/test_dmem
+EOF
+
+ echo "Booting virtme-ng VM..."
+ if run_vm --exec "bash ${VM_SCRIPT}"; then
+ ktap_test_pass "test_dmem"
+ else
+ fail "test_dmem"
+ fi
+}
+
+main "$@"
--
2.53.0
^ permalink raw reply related
* [PATCH v4 4/4] selftests: cgroup: handle vmtest-dmem -b to test locally built kernel
From: Albert Esteve @ 2026-05-19 11:03 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Shuah Khan
Cc: linux-kernel, cgroups, linux-kselftest, Albert Esteve, mripard,
echanude
In-Reply-To: <20260519-kunit_cgroups-v4-0-f6c2f498fae4@redhat.com>
Currently vmtest-dmem.sh relies on the host's running kernel or a
pre-built one when booting the virtme-ng VM, with no option to
configure and build a local kernel tree directly.
This adds friction to the development cycle: the user must manually
run vng --kconfig with the correct config fragment, build the kernel,
and pass the result to the script.
Add a -b flag that automates this workflow. When set, handle_build()
configures the kernel using vng --kconfig with the selftest config
fragment, builds it with make -j$(nproc), and run_vm() passes the
local tree to vng --run so the VM boots the freshly built kernel.
Signed-off-by: Eric Chanudet <echanude@redhat.com>
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
tools/testing/selftests/cgroup/vmtest-dmem.sh | 23 ++++++++++++++++++++++-
1 file changed, 22 insertions(+), 1 deletion(-)
diff --git a/tools/testing/selftests/cgroup/vmtest-dmem.sh b/tools/testing/selftests/cgroup/vmtest-dmem.sh
index b395b7153f635..415c1722b08e7 100755
--- a/tools/testing/selftests/cgroup/vmtest-dmem.sh
+++ b/tools/testing/selftests/cgroup/vmtest-dmem.sh
@@ -15,6 +15,7 @@ readonly KERNEL_CHECKOUT="$(realpath "${SCRIPT_DIR}"/../../../../)"
source "${SCRIPT_DIR}"/../kselftest/ktap_helpers.sh
+BUILD=0
QEMU="qemu-system-$(uname -m)"
VERBOSE=0
SHELL_MODE=0
@@ -26,6 +27,7 @@ function usage() {
cat <<EOF
$0 [OPTIONS]
Options:
+ -b Build kernel from source tree before booting
-q QEMU binary/path (default: ${QEMU})
-s Start interactive shell in VM instead of running tests
-v Verbose output (vng boot logs on stdout)
@@ -60,16 +62,33 @@ function check_deps() {
done
}
+function handle_build() {
+ [[ "${BUILD}" -eq 1 ]] || return 0
+
+ [[ -d "${KERNEL_CHECKOUT}" ]] || \
+ fail "-b requires vmtest-dmem.sh called from the kernel source tree"
+
+ pushd "${KERNEL_CHECKOUT}" &>/dev/null
+ vng --kconfig --config "${SCRIPT_DIR}"/config || \
+ fail "failed to generate .config for kernel source tree (${KERNEL_CHECKOUT})"
+ make O= KBUILD_OUTPUT= -j"$(nproc)" || \
+ fail "failed to build kernel from source tree (${KERNEL_CHECKOUT})"
+ popd &>/dev/null
+}
+
# Run vng with common flags. Extra arguments are appended by the caller:
# --exec <script> for automated test runs
# (nothing) for interactive shell mode
function run_vm() {
local verbose_opt=""
+ local kernel_opt=""
[[ "${VERBOSE}" -eq 1 ]] && verbose_opt="--verbose"
+ [[ "${BUILD}" -eq 1 ]] && kernel_opt="${KERNEL_CHECKOUT}"
vng \
--run \
+ ${kernel_opt:+"${kernel_opt}"} \
${verbose_opt:+"${verbose_opt}"} \
--qemu="$(command -v "${QEMU}")" \
--user root \
@@ -78,10 +97,11 @@ function run_vm() {
}
function main() {
- while getopts ':hvq:s' opt; do
+ while getopts ':hvq:sb' opt; do
case "${opt}" in
v) VERBOSE=1 ;;
q) QEMU="${OPTARG}" ;;
+ b) BUILD=1 ;;
s) SHELL_MODE=1 ;;
h) usage; exit 0 ;;
*) usage; exit 1 ;;
@@ -89,6 +109,7 @@ function main() {
done
check_deps
+ handle_build
if [[ "${SHELL_MODE}" -eq 1 ]]; then
echo "Starting interactive shell in VM. Exit to stop VM."
--
2.53.0
^ permalink raw reply related
* [PATCH v4 2/4] selftests: cgroup: Add dmem selftest coverage
From: Albert Esteve @ 2026-05-19 11:03 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Shuah Khan
Cc: linux-kernel, cgroups, linux-kselftest, Albert Esteve, mripard,
echanude
In-Reply-To: <20260519-kunit_cgroups-v4-0-f6c2f498fae4@redhat.com>
Currently, tools/testing/selftests/cgroup/ does not include
a dmem-specific test binary. This leaves dmem charge and
limit behavior largely unvalidated in kselftest coverage.
Add test_dmem and wire it into the cgroup selftests Makefile.
The new test exercises dmem controller behavior through the
dmem_selftest debugfs interface for the dmem_selftest region.
The test adds three complementary checks:
- test_dmem_max creates a nested hierarchy with per-leaf
dmem.max values and verifies that over-limit charges
fail while in-limit charges succeed with bounded rounding
in dmem.current.
- test_dmem_min and test_dmem_low verify that charging
from a cgroup with the corresponding protection knob
set updates dmem.current as expected.
- test_dmem_charge_byte_granularity validates accounting
bounds for non-page-aligned charge sizes and
uncharge-to-zero behavior.
This provides deterministic userspace coverage for dmem
accounting and hard-limit enforcement using a test helper
module, without requiring subsystem-specific production
drivers.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
tools/testing/selftests/cgroup/.gitignore | 1 +
tools/testing/selftests/cgroup/Makefile | 2 +
tools/testing/selftests/cgroup/config | 2 +
tools/testing/selftests/cgroup/test_dmem.c | 492 +++++++++++++++++++++++++++++
4 files changed, 497 insertions(+)
diff --git a/tools/testing/selftests/cgroup/.gitignore b/tools/testing/selftests/cgroup/.gitignore
index 952e4448bf070..ea2322598217d 100644
--- a/tools/testing/selftests/cgroup/.gitignore
+++ b/tools/testing/selftests/cgroup/.gitignore
@@ -2,6 +2,7 @@
test_core
test_cpu
test_cpuset
+test_dmem
test_freezer
test_hugetlb_memcg
test_kill
diff --git a/tools/testing/selftests/cgroup/Makefile b/tools/testing/selftests/cgroup/Makefile
index e01584c2189ac..e1a5e9316620e 100644
--- a/tools/testing/selftests/cgroup/Makefile
+++ b/tools/testing/selftests/cgroup/Makefile
@@ -10,6 +10,7 @@ TEST_GEN_FILES := wait_inotify
TEST_GEN_PROGS = test_core
TEST_GEN_PROGS += test_cpu
TEST_GEN_PROGS += test_cpuset
+TEST_GEN_PROGS += test_dmem
TEST_GEN_PROGS += test_freezer
TEST_GEN_PROGS += test_hugetlb_memcg
TEST_GEN_PROGS += test_kill
@@ -26,6 +27,7 @@ include lib/libcgroup.mk
$(OUTPUT)/test_core: $(LIBCGROUP_O)
$(OUTPUT)/test_cpu: $(LIBCGROUP_O)
$(OUTPUT)/test_cpuset: $(LIBCGROUP_O)
+$(OUTPUT)/test_dmem: $(LIBCGROUP_O)
$(OUTPUT)/test_freezer: $(LIBCGROUP_O)
$(OUTPUT)/test_hugetlb_memcg: $(LIBCGROUP_O)
$(OUTPUT)/test_kill: $(LIBCGROUP_O)
diff --git a/tools/testing/selftests/cgroup/config b/tools/testing/selftests/cgroup/config
index 39f979690dd3b..5728278310a31 100644
--- a/tools/testing/selftests/cgroup/config
+++ b/tools/testing/selftests/cgroup/config
@@ -1,6 +1,8 @@
CONFIG_CGROUPS=y
CONFIG_CGROUP_CPUACCT=y
+CONFIG_CGROUP_DMEM=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_SCHED=y
+CONFIG_DMEM_SELFTEST=m
CONFIG_MEMCG=y
CONFIG_PAGE_COUNTER=y
diff --git a/tools/testing/selftests/cgroup/test_dmem.c b/tools/testing/selftests/cgroup/test_dmem.c
new file mode 100644
index 0000000000000..0a1b9561a1b9e
--- /dev/null
+++ b/tools/testing/selftests/cgroup/test_dmem.c
@@ -0,0 +1,492 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Test the dmem (device memory) cgroup controller.
+ *
+ * Depends on dmem_selftest kernel module.
+ */
+
+#define _GNU_SOURCE
+
+#include <linux/limits.h>
+#include <errno.h>
+#include <fcntl.h>
+#include <stdio.h>
+#include <stdlib.h>
+#include <string.h>
+#include <sys/stat.h>
+#include <sys/types.h>
+#include <unistd.h>
+
+#include "kselftest.h"
+#include "cgroup_util.h"
+
+/* kernel/cgroup/dmem_selftest.c */
+#define DM_SELFTEST_REGION "dmem_selftest"
+#define DM_SELFTEST_CHARGE "/sys/kernel/debug/dmem_selftest/charge"
+#define DM_SELFTEST_UNCHARGE "/sys/kernel/debug/dmem_selftest/uncharge"
+
+/*
+ * Parse the first line of dmem.capacity (root):
+ * "<name> <size_in_bytes>"
+ * Returns 1 if a region was found, 0 if capacity is empty, -1 on read error.
+ */
+static int parse_first_region(const char *root, char *name, size_t name_len,
+ unsigned long long *size_out)
+{
+ char buf[4096];
+ char nm[256];
+ unsigned long long sz;
+
+ if (cg_read(root, "dmem.capacity", buf, sizeof(buf)) < 0)
+ return -1;
+
+ if (sscanf(buf, "%255s %llu", nm, &sz) < 2)
+ return 0;
+
+ if (name_len <= strlen(nm))
+ return -1;
+
+ strcpy(name, nm);
+ *size_out = sz;
+ return 1;
+}
+
+/*
+ * Read the numeric limit for @region_name from a multiline
+ * dmem.{min,low,max} file. Returns bytes,
+ * or -1 if the line is "<name> max", or -2 if missing/err.
+ */
+static long long dmem_read_limit_for_region(const char *cgroup, const char *ctrl,
+ const char *region_name)
+{
+ char buf[4096];
+ char *line, *saveptr = NULL;
+ char fname[256];
+ char fval[64];
+
+ if (cg_read(cgroup, ctrl, buf, sizeof(buf)) < 0)
+ return -2;
+
+ for (line = strtok_r(buf, "\n", &saveptr); line;
+ line = strtok_r(NULL, "\n", &saveptr)) {
+ if (!line[0])
+ continue;
+ if (sscanf(line, "%255s %63s", fname, fval) != 2)
+ continue;
+ if (strcmp(fname, region_name))
+ continue;
+ if (!strcmp(fval, "max"))
+ return -1;
+ return strtoll(fval, NULL, 0);
+ }
+ return -2;
+}
+
+static long long dmem_read_limit(const char *cgroup, const char *ctrl)
+{
+ return dmem_read_limit_for_region(cgroup, ctrl, DM_SELFTEST_REGION);
+}
+
+static int dmem_write_limit(const char *cgroup, const char *ctrl,
+ const char *val)
+{
+ char wr[512];
+
+ snprintf(wr, sizeof(wr), "%s %s", DM_SELFTEST_REGION, val);
+ return cg_write(cgroup, ctrl, wr);
+}
+
+static int dmem_selftest_charge_bytes(unsigned long long bytes)
+{
+ char wr[32];
+
+ snprintf(wr, sizeof(wr), "%llu", bytes);
+ return write_text(DM_SELFTEST_CHARGE, wr, strlen(wr));
+}
+
+static int dmem_selftest_uncharge(void)
+{
+ return write_text(DM_SELFTEST_UNCHARGE, "\n", 1);
+}
+
+/*
+ * First, this test creates the following hierarchy:
+ * A
+ * A/B dmem.max=1M
+ * A/B/C dmem.max=75K
+ * A/B/D dmem.max=25K
+ * A/B/E dmem.max=8K
+ * A/B/F dmem.max=0
+ *
+ * Then for each leaf cgroup it tries to charge above dmem.max
+ * and expects the charge request to fail and dmem.current to
+ * remain unchanged.
+ *
+ * For leaves with non-zero dmem.max, it additionally charges a
+ * smaller amount and verifies accounting grows within one PAGE_SIZE
+ * rounding bound, then uncharges and verifies dmem.current returns
+ * to the previous value.
+ *
+ */
+static int test_dmem_max(const char *root)
+{
+ static const char * const leaf_max[] = { "75K", "25K", "8K", "0" };
+ static const unsigned long long fail_sz[] = {
+ (75ULL * 1024ULL) + 1ULL,
+ (25ULL * 1024ULL) + 1ULL,
+ (8ULL * 1024ULL) + 1ULL,
+ 1ULL
+ };
+ static const unsigned long long pass_sz[] = {
+ 4096ULL, 4096ULL, 4096ULL, 0ULL
+ };
+ char *parent[2] = {NULL};
+ char *children[4] = {NULL};
+ unsigned long long cap;
+ char region[256];
+ long long page_size;
+ long long cur_before, cur_after;
+ int ret = KSFT_FAIL;
+ int charged = 0;
+ int in_child = 0;
+ long long v;
+ int i;
+
+ if (access(DM_SELFTEST_CHARGE, W_OK) != 0)
+ return KSFT_SKIP;
+
+ if (parse_first_region(root, region, sizeof(region), &cap) != 1)
+ return KSFT_SKIP;
+ if (strcmp(region, DM_SELFTEST_REGION) != 0)
+ return KSFT_SKIP;
+
+ page_size = sysconf(_SC_PAGESIZE);
+ if (page_size <= 0)
+ goto cleanup;
+
+ parent[0] = cg_name(root, "dmem_prot_0");
+ if (!parent[0])
+ goto cleanup;
+
+ parent[1] = cg_name(parent[0], "dmem_prot_1");
+ if (!parent[1])
+ goto cleanup;
+
+ if (cg_create(parent[0]))
+ goto cleanup;
+
+ if (cg_write(parent[0], "cgroup.subtree_control", "+dmem"))
+ goto cleanup;
+
+ if (cg_create(parent[1]))
+ goto cleanup;
+
+ if (cg_write(parent[1], "cgroup.subtree_control", "+dmem"))
+ goto cleanup;
+
+ for (i = 0; i < 4; i++) {
+ children[i] = cg_name_indexed(parent[1], "dmem_child", i);
+ if (!children[i])
+ goto cleanup;
+ if (cg_create(children[i]))
+ goto cleanup;
+ }
+
+ if (dmem_write_limit(parent[1], "dmem.max", "1M"))
+ goto cleanup;
+ for (i = 0; i < 4; i++)
+ if (dmem_write_limit(children[i], "dmem.max", leaf_max[i]))
+ goto cleanup;
+
+ v = dmem_read_limit(parent[1], "dmem.max");
+ if (!values_close(v, 1024LL * 1024LL, 3))
+ goto cleanup;
+ v = dmem_read_limit(children[0], "dmem.max");
+ if (!values_close(v, 75LL * 1024LL, 3))
+ goto cleanup;
+ v = dmem_read_limit(children[1], "dmem.max");
+ if (!values_close(v, 25LL * 1024LL, 3))
+ goto cleanup;
+ v = dmem_read_limit(children[2], "dmem.max");
+ if (!values_close(v, 8LL * 1024LL, 3))
+ goto cleanup;
+ v = dmem_read_limit(children[3], "dmem.max");
+ if (v != 0)
+ goto cleanup;
+
+ for (i = 0; i < 4; i++) {
+ if (cg_enter_current(children[i]))
+ goto cleanup;
+ in_child = 1;
+
+ cur_before = dmem_read_limit(children[i], "dmem.current");
+ if (cur_before < 0)
+ goto cleanup;
+
+ if (dmem_selftest_charge_bytes(fail_sz[i]) >= 0) {
+ charged = 1;
+ goto cleanup;
+ }
+
+ cur_after = dmem_read_limit(children[i], "dmem.current");
+ if (cur_after != cur_before)
+ goto cleanup;
+
+ if (pass_sz[i] > 0) {
+ if (dmem_selftest_charge_bytes(pass_sz[i]) < 0)
+ goto cleanup;
+ charged = 1;
+
+ cur_after = dmem_read_limit(children[i], "dmem.current");
+ if (cur_after < cur_before + (long long)pass_sz[i])
+ goto cleanup;
+ if (cur_after > cur_before + (long long)pass_sz[i] + page_size)
+ goto cleanup;
+
+ if (dmem_selftest_uncharge() < 0)
+ goto cleanup;
+ charged = 0;
+
+ cur_after = dmem_read_limit(children[i], "dmem.current");
+ if (cur_after != cur_before)
+ goto cleanup;
+ }
+
+ if (cg_enter_current(root))
+ goto cleanup;
+ in_child = 0;
+ }
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (charged)
+ dmem_selftest_uncharge();
+ if (in_child)
+ cg_enter_current(root);
+ for (i = 3; i >= 0; i--) {
+ if (!children[i])
+ continue;
+ cg_destroy(children[i]);
+ free(children[i]);
+ }
+ for (i = 1; i >= 0; i--) {
+ if (!parent[i])
+ continue;
+ cg_destroy(parent[i]);
+ free(parent[i]);
+ }
+ return ret;
+}
+
+/*
+ * This test sets dmem.min and dmem.low on a child cgroup, then charge
+ * from that context and verify dmem.current tracks the charged bytes
+ * (within one page rounding).
+ */
+static int test_dmem_charge_with_attr(const char *root, bool min)
+{
+ char region[256];
+ unsigned long long cap;
+ const unsigned long long charge_sz = 12345ULL;
+ const char *attribute = min ? "dmem.min" : "dmem.low";
+ int ret = KSFT_FAIL;
+ char *cg = NULL;
+ long long cur;
+ long long page_size;
+ int charged = 0;
+ int in_child = 0;
+
+ if (access(DM_SELFTEST_CHARGE, W_OK) != 0)
+ return KSFT_SKIP;
+
+ if (parse_first_region(root, region, sizeof(region), &cap) != 1)
+ return KSFT_SKIP;
+ if (strcmp(region, DM_SELFTEST_REGION) != 0)
+ return KSFT_SKIP;
+
+ page_size = sysconf(_SC_PAGESIZE);
+ if (page_size <= 0)
+ goto cleanup;
+
+ cg = cg_name(root, "test_dmem_attr");
+ if (!cg)
+ goto cleanup;
+
+ if (cg_create(cg))
+ goto cleanup;
+
+ if (cg_enter_current(cg))
+ goto cleanup;
+ in_child = 1;
+
+ if (dmem_write_limit(cg, attribute, "16K"))
+ goto cleanup;
+
+ if (dmem_selftest_charge_bytes(charge_sz) < 0)
+ goto cleanup;
+ charged = 1;
+
+ cur = dmem_read_limit(cg, "dmem.current");
+ if (cur < (long long)charge_sz)
+ goto cleanup;
+ if (cur > (long long)charge_sz + page_size)
+ goto cleanup;
+
+ if (dmem_selftest_uncharge() < 0)
+ goto cleanup;
+ charged = 0;
+
+ cur = dmem_read_limit(cg, "dmem.current");
+ if (cur != 0)
+ goto cleanup;
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (charged)
+ dmem_selftest_uncharge();
+ if (in_child)
+ cg_enter_current(root);
+ cg_destroy(cg);
+ free(cg);
+ return ret;
+}
+
+static int test_dmem_min(const char *root)
+{
+ return test_dmem_charge_with_attr(root, true);
+}
+
+static int test_dmem_low(const char *root)
+{
+ return test_dmem_charge_with_attr(root, false);
+}
+
+/*
+ * This test charges non-page-aligned byte sizes and verify dmem.current
+ * stays consistent: it must account at least the requested bytes and
+ * never exceed one kernel page of rounding overhead. Then uncharge must
+ * return usage to 0.
+ */
+static int test_dmem_charge_byte_granularity(const char *root)
+{
+ static const unsigned long long sizes[] = { 1ULL, 4095ULL, 4097ULL, 12345ULL };
+ char *cg = NULL;
+ unsigned long long cap;
+ char region[256];
+ long long cur;
+ long long page_size;
+ int ret = KSFT_FAIL;
+ int charged = 0;
+ int in_child = 0;
+ size_t i;
+
+ if (access(DM_SELFTEST_CHARGE, W_OK) != 0)
+ return KSFT_SKIP;
+
+ if (parse_first_region(root, region, sizeof(region), &cap) != 1)
+ return KSFT_SKIP;
+ if (strcmp(region, DM_SELFTEST_REGION) != 0)
+ return KSFT_SKIP;
+
+ page_size = sysconf(_SC_PAGESIZE);
+ if (page_size <= 0)
+ goto cleanup;
+
+ cg = cg_name(root, "dmem_dbg_byte_gran");
+ if (!cg)
+ goto cleanup;
+
+ if (cg_create(cg))
+ goto cleanup;
+
+ if (dmem_write_limit(cg, "dmem.max", "8M"))
+ goto cleanup;
+
+ if (cg_enter_current(cg))
+ goto cleanup;
+ in_child = 1;
+
+ for (i = 0; i < ARRAY_SIZE(sizes); i++) {
+ if (dmem_selftest_charge_bytes(sizes[i]) < 0)
+ goto cleanup;
+ charged = 1;
+
+ cur = dmem_read_limit(cg, "dmem.current");
+ if (cur < (long long)sizes[i])
+ goto cleanup;
+ if (cur > (long long)sizes[i] + page_size)
+ goto cleanup;
+
+ if (dmem_selftest_uncharge() < 0)
+ goto cleanup;
+ charged = 0;
+
+ cur = dmem_read_limit(cg, "dmem.current");
+ if (cur != 0)
+ goto cleanup;
+ }
+
+ ret = KSFT_PASS;
+
+cleanup:
+ if (charged)
+ dmem_selftest_uncharge();
+ if (in_child)
+ cg_enter_current(root);
+ if (cg) {
+ cg_destroy(cg);
+ free(cg);
+ }
+ return ret;
+}
+
+#define T(x) { x, #x }
+struct dmem_test {
+ int (*fn)(const char *root);
+ const char *name;
+} tests[] = {
+ T(test_dmem_max),
+ T(test_dmem_min),
+ T(test_dmem_low),
+ T(test_dmem_charge_byte_granularity),
+};
+#undef T
+
+int main(int argc, char **argv)
+{
+ char root[PATH_MAX];
+ int i;
+
+ ksft_print_header();
+ ksft_set_plan(ARRAY_SIZE(tests));
+
+ if (cg_find_unified_root(root, sizeof(root), NULL))
+ ksft_exit_skip("cgroup v2 isn't mounted\n");
+
+ if (cg_read_strstr(root, "cgroup.controllers", "dmem"))
+ ksft_exit_skip("dmem controller isn't available (CONFIG_CGROUP_DMEM?)\n");
+
+ if (cg_read_strstr(root, "cgroup.subtree_control", "dmem"))
+ if (cg_write(root, "cgroup.subtree_control", "+dmem"))
+ ksft_exit_skip("Failed to enable dmem controller\n");
+
+ for (i = 0; i < ARRAY_SIZE(tests); i++) {
+ switch (tests[i].fn(root)) {
+ case KSFT_PASS:
+ ksft_test_result_pass("%s\n", tests[i].name);
+ break;
+ case KSFT_SKIP:
+ ksft_test_result_skip(
+ "%s (need CONFIG_DMEM_SELFTEST, modprobe dmem_selftest)\n",
+ tests[i].name);
+ break;
+ default:
+ ksft_test_result_fail("%s\n", tests[i].name);
+ break;
+ }
+ }
+
+ ksft_finished();
+}
--
2.53.0
^ permalink raw reply related
* [PATCH v4 1/4] cgroup: Add dmem_selftest module
From: Albert Esteve @ 2026-05-19 11:03 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Shuah Khan
Cc: linux-kernel, cgroups, linux-kselftest, Albert Esteve, mripard,
echanude
In-Reply-To: <20260519-kunit_cgroups-v4-0-f6c2f498fae4@redhat.com>
Currently, dmem charging is driver-driven through direct
calls to dmem_cgroup_try_charge(), so cgroup selftests
do not have a generic way to trigger charge and uncharge
paths from userspace.
This limits any selftest coverage to configuration/readout
checks unless a specific driver exposing charge hooks is
present in the test environment.
Add kernel/cgroup/dmem_selftest.c as a helper module
(CONFIG_DMEM_SELFTEST) that registers a synthetic dmem region
(dmem_selftest) and exposes debugfs control files:
/sys/kernel/debug/dmem_selftest/charge
/sys/kernel/debug/dmem_selftest/uncharge
Writing a size to charge triggers dmem_cgroup_try_charge() for
the calling task's cgroup (the module calls kstrtou64()).
Writing to uncharge releases the outstanding charge via
dmem_cgroup_uncharge(). Only a single outstanding charge
is supported.
This provides a deterministic, driver-independent mechanism
for exercising dmem accounting paths in selftests.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
init/Kconfig | 12 +++
kernel/cgroup/Makefile | 1 +
kernel/cgroup/dmem_selftest.c | 198 ++++++++++++++++++++++++++++++++++++++++++
3 files changed, 211 insertions(+)
diff --git a/init/Kconfig b/init/Kconfig
index 2937c4d308aec..4620ad92f72c0 100644
--- a/init/Kconfig
+++ b/init/Kconfig
@@ -1256,6 +1256,18 @@ config CGROUP_DMEM
As an example, it allows you to restrict VRAM usage for applications
in the DRM subsystem.
+config DMEM_SELFTEST
+ tristate "dmem cgroup selftest helper module"
+ depends on CGROUP_DMEM && DEBUG_FS
+ default n
+ help
+ Builds a small loadable module that registers a dmem region named
+ "dmem_selftest" and exposes debugfs files under
+ /sys/kernel/debug/dmem_selftest/ so kselftests can trigger
+ dmem charge/uncharge operations from userspace.
+
+ Say N unless you run dmem selftests or develop the dmem controller.
+
config CGROUP_FREEZER
bool "Freezer controller"
help
diff --git a/kernel/cgroup/Makefile b/kernel/cgroup/Makefile
index ede31601a363a..febc36e60f9f9 100644
--- a/kernel/cgroup/Makefile
+++ b/kernel/cgroup/Makefile
@@ -8,4 +8,5 @@ obj-$(CONFIG_CPUSETS) += cpuset.o
obj-$(CONFIG_CPUSETS_V1) += cpuset-v1.o
obj-$(CONFIG_CGROUP_MISC) += misc.o
obj-$(CONFIG_CGROUP_DMEM) += dmem.o
+obj-$(CONFIG_DMEM_SELFTEST) += dmem_selftest.o
obj-$(CONFIG_CGROUP_DEBUG) += debug.o
diff --git a/kernel/cgroup/dmem_selftest.c b/kernel/cgroup/dmem_selftest.c
new file mode 100644
index 0000000000000..d4bd44ff246f0
--- /dev/null
+++ b/kernel/cgroup/dmem_selftest.c
@@ -0,0 +1,198 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Kselftest helper for the dmem cgroup controller.
+ *
+ * Registers a dmem region and debugfs files so tests can trigger charges
+ * from the calling task's cgroup.
+ *
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <linux/cgroup_dmem.h>
+#include <linux/debugfs.h>
+#include <linux/fs.h>
+#include <linux/init.h>
+#include <linux/kernel.h>
+#include <linux/mutex.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/uaccess.h>
+
+#include "../../tools/testing/selftests/kselftest_module.h"
+
+#define DM_SELFTEST_REGION_NAME "dmem_selftest"
+#define DM_SELFTEST_REGION_SIZE (256ULL * 1024 * 1024)
+
+KSTM_MODULE_GLOBALS();
+
+static struct dmem_cgroup_region *selftest_region;
+static struct dentry *dbg_dir;
+
+static struct dmem_cgroup_pool_state *charged_pool;
+static u64 charged_size;
+static DEFINE_MUTEX(charge_lock);
+
+static ssize_t dmem_selftest_charge_write(struct file *file, const char __user *user_buf,
+ size_t count, loff_t *ppos)
+{
+ struct dmem_cgroup_pool_state *pool = NULL, *limit = NULL;
+ u64 size;
+ char buf[32];
+ int ret;
+
+ if (!selftest_region)
+ return -ENODEV;
+
+ if (count == 0 || count >= sizeof(buf))
+ return -EINVAL;
+
+ if (copy_from_user(buf, user_buf, count))
+ return -EFAULT;
+ buf[count] = '\0';
+
+ ret = kstrtou64(strim(buf), 0, &size);
+ if (ret)
+ return ret;
+ if (!size)
+ return -EINVAL;
+
+ mutex_lock(&charge_lock);
+ if (charged_pool) {
+ mutex_unlock(&charge_lock);
+ return -EBUSY;
+ }
+
+ ret = dmem_cgroup_try_charge(selftest_region, size, &pool, &limit);
+ if (ret == -EAGAIN && limit)
+ dmem_cgroup_pool_state_put(limit);
+ if (ret) {
+ mutex_unlock(&charge_lock);
+ return ret;
+ }
+
+ charged_pool = pool;
+ charged_size = size;
+ mutex_unlock(&charge_lock);
+
+ return count;
+}
+
+static ssize_t dmem_selftest_uncharge_write(struct file *file, const char __user *user_buf,
+ size_t count, loff_t *ppos)
+{
+ if (!count)
+ return -EINVAL;
+
+ mutex_lock(&charge_lock);
+ if (!charged_pool) {
+ mutex_unlock(&charge_lock);
+ return -EINVAL;
+ }
+
+ dmem_cgroup_uncharge(charged_pool, charged_size);
+ charged_pool = NULL;
+ charged_size = 0;
+ mutex_unlock(&charge_lock);
+
+ return count;
+}
+
+static const struct file_operations dmem_selftest_charge_fops = {
+ .write = dmem_selftest_charge_write,
+ .llseek = noop_llseek,
+};
+
+static const struct file_operations dmem_selftest_uncharge_fops = {
+ .write = dmem_selftest_uncharge_write,
+ .llseek = noop_llseek,
+};
+
+static int __init dmem_selftest_register(void)
+{
+ int ret = 0;
+
+ selftest_region = dmem_cgroup_register_region(
+ DM_SELFTEST_REGION_SIZE, DM_SELFTEST_REGION_NAME);
+ if (IS_ERR(selftest_region))
+ return PTR_ERR(selftest_region);
+ if (!selftest_region)
+ return -EINVAL;
+
+ dbg_dir = debugfs_create_dir("dmem_selftest", NULL);
+ if (IS_ERR(dbg_dir)) {
+ ret = PTR_ERR(dbg_dir);
+ goto dbgfs_error;
+ }
+
+ debugfs_create_file("charge", 0200, dbg_dir, NULL, &dmem_selftest_charge_fops);
+ debugfs_create_file("uncharge", 0200, dbg_dir, NULL, &dmem_selftest_uncharge_fops);
+
+ pr_info("region '%s' registered; debugfs at dmem_selftest/{charge,uncharge}\n",
+ DM_SELFTEST_REGION_NAME);
+ return ret;
+
+dbgfs_error:
+ dmem_cgroup_unregister_region(selftest_region);
+ dbg_dir = NULL;
+ selftest_region = NULL;
+ return ret;
+}
+
+static void dmem_selftest_remove(void)
+{
+ debugfs_remove_recursive(dbg_dir);
+ dbg_dir = NULL;
+
+ if (selftest_region) {
+ dmem_cgroup_unregister_region(selftest_region);
+ selftest_region = NULL;
+ }
+
+ mutex_lock(&charge_lock);
+ if (charged_pool) {
+ dmem_cgroup_uncharge(charged_pool, charged_size);
+ charged_pool = NULL;
+ }
+ mutex_unlock(&charge_lock);
+}
+
+static void __init selftest(void)
+{
+ KSTM_CHECK_ZERO(!selftest_region);
+ KSTM_CHECK_ZERO(IS_ERR_OR_NULL(dbg_dir));
+}
+
+static int __init dmem_selftest_init(void)
+{
+ int report_rc;
+ int err;
+
+ err = dmem_selftest_register();
+ if (err)
+ return err;
+
+ pr_info("loaded.\n");
+ add_taint(TAINT_TEST, LOCKDEP_STILL_OK);
+ selftest();
+ report_rc = kstm_report(total_tests, failed_tests, skipped_tests);
+ if (report_rc) {
+ dmem_selftest_remove();
+ return report_rc;
+ }
+
+ return 0;
+}
+
+static void __exit dmem_selftest_exit(void)
+{
+ pr_info("unloaded.\n");
+ dmem_selftest_remove();
+}
+
+module_init(dmem_selftest_init);
+module_exit(dmem_selftest_exit);
+
+MODULE_AUTHOR("Albert Esteve <aesteve@redhat.com>");
+MODULE_DESCRIPTION("Kselftest helper for cgroup dmem controller");
+MODULE_LICENSE("GPL");
--
2.53.0
^ permalink raw reply related
* [PATCH v4 0/4] cgroup: dmem: add selftest helper, coverage, and VM runner
From: Albert Esteve @ 2026-05-19 11:03 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Shuah Khan
Cc: linux-kernel, cgroups, linux-kselftest, Albert Esteve, mripard,
echanude
Hi all,
This small series adds practical test coverage for the dmem
cgroup controller.
The motivation came from following the recent dmem API discussion in
thread [1]. That discussion considered changing the dmem API and
adding a new knob. Currently there are no dedicated tests covering
dmem behaviour, which makes such changes riskier.
Adding selftests has an additional challenge: dmem charging paths
are driver-driven today, so regression testing is harder unless a
suitable driver is present in the test environment.
This series addresses that by adding:
- a kernel-side selftest helper module to trigger charge/uncharge
from userspace in a controlled way,
- cgroup selftests covering dmem accounting and protection semantics
(including dmem.max enforcement and byte-granularity checks),
- a virtme-based VM runner for repeatable execution of the dmem tests.
The goal is to make dmem behavior easier to validate when evolving the API
and implementation, while keeping tests deterministic and driver-independent.
Thanks.
[1] - https://lore.kernel.org/all/aZoHfloupKvF2oSu@fedora/
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
Changes in v4:
- Fix charged_pool leak in dmem_selftest_init()
- Replace ssh-based VM approach in vmtest-dmem.sh with vng --exec
- Add main() wrapper to vmtest-dmem.sh
- Other small fixes suggested by Sashiko
- Link to v3: https://lore.kernel.org/r/20260504-kunit_cgroups-v3-0-4eac90b76f91@redhat.com
Changes in v3:
- Set charged flag on unexpected over-limit charge success.
- Add CONFIG_DMEM_SELFTEST=m to selftest config.
- Simplify -v to a boolean; single -v was a no-op.
- Unquote kernel_opt to avoid empty-string arg to vng.
- Document -b in usage() output.
- Link to v2: https://lore.kernel.org/r/20260421-kunit_cgroups-v2-0-bb6675d8249c@redhat.com
Changes in v2:
- Fix debugfs_create_dir() error check
- Fix module teardown race: call dmem_selftest_remove() before
uncharging so debugfs files are torn down
- Use IS_ERR_OR_NULL() in selftest() sanity check
- Add CONFIG_CGROUP_DMEM=y to the cgroup selftest config
- Replace config-file parsing in check_guest_requirements() with
a direct check of /sys/fs/cgroup/cgroup.controllers
- Add new patch 4 (from Eric Chanudet): vmtest-dmem.sh -b flag
to configure and build a local kernel tree
- Link to v1: https://lore.kernel.org/r/20260327-kunit_cgroups-v1-0-971b3c739a00@redhat.com
---
Albert Esteve (4):
cgroup: Add dmem_selftest module
selftests: cgroup: Add dmem selftest coverage
selftests: cgroup: Add vmtest-dmem runner script
selftests: cgroup: handle vmtest-dmem -b to test locally built kernel
init/Kconfig | 12 +
kernel/cgroup/Makefile | 1 +
kernel/cgroup/dmem_selftest.c | 198 +++++++++++
tools/testing/selftests/cgroup/.gitignore | 1 +
tools/testing/selftests/cgroup/Makefile | 4 +-
tools/testing/selftests/cgroup/config | 2 +
tools/testing/selftests/cgroup/test_dmem.c | 492 ++++++++++++++++++++++++++
tools/testing/selftests/cgroup/vmtest-dmem.sh | 177 +++++++++
8 files changed, 886 insertions(+), 1 deletion(-)
---
base-commit: 70eda68668d1476b459b64e69b8f36659fa9dfa8
change-id: 20260318-kunit_cgroups-7fb0b9e64017
Best regards,
--
Albert Esteve <aesteve@redhat.com>
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Vincent Guittot @ 2026-05-19 10:38 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260511120628.206700041@infradead.org>
On Mon, 11 May 2026 at 14:07, Peter Zijlstra <peterz@infradead.org> wrote:
>
> Change fair/cgroup to a single runqueue.
>
> Infamously fair/cgroup isn't working for a number of people; typically
> the complaint is latencies and/or overhead. The latency issue is due
> to the intermediate entries that represent a combination of tasks and
> thereby obfuscate the runnability of tasks.
>
> The approach here is to leave the cgroup hierarchy as is; including
> the intermediate enqueue/dequeue but move the actual EEVDF runqueue
> outside. This means things like the shares_weight approximation are
> fully preserved.
>
> That is, given a hierarchy like:
>
> R
> |
> se--G1
> / \
> G2--se se--G3
> / \ |
> T1--se se--T2 se--T3
>
> This is fully maintained for load tracking, however the EEVDF parts of
> cfs_rq/se go unused for the intermediates and are instead connected
> like:
>
> _R_
> / | \
> T1 T2 T3
>
> Since the effective weight of the entities is determined by the
> hierarchy, this gets recomputed on enqueue,set_next_task and tick.
>
> Notably, the effective weight (se->h_load) is computed from the
> hierarchical fraction: se->load / cfs_rq->load.
>
> Since EEVDF is now exclusive operating on rq->cfs, it needs to
> consider cfs_rq->h_nr_queued rather than cfs_rq->nr_queued. Similarly,
> only tasks can get delayed, simplifying some of the cgroup cleanup.
>
> One place where additional information was required was
> set_next_task() / put_prev_task(), where we need to track 'current'
> both in the hierarchical sense (cfs_rq->h_curr) and in the flat sense
> (cfs_rq->curr).
>
> As a result of only having a single level to pick from, much of the
> complications in pick_next_task() and preemption go away.
>
> Since many of the hierarchical operations are still there, this won't
> immediately fix the performance issues, but hopefully it will fix some
> of the latency issues.
>
> TODO: split struct cfs_rq / struct sched_entity
> TODO: try and get rid of h_curr
>
> Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
> ---
> include/linux/sched.h | 1
> kernel/sched/core.c | 5
> kernel/sched/debug.c | 9
> kernel/sched/fair.c | 789 +++++++++++++++++++++-----------------------------
> kernel/sched/pelt.c | 6
> kernel/sched/sched.h | 26 -
> 6 files changed, 366 insertions(+), 470 deletions(-)
>
> --- a/include/linux/sched.h
> +++ b/include/linux/sched.h
> @@ -575,6 +575,7 @@ struct sched_statistics {
> struct sched_entity {
> /* For load-balancing: */
> struct load_weight load;
> + struct load_weight h_load;
> struct rb_node run_node;
> u64 deadline;
> u64 min_vruntime;
> --- a/kernel/sched/core.c
> +++ b/kernel/sched/core.c
> @@ -5539,11 +5539,8 @@ EXPORT_PER_CPU_SYMBOL(kernel_cpustat);
> */
> static inline void prefetch_curr_exec_start(struct task_struct *p)
> {
> -#ifdef CONFIG_FAIR_GROUP_SCHED
> - struct sched_entity *curr = p->se.cfs_rq->curr;
> -#else
> struct sched_entity *curr = task_rq(p)->cfs.curr;
> -#endif
> +
> prefetch(curr);
> prefetch(&curr->exec_start);
> }
> --- a/kernel/sched/debug.c
> +++ b/kernel/sched/debug.c
> @@ -911,10 +911,11 @@ print_task(struct seq_file *m, struct rq
> else
> SEQ_printf(m, " %c", task_state_to_char(p));
>
> - SEQ_printf(m, " %15s %5d %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ",
> + SEQ_printf(m, " %15s %5d %10ld %9Ld.%06ld %c %9Ld.%06ld %c %9Ld.%06ld %9Ld.%06ld %9Ld %5d ",
> p->comm, task_pid_nr(p),
> + p->se.h_load.weight,
> SPLIT_NS(p->se.vruntime),
> - entity_eligible(cfs_rq_of(&p->se), &p->se) ? 'E' : 'N',
> + entity_eligible(&rq->cfs, &p->se) ? 'E' : 'N',
> SPLIT_NS(p->se.deadline),
> p->se.custom_slice ? 'S' : ' ',
> SPLIT_NS(p->se.slice),
> @@ -943,7 +944,7 @@ static void print_rq(struct seq_file *m,
>
> SEQ_printf(m, "\n");
> SEQ_printf(m, "runnable tasks:\n");
> - SEQ_printf(m, " S task PID vruntime eligible "
> + SEQ_printf(m, " S task PID weight vruntime eligible "
> "deadline slice sum-exec switches "
> "prio wait-time sum-sleep sum-block"
> #ifdef CONFIG_NUMA_BALANCING
> @@ -1051,6 +1052,8 @@ void print_cfs_rq(struct seq_file *m, in
> cfs_rq->tg_load_avg_contrib);
> SEQ_printf(m, " .%-30s: %ld\n", "tg_load_avg",
> atomic_long_read(&cfs_rq->tg->load_avg));
> + SEQ_printf(m, " .%-30s: %lu\n", "h_load",
> + cfs_rq->h_load);
> #endif /* CONFIG_FAIR_GROUP_SCHED */
> #ifdef CONFIG_CFS_BANDWIDTH
> SEQ_printf(m, " .%-30s: %d\n", "throttled",
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -296,8 +296,8 @@ static u64 __calc_delta(u64 delta_exec,
> */
> static inline u64 calc_delta_fair(u64 delta, struct sched_entity *se)
> {
> - if (unlikely(se->load.weight != NICE_0_LOAD))
> - delta = __calc_delta(delta, NICE_0_LOAD, &se->load);
> + if (se->h_load.weight != NICE_0_LOAD)
> + delta = __calc_delta(delta, NICE_0_LOAD, &se->h_load);
>
> return delta;
> }
> @@ -427,38 +427,6 @@ static inline struct sched_entity *paren
> return se->parent;
> }
>
> -static void
> -find_matching_se(struct sched_entity **se, struct sched_entity **pse)
> -{
> - int se_depth, pse_depth;
> -
> - /*
> - * preemption test can be made between sibling entities who are in the
> - * same cfs_rq i.e who have a common parent. Walk up the hierarchy of
> - * both tasks until we find their ancestors who are siblings of common
> - * parent.
> - */
> -
> - /* First walk up until both entities are at same depth */
> - se_depth = (*se)->depth;
> - pse_depth = (*pse)->depth;
> -
> - while (se_depth > pse_depth) {
> - se_depth--;
> - *se = parent_entity(*se);
> - }
> -
> - while (pse_depth > se_depth) {
> - pse_depth--;
> - *pse = parent_entity(*pse);
> - }
> -
> - while (!is_same_group(*se, *pse)) {
> - *se = parent_entity(*se);
> - *pse = parent_entity(*pse);
> - }
> -}
> -
> static int tg_is_idle(struct task_group *tg)
> {
> return tg->idle > 0;
> @@ -502,11 +470,6 @@ static inline struct sched_entity *paren
> return NULL;
> }
>
> -static inline void
> -find_matching_se(struct sched_entity **se, struct sched_entity **pse)
> -{
> -}
> -
> static inline int tg_is_idle(struct task_group *tg)
> {
> return 0;
> @@ -685,7 +648,7 @@ static inline unsigned long avg_vruntime
> static inline void
> __sum_w_vruntime_add(struct cfs_rq *cfs_rq, struct sched_entity *se)
> {
> - unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
> + unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
> s64 w_vruntime, key = entity_key(cfs_rq, se);
>
> w_vruntime = key * weight;
> @@ -702,7 +665,7 @@ sum_w_vruntime_add_paranoid(struct cfs_r
> s64 key, tmp;
>
> again:
> - weight = avg_vruntime_weight(cfs_rq, se->load.weight);
> + weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
> key = entity_key(cfs_rq, se);
>
> if (check_mul_overflow(key, weight, &key))
> @@ -748,7 +711,7 @@ sum_w_vruntime_add(struct cfs_rq *cfs_rq
> static void
> sum_w_vruntime_sub(struct cfs_rq *cfs_rq, struct sched_entity *se)
> {
> - unsigned long weight = avg_vruntime_weight(cfs_rq, se->load.weight);
> + unsigned long weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
> s64 key = entity_key(cfs_rq, se);
>
> cfs_rq->sum_w_vruntime -= key * weight;
> @@ -790,7 +753,7 @@ u64 avg_vruntime(struct cfs_rq *cfs_rq)
> s64 runtime = cfs_rq->sum_w_vruntime;
>
> if (curr) {
> - unsigned long w = avg_vruntime_weight(cfs_rq, curr->load.weight);
> + unsigned long w = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
>
> runtime += entity_key(cfs_rq, curr) * w;
> weight += w;
> @@ -861,8 +824,6 @@ bool update_entity_lag(struct cfs_rq *cf
> u64 avruntime = avg_vruntime(cfs_rq);
> s64 vlag = entity_lag(cfs_rq, se, avruntime);
>
> - WARN_ON_ONCE(!se->on_rq);
> -
> if (se->sched_delayed) {
> /* previous vlag < 0 otherwise se would not be delayed */
> vlag = max(vlag, se->vlag);
> @@ -898,7 +859,7 @@ static int vruntime_eligible(struct cfs_
> long load = cfs_rq->sum_weight;
>
> if (curr && curr->on_rq) {
> - unsigned long weight = avg_vruntime_weight(cfs_rq, curr->load.weight);
> + unsigned long weight = avg_vruntime_weight(cfs_rq, curr->h_load.weight);
>
> avg += entity_key(cfs_rq, curr) * weight;
> load += weight;
> @@ -1039,6 +1000,9 @@ RB_DECLARE_CALLBACKS(static, min_vruntim
> */
> static void __enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> {
> + WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
> + WARN_ON_ONCE(!entity_is_task(se));
> +
> sum_w_vruntime_add(cfs_rq, se);
> se->min_vruntime = se->vruntime;
> se->min_slice = se->slice;
> @@ -1048,6 +1012,9 @@ static void __enqueue_entity(struct cfs_
>
> static void __dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> {
> + WARN_ON_ONCE(&rq_of(cfs_rq)->cfs != cfs_rq);
> + WARN_ON_ONCE(!entity_is_task(se));
> +
> rb_erase_augmented_cached(&se->run_node, &cfs_rq->tasks_timeline,
> &min_vruntime_cb);
> sum_w_vruntime_sub(cfs_rq, se);
> @@ -1144,7 +1111,7 @@ static struct sched_entity *pick_eevdf(s
> * We can safely skip eligibility check if there is only one entity
> * in this cfs_rq, saving some cycles.
> */
> - if (cfs_rq->nr_queued == 1)
> + if (cfs_rq->h_nr_queued == 1)
> return curr && curr->on_rq ? curr : se;
>
> /*
> @@ -1391,8 +1358,6 @@ static s64 update_se(struct rq *rq, stru
> return delta_exec;
> }
>
> -static void set_next_buddy(struct sched_entity *se);
> -
> /*
> * Used by other classes to account runtime.
> */
> @@ -1412,7 +1377,7 @@ static void update_curr(struct cfs_rq *c
> * not necessarily be the actual task running
> * (rq->curr.se). This is easy to confuse!
> */
> - struct sched_entity *curr = cfs_rq->curr;
> + struct sched_entity *curr = cfs_rq->h_curr;
> struct rq *rq = rq_of(cfs_rq);
> s64 delta_exec;
> bool resched;
> @@ -1424,26 +1389,29 @@ static void update_curr(struct cfs_rq *c
> if (unlikely(delta_exec <= 0))
> return;
>
> + account_cfs_rq_runtime(cfs_rq, delta_exec);
> +
> + if (!entity_is_task(curr))
> + return;
> +
> + cfs_rq = &rq->cfs;
> +
> curr->vruntime += calc_delta_fair(delta_exec, curr);
> resched = update_deadline(cfs_rq, curr);
>
> - if (entity_is_task(curr)) {
> - /*
> - * If the fair_server is active, we need to account for the
> - * fair_server time whether or not the task is running on
> - * behalf of fair_server or not:
> - * - If the task is running on behalf of fair_server, we need
> - * to limit its time based on the assigned runtime.
> - * - Fair task that runs outside of fair_server should account
> - * against fair_server such that it can account for this time
> - * and possibly avoid running this period.
> - */
> - dl_server_update(&rq->fair_server, delta_exec);
> - }
> -
> - account_cfs_rq_runtime(cfs_rq, delta_exec);
> + /*
> + * If the fair_server is active, we need to account for the
> + * fair_server time whether or not the task is running on
> + * behalf of fair_server or not:
> + * - If the task is running on behalf of fair_server, we need
> + * to limit its time based on the assigned runtime.
> + * - Fair task that runs outside of fair_server should account
> + * against fair_server such that it can account for this time
> + * and possibly avoid running this period.
> + */
> + dl_server_update(&rq->fair_server, delta_exec);
>
> - if (cfs_rq->nr_queued == 1)
> + if (cfs_rq->h_nr_queued == 1)
> return;
>
> if (resched || !protect_slice(curr)) {
> @@ -1454,7 +1422,10 @@ static void update_curr(struct cfs_rq *c
>
> static void update_curr_fair(struct rq *rq)
> {
> - update_curr(cfs_rq_of(&rq->donor->se));
> + struct sched_entity *se = &rq->donor->se;
> +
> + for_each_sched_entity(se)
> + update_curr(cfs_rq_of(se));
> }
>
> static inline void
> @@ -1530,7 +1501,7 @@ update_stats_enqueue_fair(struct cfs_rq
> * Are we enqueueing a waiting task? (for current tasks
> * a dequeue/enqueue event is a NOP)
> */
> - if (se != cfs_rq->curr)
> + if (se != cfs_rq->h_curr)
> update_stats_wait_start_fair(cfs_rq, se);
>
> if (flags & ENQUEUE_WAKEUP)
> @@ -1548,7 +1519,7 @@ update_stats_dequeue_fair(struct cfs_rq
> * Mark the end of the wait period if dequeueing a
> * waiting task:
> */
> - if (se != cfs_rq->curr)
> + if (se != cfs_rq->h_curr)
> update_stats_wait_end_fair(cfs_rq, se);
>
> if ((flags & DEQUEUE_SLEEP) && entity_is_task(se)) {
> @@ -3875,6 +3846,7 @@ static inline void update_scan_period(st
> static void
> account_entity_enqueue(struct cfs_rq *cfs_rq, struct sched_entity *se)
> {
> + WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
> update_load_add(&cfs_rq->load, se->load.weight);
> if (entity_is_task(se)) {
> struct rq *rq = rq_of(cfs_rq);
> @@ -3888,6 +3860,7 @@ account_entity_enqueue(struct cfs_rq *cf
> static void
> account_entity_dequeue(struct cfs_rq *cfs_rq, struct sched_entity *se)
> {
> + WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
> update_load_sub(&cfs_rq->load, se->load.weight);
> if (entity_is_task(se)) {
> account_numa_dequeue(rq_of(cfs_rq), task_of(se));
> @@ -3965,7 +3938,7 @@ dequeue_load_avg(struct cfs_rq *cfs_rq,
> static void
> rescale_entity(struct sched_entity *se, unsigned long weight, bool rel_vprot)
> {
> - unsigned long old_weight = se->load.weight;
> + long old_weight = se->h_load.weight;
>
> /*
> * VRUNTIME
> @@ -4065,16 +4038,17 @@ rescale_entity(struct sched_entity *se,
> se->vprot = div64_long(se->vprot * old_weight, weight);
> }
>
> -static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
> - unsigned long weight)
> +static void reweight_eevdf(struct cfs_rq *cfs_rq, struct sched_entity *se,
> + unsigned long weight, bool on_rq)
> {
> bool curr = cfs_rq->curr == se;
> bool rel_vprot = false;
> u64 avruntime = 0;
>
> - if (se->on_rq) {
> - /* commit outstanding execution time */
> - update_curr(cfs_rq);
> + if (se->h_load.weight == weight)
> + return;
> +
> + if (on_rq) {
> avruntime = avg_vruntime(cfs_rq);
> se->vlag = entity_lag(cfs_rq, se, avruntime);
> se->deadline -= avruntime;
> @@ -4084,46 +4058,90 @@ static void reweight_entity(struct cfs_r
> rel_vprot = true;
> }
>
> - cfs_rq->nr_queued--;
> + cfs_rq->h_nr_queued--;
> if (!curr)
> __dequeue_entity(cfs_rq, se);
> - update_load_sub(&cfs_rq->load, se->load.weight);
> }
> - dequeue_load_avg(cfs_rq, se);
>
> rescale_entity(se, weight, rel_vprot);
>
> - update_load_set(&se->load, weight);
> + update_load_set(&se->h_load, weight);
>
> - do {
> - u32 divider = get_pelt_divider(&se->avg);
> - se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
> - } while (0);
> -
> - enqueue_load_avg(cfs_rq, se);
> - if (se->on_rq) {
> + if (on_rq) {
> if (rel_vprot)
> se->vprot += avruntime;
> se->deadline += avruntime;
> se->rel_deadline = 0;
> se->vruntime = avruntime - se->vlag;
>
> - update_load_add(&cfs_rq->load, se->load.weight);
> if (!curr)
> __enqueue_entity(cfs_rq, se);
> - cfs_rq->nr_queued++;
> + cfs_rq->h_nr_queued++;
> }
> }
>
> +static void reweight_entity(struct cfs_rq *cfs_rq, struct sched_entity *se,
> + unsigned long weight)
> +{
> + if (se->load.weight == weight)
> + return;
> +
> + if (se->on_rq) {
> + WARN_ON_ONCE(cfs_rq != cfs_rq_of(se));
> + update_load_sub(&cfs_rq->load, se->load.weight);
> + }
> + dequeue_load_avg(cfs_rq, se);
> +
> + update_load_set(&se->load, weight);
> +
> + do {
> + u32 divider = get_pelt_divider(&se->avg);
> + se->avg.load_avg = div_u64(se_weight(se) * se->avg.load_sum, divider);
> + } while (0);
> +
> + enqueue_load_avg(cfs_rq, se);
> +
> + if (se->on_rq)
> + update_load_add(&cfs_rq->load, se->load.weight);
> +}
> +
> +/*
> + * weight = NICE_0_LOAD;
> + * for_each_entity_se(se)
> + * weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
> + */
> +static __always_inline
> +unsigned long __calc_prop_weight(struct cfs_rq *cfs_rq, struct sched_entity *se,
> + unsigned long weight)
> +{
> + weight *= se->load.weight;
> + if (parent_entity(se))
> + weight /= cfs_rq->load.weight;
> + else
> + weight /= NICE_0_LOAD;
> +
> + return max(weight, MIN_SHARES);
> +}
> +
> static void reweight_task_fair(struct rq *rq, struct task_struct *p,
> const struct load_weight *lw)
> {
> struct sched_entity *se = &p->se;
> - struct cfs_rq *cfs_rq = cfs_rq_of(se);
> - struct load_weight *load = &se->load;
> + unsigned long weight = NICE_0_LOAD;
> +
> + if (se->on_rq)
> + update_curr_fair(rq);
> +
> + reweight_entity(cfs_rq_of(se), se, lw->weight);
> + se->load.inv_weight = lw->inv_weight;
> +
> + if (!se->on_rq)
> + return;
> +
> + for_each_sched_entity(se)
> + weight = __calc_prop_weight(cfs_rq_of(se), se, weight);
>
> - reweight_entity(cfs_rq, se, lw->weight);
> - load->inv_weight = lw->inv_weight;
> + reweight_eevdf(&rq->cfs, &p->se, weight, p->se.on_rq);
> }
>
> static inline int throttled_hierarchy(struct cfs_rq *cfs_rq);
> @@ -4331,7 +4349,6 @@ static long calc_group_shares(struct cfs
> static void update_cfs_group(struct sched_entity *se)
> {
> struct cfs_rq *gcfs_rq = group_cfs_rq(se);
> - long shares;
>
> /*
> * When a group becomes empty, preserve its weight. This matters for
> @@ -4340,9 +4357,7 @@ static void update_cfs_group(struct sche
> if (!gcfs_rq || !gcfs_rq->load.weight)
> return;
>
> - shares = calc_group_shares(gcfs_rq);
> - if (unlikely(se->load.weight != shares))
> - reweight_entity(cfs_rq_of(se), se, shares);
> + reweight_entity(cfs_rq_of(se), se, calc_group_shares(gcfs_rq));
> }
>
> #else /* !CONFIG_FAIR_GROUP_SCHED: */
> @@ -4460,7 +4475,7 @@ static inline bool cfs_rq_is_decayed(str
> * differential update where we store the last value we propagated. This in
> * turn allows skipping updates if the differential is 'small'.
> *
> - * Updating tg's load_avg is necessary before update_cfs_share().
> + * Updating tg's load_avg is necessary before update_cfs_group().
> */
> static inline void update_tg_load_avg(struct cfs_rq *cfs_rq)
> {
> @@ -4926,7 +4941,7 @@ static void migrate_se_pelt_lag(struct s
> * The cfs_rq avg is the direct sum of all its entities (blocked and runnable)
> * avg. The immediate corollary is that all (fair) tasks must be attached.
> *
> - * cfs_rq->avg is used for task_h_load() and update_cfs_share() for example.
> + * cfs_rq->avg is used for task_h_load() and update_cfs_group() for example.
> *
> * Return: true if the load decayed or we removed load.
> *
> @@ -5475,6 +5490,7 @@ static void
> place_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> {
> u64 vslice, vruntime = avg_vruntime(cfs_rq);
> + unsigned int nr_queued = cfs_rq->h_nr_queued;
> bool update_zero = false;
> s64 lag = 0;
>
> @@ -5482,6 +5498,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
> se->slice = sysctl_sched_base_slice;
> vslice = calc_delta_fair(se->slice, se);
>
> + if (flags & ENQUEUE_QUEUED)
> + nr_queued -= 1;
> +
> /*
> * Due to how V is constructed as the weighted average of entities,
> * adding tasks with positive lag, or removing tasks with negative lag
> @@ -5490,7 +5509,7 @@ place_entity(struct cfs_rq *cfs_rq, stru
> *
> * EEVDF: placement strategy #1 / #2
> */
> - if (sched_feat(PLACE_LAG) && cfs_rq->nr_queued && se->vlag) {
> + if (sched_feat(PLACE_LAG) && nr_queued && se->vlag) {
> struct sched_entity *curr = cfs_rq->curr;
> long load, weight;
>
> @@ -5550,9 +5569,9 @@ place_entity(struct cfs_rq *cfs_rq, stru
> */
> load = cfs_rq->sum_weight;
> if (curr && curr->on_rq)
> - load += avg_vruntime_weight(cfs_rq, curr->load.weight);
> + load += avg_vruntime_weight(cfs_rq, curr->h_load.weight);
>
> - weight = avg_vruntime_weight(cfs_rq, se->load.weight);
> + weight = avg_vruntime_weight(cfs_rq, se->h_load.weight);
> lag *= load + weight;
> if (WARN_ON_ONCE(!load))
> load = 1;
> @@ -5611,22 +5630,8 @@ static void check_enqueue_throttle(struc
> static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
>
> static void
> -requeue_delayed_entity(struct sched_entity *se);
> -
> -static void
> enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> {
> - bool curr = cfs_rq->curr == se;
> -
> - /*
> - * If we're the current task, we must renormalise before calling
> - * update_curr().
> - */
> - if (curr)
> - place_entity(cfs_rq, se, flags);
> -
> - update_curr(cfs_rq);
> -
> /*
> * When enqueuing a sched_entity, we must:
> * - Update loads to have both entity and cfs_rq synced with now.
> @@ -5645,13 +5650,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
> */
> update_cfs_group(se);
>
> - /*
> - * XXX now that the entity has been re-weighted, and it's lag adjusted,
> - * we can place the entity.
> - */
> - if (!curr)
> - place_entity(cfs_rq, se, flags);
> -
> account_entity_enqueue(cfs_rq, se);
>
> /* Entity has migrated, no longer consider this task hot */
> @@ -5660,8 +5658,6 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
>
> check_schedstat_required();
> update_stats_enqueue_fair(cfs_rq, se, flags);
> - if (!curr)
> - __enqueue_entity(cfs_rq, se);
> se->on_rq = 1;
>
> if (cfs_rq->nr_queued == 1) {
> @@ -5679,21 +5675,19 @@ enqueue_entity(struct cfs_rq *cfs_rq, st
> }
> }
>
> -static void __clear_buddies_next(struct sched_entity *se)
> +static void set_next_buddy(struct cfs_rq *cfs_rq, struct sched_entity *se)
> {
> - for_each_sched_entity(se) {
> - struct cfs_rq *cfs_rq = cfs_rq_of(se);
> - if (cfs_rq->next != se)
> - break;
> -
> - cfs_rq->next = NULL;
> - }
> + if (WARN_ON_ONCE(!se->on_rq || se->sched_delayed))
> + return;
> + if (se_is_idle(se))
> + return;
> + cfs_rq->next = se;
> }
>
> static void clear_buddies(struct cfs_rq *cfs_rq, struct sched_entity *se)
> {
> if (cfs_rq->next == se)
> - __clear_buddies_next(se);
> + cfs_rq->next = NULL;
> }
>
> static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq);
> @@ -5704,7 +5698,7 @@ static void set_delayed(struct sched_ent
>
> /*
> * Delayed se of cfs_rq have no tasks queued on them.
> - * Do not adjust h_nr_runnable since dequeue_entities()
> + * Do not adjust h_nr_runnable since __dequeue_task()
> * will account it for blocked tasks.
> */
> if (!entity_is_task(se))
> @@ -5737,37 +5731,11 @@ static void clear_delayed(struct sched_e
> }
> }
>
> -static bool
> +static void
> dequeue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
> {
> - bool sleep = flags & DEQUEUE_SLEEP;
> int action = UPDATE_TG;
>
> - update_curr(cfs_rq);
> - clear_buddies(cfs_rq, se);
> -
> - if (flags & DEQUEUE_DELAYED) {
> - WARN_ON_ONCE(!se->sched_delayed);
> - } else {
> - bool delay = sleep;
> - /*
> - * DELAY_DEQUEUE relies on spurious wakeups, special task
> - * states must not suffer spurious wakeups, excempt them.
> - */
> - if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
> - delay = false;
> -
> - WARN_ON_ONCE(delay && se->sched_delayed);
> -
> - if (sched_feat(DELAY_DEQUEUE) && delay &&
> - !entity_eligible(cfs_rq, se)) {
> - update_load_avg(cfs_rq, se, 0);
> - update_entity_lag(cfs_rq, se);
> - set_delayed(se);
> - return false;
> - }
> - }
> -
> if (entity_is_task(se) && task_on_rq_migrating(task_of(se)))
> action |= DO_DETACH;
>
> @@ -5785,14 +5753,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
>
> update_stats_dequeue_fair(cfs_rq, se, flags);
>
> - update_entity_lag(cfs_rq, se);
> - if (sched_feat(PLACE_REL_DEADLINE) && !sleep) {
> - se->deadline -= se->vruntime;
> - se->rel_deadline = 1;
> - }
> -
> - if (se != cfs_rq->curr)
> - __dequeue_entity(cfs_rq, se);
> se->on_rq = 0;
> account_entity_dequeue(cfs_rq, se);
>
> @@ -5801,9 +5761,6 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
>
> update_cfs_group(se);
>
> - if (flags & DEQUEUE_DELAYED)
> - clear_delayed(se);
> -
> if (cfs_rq->nr_queued == 0) {
> update_idle_cfs_rq_clock_pelt(cfs_rq);
> #ifdef CONFIG_CFS_BANDWIDTH
> @@ -5816,15 +5773,11 @@ dequeue_entity(struct cfs_rq *cfs_rq, st
> }
> #endif
> }
> -
> - return true;
> }
>
> static void
> -set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, bool first)
> +set_next_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> {
> - clear_buddies(cfs_rq, se);
> -
> /* 'current' is not kept within the tree. */
> if (se->on_rq) {
> /*
> @@ -5833,16 +5786,12 @@ set_next_entity(struct cfs_rq *cfs_rq, s
> * runqueue.
> */
> update_stats_wait_end_fair(cfs_rq, se);
> - __dequeue_entity(cfs_rq, se);
> update_load_avg(cfs_rq, se, UPDATE_TG);
> -
> - if (first)
> - set_protect_slice(cfs_rq, se);
> }
>
> update_stats_curr_start(cfs_rq, se);
> - WARN_ON_ONCE(cfs_rq->curr);
> - cfs_rq->curr = se;
> + WARN_ON_ONCE(cfs_rq->h_curr);
> + cfs_rq->h_curr = se;
>
> /*
> * Track our maximum slice length, if the CPU's load is at
> @@ -5862,23 +5811,17 @@ set_next_entity(struct cfs_rq *cfs_rq, s
> se->prev_sum_exec_runtime = se->sum_exec_runtime;
> }
>
> -static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags);
> +static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags);
>
> -/*
> - * Pick the next process, keeping these things in mind, in this order:
> - * 1) keep things fair between processes/task groups
> - * 2) pick the "next" process, since someone really wants that to run
> - * 3) pick the "last" process, for cache locality
> - * 4) do not run the "skip" process, if something else is available
> - */
> static struct sched_entity *
> -pick_next_entity(struct rq *rq, struct cfs_rq *cfs_rq, bool protect)
> +pick_next_entity(struct rq *rq, bool protect)
> {
> + struct cfs_rq *cfs_rq = &rq->cfs;
> struct sched_entity *se;
>
> se = pick_eevdf(cfs_rq, protect);
> if (se->sched_delayed) {
> - dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> + __dequeue_task(rq, task_of(se), DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> /*
> * Must not reference @se again, see __block_task().
> */
> @@ -5903,13 +5846,11 @@ static void put_prev_entity(struct cfs_r
>
> if (prev->on_rq) {
> update_stats_wait_start_fair(cfs_rq, prev);
> - /* Put 'current' back into the tree. */
> - __enqueue_entity(cfs_rq, prev);
> /* in !on_rq case, update occurred at dequeue */
> update_load_avg(cfs_rq, prev, 0);
> }
> - WARN_ON_ONCE(cfs_rq->curr != prev);
> - cfs_rq->curr = NULL;
> + WARN_ON_ONCE(cfs_rq->h_curr != prev);
> + cfs_rq->h_curr = NULL;
> }
>
> static void
> @@ -6062,7 +6003,7 @@ static void __account_cfs_rq_runtime(str
> * if we're unable to extend our runtime we resched so that the active
> * hierarchy can be throttled
> */
> - if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->curr))
> + if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->h_curr))
> resched_curr(rq_of(cfs_rq));
> }
>
> @@ -6420,7 +6361,7 @@ void unthrottle_cfs_rq(struct cfs_rq *cf
> assert_list_leaf_cfs_rq(rq);
>
> /* Determine whether we need to wake up potentially idle CPU: */
> - if (rq->curr == rq->idle && rq->cfs.nr_queued)
> + if (rq->curr == rq->idle && rq->cfs.h_nr_queued)
> resched_curr(rq);
> }
>
> @@ -6761,7 +6702,7 @@ static void check_enqueue_throttle(struc
> return;
>
> /* an active group must be handled by the update_curr()->put() path */
> - if (!cfs_rq->runtime_enabled || cfs_rq->curr)
> + if (!cfs_rq->runtime_enabled || cfs_rq->h_curr)
> return;
>
> /* ensure the group is not already throttled */
> @@ -7156,7 +7097,7 @@ static void hrtick_start_fair(struct rq
> resched_curr(rq);
> return;
> }
> - delta = (se->load.weight * vdelta) / NICE_0_LOAD;
> + delta = (se->h_load.weight * vdelta) / NICE_0_LOAD;
>
> /*
> * Correct for instantaneous load of other classes.
> @@ -7256,10 +7197,8 @@ static int choose_idle_cpu(int cpu, stru
> }
>
> static void
> -requeue_delayed_entity(struct sched_entity *se)
> +requeue_delayed_entity(struct cfs_rq *cfs_rq, struct sched_entity *se)
> {
> - struct cfs_rq *cfs_rq = cfs_rq_of(se);
> -
> /*
> * se->sched_delayed should imply: se->on_rq == 1.
> * Because a delayed entity is one that is still on
> @@ -7269,19 +7208,58 @@ requeue_delayed_entity(struct sched_enti
> WARN_ON_ONCE(!se->on_rq);
>
> if (update_entity_lag(cfs_rq, se)) {
> - cfs_rq->nr_queued--;
> + cfs_rq->h_nr_queued--;
> if (se != cfs_rq->curr)
> __dequeue_entity(cfs_rq, se);
> place_entity(cfs_rq, se, 0);
> if (se != cfs_rq->curr)
> __enqueue_entity(cfs_rq, se);
> - cfs_rq->nr_queued++;
> + cfs_rq->h_nr_queued++;
> }
>
> update_load_avg(cfs_rq, se, 0);
> clear_delayed(se);
> }
>
> +static unsigned long enqueue_hierarchy(struct task_struct *p, int flags)
> +{
> + unsigned long weight = NICE_0_LOAD;
> + int task_new = !(flags & ENQUEUE_WAKEUP);
> + struct sched_entity *se = &p->se;
> + int h_nr_idle = task_has_idle_policy(p);
> + int h_nr_runnable = 1;
> +
> + if (task_new && se->sched_delayed)
> + h_nr_runnable = 0;
> +
> + for_each_sched_entity(se) {
> + struct cfs_rq *cfs_rq = cfs_rq_of(se);
> +
> + update_curr(cfs_rq);
> +
> + if (!se->on_rq) {
> + enqueue_entity(cfs_rq, se, flags);
> + } else {
> + update_load_avg(cfs_rq, se, UPDATE_TG);
> + se_update_runnable(se);
> + update_cfs_group(se);
> + }
> +
> + cfs_rq->h_nr_runnable += h_nr_runnable;
> + cfs_rq->h_nr_queued++;
> + cfs_rq->h_nr_idle += h_nr_idle;
> +
> + if (cfs_rq_is_idle(cfs_rq))
> + h_nr_idle = 1;
> +
> + weight = __calc_prop_weight(cfs_rq, se, weight);
> +
> + flags = ENQUEUE_WAKEUP;
> + }
> +
> + return weight;
> +}
> +
> /*
> * The enqueue_task method is called before nr_running is
> * increased. Here we update the fair scheduling stats and
> @@ -7290,13 +7268,12 @@ requeue_delayed_entity(struct sched_enti
> static void
> enqueue_task_fair(struct rq *rq, struct task_struct *p, int flags)
> {
> - struct cfs_rq *cfs_rq;
> - struct sched_entity *se = &p->se;
> - int h_nr_idle = task_has_idle_policy(p);
> - int h_nr_runnable = 1;
> - int task_new = !(flags & ENQUEUE_WAKEUP);
> int rq_h_nr_queued = rq->cfs.h_nr_queued;
> - u64 slice = 0;
> + int task_new = !(flags & ENQUEUE_WAKEUP);
> + struct sched_entity *se = &p->se;
> + struct cfs_rq *cfs_rq = &rq->cfs;
> + unsigned long weight;
> + bool curr;
>
> if (task_is_throttled(p) && enqueue_throttled_task(p))
> return;
> @@ -7308,10 +7285,10 @@ enqueue_task_fair(struct rq *rq, struct
> * estimated utilization, before we update schedutil.
> */
> if (!p->se.sched_delayed || (flags & ENQUEUE_DELAYED))
> - util_est_enqueue(&rq->cfs, p);
> + util_est_enqueue(cfs_rq, p);
>
> if (flags & ENQUEUE_DELAYED) {
> - requeue_delayed_entity(se);
> + requeue_delayed_entity(cfs_rq, se);
> return;
> }
>
> @@ -7323,57 +7300,22 @@ enqueue_task_fair(struct rq *rq, struct
> if (p->in_iowait)
> cpufreq_update_util(rq, SCHED_CPUFREQ_IOWAIT);
>
> - if (task_new && se->sched_delayed)
> - h_nr_runnable = 0;
> -
> - for_each_sched_entity(se) {
> - if (se->on_rq) {
> - if (se->sched_delayed)
> - requeue_delayed_entity(se);
> - break;
> - }
> - cfs_rq = cfs_rq_of(se);
> -
> - /*
> - * Basically set the slice of group entries to the min_slice of
> - * their respective cfs_rq. This ensures the group can service
> - * its entities in the desired time-frame.
> - */
> - if (slice) {
> - se->slice = slice;
> - se->custom_slice = 1;
> - }
> - enqueue_entity(cfs_rq, se, flags);
> - slice = cfs_rq_min_slice(cfs_rq);
> -
> - cfs_rq->h_nr_runnable += h_nr_runnable;
> - cfs_rq->h_nr_queued++;
> - cfs_rq->h_nr_idle += h_nr_idle;
> -
> - if (cfs_rq_is_idle(cfs_rq))
> - h_nr_idle = 1;
> -
> - flags = ENQUEUE_WAKEUP;
> - }
> -
> - for_each_sched_entity(se) {
> - cfs_rq = cfs_rq_of(se);
> -
> - update_load_avg(cfs_rq, se, UPDATE_TG);
> - se_update_runnable(se);
> - update_cfs_group(se);
> + /*
> + * XXX comment on the curr thing
> + */
> + curr = (cfs_rq->curr == se);
> + if (curr)
> + place_entity(cfs_rq, se, flags);
>
> - se->slice = slice;
> - if (se != cfs_rq->curr)
> - min_vruntime_cb_propagate(&se->run_node, NULL);
> - slice = cfs_rq_min_slice(cfs_rq);
> + if (se->on_rq && se->sched_delayed)
> + requeue_delayed_entity(cfs_rq, se);
>
> - cfs_rq->h_nr_runnable += h_nr_runnable;
> - cfs_rq->h_nr_queued++;
> - cfs_rq->h_nr_idle += h_nr_idle;
> + weight = enqueue_hierarchy(p, flags);
>
> - if (cfs_rq_is_idle(cfs_rq))
> - h_nr_idle = 1;
> + if (!curr) {
> + reweight_eevdf(cfs_rq, se, weight, false);
> + place_entity(cfs_rq, se, flags | ENQUEUE_QUEUED);
> + __enqueue_entity(cfs_rq, se);
> }
>
> if (!rq_h_nr_queued && rq->cfs.h_nr_queued)
> @@ -7404,105 +7346,107 @@ enqueue_task_fair(struct rq *rq, struct
> hrtick_update(rq);
> }
>
> -/*
> - * Basically dequeue_task_fair(), except it can deal with dequeue_entity()
> - * failing half-way through and resume the dequeue later.
> - *
> - * Returns:
> - * -1 - dequeue delayed
> - * 0 - dequeue throttled
> - * 1 - dequeue complete
> - */
> -static int dequeue_entities(struct rq *rq, struct sched_entity *se, int flags)
> +static void dequeue_hierarchy(struct task_struct *p, int flags)
> {
> - bool was_sched_idle = sched_idle_rq(rq);
> + struct sched_entity *se = &p->se;
> bool task_sleep = flags & DEQUEUE_SLEEP;
> bool task_delayed = flags & DEQUEUE_DELAYED;
> bool task_throttled = flags & DEQUEUE_THROTTLE;
> - struct task_struct *p = NULL;
> - int h_nr_idle = 0;
> - int h_nr_queued = 0;
> int h_nr_runnable = 0;
> - struct cfs_rq *cfs_rq;
> - u64 slice = 0;
> + int h_nr_idle = task_has_idle_policy(p);
> + bool dequeue = true;
>
> - if (entity_is_task(se)) {
> - p = task_of(se);
> - h_nr_queued = 1;
> - h_nr_idle = task_has_idle_policy(p);
> - if (task_sleep || task_delayed || !se->sched_delayed)
> - h_nr_runnable = 1;
> - }
> + if (task_sleep || task_delayed || !se->sched_delayed)
> + h_nr_runnable = 1;
>
> for_each_sched_entity(se) {
> - cfs_rq = cfs_rq_of(se);
> + struct cfs_rq *cfs_rq = cfs_rq_of(se);
>
> - if (!dequeue_entity(cfs_rq, se, flags)) {
> - if (p && &p->se == se)
> - return -1;
> + update_curr(cfs_rq);
>
> - slice = cfs_rq_min_slice(cfs_rq);
> - break;
> + if (dequeue) {
> + dequeue_entity(cfs_rq, se, flags);
> + /* Don't dequeue parent if it has other entities besides us */
> + if (cfs_rq->load.weight)
> + dequeue = false;
> + } else {
> + update_load_avg(cfs_rq, se, UPDATE_TG);
> + se_update_runnable(se);
> + update_cfs_group(se);
> }
>
> cfs_rq->h_nr_runnable -= h_nr_runnable;
> - cfs_rq->h_nr_queued -= h_nr_queued;
> + cfs_rq->h_nr_queued--;
> cfs_rq->h_nr_idle -= h_nr_idle;
>
> if (cfs_rq_is_idle(cfs_rq))
> - h_nr_idle = h_nr_queued;
> + h_nr_idle = 1;
>
> if (throttled_hierarchy(cfs_rq) && task_throttled)
> record_throttle_clock(cfs_rq);
>
> - /* Don't dequeue parent if it has other entities besides us */
> - if (cfs_rq->load.weight) {
> - slice = cfs_rq_min_slice(cfs_rq);
> -
> - /* Avoid re-evaluating load for this entity: */
> - se = parent_entity(se);
> - /*
> - * Bias pick_next to pick a task from this cfs_rq, as
> - * p is sleeping when it is within its sched_slice.
> - */
> - if (task_sleep && se)
> - set_next_buddy(se);
> - break;
> - }
> flags |= DEQUEUE_SLEEP;
> flags &= ~(DEQUEUE_DELAYED | DEQUEUE_SPECIAL);
> }
> +}
>
> - for_each_sched_entity(se) {
> - cfs_rq = cfs_rq_of(se);
> +/*
> + * The part of dequeue_task_fair() that is needed to dequeue delayed tasks.
> + *
> + * Returns:
> + * true - dequeued
> + * false - delayed
> + */
> +static bool __dequeue_task(struct rq *rq, struct task_struct *p, int flags)
> +{
> + struct sched_entity *se = &p->se;
> + struct cfs_rq *cfs_rq = &rq->cfs;
> + bool was_sched_idle = sched_idle_rq(rq);
> + bool task_sleep = flags & DEQUEUE_SLEEP;
> + bool task_delayed = flags & DEQUEUE_DELAYED;
>
> - update_load_avg(cfs_rq, se, UPDATE_TG);
> - se_update_runnable(se);
> - update_cfs_group(se);
> + clear_buddies(cfs_rq, se);
>
> - se->slice = slice;
> - if (se != cfs_rq->curr)
> - min_vruntime_cb_propagate(&se->run_node, NULL);
> - slice = cfs_rq_min_slice(cfs_rq);
> + if (flags & DEQUEUE_DELAYED) {
> + WARN_ON_ONCE(!se->sched_delayed);
> + } else {
> + bool delay = task_sleep;
> + /*
> + * DELAY_DEQUEUE relies on spurious wakeups, special task
> + * states must not suffer spurious wakeups, excempt them.
> + */
> + if (flags & (DEQUEUE_SPECIAL | DEQUEUE_THROTTLE))
> + delay = false;
>
> - cfs_rq->h_nr_runnable -= h_nr_runnable;
> - cfs_rq->h_nr_queued -= h_nr_queued;
> - cfs_rq->h_nr_idle -= h_nr_idle;
> + WARN_ON_ONCE(delay && se->sched_delayed);
>
> - if (cfs_rq_is_idle(cfs_rq))
> - h_nr_idle = h_nr_queued;
> + if (sched_feat(DELAY_DEQUEUE) && delay &&
> + !entity_eligible(cfs_rq, se)) {
> + update_load_avg(cfs_rq_of(se), se, 0);
update_entity_lag(cfs_rq, se); is missing here. Unfortunately this
doesn't fix my regression
> + set_delayed(se);
> + return false;
> + }
> + }
>
> - if (throttled_hierarchy(cfs_rq) && task_throttled)
> - record_throttle_clock(cfs_rq);
> + dequeue_hierarchy(p, flags);
> +
> + update_entity_lag(cfs_rq, se);
> + if (sched_feat(PLACE_REL_DEADLINE) && !task_sleep) {
> + se->deadline -= se->vruntime;
> + se->rel_deadline = 1;
> }
> + if (se != cfs_rq->curr)
> + __dequeue_entity(cfs_rq, se);
>
> - sub_nr_running(rq, h_nr_queued);
> + sub_nr_running(rq, 1);
>
> /* balance early to pull high priority tasks */
> if (unlikely(!was_sched_idle && sched_idle_rq(rq)))
> rq->next_balance = jiffies;
>
> - if (p && task_delayed) {
> + if (task_delayed) {
> + clear_delayed(se);
> +
> WARN_ON_ONCE(!task_sleep);
> WARN_ON_ONCE(p->on_rq != 1);
>
> @@ -7514,7 +7458,7 @@ static int dequeue_entities(struct rq *r
> __block_task(rq, p);
> }
>
> - return 1;
> + return true;
> }
>
> /*
> @@ -7533,11 +7477,11 @@ static bool dequeue_task_fair(struct rq
> util_est_dequeue(&rq->cfs, p);
>
> util_est_update(&rq->cfs, p, flags & DEQUEUE_SLEEP);
> - if (dequeue_entities(rq, &p->se, flags) < 0)
> + if (!__dequeue_task(rq, p, flags))
> return false;
>
> /*
> - * Must not reference @p after dequeue_entities(DEQUEUE_DELAYED).
> + * Must not reference @p after __dequeue_task(DEQUEUE_DELAYED).
> */
> return true;
> }
> @@ -9021,19 +8965,6 @@ static void migrate_task_rq_fair(struct
> static void task_dead_fair(struct task_struct *p)
> {
> struct sched_entity *se = &p->se;
> -
> - if (se->sched_delayed) {
> - struct rq_flags rf;
> - struct rq *rq;
> -
> - rq = task_rq_lock(p, &rf);
> - if (se->sched_delayed) {
> - update_rq_clock(rq);
> - dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> - }
> - task_rq_unlock(rq, p, &rf);
> - }
> -
> remove_entity_load_avg(se);
> }
>
> @@ -9067,21 +8998,10 @@ static void set_cpus_allowed_fair(struct
> set_task_max_allowed_capacity(p);
> }
>
> -static void set_next_buddy(struct sched_entity *se)
> -{
> - for_each_sched_entity(se) {
> - if (WARN_ON_ONCE(!se->on_rq))
> - return;
> - if (se_is_idle(se))
> - return;
> - cfs_rq_of(se)->next = se;
> - }
> -}
> -
> enum preempt_wakeup_action {
> PREEMPT_WAKEUP_NONE, /* No preemption. */
> PREEMPT_WAKEUP_SHORT, /* Ignore slice protection. */
> - PREEMPT_WAKEUP_PICK, /* Let __pick_eevdf() decide. */
> + PREEMPT_WAKEUP_PICK, /* Let pick_eevdf() decide. */
> PREEMPT_WAKEUP_RESCHED, /* Force reschedule. */
> };
>
> @@ -9098,7 +9018,7 @@ set_preempt_buddy(struct cfs_rq *cfs_rq,
> if (cfs_rq->next && entity_before(cfs_rq->next, pse))
> return false;
>
> - set_next_buddy(pse);
> + set_next_buddy(cfs_rq, pse);
> return true;
> }
>
> @@ -9188,7 +9108,6 @@ static void wakeup_preempt_fair(struct r
> if (!sched_feat(WAKEUP_PREEMPTION))
> return;
>
> - find_matching_se(&se, &pse);
> WARN_ON_ONCE(!pse);
>
> cse_is_idle = se_is_idle(se);
> @@ -9216,8 +9135,7 @@ static void wakeup_preempt_fair(struct r
> if (unlikely(!normal_policy(p->policy)))
> return;
>
> - cfs_rq = cfs_rq_of(se);
> - update_curr(cfs_rq);
> + update_curr_fair(rq);
> /*
> * If @p has a shorter slice than current and @p is eligible, override
> * current's slice protection in order to allow preemption.
> @@ -9261,18 +9179,15 @@ static void wakeup_preempt_fair(struct r
> }
>
> pick:
> - nse = pick_next_entity(rq, cfs_rq, preempt_action != PREEMPT_WAKEUP_SHORT);
> - /* If @p has become the most eligible task, force preemption */
> - if (nse == pse)
> - goto preempt;
> -
> - /*
> - * Because p is enqueued, nse being null can only mean that we
> - * dequeued a delayed task. If there are still entities queued in
> - * cfs, check if the next one will be p.
> - */
> - if (!nse && cfs_rq->nr_queued)
> - goto pick;
> + if (cfs_rq->h_nr_queued) {
> + nse = pick_next_entity(rq, preempt_action != PREEMPT_WAKEUP_SHORT);
> + if (unlikely(!nse))
> + goto pick;
> +
> + /* If @p has become the most eligible task, force preemption */
> + if (nse == pse)
> + goto preempt;
> + }
>
> if (sched_feat(RUN_TO_PARITY))
> update_protect_slice(cfs_rq, se);
> @@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct r
> struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
> __must_hold(__rq_lockp(rq))
> {
> + struct cfs_rq *cfs_rq = &rq->cfs;
> struct sched_entity *se;
> - struct cfs_rq *cfs_rq;
> struct task_struct *p;
> - bool throttled;
> int new_tasks;
>
> again:
> - cfs_rq = &rq->cfs;
> - if (!cfs_rq->nr_queued)
> + if (!cfs_rq->h_nr_queued)
> goto idle;
>
> - throttled = false;
> -
> - do {
> - /* Might not have done put_prev_entity() */
> - if (cfs_rq->curr && cfs_rq->curr->on_rq)
> - update_curr(cfs_rq);
> -
> - throttled |= check_cfs_rq_runtime(cfs_rq);
> + /* Might not have done put_prev_entity() */
> + if (cfs_rq->curr && cfs_rq->curr->on_rq)
> + update_curr(cfs_rq);
>
> - se = pick_next_entity(rq, cfs_rq, true);
> - if (!se)
> - goto again;
> - cfs_rq = group_cfs_rq(se);
> - } while (cfs_rq);
> + se = pick_next_entity(rq, true);
> + if (!se)
> + goto again;
>
> p = task_of(se);
> - if (unlikely(throttled))
> + if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
> task_throttle_setup_work(p);
> return p;
>
> @@ -9353,7 +9259,7 @@ void fair_server_init(struct rq *rq)
> static void put_prev_task_fair(struct rq *rq, struct task_struct *prev, struct task_struct *next)
> {
> struct sched_entity *se = &prev->se;
> - struct cfs_rq *cfs_rq;
> + struct cfs_rq *cfs_rq = &rq->cfs;
> struct sched_entity *nse = NULL;
>
> #ifdef CONFIG_FAIR_GROUP_SCHED
> @@ -9363,7 +9269,7 @@ static void put_prev_task_fair(struct rq
>
> while (se) {
> cfs_rq = cfs_rq_of(se);
> - if (!nse || cfs_rq->curr)
> + if (!nse || cfs_rq->h_curr)
> put_prev_entity(cfs_rq, se);
> #ifdef CONFIG_FAIR_GROUP_SCHED
> if (nse) {
> @@ -9382,6 +9288,14 @@ static void put_prev_task_fair(struct rq
> #endif
> se = parent_entity(se);
> }
> +
> + /* Put 'current' back into the tree. */
> + cfs_rq = &rq->cfs;
> + se = &prev->se;
> + WARN_ON_ONCE(cfs_rq->curr != se);
> + cfs_rq->curr = NULL;
> + if (se->on_rq)
> + __enqueue_entity(cfs_rq, se);
> }
>
> /*
> @@ -9390,8 +9304,8 @@ static void put_prev_task_fair(struct rq
> static void yield_task_fair(struct rq *rq)
> {
> struct task_struct *curr = rq->donor;
> - struct cfs_rq *cfs_rq = task_cfs_rq(curr);
> struct sched_entity *se = &curr->se;
> + struct cfs_rq *cfs_rq = &rq->cfs;
>
> /*
> * Are we the only task in the tree?
> @@ -9432,11 +9346,11 @@ static bool yield_to_task_fair(struct rq
> struct sched_entity *se = &p->se;
>
> /* !se->on_rq also covers throttled task */
> - if (!se->on_rq)
> + if (!se->on_rq || se->sched_delayed)
> return false;
>
> /* Tell the scheduler that we'd really like se to run next. */
> - set_next_buddy(se);
> + set_next_buddy(&task_rq(p)->cfs, se);
>
> yield_task_fair(rq);
>
> @@ -9762,15 +9676,10 @@ static inline long migrate_degrades_loca
> */
> static inline int task_is_ineligible_on_dst_cpu(struct task_struct *p, int dest_cpu)
> {
> - struct cfs_rq *dst_cfs_rq;
> + struct cfs_rq *dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
>
> -#ifdef CONFIG_FAIR_GROUP_SCHED
> - dst_cfs_rq = task_group(p)->cfs_rq[dest_cpu];
> -#else
> - dst_cfs_rq = &cpu_rq(dest_cpu)->cfs;
> -#endif
> - if (sched_feat(PLACE_LAG) && dst_cfs_rq->nr_queued &&
> - !entity_eligible(task_cfs_rq(p), &p->se))
> + if (sched_feat(PLACE_LAG) && dst_cfs_rq->h_nr_queued &&
> + !entity_eligible(&task_rq(p)->cfs, &p->se))
> return 1;
>
> return 0;
> @@ -10240,7 +10149,7 @@ static void update_cfs_rq_h_load(struct
> while ((se = READ_ONCE(cfs_rq->h_load_next)) != NULL) {
> load = cfs_rq->h_load;
> load = div64_ul(load * se->avg.load_avg,
> - cfs_rq_load_avg(cfs_rq) + 1);
> + cfs_rq_load_avg(cfs_rq) + 1);
> cfs_rq = group_cfs_rq(se);
> cfs_rq->h_load = load;
> cfs_rq->last_h_load_update = now;
> @@ -13459,7 +13368,7 @@ static inline void task_tick_core(struct
> * MIN_NR_TASKS_DURING_FORCEIDLE - 1 tasks and use that to check
> * if we need to give up the CPU.
> */
> - if (rq->core->core_forceidle_count && rq->cfs.nr_queued == 1 &&
> + if (rq->core->core_forceidle_count && rq->cfs.h_nr_queued == 1 &&
> __entity_slice_used(&curr->se, MIN_NR_TASKS_DURING_FORCEIDLE))
> resched_curr(rq);
> }
> @@ -13668,30 +13577,8 @@ bool cfs_prio_less(const struct task_str
>
> WARN_ON_ONCE(task_rq(b)->core != rq->core);
>
> -#ifdef CONFIG_FAIR_GROUP_SCHED
> - /*
> - * Find an se in the hierarchy for tasks a and b, such that the se's
> - * are immediate siblings.
> - */
> - while (sea->cfs_rq->tg != seb->cfs_rq->tg) {
> - int sea_depth = sea->depth;
> - int seb_depth = seb->depth;
> -
> - if (sea_depth >= seb_depth)
> - sea = parent_entity(sea);
> - if (sea_depth <= seb_depth)
> - seb = parent_entity(seb);
> - }
> -
> - se_fi_update(sea, rq->core->core_forceidle_seq, in_fi);
> - se_fi_update(seb, rq->core->core_forceidle_seq, in_fi);
> -
> - cfs_rqa = sea->cfs_rq;
> - cfs_rqb = seb->cfs_rq;
> -#else /* !CONFIG_FAIR_GROUP_SCHED: */
> cfs_rqa = &task_rq(a)->cfs;
> cfs_rqb = &task_rq(b)->cfs;
> -#endif /* !CONFIG_FAIR_GROUP_SCHED */
>
> /*
> * Find delta after normalizing se's vruntime with its cfs_rq's
> @@ -13729,14 +13616,20 @@ static inline void task_tick_core(struct
> */
> static void task_tick_fair(struct rq *rq, struct task_struct *curr, int queued)
> {
> - struct cfs_rq *cfs_rq;
> struct sched_entity *se = &curr->se;
> + unsigned long weight = NICE_0_LOAD;
> + struct cfs_rq *cfs_rq;
>
> for_each_sched_entity(se) {
> cfs_rq = cfs_rq_of(se);
> entity_tick(cfs_rq, se, queued);
> +
> + weight = __calc_prop_weight(cfs_rq, se, weight);
> }
>
> + se = &curr->se;
> + reweight_eevdf(cfs_rq, se, weight, se->on_rq);
> +
> if (queued)
> return;
>
> @@ -13772,7 +13665,7 @@ prio_changed_fair(struct rq *rq, struct
> if (p->prio == oldprio)
> return;
>
> - if (rq->cfs.nr_queued == 1)
> + if (rq->cfs.h_nr_queued == 1)
> return;
>
> /*
> @@ -13901,29 +13794,40 @@ static void switched_to_fair(struct rq *
> }
> }
>
> -/*
> - * Account for a task changing its policy or group.
> - *
> - * This routine is mostly called to set cfs_rq->curr field when a task
> - * migrates between groups/classes.
> - */
> static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> {
> struct sched_entity *se = &p->se;
> + struct cfs_rq *cfs_rq = &rq->cfs;
> + unsigned long weight = NICE_0_LOAD;
> + bool on_rq = se->on_rq;
> +
> + clear_buddies(cfs_rq, se);
> +
> + if (on_rq)
> + __dequeue_entity(cfs_rq, se);
>
> for_each_sched_entity(se) {
> - struct cfs_rq *cfs_rq = cfs_rq_of(se);
> + cfs_rq = cfs_rq_of(se);
>
> - if (IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) &&
> - first && cfs_rq->curr)
> - break;
> + if (!IS_ENABLED(CONFIG_FAIR_GROUP_SCHED) ||
> + !first || !cfs_rq->h_curr)
> + set_next_entity(cfs_rq, se);
>
> - set_next_entity(cfs_rq, se, first);
> /* ensure bandwidth has been allocated on our new cfs_rq */
> account_cfs_rq_runtime(cfs_rq, 0);
> +
> + if (on_rq)
> + weight = __calc_prop_weight(cfs_rq, se, weight);
> }
>
> se = &p->se;
> + cfs_rq->curr = se;
> +
> + if (on_rq) {
> + reweight_eevdf(cfs_rq, se, weight, se->on_rq);
> + if (first)
> + set_protect_slice(cfs_rq, se);
> + }
>
> if (task_on_rq_queued(p)) {
> /*
> @@ -14054,17 +13958,8 @@ void unregister_fair_sched_group(struct
> struct sched_entity *se = tg->se[cpu];
> struct rq *rq = cpu_rq(cpu);
>
> - if (se) {
> - if (se->sched_delayed) {
> - guard(rq_lock_irqsave)(rq);
> - if (se->sched_delayed) {
> - update_rq_clock(rq);
> - dequeue_entities(rq, se, DEQUEUE_SLEEP | DEQUEUE_DELAYED);
> - }
> - list_del_leaf_cfs_rq(cfs_rq);
> - }
> + if (se)
> remove_entity_load_avg(se);
> - }
>
> /*
> * Only empty task groups can be destroyed; so we can speculatively
> --- a/kernel/sched/pelt.c
> +++ b/kernel/sched/pelt.c
> @@ -206,7 +206,7 @@ ___update_load_sum(u64 now, struct sched
> /*
> * running is a subset of runnable (weight) so running can't be set if
> * runnable is clear. But there are some corner cases where the current
> - * se has been already dequeued but cfs_rq->curr still points to it.
> + * se has been already dequeued but cfs_rq->h_curr still points to it.
> * This means that weight will be 0 but not running for a sched_entity
> * but also for a cfs_rq if the latter becomes idle. As an example,
> * this happens during sched_balance_newidle() which calls
> @@ -307,7 +307,7 @@ int __update_load_avg_blocked_se(u64 now
> int __update_load_avg_se(u64 now, struct cfs_rq *cfs_rq, struct sched_entity *se)
> {
> if (___update_load_sum(now, &se->avg, !!se->on_rq, se_runnable(se),
> - cfs_rq->curr == se)) {
> + cfs_rq->h_curr == se)) {
>
> ___update_load_avg(&se->avg, se_weight(se));
> cfs_se_util_change(&se->avg);
> @@ -323,7 +323,7 @@ int __update_load_avg_cfs_rq(u64 now, st
> if (___update_load_sum(now, &cfs_rq->avg,
> scale_load_down(cfs_rq->load.weight),
> cfs_rq->h_nr_runnable,
> - cfs_rq->curr != NULL)) {
> + cfs_rq->h_curr != NULL)) {
>
> ___update_load_avg(&cfs_rq->avg, 1);
> trace_pelt_cfs_tp(cfs_rq);
> --- a/kernel/sched/sched.h
> +++ b/kernel/sched/sched.h
> @@ -528,21 +528,8 @@ struct task_group {
>
> };
>
> -#ifdef CONFIG_GROUP_SCHED_WEIGHT
> #define ROOT_TASK_GROUP_LOAD NICE_0_LOAD
>
> -/*
> - * A weight of 0 or 1 can cause arithmetics problems.
> - * A weight of a cfs_rq is the sum of weights of which entities
> - * are queued on this cfs_rq, so a weight of a entity should not be
> - * too large, so as the shares value of a task group.
> - * (The default weight is 1024 - so there's no practical
> - * limitation from this.)
> - */
> -#define MIN_SHARES (1UL << 1)
> -#define MAX_SHARES (1UL << 18)
> -#endif
> -
> typedef int (*tg_visitor)(struct task_group *, void *);
>
> extern int walk_tg_tree_from(struct task_group *from,
> @@ -629,6 +616,17 @@ static inline bool cfs_task_bw_constrain
>
> #endif /* !CONFIG_CGROUP_SCHED */
>
> +/*
> + * A weight of 0 or 1 can cause arithmetics problems.
> + * A weight of a cfs_rq is the sum of weights of which entities
> + * are queued on this cfs_rq, so a weight of a entity should not be
> + * too large, so as the shares value of a task group.
> + * (The default weight is 1024 - so there's no practical
> + * limitation from this.)
> + */
> +#define MIN_SHARES (1UL << 1)
> +#define MAX_SHARES (1UL << 18)
> +
> extern void unregister_rt_sched_group(struct task_group *tg);
> extern void free_rt_sched_group(struct task_group *tg);
> extern int alloc_rt_sched_group(struct task_group *tg, struct task_group *parent);
> @@ -707,6 +705,7 @@ struct cfs_rq {
> /*
> * CFS load tracking
> */
> + struct sched_entity *h_curr;
> struct sched_avg avg;
> #ifndef CONFIG_64BIT
> u64 last_update_time_copy;
> @@ -2509,6 +2508,7 @@ extern const u32 sched_prio_to_wmult[40
> #define ENQUEUE_MIGRATED 0x00040000
> #define ENQUEUE_INITIAL 0x00080000
> #define ENQUEUE_RQ_SELECTED 0x00100000
> +#define ENQUEUE_QUEUED 0x00200000
>
> #define RETRY_TASK ((void *)-1UL)
>
>
>
^ permalink raw reply
* [tj-cgroup:for-next] BUILD SUCCESS 81807796db07bb1a4c066c75ccb5fcf04cbea3ed
From: kernel test robot @ 2026-05-19 10:16 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-next
branch HEAD: 81807796db07bb1a4c066c75ccb5fcf04cbea3ed Merge branch 'for-7.1-fixes' into for-next
elapsed time: 855m
configs tested: 263
configs skipped: 13
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-15.2.0
alpha allyesconfig gcc-15.2.0
alpha defconfig gcc-15.2.0
arc allmodconfig clang-16
arc allmodconfig gcc-15.2.0
arc allnoconfig gcc-15.2.0
arc allyesconfig clang-23
arc allyesconfig gcc-15.2.0
arc defconfig gcc-15.2.0
arc randconfig-001-20260519 clang-23
arc randconfig-001-20260519 gcc-9.5.0
arc randconfig-002-20260519 clang-23
arc randconfig-002-20260519 gcc-10.5.0
arm allnoconfig clang-23
arm allnoconfig gcc-15.2.0
arm allyesconfig clang-16
arm allyesconfig gcc-15.2.0
arm defconfig gcc-15.2.0
arm randconfig-001-20260519 clang-23
arm randconfig-002-20260519 clang-23
arm randconfig-003-20260519 clang-23
arm randconfig-004-20260519 clang-23
arm randconfig-004-20260519 gcc-8.5.0
arm spear13xx_defconfig gcc-15.2.0
arm64 allmodconfig clang-19
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-15.2.0
arm64 defconfig gcc-15.2.0
arm64 randconfig-001-20260519 clang-18
arm64 randconfig-001-20260519 gcc-8.5.0
arm64 randconfig-002-20260519 gcc-11.5.0
arm64 randconfig-002-20260519 gcc-8.5.0
arm64 randconfig-003-20260519 gcc-8.5.0
arm64 randconfig-004-20260519 gcc-8.5.0
arm64 randconfig-004-20260519 gcc-9.5.0
csky allmodconfig gcc-15.2.0
csky allnoconfig gcc-15.2.0
csky defconfig gcc-15.2.0
csky randconfig-001-20260519 gcc-12.5.0
csky randconfig-001-20260519 gcc-8.5.0
csky randconfig-002-20260519 gcc-11.5.0
csky randconfig-002-20260519 gcc-8.5.0
hexagon allmodconfig clang-17
hexagon allmodconfig gcc-15.2.0
hexagon allnoconfig clang-23
hexagon allnoconfig gcc-15.2.0
hexagon defconfig gcc-15.2.0
hexagon randconfig-001 gcc-11.5.0
hexagon randconfig-001-20260519 clang-23
hexagon randconfig-001-20260519 gcc-10.5.0
hexagon randconfig-001-20260519 gcc-11.5.0
hexagon randconfig-002 gcc-11.5.0
hexagon randconfig-002-20260519 clang-23
hexagon randconfig-002-20260519 gcc-10.5.0
hexagon randconfig-002-20260519 gcc-11.5.0
i386 allmodconfig clang-20
i386 allmodconfig gcc-14
i386 allnoconfig gcc-14
i386 allnoconfig gcc-15.2.0
i386 allyesconfig clang-20
i386 buildonly-randconfig-001-20260519 gcc-12
i386 buildonly-randconfig-002-20260519 gcc-12
i386 buildonly-randconfig-002-20260519 gcc-14
i386 buildonly-randconfig-003-20260519 clang-20
i386 buildonly-randconfig-003-20260519 gcc-12
i386 buildonly-randconfig-004-20260519 gcc-12
i386 buildonly-randconfig-004-20260519 gcc-14
i386 buildonly-randconfig-005-20260519 gcc-12
i386 buildonly-randconfig-005-20260519 gcc-14
i386 buildonly-randconfig-006-20260519 clang-20
i386 buildonly-randconfig-006-20260519 gcc-12
i386 defconfig gcc-15.2.0
i386 randconfig-001-20260519 gcc-14
i386 randconfig-002-20260519 gcc-14
i386 randconfig-003-20260519 gcc-14
i386 randconfig-004-20260519 gcc-14
i386 randconfig-005-20260519 gcc-14
i386 randconfig-006-20260519 gcc-14
i386 randconfig-007-20260519 gcc-14
loongarch allmodconfig clang-19
loongarch allmodconfig clang-23
loongarch allnoconfig clang-23
loongarch allnoconfig gcc-15.2.0
loongarch defconfig clang-19
loongarch randconfig-001 gcc-11.5.0
loongarch randconfig-001-20260519 clang-18
loongarch randconfig-001-20260519 gcc-10.5.0
loongarch randconfig-001-20260519 gcc-11.5.0
loongarch randconfig-002 gcc-11.5.0
loongarch randconfig-002-20260519 gcc-10.5.0
loongarch randconfig-002-20260519 gcc-11.5.0
loongarch randconfig-002-20260519 gcc-15.2.0
m68k allmodconfig gcc-15.2.0
m68k allnoconfig gcc-15.2.0
m68k allyesconfig clang-16
m68k allyesconfig gcc-15.2.0
m68k defconfig clang-19
m68k defconfig gcc-15.2.0
microblaze allnoconfig gcc-15.2.0
microblaze allyesconfig gcc-15.2.0
microblaze defconfig clang-19
microblaze defconfig gcc-15.2.0
mips allmodconfig gcc-15.2.0
mips allnoconfig gcc-15.2.0
mips allyesconfig gcc-15.2.0
nios2 10m50_defconfig gcc-11.5.0
nios2 allmodconfig clang-23
nios2 allmodconfig gcc-11.5.0
nios2 allnoconfig clang-23
nios2 allnoconfig gcc-11.5.0
nios2 defconfig clang-19
nios2 defconfig gcc-11.5.0
nios2 randconfig-001 gcc-11.5.0
nios2 randconfig-001-20260519 gcc-10.5.0
nios2 randconfig-001-20260519 gcc-11.5.0
nios2 randconfig-002 gcc-11.5.0
nios2 randconfig-002-20260519 gcc-10.5.0
nios2 randconfig-002-20260519 gcc-11.5.0
openrisc allmodconfig clang-23
openrisc allmodconfig gcc-15.2.0
openrisc allnoconfig clang-23
openrisc allnoconfig gcc-15.2.0
openrisc defconfig gcc-15.2.0
parisc allmodconfig gcc-15.2.0
parisc allnoconfig clang-23
parisc allnoconfig gcc-15.2.0
parisc allyesconfig clang-19
parisc allyesconfig gcc-15.2.0
parisc defconfig gcc-15.2.0
parisc randconfig-001 gcc-8.5.0
parisc randconfig-001-20260519 gcc-12.5.0
parisc randconfig-001-20260519 gcc-8.5.0
parisc randconfig-002 gcc-8.5.0
parisc randconfig-002-20260519 gcc-8.5.0
parisc64 defconfig clang-19
parisc64 defconfig gcc-15.2.0
powerpc allmodconfig gcc-15.2.0
powerpc allnoconfig clang-23
powerpc allnoconfig gcc-15.2.0
powerpc mpc832x_rdb_defconfig gcc-15.2.0
powerpc randconfig-001 gcc-8.5.0
powerpc randconfig-001-20260519 clang-23
powerpc randconfig-001-20260519 gcc-8.5.0
powerpc randconfig-002 gcc-8.5.0
powerpc randconfig-002-20260519 gcc-8.5.0
powerpc64 randconfig-001 gcc-8.5.0
powerpc64 randconfig-001-20260519 clang-23
powerpc64 randconfig-001-20260519 gcc-8.5.0
powerpc64 randconfig-002-20260519 gcc-14.3.0
powerpc64 randconfig-002-20260519 gcc-8.5.0
riscv allmodconfig clang-23
riscv allnoconfig clang-23
riscv allnoconfig gcc-15.2.0
riscv allyesconfig clang-16
riscv defconfig clang-23
riscv defconfig gcc-15.2.0
riscv randconfig-001-20260519 gcc-13.4.0
riscv randconfig-001-20260519 gcc-8.5.0
riscv randconfig-002-20260519 clang-17
riscv randconfig-002-20260519 gcc-13.4.0
s390 allmodconfig clang-18
s390 allmodconfig clang-19
s390 allnoconfig clang-23
s390 allyesconfig gcc-15.2.0
s390 defconfig clang-23
s390 defconfig gcc-15.2.0
s390 randconfig-001-20260519 clang-23
s390 randconfig-001-20260519 gcc-13.4.0
s390 randconfig-002-20260519 clang-18
s390 randconfig-002-20260519 gcc-13.4.0
sh allmodconfig gcc-15.2.0
sh allnoconfig clang-23
sh allnoconfig gcc-15.2.0
sh allyesconfig clang-19
sh allyesconfig gcc-15.2.0
sh defconfig gcc-14
sh defconfig gcc-15.2.0
sh randconfig-001-20260519 gcc-13.4.0
sh randconfig-001-20260519 gcc-15.2.0
sh randconfig-002-20260519 gcc-13.4.0
sparc allnoconfig clang-23
sparc allnoconfig gcc-15.2.0
sparc defconfig gcc-15.2.0
sparc randconfig-001-20260519 gcc-14.3.0
sparc randconfig-002-20260519 gcc-11.5.0
sparc randconfig-002-20260519 gcc-14.3.0
sparc64 allmodconfig clang-23
sparc64 defconfig clang-20
sparc64 defconfig gcc-14
sparc64 randconfig-001-20260519 gcc-14.3.0
sparc64 randconfig-001-20260519 gcc-8.5.0
sparc64 randconfig-002-20260519 gcc-14.3.0
sparc64 randconfig-002-20260519 gcc-8.5.0
um allmodconfig clang-19
um allnoconfig clang-23
um allyesconfig gcc-14
um allyesconfig gcc-15.2.0
um defconfig clang-23
um defconfig gcc-14
um i386_defconfig gcc-14
um randconfig-001-20260519 clang-23
um randconfig-001-20260519 gcc-14.3.0
um randconfig-002-20260519 clang-23
um randconfig-002-20260519 gcc-14.3.0
um x86_64_defconfig clang-23
um x86_64_defconfig gcc-14
x86_64 allmodconfig clang-20
x86_64 allnoconfig clang-20
x86_64 allnoconfig clang-23
x86_64 allyesconfig clang-20
x86_64 buildonly-randconfig-001-20260519 gcc-14
x86_64 buildonly-randconfig-002-20260519 gcc-14
x86_64 buildonly-randconfig-003-20260519 clang-20
x86_64 buildonly-randconfig-003-20260519 gcc-14
x86_64 buildonly-randconfig-004-20260519 clang-20
x86_64 buildonly-randconfig-004-20260519 gcc-14
x86_64 buildonly-randconfig-005-20260519 gcc-14
x86_64 buildonly-randconfig-006-20260519 gcc-14
x86_64 defconfig gcc-14
x86_64 kexec clang-20
x86_64 randconfig-001-20260519 clang-20
x86_64 randconfig-001-20260519 gcc-14
x86_64 randconfig-002-20260519 clang-20
x86_64 randconfig-002-20260519 gcc-13
x86_64 randconfig-003-20260519 clang-20
x86_64 randconfig-004-20260519 clang-20
x86_64 randconfig-005-20260519 clang-20
x86_64 randconfig-006-20260519 clang-20
x86_64 randconfig-011 clang-20
x86_64 randconfig-011-20260519 clang-20
x86_64 randconfig-011-20260519 gcc-14
x86_64 randconfig-012 clang-20
x86_64 randconfig-012-20260519 clang-20
x86_64 randconfig-013 clang-20
x86_64 randconfig-013-20260519 clang-20
x86_64 randconfig-014 clang-20
x86_64 randconfig-014-20260519 clang-20
x86_64 randconfig-014-20260519 gcc-14
x86_64 randconfig-015 clang-20
x86_64 randconfig-015-20260519 clang-20
x86_64 randconfig-016 clang-20
x86_64 randconfig-016-20260519 clang-20
x86_64 randconfig-071-20260519 gcc-14
x86_64 randconfig-072-20260519 gcc-14
x86_64 randconfig-073-20260519 gcc-14
x86_64 randconfig-074-20260519 gcc-14
x86_64 randconfig-075-20260519 gcc-14
x86_64 randconfig-076-20260519 gcc-14
x86_64 rhel-9.4 clang-20
x86_64 rhel-9.4-bpf gcc-14
x86_64 rhel-9.4-func clang-20
x86_64 rhel-9.4-kselftests clang-20
x86_64 rhel-9.4-kunit gcc-14
x86_64 rhel-9.4-ltp gcc-14
x86_64 rhel-9.4-rust clang-20
xtensa allnoconfig clang-23
xtensa allnoconfig gcc-15.2.0
xtensa allyesconfig clang-23
xtensa allyesconfig gcc-15.2.0
xtensa randconfig-001-20260519 gcc-10.5.0
xtensa randconfig-001-20260519 gcc-14.3.0
xtensa randconfig-002-20260519 gcc-10.5.0
xtensa randconfig-002-20260519 gcc-14.3.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH v2 00/10] sched: Flatten the pick
From: Vincent Guittot @ 2026-05-19 10:13 UTC (permalink / raw)
To: Peter Zijlstra
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <20260518211239.GY3102624@noisy.programming.kicks-ass.net>
On Mon, 18 May 2026 at 23:12, Peter Zijlstra <peterz@infradead.org> wrote:
>
> On Mon, May 18, 2026 at 03:34:51PM +0200, Vincent Guittot wrote:
> > On Wed, 13 May 2026 at 13:35, Peter Zijlstra <peterz@infradead.org> wrote:
> > >
> > > On Tue, May 12, 2026 at 10:42:33AM +0200, Vincent Guittot wrote:
> > >
> > > > I haven't reviewed the patches yet but I ran some tests with it while
> > > > testing sched latency related changes for short slice wakeup
> > > > preemption. I have some large hackbench regressions with this series
> > > > on HMP system with and without EAS. those figures are unexpected
> > > > because the benchs run on root cfs
> > > >
> > > > One example with hackbench 8 groups thread pipe
> > > > tip/sched/core tip/sched/core +this patchset +this patchset
> > > > slice 2.8ms 16ms 2.8ms 16ms
> > > > dragonboard rb5 with EAS
> > > > 0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
> > > > 0,689(+/- 9.1%) +8%
> > > >
> > > > radxa orion6 HMP without EAS
> > > > 0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
> > > > 1,071(+/-5.9%) -82%
> > > >
> > > > Increasing the slice partly removes regressions but tis is surprising
> > > > because the bench runs at root cfs and I thought that results will not
> > > > change in such a case
> > >
> > > D'oh :/
> > >
> > > diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> > > index e54da4c6c945..77d0e1937f2c 100644
> > > --- a/kernel/sched/fair.c
> > > +++ b/kernel/sched/fair.c
> > > @@ -9071,7 +9071,7 @@ static void wakeup_preempt_fair(struct rq *rq, struct task_struct *p, int wake_f
> > > enum preempt_wakeup_action preempt_action = PREEMPT_WAKEUP_PICK;
> > > struct task_struct *donor = rq->donor;
> > > struct sched_entity *nse, *se = &donor->se, *pse = &p->se;
> > > - struct cfs_rq *cfs_rq = task_cfs_rq(donor);
> > > + struct cfs_rq *cfs_rq = &rq->cfs;
> >
> > I tested this patch on top of the series but it doesn't fix the perf
> > regression on rb5
> >
> > hackbench 8 groups thread pipe is still at 1.907(+/-7.6%) with default
> > slice duration
>
> Weird, I can't reproduce anymore with this fixed :/
>
> I'll try more hackbench variants tomorrow I suppose.
I tried several conf :
- HMP with EAS enabled
- HMP without EAS enabled (perf cpufreq gov)
- SMP (only the 4 little cores)
All of them show large regressions with hackbench which are almost
recovered when increasing the slice from 2.8 to 16ms
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox