* Re: [PATCH 4/4] memcg: multi objcg charge support
From: Harry Yoo @ 2026-05-20 9:35 UTC (permalink / raw)
To: Shakeel Butt, Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Meta kernel team,
linux-mm, cgroups, linux-kernel, kernel test robot
In-Reply-To: <20260520053123.2709959-5-shakeel.butt@linux.dev>
On 5/20/26 2:31 PM, Shakeel Butt wrote:
> Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
> per-node type") split a memcg's single obj_cgroup into one per NUMA
> node so that reparenting LRU folios can take per-node lru locks. As a
> side effect, the per-CPU obj_stock_pcp -- which caches exactly one
> cached_objcg -- thrashes on workloads where threads of the same memcg
> run on different NUMA nodes. The kernel test robot reported a 67.7%
> regression on stress-ng.switch.ops_per_sec from this pattern.
>
> Mirror the multi-slot pattern already used by memcg_stock_pcp: turn
> nr_bytes and cached_objcg into NR_OBJ_STOCK-element arrays, scan all
> slots on consume/refill/account, prefer empty slots when inserting,
> and evict a random slot only when full. With multiple slots a CPU can
> hold the per-node objcg variants of one memcg plus a few siblings
> without ever forcing a drain.
>
> A single int8_t index records which slot the cached slab stats belong
> to; the stats are flushed on slot or pgdat change. With NR_OBJ_STOCK
> = 5 the layout (verified with pahole) is:
>
> offset 0 : lock(1) + index(1) + node_id(2) + slab stats(4) = 8B
> offset 8 : nr_bytes[5] = 10B
> offset 18 : padding = 6B
> offset 24 : cached[5] = 40B
> offset 64 : (line 2) work_struct + flags (cold)
>
> so consume_obj_stock, refill_obj_stock and the slab account path each
> touch exactly one 64-byte cache line on non-debug 64-bit builds.
>
> Reported-by: kernel test robot <oliver.sang@intel.com>
> Closes: https://lore.kernel.org/oe-lkp/202605121641.b6a60cb0-lkp@intel.com
> Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> Tested-by: kernel test robot <oliver.sang@intel.com>
> ---
> @@ -3350,19 +3405,45 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> goto out;
> }
>
> - stock_nr_bytes = stock->nr_bytes;
> - if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
> - drain_obj_stock(stock);
> + for (i = 0; i < NR_OBJ_STOCK; ++i) {
> + struct obj_cgroup *cached = READ_ONCE(stock->cached[i]);
> +
> + if (!cached) {
> + if (empty_slot == -1)
> + empty_slot = i;
> + continue;
> + }
> + if (cached == objcg) {
> + slot = i;
> + break;
> + }
> + }
> +
> + if (slot == -1) {
> + slot = empty_slot;
> + if (slot == -1) {
> + slot = get_random_u32_below(NR_OBJ_STOCK);
It would break kmalloc_nolock() because _get_random_bytes() uses a
spinlock. perhaps prandom_u32_state() should be sufficient in this case.
Is there a reason why it uses random eviction, unlike multi-memcg percpu
charge cache?
Otherwise LGTM!
--
Cheers,
Harry / Hyeonggon
^ permalink raw reply
* [PATCH] selftests/cgroup: enable memory controller in hugetlb memcg test
From: Guopeng Zhang @ 2026-05-20 9:31 UTC (permalink / raw)
To: Shuah Khan, Tejun Heo, Michal Koutný
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Oscar Salvador, David Hildenbrand, cgroups, linux-mm,
linux-kselftest, linux-kernel, Guopeng Zhang
test_hugetlb_memcg creates a child cgroup and then writes memory.max and
memory.swap.max. When the test is run standalone, the memory controller
may not be enabled in the test root cgroup's subtree_control.
In that case, the child cgroup is created without the memory control
files, and the test fails during setup before reaching the hugetlb memcg
accounting checks.
Skip the test when the memory controller is unavailable. Otherwise, enable
it in subtree_control before creating the test cgroup.
Signed-off-by: Guopeng Zhang <zhangguopeng@kylinos.cn>
---
Tested with a cgroup namespace where memory is available in
cgroup.controllers but not enabled in cgroup.subtree_control:
before: test_hugetlb_memcg failed with "fail to set cgroup memory limit"
after: test_hugetlb_memcg passed and cgroup.subtree_control contained memory
tools/testing/selftests/cgroup/test_hugetlb_memcg.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/tools/testing/selftests/cgroup/test_hugetlb_memcg.c b/tools/testing/selftests/cgroup/test_hugetlb_memcg.c
index f451aa449be6..b627d84358b1 100644
--- a/tools/testing/selftests/cgroup/test_hugetlb_memcg.c
+++ b/tools/testing/selftests/cgroup/test_hugetlb_memcg.c
@@ -217,6 +217,14 @@ int main(int argc, char **argv)
if (cg_find_unified_root(root, sizeof(root), NULL))
ksft_exit_skip("cgroup v2 isn't mounted\n");
+ if (cg_read_strstr(root, "cgroup.controllers", "memory"))
+ ksft_exit_skip("memory controller isn't available\n");
+
+ if (cg_read_strstr(root, "cgroup.subtree_control", "memory")) {
+ if (cg_write(root, "cgroup.subtree_control", "+memory"))
+ ksft_exit_skip("Failed to set memory controller\n");
+ }
+
switch (test_hugetlb_memcg(root)) {
case KSFT_PASS:
ksft_test_result_pass("test_hugetlb_memcg\n");
--
2.43.0
^ permalink raw reply related
* Re: [PATCH 0/8] per-memcg-per-node kmem accounting
From: Alexandre Ghiti @ 2026-05-20 8:39 UTC (permalink / raw)
To: Joshua Hahn
Cc: Andrew Morton, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Dennis Zhou, Tejun Heo, Christoph Lameter,
Vlastimil Babka, Yosry Ahmed, Nhat Pham, Sergey Senozhatsky,
Chengming Zhou, Suren Baghdasaryan, Qi Zheng, David Hildenbrand,
Lorenzo Stoakes, Minchan Kim, Mike Rapoport, Axel Rasmussen,
Barry Song, Kairui Song, Wei Xu, Yuanchu Xie, Liam R . Howlett,
linux-mm, linux-kernel, cgroups
In-Reply-To: <20260518145732.349196-1-joshua.hahnjy@gmail.com>
Hi Joshua,
On 5/18/26 16:57, Joshua Hahn wrote:
> On Mon, 11 May 2026 22:20:35 +0200 Alexandre Ghiti <alex@ghiti.fr> wrote:
>
>> This series pursues the work initiated by Joshua [1]. We need kernel
>> memory to be accounted on a per-node basis in order to be able to
>> know the memcg and physical memory association.
>>
>> This series takes advantage of the recent introduction of per-node
>> obj_cgroup [2] and makes those obj_cgroup tied to their numa node.
>>
>> The bulk of the series is percpu per-node accounting: percpu
>> "precharges" the memcg before we know the actual location of the pages
>> it uses, so charging and accounting had to be split. All other kmem
>> users (slab, zswap, __memcg_kmem_charge_page) are straightforward
>> conversions (zswap support is limited in this series because Joshua
>> is working on it in parallel [3]).
>>
>> Thanks Joshua for your early feedbacks!
> Hello Alex,
>
> Thank you for your work!
>
> Overall I think the direction makes sense to me. Pre-overcharging makes sense to
> me as an approach, we would much rather overaccount than underaccount and
> later have to breach limits.
>
> I do have some concerns on performance, though. Namely, I think there are
> some expensive operations that I think would benefit from some performane
> benchmarking with this patch added (maybe some simple microbenchmarks that
> demonstrates kernel allocation overhead could be useful).
>
> From what I can tell, there is some additional performance overhead that has
> to do with iterating over num_possible_cpus() x pages_per_alloc, which
> doesn't seem trivial to me.
Indeed, let me microbenchmark the overhead on a large system.
>
> Another concern that I see is the stock credit system. Maybe we could be
> bypassing the stock check leading to more time spent doing the atomic
> operations.
I'm not following on this one, which atomic operations do you see that
could be bypassed?
>
> obj_stock caches a single obj_cgroup, which means that if we split the objcg
> to be per-node (in patch 6), then the obj_stock basically gets invalidated
> every operation since we iterate over more objcgs (even though we are in
> the same logical objcg). Maybe I'm missing something?
The objcg split comes from commit 01b9da291c49 ("mm: memcontrol: convert
objcg to be per-memcg per-node type") and the problem you describe is
exactly what Shakeel is trying to fix [1].
But I remember trying a microbenchmark and noticed a +5% regression (on
top of the 67% then...), I'll rebase this series on top of Shakeel's and
re-run.
[1]
https://lore.kernel.org/linux-mm/20260520053123.2709959-1-shakeel.butt@linux.dev/T/#m127d4969b105c046a2a21e3c79c963771007583d
>
> I haven't taken a deep look at the implementation details but just wanted to
> raise some high level items that I noticed. Of course, all of these concerns
> are just theoretical, if you can show that the performance delta is not
> noticable then all of my concerns don't matter.
>
> I also want to talk more about the local credit system but let's first see
> what the numbers are first.
>
> Thanks again, Alex. And I really like patch 2 because it is a solution to
> a problem that I ran into in my percpu tracking series that I couldn't think
> of before! Thank you for solving my problem too : -)
Great then, thanks :)
Alex
>
> Have a great day!
> Joshua
>
>> [1] https://lore.kernel.org/linux-mm/20260404033844.1892595-1-joshua.hahnjy@gmail.com/
>> [2] https://lore.kernel.org/linux-mm/56c04b1c5d54f75ccdc12896df6c1ca35403ecc3.1772711148.git.zhengqi.arch@bytedance.com/
>> [3] https://lore.kernel.org/linux-mm/20260311195153.4013476-1-joshua.hahnjy@gmail.com/
>>
>> Alexandre Ghiti (8):
>> mm: memcontrol: propagate NMI slab stats to memcg vmstats
>> mm: percpu: charge obj_exts allocation with __GFP_ACCOUNT
>> mm: percpu: Split memcg charging and kmem accounting
>> mm: memcontrol: track MEMCG_KMEM per NUMA node
>> mm: memcontrol: per-node kmem accounting for page charges
>> mm: slab: per-node kmem accounting for slab
>> mm: percpu: per-node kmem accounting using local credit
>> mm: zswap: per-node kmem accounting for zswap/zsmalloc
>>
>> include/linux/memcontrol.h | 27 +++++--
>> include/linux/mmzone.h | 1 +
>> include/linux/zsmalloc.h | 2 +
>> mm/memcontrol.c | 150 ++++++++++++++++++++++++++++---------
>> mm/percpu-internal.h | 16 +---
>> mm/percpu.c | 90 ++++++++++++++++++++--
>> mm/vmstat.c | 1 +
>> mm/zsmalloc.c | 11 +++
>> mm/zswap.c | 9 ++-
>> 9 files changed, 242 insertions(+), 65 deletions(-)
>>
>> --
>> 2.54.0
>>
>>
^ permalink raw reply
* Re: [PATCH cgroup/for-next v2 0/5] cgroup/cpuset: Support multiple source/destination cpusets for cpuset_*attach()
From: Ridong Chen @ 2026-05-20 8:29 UTC (permalink / raw)
To: Waiman Long, Chen Ridong, Tejun Heo, Johannes Weiner,
Michal Koutný, Ingo Molnar, Peter Zijlstra, Juri Lelli,
Vincent Guittot, Dietmar Eggemann, Steven Rostedt, Ben Segall,
Mel Gorman, Valentin Schneider, K Prateek Nayak
Cc: cgroups, linux-kernel, Aaron Tomlin
In-Reply-To: <20260516042448.698216-1-longman@redhat.com>
On 2026/5/16 12:24, Waiman Long wrote:
> Sashiko AI review of another cpuset patch had found that cpuset_attach()
> and cpuset_can_attach() can be passed a cgroup_taskset with tasks
> migrating from one source cpuset to multiple destination cpusets and
> vice versa. Further testing of the cpuset code indicates that this is
> indeed the case when the v2 cpuset controller is enabled or disabled.
>
> Unfortunately, cpuset_attach() and cpuset_can_attach() still assume that
> there will be one source and one destinaton cpuset which may result in
> inocrrect behavior.
>
Hi Longman,
I am thinking whether we can use the pids subsystem's approach to solve
this issue, which I think could be much simpler.
For the DL task accounting, we can handle it the same way
pids_can_attach() does - just call task_cs(task) for each task
individually inside the can_attach() loop and do the nr_deadline_tasks
adjustment right there. This eliminates the need to pass per-task source
cpuset information to the attach() callback entirely for DL accounting
purposes.
For cpuset_migrate_mm(), I don't think we need per-task oldcs storage in
task_struct either. The scenarios where multiple source cpusets are
involved are:
enable cpuset controller: child cpusets inherit parent's effective_mems,
so attach_mems_updated is false and cpuset_migrate_mm() is never called.
disable cpuset controller: tasks move from children to parent. Since
children's effective_mems is always a subset of parent's effective_mems,
even if cpuset_migrate_mm() is triggered, it's effectively a noop (no
pages need to move from a subset to its superset).
cgroup.procs write with threads in different cpusets: this is a
many-to-one migration with a single process, so there is only one
group_leader and one mm. We only need to record the leader's oldcs,
which a single static variable can handle.
So in all cases, the migration path only needs one oldcs for the leader.
We don't need to add a field to task_struct.
What do you think?
> This patch series is created to fix this issue. The first 2 patches are
> just preparatory patches to make the remaining patches easier to review.
>
> Patch 3 adds a new attach_old_cs field into task_struct to track the
> old cpuset to be used in case when cpuset_migrate_mm() needs to be
> called in cpuset_attach().
>
> Patch 4 moves mpol_rebind_mm() and cpuset_migrate_mm() inside
> cpuset_attach_task() to make CLONE_INTO_CGROUP flag of clone(2) works
> more like moving task from one cpuset to another one, while also make
> supporting multiple source and destination cpusets easier.
>
> Patch 5 makes the necessary changes to enable the support of multiple
> source and destination cpusets by keeping all the source and destination
> cpusets found during task iterations in two singly linked lists for
> source and destination cpusets respectively.
>
> Waiman Long (5):
> cgroup/cpuset: Add a cpuset_reserve_dl_bw() helper
> cgroup/cpuset: Expand the scope of cpuset_can_attach_check()
> cgroup/cpuset: Replace cpuset_attach_old_cs by a new attach_old_cs
> field in task_struct
> cgroup/cpuset: Move mpol_rebind_mm/cpuset_migrate_mm() calls inside
> cpuset_attach_task()
> cgroup/cpuset: Support multiple source/destination cpusets for
> cpuset_*attach()
>
> include/linux/sched.h | 3 +
> kernel/cgroup/cpuset-internal.h | 6 +
> kernel/cgroup/cpuset.c | 358 +++++++++++++++++++++-----------
> 3 files changed, 249 insertions(+), 118 deletions(-)
>
--
Best regards,
Ridong
^ permalink raw reply
* Re: [PATCH 3/4] memcg: int16_t for cached slab stats
From: Harry Yoo @ 2026-05-20 7:25 UTC (permalink / raw)
To: Shakeel Butt, Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Meta kernel team,
linux-mm, cgroups, linux-kernel, kernel test robot
In-Reply-To: <20260520053123.2709959-4-shakeel.butt@linux.dev>
On 5/20/26 2:31 PM, Shakeel Butt wrote:
> Currently struct obj_stock_pcp stores cached slab stats in 'int' which
> is 4 bytes per counter on 64-bit machines. Switch them to int16_t to
> shrink the cached metadata.
>
> The existing PAGE_SIZE flush in __account_obj_stock() bounds *bytes at
> PAGE_SIZE on 4KiB and 16KiB page archs, well within int16_t. On 64KiB
> pages PAGE_SIZE is well above S16_MAX so that flush never fires, and a
> sufficiently long run of accumulations would overflow the cache. Add
> an explicit S16_MAX guard before each add: when the next add would
> push abs(*bytes) past S16_MAX, fold the cached value into @nr and
> flush directly via mod_objcg_mlstate() before the accumulation.
>
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> Tested-by: kernel test robot <oliver.sang@intel.com>
> ---
So arches with 64KiB sizes won't benefit from
"Even for large object >= PAGE_SIZE, the vmstat data will still be
cached locally at least once before pushing it out" case.
But its benefit is questionable (to me), might be ok.
> mm/memcontrol.c | 16 +++++++++++++---
> 1 file changed, 13 insertions(+), 3 deletions(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index b3d63d9f267c..1ed27fd06850 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -3158,7 +3158,7 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
> struct obj_stock_pcp *stock, int nr,
> struct pglist_data *pgdat, enum node_stat_item idx)
> {
> - int *bytes;
> + int16_t *bytes;
>
> /*
> * Though at the moment MAX_NUMNODES <= 1024 in all archs but let's make
> @@ -3195,6 +3195,16 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
>
> bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b
> : &stock->nr_slab_unreclaimable_b;
> + /*
> + * To avoid overflow or underflow, flush directly if accumulating @nr
> + * would push the cached value past S16_MAX.
> + */
> + if (abs(nr + *bytes) >= S16_MAX) {
nit: should be > S16_MAX?
> + nr += *bytes;
> + *bytes = 0;
> + goto direct;
> + }
> +
> /*
> * Even for large object >= PAGE_SIZE, the vmstat data will still be
> * cached locally at least once before pushing it out.
FWIW:
Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
--
Cheers,
Harry / Hyeonggon
^ permalink raw reply
* Re: [PATCH v2 1/2] mm/memcontrol: add dmem charge/uncharge functions
From: Albert Esteve @ 2026-05-20 7:22 UTC (permalink / raw)
To: Eric Chanudet
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Shakeel Butt,
Muchun Song, Andrew Morton, Maarten Lankhorst, Maxime Ripard,
Natalie Vock, Tejun Heo, Michal Koutný, Jonathan Corbet,
Shuah Khan, cgroups, linux-mm, linux-kernel, dri-devel,
T.J. Mercier, Christian König, Maxime Ripard, Dave Airlie,
linux-doc
In-Reply-To: <20260519-cgroup-dmem-memcg-double-charge-v2-1-db4d1407062b@redhat.com>
On Tue, May 19, 2026 at 6:01 PM Eric Chanudet <echanude@redhat.com> wrote:
>
> Add mem_cgroup_dmem_charge() and mem_cgroup_dmem_uncharge() to allow
> dmem pool allocations to optionally be double-charged against the memory
> controller. Take the struct cgroup from the dmem pool's css as there is
> no convenient object exported to represent these allocations. These will
> resolve the effective memory css from that cgroup and perform the
> charge.
>
> Introduce a MEMCG_DMEM stat counter to memory.stat to make the cgroup's
> dmem charge visible.
>
> Signed-off-by: Eric Chanudet <echanude@redhat.com>
Reviewed-by: Albert Esteve <aesteve@redhat.com>
> ---
> include/linux/memcontrol.h | 16 ++++++++++++
> mm/memcontrol.c | 65 ++++++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 81 insertions(+)
>
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index dc3fa687759b45748b2acee6d7f43da325eb50c1..8e1d49b87fb64e6114f3eb920293e14920290fe7 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -39,6 +39,7 @@ enum memcg_stat_item {
> MEMCG_ZSWAP_B,
> MEMCG_ZSWAPPED,
> MEMCG_ZSWAP_INCOMP,
> + MEMCG_DMEM,
> MEMCG_NR_STAT,
> };
>
> @@ -1872,6 +1873,21 @@ static inline bool mem_cgroup_zswap_writeback_enabled(struct mem_cgroup *memcg)
> }
> #endif
>
> +#if defined(CONFIG_MEMCG) && defined(CONFIG_CGROUP_DMEM)
> +bool mem_cgroup_dmem_charge(struct cgroup *cgrp, unsigned int nr_pages,
> + gfp_t gfp_mask);
> +void mem_cgroup_dmem_uncharge(struct cgroup *cgrp, unsigned int nr_pages);
> +#else
> +static inline bool mem_cgroup_dmem_charge(struct cgroup *cgrp,
> + unsigned int nr_pages, gfp_t gfp_mask)
> +{
> + return true;
> +}
> +static inline void mem_cgroup_dmem_uncharge(struct cgroup *cgrp,
> + unsigned int nr_pages)
> +{
> +}
> +#endif
>
> /* Cgroup v1-related declarations */
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index c03d4787d466803db49cdaa90e6d6ba426b7afe2..91a7ac16b6eac2d6c3700b6885a068bf8b640706 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -433,6 +433,7 @@ static const unsigned int memcg_stat_items[] = {
> MEMCG_ZSWAP_B,
> MEMCG_ZSWAPPED,
> MEMCG_ZSWAP_INCOMP,
> + MEMCG_DMEM,
> };
>
> #define NR_MEMCG_NODE_STAT_ITEMS ARRAY_SIZE(memcg_node_stat_items)
> @@ -1606,6 +1607,9 @@ static const struct memory_stat memory_stats[] = {
> #ifdef CONFIG_NUMA_BALANCING
> { "pgpromote_success", PGPROMOTE_SUCCESS },
> #endif
> +#ifdef CONFIG_CGROUP_DMEM
> + { "dmem", MEMCG_DMEM },
> +#endif
> };
>
> /* The actual unit of the state item, not the same as the output unit */
> @@ -5909,6 +5913,67 @@ static struct cftype zswap_files[] = {
> };
> #endif /* CONFIG_ZSWAP */
>
> +#ifdef CONFIG_CGROUP_DMEM
> +/**
> + * mem_cgroup_dmem_charge - charge memcg for a dmem pool allocation
> + * @cgrp: cgroup of the dmem pool
> + * @nr_pages: number of pages to charge
> + * @gfp_mask: reclaim mode
> + *
> + * Charges @nr_pages to @memcg. Returns %true if the charge fit within
> + * @memcg's configured limit, %false if it doesn't.
> + */
> +bool mem_cgroup_dmem_charge(struct cgroup *cgrp, unsigned int nr_pages,
> + gfp_t gfp_mask)
> +{
> + struct cgroup_subsys_state *mem_css;
> + struct mem_cgroup *memcg;
> +
> + /* CGROUP_DMEM and MEMCG guarantees this cannot be NULL. */
> + mem_css = cgroup_get_e_css(cgrp, &memory_cgrp_subsys);
> +
> + /* Use the memcg, if any, of the dmem cgroup. */
> + memcg = mem_cgroup_from_css(mem_css);
> + if (!memcg || mem_cgroup_is_root(memcg)) {
> + css_put(mem_css);
> + return false;
> + }
> +
> + if (try_charge_memcg(memcg, gfp_mask, nr_pages)) {
> + css_put(mem_css);
> + return false;
> + }
> +
> + mod_memcg_state(memcg, MEMCG_DMEM, nr_pages);
> + css_put(mem_css);
> + return true;
> +}
> +
> +/**
> + * mem_cgroup_dmem_uncharge - uncharge memcg from a dmem pool allocation
> + * @cgrp: cgroup of the dmem pool
> + * @nr_pages: number of pages to uncharge
> + */
> +void mem_cgroup_dmem_uncharge(struct cgroup *cgrp, unsigned int nr_pages)
> +{
> + struct cgroup_subsys_state *mem_css;
> + struct mem_cgroup *memcg;
> +
> + /* CGROUP_DMEM and MEMCG guarantees this cannot be NULL. */
> + mem_css = cgroup_get_e_css(cgrp, &memory_cgrp_subsys);
> +
> + memcg = mem_cgroup_from_css(mem_css);
> + if (!memcg || mem_cgroup_is_root(memcg)) {
> + css_put(mem_css);
> + return;
> + }
> +
> + mod_memcg_state(memcg, MEMCG_DMEM, -nr_pages);
> + refill_stock(memcg, nr_pages);
> + css_put(mem_css);
> +}
> +#endif /* CONFIG_CGROUP_DMEM */
> +
> static int __init mem_cgroup_swap_init(void)
> {
> if (mem_cgroup_disabled())
>
> --
> 2.52.0
>
^ permalink raw reply
* Re: [PATCH 2/4] memcg: uint16_t for nr_bytes in obj_stock_pcp
From: Harry Yoo @ 2026-05-20 7:01 UTC (permalink / raw)
To: Shakeel Butt, Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Meta kernel team,
linux-mm, cgroups, linux-kernel, kernel test robot
In-Reply-To: <20260520053123.2709959-3-shakeel.butt@linux.dev>
On 5/20/26 2:31 PM, Shakeel Butt wrote:
> Currently struct obj_stock_pcp stores nr_bytes in an 'unsigned int'
> which is 4 bytes on 64-bit machines. Switch the field to uint16_t to
> shrink the per-CPU cache.
>
> The kernel supports PAGE_SIZE_4KB, _8KB, _16KB, _32KB, _64KB and
> _256KB (see HAVE_PAGE_SIZE_* in arch/Kconfig). After the
> PAGE_SIZE-aligned flush in __refill_obj_stock(), the sub-page
> remainder fits in uint16_t up through 64KiB pages where PAGE_SIZE - 1
> == U16_MAX, but on 256KiB pages PAGE_SIZE - 1 == 0x3FFFF exceeds
> U16_MAX. The accumulator also needs to stay within uint16_t between
> page-aligned flushes on 64KiB pages where PAGE_SIZE itself is
> U16_MAX + 1.
>
> Accumulate the new total in an 'unsigned int' local, then:
>
> 1. Flush whenever the accumulator would hit U16_MAX. Together with
> the existing allow_uncharge flush at PAGE_SIZE, this keeps the
> uint16_t safe on PAGE_SIZE <= 64KiB.
>
> 2. On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on hexagon and
> powerpc 44x), push any sub-page remainder above U16_MAX into
> objcg->nr_charged_bytes via atomic_add before storing back, so
> the store cannot silently truncate. The PAGE_SHIFT > 16 guard
> folds the branch out at compile time on smaller page sizes.
>
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> Tested-by: kernel test robot <oliver.sang@intel.com>
> ---
> mm/memcontrol.c | 33 +++++++++++++++++++++++++++------
> 1 file changed, 27 insertions(+), 6 deletions(-)
>
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index d7c162946719..b3d63d9f267c 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -3339,21 +3340,41 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> goto out;
> }
>
> + stock_nr_bytes = stock->nr_bytes;
> if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
> drain_obj_stock(stock);
> obj_cgroup_get(objcg);
> - stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
> + stock_nr_bytes = atomic_read(&objcg->nr_charged_bytes)
> ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
> WRITE_ONCE(stock->cached_objcg, objcg);
>
> allow_uncharge = true; /* Allow uncharge when objcg changes */
> }
> - stock->nr_bytes += nr_bytes;
> + stock_nr_bytes += nr_bytes;
> +
> + /* Since stock->nr_bytes is uint16_t, don't refill >= U16_MAX */
> + if ((allow_uncharge && (stock_nr_bytes > PAGE_SIZE)) ||
> + stock_nr_bytes >= U16_MAX) {
nit: This should be > U16_MAX?
> + nr_pages = stock_nr_bytes >> PAGE_SHIFT;
> + stock_nr_bytes &= (PAGE_SIZE - 1);
> +
> + /*
> + * On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on
> + * hexagon and powerpc 44x), the sub-page remainder can
> + * still exceed U16_MAX. Push the excess back to
> + * objcg->nr_charged_bytes so the store into uint16_t
> + * cannot silently truncate; folded out at compile time
> + * on smaller page sizes.
> + */
> + if (PAGE_SHIFT > 16 && stock_nr_bytes > U16_MAX) {
> + unsigned int kept = stock_nr_bytes & U16_MAX;
>
> - if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) {
> - nr_pages = stock->nr_bytes >> PAGE_SHIFT;
> - stock->nr_bytes &= (PAGE_SIZE - 1);
> + atomic_add(stock_nr_bytes - kept,
> + &objcg->nr_charged_bytes);
> + stock_nr_bytes = kept;
> + }
> }
> + stock->nr_bytes = stock_nr_bytes;
>
> out:
> if (nr_pages)
--
Cheers,
Harry / Hyeonggon
^ permalink raw reply
* Re: [PATCH 2/4] memcg: uint16_t for nr_bytes in obj_stock_pcp
From: Harry Yoo @ 2026-05-20 6:41 UTC (permalink / raw)
To: Shakeel Butt, Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Meta kernel team,
linux-mm, cgroups, linux-kernel, kernel test robot
In-Reply-To: <20260520053123.2709959-3-shakeel.butt@linux.dev>
On 5/20/26 2:31 PM, Shakeel Butt wrote:
> Currently struct obj_stock_pcp stores nr_bytes in an 'unsigned int'
> which is 4 bytes on 64-bit machines. Switch the field to uint16_t to
> shrink the per-CPU cache.
>
> The kernel supports PAGE_SIZE_4KB, _8KB, _16KB, _32KB, _64KB and
> _256KB (see HAVE_PAGE_SIZE_* in arch/Kconfig). After the
> PAGE_SIZE-aligned flush in __refill_obj_stock(), the sub-page
> remainder fits in uint16_t up through 64KiB pages where PAGE_SIZE - 1
> == U16_MAX, but on 256KiB pages PAGE_SIZE - 1 == 0x3FFFF exceeds
> U16_MAX. The accumulator also needs to stay within uint16_t between
> page-aligned flushes on 64KiB pages where PAGE_SIZE itself is
> U16_MAX + 1.
>
> Accumulate the new total in an 'unsigned int' local, then:
>
> 1. Flush whenever the accumulator would hit U16_MAX. Together with
> the existing allow_uncharge flush at PAGE_SIZE, this keeps the
> uint16_t safe on PAGE_SIZE <= 64KiB.
>
> 2. On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on hexagon and
> powerpc 44x), push any sub-page remainder above U16_MAX into
> objcg->nr_charged_bytes via atomic_add before storing back, so
> the store cannot silently truncate. The PAGE_SHIFT > 16 guard
> folds the branch out at compile time on smaller page sizes.
>
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> Tested-by: kernel test robot <oliver.sang@intel.com>
> ---
Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
--
Cheers,
Harry / Hyeonggon
^ permalink raw reply
* [PATCH] blk-throttle: schedule parent dispatch in tg_flush_bios()
From: Tao Cui @ 2026-05-20 6:24 UTC (permalink / raw)
To: tj, josef, axboe, cgroups; +Cc: linux-block, Tao Cui
tg_flush_bios() schedules pending_timer on the child tg's own
service_queue, which causes throtl_pending_timer_fn() to dispatch from
the child's pending_tree. For leaf cgroups this tree is empty, so the
timer fires and exits without dispatching the throttled bio.
The throttled bio sits in the parent's pending_tree with disptime set
to jiffies (THROTL_TG_CANCELING zeroes all dispatch times), but the
parent's timer is never explicitly rescheduled. The bio only gets
dispatched when the parent timer eventually fires at its previously
scheduled expiry.
Fix by calling throtl_schedule_next_dispatch(sq->parent_sq, true)
instead, matching what tg_set_limit() already does. This forces the
parent's dispatch cycle to run immediately and flush all canceling
bios without waiting for a stale timer.
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
---
block/blk-throttle.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/block/blk-throttle.c b/block/blk-throttle.c
index 2d39977ba9de..0ad30b688ce1 100644
--- a/block/blk-throttle.c
+++ b/block/blk-throttle.c
@@ -1652,7 +1652,7 @@ static void tg_flush_bios(struct throtl_grp *tg)
*/
tg_update_disptime(tg);
- throtl_schedule_pending_timer(sq, jiffies + 1);
+ throtl_schedule_next_dispatch(sq->parent_sq, true);
}
static void throtl_pd_offline(struct blkg_policy_data *pd)
--
2.43.0
^ permalink raw reply related
* Re: [PATCH 1/4] memcg: store node_id instead of pglist_data pointer
From: Muchun Song @ 2026-05-20 6:13 UTC (permalink / raw)
To: Shakeel Butt
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Roman Gushchin,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260520053123.2709959-2-shakeel.butt@linux.dev>
> On May 20, 2026, at 13:31, Shakeel Butt <shakeel.butt@linux.dev> wrote:
>
> The struct obj_stock_pcp stores a pointer to pglist_data for the slab
> stats cached on the cpu. On 64-bit machines, this costs 8 bytes. The
> pointer is not strictly required: NODE_DATA() can recover it from the
> node id. Replace cached_pgdat with int16_t node_id and use NUMA_NO_NODE
> as the "no stats cached" sentinel.
>
> At the moment all the archs limit MAX_NUMNODES to 1024 so int16_t is
> plenty; a BUILD_BUG_ON() makes sure we notice if that ever changes.
>
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> Tested-by: kernel test robot <oliver.sang@intel.com>
Acked-by: Muchun Song <muchun.song@linux.dev>
Thanks.
^ permalink raw reply
* [PATCH] cgroup/dmem: implement dmem.high soft limit and throttling
From: Qiliang Yuan @ 2026-05-20 6:07 UTC (permalink / raw)
To: Maarten Lankhorst, Maxime Ripard, Natalie Vock, Tejun Heo,
Johannes Weiner, Michal Koutný
Cc: cgroups, dri-devel, linux-kernel, Qiliang Yuan
Introduce the "high" soft limit for the dmem cgroup v2 controller.
When a cgroup's device memory usage exceeds its high limit, tasks
belonging to that cgroup are throttled by being forced into a sleep
before returning to user space, instead of being failed outright
as with the "max" limit.
Key changes:
- Add high counter configuration to dmem_cgroup_pool.
- Add over-high check in the try_charge path and set TIF_NOTIFY_RESUME.
- Inject the dmem throttling handler into resume_user_mode_work.
- Implement the handler to perform a 100ms interruptible sleep for
over-limit tasks.
This mechanism provides smoother over-subscription support for device
memory resources.
Signed-off-by: Qiliang Yuan <realwujing@gmail.com>
---
This series introduces the "high" soft limit and associated task
throttling mechanism to the dmem cgroup v2 controller.
The device memory (VRAM) management currently only supports hard limits
(max), which leads to immediate allocation failures when reached. This
can be disruptive for GPU-bound AI workloads. By introducing a soft
limit, we allow cgroups to exceed their quota temporarily while
applying backpressure via task throttling before the process returns
to user space.
The mechanism is inspired by the memory cgroup's high limit:
- When usage > high, the task is marked with TIF_NOTIFY_RESUME.
- Upon returning to user space, it triggers a 100ms sleep.
- This provides a smoother over-subscription model for GPU resources.
Qiliang Yuan (1):
cgroup/dmem: implement dmem.high soft limit and throttling
---
To: Maarten Lankhorst <dev@lankhorst.se>
To: Maxime Ripard <mripard@kernel.org>
To: Natalie Vock <natalie.vock@gmx.de>
To: Tejun Heo <tj@kernel.org>
To: Johannes Weiner <hannes@cmpxchg.org>
To: Michal Koutný <mkoutny@suse.com>
Cc: cgroups@vger.kernel.org
Cc: dri-devel@lists.freedesktop.org
Cc: linux-kernel@vger.kernel.org
---
include/linux/cgroup_dmem.h | 10 +++++++
include/linux/resume_user_mode.h | 2 ++
kernel/cgroup/dmem.c | 60 +++++++++++++++++++++++++++++++++++++++-
3 files changed, 71 insertions(+), 1 deletion(-)
diff --git a/include/linux/cgroup_dmem.h b/include/linux/cgroup_dmem.h
index dd4869f1d736e..d58972de7c910 100644
--- a/include/linux/cgroup_dmem.h
+++ b/include/linux/cgroup_dmem.h
@@ -21,6 +21,13 @@ int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size,
struct dmem_cgroup_pool_state **ret_pool,
struct dmem_cgroup_pool_state **ret_limit_pool);
void dmem_cgroup_uncharge(struct dmem_cgroup_pool_state *pool, u64 size);
+void __dmem_cgroup_handle_over_high(void);
+
+static inline void dmem_cgroup_handle_over_high(void)
+{
+ __dmem_cgroup_handle_over_high();
+}
+
bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool,
struct dmem_cgroup_pool_state *test_pool,
bool ignore_low, bool *ret_hit_low);
@@ -51,6 +58,9 @@ static inline int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64
static inline void dmem_cgroup_uncharge(struct dmem_cgroup_pool_state *pool, u64 size)
{ }
+static inline void dmem_cgroup_handle_over_high(void)
+{ }
+
static inline
bool dmem_cgroup_state_evict_valuable(struct dmem_cgroup_pool_state *limit_pool,
struct dmem_cgroup_pool_state *test_pool,
diff --git a/include/linux/resume_user_mode.h b/include/linux/resume_user_mode.h
index bf92227c78d0d..afcab20998c41 100644
--- a/include/linux/resume_user_mode.h
+++ b/include/linux/resume_user_mode.h
@@ -8,6 +8,7 @@
#include <linux/memcontrol.h>
#include <linux/rseq.h>
#include <linux/blk-cgroup.h>
+#include <linux/cgroup_dmem.h>
/**
* set_notify_resume - cause resume_user_mode_work() to be called
@@ -58,6 +59,7 @@ static inline void resume_user_mode_work(struct pt_regs *regs)
mem_cgroup_handle_over_high(GFP_KERNEL);
blkcg_maybe_throttle_current();
+ dmem_cgroup_handle_over_high();
rseq_handle_slowpath(regs);
}
diff --git a/kernel/cgroup/dmem.c b/kernel/cgroup/dmem.c
index 4753a67d0f0f2..f77c692b887b1 100644
--- a/kernel/cgroup/dmem.c
+++ b/kernel/cgroup/dmem.c
@@ -15,6 +15,7 @@
#include <linux/page_counter.h>
#include <linux/parser.h>
#include <linux/refcount.h>
+#include <linux/resume_user_mode.h>
#include <linux/rculist.h>
#include <linux/slab.h>
@@ -156,6 +157,12 @@ set_resource_low(struct dmem_cgroup_pool_state *pool, u64 val)
page_counter_set_low(&pool->cnt, val);
}
+static void
+set_resource_high(struct dmem_cgroup_pool_state *pool, u64 val)
+{
+ page_counter_set_high(&pool->cnt, val);
+}
+
static void
set_resource_max(struct dmem_cgroup_pool_state *pool, u64 val)
{
@@ -167,6 +174,11 @@ static u64 get_resource_low(struct dmem_cgroup_pool_state *pool)
return pool ? READ_ONCE(pool->cnt.low) : 0;
}
+static u64 get_resource_high(struct dmem_cgroup_pool_state *pool)
+{
+ return pool ? READ_ONCE(pool->cnt.high) : 0;
+}
+
static u64 get_resource_min(struct dmem_cgroup_pool_state *pool)
{
return pool ? READ_ONCE(pool->cnt.min) : 0;
@@ -186,6 +198,7 @@ static void reset_all_resource_limits(struct dmem_cgroup_pool_state *rpool)
{
set_resource_min(rpool, 0);
set_resource_low(rpool, 0);
+ set_resource_high(rpool, PAGE_COUNTER_MAX);
set_resource_max(rpool, PAGE_COUNTER_MAX);
}
@@ -685,6 +698,9 @@ int dmem_cgroup_try_charge(struct dmem_cgroup_region *region, u64 size,
goto err;
}
+ if (page_counter_read(&pool->cnt) > READ_ONCE(pool->cnt.high))
+ set_notify_resume(current);
+
/* On success, reference from get_current_dmemcs is transferred to *ret_pool */
*ret_pool = pool;
return 0;
@@ -835,13 +851,24 @@ static ssize_t dmem_cgroup_region_low_write(struct kernfs_open_file *of,
return dmemcg_limit_write(of, buf, nbytes, off, set_resource_low);
}
+static int dmem_cgroup_region_high_show(struct seq_file *sf, void *v)
+{
+ return dmemcg_limit_show(sf, v, get_resource_high);
+}
+
+static ssize_t dmem_cgroup_region_high_write(struct kernfs_open_file *of,
+ char *buf, size_t nbytes, loff_t off)
+{
+ return dmemcg_limit_write(of, buf, nbytes, off, set_resource_high);
+}
+
static int dmem_cgroup_region_max_show(struct seq_file *sf, void *v)
{
return dmemcg_limit_show(sf, v, get_resource_max);
}
static ssize_t dmem_cgroup_region_max_write(struct kernfs_open_file *of,
- char *buf, size_t nbytes, loff_t off)
+ char *buf, size_t nbytes, loff_t off)
{
return dmemcg_limit_write(of, buf, nbytes, off, set_resource_max);
}
@@ -868,6 +895,12 @@ static struct cftype files[] = {
.seq_show = dmem_cgroup_region_low_show,
.flags = CFTYPE_NOT_ON_ROOT,
},
+ {
+ .name = "high",
+ .write = dmem_cgroup_region_high_write,
+ .seq_show = dmem_cgroup_region_high_show,
+ .flags = CFTYPE_NOT_ON_ROOT,
+ },
{
.name = "max",
.write = dmem_cgroup_region_max_write,
@@ -877,6 +910,31 @@ static struct cftype files[] = {
{ } /* Zero entry terminates. */
};
+void __dmem_cgroup_handle_over_high(void)
+{
+ struct dmemcg_state *dmemcs;
+ struct dmem_cgroup_pool_state *pool;
+
+ dmemcs = css_to_dmemcs(task_get_css(current, dmem_cgrp_id));
+ if (!dmemcs)
+ return;
+
+ rcu_read_lock();
+ list_for_each_entry_rcu(pool, &dmemcs->pools, css_node) {
+ unsigned long usage, high;
+
+ usage = page_counter_read(&pool->cnt);
+ high = READ_ONCE(pool->cnt.high);
+
+ if (usage > high)
+ schedule_timeout_killable(HZ / 10);
+ }
+ rcu_read_unlock();
+
+ css_put(&dmemcs->css);
+}
+EXPORT_SYMBOL_GPL(__dmem_cgroup_handle_over_high);
+
struct cgroup_subsys dmem_cgrp_subsys = {
.css_alloc = dmemcs_alloc,
.css_free = dmemcs_free,
---
base-commit: ab5fce87a778cb780a05984a2ca448f2b41aafbf
change-id: 20260519-feature-dmem-high-16997148dc38
Best regards,
--
Qiliang Yuan <realwujing@gmail.com>
^ permalink raw reply related
* Re: [PATCH 1/4] memcg: store node_id instead of pglist_data pointer
From: Harry Yoo @ 2026-05-20 6:01 UTC (permalink / raw)
To: Shakeel Butt, Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Meta kernel team,
linux-mm, cgroups, linux-kernel, kernel test robot
In-Reply-To: <20260520053123.2709959-2-shakeel.butt@linux.dev>
On 5/20/26 2:31 PM, Shakeel Butt wrote:
> The struct obj_stock_pcp stores a pointer to pglist_data for the slab
> stats cached on the cpu. On 64-bit machines, this costs 8 bytes. The
> pointer is not strictly required: NODE_DATA() can recover it from the
> node id. Replace cached_pgdat with int16_t node_id and use NUMA_NO_NODE
> as the "no stats cached" sentinel.
>
> At the moment all the archs limit MAX_NUMNODES to 1024 so int16_t is
> plenty; a BUILD_BUG_ON() makes sure we notice if that ever changes.
>
> Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> Tested-by: kernel test robot <oliver.sang@intel.com>
> ---
Reviewed-by: Harry Yoo (Oracle) <harry@kernel.org>
--
Cheers,
Harry / Hyeonggon
^ permalink raw reply
* [PATCH 4/4] memcg: multi objcg charge support
From: Shakeel Butt @ 2026-05-20 5:31 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260520053123.2709959-1-shakeel.butt@linux.dev>
Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
per-node type") split a memcg's single obj_cgroup into one per NUMA
node so that reparenting LRU folios can take per-node lru locks. As a
side effect, the per-CPU obj_stock_pcp -- which caches exactly one
cached_objcg -- thrashes on workloads where threads of the same memcg
run on different NUMA nodes. The kernel test robot reported a 67.7%
regression on stress-ng.switch.ops_per_sec from this pattern.
Mirror the multi-slot pattern already used by memcg_stock_pcp: turn
nr_bytes and cached_objcg into NR_OBJ_STOCK-element arrays, scan all
slots on consume/refill/account, prefer empty slots when inserting,
and evict a random slot only when full. With multiple slots a CPU can
hold the per-node objcg variants of one memcg plus a few siblings
without ever forcing a drain.
A single int8_t index records which slot the cached slab stats belong
to; the stats are flushed on slot or pgdat change. With NR_OBJ_STOCK
= 5 the layout (verified with pahole) is:
offset 0 : lock(1) + index(1) + node_id(2) + slab stats(4) = 8B
offset 8 : nr_bytes[5] = 10B
offset 18 : padding = 6B
offset 24 : cached[5] = 40B
offset 64 : (line 2) work_struct + flags (cold)
so consume_obj_stock, refill_obj_stock and the slab account path each
touch exactly one 64-byte cache line on non-debug 64-bit builds.
Reported-by: kernel test robot <oliver.sang@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202605121641.b6a60cb0-lkp@intel.com
Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Tested-by: kernel test robot <oliver.sang@intel.com>
---
mm/memcontrol.c | 185 ++++++++++++++++++++++++++++++++++--------------
1 file changed, 133 insertions(+), 52 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 1ed27fd06850..52104cbb8e7c 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -150,14 +150,14 @@ static void obj_cgroup_release(struct percpu_ref *ref)
* However, it can be PAGE_SIZE or (x * PAGE_SIZE).
*
* The following sequence can lead to it:
- * 1) CPU0: objcg == stock->cached_objcg
+ * 1) CPU0: objcg cached in one of stock->cached[i]
* 2) CPU1: we do a small allocation (e.g. 92 bytes),
* PAGE_SIZE bytes are charged
* 3) CPU1: a process from another memcg is allocating something,
* the stock if flushed,
* objcg->nr_charged_bytes = PAGE_SIZE - 92
* 5) CPU0: we do release this object,
- * 92 bytes are added to stock->nr_bytes
+ * 92 bytes are added to stock->nr_bytes[i]
* 6) CPU0: stock is flushed,
* 92 bytes are added to objcg->nr_charged_bytes
*
@@ -2017,13 +2017,25 @@ static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
.lock = INIT_LOCAL_TRYLOCK(lock),
};
+/*
+ * NR_OBJ_STOCK is sized so the entire hot path of obj_stock_pcp
+ * (lock, accounting metadata, nr_bytes[] and cached[]) fits within a
+ * single 64-byte cache line on non-debug 64-bit builds. With 5 slots:
+ * lock(1) + index(1) + node_id(2) + slab stats(4) + nr_bytes(10)
+ * + pad(6) + cached(40) == 64 bytes.
+ * A CPU can thus consume/refill/account against five different objcgs
+ * (typically per-node variants of the same memcg) while incurring at
+ * most one cache miss on the stock.
+ */
+#define NR_OBJ_STOCK 5
struct obj_stock_pcp {
local_trylock_t lock;
- uint16_t nr_bytes;
- struct obj_cgroup *cached_objcg;
+ int8_t index;
int16_t node_id;
int16_t nr_slab_reclaimable_b;
int16_t nr_slab_unreclaimable_b;
+ uint16_t nr_bytes[NR_OBJ_STOCK];
+ struct obj_cgroup *cached[NR_OBJ_STOCK];
struct work_struct work;
unsigned long flags;
@@ -2031,11 +2043,13 @@ struct obj_stock_pcp {
static DEFINE_PER_CPU_ALIGNED(struct obj_stock_pcp, obj_stock) = {
.lock = INIT_LOCAL_TRYLOCK(lock),
+ .index = -1,
.node_id = NUMA_NO_NODE,
};
static DEFINE_MUTEX(percpu_charge_mutex);
+static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i);
static void drain_obj_stock(struct obj_stock_pcp *stock);
static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
struct mem_cgroup *root_memcg);
@@ -3153,12 +3167,13 @@ static void unlock_stock(struct obj_stock_pcp *stock)
local_unlock(&obj_stock.lock);
}
-/* Call after __refill_obj_stock() to ensure stock->cached_objg == objcg */
+/* Call after __refill_obj_stock() so a slot for objcg exists in the stock */
static void __account_obj_stock(struct obj_cgroup *objcg,
struct obj_stock_pcp *stock, int nr,
struct pglist_data *pgdat, enum node_stat_item idx)
{
int16_t *bytes;
+ int i;
/*
* Though at the moment MAX_NUMNODES <= 1024 in all archs but let's make
@@ -3167,29 +3182,39 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
*/
BUILD_BUG_ON(MAX_NUMNODES >= S16_MAX);
- if (!stock || READ_ONCE(stock->cached_objcg) != objcg)
+ if (!stock)
+ goto direct;
+
+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
+ if (READ_ONCE(stock->cached[i]) == objcg)
+ break;
+ }
+ if (i == NR_OBJ_STOCK)
goto direct;
/*
* Save vmstat data in stock and skip vmstat array update unless
- * accumulating over a page of vmstat data or when pgdat changes.
+ * accumulating over a page of vmstat data or when the objcg slot or
+ * pgdat the stats belong to changes.
*/
- if (stock->node_id == NUMA_NO_NODE) {
+ if (stock->index < 0) {
+ stock->index = i;
stock->node_id = pgdat->node_id;
- } else if (stock->node_id != pgdat->node_id) {
- /* Flush the existing cached vmstat data */
+ } else if (stock->index != i || stock->node_id != pgdat->node_id) {
+ struct obj_cgroup *old = READ_ONCE(stock->cached[stock->index]);
struct pglist_data *oldpg = NODE_DATA(stock->node_id);
if (stock->nr_slab_reclaimable_b) {
- mod_objcg_mlstate(objcg, oldpg, NR_SLAB_RECLAIMABLE_B,
+ mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
stock->nr_slab_reclaimable_b);
stock->nr_slab_reclaimable_b = 0;
}
if (stock->nr_slab_unreclaimable_b) {
- mod_objcg_mlstate(objcg, oldpg, NR_SLAB_UNRECLAIMABLE_B,
+ mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
stock->nr_slab_unreclaimable_b);
stock->nr_slab_unreclaimable_b = 0;
}
+ stock->index = i;
stock->node_id = pgdat->node_id;
}
@@ -3230,10 +3255,16 @@ static bool __consume_obj_stock(struct obj_cgroup *objcg,
struct obj_stock_pcp *stock,
unsigned int nr_bytes)
{
- if (objcg == READ_ONCE(stock->cached_objcg) &&
- stock->nr_bytes >= nr_bytes) {
- stock->nr_bytes -= nr_bytes;
- return true;
+ int i;
+
+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
+ if (READ_ONCE(stock->cached[i]) != objcg)
+ continue;
+ if (stock->nr_bytes[i] >= nr_bytes) {
+ stock->nr_bytes[i] -= nr_bytes;
+ return true;
+ }
+ return false;
}
return false;
@@ -3254,16 +3285,42 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes)
return ret;
}
-static void drain_obj_stock(struct obj_stock_pcp *stock)
+/* Flush the cached slab stats (if any) back to their owning objcg/pgdat. */
+static void drain_obj_stock_stats(struct obj_stock_pcp *stock)
{
- struct obj_cgroup *old = READ_ONCE(stock->cached_objcg);
+ struct obj_cgroup *old;
+ struct pglist_data *oldpg;
+
+ if (stock->index < 0)
+ return;
+
+ old = READ_ONCE(stock->cached[stock->index]);
+ oldpg = NODE_DATA(stock->node_id);
+
+ if (stock->nr_slab_reclaimable_b) {
+ mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
+ stock->nr_slab_reclaimable_b);
+ stock->nr_slab_reclaimable_b = 0;
+ }
+ if (stock->nr_slab_unreclaimable_b) {
+ mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
+ stock->nr_slab_unreclaimable_b);
+ stock->nr_slab_unreclaimable_b = 0;
+ }
+ stock->index = -1;
+ stock->node_id = NUMA_NO_NODE;
+}
+
+static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i)
+{
+ struct obj_cgroup *old = READ_ONCE(stock->cached[i]);
if (!old)
return;
- if (stock->nr_bytes) {
- unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT;
- unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1);
+ if (stock->nr_bytes[i]) {
+ unsigned int nr_pages = stock->nr_bytes[i] >> PAGE_SHIFT;
+ unsigned int nr_bytes = stock->nr_bytes[i] & (PAGE_SIZE - 1);
if (nr_pages) {
struct mem_cgroup *memcg;
@@ -3289,46 +3346,43 @@ static void drain_obj_stock(struct obj_stock_pcp *stock)
* so it might be changed in the future.
*/
atomic_add(nr_bytes, &old->nr_charged_bytes);
- stock->nr_bytes = 0;
+ stock->nr_bytes[i] = 0;
}
- /*
- * Flush the vmstat data in current stock
- */
- if (stock->nr_slab_reclaimable_b || stock->nr_slab_unreclaimable_b) {
- struct pglist_data *oldpg = NODE_DATA(stock->node_id);
-
- if (stock->nr_slab_reclaimable_b) {
- mod_objcg_mlstate(old, oldpg,
- NR_SLAB_RECLAIMABLE_B,
- stock->nr_slab_reclaimable_b);
- stock->nr_slab_reclaimable_b = 0;
- }
- if (stock->nr_slab_unreclaimable_b) {
- mod_objcg_mlstate(old, oldpg,
- NR_SLAB_UNRECLAIMABLE_B,
- stock->nr_slab_unreclaimable_b);
- stock->nr_slab_unreclaimable_b = 0;
- }
- stock->node_id = NUMA_NO_NODE;
- }
+ /* Flush vmstat data when its owning slot is being drained. */
+ if (stock->index == i)
+ drain_obj_stock_stats(stock);
- WRITE_ONCE(stock->cached_objcg, NULL);
+ WRITE_ONCE(stock->cached[i], NULL);
obj_cgroup_put(old);
}
+static void drain_obj_stock(struct obj_stock_pcp *stock)
+{
+ int i;
+
+ for (i = 0; i < NR_OBJ_STOCK; ++i)
+ drain_obj_stock_slot(stock, i);
+}
+
static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
struct mem_cgroup *root_memcg)
{
- struct obj_cgroup *objcg = READ_ONCE(stock->cached_objcg);
+ struct obj_cgroup *objcg;
struct mem_cgroup *memcg;
bool flush = false;
+ int i;
rcu_read_lock();
- if (objcg) {
+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
+ objcg = READ_ONCE(stock->cached[i]);
+ if (!objcg)
+ continue;
memcg = obj_cgroup_memcg(objcg);
- if (memcg && mem_cgroup_is_descendant(memcg, root_memcg))
+ if (memcg && mem_cgroup_is_descendant(memcg, root_memcg)) {
flush = true;
+ break;
+ }
}
rcu_read_unlock();
@@ -3342,6 +3396,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
{
unsigned int nr_pages = 0;
unsigned int stock_nr_bytes;
+ int i, slot = -1, empty_slot = -1;
if (!stock) {
nr_pages = nr_bytes >> PAGE_SHIFT;
@@ -3350,19 +3405,45 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
goto out;
}
- stock_nr_bytes = stock->nr_bytes;
- if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
- drain_obj_stock(stock);
+ for (i = 0; i < NR_OBJ_STOCK; ++i) {
+ struct obj_cgroup *cached = READ_ONCE(stock->cached[i]);
+
+ if (!cached) {
+ if (empty_slot == -1)
+ empty_slot = i;
+ continue;
+ }
+ if (cached == objcg) {
+ slot = i;
+ break;
+ }
+ }
+
+ if (slot == -1) {
+ slot = empty_slot;
+ if (slot == -1) {
+ slot = get_random_u32_below(NR_OBJ_STOCK);
+ drain_obj_stock_slot(stock, slot);
+ }
obj_cgroup_get(objcg);
+ /*
+ * Keep the xchg result in the unsigned int local; storing
+ * it directly into stock->nr_bytes[slot] (uint16_t) would
+ * silently truncate values >= U16_MAX and bypass the flush
+ * guard below, leaking page-counter charges.
+ */
stock_nr_bytes = atomic_read(&objcg->nr_charged_bytes)
? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
- WRITE_ONCE(stock->cached_objcg, objcg);
+ WRITE_ONCE(stock->cached[slot], objcg);
allow_uncharge = true; /* Allow uncharge when objcg changes */
+ } else {
+ stock_nr_bytes = stock->nr_bytes[slot];
}
+
stock_nr_bytes += nr_bytes;
- /* Since stock->nr_bytes is uint16_t, don't refill >= U16_MAX */
+ /* nr_bytes[] is uint16_t; flush if we would refill >= U16_MAX. */
if ((allow_uncharge && (stock_nr_bytes > PAGE_SIZE)) ||
stock_nr_bytes >= U16_MAX) {
nr_pages = stock_nr_bytes >> PAGE_SHIFT;
@@ -3384,7 +3465,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
stock_nr_bytes = kept;
}
}
- stock->nr_bytes = stock_nr_bytes;
+ stock->nr_bytes[slot] = stock_nr_bytes;
out:
if (nr_pages)
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 3/4] memcg: int16_t for cached slab stats
From: Shakeel Butt @ 2026-05-20 5:31 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260520053123.2709959-1-shakeel.butt@linux.dev>
Currently struct obj_stock_pcp stores cached slab stats in 'int' which
is 4 bytes per counter on 64-bit machines. Switch them to int16_t to
shrink the cached metadata.
The existing PAGE_SIZE flush in __account_obj_stock() bounds *bytes at
PAGE_SIZE on 4KiB and 16KiB page archs, well within int16_t. On 64KiB
pages PAGE_SIZE is well above S16_MAX so that flush never fires, and a
sufficiently long run of accumulations would overflow the cache. Add
an explicit S16_MAX guard before each add: when the next add would
push abs(*bytes) past S16_MAX, fold the cached value into @nr and
flush directly via mod_objcg_mlstate() before the accumulation.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Tested-by: kernel test robot <oliver.sang@intel.com>
---
mm/memcontrol.c | 16 +++++++++++++---
1 file changed, 13 insertions(+), 3 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index b3d63d9f267c..1ed27fd06850 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2022,8 +2022,8 @@ struct obj_stock_pcp {
uint16_t nr_bytes;
struct obj_cgroup *cached_objcg;
int16_t node_id;
- int nr_slab_reclaimable_b;
- int nr_slab_unreclaimable_b;
+ int16_t nr_slab_reclaimable_b;
+ int16_t nr_slab_unreclaimable_b;
struct work_struct work;
unsigned long flags;
@@ -3158,7 +3158,7 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
struct obj_stock_pcp *stock, int nr,
struct pglist_data *pgdat, enum node_stat_item idx)
{
- int *bytes;
+ int16_t *bytes;
/*
* Though at the moment MAX_NUMNODES <= 1024 in all archs but let's make
@@ -3195,6 +3195,16 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b
: &stock->nr_slab_unreclaimable_b;
+ /*
+ * To avoid overflow or underflow, flush directly if accumulating @nr
+ * would push the cached value past S16_MAX.
+ */
+ if (abs(nr + *bytes) >= S16_MAX) {
+ nr += *bytes;
+ *bytes = 0;
+ goto direct;
+ }
+
/*
* Even for large object >= PAGE_SIZE, the vmstat data will still be
* cached locally at least once before pushing it out.
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 2/4] memcg: uint16_t for nr_bytes in obj_stock_pcp
From: Shakeel Butt @ 2026-05-20 5:31 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260520053123.2709959-1-shakeel.butt@linux.dev>
Currently struct obj_stock_pcp stores nr_bytes in an 'unsigned int'
which is 4 bytes on 64-bit machines. Switch the field to uint16_t to
shrink the per-CPU cache.
The kernel supports PAGE_SIZE_4KB, _8KB, _16KB, _32KB, _64KB and
_256KB (see HAVE_PAGE_SIZE_* in arch/Kconfig). After the
PAGE_SIZE-aligned flush in __refill_obj_stock(), the sub-page
remainder fits in uint16_t up through 64KiB pages where PAGE_SIZE - 1
== U16_MAX, but on 256KiB pages PAGE_SIZE - 1 == 0x3FFFF exceeds
U16_MAX. The accumulator also needs to stay within uint16_t between
page-aligned flushes on 64KiB pages where PAGE_SIZE itself is
U16_MAX + 1.
Accumulate the new total in an 'unsigned int' local, then:
1. Flush whenever the accumulator would hit U16_MAX. Together with
the existing allow_uncharge flush at PAGE_SIZE, this keeps the
uint16_t safe on PAGE_SIZE <= 64KiB.
2. On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on hexagon and
powerpc 44x), push any sub-page remainder above U16_MAX into
objcg->nr_charged_bytes via atomic_add before storing back, so
the store cannot silently truncate. The PAGE_SHIFT > 16 guard
folds the branch out at compile time on smaller page sizes.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Tested-by: kernel test robot <oliver.sang@intel.com>
---
mm/memcontrol.c | 33 +++++++++++++++++++++++++++------
1 file changed, 27 insertions(+), 6 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index d7c162946719..b3d63d9f267c 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2019,7 +2019,7 @@ static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
struct obj_stock_pcp {
local_trylock_t lock;
- unsigned int nr_bytes;
+ uint16_t nr_bytes;
struct obj_cgroup *cached_objcg;
int16_t node_id;
int nr_slab_reclaimable_b;
@@ -3331,6 +3331,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
bool allow_uncharge)
{
unsigned int nr_pages = 0;
+ unsigned int stock_nr_bytes;
if (!stock) {
nr_pages = nr_bytes >> PAGE_SHIFT;
@@ -3339,21 +3340,41 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
goto out;
}
+ stock_nr_bytes = stock->nr_bytes;
if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
drain_obj_stock(stock);
obj_cgroup_get(objcg);
- stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
+ stock_nr_bytes = atomic_read(&objcg->nr_charged_bytes)
? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
WRITE_ONCE(stock->cached_objcg, objcg);
allow_uncharge = true; /* Allow uncharge when objcg changes */
}
- stock->nr_bytes += nr_bytes;
+ stock_nr_bytes += nr_bytes;
+
+ /* Since stock->nr_bytes is uint16_t, don't refill >= U16_MAX */
+ if ((allow_uncharge && (stock_nr_bytes > PAGE_SIZE)) ||
+ stock_nr_bytes >= U16_MAX) {
+ nr_pages = stock_nr_bytes >> PAGE_SHIFT;
+ stock_nr_bytes &= (PAGE_SIZE - 1);
+
+ /*
+ * On configs with PAGE_SHIFT > 16 (PAGE_SIZE_256KB on
+ * hexagon and powerpc 44x), the sub-page remainder can
+ * still exceed U16_MAX. Push the excess back to
+ * objcg->nr_charged_bytes so the store into uint16_t
+ * cannot silently truncate; folded out at compile time
+ * on smaller page sizes.
+ */
+ if (PAGE_SHIFT > 16 && stock_nr_bytes > U16_MAX) {
+ unsigned int kept = stock_nr_bytes & U16_MAX;
- if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) {
- nr_pages = stock->nr_bytes >> PAGE_SHIFT;
- stock->nr_bytes &= (PAGE_SIZE - 1);
+ atomic_add(stock_nr_bytes - kept,
+ &objcg->nr_charged_bytes);
+ stock_nr_bytes = kept;
+ }
}
+ stock->nr_bytes = stock_nr_bytes;
out:
if (nr_pages)
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 1/4] memcg: store node_id instead of pglist_data pointer
From: Shakeel Butt @ 2026-05-20 5:31 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <20260520053123.2709959-1-shakeel.butt@linux.dev>
The struct obj_stock_pcp stores a pointer to pglist_data for the slab
stats cached on the cpu. On 64-bit machines, this costs 8 bytes. The
pointer is not strictly required: NODE_DATA() can recover it from the
node id. Replace cached_pgdat with int16_t node_id and use NUMA_NO_NODE
as the "no stats cached" sentinel.
At the moment all the archs limit MAX_NUMNODES to 1024 so int16_t is
plenty; a BUILD_BUG_ON() makes sure we notice if that ever changes.
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Tested-by: kernel test robot <oliver.sang@intel.com>
---
mm/memcontrol.c | 26 +++++++++++++++++++-------
1 file changed, 19 insertions(+), 7 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index b8caeb7ccaa3..d7c162946719 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -2021,7 +2021,7 @@ struct obj_stock_pcp {
local_trylock_t lock;
unsigned int nr_bytes;
struct obj_cgroup *cached_objcg;
- struct pglist_data *cached_pgdat;
+ int16_t node_id;
int nr_slab_reclaimable_b;
int nr_slab_unreclaimable_b;
@@ -2031,6 +2031,7 @@ struct obj_stock_pcp {
static DEFINE_PER_CPU_ALIGNED(struct obj_stock_pcp, obj_stock) = {
.lock = INIT_LOCAL_TRYLOCK(lock),
+ .node_id = NUMA_NO_NODE,
};
static DEFINE_MUTEX(percpu_charge_mutex);
@@ -3159,6 +3160,13 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
{
int *bytes;
+ /*
+ * Though at the moment MAX_NUMNODES <= 1024 in all archs but let's make
+ * sure it does not exceed S16_MAX otherwise we need to fix node_id type
+ * in struct obj_stock_pcp.
+ */
+ BUILD_BUG_ON(MAX_NUMNODES >= S16_MAX);
+
if (!stock || READ_ONCE(stock->cached_objcg) != objcg)
goto direct;
@@ -3166,9 +3174,11 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
* Save vmstat data in stock and skip vmstat array update unless
* accumulating over a page of vmstat data or when pgdat changes.
*/
- if (stock->cached_pgdat != pgdat) {
+ if (stock->node_id == NUMA_NO_NODE) {
+ stock->node_id = pgdat->node_id;
+ } else if (stock->node_id != pgdat->node_id) {
/* Flush the existing cached vmstat data */
- struct pglist_data *oldpg = stock->cached_pgdat;
+ struct pglist_data *oldpg = NODE_DATA(stock->node_id);
if (stock->nr_slab_reclaimable_b) {
mod_objcg_mlstate(objcg, oldpg, NR_SLAB_RECLAIMABLE_B,
@@ -3180,7 +3190,7 @@ static void __account_obj_stock(struct obj_cgroup *objcg,
stock->nr_slab_unreclaimable_b);
stock->nr_slab_unreclaimable_b = 0;
}
- stock->cached_pgdat = pgdat;
+ stock->node_id = pgdat->node_id;
}
bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b
@@ -3276,19 +3286,21 @@ static void drain_obj_stock(struct obj_stock_pcp *stock)
* Flush the vmstat data in current stock
*/
if (stock->nr_slab_reclaimable_b || stock->nr_slab_unreclaimable_b) {
+ struct pglist_data *oldpg = NODE_DATA(stock->node_id);
+
if (stock->nr_slab_reclaimable_b) {
- mod_objcg_mlstate(old, stock->cached_pgdat,
+ mod_objcg_mlstate(old, oldpg,
NR_SLAB_RECLAIMABLE_B,
stock->nr_slab_reclaimable_b);
stock->nr_slab_reclaimable_b = 0;
}
if (stock->nr_slab_unreclaimable_b) {
- mod_objcg_mlstate(old, stock->cached_pgdat,
+ mod_objcg_mlstate(old, oldpg,
NR_SLAB_UNRECLAIMABLE_B,
stock->nr_slab_unreclaimable_b);
stock->nr_slab_unreclaimable_b = 0;
}
- stock->cached_pgdat = NULL;
+ stock->node_id = NUMA_NO_NODE;
}
WRITE_ONCE(stock->cached_objcg, NULL);
--
2.53.0-Meta
^ permalink raw reply related
* [PATCH 0/4] memcg: shrink obj_stock_pcp and cache multiple objcgs
From: Shakeel Butt @ 2026-05-20 5:31 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
per-node type") split a memcg's single obj_cgroup into one per NUMA
node so that reparenting LRU folios can take per-node lru locks. As a
side effect, the per-CPU obj_stock_pcp -- which caches a single
cached_objcg pointer -- thrashes on workloads where threads of the
same memcg run on different NUMA nodes. The kernel test robot reported
a 67.7% regression on stress-ng.switch.ops_per_sec from this pattern.
Commit d0211878ce06 ("memcg: cache obj_stock by memcg, not by objcg
pointer") landed as a temporary fix by treating sibling per-node
objcgs as equivalent for the cache lookup, intended to be reverted
once per-node kmem accounting is introduced. This series takes a more
general approach: cache multiple objcgs per CPU using the multi-slot
pattern memcg_stock_pcp already uses, so the per-node objcg variants
of one memcg can all coexist in the stock without ever forcing a
drain. The temporary fix can then be reverted.
To avoid increasing the per-CPU cache footprint, the first three
patches shrink the existing single-slot obj_stock_pcp fields.
The final patch converts cached_objcg and nr_bytes into
NR_OBJ_STOCK=5 slot arrays and reorders the struct so the entire
consume/refill/account hot path fits within a single 64-byte cache
line on non-debug 64-bit builds (verified with pahole).
Reported-by: kernel test robot <oliver.sang@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202605121641.b6a60cb0-lkp@intel.com
Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
Tested-by: kernel test robot <oliver.sang@intel.com>
Shakeel Butt (4):
memcg: store node_id instead of pglist_data pointer
memcg: uint16_t for nr_bytes in obj_stock_pcp
memcg: int16_t for cached slab stats
memcg: multi objcg charge support
mm/memcontrol.c | 214 +++++++++++++++++++++++++++++++++++-------------
1 file changed, 157 insertions(+), 57 deletions(-)
--
2.53.0-Meta
^ permalink raw reply
* [PATCH 0/4] memcg: shrink obj_stock_pcp and cache multiple objcgs
From: Shakeel Butt @ 2026-05-20 5:29 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Alexandre Ghiti, Joshua Hahn, Harry Yoo,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
per-node type") split a memcg's single obj_cgroup into one per NUMA
node so that reparenting LRU folios can take per-node lru locks. As a
side effect, the per-CPU obj_stock_pcp -- which caches a single
cached_objcg pointer -- thrashes on workloads where threads of the
same memcg run on different NUMA nodes. The kernel test robot reported
a 67.7% regression on stress-ng.switch.ops_per_sec from this pattern.
Commit d0211878ce06 ("memcg: cache obj_stock by memcg, not by objcg
pointer") landed as a temporary fix by treating sibling per-node
objcgs as equivalent for the cache lookup, intended to be reverted
once per-node kmem accounting is introduced. This series takes a more
general approach: cache multiple objcgs per CPU using the multi-slot
pattern memcg_stock_pcp already uses, so the per-node objcg variants
of one memcg can all coexist in the stock without ever forcing a
drain. The temporary fix can then be reverted.
To avoid increasing the per-CPU cache footprint, the first three
patches shrink the existing single-slot obj_stock_pcp fields.
The final patch converts cached_objcg and nr_bytes into
NR_OBJ_STOCK=5 slot arrays and reorders the struct so the entire
consume/refill/account hot path fits within a single 64-byte cache
line on non-debug 64-bit builds (verified with pahole).
Reported-by: kernel test robot <oliver.sang@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202605121641.b6a60cb0-lkp@intel.com
Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
Tested-by: kernel test robot <oliver.sang@intel.com>
Shakeel Butt (4):
memcg: store node_id instead of pglist_data pointer
memcg: uint16_t for nr_bytes in obj_stock_pcp
memcg: int16_t for cached slab stats
memcg: multi objcg charge support
mm/memcontrol.c | 214 +++++++++++++++++++++++++++++++++++-------------
1 file changed, 157 insertions(+), 57 deletions(-)
--
2.53.0-Meta
^ permalink raw reply
* Re: [linus:master] [mm] 01b9da291c: stress-ng.switch.ops_per_sec 67.7% regression
From: Shakeel Butt @ 2026-05-20 4:12 UTC (permalink / raw)
To: Oliver Sang
Cc: Qi Zheng, oe-lkp, lkp, linux-kernel, Andrew Morton, David Carlier,
Allen Pais, Axel Rasmussen, Baoquan He, Chengming Zhou,
Chen Ridong, David Hildenbrand, Hamza Mahfooz, Harry Yoo,
Hugh Dickins, Imran Khan, Johannes Weiner, Kamalesh Babulal,
Lance Yang, Liam Howlett, Lorenzo Stoakes, Michal Hocko,
Michal Koutný, Mike Rapoport, Muchun Song, Muchun Song,
Nhat Pham, Roman Gushchin, Suren Baghdasaryan, Usama Arif,
Vlastimil Babka, Wei Xu, Yosry Ahmed, Yuanchu Xie, Zi Yan,
Usama Arif, cgroups, linux-mm
In-Reply-To: <ag0fyi8GjHHf8bdC@xsang-OptiPlex-9020>
On Wed, May 20, 2026 at 10:43:22AM +0800, Oliver Sang wrote:
> hi, Shakeel,
>
> On Tue, May 19, 2026 at 07:22:52AM -0700, Shakeel Butt wrote:
> > Hi Oliver,
> >
>
> [...]
>
> >
> > > > >
> > > > > Also I am rethinking the approach, so I will send a prototype in response on
> > > > > this email for which I will need your help in testing.
> > > >
> > > > Hi Oliver, can you please test the following patch?
> > >
> > > got it. will change to test following patch. and this looks quite different
> > > with v2 or v3, so if you still want us to test v3, please let me know. thanks!
> > >
> >
> > No need to test v3 as it is similar to v2. Please test the following patch as it
> > is a direction I want to pursue and wanted an early signal if this is the right
> > direction.
>
> FYI. in our tests, the following patch also recovers the regresion.
>
> Tested-by: kernel test robot <oliver.sang@intel.com>
Thanks a lot. I will send out the formal patch with your tested-by tag.
^ permalink raw reply
* [PATCH] cgroup: rstat: relax NMI guard after switch to try_cmpxchg
From: Cunlong Li @ 2026-05-20 3:30 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Shakeel Butt
Cc: cgroups, linux-kernel, Cunlong Li
Commit 36df6e3dbd7e ("cgroup: make css_rstat_updated nmi safe") used
this_cpu_cmpxchg() for the lockless insertion, and therefore required
both ARCH_HAVE_NMI_SAFE_CMPXCHG and ARCH_HAS_NMI_SAFE_THIS_CPU_OPS in
the NMI guard: on archs without the latter, this_cpu_cmpxchg() falls
back to "local_irq_save() + plain cmpxchg", and local_irq_save()
cannot mask NMIs.
Commit 3309b63a2281 ("cgroup: rstat: use LOCK CMPXCHG in
css_rstat_updated") later replaced this_cpu_cmpxchg() with plain
try_cmpxchg() to fix cross-CPU lockless-list corruption, but left the
NMI guard untouched. After that switch, css_rstat_updated() no longer
performs any this_cpu_*() RMW operations and only relies on the arch
having NMI-safe cmpxchg, so ARCH_HAS_NMI_SAFE_THIS_CPU_OPS is no
longer required in the guard.
Relax the guard accordingly so that archs which have HAVE_NMI and
ARCH_HAVE_NMI_SAFE_CMPXCHG but not ARCH_HAS_NMI_SAFE_THIS_CPU_OPS
(e.g. sparc, powerpc on PPC64/BOOK3S) can benefit from the existing
CONFIG_MEMCG_NMI_SAFETY_REQUIRES_ATOMIC path. Without this, the css
is never queued in NMI on those archs, and the atomics staged by
account_{slab,kmem}_nmi_safe() are not drained by flush_nmi_stats().
Fixes: 3309b63a2281 ("cgroup: rstat: use LOCK CMPXCHG in css_rstat_updated")
Signed-off-by: Cunlong Li <shenxiaogll@gmail.com>
---
kernel/cgroup/rstat.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c
index 150e5871e66f..fa46611098a5 100644
--- a/kernel/cgroup/rstat.c
+++ b/kernel/cgroup/rstat.c
@@ -83,11 +83,10 @@ __bpf_kfunc void css_rstat_updated(struct cgroup_subsys_state *css, int cpu)
lockdep_assert_preemption_disabled();
/*
- * For archs withnot nmi safe cmpxchg or percpu ops support, ignore
- * the requests from nmi context.
+ * The lockless insertion below relies on NMI-safe cmpxchg;
+ * bail out in NMI on archs that don't provide it.
*/
- if ((!IS_ENABLED(CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG) ||
- !IS_ENABLED(CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS)) && in_nmi())
+ if (!IS_ENABLED(CONFIG_ARCH_HAVE_NMI_SAFE_CMPXCHG) && in_nmi())
return;
rstatc = css_rstat_cpu(css, cpu);
---
base-commit: 27fa82620cbaa89a7fc11ac3057701d598813e87
change-id: 20260520-nmi-0493a1569716
Best regards,
--
Cunlong Li <shenxiaogll@gmail.com>
^ permalink raw reply related
* Re: [linus:master] [mm] 01b9da291c: stress-ng.switch.ops_per_sec 67.7% regression
From: Oliver Sang @ 2026-05-20 2:43 UTC (permalink / raw)
To: Shakeel Butt
Cc: Qi Zheng, oe-lkp, lkp, linux-kernel, Andrew Morton, David Carlier,
Allen Pais, Axel Rasmussen, Baoquan He, Chengming Zhou,
Chen Ridong, David Hildenbrand, Hamza Mahfooz, Harry Yoo,
Hugh Dickins, Imran Khan, Johannes Weiner, Kamalesh Babulal,
Lance Yang, Liam Howlett, Lorenzo Stoakes, Michal Hocko,
Michal Koutný, Mike Rapoport, Muchun Song, Muchun Song,
Nhat Pham, Roman Gushchin, Suren Baghdasaryan, Usama Arif,
Vlastimil Babka, Wei Xu, Yosry Ahmed, Yuanchu Xie, Zi Yan,
Usama Arif, cgroups, linux-mm, oliver.sang
In-Reply-To: <agxxuHOfNLX-32kI@linux.dev>
hi, Shakeel,
On Tue, May 19, 2026 at 07:22:52AM -0700, Shakeel Butt wrote:
> Hi Oliver,
>
[...]
>
> > > >
> > > > Also I am rethinking the approach, so I will send a prototype in response on
> > > > this email for which I will need your help in testing.
> > >
> > > Hi Oliver, can you please test the following patch?
> >
> > got it. will change to test following patch. and this looks quite different
> > with v2 or v3, so if you still want us to test v3, please let me know. thanks!
> >
>
> No need to test v3 as it is similar to v2. Please test the following patch as it
> is a direction I want to pursue and wanted an early signal if this is the right
> direction.
FYI. in our tests, the following patch also recovers the regresion.
Tested-by: kernel test robot <oliver.sang@intel.com>
=========================================================================================
compiler/cpufreq_governor/kconfig/method/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-14/performance/x86_64-rhel-9.4/mq/100%/debian-13-x86_64-20250902.cgz/lkp-spr-r02/switch/stress-ng/60s
commit:
8285917d6f ("mm: memcontrol: prepare for reparenting non-hierarchical stats")
01b9da291c ("mm: memcontrol: convert objcg to be per-memcg per-node type")
542b7317fe ("memcg: shrink obj_stock_pcp and cache multiple objcgs")
8285917d6f383aef 01b9da291c4969354807b52956f 542b7317fe304742e79a9a2215e
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
5849 +210.2% 18145 ± 3% +1.7% 5949 stress-ng.switch.nanosecs_per_context_switch_mq_method
2.296e+09 -67.7% 7.408e+08 ± 3% -1.7% 2.257e+09 stress-ng.switch.ops
38288993 -67.7% 12355813 ± 3% -1.7% 37646995 stress-ng.switch.ops_per_sec
93416932 -68.6% 29310048 ± 3% -0.7% 92764788 stress-ng.time.involuntary_context_switches
15845 +11.0% 17584 +0.2% 15882 stress-ng.time.percent_of_cpu_this_job_got
8556 +18.2% 10115 +0.4% 8589 stress-ng.time.system_time
963.36 -53.5% 447.72 ± 3% -1.1% 952.66 stress-ng.time.user_time
1.518e+09 -69.7% 4.607e+08 ± 2% -1.6% 1.494e+09 stress-ng.time.voluntary_context_switches
1124 ± 17% +34.3% 1509 ± 8% -4.0% 1079 ± 27% perf-c2c.HITM.remote
2.55e+09 -12.3% 2.236e+09 ± 3% +5.3% 2.685e+09 ± 6% cpuidle..time
8.29e+08 -71.8% 2.337e+08 ± 2% -2.1% 8.12e+08 cpuidle..usage
14184409 ± 2% -16.4% 11860068 +0.5% 14258186 vmstat.memory.cache
39204964 -69.7% 11868752 ± 2% -2.9% 38078258 vmstat.system.cs
1808848 -38.5% 1111830 -0.6% 1798192 vmstat.system.in
389109 ± 23% -20.4% 309611 ± 49% +27.8% 497404 ± 8% numa-numastat.node0.local_node
504867 ± 9% -13.8% 435237 ± 27% +23.0% 621221 ± 10% numa-numastat.node0.numa_hit
4102960 ± 5% -19.0% 3324393 ± 4% -2.5% 3998654 numa-numastat.node1.local_node
4218983 ± 3% -18.7% 3430325 ± 3% -2.6% 4107779 ± 2% numa-numastat.node1.numa_hit
93416932 -68.6% 29310048 ± 3% -0.7% 92764788 time.involuntary_context_switches
15845 +11.0% 17584 +0.2% 15882 time.percent_of_cpu_this_job_got
8556 +18.2% 10115 +0.4% 8589 time.system_time
963.36 -53.5% 447.72 ± 3% -1.1% 952.66 time.user_time
1.518e+09 -69.7% 4.607e+08 ± 2% -1.6% 1.494e+09 time.voluntary_context_switches
22.48 -4.4 18.08 +0.5 22.97 ± 4% mpstat.cpu.all.idle%
1.13 -0.4 0.73 -0.0 1.12 mpstat.cpu.all.irq%
0.10 -0.0 0.09 -0.0 0.09 mpstat.cpu.all.soft%
67.98 +9.1 77.06 -0.3 67.69 mpstat.cpu.all.sys%
8.32 -4.3 4.04 ± 2% -0.2 8.14 mpstat.cpu.all.usr%
17.33 ± 2% +15.4% 20.00 ± 4% +2.4% 17.75 ± 4% mpstat.max_utilization.seconds
10939677 ± 2% -21.0% 8641166 +0.8% 11022621 ± 2% meminfo.Active
10939661 ± 2% -21.0% 8641149 +0.8% 11022605 ± 2% meminfo.Active(anon)
13917673 ± 2% -16.4% 11633722 +0.6% 14000937 meminfo.Cached
14400924 ± 2% -16.0% 12102150 +0.3% 14443477 meminfo.Committed_AS
8394752 ± 5% +16.7% 9796949 ± 8% +10.4% 9266688 ± 5% meminfo.DirectMap2M
617671 -12.0% 543559 +3.0% 636127 ± 3% meminfo.Mapped
18364992 -12.5% 16065468 +0.3% 18427193 meminfo.Memused
10124702 ± 2% -22.6% 7839682 +0.8% 10208910 ± 2% meminfo.Shmem
18393665 -12.5% 16100473 +0.4% 18462892 meminfo.max_used_kB
60023 ± 85% +48.1% 88868 ± 8% +103.5% 122159 ± 17% numa-meminfo.node0.Mapped
1370772 ±124% +153.5% 3474876 +175.9% 3782461 numa-meminfo.node0.Unevictable
10552401 ± 4% -23.3% 8092823 +1.9% 10752961 numa-meminfo.node1.Active
10552392 ± 4% -23.3% 8092820 +1.9% 10752957 numa-meminfo.node1.Active(anon)
89021 ± 81% -1.6% 87590 ± 71% +108.4% 185539 ± 2% numa-meminfo.node1.AnonHugePages
12454155 ± 15% -34.9% 8106052 -18.8% 10108844 ± 2% numa-meminfo.node1.FilePages
20917 ± 9% -3.6% 20156 ± 9% +8.5% 22693 ± 6% numa-meminfo.node1.KernelStack
559046 ± 8% -19.2% 451929 ± 2% -7.8% 515612 ± 4% numa-meminfo.node1.Mapped
14688311 ± 13% -30.0% 10285394 ± 2% -15.7% 12377651 ± 3% numa-meminfo.node1.MemUsed
10028979 ± 3% -22.4% 7783864 +0.7% 10096369 ± 2% numa-meminfo.node1.Shmem
2425126 ± 70% -86.7% 322088 -99.5% 12421 ±122% numa-meminfo.node1.Unevictable
10.59 -7.6 2.97 ± 8% +0.1 10.69 turbostat.C1%
0.85 ± 3% +9.1 9.96 ± 2% +0.0 0.87 ± 3% turbostat.C1E%
1.29 ± 6% +19.4% 1.54 ± 2% -1.4% 1.27 ± 16% turbostat.CPU%c1
48.67 ± 2% -15.1% 41.33 ± 3% -4.5% 46.50 ± 2% turbostat.CoreTmp
0.56 -60.7% 0.22 ± 3% +0.9% 0.56 turbostat.IPC
1.153e+08 -38.7% 70680365 +0.6% 1.16e+08 turbostat.IRQ
10242404 -14.8% 8723704 -0.3% 10215225 turbostat.NMI
88.65 -84.0 4.67 ± 33% -2.9 85.77 ± 2% turbostat.PKG_%
3.82 -3.8 0.04 ± 10% -0.2 3.60 ± 2% turbostat.POLL%
48.67 ± 2% -13.7% 42.00 ± 3% -4.5% 46.50 ± 2% turbostat.PkgTmp
683.77 -13.1% 594.00 -0.2% 682.36 turbostat.PkgWatt
18.74 -3.3% 18.13 -0.6% 18.62 turbostat.RAMWatt
2735312 ± 2% -21.0% 2160742 +0.8% 2756101 ± 2% proc-vmstat.nr_active_anon
204708 -1.6% 201435 -0.1% 204540 proc-vmstat.nr_anon_pages
3479812 ± 2% -16.4% 2908863 +0.6% 3500665 proc-vmstat.nr_file_pages
154477 -12.0% 135959 +3.0% 159129 ± 3% proc-vmstat.nr_mapped
2531568 ± 2% -22.6% 1960353 +0.8% 2552660 ± 2% proc-vmstat.nr_shmem
42010 -3.5% 40543 +0.7% 42288 proc-vmstat.nr_slab_reclaimable
2735312 ± 2% -21.0% 2160742 +0.8% 2756101 ± 2% proc-vmstat.nr_zone_active_anon
210167 ± 5% -11.5% 185950 ± 11% -6.6% 196220 ± 5% proc-vmstat.numa_hint_faults
4730338 ± 2% -18.2% 3871343 +0.1% 4733860 proc-vmstat.numa_hit
4498551 ± 2% -19.1% 3639783 +0.1% 4500902 ± 2% proc-vmstat.numa_local
4808959 ± 2% -17.8% 3954157 -0.3% 4796169 proc-vmstat.pgalloc_normal
806619 -5.1% 765525 ± 2% +1.1% 815650 ± 2% proc-vmstat.pgfault
34098 ± 3% -14.8% 29054 +0.7% 34328 ± 8% proc-vmstat.pgreuse
15033 ± 85% +47.8% 22227 ± 8% +103.3% 30562 ± 17% numa-vmstat.node0.nr_mapped
342693 ±124% +153.5% 868719 +175.9% 945615 numa-vmstat.node0.nr_unevictable
342693 ±124% +153.5% 868719 +175.9% 945615 numa-vmstat.node0.nr_zone_unevictable
505319 ± 9% -14.0% 434390 ± 27% +23.1% 622018 ± 10% numa-vmstat.node0.numa_hit
389561 ± 23% -20.7% 308764 ± 50% +27.9% 498200 ± 8% numa-vmstat.node0.numa_local
2638537 ± 4% -23.3% 2022639 +1.9% 2688595 numa-vmstat.node1.nr_active_anon
3113944 ± 15% -34.9% 2025946 -18.8% 2527547 ± 2% numa-vmstat.node1.nr_file_pages
20917 ± 9% -3.6% 20157 ± 9% +8.5% 22691 ± 6% numa-vmstat.node1.nr_kernel_stack
139848 ± 9% -19.3% 112912 ± 2% -7.8% 128986 ± 4% numa-vmstat.node1.nr_mapped
2507650 ± 3% -22.4% 1945399 +0.7% 2524428 ± 2% numa-vmstat.node1.nr_shmem
606281 ± 70% -86.7% 80522 -99.5% 3105 ±122% numa-vmstat.node1.nr_unevictable
2638531 ± 4% -23.3% 2022634 +1.9% 2688587 numa-vmstat.node1.nr_zone_active_anon
606281 ± 70% -86.7% 80522 -99.5% 3105 ±122% numa-vmstat.node1.nr_zone_unevictable
4219206 ± 3% -18.7% 3430093 ± 3% -2.6% 4108044 ± 2% numa-vmstat.node1.numa_hit
4103183 ± 4% -19.0% 3324161 ± 4% -2.5% 3998968 numa-vmstat.node1.numa_local
0.11 +59.9% 0.17 ± 3% -0.9% 0.11 ± 2% perf-stat.i.MPKI
6.653e+10 -61.7% 2.546e+10 ± 2% +0.2% 6.669e+10 perf-stat.i.branch-instructions
0.76 +0.1 0.89 +0.1 0.82 perf-stat.i.branch-miss-rate%
4.685e+08 -59.7% 1.888e+08 ± 2% +8.4% 5.079e+08 perf-stat.i.branch-misses
1.12 +0.6 1.76 ± 3% +0.0 1.13 ± 3% perf-stat.i.cache-miss-rate%
35553724 ± 3% -40.4% 21188697 -0.8% 35253639 ± 2% perf-stat.i.cache-misses
4.194e+09 -68.3% 1.331e+09 ± 2% -1.6% 4.129e+09 perf-stat.i.cache-references
40710745 -69.6% 12395879 ± 2% -1.5% 40099753 perf-stat.i.context-switches
1.84 +189.1% 5.31 ± 2% -0.5% 1.83 perf-stat.i.cpi
5.965e+11 -2.0% 5.848e+11 -0.2% 5.956e+11 perf-stat.i.cpu-cycles
8813175 -64.5% 3125097 ± 2% -1.3% 8695118 perf-stat.i.cpu-migrations
24447 ± 3% +68.5% 41184 ± 2% -0.4% 24353 perf-stat.i.cycles-between-cache-misses
3.374e+11 -61.8% 1.287e+11 ± 2% +0.1% 3.377e+11 perf-stat.i.instructions
0.57 -60.8% 0.22 ± 2% +0.3% 0.57 perf-stat.i.ipc
221.10 -68.6% 69.32 ± 2% -1.5% 217.85 perf-stat.i.metric.K/sec
11782 ± 3% -6.1% 11068 ± 3% -2.7% 11463 ± 2% perf-stat.i.minor-faults
11782 ± 3% -6.1% 11068 ± 3% -2.7% 11463 ± 2% perf-stat.i.page-faults
0.10 ± 2% +59.2% 0.17 ± 3% -0.8% 0.10 perf-stat.overall.MPKI
0.71 +0.0 0.75 +0.1 0.77 perf-stat.overall.branch-miss-rate%
0.83 ± 3% +0.7 1.56 ± 3% +0.0 0.83 perf-stat.overall.cache-miss-rate%
1.78 +162.2% 4.67 ± 2% -0.4% 1.78 perf-stat.overall.cpi
17181 ± 3% +64.6% 28283 +0.4% 17248 perf-stat.overall.cycles-between-cache-misses
0.56 -61.8% 0.21 ± 2% +0.4% 0.56 perf-stat.overall.ipc
6.388e+10 -62.3% 2.409e+10 ± 2% +0.8% 6.439e+10 perf-stat.ps.branch-instructions
4.538e+08 -60.0% 1.817e+08 ± 2% +8.9% 4.941e+08 perf-stat.ps.branch-misses
33674051 ± 3% -40.1% 20155290 -0.2% 33597494 perf-stat.ps.cache-misses
4.077e+09 -68.2% 1.296e+09 ± 2% -1.2% 4.027e+09 perf-stat.ps.cache-references
39570629 -69.5% 12072702 ± 2% -1.2% 39106938 perf-stat.ps.context-switches
5.78e+11 -1.4% 5.7e+11 +0.2% 5.793e+11 perf-stat.ps.cpu-cycles
8584979 -64.5% 3051930 ± 2% -1.0% 8495000 perf-stat.ps.cpu-migrations
3.243e+11 -62.4% 1.22e+11 ± 2% +0.6% 3.264e+11 perf-stat.ps.instructions
11022 ± 4% -6.5% 10300 ± 3% -1.8% 10820 perf-stat.ps.minor-faults
11022 ± 4% -6.5% 10300 ± 3% -1.8% 10821 perf-stat.ps.page-faults
1.941e+13 -61.9% 7.405e+12 ± 3% +2.3% 1.986e+13 ± 2% perf-stat.total.instructions
18451 +9.9% 20272 -0.5% 18360 sched_debug.cfs_rq:/.avg_vruntime.avg
5869 ± 4% -7.4% 5437 ± 5% -10.1% 5278 ± 2% sched_debug.cfs_rq:/.avg_vruntime.stddev
0.68 ± 2% -12.9% 0.59 ± 3% -2.7% 0.66 ± 4% sched_debug.cfs_rq:/.h_nr_queued.stddev
0.62 ± 6% -12.9% 0.54 ± 2% -3.3% 0.60 ± 4% sched_debug.cfs_rq:/.h_nr_runnable.stddev
8469 +12.7% 9544 ± 2% +1.3% 8581 ± 3% sched_debug.cfs_rq:/.left_deadline.stddev
8467 +12.7% 9542 ± 2% +1.3% 8579 ± 3% sched_debug.cfs_rq:/.left_vruntime.stddev
3513124 ± 25% -30.0% 2459550 ± 10% -13.8% 3028548 ± 14% sched_debug.cfs_rq:/.load.max
588329 ± 5% -11.2% 522578 ± 5% +0.6% 591849 ± 3% sched_debug.cfs_rq:/.load.stddev
50699 ± 17% -19.8% 40655 ± 7% -24.8% 38144 ± 5% sched_debug.cfs_rq:/.load_avg.max
7968 ± 26% -9.9% 7176 ± 15% -30.7% 5519 ± 9% sched_debug.cfs_rq:/.load_avg.stddev
0.68 ± 2% -12.9% 0.59 ± 3% -2.8% 0.66 ± 4% sched_debug.cfs_rq:/.nr_queued.stddev
38.80 ± 32% +108.5% 80.90 ± 16% -14.7% 33.09 ± 15% sched_debug.cfs_rq:/.removed.load_avg.avg
857.83 ± 12% +61.0% 1381 ± 12% +1.4% 870.00 ± 17% sched_debug.cfs_rq:/.removed.load_avg.max
152.02 ± 18% +57.2% 239.02 ± 11% -8.0% 139.91 ± 8% sched_debug.cfs_rq:/.removed.load_avg.stddev
26.08 ± 28% +143.0% 63.37 ± 14% -6.3% 24.43 ± 18% sched_debug.cfs_rq:/.removed.runnable_avg.avg
547.00 ± 13% +88.7% 1032 ± 12% +2.7% 561.75 ± 22% sched_debug.cfs_rq:/.removed.runnable_avg.max
94.86 ± 17% +84.3% 174.82 ± 9% -2.9% 92.15 ± 9% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
9.09 ± 52% +253.3% 32.11 ± 17% -34.3% 5.97 ± 15% sched_debug.cfs_rq:/.removed.util_avg.avg
275.17 ± 3% +130.3% 633.67 ± 9% +0.5% 276.62 sched_debug.cfs_rq:/.removed.util_avg.max
44.90 ± 30% +126.4% 101.66 ± 11% -13.2% 38.95 ± 8% sched_debug.cfs_rq:/.removed.util_avg.stddev
8467 +12.7% 9542 ± 2% +1.3% 8579 ± 3% sched_debug.cfs_rq:/.right_vruntime.stddev
659.63 ± 3% +13.0% 745.47 -1.3% 651.21 sched_debug.cfs_rq:/.runnable_avg.avg
271.34 ± 2% +31.2% 355.98 ± 3% -3.1% 262.84 ± 4% sched_debug.cfs_rq:/.runnable_avg.stddev
0.00 ± 26% +110.4% 0.00 ± 45% +1.8% 0.00 ± 32% sched_debug.cfs_rq:/.spread.avg
0.01 ± 13% +174.3% 0.02 ± 25% -28.3% 0.00 ± 26% sched_debug.cfs_rq:/.spread.max
0.00 ± 7% +146.2% 0.00 ± 27% -13.4% 0.00 ± 26% sched_debug.cfs_rq:/.spread.stddev
431.00 +14.5% 493.62 -2.7% 419.48 sched_debug.cfs_rq:/.util_avg.avg
1061 ± 3% +26.4% 1341 ± 3% -0.6% 1055 ± 2% sched_debug.cfs_rq:/.util_avg.max
151.53 ± 5% +50.1% 227.46 ± 2% -9.2% 137.60 ± 5% sched_debug.cfs_rq:/.util_avg.stddev
206.96 +17.5% 243.18 ± 3% +8.4% 224.27 ± 8% sched_debug.cfs_rq:/.util_est.avg
18451 +9.9% 20272 -0.5% 18360 sched_debug.cfs_rq:/.zero_vruntime.avg
5869 ± 4% -7.4% 5437 ± 5% -10.1% 5278 ± 2% sched_debug.cfs_rq:/.zero_vruntime.stddev
2345 +33.6% 3133 ± 5% +20.6% 2829 ± 11% sched_debug.cpu.avg_idle.min
13.18 ± 2% +39.8% 18.42 ± 6% +4.1% 13.72 sched_debug.cpu.clock.stddev
3961 +14.6% 4541 +6.0% 4197 ± 4% sched_debug.cpu.curr->pid.avg
3213 -15.4% 2718 -5.8% 3028 ± 3% sched_debug.cpu.curr->pid.stddev
0.00 ± 29% +157.3% 0.00 ± 35% +101.7% 0.00 ± 51% sched_debug.cpu.next_balance.stddev
0.70 -15.8% 0.59 ± 3% -5.0% 0.66 ± 2% sched_debug.cpu.nr_running.stddev
5474800 -69.7% 1660250 ± 2% -1.6% 5386873 sched_debug.cpu.nr_switches.avg
5648642 -65.5% 1946319 ± 5% -1.5% 5561645 sched_debug.cpu.nr_switches.max
2229198 ± 8% -67.1% 734011 ± 20% +9.1% 2431460 ± 15% sched_debug.cpu.nr_switches.min
297592 ± 6% -25.9% 220513 ± 18% -3.5% 287202 ± 9% sched_debug.cpu.nr_switches.stddev
23.75 -10.9 12.88 -6.3 17.48 ± 57% perf-profile.calltrace.cycles-pp.common_startup_64
23.65 -10.8 12.82 -6.2 17.41 ± 57% perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
23.62 -10.8 12.81 -6.2 17.39 ± 57% perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
23.51 -10.8 12.76 -6.2 17.30 ± 57% perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
12.93 -7.0 5.94 ± 4% -3.3 9.65 ± 57% perf-profile.calltrace.cycles-pp.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
12.78 -6.9 5.89 ± 4% -3.2 9.54 ± 57% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
11.30 -5.2 6.07 ± 3% -3.0 8.34 ± 57% perf-profile.calltrace.cycles-pp.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
11.17 -5.2 6.02 ± 3% -2.9 8.24 ± 57% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
9.89 -4.8 5.08 ± 4% -2.6 7.32 ± 57% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q
4.52 -4.5 0.00 -1.3 3.21 ± 57% perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
12.41 -4.4 7.96 -3.3 9.09 ± 57% perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
9.19 -4.4 4.77 ± 4% -2.4 6.79 ± 57% perf-profile.calltrace.cycles-pp.select_idle_cpu.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up
11.29 -4.0 7.29 -3.0 8.27 ± 57% perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
11.38 -4.0 7.40 -3.0 8.34 ± 57% perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
8.04 -3.9 4.13 ± 4% -2.1 5.94 ± 57% perf-profile.calltrace.cycles-pp.select_idle_core.select_idle_cpu.select_idle_sibling.select_task_rq_fair.select_task_rq
6.91 -3.8 3.08 ± 4% -1.8 5.15 ± 57% perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive
6.76 -3.8 3.00 ± 4% -1.7 5.04 ± 57% perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedreceive
8.71 -3.7 5.00 -2.2 6.52 ± 57% perf-profile.calltrace.cycles-pp.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.71 -3.4 2.26 ± 4% -1.5 4.21 ± 57% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.common_startup_64
8.12 -3.4 4.72 -2.1 6.07 ± 57% perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
7.76 -3.2 4.55 -2.0 5.80 ± 57% perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend
8.39 -3.1 5.26 -2.1 6.26 ± 57% perf-profile.calltrace.cycles-pp.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
7.47 -3.1 4.35 -1.9 5.59 ± 57% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend
4.92 -2.9 2.00 ± 4% -1.3 3.62 ± 57% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
7.79 -2.8 4.97 -2.0 5.79 ± 57% perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
7.48 -2.7 4.79 -1.9 5.57 ± 57% perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive
7.17 -2.6 4.60 -1.8 5.34 ± 57% perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive
5.54 -2.3 3.24 ± 4% -1.5 4.05 ± 57% perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend
5.41 -2.2 3.16 ± 4% -1.5 3.95 ± 57% perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedsend
3.88 -2.2 1.67 ± 4% -1.0 2.86 ± 57% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry
4.04 -2.2 1.88 -1.0 3.00 ± 57% perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.common_startup_64
3.44 -2.2 1.28 ± 3% -0.9 2.54 ± 57% perf-profile.calltrace.cycles-pp.store_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.84 -2.0 1.80 -1.0 2.84 ± 57% perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
4.83 -2.0 2.85 -1.2 3.62 ± 57% perf-profile.calltrace.cycles-pp.try_to_block_task.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep
4.71 -1.9 2.78 -1.2 3.53 ± 57% perf-profile.calltrace.cycles-pp.dequeue_task_fair.try_to_block_task.__schedule.schedule.schedule_hrtimeout_range_clock
1.84 -1.8 0.00 -0.5 1.35 ± 57% perf-profile.calltrace.cycles-pp.wake_affine.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q
4.39 -1.8 2.58 -1.1 3.30 ± 57% perf-profile.calltrace.cycles-pp.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule.schedule
2.37 -1.5 0.84 ± 3% -0.6 1.76 ± 57% perf-profile.calltrace.cycles-pp._copy_to_user.store_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
2.66 -1.4 1.26 ± 4% -0.7 1.96 ± 57% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle
2.73 -1.4 1.33 ± 4% -0.7 2.01 ± 57% perf-profile.calltrace.cycles-pp.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.50 -1.3 2.17 -0.9 2.62 ± 57% perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule
2.37 -1.3 1.06 ± 2% -0.6 1.78 ± 57% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.29 -1.3 0.00 -0.4 0.94 ± 57% perf-profile.calltrace.cycles-pp._raw_spin_lock.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.23 -1.2 0.00 -0.3 0.90 ± 57% perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule_idle.do_idle.cpu_startup_entry
1.16 -1.2 0.00 -0.3 0.86 ± 57% perf-profile.calltrace.cycles-pp.ttwu_queue_wakelist.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend
1.15 -1.2 0.00 -0.6 0.59 ± 67% perf-profile.calltrace.cycles-pp.task_h_load.wake_affine.select_task_rq_fair.select_task_rq.try_to_wake_up
1.15 -1.1 0.00 -0.3 0.84 ± 57% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule_idle.do_idle
4.10 -1.1 3.03 -1.0 3.07 ± 57% perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep
2.20 -1.1 1.13 ± 5% -0.6 1.62 ± 57% perf-profile.calltrace.cycles-pp.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue
2.20 -1.1 1.15 ± 4% -0.6 1.62 ± 57% perf-profile.calltrace.cycles-pp.switch_fpu_return.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.20 -1.0 0.17 ±141% -0.3 0.89 ± 57% perf-profile.calltrace.cycles-pp.do_perf_trace_sched_wakeup_template.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedsend
1.74 -1.0 0.73 ± 3% -0.4 1.34 ± 57% perf-profile.calltrace.cycles-pp.msg_get.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.32 -1.0 0.37 ± 70% -0.3 0.98 ± 57% perf-profile.calltrace.cycles-pp.do_perf_trace_sched_wakeup_template.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedreceive
1.93 -0.9 0.99 ± 4% -0.5 1.42 ± 57% perf-profile.calltrace.cycles-pp.enqueue_task_fair.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue
0.93 -0.9 0.00 -0.3 0.67 ± 57% perf-profile.calltrace.cycles-pp.__check_object_size.store_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
0.88 -0.9 0.00 -0.2 0.64 ± 57% perf-profile.calltrace.cycles-pp.llist_reverse_order.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry
1.34 -0.8 0.54 ± 5% -0.3 1.00 ± 57% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive
1.52 -0.8 0.73 ± 4% -0.4 1.12 ± 57% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.enqueue_task.ttwu_do_activate.sched_ttwu_pending
0.74 -0.7 0.00 -0.2 0.55 ± 57% perf-profile.calltrace.cycles-pp.__smp_call_single_queue.ttwu_queue_wakelist.try_to_wake_up.wake_up_q.do_mq_timedsend
1.05 -0.7 0.34 ± 70% -0.3 0.78 ± 57% perf-profile.calltrace.cycles-pp.__switch_to
1.57 -0.7 0.90 ± 6% -0.4 1.17 ± 57% perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.67 -0.7 0.00 -0.2 0.46 ± 57% perf-profile.calltrace.cycles-pp.__wake_up.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.66 -0.7 0.00 -0.2 0.49 ± 57% perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.enqueue_task.ttwu_do_activate
0.60 -0.6 0.00 -0.2 0.42 ± 57% perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
0.57 -0.6 0.00 -0.2 0.42 ± 57% perf-profile.calltrace.cycles-pp.set_task_cpu.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend
2.56 -0.3 2.28 -0.6 1.91 ± 57% perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule.schedule_hrtimeout_range_clock
5.87 +0.9 6.78 -1.5 4.41 ± 57% perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.00 +0.9 0.91 ± 30% +0.0 0.00 perf-profile.calltrace.cycles-pp.sched_balance_newidle.pick_next_task_fair.__pick_next_task.__schedule.schedule
35.80 +3.5 39.29 -8.5 27.26 ± 57% perf-profile.calltrace.cycles-pp.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
35.59 +3.6 39.21 -8.5 27.09 ± 57% perf-profile.calltrace.cycles-pp.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +4.4 4.37 ± 3% +0.0 0.00 perf-profile.calltrace.cycles-pp.drain_obj_stock.__refill_obj_stock.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg
0.00 +4.4 4.39 ± 4% +0.0 0.00 perf-profile.calltrace.cycles-pp.drain_obj_stock.__refill_obj_stock.__memcg_slab_free_hook.kfree.free_msg
0.00 +8.0 8.01 ± 4% +0.0 0.00 perf-profile.calltrace.cycles-pp.__refill_obj_stock.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg.do_mq_timedsend
0.00 +8.3 8.29 ± 4% +0.0 0.00 perf-profile.calltrace.cycles-pp.__refill_obj_stock.__memcg_slab_free_hook.kfree.free_msg.do_mq_timedreceive
28.51 +13.5 41.99 -7.2 21.32 ± 57% perf-profile.calltrace.cycles-pp.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
28.23 +13.5 41.71 -7.1 21.11 ± 57% perf-profile.calltrace.cycles-pp.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
70.69 +13.7 84.35 -17.4 53.34 ± 57% perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
70.44 +13.8 84.26 -17.3 53.14 ± 57% perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.99 +20.2 23.24 ± 2% -0.3 2.70 ± 57% perf-profile.calltrace.cycles-pp.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.79 +20.4 23.15 ± 2% -0.2 2.56 ± 57% perf-profile.calltrace.cycles-pp.kfree.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
2.43 +20.6 22.98 ± 2% -0.1 2.28 ± 57% perf-profile.calltrace.cycles-pp.__memcg_slab_free_hook.kfree.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive
2.26 +26.0 28.23 ± 2% -0.2 2.03 ± 57% perf-profile.calltrace.cycles-pp.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.99 +26.8 27.80 ± 2% +0.1 1.09 ± 57% perf-profile.calltrace.cycles-pp.__kmalloc_node_noprof.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
0.65 +27.0 27.62 ± 2% +0.2 0.84 ± 57% perf-profile.calltrace.cycles-pp.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend
24.25 -12.2 12.02 ± 4% -6.2 18.00 ± 57% perf-profile.children.cycles-pp.wake_up_q
24.00 -12.1 11.93 ± 4% -6.2 17.82 ± 57% perf-profile.children.cycles-pp.try_to_wake_up
23.75 -10.9 12.88 -6.3 17.48 ± 57% perf-profile.children.cycles-pp.common_startup_64
23.75 -10.9 12.88 -6.3 17.48 ± 57% perf-profile.children.cycles-pp.cpu_startup_entry
23.68 -10.8 12.85 -6.3 17.43 ± 57% perf-profile.children.cycles-pp.do_idle
23.65 -10.8 12.82 -6.2 17.41 ± 57% perf-profile.children.cycles-pp.start_secondary
19.65 -8.3 11.33 -5.0 14.66 ± 57% perf-profile.children.cycles-pp.__schedule
17.14 -6.9 10.28 -4.3 12.80 ± 57% perf-profile.children.cycles-pp.wq_sleep
16.24 -6.4 9.82 -4.1 12.13 ± 57% perf-profile.children.cycles-pp.schedule
15.92 -6.2 9.69 -4.0 11.88 ± 57% perf-profile.children.cycles-pp.schedule_hrtimeout_range_clock
12.46 -6.1 6.32 ± 4% -3.3 9.20 ± 57% perf-profile.children.cycles-pp.select_task_rq
12.19 -6.0 6.17 ± 4% -3.2 9.00 ± 57% perf-profile.children.cycles-pp.select_task_rq_fair
9.91 -4.8 5.09 ± 4% -2.6 7.33 ± 57% perf-profile.children.cycles-pp.select_idle_sibling
4.57 -4.6 0.02 ±141% -1.3 3.24 ± 57% perf-profile.children.cycles-pp.poll_idle
9.27 -4.5 4.80 ± 4% -2.4 6.85 ± 57% perf-profile.children.cycles-pp.select_idle_cpu
12.49 -4.5 8.03 -3.3 9.15 ± 57% perf-profile.children.cycles-pp.cpuidle_idle_call
11.41 -4.0 7.39 -3.1 8.36 ± 57% perf-profile.children.cycles-pp.cpuidle_enter_state
11.44 -4.0 7.43 -3.1 8.38 ± 57% perf-profile.children.cycles-pp.cpuidle_enter
8.17 -4.0 4.18 ± 4% -2.1 6.04 ± 57% perf-profile.children.cycles-pp.select_idle_core
5.76 -3.5 2.28 ± 4% -1.5 4.25 ± 57% perf-profile.children.cycles-pp.flush_smp_call_function_queue
5.47 -3.1 2.35 ± 4% -1.4 4.04 ± 57% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
4.30 -2.4 1.94 ± 4% -1.1 3.18 ± 57% perf-profile.children.cycles-pp.sched_ttwu_pending
4.08 -2.2 1.90 -1.1 3.02 ± 57% perf-profile.children.cycles-pp.schedule_idle
3.47 -2.2 1.30 ± 2% -0.9 2.57 ± 57% perf-profile.children.cycles-pp.store_msg
5.87 -2.1 3.78 -1.5 4.38 ± 57% perf-profile.children.cycles-pp.__pick_next_task
4.84 -2.0 2.86 -1.2 3.63 ± 57% perf-profile.children.cycles-pp.try_to_block_task
4.73 -1.9 2.79 -1.2 3.54 ± 57% perf-profile.children.cycles-pp.dequeue_entities
4.73 -1.9 2.82 -1.2 3.55 ± 57% perf-profile.children.cycles-pp.dequeue_task_fair
4.04 -1.8 2.26 -1.1 2.98 ± 57% perf-profile.children.cycles-pp._raw_spin_lock
3.72 -1.7 2.03 ± 4% -0.9 2.77 ± 57% perf-profile.children.cycles-pp.enqueue_task
2.40 -1.6 0.85 ± 3% -0.6 1.79 ± 57% perf-profile.children.cycles-pp._copy_to_user
3.39 -1.5 1.86 ± 3% -0.9 2.52 ± 57% perf-profile.children.cycles-pp.ttwu_do_activate
2.56 -1.5 1.04 ± 5% -0.7 1.90 ± 57% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
2.54 -1.5 1.03 ± 5% -0.7 1.88 ± 57% perf-profile.children.cycles-pp.do_perf_trace_sched_wakeup_template
2.76 -1.4 1.34 ± 4% -0.7 2.03 ± 57% perf-profile.children.cycles-pp.arch_exit_to_user_mode_prepare
3.75 -1.4 2.34 -0.9 2.81 ± 57% perf-profile.children.cycles-pp.dequeue_entity
2.32 -1.4 0.95 ± 4% -0.6 1.72 ± 57% perf-profile.children.cycles-pp.ttwu_queue_wakelist
2.72 -1.3 1.38 ± 2% -0.7 2.05 ± 57% perf-profile.children.cycles-pp.update_curr
2.38 -1.3 1.06 ± 2% -0.6 1.79 ± 57% perf-profile.children.cycles-pp.exit_to_user_mode_loop
4.24 -1.2 2.99 -1.1 3.16 ± 57% perf-profile.children.cycles-pp.pick_next_task_fair
2.21 -1.1 1.15 ± 5% -0.6 1.63 ± 57% perf-profile.children.cycles-pp.switch_fpu_return
1.76 -1.0 0.73 ± 3% -0.4 1.37 ± 57% perf-profile.children.cycles-pp.msg_get
1.67 -1.0 0.65 ± 3% -0.6 1.10 ± 57% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
2.47 -1.0 1.47 ± 3% -0.6 1.84 ± 57% perf-profile.children.cycles-pp.enqueue_task_fair
2.39 -1.0 1.42 -0.6 1.78 ± 57% perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
2.29 -1.0 1.32 -0.6 1.69 ± 57% perf-profile.children.cycles-pp.update_load_avg
1.60 -1.0 0.64 -0.4 1.20 ± 57% perf-profile.children.cycles-pp.switch_mm_irqs_off
1.51 -0.9 0.57 ± 5% -0.4 1.12 ± 57% perf-profile.children.cycles-pp.__smp_call_single_queue
1.84 -0.9 0.93 ± 6% -0.5 1.35 ± 57% perf-profile.children.cycles-pp.wake_affine
1.49 -0.9 0.59 -0.4 1.10 ± 57% perf-profile.children.cycles-pp.__check_object_size
1.61 -0.9 0.72 ± 5% -0.4 1.21 ± 57% perf-profile.children.cycles-pp.update_rq_clock_task
1.73 -0.9 0.87 ± 2% -0.4 1.30 ± 57% perf-profile.children.cycles-pp.update_se
1.51 -0.9 0.65 ± 2% -0.4 1.14 ± 57% perf-profile.children.cycles-pp.wakeup_preempt
2.00 -0.8 1.15 ± 3% -0.5 1.48 ± 57% perf-profile.children.cycles-pp.enqueue_entity
1.26 -0.8 0.42 ± 3% -0.5 0.80 ± 57% perf-profile.children.cycles-pp.__wake_up
1.31 -0.7 0.58 ± 5% -0.3 0.97 ± 57% perf-profile.children.cycles-pp.set_task_cpu
1.15 -0.7 0.45 ± 3% -0.3 0.85 ± 57% perf-profile.children.cycles-pp.msg_insert
1.04 -0.7 0.35 ± 5% -0.3 0.78 ± 57% perf-profile.children.cycles-pp.perf_trace_buf_alloc
1.57 -0.7 0.90 ± 5% -0.4 1.17 ± 57% perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
1.00 -0.7 0.34 ± 7% -0.3 0.74 ± 57% perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
0.95 -0.7 0.29 ± 4% -0.3 0.70 ± 57% perf-profile.children.cycles-pp.llist_reverse_order
1.14 -0.6 0.50 ± 5% -0.3 0.84 ± 57% perf-profile.children.cycles-pp.migrate_task_rq_fair
1.11 -0.6 0.51 -0.3 0.82 ± 57% perf-profile.children.cycles-pp.set_next_entity
0.95 -0.6 0.39 ± 2% -0.2 0.71 ± 57% perf-profile.children.cycles-pp.__update_idle_core
1.04 -0.6 0.49 ± 2% -0.3 0.77 ± 57% perf-profile.children.cycles-pp.pick_task_fair
1.15 -0.6 0.60 ± 7% -0.3 0.84 ± 57% perf-profile.children.cycles-pp.task_h_load
1.06 -0.5 0.52 -0.3 0.79 ± 57% perf-profile.children.cycles-pp.set_next_task_idle
0.84 -0.5 0.30 ± 5% -0.2 0.63 ± 57% perf-profile.children.cycles-pp.call_function_single_prep_ipi
1.04 -0.5 0.51 -0.3 0.76 ± 57% perf-profile.children.cycles-pp._find_next_bit
1.38 -0.5 0.85 -0.4 1.02 ± 57% perf-profile.children.cycles-pp.update_cfs_rq_load_avg
1.28 -0.5 0.75 -0.3 0.95 ± 57% perf-profile.children.cycles-pp.__switch_to
0.85 -0.5 0.34 ± 3% -0.2 0.66 ± 57% perf-profile.children.cycles-pp.cpuacct_charge
0.77 ± 4% -0.5 0.25 -0.2 0.55 ± 57% perf-profile.children.cycles-pp.__bitmap_andnot
0.88 -0.5 0.41 ± 6% -0.2 0.66 ± 57% perf-profile.children.cycles-pp.update_entity_lag
0.94 -0.5 0.48 -0.2 0.69 ± 57% perf-profile.children.cycles-pp.prepare_task_switch
0.74 -0.4 0.31 ± 2% -0.2 0.56 ± 57% perf-profile.children.cycles-pp.check_heap_object
0.75 -0.4 0.32 ± 5% -0.2 0.56 ± 57% perf-profile.children.cycles-pp.requeue_delayed_entity
0.80 -0.4 0.40 -0.2 0.60 ± 57% perf-profile.children.cycles-pp.wakeup_preempt_fair
0.55 -0.4 0.16 ± 2% -0.1 0.40 ± 57% perf-profile.children.cycles-pp.native_sched_clock
0.57 -0.4 0.18 ± 4% -0.2 0.41 ± 57% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
0.55 -0.4 0.17 -0.1 0.41 ± 57% perf-profile.children.cycles-pp.os_xsave
0.58 -0.4 0.20 -0.2 0.43 ± 57% perf-profile.children.cycles-pp.sched_clock_cpu
1.39 -0.4 1.01 -0.4 1.03 ± 57% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
1.18 -0.4 0.81 -0.3 0.88 ± 57% perf-profile.children.cycles-pp.___task_rq_lock
0.51 ± 3% -0.4 0.14 ± 3% -0.2 0.36 ± 58% perf-profile.children.cycles-pp._copy_from_user
0.56 -0.4 0.19 ± 2% -0.1 0.41 ± 57% perf-profile.children.cycles-pp.update_rq_clock
0.51 -0.3 0.17 ± 2% -0.1 0.38 ± 57% perf-profile.children.cycles-pp.sched_clock
0.56 -0.3 0.23 ± 3% -0.1 0.44 ± 57% perf-profile.children.cycles-pp.simple_inode_init_ts
0.89 -0.3 0.56 -0.2 0.67 ± 57% perf-profile.children.cycles-pp.put_prev_entity
0.53 ± 3% -0.3 0.21 ± 2% -0.2 0.38 ± 57% perf-profile.children.cycles-pp.__put_user_4
0.68 ± 12% -0.3 0.36 ± 4% -0.1 0.57 ± 57% perf-profile.children.cycles-pp.stress_switch_mq
0.52 -0.3 0.20 -0.1 0.37 ± 57% perf-profile.children.cycles-pp.set_next_task_fair
0.55 -0.3 0.24 ± 5% -0.1 0.40 ± 57% perf-profile.children.cycles-pp.__switch_to_asm
0.51 -0.3 0.21 ± 3% -0.1 0.40 ± 57% perf-profile.children.cycles-pp.inode_set_ctime_current
0.56 -0.3 0.26 ± 3% -0.1 0.42 ± 57% perf-profile.children.cycles-pp.fdget
0.52 -0.3 0.23 ± 3% -0.1 0.39 ± 57% perf-profile.children.cycles-pp.mm_cid_switch_to
0.54 -0.3 0.25 ± 3% -0.1 0.40 ± 57% perf-profile.children.cycles-pp.remove_entity_load_avg
0.79 -0.3 0.51 -0.2 0.59 ± 57% perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.37 -0.3 0.11 ± 7% -0.1 0.27 ± 57% perf-profile.children.cycles-pp.__resched_curr
0.57 -0.2 0.32 -0.1 0.42 ± 57% perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.35 -0.2 0.12 ± 4% -0.1 0.26 ± 57% perf-profile.children.cycles-pp.avg_vruntime
0.62 -0.2 0.39 ± 2% -0.2 0.47 ± 57% perf-profile.children.cycles-pp.sysvec_call_function_single
0.34 -0.2 0.11 -0.1 0.25 ± 57% perf-profile.children.cycles-pp.entry_SYSCALL_64
0.58 -0.2 0.36 -0.1 0.44 ± 57% perf-profile.children.cycles-pp.__enqueue_entity
0.52 -0.2 0.30 ± 3% -0.1 0.38 ± 57% perf-profile.children.cycles-pp.perf_tp_event
0.46 -0.2 0.24 -0.1 0.34 ± 57% perf-profile.children.cycles-pp.__pick_eevdf
0.35 -0.2 0.14 ± 3% -0.1 0.26 ± 57% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.56 -0.2 0.36 -0.1 0.42 ± 57% perf-profile.children.cycles-pp.__sysvec_call_function_single
0.33 -0.2 0.13 ± 3% -0.1 0.24 ± 57% perf-profile.children.cycles-pp.__update_load_avg_se
0.31 ± 2% -0.2 0.12 ± 4% -0.1 0.24 ± 57% perf-profile.children.cycles-pp.__virt_addr_valid
0.58 -0.2 0.40 -0.1 0.44 ± 57% perf-profile.children.cycles-pp.menu_select
0.32 ± 5% -0.2 0.13 -0.1 0.22 ± 57% perf-profile.children.cycles-pp.__check_heap_object
0.31 -0.2 0.13 ± 3% -0.1 0.23 ± 57% perf-profile.children.cycles-pp.place_entity
0.32 -0.2 0.14 -0.1 0.24 ± 57% perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.36 -0.2 0.18 ± 2% -0.1 0.27 ± 57% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.29 -0.2 0.11 -0.1 0.22 ± 59% perf-profile.children.cycles-pp.syscall_return_via_sysret
0.33 -0.2 0.16 ± 3% -0.1 0.24 ± 57% perf-profile.children.cycles-pp.do_perf_trace_sched_stat_runtime
0.35 -0.2 0.18 ± 2% -0.1 0.26 ± 57% perf-profile.children.cycles-pp.ktime_get
0.25 -0.2 0.08 ± 5% -0.1 0.18 ± 57% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.37 -0.2 0.20 ± 4% -0.1 0.27 ± 57% perf-profile.children.cycles-pp.___perf_sw_event
0.23 -0.2 0.07 ± 6% -0.1 0.17 ± 57% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.22 -0.2 0.07 -0.1 0.16 ± 57% perf-profile.children.cycles-pp.read_tsc
0.21 -0.1 0.06 ± 7% -0.1 0.16 ± 57% perf-profile.children.cycles-pp.__rdgsbase_inactive
0.25 -0.1 0.11 ± 7% -0.1 0.18 ± 57% perf-profile.children.cycles-pp.strnlen
0.16 -0.1 0.02 ±141% -0.0 0.11 ± 57% perf-profile.children.cycles-pp.ct_idle_exit
0.32 -0.1 0.18 -0.1 0.24 ± 57% perf-profile.children.cycles-pp.__dequeue_entity
0.23 ± 2% -0.1 0.10 ± 4% -0.1 0.18 ± 57% perf-profile.children.cycles-pp.check_stack_object
0.53 -0.1 0.40 ± 2% -0.1 0.40 ± 57% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.23 ± 2% -0.1 0.09 ± 5% -0.1 0.17 ± 57% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.30 ± 3% -0.1 0.17 ± 4% -0.1 0.23 ± 57% perf-profile.children.cycles-pp.tick_nohz_handler
0.20 -0.1 0.07 +0.3 0.46 ± 57% perf-profile.children.cycles-pp.__account_obj_stock
0.13 -0.1 0.00 -0.0 0.09 ± 57% perf-profile.children.cycles-pp.ct_kernel_exit_state
0.41 -0.1 0.28 ± 2% -0.1 0.30 ± 57% perf-profile.children.cycles-pp.hrtimer_interrupt
0.41 ± 2% -0.1 0.29 ± 3% -0.1 0.31 ± 57% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.31 ± 2% -0.1 0.18 ± 2% -0.1 0.24 ± 57% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.28 ± 2% -0.1 0.16 ± 6% -0.1 0.21 ± 57% perf-profile.children.cycles-pp.update_process_times
0.33 -0.1 0.21 ± 6% -0.1 0.25 ± 57% perf-profile.children.cycles-pp.attach_entity_load_avg
0.12 ± 3% -0.1 0.00 -0.1 0.05 ± 58% perf-profile.children.cycles-pp.__do_notify
0.19 ± 2% -0.1 0.07 ± 7% -0.0 0.14 ± 57% perf-profile.children.cycles-pp.wake_q_add_safe
0.23 ± 2% -0.1 0.11 -0.1 0.17 ± 57% perf-profile.children.cycles-pp.__kmalloc_cache_noprof
0.18 -0.1 0.07 ± 6% -0.1 0.13 ± 57% perf-profile.children.cycles-pp.nohz_run_idle_balance
0.56 -0.1 0.46 -0.1 0.42 ± 57% perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.15 -0.1 0.05 -0.0 0.11 ± 57% perf-profile.children.cycles-pp._raw_spin_unlock
0.17 -0.1 0.08 -0.1 0.12 ± 57% perf-profile.children.cycles-pp.security_msg_msg_free
0.15 -0.1 0.06 -0.0 0.12 ± 57% perf-profile.children.cycles-pp.inode_set_ctime_to_ts
0.16 ± 2% -0.1 0.08 ± 5% -0.0 0.12 ± 57% perf-profile.children.cycles-pp.dl_server_update
0.13 -0.1 0.06 ± 8% -0.0 0.10 ± 57% perf-profile.children.cycles-pp.timestamp_truncate
0.15 ± 3% -0.1 0.08 -0.0 0.12 ± 57% perf-profile.children.cycles-pp.perf_trace_buf_update
0.07 -0.1 0.00 -0.0 0.04 ± 57% perf-profile.children.cycles-pp.kick_ilb
0.13 ± 3% -0.1 0.06 -0.0 0.09 ± 57% perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64_mg
0.13 ± 3% -0.1 0.06 -0.0 0.10 ± 57% perf-profile.children.cycles-pp.migrate_disable_switch
0.12 ± 6% -0.1 0.05 ± 8% -0.0 0.08 ± 58% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.11 ± 4% -0.1 0.05 -0.0 0.08 ± 57% perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.14 -0.1 0.08 ± 5% -0.0 0.10 ± 57% perf-profile.children.cycles-pp.update_curr_dl_se
0.10 -0.1 0.05 -0.0 0.08 ± 57% perf-profile.children.cycles-pp.ct_kernel_exit
0.11 -0.1 0.06 -0.0 0.08 ± 57% perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
0.10 ± 4% -0.0 0.05 -0.0 0.08 ± 57% perf-profile.children.cycles-pp.__rb_insert_augmented
0.10 -0.0 0.06 ± 8% -0.0 0.08 ± 57% perf-profile.children.cycles-pp.rest_init
0.10 -0.0 0.06 ± 8% -0.0 0.08 ± 57% perf-profile.children.cycles-pp.start_kernel
0.10 -0.0 0.06 ± 8% -0.0 0.08 ± 57% perf-profile.children.cycles-pp.x86_64_start_kernel
0.10 -0.0 0.06 ± 8% -0.0 0.08 ± 57% perf-profile.children.cycles-pp.x86_64_start_reservations
0.08 ± 10% -0.0 0.04 ± 71% -0.0 0.06 ± 59% perf-profile.children.cycles-pp.mq_timedreceive
0.15 -0.0 0.11 ± 4% -0.0 0.11 ± 57% perf-profile.children.cycles-pp.vruntime_eligible
0.13 -0.0 0.09 -0.0 0.10 ± 57% perf-profile.children.cycles-pp.put_prev_task_fair
0.09 -0.0 0.05 ± 8% -0.0 0.07 ± 57% perf-profile.children.cycles-pp.native_irq_return_iret
0.08 -0.0 0.05 -0.0 0.06 ± 57% perf-profile.children.cycles-pp.choose_new_asid
0.13 -0.0 0.11 -0.0 0.10 ± 57% perf-profile.children.cycles-pp.__irq_exit_rcu
0.07 -0.0 0.05 -0.0 0.05 ± 57% perf-profile.children.cycles-pp.__set_next_task_fair
0.76 -0.0 0.74 -0.2 0.56 ± 57% perf-profile.children.cycles-pp.finish_task_switch
0.09 -0.0 0.07 ± 6% -0.0 0.07 ± 57% perf-profile.children.cycles-pp.propagate_entity_load_avg
0.10 -0.0 0.09 -0.0 0.08 ± 57% perf-profile.children.cycles-pp.handle_softirqs
0.07 -0.0 0.06 -0.0 0.05 ± 57% perf-profile.children.cycles-pp.clockevents_program_event
0.07 +0.0 0.08 -0.0 0.05 ± 57% perf-profile.children.cycles-pp.perf_swevent_event
0.48 +0.0 0.49 -0.1 0.36 ± 57% perf-profile.children.cycles-pp.process_simple
0.05 +0.0 0.06 ± 7% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.sched_update_worker
0.05 +0.0 0.07 -0.0 0.04 ± 57% perf-profile.children.cycles-pp.arch_cpu_idle_enter
0.07 ± 11% +0.0 0.09 ± 5% +0.0 0.07 ± 58% perf-profile.children.cycles-pp.mq_timedsend
0.15 ± 3% +0.0 0.20 ± 4% -0.0 0.12 ± 57% perf-profile.children.cycles-pp.x64_sys_call
0.00 +0.1 0.05 +0.0 0.00 perf-profile.children.cycles-pp.__sched_balance_update_blocked_averages
0.00 +0.1 0.05 +0.0 0.00 perf-profile.children.cycles-pp.update_cfs_group
0.00 +0.1 0.06 ± 23% +0.0 0.00 perf-profile.children.cycles-pp.generic_perform_write
0.00 +0.1 0.06 ± 7% +0.0 0.00 perf-profile.children.cycles-pp.detach_tasks
0.00 +0.1 0.06 ± 29% +0.0 0.00 perf-profile.children.cycles-pp.shmem_file_write_iter
0.00 +0.1 0.06 ± 29% +0.0 0.00 perf-profile.children.cycles-pp.vfs_write
0.00 +0.1 0.07 ± 25% +0.0 0.00 perf-profile.children.cycles-pp.ksys_write
0.00 +0.1 0.08 ± 30% +0.0 0.00 perf-profile.children.cycles-pp.record__pushfn
0.04 ± 71% +0.1 0.12 ± 35% -0.0 0.01 ±173% perf-profile.children.cycles-pp.perf_mmap__push
0.54 ± 2% +0.1 0.62 ± 7% -0.1 0.40 ± 57% perf-profile.children.cycles-pp.cmd_record
0.04 ± 71% +0.1 0.13 ± 35% -0.0 0.04 ± 58% perf-profile.children.cycles-pp.handle_internal_command
0.04 ± 71% +0.1 0.13 ± 35% -0.0 0.04 ± 58% perf-profile.children.cycles-pp.main
0.04 ± 71% +0.1 0.13 ± 35% -0.0 0.04 ± 58% perf-profile.children.cycles-pp.run_builtin
0.04 ± 70% +0.1 0.13 ± 32% -0.0 0.04 ± 57% perf-profile.children.cycles-pp.record__mmap_read_evlist
0.10 ± 4% +0.1 0.20 ± 4% -0.0 0.08 ± 57% perf-profile.children.cycles-pp.do_perf_trace_sched_switch
0.00 +0.1 0.10 ± 4% +0.0 0.00 perf-profile.children.cycles-pp.ct_idle_enter
0.13 ± 3% +0.2 0.29 ± 4% -0.0 0.10 ± 57% perf-profile.children.cycles-pp.perf_trace_sched_switch
0.21 ± 6% +0.6 0.78 ± 3% -0.0 0.16 ± 57% perf-profile.children.cycles-pp.update_sg_lb_stats
0.22 ± 6% +0.6 0.81 ± 4% -0.1 0.17 ± 57% perf-profile.children.cycles-pp.update_sd_lb_stats
0.22 ± 6% +0.6 0.81 ± 3% -0.1 0.17 ± 57% perf-profile.children.cycles-pp.sched_balance_find_src_group
0.40 ± 3% +0.7 1.08 ± 4% -0.1 0.30 ± 57% perf-profile.children.cycles-pp.sched_balance_newidle
0.27 ± 6% +0.7 0.97 ± 4% -0.1 0.21 ± 57% perf-profile.children.cycles-pp.sched_balance_rq
5.90 +0.9 6.81 -1.5 4.43 ± 57% perf-profile.children.cycles-pp.intel_idle
35.82 +3.5 39.30 -8.6 27.27 ± 57% perf-profile.children.cycles-pp.__x64_sys_mq_timedreceive
35.70 +3.5 39.25 -8.5 27.18 ± 57% perf-profile.children.cycles-pp.do_mq_timedreceive
0.00 +8.8 8.78 ± 3% +0.0 0.00 perf-profile.children.cycles-pp.drain_obj_stock
28.34 +13.4 41.76 -7.1 21.20 ± 57% perf-profile.children.cycles-pp.do_mq_timedsend
28.52 +13.5 41.99 -7.2 21.34 ± 57% perf-profile.children.cycles-pp.__x64_sys_mq_timedsend
70.76 +13.7 84.45 -17.4 53.40 ± 57% perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
70.56 +13.8 84.39 -17.3 53.24 ± 57% perf-profile.children.cycles-pp.do_syscall_64
0.08 +16.2 16.31 ± 4% +0.1 0.15 ± 57% perf-profile.children.cycles-pp.__refill_obj_stock
3.22 +20.1 23.33 ± 2% -0.3 2.91 ± 57% perf-profile.children.cycles-pp.kfree
3.01 +20.2 23.25 ± 2% -0.3 2.72 ± 57% perf-profile.children.cycles-pp.free_msg
2.46 +20.5 23.00 ± 2% -0.1 2.31 ± 57% perf-profile.children.cycles-pp.__memcg_slab_free_hook
2.31 +25.9 28.25 ± 2% -0.2 2.07 ± 57% perf-profile.children.cycles-pp.load_msg
1.00 +26.8 27.81 ± 2% +0.1 1.10 ± 57% perf-profile.children.cycles-pp.__kmalloc_node_noprof
0.68 +27.0 27.65 ± 2% +0.2 0.86 ± 57% perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
4.41 -4.4 0.02 ±141% -1.3 3.13 ± 57% perf-profile.self.cycles-pp.poll_idle
6.79 -3.2 3.63 ± 5% -1.8 5.04 ± 57% perf-profile.self.cycles-pp.select_idle_core
3.07 -1.7 1.33 ± 4% -0.7 2.36 ± 57% perf-profile.self.cycles-pp.do_mq_timedreceive
2.90 -1.6 1.31 ± 3% -0.7 2.16 ± 57% perf-profile.self.cycles-pp.__schedule
2.37 -1.5 0.84 ± 3% -0.6 1.77 ± 57% perf-profile.self.cycles-pp._copy_to_user
2.73 -1.5 1.22 ± 3% -0.7 2.00 ± 57% perf-profile.self.cycles-pp.do_mq_timedsend
2.63 -1.4 1.25 ± 3% -0.7 1.93 ± 57% perf-profile.self.cycles-pp._raw_spin_lock
1.64 -1.0 0.64 ± 3% -0.6 1.08 ± 57% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
1.46 -0.9 0.55 ± 2% -0.4 1.10 ± 57% perf-profile.self.cycles-pp.switch_mm_irqs_off
1.54 -0.9 0.67 ± 6% -0.4 1.16 ± 57% perf-profile.self.cycles-pp.update_rq_clock_task
1.48 -0.9 0.62 ± 3% -0.4 1.13 ± 57% perf-profile.self.cycles-pp.msg_get
1.36 -0.8 0.58 ± 3% -0.3 1.02 ± 57% perf-profile.self.cycles-pp.exit_to_user_mode_loop
1.11 -0.7 0.44 ± 4% -0.3 0.82 ± 57% perf-profile.self.cycles-pp.msg_insert
1.56 -0.7 0.90 ± 6% -0.4 1.17 ± 57% perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
0.95 -0.7 0.29 ± 5% -0.3 0.70 ± 57% perf-profile.self.cycles-pp.llist_reverse_order
1.00 -0.7 0.34 ± 7% -0.3 0.74 ± 57% perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
1.19 -0.6 0.59 ± 3% -0.3 0.90 ± 57% perf-profile.self.cycles-pp.wq_sleep
0.92 -0.6 0.36 ± 6% -0.2 0.69 ± 57% perf-profile.self.cycles-pp.do_perf_trace_sched_wakeup_template
0.90 -0.6 0.34 ± 2% -0.2 0.68 ± 57% perf-profile.self.cycles-pp.__update_idle_core
1.15 -0.5 0.60 ± 7% -0.3 0.84 ± 57% perf-profile.self.cycles-pp.task_h_load
0.83 -0.5 0.30 ± 5% -0.2 0.62 ± 57% perf-profile.self.cycles-pp.call_function_single_prep_ipi
0.97 -0.5 0.44 ± 2% -0.2 0.73 ± 57% perf-profile.self.cycles-pp.dequeue_entities
0.84 -0.5 0.34 ± 3% -0.2 0.65 ± 57% perf-profile.self.cycles-pp.cpuacct_charge
0.96 -0.5 0.47 ± 4% -0.2 0.72 ± 57% perf-profile.self.cycles-pp.do_syscall_64
1.23 -0.5 0.74 -0.3 0.91 ± 57% perf-profile.self.cycles-pp.__switch_to
0.73 ± 4% -0.5 0.24 ± 3% -0.2 0.52 ± 57% perf-profile.self.cycles-pp.__bitmap_andnot
0.69 -0.5 0.20 ± 4% -0.2 0.51 ± 57% perf-profile.self.cycles-pp.flush_smp_call_function_queue
0.94 -0.5 0.47 -0.2 0.69 ± 57% perf-profile.self.cycles-pp._find_next_bit
0.77 -0.4 0.36 ± 6% -0.2 0.58 ± 57% perf-profile.self.cycles-pp.update_entity_lag
0.76 -0.4 0.36 ± 3% -0.2 0.56 ± 57% perf-profile.self.cycles-pp.update_load_avg
0.67 -0.4 0.27 ± 5% -0.2 0.49 ± 57% perf-profile.self.cycles-pp.__smp_call_single_queue
0.70 -0.4 0.31 ± 2% -0.2 0.52 ± 57% perf-profile.self.cycles-pp.pick_next_task_fair
0.55 -0.4 0.17 ± 2% -0.1 0.41 ± 57% perf-profile.self.cycles-pp.os_xsave
0.56 -0.4 0.18 ± 4% -0.1 0.41 ± 57% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
0.63 -0.4 0.25 -0.2 0.46 ± 57% perf-profile.self.cycles-pp.switch_fpu_return
1.38 -0.4 1.01 -0.4 1.02 ± 57% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.69 -0.4 0.33 ± 5% -0.2 0.51 ± 57% perf-profile.self.cycles-pp.wake_affine
0.53 -0.4 0.16 -0.1 0.38 ± 57% perf-profile.self.cycles-pp.native_sched_clock
0.67 -0.4 0.30 ± 2% -0.1 0.52 ± 57% perf-profile.self.cycles-pp.kfree
0.75 -0.4 0.39 ± 3% -0.2 0.57 ± 57% perf-profile.self.cycles-pp.update_curr
0.67 -0.4 0.31 ± 5% -0.2 0.49 ± 57% perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.53 -0.4 0.18 ± 2% -0.1 0.38 ± 57% perf-profile.self.cycles-pp.arch_exit_to_user_mode_prepare
0.49 ± 2% -0.4 0.14 ± 3% -0.1 0.35 ± 58% perf-profile.self.cycles-pp._copy_from_user
0.63 -0.3 0.28 ± 3% -0.2 0.47 ± 57% perf-profile.self.cycles-pp.schedule_hrtimeout_range_clock
0.58 -0.3 0.23 ± 4% -0.1 0.43 ± 57% perf-profile.self.cycles-pp.select_idle_sibling
0.58 -0.3 0.24 ± 7% -0.2 0.42 ± 57% perf-profile.self.cycles-pp.migrate_task_rq_fair
0.64 -0.3 0.31 -0.2 0.47 ± 57% perf-profile.self.cycles-pp.prepare_task_switch
0.82 -0.3 0.49 ± 3% -0.2 0.62 ± 57% perf-profile.self.cycles-pp.select_idle_cpu
0.52 ± 3% -0.3 0.21 ± 2% -0.1 0.37 ± 57% perf-profile.self.cycles-pp.__put_user_4
0.63 ± 11% -0.3 0.32 ± 5% -0.1 0.52 ± 57% perf-profile.self.cycles-pp.stress_switch_mq
0.54 -0.3 0.24 ± 5% -0.1 0.40 ± 57% perf-profile.self.cycles-pp.__switch_to_asm
0.54 -0.3 0.25 ± 3% -0.1 0.40 ± 57% perf-profile.self.cycles-pp.fdget
0.51 -0.3 0.22 ± 4% -0.1 0.38 ± 57% perf-profile.self.cycles-pp.mm_cid_switch_to
0.44 -0.3 0.15 ± 6% -0.1 0.33 ± 57% perf-profile.self.cycles-pp.select_task_rq_fair
0.43 -0.3 0.15 ± 5% -0.1 0.32 ± 57% perf-profile.self.cycles-pp.sched_ttwu_pending
0.49 -0.3 0.23 ± 6% -0.1 0.37 ± 57% perf-profile.self.cycles-pp.enqueue_task
0.59 -0.2 0.34 ± 2% -0.1 0.44 ± 57% perf-profile.self.cycles-pp.try_to_wake_up
0.73 -0.2 0.48 -0.2 0.54 ± 57% perf-profile.self.cycles-pp.update_cfs_rq_load_avg
0.35 -0.2 0.11 ± 4% -0.1 0.26 ± 57% perf-profile.self.cycles-pp.__resched_curr
0.40 -0.2 0.16 ± 2% -0.1 0.30 ± 57% perf-profile.self.cycles-pp.__pick_next_task
0.36 -0.2 0.12 ± 3% -0.1 0.26 ± 57% perf-profile.self.cycles-pp.cpuidle_idle_call
0.34 -0.2 0.11 -0.1 0.25 ± 57% perf-profile.self.cycles-pp.entry_SYSCALL_64
0.57 -0.2 0.36 -0.1 0.43 ± 57% perf-profile.self.cycles-pp.__enqueue_entity
0.51 -0.2 0.31 -0.1 0.38 ± 57% perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.34 -0.2 0.14 ± 5% -0.1 0.26 ± 57% perf-profile.self.cycles-pp.wakeup_preempt
0.34 -0.2 0.14 ± 3% -0.1 0.25 ± 57% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.39 -0.2 0.20 ± 2% -0.1 0.29 ± 57% perf-profile.self.cycles-pp.do_idle
0.31 ± 5% -0.2 0.11 -0.1 0.21 ± 58% perf-profile.self.cycles-pp.__check_heap_object
0.37 -0.2 0.17 ± 2% -0.1 0.27 ± 57% perf-profile.self.cycles-pp.check_heap_object
0.51 -0.2 0.32 -0.1 0.38 ± 57% perf-profile.self.cycles-pp.schedule
0.36 -0.2 0.18 ± 2% -0.1 0.27 ± 57% perf-profile.self.cycles-pp.__pick_eevdf
0.61 -0.2 0.43 -0.2 0.46 ± 57% perf-profile.self.cycles-pp.dequeue_entity
0.29 ± 2% -0.2 0.11 ± 4% -0.1 0.22 ± 57% perf-profile.self.cycles-pp.__virt_addr_valid
0.30 -0.2 0.12 -0.1 0.22 ± 57% perf-profile.self.cycles-pp.__update_load_avg_se
0.27 -0.2 0.10 ± 4% -0.1 0.20 ± 57% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.25 -0.2 0.08 ± 5% -0.1 0.18 ± 57% perf-profile.self.cycles-pp.__check_object_size
0.26 -0.2 0.11 ± 4% -0.1 0.19 ± 57% perf-profile.self.cycles-pp.place_entity
0.45 -0.2 0.30 ± 3% -0.1 0.34 ± 57% perf-profile.self.cycles-pp.enqueue_entity
0.44 -0.2 0.29 ± 5% -0.1 0.33 ± 57% perf-profile.self.cycles-pp.enqueue_task_fair
0.24 -0.2 0.09 -0.1 0.18 ± 57% perf-profile.self.cycles-pp.wake_up_q
0.24 -0.1 0.09 ± 5% -0.1 0.18 ± 57% perf-profile.self.cycles-pp.___perf_sw_event
0.22 ± 2% -0.1 0.07 -0.0 0.17 ± 59% perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.32 -0.1 0.18 ± 2% -0.1 0.24 ± 57% perf-profile.self.cycles-pp.dequeue_task_fair
0.21 ± 2% -0.1 0.06 ± 7% -0.1 0.15 ± 57% perf-profile.self.cycles-pp.read_tsc
0.20 -0.1 0.06 -0.1 0.15 ± 57% perf-profile.self.cycles-pp.__rdgsbase_inactive
0.36 -0.1 0.22 ± 4% -0.1 0.26 ± 57% perf-profile.self.cycles-pp.perf_tp_event
0.28 -0.1 0.14 -0.1 0.21 ± 57% perf-profile.self.cycles-pp.__kmalloc_node_noprof
0.24 -0.1 0.11 ± 4% -0.1 0.18 ± 57% perf-profile.self.cycles-pp.strnlen
0.21 -0.1 0.08 ± 6% -0.0 0.16 ± 57% perf-profile.self.cycles-pp.load_msg
0.33 -0.1 0.20 ± 4% -0.1 0.24 ± 57% perf-profile.self.cycles-pp.attach_entity_load_avg
0.12 ± 3% -0.1 0.00 -0.1 0.05 ± 58% perf-profile.self.cycles-pp.__do_notify
0.44 -0.1 0.33 -0.1 0.33 ± 57% perf-profile.self.cycles-pp.menu_select
0.41 -0.1 0.29 -0.1 0.30 ± 57% perf-profile.self.cycles-pp.update_se
0.27 -0.1 0.15 ± 6% -0.1 0.20 ± 57% perf-profile.self.cycles-pp.select_task_rq
0.17 ± 2% -0.1 0.06 ± 8% -0.1 0.12 ± 57% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.23 ± 2% -0.1 0.11 ± 4% -0.1 0.16 ± 57% perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.19 -0.1 0.08 ± 6% -0.1 0.14 ± 57% perf-profile.self.cycles-pp.update_rq_clock
0.20 ± 2% -0.1 0.09 -0.0 0.16 ± 57% perf-profile.self.cycles-pp.check_stack_object
0.18 ± 2% -0.1 0.06 ± 7% -0.0 0.14 ± 57% perf-profile.self.cycles-pp.wake_q_add_safe
0.18 -0.1 0.07 +0.2 0.38 ± 57% perf-profile.self.cycles-pp.__account_obj_stock
0.21 ± 2% -0.1 0.10 ± 4% -0.0 0.16 ± 57% perf-profile.self.cycles-pp.__kmalloc_cache_noprof
0.24 -0.1 0.14 -0.1 0.18 ± 57% perf-profile.self.cycles-pp.__dequeue_entity
0.19 -0.1 0.10 ± 4% -0.0 0.14 ± 57% perf-profile.self.cycles-pp.pick_task_fair
0.14 ± 3% -0.1 0.05 -0.0 0.11 ± 57% perf-profile.self.cycles-pp.inode_set_ctime_current
0.16 ± 3% -0.1 0.06 ± 7% -0.0 0.11 ± 57% perf-profile.self.cycles-pp.nohz_run_idle_balance
0.16 -0.1 0.07 -0.0 0.12 ± 57% perf-profile.self.cycles-pp.schedule_idle
0.15 -0.1 0.06 -0.0 0.11 ± 57% perf-profile.self.cycles-pp.avg_vruntime
0.08 -0.1 0.00 -0.0 0.05 ± 57% perf-profile.self.cycles-pp.security_msg_msg_free
0.07 -0.1 0.00 -0.0 0.04 ± 57% perf-profile.self.cycles-pp.ct_kernel_enter
0.15 -0.1 0.08 ± 5% -0.0 0.11 ± 57% perf-profile.self.cycles-pp.dl_server_update
0.15 -0.1 0.08 ± 5% -0.0 0.11 ± 57% perf-profile.self.cycles-pp.raw_spin_rq_lock_nested
0.12 -0.1 0.05 ± 8% -0.0 0.09 ± 57% perf-profile.self.cycles-pp.migrate_disable_switch
0.12 ± 4% -0.1 0.05 -0.0 0.09 ± 57% perf-profile.self.cycles-pp.__x64_sys_mq_timedreceive
0.13 -0.1 0.07 ± 7% -0.0 0.09 ± 57% perf-profile.self.cycles-pp.___task_rq_lock
0.09 ± 5% -0.1 0.03 ± 70% -0.0 0.08 ± 57% perf-profile.self.cycles-pp.inode_set_ctime_to_ts
0.22 -0.1 0.16 -0.1 0.16 ± 57% perf-profile.self.cycles-pp.cpuidle_enter_state
0.12 -0.1 0.06 -0.0 0.08 ± 57% perf-profile.self.cycles-pp.store_msg
0.12 -0.1 0.06 -0.0 0.08 ± 57% perf-profile.self.cycles-pp.wakeup_preempt_fair
0.11 ± 4% -0.1 0.05 ± 8% -0.0 0.08 ± 57% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64_mg
0.11 -0.1 0.05 -0.0 0.09 ± 57% perf-profile.self.cycles-pp.timestamp_truncate
0.12 ± 4% -0.1 0.06 ± 7% -0.0 0.09 ± 57% perf-profile.self.cycles-pp.do_perf_trace_sched_stat_runtime
0.11 ± 4% -0.1 0.06 ± 8% -0.0 0.08 ± 57% perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
0.13 ± 3% -0.0 0.08 -0.0 0.10 ± 57% perf-profile.self.cycles-pp.update_curr_dl_se
0.15 -0.0 0.11 ± 4% -0.0 0.11 ± 57% perf-profile.self.cycles-pp.sched_balance_newidle
0.09 -0.0 0.05 ± 8% -0.0 0.07 ± 57% perf-profile.self.cycles-pp.native_irq_return_iret
0.14 ± 3% -0.0 0.10 ± 4% -0.0 0.10 ± 57% perf-profile.self.cycles-pp.vruntime_eligible
0.14 ± 3% -0.0 0.11 -0.0 0.10 ± 57% perf-profile.self.cycles-pp.ktime_get
0.07 ± 7% -0.0 0.03 ± 70% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.__set_next_task_fair
0.15 ± 3% -0.0 0.13 -0.0 0.12 ± 57% perf-profile.self.cycles-pp.put_prev_entity
0.11 ± 4% +0.0 0.12 ± 3% -0.0 0.08 ± 57% perf-profile.self.cycles-pp.set_next_task_idle
0.06 +0.0 0.08 ± 6% -0.0 0.04 ± 57% perf-profile.self.cycles-pp.perf_swevent_event
0.07 ± 7% +0.0 0.09 ± 10% +0.0 0.07 ± 58% perf-profile.self.cycles-pp.mq_timedsend
0.00 +0.1 0.05 +0.0 0.00 perf-profile.self.cycles-pp.ct_idle_enter
0.00 +0.1 0.05 +0.0 0.00 perf-profile.self.cycles-pp.perf_trace_sched_switch
0.00 +0.1 0.06 +0.0 0.00 perf-profile.self.cycles-pp.sched_update_worker
0.12 +0.1 0.18 ± 6% -0.0 0.09 ± 57% perf-profile.self.cycles-pp.x64_sys_call
0.38 ± 2% +0.1 0.46 -0.1 0.28 ± 57% perf-profile.self.cycles-pp.finish_task_switch
0.08 ± 5% +0.1 0.20 ± 6% -0.0 0.06 ± 57% perf-profile.self.cycles-pp.do_perf_trace_sched_switch
0.18 ± 5% +0.5 0.71 ± 3% -0.0 0.14 ± 57% perf-profile.self.cycles-pp.update_sg_lb_stats
5.89 +0.9 6.81 -1.5 4.43 ± 57% perf-profile.self.cycles-pp.intel_idle
0.07 +7.4 7.48 ± 4% +0.1 0.14 ± 57% perf-profile.self.cycles-pp.__refill_obj_stock
0.00 +8.7 8.70 ± 3% +0.0 0.00 perf-profile.self.cycles-pp.drain_obj_stock
2.25 +12.4 14.61 -0.3 1.90 ± 57% perf-profile.self.cycles-pp.__memcg_slab_free_hook
0.53 +19.0 19.48 +0.1 0.60 ± 57% perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
>
> > >
> > > From: Shakeel Butt <shakeel.butt@linux.dev>
> > > Subject: [PATCH] memcg: shrink obj_stock_pcp and cache multiple objcgs
> > >
> > >
> > > Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> > > ---
> > > mm/memcontrol.c | 213 +++++++++++++++++++++++++++++++++++-------------
> > > 1 file changed, 156 insertions(+), 57 deletions(-)
> > >
> > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > > index d978e18b9b2d..2a9e5136a956 100644
> > > --- a/mm/memcontrol.c
> > > +++ b/mm/memcontrol.c
> > > @@ -150,14 +150,14 @@ static void obj_cgroup_release(struct percpu_ref *ref)
> > > * However, it can be PAGE_SIZE or (x * PAGE_SIZE).
> > > *
> > > * The following sequence can lead to it:
> > > - * 1) CPU0: objcg == stock->cached_objcg
> > > + * 1) CPU0: objcg cached in one of stock->cached[i]
> > > * 2) CPU1: we do a small allocation (e.g. 92 bytes),
> > > * PAGE_SIZE bytes are charged
> > > * 3) CPU1: a process from another memcg is allocating something,
> > > * the stock if flushed,
> > > * objcg->nr_charged_bytes = PAGE_SIZE - 92
> > > * 5) CPU0: we do release this object,
> > > - * 92 bytes are added to stock->nr_bytes
> > > + * 92 bytes are added to stock->nr_bytes[i]
> > > * 6) CPU0: stock is flushed,
> > > * 92 bytes are added to objcg->nr_charged_bytes
> > > *
> > > @@ -2017,13 +2017,25 @@ static DEFINE_PER_CPU_ALIGNED(struct memcg_stock_pcp, memcg_stock) = {
> > > .lock = INIT_LOCAL_TRYLOCK(lock),
> > > };
> > >
> > > +/*
> > > + * NR_OBJ_STOCK is sized so the entire hot path of obj_stock_pcp
> > > + * (lock, accounting metadata, nr_bytes[] and cached[]) fits within a
> > > + * single 64-byte cache line on non-debug 64-bit builds. With 5 slots:
> > > + * lock(1) + index(1) + node_id(2) + slab stats(4) + nr_bytes(10)
> > > + * + pad(6) + cached(40) == 64 bytes.
> > > + * A CPU can thus consume/refill/account against five different objcgs
> > > + * (typically per-node variants of the same memcg) while incurring at
> > > + * most one cache miss on the stock.
> > > + */
> > > +#define NR_OBJ_STOCK 5
> > > struct obj_stock_pcp {
> > > local_trylock_t lock;
> > > - unsigned int nr_bytes;
> > > - struct obj_cgroup *cached_objcg;
> > > - struct pglist_data *cached_pgdat;
> > > - int nr_slab_reclaimable_b;
> > > - int nr_slab_unreclaimable_b;
> > > + int8_t index;
> > > + int16_t node_id;
> > > + int16_t nr_slab_reclaimable_b;
> > > + int16_t nr_slab_unreclaimable_b;
> > > + uint16_t nr_bytes[NR_OBJ_STOCK];
> > > + struct obj_cgroup *cached[NR_OBJ_STOCK];
> > >
> > > struct work_struct work;
> > > unsigned long flags;
> > > @@ -2031,10 +2043,13 @@ struct obj_stock_pcp {
> > >
> > > static DEFINE_PER_CPU_ALIGNED(struct obj_stock_pcp, obj_stock) = {
> > > .lock = INIT_LOCAL_TRYLOCK(lock),
> > > + .index = -1,
> > > + .node_id = NUMA_NO_NODE,
> > > };
> > >
> > > static DEFINE_MUTEX(percpu_charge_mutex);
> > >
> > > +static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i);
> > > static void drain_obj_stock(struct obj_stock_pcp *stock);
> > > static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
> > > struct mem_cgroup *root_memcg);
> > > @@ -3152,39 +3167,68 @@ static void unlock_stock(struct obj_stock_pcp *stock)
> > > local_unlock(&obj_stock.lock);
> > > }
> > >
> > > -/* Call after __refill_obj_stock() to ensure stock->cached_objg == objcg */
> > > +/* Call after __refill_obj_stock() so a slot for objcg exists in the stock */
> > > static void __account_obj_stock(struct obj_cgroup *objcg,
> > > struct obj_stock_pcp *stock, int nr,
> > > struct pglist_data *pgdat, enum node_stat_item idx)
> > > {
> > > - int *bytes;
> > > + int16_t *bytes;
> > > + int i;
> > >
> > > - if (!stock || READ_ONCE(stock->cached_objcg) != objcg)
> > > + /*
> > > + * node_id is stored as int16_t and -1 is used as the "no pgdat
> > > + * cached" sentinel, so MAX_NUMNODES must fit in a positive int16_t.
> > > + */
> > > + BUILD_BUG_ON(MAX_NUMNODES >= S16_MAX);
> > > +
> > > + if (!stock)
> > > + goto direct;
> > > +
> > > + for (i = 0; i < NR_OBJ_STOCK; ++i) {
> > > + if (READ_ONCE(stock->cached[i]) == objcg)
> > > + break;
> > > + }
> > > + if (i == NR_OBJ_STOCK)
> > > goto direct;
> > >
> > > /*
> > > * Save vmstat data in stock and skip vmstat array update unless
> > > - * accumulating over a page of vmstat data or when pgdat changes.
> > > + * accumulating over a page of vmstat data or when the objcg slot or
> > > + * pgdat the stats belong to changes.
> > > */
> > > - if (stock->cached_pgdat != pgdat) {
> > > - /* Flush the existing cached vmstat data */
> > > - struct pglist_data *oldpg = stock->cached_pgdat;
> > > + if (stock->index < 0) {
> > > + stock->index = i;
> > > + stock->node_id = pgdat->node_id;
> > > + } else if (stock->index != i || stock->node_id != pgdat->node_id) {
> > > + struct obj_cgroup *old = READ_ONCE(stock->cached[stock->index]);
> > > + struct pglist_data *oldpg = NODE_DATA(stock->node_id);
> > >
> > > if (stock->nr_slab_reclaimable_b) {
> > > - mod_objcg_mlstate(objcg, oldpg, NR_SLAB_RECLAIMABLE_B,
> > > + mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
> > > stock->nr_slab_reclaimable_b);
> > > stock->nr_slab_reclaimable_b = 0;
> > > }
> > > if (stock->nr_slab_unreclaimable_b) {
> > > - mod_objcg_mlstate(objcg, oldpg, NR_SLAB_UNRECLAIMABLE_B,
> > > + mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
> > > stock->nr_slab_unreclaimable_b);
> > > stock->nr_slab_unreclaimable_b = 0;
> > > }
> > > - stock->cached_pgdat = pgdat;
> > > + stock->index = i;
> > > + stock->node_id = pgdat->node_id;
> > > }
> > >
> > > bytes = (idx == NR_SLAB_RECLAIMABLE_B) ? &stock->nr_slab_reclaimable_b
> > > : &stock->nr_slab_unreclaimable_b;
> > > + /*
> > > + * Cached stats are int16_t; flush directly if accumulating @nr would
> > > + * overflow or underflow the cache.
> > > + */
> > > + if (abs(nr + *bytes) >= S16_MAX) {
> > > + nr += *bytes;
> > > + *bytes = 0;
> > > + goto direct;
> > > + }
> > > +
> > > /*
> > > * Even for large object >= PAGE_SIZE, the vmstat data will still be
> > > * cached locally at least once before pushing it out.
> > > @@ -3210,10 +3254,16 @@ static bool __consume_obj_stock(struct obj_cgroup *objcg,
> > > struct obj_stock_pcp *stock,
> > > unsigned int nr_bytes)
> > > {
> > > - if (objcg == READ_ONCE(stock->cached_objcg) &&
> > > - stock->nr_bytes >= nr_bytes) {
> > > - stock->nr_bytes -= nr_bytes;
> > > - return true;
> > > + int i;
> > > +
> > > + for (i = 0; i < NR_OBJ_STOCK; ++i) {
> > > + if (READ_ONCE(stock->cached[i]) != objcg)
> > > + continue;
> > > + if (stock->nr_bytes[i] >= nr_bytes) {
> > > + stock->nr_bytes[i] -= nr_bytes;
> > > + return true;
> > > + }
> > > + return false;
> > > }
> > >
> > > return false;
> > > @@ -3234,16 +3284,42 @@ static bool consume_obj_stock(struct obj_cgroup *objcg, unsigned int nr_bytes)
> > > return ret;
> > > }
> > >
> > > -static void drain_obj_stock(struct obj_stock_pcp *stock)
> > > +/* Flush the cached slab stats (if any) back to their owning objcg/pgdat. */
> > > +static void drain_obj_stock_stats(struct obj_stock_pcp *stock)
> > > +{
> > > + struct obj_cgroup *old;
> > > + struct pglist_data *oldpg;
> > > +
> > > + if (stock->index < 0)
> > > + return;
> > > +
> > > + old = READ_ONCE(stock->cached[stock->index]);
> > > + oldpg = NODE_DATA(stock->node_id);
> > > +
> > > + if (stock->nr_slab_reclaimable_b) {
> > > + mod_objcg_mlstate(old, oldpg, NR_SLAB_RECLAIMABLE_B,
> > > + stock->nr_slab_reclaimable_b);
> > > + stock->nr_slab_reclaimable_b = 0;
> > > + }
> > > + if (stock->nr_slab_unreclaimable_b) {
> > > + mod_objcg_mlstate(old, oldpg, NR_SLAB_UNRECLAIMABLE_B,
> > > + stock->nr_slab_unreclaimable_b);
> > > + stock->nr_slab_unreclaimable_b = 0;
> > > + }
> > > + stock->index = -1;
> > > + stock->node_id = NUMA_NO_NODE;
> > > +}
> > > +
> > > +static void drain_obj_stock_slot(struct obj_stock_pcp *stock, int i)
> > > {
> > > - struct obj_cgroup *old = READ_ONCE(stock->cached_objcg);
> > > + struct obj_cgroup *old = READ_ONCE(stock->cached[i]);
> > >
> > > if (!old)
> > > return;
> > >
> > > - if (stock->nr_bytes) {
> > > - unsigned int nr_pages = stock->nr_bytes >> PAGE_SHIFT;
> > > - unsigned int nr_bytes = stock->nr_bytes & (PAGE_SIZE - 1);
> > > + if (stock->nr_bytes[i]) {
> > > + unsigned int nr_pages = stock->nr_bytes[i] >> PAGE_SHIFT;
> > > + unsigned int nr_bytes = stock->nr_bytes[i] & (PAGE_SIZE - 1);
> > >
> > > if (nr_pages) {
> > > struct mem_cgroup *memcg;
> > > @@ -3269,44 +3345,43 @@ static void drain_obj_stock(struct obj_stock_pcp *stock)
> > > * so it might be changed in the future.
> > > */
> > > atomic_add(nr_bytes, &old->nr_charged_bytes);
> > > - stock->nr_bytes = 0;
> > > + stock->nr_bytes[i] = 0;
> > > }
> > >
> > > - /*
> > > - * Flush the vmstat data in current stock
> > > - */
> > > - if (stock->nr_slab_reclaimable_b || stock->nr_slab_unreclaimable_b) {
> > > - if (stock->nr_slab_reclaimable_b) {
> > > - mod_objcg_mlstate(old, stock->cached_pgdat,
> > > - NR_SLAB_RECLAIMABLE_B,
> > > - stock->nr_slab_reclaimable_b);
> > > - stock->nr_slab_reclaimable_b = 0;
> > > - }
> > > - if (stock->nr_slab_unreclaimable_b) {
> > > - mod_objcg_mlstate(old, stock->cached_pgdat,
> > > - NR_SLAB_UNRECLAIMABLE_B,
> > > - stock->nr_slab_unreclaimable_b);
> > > - stock->nr_slab_unreclaimable_b = 0;
> > > - }
> > > - stock->cached_pgdat = NULL;
> > > - }
> > > + /* Flush vmstat data when its owning slot is being drained. */
> > > + if (stock->index == i)
> > > + drain_obj_stock_stats(stock);
> > >
> > > - WRITE_ONCE(stock->cached_objcg, NULL);
> > > + WRITE_ONCE(stock->cached[i], NULL);
> > > obj_cgroup_put(old);
> > > }
> > >
> > > +static void drain_obj_stock(struct obj_stock_pcp *stock)
> > > +{
> > > + int i;
> > > +
> > > + for (i = 0; i < NR_OBJ_STOCK; ++i)
> > > + drain_obj_stock_slot(stock, i);
> > > +}
> > > +
> > > static bool obj_stock_flush_required(struct obj_stock_pcp *stock,
> > > struct mem_cgroup *root_memcg)
> > > {
> > > - struct obj_cgroup *objcg = READ_ONCE(stock->cached_objcg);
> > > + struct obj_cgroup *objcg;
> > > struct mem_cgroup *memcg;
> > > bool flush = false;
> > > + int i;
> > >
> > > rcu_read_lock();
> > > - if (objcg) {
> > > + for (i = 0; i < NR_OBJ_STOCK; ++i) {
> > > + objcg = READ_ONCE(stock->cached[i]);
> > > + if (!objcg)
> > > + continue;
> > > memcg = obj_cgroup_memcg(objcg);
> > > - if (memcg && mem_cgroup_is_descendant(memcg, root_memcg))
> > > + if (memcg && mem_cgroup_is_descendant(memcg, root_memcg)) {
> > > flush = true;
> > > + break;
> > > + }
> > > }
> > > rcu_read_unlock();
> > >
> > > @@ -3319,6 +3394,8 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > > bool allow_uncharge)
> > > {
> > > unsigned int nr_pages = 0;
> > > + unsigned int stock_nr_bytes;
> > > + int i, slot = -1, empty_slot = -1;
> > >
> > > if (!stock) {
> > > nr_pages = nr_bytes >> PAGE_SHIFT;
> > > @@ -3327,21 +3404,43 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > > goto out;
> > > }
> > >
> > > - if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
> > > - drain_obj_stock(stock);
> > > + for (i = 0; i < NR_OBJ_STOCK; ++i) {
> > > + struct obj_cgroup *cached = READ_ONCE(stock->cached[i]);
> > > +
> > > + if (!cached) {
> > > + if (empty_slot == -1)
> > > + empty_slot = i;
> > > + continue;
> > > + }
> > > + if (cached == objcg) {
> > > + slot = i;
> > > + break;
> > > + }
> > > + }
> > > +
> > > + if (slot == -1) {
> > > + slot = empty_slot;
> > > + if (slot == -1) {
> > > + slot = get_random_u32_below(NR_OBJ_STOCK);
> > > + drain_obj_stock_slot(stock, slot);
> > > + }
> > > obj_cgroup_get(objcg);
> > > - stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
> > > + stock->nr_bytes[slot] = atomic_read(&objcg->nr_charged_bytes)
> > > ? atomic_xchg(&objcg->nr_charged_bytes, 0) : 0;
> > > - WRITE_ONCE(stock->cached_objcg, objcg);
> > > + WRITE_ONCE(stock->cached[slot], objcg);
> > >
> > > allow_uncharge = true; /* Allow uncharge when objcg changes */
> > > }
> > > - stock->nr_bytes += nr_bytes;
> > >
> > > - if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) {
> > > - nr_pages = stock->nr_bytes >> PAGE_SHIFT;
> > > - stock->nr_bytes &= (PAGE_SIZE - 1);
> > > + stock_nr_bytes = (unsigned int)stock->nr_bytes[slot] + nr_bytes;
> > > +
> > > + /* nr_bytes[] is uint16_t; flush if we would refill >= U16_MAX. */
> > > + if ((allow_uncharge && (stock_nr_bytes > PAGE_SIZE)) ||
> > > + stock_nr_bytes >= U16_MAX) {
> > > + nr_pages = stock_nr_bytes >> PAGE_SHIFT;
> > > + stock_nr_bytes &= (PAGE_SIZE - 1);
> > > }
> > > + stock->nr_bytes[slot] = stock_nr_bytes;
> > >
> > > out:
> > > if (nr_pages)
> > > --
> > > 2.53.0-Meta
> > >
^ permalink raw reply
* Re: [PATCH v3] memcg: cache obj_stock by memcg, not by objcg pointer
From: Harry Yoo @ 2026-05-19 23:39 UTC (permalink / raw)
To: Shakeel Butt
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Roman Gushchin,
Muchun Song, Qi Zheng, Alexandre Ghiti, Joshua Hahn,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <agynzVVBb4CYJTYG@linux.dev>
On 5/20/26 5:11 AM, Shakeel Butt wrote:
> On Wed, May 20, 2026 at 12:00:16AM +0900, Harry Yoo wrote:
>>
>>
> [...]
>>>
>>> The full clean solution might take one more cycle and I think we can not just
>>> ignore 67% regression on 7.1.
>>
>> That is valid point, unfortunately.
>>
>> One more thing I have to ask... for v7.1, wouldn't it be a safer option to
>> revert the per-node object change and re-introduce it once we have a cleaner
>> solution?
>
> The issue with that revert is that we reintroduce all node lru locking in the
> objcg reparenting path.
I'm not sure the problems with all-node locking are serious enough to
rule it out as an option for 7.1.
It is not ideal, but given that the critical section for reparenting is
independent of folio count, would this actually be a significant problem
in practice? (even large servers rarely go beyond 8 NUMA nodes...)
>> This change was introduced in v5, but the implementation before v4 had been
>> exposed in -next for a while, and I think we don't have enough justification
>> to keep the per-node objcgs change, at least for v7.1, given that we have an
>> unexpected last-minute regression and
>> correctness concerns (albeit slight).
>
> I am waiting for Oliver to test the multi-objcg patch I sent. If that also
> resolves the regression then we have one more option i.e. backport that to 7.1
> to fix the regression.
Yeah, If per-node objcg is required in future kernels anyway, this
option would be more ideal (if available).
> So to summarize, for future kernels we will be having multi-objcg in some form.
> For 7.1, we have to decide between (1) do nothing (2) this patch or (3) backport
> the multi-objcg path to 7.1.
Ack.
Thanks!
--
Cheers,
Harry / Hyeonggon
^ permalink raw reply
* Re: [BULK] Re: [RFC PATCH v5 20/29] sched/deadline: Allow deeper hierarchies of RT cgroups
From: luca abeni @ 2026-05-19 21:02 UTC (permalink / raw)
To: Tejun Heo
Cc: Yuri Andriaccio, Peter Zijlstra, Yuri Andriaccio, Ingo Molnar,
Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, linux-kernel, hannes,
mkoutny, cgroups
In-Reply-To: <agteySwOHszMVMp8@slm.duckdns.org>
Hi,
I think we are converging... But I still have some doubts (probably due
to the fact that I do not know the cgroup v2 API well):
On Mon, 18 May 2026 08:47:37 -1000
Tejun Heo <tj@kernel.org> wrote:
[...]
> I wonder whether it can be generalized more. Would something like the
> following work? I'm going to ignore period for the sake of simplicity
> as it doesn't seem to affect admission decisions.
>
> - There is no root cgroup.rt.max in line with other control knobs.
Well, the reason we had "rt.{runtime,period}_us" (now "rt.max") in the
root cgroup is that RT cgroups are scheduled by dl entities (one dl
entity per cpu), and these dl entities must be accounted for in the
SCHED_DEADLINE admission test... The easiest way to do this is to
reserve a fixed fraction of the CPU time to RT cgroups, leaving the
remaining fraction to SCHED_DEADLINE tasks. And we used rt.max to
configure the fraction of CPU time reserved for RT cgroups; do you have
suggestions about alternative interfaces for this configuration?
> - max means running in the nearest ancestor that has budget
> configuration. Obviously, if no one has budget configured, run in
> root.
Uh... OK; I understand this, now, but I suspect this will increase the
complexity of the admission control... Yuri, what do you think?
> - Setting a budget is subject to admission control in both directions
> - the budget source (the nearest budgeted ancestor, or the root pool
> if none) should have enough to give out and the target budget should
> be big enough to contain the actual usages and !max descendants in
> the subtree. Going to max is always fine - the source previously gave
> the budget out, so it has room to take everything back.
OK... Just to understand: if we consider this situation
root cgroup -> G1 (50, 100) -> G2 (10, 100)
and G1 switches to "max", what happens to G2? Does it stay (10, 100),
or is it forced to switch to "max", too?
I was thinking about enforcing that a cgroup can have runtime > 0 only
if it is a direct child of the root cgroup, or if its parent has
runtime > 0 and is not "max" (so, in the previous example G1 can switch
to "max" only if G2 sets its runtime to 0). Could this be acceptable?
Thanks,
Luca
>
> It seems like the above would give fairly generic behavior without
> abrupt system-wide switches while staying relatively close to the
> behaviors of other resource knobs. I could be missing something tho.
> Would something like this work?
>
> Thanks.
>
^ permalink raw reply
* Re: [PATCH v3] memcg: cache obj_stock by memcg, not by objcg pointer
From: Andrew Morton @ 2026-05-19 20:49 UTC (permalink / raw)
To: Shakeel Butt
Cc: Harry Yoo, Johannes Weiner, Michal Hocko, Roman Gushchin,
Muchun Song, Qi Zheng, Alexandre Ghiti, Joshua Hahn,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <agynzVVBb4CYJTYG@linux.dev>
On Tue, 19 May 2026 13:11:13 -0700 Shakeel Butt <shakeel.butt@linux.dev> wrote:
> Andrew, please don't send this patch to Linus until we decide on the option.
No probs, I added a note-to-self.
^ permalink raw reply
* Re: [PATCH v3] memcg: cache obj_stock by memcg, not by objcg pointer
From: Shakeel Butt @ 2026-05-19 20:11 UTC (permalink / raw)
To: Harry Yoo
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Roman Gushchin,
Muchun Song, Qi Zheng, Alexandre Ghiti, Joshua Hahn,
Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <ca8e655d-5fe7-4957-8a36-6667616be8b6@kernel.org>
On Wed, May 20, 2026 at 12:00:16AM +0900, Harry Yoo wrote:
>
>
[...]
> >
> > The full clean solution might take one more cycle and I think we can not just
> > ignore 67% regression on 7.1.
>
> That is valid point, unfortunately.
>
> One more thing I have to ask... for v7.1, wouldn't it be a safer option to
> revert the per-node object change and re-introduce it once we have a cleaner
> solution?
The issue with that revert is that we reintroduce all node lru locking in the
objcg reparenting path.
>
> This change was introduced in v5, but the implementation before v4 had been
> exposed in -next for a while, and I think we don't have enough justification
> to keep the per-node objcgs change, at least for v7.1, given that we have an
> unexpected last-minute regression and
> correctness concerns (albeit slight).
I am waiting for Oliver to test the multi-objcg patch I sent. If that also
resolves the regression then we have one more option i.e. backport that to 7.1
to fix the regression.
So to summarize, for future kernels we will be having multi-objcg in some form.
For 7.1, we have to decide between (1) do nothing (2) this patch or (3) backport
the multi-objcg path to 7.1.
Andrew, please don't send this patch to Linus until we decide on the option.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox