* Re: [RFC PATCH v5 20/29] sched/deadline: Allow deeper hierarchies of RT cgroups
From: Tejun Heo @ 2026-05-12 18:20 UTC (permalink / raw)
To: Yuri Andriaccio
Cc: luca abeni, Peter Zijlstra, Yuri Andriaccio, Ingo Molnar,
Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, linux-kernel, hannes,
mkoutny, cgroups
In-Reply-To: <b549b3cb062f2823ba6d4723b7b9260b@kernel.org>
On Tue, May 12, 2026 at 08:19:02AM -1000, Tejun Heo wrote:
> How is a delegated subtree prevented from setting cpu.rt.min = 'root' and
> escaping its ancestors' cpu.rt.max budget?
Hmm.. I guess the same problem exists w/ separate rt controller too. If the
users on the system already started using rt, how do you enable the
controller from the top down with budgets already being used down in the
hierarchy?
Thanks.
--
tejun
^ permalink raw reply
* Re: [RFC PATCH v5 20/29] sched/deadline: Allow deeper hierarchies of RT cgroups
From: Tejun Heo @ 2026-05-12 18:19 UTC (permalink / raw)
To: Yuri Andriaccio
Cc: luca abeni, Peter Zijlstra, Yuri Andriaccio, Ingo Molnar,
Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, linux-kernel, hannes,
mkoutny, cgroups
In-Reply-To: <c446b9be-38d7-425c-9ca8-eda721fe1c9e@santannapisa.it>
Hello,
How is a delegated subtree prevented from setting cpu.rt.min = 'root' and
escaping its ancestors' cpu.rt.max budget?
Thanks.
--
tejun
^ permalink raw reply
* [tj-cgroup:for-7.1-fixes] BUILD SUCCESS 5dd74441cbf42c22e874450eb6a6bbb19390a216
From: kernel test robot @ 2026-05-12 17:55 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-7.1-fixes
branch HEAD: 5dd74441cbf42c22e874450eb6a6bbb19390a216 cgroup/cpuset: Reserve DL bandwidth only for root-domain moves
elapsed time: 736m
configs tested: 248
configs skipped: 2
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-15.2.0
alpha allyesconfig gcc-15.2.0
alpha defconfig gcc-15.2.0
arc allmodconfig clang-16
arc allnoconfig gcc-15.2.0
arc allyesconfig clang-23
arc defconfig gcc-15.2.0
arc haps_hs_defconfig gcc-15.2.0
arc randconfig-001 gcc-8.5.0
arc randconfig-001-20260512 gcc-11.5.0
arc randconfig-001-20260512 gcc-8.5.0
arc randconfig-002 gcc-8.5.0
arc randconfig-002-20260512 gcc-11.5.0
arc randconfig-002-20260512 gcc-8.5.0
arm allnoconfig gcc-15.2.0
arm allyesconfig clang-16
arm defconfig gcc-15.2.0
arm footbridge_defconfig clang-17
arm randconfig-001 gcc-8.5.0
arm randconfig-001-20260512 gcc-11.5.0
arm randconfig-001-20260512 gcc-8.5.0
arm randconfig-002 gcc-8.5.0
arm randconfig-002-20260512 gcc-11.5.0
arm randconfig-002-20260512 gcc-8.5.0
arm randconfig-003 gcc-8.5.0
arm randconfig-003-20260512 gcc-11.5.0
arm randconfig-003-20260512 gcc-8.5.0
arm randconfig-004 gcc-8.5.0
arm randconfig-004-20260512 gcc-11.5.0
arm randconfig-004-20260512 gcc-8.5.0
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-15.2.0
arm64 defconfig gcc-15.2.0
arm64 randconfig-001 clang-23
arm64 randconfig-001-20260512 clang-23
arm64 randconfig-001-20260512 gcc-14.3.0
arm64 randconfig-002 clang-23
arm64 randconfig-002-20260512 clang-23
arm64 randconfig-002-20260512 gcc-14.3.0
arm64 randconfig-003 clang-23
arm64 randconfig-003-20260512 clang-23
arm64 randconfig-003-20260512 gcc-14.3.0
arm64 randconfig-004 clang-23
arm64 randconfig-004-20260512 clang-23
arm64 randconfig-004-20260512 gcc-14.3.0
csky allmodconfig gcc-15.2.0
csky allnoconfig gcc-15.2.0
csky defconfig gcc-15.2.0
csky randconfig-001 clang-23
csky randconfig-001-20260512 clang-23
csky randconfig-001-20260512 gcc-14.3.0
csky randconfig-002 clang-23
csky randconfig-002-20260512 clang-23
csky randconfig-002-20260512 gcc-14.3.0
hexagon allmodconfig gcc-15.2.0
hexagon allnoconfig gcc-15.2.0
hexagon defconfig gcc-15.2.0
hexagon randconfig-001 gcc-11.5.0
hexagon randconfig-001-20260512 gcc-10.5.0
hexagon randconfig-001-20260512 gcc-11.5.0
hexagon randconfig-002 gcc-11.5.0
hexagon randconfig-002-20260512 gcc-10.5.0
hexagon randconfig-002-20260512 gcc-11.5.0
i386 allmodconfig clang-20
i386 allnoconfig gcc-15.2.0
i386 allyesconfig clang-20
i386 buildonly-randconfig-001-20260512 gcc-14
i386 buildonly-randconfig-002-20260512 gcc-14
i386 buildonly-randconfig-003-20260512 gcc-14
i386 buildonly-randconfig-004-20260512 gcc-14
i386 buildonly-randconfig-005-20260512 gcc-14
i386 buildonly-randconfig-006-20260512 gcc-14
i386 defconfig gcc-15.2.0
i386 randconfig-001 gcc-14
i386 randconfig-001-20260512 gcc-14
i386 randconfig-002 gcc-14
i386 randconfig-002-20260512 gcc-14
i386 randconfig-003 gcc-14
i386 randconfig-003-20260512 gcc-14
i386 randconfig-004 gcc-14
i386 randconfig-004-20260512 gcc-14
i386 randconfig-005 gcc-14
i386 randconfig-005-20260512 gcc-14
i386 randconfig-006 gcc-14
i386 randconfig-006-20260512 gcc-14
i386 randconfig-007 gcc-14
i386 randconfig-007-20260512 gcc-14
i386 randconfig-011 clang-20
i386 randconfig-011-20260512 clang-20
i386 randconfig-012 clang-20
i386 randconfig-012-20260512 clang-20
i386 randconfig-013 clang-20
i386 randconfig-013-20260512 clang-20
i386 randconfig-014 clang-20
i386 randconfig-014-20260512 clang-20
i386 randconfig-015 clang-20
i386 randconfig-015-20260512 clang-20
i386 randconfig-016 clang-20
i386 randconfig-016-20260512 clang-20
i386 randconfig-017 clang-20
i386 randconfig-017-20260512 clang-20
loongarch allmodconfig clang-23
loongarch allnoconfig gcc-15.2.0
loongarch defconfig clang-19
loongarch randconfig-001 gcc-11.5.0
loongarch randconfig-001-20260512 gcc-10.5.0
loongarch randconfig-001-20260512 gcc-11.5.0
loongarch randconfig-002 gcc-11.5.0
loongarch randconfig-002-20260512 gcc-10.5.0
loongarch randconfig-002-20260512 gcc-11.5.0
m68k allmodconfig gcc-15.2.0
m68k allnoconfig gcc-15.2.0
m68k allyesconfig clang-16
m68k amiga_defconfig gcc-15.2.0
m68k atari_defconfig gcc-15.2.0
m68k defconfig clang-19
microblaze allnoconfig gcc-15.2.0
microblaze allyesconfig gcc-15.2.0
microblaze defconfig clang-19
mips allmodconfig gcc-15.2.0
mips allnoconfig gcc-15.2.0
mips allyesconfig gcc-15.2.0
mips cu1000-neo_defconfig gcc-15.2.0
nios2 allmodconfig clang-23
nios2 allnoconfig clang-23
nios2 defconfig clang-19
nios2 randconfig-001 gcc-11.5.0
nios2 randconfig-001-20260512 gcc-10.5.0
nios2 randconfig-001-20260512 gcc-11.5.0
nios2 randconfig-002 gcc-11.5.0
nios2 randconfig-002-20260512 gcc-10.5.0
nios2 randconfig-002-20260512 gcc-11.5.0
openrisc allmodconfig clang-23
openrisc allnoconfig clang-23
openrisc defconfig gcc-15.2.0
parisc allmodconfig gcc-15.2.0
parisc allnoconfig clang-23
parisc allyesconfig clang-19
parisc defconfig gcc-15.2.0
parisc randconfig-001-20260512 gcc-12.5.0
parisc randconfig-002-20260512 gcc-12.5.0
parisc64 defconfig clang-19
powerpc allmodconfig gcc-15.2.0
powerpc allnoconfig clang-23
powerpc randconfig-001-20260512 gcc-12.5.0
powerpc randconfig-002-20260512 gcc-12.5.0
powerpc64 randconfig-001-20260512 gcc-12.5.0
powerpc64 randconfig-002-20260512 gcc-12.5.0
riscv allmodconfig clang-23
riscv allnoconfig clang-23
riscv allyesconfig clang-16
riscv defconfig gcc-15.2.0
riscv randconfig-001 gcc-15.2.0
riscv randconfig-001-20260512 gcc-15.2.0
riscv randconfig-002 gcc-15.2.0
riscv randconfig-002-20260512 gcc-15.2.0
s390 allmodconfig clang-19
s390 allnoconfig clang-23
s390 allyesconfig gcc-15.2.0
s390 defconfig gcc-15.2.0
s390 randconfig-001 gcc-15.2.0
s390 randconfig-001-20260512 gcc-15.2.0
s390 randconfig-002 gcc-15.2.0
s390 randconfig-002-20260512 gcc-15.2.0
sh allmodconfig gcc-15.2.0
sh allnoconfig clang-23
sh allyesconfig clang-19
sh defconfig gcc-14
sh randconfig-001 gcc-15.2.0
sh randconfig-001-20260512 gcc-15.2.0
sh randconfig-002 gcc-15.2.0
sh randconfig-002-20260512 gcc-15.2.0
sparc allnoconfig clang-23
sparc defconfig gcc-15.2.0
sparc randconfig-001 gcc-13.4.0
sparc randconfig-001-20260512 gcc-13.4.0
sparc randconfig-002 gcc-13.4.0
sparc randconfig-002-20260512 gcc-13.4.0
sparc64 allmodconfig clang-23
sparc64 defconfig gcc-14
sparc64 randconfig-001 gcc-13.4.0
sparc64 randconfig-001-20260512 gcc-13.4.0
sparc64 randconfig-002 gcc-13.4.0
sparc64 randconfig-002-20260512 gcc-13.4.0
um allmodconfig clang-19
um allnoconfig clang-23
um allyesconfig gcc-15.2.0
um defconfig gcc-14
um i386_defconfig gcc-14
um randconfig-001 gcc-13.4.0
um randconfig-001-20260512 gcc-13.4.0
um randconfig-002 gcc-13.4.0
um randconfig-002-20260512 gcc-13.4.0
um x86_64_defconfig gcc-14
x86_64 allmodconfig clang-20
x86_64 allnoconfig clang-23
x86_64 allyesconfig clang-20
x86_64 buildonly-randconfig-001 gcc-14
x86_64 buildonly-randconfig-001-20260512 gcc-14
x86_64 buildonly-randconfig-002 gcc-14
x86_64 buildonly-randconfig-002-20260512 gcc-14
x86_64 buildonly-randconfig-003 gcc-14
x86_64 buildonly-randconfig-003-20260512 gcc-14
x86_64 buildonly-randconfig-004 gcc-14
x86_64 buildonly-randconfig-004-20260512 gcc-14
x86_64 buildonly-randconfig-005 gcc-14
x86_64 buildonly-randconfig-005-20260512 gcc-14
x86_64 buildonly-randconfig-006 gcc-14
x86_64 buildonly-randconfig-006-20260512 gcc-14
x86_64 defconfig gcc-14
x86_64 kexec clang-20
x86_64 randconfig-001 gcc-14
x86_64 randconfig-001-20260512 gcc-14
x86_64 randconfig-002 gcc-14
x86_64 randconfig-002-20260512 gcc-14
x86_64 randconfig-003 gcc-14
x86_64 randconfig-003-20260512 gcc-14
x86_64 randconfig-004 gcc-14
x86_64 randconfig-004-20260512 gcc-14
x86_64 randconfig-005 gcc-14
x86_64 randconfig-005-20260512 gcc-14
x86_64 randconfig-006 gcc-14
x86_64 randconfig-006-20260512 gcc-14
x86_64 randconfig-011-20260512 clang-20
x86_64 randconfig-012-20260512 clang-20
x86_64 randconfig-013-20260512 clang-20
x86_64 randconfig-014-20260512 clang-20
x86_64 randconfig-015-20260512 clang-20
x86_64 randconfig-016-20260512 clang-20
x86_64 randconfig-071-20260512 clang-20
x86_64 randconfig-072-20260512 clang-20
x86_64 randconfig-073-20260512 clang-20
x86_64 randconfig-074-20260512 clang-20
x86_64 randconfig-075-20260512 clang-20
x86_64 randconfig-076-20260512 clang-20
x86_64 rhel-9.4 clang-20
x86_64 rhel-9.4-bpf gcc-14
x86_64 rhel-9.4-func clang-20
x86_64 rhel-9.4-kselftests clang-20
x86_64 rhel-9.4-kunit gcc-14
x86_64 rhel-9.4-ltp gcc-14
x86_64 rhel-9.4-rust clang-20
xtensa allnoconfig clang-23
xtensa allyesconfig clang-23
xtensa randconfig-001 gcc-13.4.0
xtensa randconfig-001-20260512 gcc-13.4.0
xtensa randconfig-002 gcc-13.4.0
xtensa randconfig-002-20260512 gcc-13.4.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH 0/3] cgroup/rdma: add rdma.peak and rdma.events[.local]
From: Tejun Heo @ 2026-05-12 17:49 UTC (permalink / raw)
To: Tao Cui; +Cc: hannes, mkoutny, cgroups
In-Reply-To: <20260512031719.273507-1-cuitao@kylinos.cn>
Hello,
The list below is from an AI-assisted review with some input from me.
* Patches 2 and 3 don't extend the rpool-free condition in
uncharge_cg_locked() and rdmacg_resource_set_max() to the new event
counters, so a "set limit -> hit limit -> uncharge to 0 -> write
'max max'" sequence frees the rpool and zeros the counts.
* rdmacg_event_locked() creates rpools in ancestors of over_cg via
get_cg_rpool_locked() just to host event counters. Those rpools have
usage_sum==0, num_max_cnt==max, peak==0, so the next real uncharge
through any such ancestor frees them.
* Patch 3 says failcnt covers "this cgroup (or its descendants)" but
the code only increments the directly-requesting cgroup. Either the
description or the propagation is wrong.
* rdma.events / rdma.events.local print "mlx4_0 hca_handle.max 5
hca_object.max 0 " (trailing space). That doesn't match any of the
formats in Documentation/admin-guide/cgroup-v2.rst. rdma.current and
rdma.max are nested-keyed; the new files should be too:
"mlx4_0 hca_handle.max=5 hca_object.max=0".
* Please document rdma.peak / rdma.events / rdma.events.local in
Documentation/admin-guide/cgroup-v2.rst.
* "failcnt" is cgroup-v1 vocabulary; pids.events.local uses
"fork_fail" for the same role.
* Event counters are atomic64_t but all updates are under
rdmacg_mutex. Plain u64 with READ_ONCE on the read side would do.
* Patch 1 reflows an unrelated comment ("No user of the rpool ...");
please drop the churn.
Thanks.
--
tejun
^ permalink raw reply
* Re: [RFC PATCH v5 20/29] sched/deadline: Allow deeper hierarchies of RT cgroups
From: Yuri Andriaccio @ 2026-05-12 17:38 UTC (permalink / raw)
To: Tejun Heo
Cc: luca abeni, Peter Zijlstra, Yuri Andriaccio, Ingo Molnar,
Juri Lelli, Vincent Guittot, Dietmar Eggemann, Steven Rostedt,
Ben Segall, Mel Gorman, Valentin Schneider, linux-kernel, hannes,
mkoutny, cgroups
In-Reply-To: <agIfvZuvXEtK45em@slm.duckdns.org>
Hello,
I've been thinking and experimenting with some of the ideas for the rt
controller, and I've come up with the following interface, keeping
everything in the standard cpu controller:
- cpu.rt.max <runtime_us> <period_us>
Sets the bandwidth reserved to the hierarchy that has that specific
cgroup as root, but does
not set any deadline servers.
The default value for this file is '0 0'.
- cpu.rt.min <runtime_us | 'root'> <period_us>
If the runtime part is equal to 'root', the tasks are scheduled on
the root runqueue.
If the runtime is equal to zero, no FIFO/RR tasks can be scheduled.
If the runtime is > zero, FIFO/RR tasks are scheduled under
reservation/HCBS.
This file is not available in the root cgroup, as it does not make
use of dl-servers,
rather only reserves the total bandwidth for the hierarchy.
The default value for this file is 'root 0', meaning that tasks in
this cgroups are
by default scheduled on the root runqueue.
Of course you can imagine that all the admission tests have been updated
accordingly, as an example a cgroups rt.max bw must be >= than the sum
of the rt.max bws of its children + its rt.min bw. I'm also skipping
some details which are only meaningful if we decide to adopt this solution.
What do you think of this interface?
Thanks,
Yuri
^ permalink raw reply
* Re: [PATCH v2] selftests/cgroup: check malloc return value in alloc_anon functions
From: Tejun Heo @ 2026-05-12 17:37 UTC (permalink / raw)
To: Hongfu Li
Cc: hannes, mhocko, roman.gushchin, shakeel.butt, muchun.song,
mkoutny, shuah, vishal.moola, cgroups, linux-mm, linux-kselftest,
linux-kernel
In-Reply-To: <20260512071424.59449-1-lihongfu@kylinos.cn>
Hello,
> Hongfu Li (1):
> selftests/cgroup: check malloc return value in alloc_anon functions
Applied to cgroup/for-7.2.
Thanks.
--
tejun
^ permalink raw reply
* Re: [PATCH 1/8] mm: memcontrol: propagate NMI slab stats to memcg vmstats
From: Shakeel Butt @ 2026-05-12 16:08 UTC (permalink / raw)
To: Alexandre Ghiti
Cc: Andrew Morton, Johannes Weiner, Michal Hocko, Roman Gushchin,
Muchun Song, Dennis Zhou, Tejun Heo, Christoph Lameter,
Vlastimil Babka, Yosry Ahmed, Nhat Pham, Sergey Senozhatsky,
Chengming Zhou, Suren Baghdasaryan, Qi Zheng, David Hildenbrand,
Lorenzo Stoakes, Minchan Kim, Mike Rapoport, Axel Rasmussen,
Barry Song, Kairui Song, Wei Xu, Yuanchu Xie, Liam R . Howlett,
Joshua Hahn, linux-mm, linux-kernel, cgroups
In-Reply-To: <agJYrraKk_wbHe07@linux.dev>
On Mon, May 11, 2026 at 03:49:36PM -0700, Shakeel Butt wrote:
> On Mon, May 11, 2026 at 10:20:36PM +0200, Alexandre Ghiti wrote:
> > flush_nmi_stats() drains per-node NMI slab atomics into the per-node
> > lruvec_stats, but does not propagate them to the memcg-level vmstats.
> >
> > This is inconsistent with account_slab_nmi_safe() which updates both,
>
[...]
>
> > so fix this by propagating the NMI slab stats to the memcg-level vmstats.
> >
> > Fixes: 940b01fc8dc1 ("memcg: nmi safe memcg stats for specific archs")
> > Signed-off-by: Alexandre Ghiti <alex@ghiti.fr>
Please resend this patch as independent patch and CC stable.
^ permalink raw reply
* Re: [linus:master] [mm] 01b9da291c: stress-ng.switch.ops_per_sec 67.7% regression
From: Shakeel Butt @ 2026-05-12 16:03 UTC (permalink / raw)
To: kernel test robot
Cc: Qi Zheng, oe-lkp, lkp, linux-kernel, Andrew Morton, David Carlier,
Allen Pais, Axel Rasmussen, Baoquan He, Chengming Zhou,
Chen Ridong, David Hildenbrand, Hamza Mahfooz, Harry Yoo,
Hugh Dickins, Imran Khan, Johannes Weiner, Kamalesh Babulal,
Lance Yang, Liam Howlett, Lorenzo Stoakes, Michal Hocko,
Michal Koutný, Mike Rapoport, Muchun Song, Muchun Song,
Nhat Pham, Roman Gushchin, Suren Baghdasaryan, Usama Arif,
Vlastimil Babka, Wei Xu, Yosry Ahmed, Yuanchu Xie, Zi Yan,
Usama Arif, cgroups, linux-mm
In-Reply-To: <202605121641.b6a60cb0-lkp@intel.com>
On Tue, May 12, 2026 at 08:56:52PM +0800, kernel test robot wrote:
>
>
> Hello,
>
> kernel test robot noticed a 67.7% regression of stress-ng.switch.ops_per_sec on:
>
>
> commit: 01b9da291c4969354807b52956f4aae1f41b4924 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
This is most probably due to shuffling of struct mem_cgroup and struct
mem_cgroup_per_node members.
I will try to reproduce and will followup on this.
^ permalink raw reply
* Re: [PATCH 2/3] mm/zswap: Implement proactive writeback
From: Nhat Pham @ 2026-05-12 15:47 UTC (permalink / raw)
To: Hao Jia
Cc: Yosry Ahmed, akpm, tj, hannes, shakeel.butt, mhocko, mkoutny,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia, Alexandre Ghiti
In-Reply-To: <12e4784e-2add-d849-7e54-bde8abfa6e78@gmail.com>
On Tue, May 12, 2026 at 2:32 AM Hao Jia <jiahao.kernel@gmail.com> wrote:
>
>
>
> On 2026/5/12 03:57, Yosry Ahmed wrote:
> > On Mon, May 11, 2026 at 12:49 PM Nhat Pham <nphamcs@gmail.com> wrote:
> >>
> >> On Mon, May 11, 2026 at 3:52 AM Hao Jia <jiahao.kernel@gmail.com> wrote:
> >>>
> >>> From: Hao Jia <jiahao1@lixiang.com>
> >>>
> >>> Zswap currently writes back pages to backing swap devices reactively,
> >>> triggered either by memory pressure via the shrinker or by the pool
> >>> reaching its size limit. This reactive approach offers no precise
> >>> control over when writeback happens, which can disturb latency-sensitive
> >>> workloads, and it cannot direct writeback at a specific memory cgroup.
> >>> However, there are scenarios where users might want to proactively
> >>> write back cold pages from zswap to the backing swap device, for
> >>> example, to free up memory for other applications or to prepare for
> >>> upcoming memory-intensive workloads.
> >>>
> >>> Therefore, implement a proactive writeback mechanism for zswap by
> >>> adding a new cgroup interface file memory.zswap.proactive_writeback
> >>> within the memory controller.
> >>
>
> Thanks Nhat, Yosry — let me address both comments together.
>
> >>
> >> We already have memory.reclaim, no? Would that not work to create
> >> headroom generally for your use case? Is there a reason why we are
> >> treating zswap memory as special here?
> >
>
> Apologies for the lack of detailed explanation in the patch description,
> which led to the confusion.
>
> While we are already utilizing memory.reclaim, it does not fully address
> our requirements.
>
> Our deployment runs a userspace proactive reclaimer that drives
> memory.reclaim based on the system's runtime state (memory/CPU/IO
> pressure, refault rate, ...) and workload-specific
> policy. That first stage compresses cold anon pages into zswap. Entries
> that then remain in zswap past a policy-defined age threshold are
> considered "twice cold", and the reclaimer wants
> to write them back to the backing swap device at a moment of its own
> choosing, to further reclaim the DRAM still held by the compressed data.
>
> This is the "second-level offloading" pattern described in Meta's TMO
> paper [1]. zswap proactive writeback is what this series introduces to
> address that second-level offloading stage.
>
> [1] https://www.pdl.cmu.edu/ftp/NVM/tmo_asplos22.pdf
Yeah that's what we've been trying to work on as well :) We are
working on a couple of improvements to the mechanism side of this path
(cc Alex) - hopefully it will help your use case too!
Anyway, back to my original inquiry: I understand your use case. It's
pretty similar to our goal. What I'm not getting is why is
memory.reclaim (which you already use) not sufficient for zswap ->
disk swap offloading too?
Zswap objects are organized into LRU and exposed to the shrinker
interface. Echo-ing to memory.reclaim should also offload some zswap
entries, correct? Are there still cold zswap entries that escape this,
somehow?
Furthermore, we already have a way to detect the "twice cold" entries
you mentioned: the referenced bit. This is analogous to the way we
treat uncompressed pages.
>
>
> > +1, why do we need to specifically proactively reclaim the compressed memory?
> >
> > Also, if we do need to minimize the compressed memory and force higher
> > writeback rates, we can do so with memory.zswap.max, right?
>
> Here are a few reasons why memory.zswap.max is not enough:
>
> 1. Writing memory.zswap.max itself does not trigger any writeback
> immediately. For a memcg that has reached steady state (on which the
> userspace reclaimer is no longer invoking
> memory.reclaim), after enough time has passed, the reclaimer has no good
> way to trigger proactive writeback for second-level offloading by
> lowering memory.zswap.max, because in steady
> state nothing drives the zswap_store() -> shrink_memcg() path. The
> userspace reclaimer still has no control over when proactive writeback
> happens.
>
> 2. memory.zswap.max currently triggers zswap writeback via zswap_store()
> -> shrink_memcg(), and each over-limit event can write back at most
> NR_NODES entries. If zswap residency is far
> above memory.zswap.max, converging to the target size requires at least
> O(over-limit pages / NR_NODES) zswap_store() events, with no batching —
> proactive writeback therefore has
> significant latency.
>
> 3. memory.zswap.max is a stateful interface. If the userspace reclaimer
> crashes for any reason mid-operation, it may leave memory.zswap.max at
> some set value, putting the application in a
> persistently throttled bad state.
>
> 4. Once the userspace reclaimer has lowered memory.zswap.max, if the
> workload is rapidly expanding and triggers memory reclaim via
> memory.high / kswapd / etc., the actual amount written
> back can exceed what was intended.
One more reason: IIRC, when you set memory.zswap.max to a value other
than 0 max, every zswap store incurs a pretty expensive check
(obj_cgroup_may_zswap), which does a force flush
(__mem_cgroup_flush_stats). That was pretty expensive last time some
of our internal services played with it. So yeah, it's not ideal...
(if you're using this, might wanna profile this as well).
>
> Thanks,
> Hao
^ permalink raw reply
* Re: [PATCH v3 02/12] mm, swap: move common swap cache operations into standalone helpers
From: Kairui Song @ 2026-05-12 14:48 UTC (permalink / raw)
To: Chris Li
Cc: linux-mm, Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang,
Barry Song, Hugh Dickins, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Qi Zheng, linux-kernel, cgroups,
Yosry Ahmed, Lorenzo Stoakes, Dev Jain, Lance Yang, Michal Hocko,
Michal Hocko, Suren Baghdasaryan, Axel Rasmussen
In-Reply-To: <CACePvbX3Bead1Ea+UJfUAq0e4JNJ0P+yN0=zf5rhdPsH8y9PUg@mail.gmail.com>
On Wed, May 6, 2026 at 10:46 PM Chris Li <chrisl@kernel.org> wrote:
>
> On Tue, Apr 21, 2026 at 8:16 AM Kairui Song via B4 Relay
> <devnull+kasong.tencent.com@kernel.org> wrote:
> >
> > From: Kairui Song <kasong@tencent.com>
> >
> > Move a few swap cache checking, adding, and deletion operations
> > into standalone helpers to be used later. And while at it, add
> > proper kernel doc.
> >
> > No feature or behavior change.
> >
> > Signed-off-by: Kairui Song <kasong@tencent.com>
>
> Acked-by: Chris Li <chrisl@kernel.org>
>
>
> > ---
> > mm/swap_state.c | 141 ++++++++++++++++++++++++++++++++++++++------------------
> > 1 file changed, 95 insertions(+), 46 deletions(-)
> >
> > diff --git a/mm/swap_state.c b/mm/swap_state.c
> > index 204a9499d50c..3da285a891b2 100644
> > --- a/mm/swap_state.c
> > +++ b/mm/swap_state.c
> > @@ -137,8 +137,42 @@ void *swap_cache_get_shadow(swp_entry_t entry)
> > return NULL;
> > }
> >
> > -void __swap_cache_add_folio(struct swap_cluster_info *ci,
> > - struct folio *folio, swp_entry_t entry)
> > +/**
> > + * __swap_cache_add_check - Check if a range is suitable for adding a folio.
> > + * @ci: The locked swap cluster.
> > + * @ci_off: Range start offset.
> > + * @nr: Number of slots to check.
> > + * @shadow: Returns the shadow value if one exists in the range.
> > + *
> > + * Check if all slots covered by given range have a swap count >= 1.
> > + * Retrieves the shadow if there is one.
> > + *
> > + * Context: Caller must lock the cluster.
> > + */
> > +static int __swap_cache_add_check(struct swap_cluster_info *ci,
> > + unsigned int ci_off, unsigned int nr,
> > + void **shadow)
> > +{
> > + unsigned int ci_end = ci_off + nr;
> > + unsigned long old_tb;
> > +
>
> Nitpick: Can add lockdep_assert_held(&ci->lock);
>
> Can check ci_end < SWAPFILE_CLUSTER and bail out on error.
Ack.
>
> > + if (unlikely(!ci->table))
> > + return -ENOENT;
> > + do {
> > + old_tb = __swap_table_get(ci, ci_off);
> > + if (unlikely(swp_tb_is_folio(old_tb)))
> > + return -EEXIST;
> > + if (unlikely(!__swp_tb_get_count(old_tb)))
> > + return -ENOENT;
> > + if (swp_tb_is_shadow(old_tb))
> > + *shadow = swp_tb_to_shadow(old_tb);
>
> Nitpick: You can create a local variable for the shadow and assign it
> at the end. Because it is a pointer, the compiler can't optimize the
> store away.
This part will be reworked very soon but using a local variable here
is good for an intermediate commit.
> Chris
Thanks.
^ permalink raw reply
* [linus:master] [mm] 01b9da291c: stress-ng.switch.ops_per_sec 67.7% regression
From: kernel test robot @ 2026-05-12 12:56 UTC (permalink / raw)
To: Qi Zheng
Cc: oe-lkp, lkp, linux-kernel, Andrew Morton, David Carlier,
Shakeel Butt, Allen Pais, Axel Rasmussen, Baoquan He,
Chengming Zhou, Chen Ridong, David Hildenbrand, Hamza Mahfooz,
Harry Yoo, Hugh Dickins, Imran Khan, Johannes Weiner,
Kamalesh Babulal, Lance Yang, Liam Howlett, Lorenzo Stoakes,
Michal Hocko, Michal Koutný, Mike Rapoport, Muchun Song,
Muchun Song, Nhat Pham, Roman Gushchin, Suren Baghdasaryan,
Usama Arif, Vlastimil Babka, Wei Xu, Yosry Ahmed, Yuanchu Xie,
Zi Yan, Usama Arif, cgroups, linux-mm, oliver.sang
Hello,
kernel test robot noticed a 67.7% regression of stress-ng.switch.ops_per_sec on:
commit: 01b9da291c4969354807b52956f4aae1f41b4924 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
[still regression on linus/master 66edb901bf874d9e0787326ba12d3548b2da8700]
[still regression on linux-next/master b9303e6bff706758c167af686b5315ad00233bf8]
testcase: stress-ng
config: x86_64-rhel-9.4
compiler: gcc-14
test machine: 224 threads 2 sockets Intel(R) Xeon(R) Platinum 8480CTDX (Sapphire Rapids) with 256G memory
parameters:
nr_threads: 100%
testtime: 60s
test: switch
method: mq
cpufreq_governor: performance
In addition to that, the commit also has significant impact on the following tests:
+------------------+--------------------------------------------------+
| testcase: change | unixbench: unixbench.throughput 3.8% regression |
| test parameters | cpufreq_governor=performance |
| | nr_task=100% |
| | runtime=300s |
| | test=spawn |
+------------------+--------------------------------------------------+
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <oliver.sang@intel.com>
| Closes: https://lore.kernel.org/oe-lkp/202605121641.b6a60cb0-lkp@intel.com
Details are as below:
-------------------------------------------------------------------------------------------------->
The kernel config and materials to reproduce are available at:
https://download.01.org/0day-ci/archive/20260512/202605121641.b6a60cb0-lkp@intel.com
=========================================================================================
compiler/cpufreq_governor/kconfig/method/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-14/performance/x86_64-rhel-9.4/mq/100%/debian-13-x86_64-20250902.cgz/lkp-spr-r02/switch/stress-ng/60s
commit:
8285917d6f ("mm: memcontrol: prepare for reparenting non-hierarchical stats")
01b9da291c ("mm: memcontrol: convert objcg to be per-memcg per-node type")
8285917d6f383aef 01b9da291c4969354807b52956f
---------------- ---------------------------
%stddev %change %stddev
\ | \
5849 +210.2% 18145 ± 3% stress-ng.switch.nanosecs_per_context_switch_mq_method
2.296e+09 -67.7% 7.408e+08 ± 3% stress-ng.switch.ops
38288993 -67.7% 12355813 ± 3% stress-ng.switch.ops_per_sec
93416932 -68.6% 29310048 ± 3% stress-ng.time.involuntary_context_switches
15845 +11.0% 17584 stress-ng.time.percent_of_cpu_this_job_got
8556 +18.2% 10115 stress-ng.time.system_time
963.36 -53.5% 447.72 ± 3% stress-ng.time.user_time
1.518e+09 -69.7% 4.607e+08 ± 2% stress-ng.time.voluntary_context_switches
1124 ± 17% +34.3% 1509 ± 8% perf-c2c.HITM.remote
2.55e+09 -12.3% 2.236e+09 ± 3% cpuidle..time
8.29e+08 -71.8% 2.337e+08 ± 2% cpuidle..usage
4102960 ± 5% -19.0% 3324393 ± 4% numa-numastat.node1.local_node
4218983 ± 3% -18.7% 3430325 ± 3% numa-numastat.node1.numa_hit
14184409 ± 2% -16.4% 11860068 vmstat.memory.cache
39204964 -69.7% 11868752 ± 2% vmstat.system.cs
1808848 -38.5% 1111830 vmstat.system.in
22.48 -4.4 18.08 mpstat.cpu.all.idle%
1.13 -0.4 0.73 mpstat.cpu.all.irq%
0.10 -0.0 0.09 mpstat.cpu.all.soft%
67.98 +9.1 77.06 mpstat.cpu.all.sys%
8.32 -4.3 4.04 ± 2% mpstat.cpu.all.usr%
17.33 ± 2% +15.4% 20.00 ± 4% mpstat.max_utilization.seconds
10552401 ± 4% -23.3% 8092823 numa-meminfo.node1.Active
10552392 ± 4% -23.3% 8092820 numa-meminfo.node1.Active(anon)
12454155 ± 15% -34.9% 8106052 numa-meminfo.node1.FilePages
559046 ± 8% -19.2% 451929 ± 2% numa-meminfo.node1.Mapped
14688311 ± 13% -30.0% 10285394 ± 2% numa-meminfo.node1.MemUsed
10028979 ± 3% -22.4% 7783864 numa-meminfo.node1.Shmem
2638537 ± 4% -23.3% 2022639 numa-vmstat.node1.nr_active_anon
3113944 ± 15% -34.9% 2025946 numa-vmstat.node1.nr_file_pages
139848 ± 9% -19.3% 112912 ± 2% numa-vmstat.node1.nr_mapped
2507650 ± 3% -22.4% 1945399 numa-vmstat.node1.nr_shmem
2638531 ± 4% -23.3% 2022634 numa-vmstat.node1.nr_zone_active_anon
4219206 ± 3% -18.7% 3430093 ± 3% numa-vmstat.node1.numa_hit
4103183 ± 4% -19.0% 3324161 ± 4% numa-vmstat.node1.numa_local
10939677 ± 2% -21.0% 8641166 meminfo.Active
10939661 ± 2% -21.0% 8641149 meminfo.Active(anon)
13917673 ± 2% -16.4% 11633722 meminfo.Cached
14400924 ± 2% -16.0% 12102150 meminfo.Committed_AS
8394752 ± 5% +16.7% 9796949 ± 8% meminfo.DirectMap2M
617671 -12.0% 543559 meminfo.Mapped
18364992 -12.5% 16065468 meminfo.Memused
10124702 ± 2% -22.6% 7839682 meminfo.Shmem
18393665 -12.5% 16100473 meminfo.max_used_kB
10.59 -7.6 2.97 ± 8% turbostat.C1%
0.85 ± 3% +9.1 9.96 ± 2% turbostat.C1E%
1.29 ± 6% +19.4% 1.54 ± 2% turbostat.CPU%c1
48.67 ± 2% -15.1% 41.33 ± 3% turbostat.CoreTmp
0.56 -60.7% 0.22 ± 3% turbostat.IPC
1.153e+08 -38.7% 70680365 turbostat.IRQ
10242404 -14.8% 8723704 turbostat.NMI
88.65 -84.0 4.67 ± 33% turbostat.PKG_%
3.82 -3.8 0.04 ± 10% turbostat.POLL%
48.67 ± 2% -13.7% 42.00 ± 3% turbostat.PkgTmp
683.77 -13.1% 594.00 turbostat.PkgWatt
18.74 -3.3% 18.13 turbostat.RAMWatt
2735312 ± 2% -21.0% 2160742 proc-vmstat.nr_active_anon
204708 -1.6% 201435 proc-vmstat.nr_anon_pages
3479812 ± 2% -16.4% 2908863 proc-vmstat.nr_file_pages
154477 -12.0% 135959 proc-vmstat.nr_mapped
2531568 ± 2% -22.6% 1960353 proc-vmstat.nr_shmem
42010 -3.5% 40543 proc-vmstat.nr_slab_reclaimable
2735312 ± 2% -21.0% 2160742 proc-vmstat.nr_zone_active_anon
210167 ± 5% -11.5% 185950 ± 11% proc-vmstat.numa_hint_faults
4730338 ± 2% -18.2% 3871343 proc-vmstat.numa_hit
4498551 ± 2% -19.1% 3639783 proc-vmstat.numa_local
4808959 ± 2% -17.8% 3954157 proc-vmstat.pgalloc_normal
806619 -5.1% 765525 ± 2% proc-vmstat.pgfault
34098 ± 3% -14.8% 29054 proc-vmstat.pgreuse
0.11 +59.9% 0.17 ± 3% perf-stat.i.MPKI
6.653e+10 -61.7% 2.546e+10 ± 2% perf-stat.i.branch-instructions
0.76 +0.1 0.89 perf-stat.i.branch-miss-rate%
4.685e+08 -59.7% 1.888e+08 ± 2% perf-stat.i.branch-misses
1.12 +0.6 1.76 ± 3% perf-stat.i.cache-miss-rate%
35553724 ± 3% -40.4% 21188697 perf-stat.i.cache-misses
4.194e+09 -68.3% 1.331e+09 ± 2% perf-stat.i.cache-references
40710745 -69.6% 12395879 ± 2% perf-stat.i.context-switches
1.84 +189.1% 5.31 ± 2% perf-stat.i.cpi
5.965e+11 -2.0% 5.848e+11 perf-stat.i.cpu-cycles
8813175 -64.5% 3125097 ± 2% perf-stat.i.cpu-migrations
24447 ± 3% +68.5% 41184 ± 2% perf-stat.i.cycles-between-cache-misses
3.374e+11 -61.8% 1.287e+11 ± 2% perf-stat.i.instructions
0.57 -60.8% 0.22 ± 2% perf-stat.i.ipc
221.10 -68.6% 69.32 ± 2% perf-stat.i.metric.K/sec
11782 ± 3% -6.1% 11068 ± 3% perf-stat.i.minor-faults
11782 ± 3% -6.1% 11068 ± 3% perf-stat.i.page-faults
0.10 ± 2% +59.2% 0.17 ± 3% perf-stat.overall.MPKI
0.71 +0.0 0.75 perf-stat.overall.branch-miss-rate%
0.83 ± 3% +0.7 1.56 ± 3% perf-stat.overall.cache-miss-rate%
1.78 +162.2% 4.67 ± 2% perf-stat.overall.cpi
17181 ± 3% +64.6% 28283 perf-stat.overall.cycles-between-cache-misses
0.56 -61.8% 0.21 ± 2% perf-stat.overall.ipc
6.388e+10 -62.3% 2.409e+10 ± 2% perf-stat.ps.branch-instructions
4.538e+08 -60.0% 1.817e+08 ± 2% perf-stat.ps.branch-misses
33674051 ± 3% -40.1% 20155290 perf-stat.ps.cache-misses
4.077e+09 -68.2% 1.296e+09 ± 2% perf-stat.ps.cache-references
39570629 -69.5% 12072702 ± 2% perf-stat.ps.context-switches
5.78e+11 -1.4% 5.7e+11 perf-stat.ps.cpu-cycles
8584979 -64.5% 3051930 ± 2% perf-stat.ps.cpu-migrations
3.243e+11 -62.4% 1.22e+11 ± 2% perf-stat.ps.instructions
11022 ± 4% -6.5% 10300 ± 3% perf-stat.ps.minor-faults
11022 ± 4% -6.5% 10300 ± 3% perf-stat.ps.page-faults
1.941e+13 -61.9% 7.405e+12 ± 3% perf-stat.total.instructions
18451 +9.9% 20272 sched_debug.cfs_rq:/.avg_vruntime.avg
5869 ± 4% -7.4% 5437 ± 5% sched_debug.cfs_rq:/.avg_vruntime.stddev
0.68 ± 2% -12.9% 0.59 ± 3% sched_debug.cfs_rq:/.h_nr_queued.stddev
0.62 ± 6% -12.9% 0.54 ± 2% sched_debug.cfs_rq:/.h_nr_runnable.stddev
8469 +12.7% 9544 ± 2% sched_debug.cfs_rq:/.left_deadline.stddev
8467 +12.7% 9542 ± 2% sched_debug.cfs_rq:/.left_vruntime.stddev
3513124 ± 25% -30.0% 2459550 ± 10% sched_debug.cfs_rq:/.load.max
588329 ± 5% -11.2% 522578 ± 5% sched_debug.cfs_rq:/.load.stddev
50699 ± 17% -19.8% 40655 ± 7% sched_debug.cfs_rq:/.load_avg.max
0.68 ± 2% -12.9% 0.59 ± 3% sched_debug.cfs_rq:/.nr_queued.stddev
38.80 ± 32% +108.5% 80.90 ± 16% sched_debug.cfs_rq:/.removed.load_avg.avg
857.83 ± 12% +61.0% 1381 ± 12% sched_debug.cfs_rq:/.removed.load_avg.max
152.02 ± 18% +57.2% 239.02 ± 11% sched_debug.cfs_rq:/.removed.load_avg.stddev
26.08 ± 28% +143.0% 63.37 ± 14% sched_debug.cfs_rq:/.removed.runnable_avg.avg
547.00 ± 13% +88.7% 1032 ± 12% sched_debug.cfs_rq:/.removed.runnable_avg.max
94.86 ± 17% +84.3% 174.82 ± 9% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
9.09 ± 52% +253.3% 32.11 ± 17% sched_debug.cfs_rq:/.removed.util_avg.avg
275.17 ± 3% +130.3% 633.67 ± 9% sched_debug.cfs_rq:/.removed.util_avg.max
44.90 ± 30% +126.4% 101.66 ± 11% sched_debug.cfs_rq:/.removed.util_avg.stddev
8467 +12.7% 9542 ± 2% sched_debug.cfs_rq:/.right_vruntime.stddev
659.63 ± 3% +13.0% 745.47 sched_debug.cfs_rq:/.runnable_avg.avg
271.34 ± 2% +31.2% 355.98 ± 3% sched_debug.cfs_rq:/.runnable_avg.stddev
0.00 ± 26% +110.4% 0.00 ± 45% sched_debug.cfs_rq:/.spread.avg
0.01 ± 13% +174.3% 0.02 ± 25% sched_debug.cfs_rq:/.spread.max
0.00 ± 7% +146.2% 0.00 ± 27% sched_debug.cfs_rq:/.spread.stddev
431.00 +14.5% 493.62 sched_debug.cfs_rq:/.util_avg.avg
1061 ± 3% +26.4% 1341 ± 3% sched_debug.cfs_rq:/.util_avg.max
151.53 ± 5% +50.1% 227.46 ± 2% sched_debug.cfs_rq:/.util_avg.stddev
206.96 +17.5% 243.18 ± 3% sched_debug.cfs_rq:/.util_est.avg
18451 +9.9% 20272 sched_debug.cfs_rq:/.zero_vruntime.avg
5869 ± 4% -7.4% 5437 ± 5% sched_debug.cfs_rq:/.zero_vruntime.stddev
2345 +33.6% 3133 ± 5% sched_debug.cpu.avg_idle.min
13.18 ± 2% +39.8% 18.42 ± 6% sched_debug.cpu.clock.stddev
3961 +14.6% 4541 sched_debug.cpu.curr->pid.avg
3213 -15.4% 2718 sched_debug.cpu.curr->pid.stddev
0.00 ± 29% +157.3% 0.00 ± 35% sched_debug.cpu.next_balance.stddev
0.70 -15.8% 0.59 ± 3% sched_debug.cpu.nr_running.stddev
5474800 -69.7% 1660250 ± 2% sched_debug.cpu.nr_switches.avg
5648642 -65.5% 1946319 ± 5% sched_debug.cpu.nr_switches.max
2229198 ± 8% -67.1% 734011 ± 20% sched_debug.cpu.nr_switches.min
297592 ± 6% -25.9% 220513 ± 18% sched_debug.cpu.nr_switches.stddev
23.75 -10.9 12.88 perf-profile.calltrace.cycles-pp.common_startup_64
23.65 -10.8 12.82 perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
23.62 -10.8 12.81 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
23.51 -10.8 12.76 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
12.93 -7.0 5.94 ± 4% perf-profile.calltrace.cycles-pp.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
12.78 -6.9 5.89 ± 4% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
11.30 -5.2 6.07 ± 3% perf-profile.calltrace.cycles-pp.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
11.17 -5.2 6.02 ± 3% perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
9.89 -4.8 5.08 ± 4% perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q
12.41 -4.4 7.96 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
9.19 -4.4 4.77 ± 4% perf-profile.calltrace.cycles-pp.select_idle_cpu.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up
11.29 -4.0 7.29 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
11.38 -4.0 7.40 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
8.04 -3.9 4.13 ± 4% perf-profile.calltrace.cycles-pp.select_idle_core.select_idle_cpu.select_idle_sibling.select_task_rq_fair.select_task_rq
6.91 -3.8 3.08 ± 4% perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive
6.76 -3.8 3.00 ± 4% perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedreceive
8.71 -3.7 5.00 perf-profile.calltrace.cycles-pp.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.71 -3.4 2.26 ± 4% perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.common_startup_64
8.12 -3.4 4.72 perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
7.76 -3.2 4.55 perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend
8.39 -3.1 5.26 perf-profile.calltrace.cycles-pp.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
7.47 -3.1 4.35 perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend
4.92 -2.9 2.00 ± 4% perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
7.79 -2.8 4.97 perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
7.48 -2.7 4.79 perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive
7.17 -2.6 4.60 perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive
5.54 -2.3 3.24 ± 4% perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend
5.41 -2.2 3.16 ± 4% perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedsend
3.88 -2.2 1.67 ± 4% perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry
4.04 -2.2 1.88 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.common_startup_64
3.44 -2.2 1.28 ± 3% perf-profile.calltrace.cycles-pp.store_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.84 -2.0 1.80 perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
4.83 -2.0 2.85 perf-profile.calltrace.cycles-pp.try_to_block_task.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep
4.71 -1.9 2.78 perf-profile.calltrace.cycles-pp.dequeue_task_fair.try_to_block_task.__schedule.schedule.schedule_hrtimeout_range_clock
4.39 -1.8 2.58 perf-profile.calltrace.cycles-pp.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule.schedule
2.37 -1.5 0.84 ± 3% perf-profile.calltrace.cycles-pp._copy_to_user.store_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
2.66 -1.4 1.26 ± 4% perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle
2.73 -1.4 1.33 ± 4% perf-profile.calltrace.cycles-pp.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.50 -1.3 2.17 perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule
2.37 -1.3 1.06 ± 2% perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
4.10 -1.1 3.03 perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep
2.20 -1.1 1.13 ± 5% perf-profile.calltrace.cycles-pp.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue
2.20 -1.1 1.15 ± 4% perf-profile.calltrace.cycles-pp.switch_fpu_return.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.74 -1.0 0.73 ± 3% perf-profile.calltrace.cycles-pp.msg_get.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.32 -1.0 0.37 ± 70% perf-profile.calltrace.cycles-pp.do_perf_trace_sched_wakeup_template.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedreceive
1.93 -0.9 0.99 ± 4% perf-profile.calltrace.cycles-pp.enqueue_task_fair.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue
1.34 -0.8 0.54 ± 5% perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive
1.52 -0.8 0.73 ± 4% perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.enqueue_task.ttwu_do_activate.sched_ttwu_pending
1.05 -0.7 0.34 ± 70% perf-profile.calltrace.cycles-pp.__switch_to
1.57 -0.7 0.90 ± 6% perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.56 -0.3 2.28 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule.schedule_hrtimeout_range_clock
5.87 +0.9 6.78 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.00 +0.9 0.91 ± 30% perf-profile.calltrace.cycles-pp.sched_balance_newidle.pick_next_task_fair.__pick_next_task.__schedule.schedule
35.80 +3.5 39.29 perf-profile.calltrace.cycles-pp.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
35.59 +3.6 39.21 perf-profile.calltrace.cycles-pp.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +4.4 4.37 ± 3% perf-profile.calltrace.cycles-pp.drain_obj_stock.__refill_obj_stock.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg
0.00 +4.4 4.39 ± 4% perf-profile.calltrace.cycles-pp.drain_obj_stock.__refill_obj_stock.__memcg_slab_free_hook.kfree.free_msg
0.00 +8.0 8.01 ± 4% perf-profile.calltrace.cycles-pp.__refill_obj_stock.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg.do_mq_timedsend
0.00 +8.3 8.29 ± 4% perf-profile.calltrace.cycles-pp.__refill_obj_stock.__memcg_slab_free_hook.kfree.free_msg.do_mq_timedreceive
28.51 +13.5 41.99 perf-profile.calltrace.cycles-pp.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
28.23 +13.5 41.71 perf-profile.calltrace.cycles-pp.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
70.69 +13.7 84.35 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
70.44 +13.8 84.26 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.99 +20.2 23.24 ± 2% perf-profile.calltrace.cycles-pp.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.79 +20.4 23.15 ± 2% perf-profile.calltrace.cycles-pp.kfree.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
2.43 +20.6 22.98 ± 2% perf-profile.calltrace.cycles-pp.__memcg_slab_free_hook.kfree.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive
2.26 +26.0 28.23 ± 2% perf-profile.calltrace.cycles-pp.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.99 +26.8 27.80 ± 2% perf-profile.calltrace.cycles-pp.__kmalloc_node_noprof.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
0.65 +27.0 27.62 ± 2% perf-profile.calltrace.cycles-pp.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend
24.25 -12.2 12.02 ± 4% perf-profile.children.cycles-pp.wake_up_q
24.00 -12.1 11.93 ± 4% perf-profile.children.cycles-pp.try_to_wake_up
23.75 -10.9 12.88 perf-profile.children.cycles-pp.common_startup_64
23.75 -10.9 12.88 perf-profile.children.cycles-pp.cpu_startup_entry
23.68 -10.8 12.85 perf-profile.children.cycles-pp.do_idle
23.65 -10.8 12.82 perf-profile.children.cycles-pp.start_secondary
19.65 -8.3 11.33 perf-profile.children.cycles-pp.__schedule
17.14 -6.9 10.28 perf-profile.children.cycles-pp.wq_sleep
16.24 -6.4 9.82 perf-profile.children.cycles-pp.schedule
15.92 -6.2 9.69 perf-profile.children.cycles-pp.schedule_hrtimeout_range_clock
12.46 -6.1 6.32 ± 4% perf-profile.children.cycles-pp.select_task_rq
12.19 -6.0 6.17 ± 4% perf-profile.children.cycles-pp.select_task_rq_fair
9.91 -4.8 5.09 ± 4% perf-profile.children.cycles-pp.select_idle_sibling
9.27 -4.5 4.80 ± 4% perf-profile.children.cycles-pp.select_idle_cpu
12.49 -4.5 8.03 perf-profile.children.cycles-pp.cpuidle_idle_call
11.41 -4.0 7.39 perf-profile.children.cycles-pp.cpuidle_enter_state
11.44 -4.0 7.43 perf-profile.children.cycles-pp.cpuidle_enter
8.17 -4.0 4.18 ± 4% perf-profile.children.cycles-pp.select_idle_core
5.76 -3.5 2.28 ± 4% perf-profile.children.cycles-pp.flush_smp_call_function_queue
5.47 -3.1 2.35 ± 4% perf-profile.children.cycles-pp.__flush_smp_call_function_queue
4.30 -2.4 1.94 ± 4% perf-profile.children.cycles-pp.sched_ttwu_pending
4.08 -2.2 1.90 perf-profile.children.cycles-pp.schedule_idle
3.47 -2.2 1.30 ± 2% perf-profile.children.cycles-pp.store_msg
5.87 -2.1 3.78 perf-profile.children.cycles-pp.__pick_next_task
4.84 -2.0 2.86 perf-profile.children.cycles-pp.try_to_block_task
4.73 -1.9 2.79 perf-profile.children.cycles-pp.dequeue_entities
4.73 -1.9 2.82 perf-profile.children.cycles-pp.dequeue_task_fair
4.04 -1.8 2.26 perf-profile.children.cycles-pp._raw_spin_lock
3.72 -1.7 2.03 ± 4% perf-profile.children.cycles-pp.enqueue_task
2.40 -1.6 0.85 ± 3% perf-profile.children.cycles-pp._copy_to_user
3.39 -1.5 1.86 ± 3% perf-profile.children.cycles-pp.ttwu_do_activate
2.56 -1.5 1.04 ± 5% perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
2.54 -1.5 1.03 ± 5% perf-profile.children.cycles-pp.do_perf_trace_sched_wakeup_template
2.76 -1.4 1.34 ± 4% perf-profile.children.cycles-pp.arch_exit_to_user_mode_prepare
3.75 -1.4 2.34 perf-profile.children.cycles-pp.dequeue_entity
2.32 -1.4 0.95 ± 4% perf-profile.children.cycles-pp.ttwu_queue_wakelist
2.72 -1.3 1.38 ± 2% perf-profile.children.cycles-pp.update_curr
2.38 -1.3 1.06 ± 2% perf-profile.children.cycles-pp.exit_to_user_mode_loop
4.24 -1.2 2.99 perf-profile.children.cycles-pp.pick_next_task_fair
2.21 -1.1 1.15 ± 5% perf-profile.children.cycles-pp.switch_fpu_return
1.76 -1.0 0.73 ± 3% perf-profile.children.cycles-pp.msg_get
1.67 -1.0 0.65 ± 3% perf-profile.children.cycles-pp._raw_spin_lock_irqsave
2.47 -1.0 1.47 ± 3% perf-profile.children.cycles-pp.enqueue_task_fair
2.39 -1.0 1.42 perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
2.29 -1.0 1.32 perf-profile.children.cycles-pp.update_load_avg
1.60 -1.0 0.64 perf-profile.children.cycles-pp.switch_mm_irqs_off
1.51 -0.9 0.57 ± 5% perf-profile.children.cycles-pp.__smp_call_single_queue
1.84 -0.9 0.93 ± 6% perf-profile.children.cycles-pp.wake_affine
1.49 -0.9 0.59 perf-profile.children.cycles-pp.__check_object_size
1.61 -0.9 0.72 ± 5% perf-profile.children.cycles-pp.update_rq_clock_task
1.73 -0.9 0.87 ± 2% perf-profile.children.cycles-pp.update_se
1.51 -0.9 0.65 ± 2% perf-profile.children.cycles-pp.wakeup_preempt
2.00 -0.8 1.15 ± 3% perf-profile.children.cycles-pp.enqueue_entity
1.26 -0.8 0.42 ± 3% perf-profile.children.cycles-pp.__wake_up
1.31 -0.7 0.58 ± 5% perf-profile.children.cycles-pp.set_task_cpu
1.15 -0.7 0.45 ± 3% perf-profile.children.cycles-pp.msg_insert
1.04 -0.7 0.35 ± 5% perf-profile.children.cycles-pp.perf_trace_buf_alloc
1.57 -0.7 0.90 ± 5% perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
1.00 -0.7 0.34 ± 7% perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
0.95 -0.7 0.29 ± 4% perf-profile.children.cycles-pp.llist_reverse_order
1.14 -0.6 0.50 ± 5% perf-profile.children.cycles-pp.migrate_task_rq_fair
1.11 -0.6 0.51 perf-profile.children.cycles-pp.set_next_entity
0.95 -0.6 0.39 ± 2% perf-profile.children.cycles-pp.__update_idle_core
1.04 -0.6 0.49 ± 2% perf-profile.children.cycles-pp.pick_task_fair
1.15 -0.6 0.60 ± 7% perf-profile.children.cycles-pp.task_h_load
0.84 -0.5 0.30 ± 5% perf-profile.children.cycles-pp.call_function_single_prep_ipi
1.06 -0.5 0.52 perf-profile.children.cycles-pp.set_next_task_idle
1.04 -0.5 0.51 perf-profile.children.cycles-pp._find_next_bit
1.28 -0.5 0.75 perf-profile.children.cycles-pp.__switch_to
1.38 -0.5 0.85 perf-profile.children.cycles-pp.update_cfs_rq_load_avg
0.85 -0.5 0.34 ± 3% perf-profile.children.cycles-pp.cpuacct_charge
0.77 ± 4% -0.5 0.25 perf-profile.children.cycles-pp.__bitmap_andnot
0.88 -0.5 0.41 ± 6% perf-profile.children.cycles-pp.update_entity_lag
0.94 -0.5 0.48 perf-profile.children.cycles-pp.prepare_task_switch
0.74 -0.4 0.31 ± 2% perf-profile.children.cycles-pp.check_heap_object
0.75 -0.4 0.32 ± 5% perf-profile.children.cycles-pp.requeue_delayed_entity
0.80 -0.4 0.40 perf-profile.children.cycles-pp.wakeup_preempt_fair
0.55 -0.4 0.16 ± 2% perf-profile.children.cycles-pp.native_sched_clock
0.57 -0.4 0.18 ± 4% perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
0.55 -0.4 0.17 perf-profile.children.cycles-pp.os_xsave
0.58 -0.4 0.20 perf-profile.children.cycles-pp.sched_clock_cpu
1.18 -0.4 0.81 perf-profile.children.cycles-pp.___task_rq_lock
1.39 -0.4 1.01 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
0.51 ± 3% -0.4 0.14 ± 3% perf-profile.children.cycles-pp._copy_from_user
0.56 -0.4 0.19 ± 2% perf-profile.children.cycles-pp.update_rq_clock
0.51 -0.3 0.17 ± 2% perf-profile.children.cycles-pp.sched_clock
0.56 -0.3 0.23 ± 3% perf-profile.children.cycles-pp.simple_inode_init_ts
0.89 -0.3 0.56 perf-profile.children.cycles-pp.put_prev_entity
0.53 ± 3% -0.3 0.21 ± 2% perf-profile.children.cycles-pp.__put_user_4
0.68 ± 12% -0.3 0.36 ± 4% perf-profile.children.cycles-pp.stress_switch_mq
0.52 -0.3 0.20 perf-profile.children.cycles-pp.set_next_task_fair
0.55 -0.3 0.24 ± 5% perf-profile.children.cycles-pp.__switch_to_asm
0.51 -0.3 0.21 ± 3% perf-profile.children.cycles-pp.inode_set_ctime_current
0.56 -0.3 0.26 ± 3% perf-profile.children.cycles-pp.fdget
0.52 -0.3 0.23 ± 3% perf-profile.children.cycles-pp.mm_cid_switch_to
0.54 -0.3 0.25 ± 3% perf-profile.children.cycles-pp.remove_entity_load_avg
0.79 -0.3 0.51 perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.37 -0.3 0.11 ± 7% perf-profile.children.cycles-pp.__resched_curr
0.57 -0.2 0.32 perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.35 -0.2 0.12 ± 4% perf-profile.children.cycles-pp.avg_vruntime
0.62 -0.2 0.39 ± 2% perf-profile.children.cycles-pp.sysvec_call_function_single
0.34 -0.2 0.11 perf-profile.children.cycles-pp.entry_SYSCALL_64
0.58 -0.2 0.36 perf-profile.children.cycles-pp.__enqueue_entity
0.46 -0.2 0.24 perf-profile.children.cycles-pp.__pick_eevdf
0.52 -0.2 0.30 ± 3% perf-profile.children.cycles-pp.perf_tp_event
0.35 -0.2 0.14 ± 3% perf-profile.children.cycles-pp.__wrgsbase_inactive
0.56 -0.2 0.36 perf-profile.children.cycles-pp.__sysvec_call_function_single
0.33 -0.2 0.13 ± 3% perf-profile.children.cycles-pp.__update_load_avg_se
0.31 ± 2% -0.2 0.12 ± 4% perf-profile.children.cycles-pp.__virt_addr_valid
0.58 -0.2 0.40 perf-profile.children.cycles-pp.menu_select
0.32 ± 5% -0.2 0.13 perf-profile.children.cycles-pp.__check_heap_object
0.31 -0.2 0.13 ± 3% perf-profile.children.cycles-pp.place_entity
0.32 -0.2 0.14 perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.36 -0.2 0.18 ± 2% perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.29 -0.2 0.11 perf-profile.children.cycles-pp.syscall_return_via_sysret
0.33 -0.2 0.16 ± 3% perf-profile.children.cycles-pp.do_perf_trace_sched_stat_runtime
0.35 -0.2 0.18 ± 2% perf-profile.children.cycles-pp.ktime_get
0.25 -0.2 0.08 ± 5% perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.37 -0.2 0.20 ± 4% perf-profile.children.cycles-pp.___perf_sw_event
0.23 -0.2 0.07 ± 6% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.22 -0.2 0.07 perf-profile.children.cycles-pp.read_tsc
0.21 -0.1 0.06 ± 7% perf-profile.children.cycles-pp.__rdgsbase_inactive
0.25 -0.1 0.11 ± 7% perf-profile.children.cycles-pp.strnlen
0.32 -0.1 0.18 perf-profile.children.cycles-pp.__dequeue_entity
0.23 ± 2% -0.1 0.10 ± 4% perf-profile.children.cycles-pp.check_stack_object
0.23 ± 2% -0.1 0.09 ± 5% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.53 -0.1 0.40 ± 2% perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.30 ± 3% -0.1 0.17 ± 4% perf-profile.children.cycles-pp.tick_nohz_handler
0.20 -0.1 0.07 perf-profile.children.cycles-pp.__account_obj_stock
0.31 ± 2% -0.1 0.18 ± 2% perf-profile.children.cycles-pp.__hrtimer_run_queues
0.41 ± 2% -0.1 0.29 ± 3% perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.41 -0.1 0.28 ± 2% perf-profile.children.cycles-pp.hrtimer_interrupt
0.28 ± 2% -0.1 0.16 ± 6% perf-profile.children.cycles-pp.update_process_times
0.33 -0.1 0.21 ± 6% perf-profile.children.cycles-pp.attach_entity_load_avg
0.19 ± 2% -0.1 0.07 ± 7% perf-profile.children.cycles-pp.wake_q_add_safe
0.23 ± 2% -0.1 0.11 perf-profile.children.cycles-pp.__kmalloc_cache_noprof
0.18 -0.1 0.07 ± 6% perf-profile.children.cycles-pp.nohz_run_idle_balance
0.56 -0.1 0.46 perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.15 -0.1 0.05 perf-profile.children.cycles-pp._raw_spin_unlock
0.17 -0.1 0.08 perf-profile.children.cycles-pp.security_msg_msg_free
0.15 -0.1 0.06 perf-profile.children.cycles-pp.inode_set_ctime_to_ts
0.16 ± 2% -0.1 0.08 ± 5% perf-profile.children.cycles-pp.dl_server_update
0.13 -0.1 0.06 ± 8% perf-profile.children.cycles-pp.timestamp_truncate
0.15 ± 3% -0.1 0.08 perf-profile.children.cycles-pp.perf_trace_buf_update
0.13 ± 3% -0.1 0.06 perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64_mg
0.13 ± 3% -0.1 0.06 perf-profile.children.cycles-pp.migrate_disable_switch
0.12 ± 6% -0.1 0.05 ± 8% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.11 ± 4% -0.1 0.05 perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.14 -0.1 0.08 ± 5% perf-profile.children.cycles-pp.update_curr_dl_se
0.10 -0.1 0.05 perf-profile.children.cycles-pp.ct_kernel_exit
0.11 -0.1 0.06 perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
0.10 ± 4% -0.0 0.05 perf-profile.children.cycles-pp.__rb_insert_augmented
0.10 -0.0 0.06 ± 8% perf-profile.children.cycles-pp.rest_init
0.10 -0.0 0.06 ± 8% perf-profile.children.cycles-pp.start_kernel
0.10 -0.0 0.06 ± 8% perf-profile.children.cycles-pp.x86_64_start_kernel
0.10 -0.0 0.06 ± 8% perf-profile.children.cycles-pp.x86_64_start_reservations
0.08 ± 10% -0.0 0.04 ± 71% perf-profile.children.cycles-pp.mq_timedreceive
0.15 -0.0 0.11 ± 4% perf-profile.children.cycles-pp.vruntime_eligible
0.13 -0.0 0.09 perf-profile.children.cycles-pp.put_prev_task_fair
0.09 -0.0 0.05 ± 8% perf-profile.children.cycles-pp.native_irq_return_iret
0.08 -0.0 0.05 perf-profile.children.cycles-pp.choose_new_asid
0.13 -0.0 0.11 perf-profile.children.cycles-pp.__irq_exit_rcu
0.07 -0.0 0.05 perf-profile.children.cycles-pp.__set_next_task_fair
0.76 -0.0 0.74 perf-profile.children.cycles-pp.finish_task_switch
0.09 -0.0 0.07 ± 6% perf-profile.children.cycles-pp.propagate_entity_load_avg
0.07 -0.0 0.06 perf-profile.children.cycles-pp.clockevents_program_event
0.10 -0.0 0.09 perf-profile.children.cycles-pp.handle_softirqs
0.07 +0.0 0.08 perf-profile.children.cycles-pp.perf_swevent_event
0.48 +0.0 0.49 perf-profile.children.cycles-pp.process_simple
0.05 +0.0 0.06 ± 7% perf-profile.children.cycles-pp.sched_update_worker
0.05 +0.0 0.07 perf-profile.children.cycles-pp.arch_cpu_idle_enter
0.07 ± 11% +0.0 0.09 ± 5% perf-profile.children.cycles-pp.mq_timedsend
0.15 ± 3% +0.0 0.20 ± 4% perf-profile.children.cycles-pp.x64_sys_call
0.00 +0.1 0.05 perf-profile.children.cycles-pp.__sched_balance_update_blocked_averages
0.00 +0.1 0.05 perf-profile.children.cycles-pp.update_cfs_group
0.00 +0.1 0.06 ± 23% perf-profile.children.cycles-pp.generic_perform_write
0.00 +0.1 0.06 ± 7% perf-profile.children.cycles-pp.detach_tasks
0.00 +0.1 0.06 ± 29% perf-profile.children.cycles-pp.shmem_file_write_iter
0.00 +0.1 0.06 ± 29% perf-profile.children.cycles-pp.vfs_write
0.00 +0.1 0.07 ± 25% perf-profile.children.cycles-pp.ksys_write
0.00 +0.1 0.08 ± 30% perf-profile.children.cycles-pp.record__pushfn
0.04 ± 71% +0.1 0.12 ± 35% perf-profile.children.cycles-pp.perf_mmap__push
0.54 ± 2% +0.1 0.62 ± 7% perf-profile.children.cycles-pp.cmd_record
0.04 ± 71% +0.1 0.13 ± 35% perf-profile.children.cycles-pp.handle_internal_command
0.04 ± 71% +0.1 0.13 ± 35% perf-profile.children.cycles-pp.main
0.04 ± 70% +0.1 0.13 ± 32% perf-profile.children.cycles-pp.record__mmap_read_evlist
0.04 ± 71% +0.1 0.13 ± 35% perf-profile.children.cycles-pp.run_builtin
0.10 ± 4% +0.1 0.20 ± 4% perf-profile.children.cycles-pp.do_perf_trace_sched_switch
0.00 +0.1 0.10 ± 4% perf-profile.children.cycles-pp.ct_idle_enter
0.13 ± 3% +0.2 0.29 ± 4% perf-profile.children.cycles-pp.perf_trace_sched_switch
0.21 ± 6% +0.6 0.78 ± 3% perf-profile.children.cycles-pp.update_sg_lb_stats
0.22 ± 6% +0.6 0.81 ± 4% perf-profile.children.cycles-pp.update_sd_lb_stats
0.22 ± 6% +0.6 0.81 ± 3% perf-profile.children.cycles-pp.sched_balance_find_src_group
0.40 ± 3% +0.7 1.08 ± 4% perf-profile.children.cycles-pp.sched_balance_newidle
0.27 ± 6% +0.7 0.97 ± 4% perf-profile.children.cycles-pp.sched_balance_rq
5.90 +0.9 6.81 perf-profile.children.cycles-pp.intel_idle
35.82 +3.5 39.30 perf-profile.children.cycles-pp.__x64_sys_mq_timedreceive
35.70 +3.5 39.25 perf-profile.children.cycles-pp.do_mq_timedreceive
0.00 +8.8 8.78 ± 3% perf-profile.children.cycles-pp.drain_obj_stock
28.34 +13.4 41.76 perf-profile.children.cycles-pp.do_mq_timedsend
28.52 +13.5 41.99 perf-profile.children.cycles-pp.__x64_sys_mq_timedsend
70.76 +13.7 84.45 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
70.56 +13.8 84.39 perf-profile.children.cycles-pp.do_syscall_64
0.08 +16.2 16.31 ± 4% perf-profile.children.cycles-pp.__refill_obj_stock
3.22 +20.1 23.33 ± 2% perf-profile.children.cycles-pp.kfree
3.01 +20.2 23.25 ± 2% perf-profile.children.cycles-pp.free_msg
2.46 +20.5 23.00 ± 2% perf-profile.children.cycles-pp.__memcg_slab_free_hook
2.31 +25.9 28.25 ± 2% perf-profile.children.cycles-pp.load_msg
1.00 +26.8 27.81 ± 2% perf-profile.children.cycles-pp.__kmalloc_node_noprof
0.68 +27.0 27.65 ± 2% perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
6.79 -3.2 3.63 ± 5% perf-profile.self.cycles-pp.select_idle_core
3.07 -1.7 1.33 ± 4% perf-profile.self.cycles-pp.do_mq_timedreceive
2.90 -1.6 1.31 ± 3% perf-profile.self.cycles-pp.__schedule
2.37 -1.5 0.84 ± 3% perf-profile.self.cycles-pp._copy_to_user
2.73 -1.5 1.22 ± 3% perf-profile.self.cycles-pp.do_mq_timedsend
2.63 -1.4 1.25 ± 3% perf-profile.self.cycles-pp._raw_spin_lock
1.64 -1.0 0.64 ± 3% perf-profile.self.cycles-pp._raw_spin_lock_irqsave
1.46 -0.9 0.55 ± 2% perf-profile.self.cycles-pp.switch_mm_irqs_off
1.54 -0.9 0.67 ± 6% perf-profile.self.cycles-pp.update_rq_clock_task
1.48 -0.9 0.62 ± 3% perf-profile.self.cycles-pp.msg_get
1.36 -0.8 0.58 ± 3% perf-profile.self.cycles-pp.exit_to_user_mode_loop
1.11 -0.7 0.44 ± 4% perf-profile.self.cycles-pp.msg_insert
1.56 -0.7 0.90 ± 6% perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
0.95 -0.7 0.29 ± 5% perf-profile.self.cycles-pp.llist_reverse_order
1.00 -0.7 0.34 ± 7% perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
1.19 -0.6 0.59 ± 3% perf-profile.self.cycles-pp.wq_sleep
0.92 -0.6 0.36 ± 6% perf-profile.self.cycles-pp.do_perf_trace_sched_wakeup_template
0.90 -0.6 0.34 ± 2% perf-profile.self.cycles-pp.__update_idle_core
1.15 -0.5 0.60 ± 7% perf-profile.self.cycles-pp.task_h_load
0.83 -0.5 0.30 ± 5% perf-profile.self.cycles-pp.call_function_single_prep_ipi
0.97 -0.5 0.44 ± 2% perf-profile.self.cycles-pp.dequeue_entities
0.84 -0.5 0.34 ± 3% perf-profile.self.cycles-pp.cpuacct_charge
0.96 -0.5 0.47 ± 4% perf-profile.self.cycles-pp.do_syscall_64
1.23 -0.5 0.74 perf-profile.self.cycles-pp.__switch_to
0.73 ± 4% -0.5 0.24 ± 3% perf-profile.self.cycles-pp.__bitmap_andnot
0.69 -0.5 0.20 ± 4% perf-profile.self.cycles-pp.flush_smp_call_function_queue
0.94 -0.5 0.47 perf-profile.self.cycles-pp._find_next_bit
0.77 -0.4 0.36 ± 6% perf-profile.self.cycles-pp.update_entity_lag
0.76 -0.4 0.36 ± 3% perf-profile.self.cycles-pp.update_load_avg
0.67 -0.4 0.27 ± 5% perf-profile.self.cycles-pp.__smp_call_single_queue
0.70 -0.4 0.31 ± 2% perf-profile.self.cycles-pp.pick_next_task_fair
0.55 -0.4 0.17 ± 2% perf-profile.self.cycles-pp.os_xsave
0.56 -0.4 0.18 ± 4% perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
0.63 -0.4 0.25 perf-profile.self.cycles-pp.switch_fpu_return
1.38 -0.4 1.01 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.53 -0.4 0.16 perf-profile.self.cycles-pp.native_sched_clock
0.69 -0.4 0.33 ± 5% perf-profile.self.cycles-pp.wake_affine
0.67 -0.4 0.30 ± 2% perf-profile.self.cycles-pp.kfree
0.75 -0.4 0.39 ± 3% perf-profile.self.cycles-pp.update_curr
0.67 -0.4 0.31 ± 5% perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.53 -0.4 0.18 ± 2% perf-profile.self.cycles-pp.arch_exit_to_user_mode_prepare
0.49 ± 2% -0.4 0.14 ± 3% perf-profile.self.cycles-pp._copy_from_user
0.63 -0.3 0.28 ± 3% perf-profile.self.cycles-pp.schedule_hrtimeout_range_clock
0.58 -0.3 0.23 ± 4% perf-profile.self.cycles-pp.select_idle_sibling
0.58 -0.3 0.24 ± 7% perf-profile.self.cycles-pp.migrate_task_rq_fair
0.64 -0.3 0.31 perf-profile.self.cycles-pp.prepare_task_switch
0.82 -0.3 0.49 ± 3% perf-profile.self.cycles-pp.select_idle_cpu
0.52 ± 3% -0.3 0.21 ± 2% perf-profile.self.cycles-pp.__put_user_4
0.63 ± 11% -0.3 0.32 ± 5% perf-profile.self.cycles-pp.stress_switch_mq
0.54 -0.3 0.24 ± 5% perf-profile.self.cycles-pp.__switch_to_asm
0.54 -0.3 0.25 ± 3% perf-profile.self.cycles-pp.fdget
0.51 -0.3 0.22 ± 4% perf-profile.self.cycles-pp.mm_cid_switch_to
0.44 -0.3 0.15 ± 6% perf-profile.self.cycles-pp.select_task_rq_fair
0.43 -0.3 0.15 ± 5% perf-profile.self.cycles-pp.sched_ttwu_pending
0.49 -0.3 0.23 ± 6% perf-profile.self.cycles-pp.enqueue_task
0.59 -0.2 0.34 ± 2% perf-profile.self.cycles-pp.try_to_wake_up
0.73 -0.2 0.48 perf-profile.self.cycles-pp.update_cfs_rq_load_avg
0.35 -0.2 0.11 ± 4% perf-profile.self.cycles-pp.__resched_curr
0.40 -0.2 0.16 ± 2% perf-profile.self.cycles-pp.__pick_next_task
0.36 -0.2 0.12 ± 3% perf-profile.self.cycles-pp.cpuidle_idle_call
0.34 -0.2 0.11 perf-profile.self.cycles-pp.entry_SYSCALL_64
0.57 -0.2 0.36 perf-profile.self.cycles-pp.__enqueue_entity
0.51 -0.2 0.31 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.34 -0.2 0.14 ± 5% perf-profile.self.cycles-pp.wakeup_preempt
0.34 -0.2 0.14 ± 3% perf-profile.self.cycles-pp.__wrgsbase_inactive
0.39 -0.2 0.20 ± 2% perf-profile.self.cycles-pp.do_idle
0.31 ± 5% -0.2 0.11 perf-profile.self.cycles-pp.__check_heap_object
0.37 -0.2 0.17 ± 2% perf-profile.self.cycles-pp.check_heap_object
0.51 -0.2 0.32 perf-profile.self.cycles-pp.schedule
0.36 -0.2 0.18 ± 2% perf-profile.self.cycles-pp.__pick_eevdf
0.29 ± 2% -0.2 0.11 ± 4% perf-profile.self.cycles-pp.__virt_addr_valid
0.61 -0.2 0.43 perf-profile.self.cycles-pp.dequeue_entity
0.30 -0.2 0.12 perf-profile.self.cycles-pp.__update_load_avg_se
0.27 -0.2 0.10 ± 4% perf-profile.self.cycles-pp.syscall_return_via_sysret
0.25 -0.2 0.08 ± 5% perf-profile.self.cycles-pp.__check_object_size
0.26 -0.2 0.11 ± 4% perf-profile.self.cycles-pp.place_entity
0.45 -0.2 0.30 ± 3% perf-profile.self.cycles-pp.enqueue_entity
0.44 -0.2 0.29 ± 5% perf-profile.self.cycles-pp.enqueue_task_fair
0.24 -0.2 0.09 perf-profile.self.cycles-pp.wake_up_q
0.24 -0.1 0.09 ± 5% perf-profile.self.cycles-pp.___perf_sw_event
0.22 ± 2% -0.1 0.07 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.32 -0.1 0.18 ± 2% perf-profile.self.cycles-pp.dequeue_task_fair
0.21 ± 2% -0.1 0.06 ± 7% perf-profile.self.cycles-pp.read_tsc
0.20 -0.1 0.06 perf-profile.self.cycles-pp.__rdgsbase_inactive
0.36 -0.1 0.22 ± 4% perf-profile.self.cycles-pp.perf_tp_event
0.28 -0.1 0.14 perf-profile.self.cycles-pp.__kmalloc_node_noprof
0.24 -0.1 0.11 ± 4% perf-profile.self.cycles-pp.strnlen
0.21 -0.1 0.08 ± 6% perf-profile.self.cycles-pp.load_msg
0.33 -0.1 0.20 ± 4% perf-profile.self.cycles-pp.attach_entity_load_avg
0.44 -0.1 0.33 perf-profile.self.cycles-pp.menu_select
0.41 -0.1 0.29 perf-profile.self.cycles-pp.update_se
0.17 ± 2% -0.1 0.06 ± 8% perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.27 -0.1 0.15 ± 6% perf-profile.self.cycles-pp.select_task_rq
0.23 ± 2% -0.1 0.11 ± 4% perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.19 -0.1 0.08 ± 6% perf-profile.self.cycles-pp.update_rq_clock
0.20 ± 2% -0.1 0.09 perf-profile.self.cycles-pp.check_stack_object
0.18 ± 2% -0.1 0.06 ± 7% perf-profile.self.cycles-pp.wake_q_add_safe
0.18 -0.1 0.07 perf-profile.self.cycles-pp.__account_obj_stock
0.21 ± 2% -0.1 0.10 ± 4% perf-profile.self.cycles-pp.__kmalloc_cache_noprof
0.24 -0.1 0.14 perf-profile.self.cycles-pp.__dequeue_entity
0.19 -0.1 0.10 ± 4% perf-profile.self.cycles-pp.pick_task_fair
0.14 ± 3% -0.1 0.05 perf-profile.self.cycles-pp.inode_set_ctime_current
0.16 ± 3% -0.1 0.06 ± 7% perf-profile.self.cycles-pp.nohz_run_idle_balance
0.15 -0.1 0.06 perf-profile.self.cycles-pp.avg_vruntime
0.16 -0.1 0.07 perf-profile.self.cycles-pp.schedule_idle
0.15 -0.1 0.08 ± 5% perf-profile.self.cycles-pp.dl_server_update
0.15 -0.1 0.08 ± 5% perf-profile.self.cycles-pp.raw_spin_rq_lock_nested
0.12 ± 4% -0.1 0.05 perf-profile.self.cycles-pp.__x64_sys_mq_timedreceive
0.12 -0.1 0.05 ± 8% perf-profile.self.cycles-pp.migrate_disable_switch
0.13 -0.1 0.07 ± 7% perf-profile.self.cycles-pp.___task_rq_lock
0.09 ± 5% -0.1 0.03 ± 70% perf-profile.self.cycles-pp.inode_set_ctime_to_ts
0.22 -0.1 0.16 perf-profile.self.cycles-pp.cpuidle_enter_state
0.12 -0.1 0.06 perf-profile.self.cycles-pp.store_msg
0.12 -0.1 0.06 perf-profile.self.cycles-pp.wakeup_preempt_fair
0.11 ± 4% -0.1 0.05 ± 8% perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64_mg
0.11 -0.1 0.05 perf-profile.self.cycles-pp.timestamp_truncate
0.12 ± 4% -0.1 0.06 ± 7% perf-profile.self.cycles-pp.do_perf_trace_sched_stat_runtime
0.11 ± 4% -0.1 0.06 ± 8% perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
0.13 ± 3% -0.0 0.08 perf-profile.self.cycles-pp.update_curr_dl_se
0.15 -0.0 0.11 ± 4% perf-profile.self.cycles-pp.sched_balance_newidle
0.09 -0.0 0.05 ± 8% perf-profile.self.cycles-pp.native_irq_return_iret
0.14 ± 3% -0.0 0.10 ± 4% perf-profile.self.cycles-pp.vruntime_eligible
0.14 ± 3% -0.0 0.11 perf-profile.self.cycles-pp.ktime_get
0.07 ± 7% -0.0 0.03 ± 70% perf-profile.self.cycles-pp.__set_next_task_fair
0.15 ± 3% -0.0 0.13 perf-profile.self.cycles-pp.put_prev_entity
0.11 ± 4% +0.0 0.12 ± 3% perf-profile.self.cycles-pp.set_next_task_idle
0.06 +0.0 0.08 ± 6% perf-profile.self.cycles-pp.perf_swevent_event
0.07 ± 7% +0.0 0.09 ± 10% perf-profile.self.cycles-pp.mq_timedsend
0.00 +0.1 0.05 perf-profile.self.cycles-pp.ct_idle_enter
0.00 +0.1 0.05 perf-profile.self.cycles-pp.perf_trace_sched_switch
0.00 +0.1 0.06 perf-profile.self.cycles-pp.sched_update_worker
0.12 +0.1 0.18 ± 6% perf-profile.self.cycles-pp.x64_sys_call
0.38 ± 2% +0.1 0.46 perf-profile.self.cycles-pp.finish_task_switch
0.08 ± 5% +0.1 0.20 ± 6% perf-profile.self.cycles-pp.do_perf_trace_sched_switch
0.18 ± 5% +0.5 0.71 ± 3% perf-profile.self.cycles-pp.update_sg_lb_stats
5.89 +0.9 6.81 perf-profile.self.cycles-pp.intel_idle
0.07 +7.4 7.48 ± 4% perf-profile.self.cycles-pp.__refill_obj_stock
0.00 +8.7 8.70 ± 3% perf-profile.self.cycles-pp.drain_obj_stock
2.25 +12.4 14.61 perf-profile.self.cycles-pp.__memcg_slab_free_hook
0.53 +19.0 19.48 perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
***************************************************************************************************
=========================================================================================
compiler/cpufreq_governor/kconfig/nr_task/rootfs/runtime/tbox_group/test/testcase:
gcc-14/performance/x86_64-rhel-9.4/100%/debian-13-x86_64-20250902.cgz/300s/lkp-icl-2sp9/spawn/unixbench
commit:
8285917d6f ("mm: memcontrol: prepare for reparenting non-hierarchical stats")
01b9da291c ("mm: memcontrol: convert objcg to be per-memcg per-node type")
8285917d6f383aef 01b9da291c4969354807b52956f
---------------- ---------------------------
%stddev %change %stddev
\ | \
12631 -3.8% 12146 unixbench.score
159157 -3.8% 153044 unixbench.throughput
19807234 -4.8% 18852799 unixbench.time.involuntary_context_switches
1.413e+09 -3.9% 1.357e+09 unixbench.time.minor_page_faults
8608 +2.8% 8852 unixbench.time.system_time
6616 -2.4% 6458 unixbench.time.user_time
94616535 -4.1% 90778327 unixbench.time.voluntary_context_switches
52575303 -3.9% 50543226 unixbench.workload
1110579 +8.8% 1208410 ± 2% meminfo.AnonPages
210425 -12.6% 183972 ± 5% meminfo.DirectMap4k
0.05 ± 4% +0.0 0.06 ± 2% mpstat.cpu.all.iowait%
1.51 +0.4 1.93 mpstat.cpu.all.soft%
557785 ± 44% +49.6% 834357 ± 25% numa-meminfo.node0.AnonPages.max
736507 ± 5% +8.1% 796466 ± 3% numa-meminfo.node1.Mapped
6.802e+08 -5.2% 6.446e+08 numa-numastat.node0.local_node
6.804e+08 -5.3% 6.445e+08 numa-numastat.node0.numa_hit
0.04 +7.1% 0.05 perf-sched.sch_delay.avg.ms.[unknown].[unknown].[unknown].[unknown].[unknown]
0.04 +7.1% 0.05 perf-sched.total_sch_delay.average.ms
23.64 -0.9 22.72 turbostat.C1%
22.97 ± 10% -5.2 17.74 ± 5% turbostat.PKG_%
6.804e+08 -5.3% 6.445e+08 numa-vmstat.node0.numa_hit
6.802e+08 -5.2% 6.446e+08 numa-vmstat.node0.numa_local
184186 ± 5% +8.2% 199205 ± 3% numa-vmstat.node1.nr_mapped
0.03 ± 67% +680.5% 0.22 ± 62% vmstat.procs.b
564221 -3.9% 542439 vmstat.system.cs
380977 -2.6% 371055 vmstat.system.in
769143 +1.6% 781758 proc-vmstat.nr_active_anon
277665 +8.8% 302174 ± 2% proc-vmstat.nr_anon_pages
30969 -3.5% 29900 proc-vmstat.nr_page_table_pages
491673 -2.4% 479782 proc-vmstat.nr_shmem
57770 -1.0% 57209 proc-vmstat.nr_slab_unreclaimable
769143 +1.6% 781758 proc-vmstat.nr_zone_active_anon
252183 ± 14% +35.3% 341208 ± 18% proc-vmstat.numa_hint_faults
1.254e+09 -3.8% 1.207e+09 proc-vmstat.numa_hit
1.254e+09 -3.8% 1.207e+09 proc-vmstat.numa_local
317222 ± 15% +38.4% 438924 ± 20% proc-vmstat.numa_pte_updates
1.338e+09 -3.8% 1.287e+09 proc-vmstat.pgalloc_normal
1.415e+09 -3.9% 1.359e+09 proc-vmstat.pgfault
1.337e+09 -3.8% 1.286e+09 proc-vmstat.pgfree
1.598e+08 -4.1% 1.533e+08 proc-vmstat.pgreuse
482323 ± 4% +17.7% 567902 ± 7% sched_debug.cfs_rq:/.left_deadline.avg
1301805 ± 3% +10.2% 1435059 ± 3% sched_debug.cfs_rq:/.left_deadline.stddev
482320 ± 4% +17.7% 567899 ± 7% sched_debug.cfs_rq:/.left_vruntime.avg
1301796 ± 3% +10.2% 1435050 ± 3% sched_debug.cfs_rq:/.left_vruntime.stddev
135389 ± 2% +23.4% 167033 ± 6% sched_debug.cfs_rq:/.load.avg
327477 +12.0% 366681 ± 5% sched_debug.cfs_rq:/.load.stddev
482320 ± 4% +17.7% 567899 ± 7% sched_debug.cfs_rq:/.right_vruntime.avg
1301796 ± 3% +10.2% 1435050 ± 3% sched_debug.cfs_rq:/.right_vruntime.stddev
0.02 ± 9% +262.4% 0.06 ± 56% sched_debug.cfs_rq:/.spread.avg
0.92 ± 28% +264.8% 3.37 ± 57% sched_debug.cfs_rq:/.spread.max
0.12 ± 21% +260.0% 0.43 ± 57% sched_debug.cfs_rq:/.spread.stddev
405099 ± 2% +6.3% 430541 ± 4% sched_debug.cpu.avg_idle.max
1430164 -4.9% 1359536 sched_debug.cpu.nr_switches.max
90912 ± 2% -12.6% 79477 ± 9% sched_debug.cpu.nr_switches.stddev
1.418e+10 -2.4% 1.385e+10 perf-stat.i.branch-instructions
88090016 -2.4% 86013654 perf-stat.i.branch-misses
27.32 +0.5 27.79 perf-stat.i.cache-miss-rate%
3.981e+08 -2.5% 3.882e+08 perf-stat.i.cache-misses
1.414e+09 -3.9% 1.359e+09 perf-stat.i.cache-references
570181 -3.7% 548874 perf-stat.i.context-switches
2.41 +4.0% 2.50 perf-stat.i.cpi
1.81e+11 +1.2% 1.833e+11 perf-stat.i.cpu-cycles
109140 -4.5% 104200 perf-stat.i.cpu-migrations
7.268e+10 -2.4% 7.092e+10 perf-stat.i.instructions
143.18 -3.7% 137.84 perf-stat.i.metric.K/sec
4249150 -3.6% 4095509 perf-stat.i.minor-faults
4249372 -3.6% 4095721 perf-stat.i.page-faults
27.38 +0.5 27.93 perf-stat.overall.cache-miss-rate%
2.41 +4.1% 2.51 perf-stat.overall.cpi
456.68 +3.6% 473.23 perf-stat.overall.cycles-between-cache-misses
0.41 -3.9% 0.40 perf-stat.overall.ipc
468146 +1.0% 472826 perf-stat.overall.path-length
1.426e+10 -2.6% 1.39e+10 perf-stat.ps.branch-instructions
85809605 -2.2% 83901451 perf-stat.ps.branch-misses
3.912e+08 -2.3% 3.822e+08 perf-stat.ps.cache-misses
1.429e+09 -4.2% 1.368e+09 perf-stat.ps.cache-references
567141 -3.8% 545697 perf-stat.ps.context-switches
106475 -4.1% 102109 perf-stat.ps.cpu-migrations
7.401e+10 -2.7% 7.198e+10 perf-stat.ps.instructions
4240337 -3.8% 4081092 perf-stat.ps.minor-faults
4240545 -3.8% 4081293 perf-stat.ps.page-faults
2.461e+13 -2.9% 2.39e+13 perf-stat.total.instructions
18.76 -1.1 17.70 perf-profile.calltrace.cycles-pp.asm_exc_page_fault
17.73 -1.0 16.72 perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault
17.66 -1.0 16.66 perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
16.03 ± 2% -0.9 15.10 ± 2% perf-profile.calltrace.cycles-pp.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
15.61 ± 2% -0.9 14.70 ± 2% perf-profile.calltrace.cycles-pp.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
10.73 ± 2% -0.8 9.96 ± 3% perf-profile.calltrace.cycles-pp.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
10.75 ± 2% -0.8 9.98 ± 3% perf-profile.calltrace.cycles-pp.do_fault.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
10.38 ± 3% -0.7 9.64 ± 3% perf-profile.calltrace.cycles-pp.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault.handle_mm_fault
1.53 -0.6 0.92 perf-profile.calltrace.cycles-pp.asm_sysvec_call_function_single.pv_native_safe_halt.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter
3.01 -0.1 2.86 perf-profile.calltrace.cycles-pp.common_startup_64
2.96 -0.1 2.81 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
2.96 -0.1 2.82 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
2.96 -0.1 2.82 perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
2.82 -0.1 2.70 perf-profile.calltrace.cycles-pp.mm_init.dup_mm.copy_process.kernel_clone.__do_sys_clone
1.72 ± 2% -0.1 1.60 ± 2% perf-profile.calltrace.cycles-pp.set_pte_range.filemap_map_pages.do_read_fault.do_fault.__handle_mm_fault
1.41 ± 2% -0.1 1.29 ± 3% perf-profile.calltrace.cycles-pp.folio_add_file_rmap_ptes.set_pte_range.filemap_map_pages.do_read_fault.do_fault
1.83 -0.1 1.72 perf-profile.calltrace.cycles-pp.schedule.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.82 -0.1 1.71 perf-profile.calltrace.cycles-pp.__schedule.schedule.do_wait.kernel_wait4.do_syscall_64
2.13 -0.1 2.04 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
1.96 -0.1 1.87 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
1.94 -0.1 1.85 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
1.29 -0.1 1.20 perf-profile.calltrace.cycles-pp.do_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault.exc_page_fault
1.87 -0.1 1.79 perf-profile.calltrace.cycles-pp.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
1.86 -0.1 1.78 perf-profile.calltrace.cycles-pp.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call
1.86 -0.1 1.78 perf-profile.calltrace.cycles-pp.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state.cpuidle_enter
1.43 -0.1 1.35 perf-profile.calltrace.cycles-pp.select_task_rq_fair.wake_up_new_task.kernel_clone.__do_sys_clone.do_syscall_64
2.09 -0.1 2.01 perf-profile.calltrace.cycles-pp.schedule_tail.ret_from_fork.ret_from_fork_asm
1.08 ± 2% -0.1 1.00 ± 2% perf-profile.calltrace.cycles-pp.__percpu_counter_init_many.mm_init.dup_mm.copy_process.kernel_clone
1.26 -0.1 1.20 perf-profile.calltrace.cycles-pp.sched_balance_find_dst_group.select_task_rq_fair.wake_up_new_task.kernel_clone.__do_sys_clone
1.16 -0.1 1.10 perf-profile.calltrace.cycles-pp.update_sg_wakeup_stats.sched_balance_find_dst_group.select_task_rq_fair.wake_up_new_task.kernel_clone
0.88 ± 2% -0.1 0.82 perf-profile.calltrace.cycles-pp.lru_add_drain.do_wp_page.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
0.87 -0.1 0.82 perf-profile.calltrace.cycles-pp.lru_add_drain_cpu.lru_add_drain.do_wp_page.__handle_mm_fault.handle_mm_fault
0.91 ± 2% -0.1 0.86 perf-profile.calltrace.cycles-pp.pcpu_alloc_noprof.__percpu_counter_init_many.mm_init.dup_mm.copy_process
0.86 -0.1 0.80 perf-profile.calltrace.cycles-pp.folio_batch_move_lru.lru_add_drain_cpu.lru_add_drain.do_wp_page.__handle_mm_fault
1.10 -0.1 1.05 perf-profile.calltrace.cycles-pp.lock_vma_under_rcu.do_user_addr_fault.exc_page_fault.asm_exc_page_fault
1.25 -0.1 1.20 perf-profile.calltrace.cycles-pp.__vma_start_write.dup_mmap.dup_mm.copy_process.kernel_clone
2.11 -0.1 2.06 perf-profile.calltrace.cycles-pp.wake_up_new_task.kernel_clone.__do_sys_clone.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.72 -0.1 0.67 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.common_startup_64
0.87 -0.0 0.82 perf-profile.calltrace.cycles-pp.__wp_page_copy_user.wp_page_copy.__handle_mm_fault.handle_mm_fault.do_user_addr_fault
1.14 -0.0 1.10 perf-profile.calltrace.cycles-pp.alloc_thread_stack_node.dup_task_struct.copy_process.kernel_clone.__do_sys_clone
0.85 -0.0 0.80 ± 2% perf-profile.calltrace.cycles-pp.copy_mc_enhanced_fast_string.__wp_page_copy_user.wp_page_copy.__handle_mm_fault.handle_mm_fault
0.79 -0.0 0.74 perf-profile.calltrace.cycles-pp.__mmdrop.finish_task_switch.__schedule.schedule.do_wait
0.71 -0.0 0.66 perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
0.88 -0.0 0.83 perf-profile.calltrace.cycles-pp.finish_task_switch.__schedule.schedule.do_wait.kernel_wait4
1.00 -0.0 0.96 perf-profile.calltrace.cycles-pp.__mmdrop.finish_task_switch.schedule_tail.ret_from_fork.ret_from_fork_asm
0.81 -0.0 0.76 perf-profile.calltrace.cycles-pp.sync_regs.asm_exc_page_fault
0.68 -0.0 0.64 perf-profile.calltrace.cycles-pp.copy_present_ptes.copy_pte_range.copy_p4d_range.copy_page_range.dup_mmap
2.05 -0.0 2.01 perf-profile.calltrace.cycles-pp.copy_pte_range.copy_p4d_range.copy_page_range.dup_mmap.dup_mm
0.82 -0.0 0.78 perf-profile.calltrace.cycles-pp.__vmalloc_node_noprof.alloc_thread_stack_node.dup_task_struct.copy_process.kernel_clone
0.82 -0.0 0.78 perf-profile.calltrace.cycles-pp.__vmalloc_node_range_noprof.__vmalloc_node_noprof.alloc_thread_stack_node.dup_task_struct.copy_process
3.37 -0.0 3.33 perf-profile.calltrace.cycles-pp.ret_from_fork_asm
0.65 -0.0 0.61 perf-profile.calltrace.cycles-pp.sched_move_task.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
1.10 -0.0 1.07 perf-profile.calltrace.cycles-pp.finish_task_switch.schedule_tail.ret_from_fork.ret_from_fork_asm
0.61 -0.0 0.58 perf-profile.calltrace.cycles-pp.__vma_start_write.free_pgtables.exit_mmap.__mmput.exit_mm
0.56 -0.0 0.52 perf-profile.calltrace.cycles-pp.process_one_work.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
0.57 -0.0 0.54 perf-profile.calltrace.cycles-pp.__wake_up_sync_key.do_notify_parent.exit_notify.do_exit.do_group_exit
0.55 -0.0 0.51 perf-profile.calltrace.cycles-pp.__wake_up_common.__wake_up_sync_key.do_notify_parent.exit_notify.do_exit
0.88 -0.0 0.85 perf-profile.calltrace.cycles-pp.asm_exc_page_fault.__put_user_4.schedule_tail.ret_from_fork.ret_from_fork_asm
0.61 -0.0 0.58 perf-profile.calltrace.cycles-pp.__vma_start_exclude_readers.__vma_start_write.dup_mmap.dup_mm.copy_process
0.61 -0.0 0.58 perf-profile.calltrace.cycles-pp.worker_thread.kthread.ret_from_fork.ret_from_fork_asm
0.66 -0.0 0.63 perf-profile.calltrace.cycles-pp.do_notify_parent.exit_notify.do_exit.do_group_exit.__x64_sys_exit_group
0.71 -0.0 0.68 perf-profile.calltrace.cycles-pp.exc_page_fault.asm_exc_page_fault.__put_user_4.schedule_tail.ret_from_fork
1.67 -0.0 1.65 perf-profile.calltrace.cycles-pp.dup_task_struct.copy_process.kernel_clone.__do_sys_clone.do_syscall_64
0.54 -0.0 0.52 perf-profile.calltrace.cycles-pp.get_page_from_freelist.__alloc_frozen_pages_noprof.alloc_pages_mpol.alloc_pages_noprof.pte_alloc_one
0.76 -0.0 0.73 perf-profile.calltrace.cycles-pp.__put_user_4.schedule_tail.ret_from_fork.ret_from_fork_asm
0.55 -0.0 0.52 perf-profile.calltrace.cycles-pp.__vmalloc_area_node.__vmalloc_node_range_noprof.__vmalloc_node_noprof.alloc_thread_stack_node.dup_task_struct
0.54 -0.0 0.51 perf-profile.calltrace.cycles-pp.asm_sysvec_reschedule_ipi.pv_native_safe_halt.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter
0.81 -0.0 0.78 perf-profile.calltrace.cycles-pp.pv_native_safe_halt.acpi_safe_halt.acpi_idle_do_entry.acpi_idle_enter.cpuidle_enter_state
0.70 -0.0 0.67 perf-profile.calltrace.cycles-pp.do_user_addr_fault.exc_page_fault.asm_exc_page_fault.__put_user_4.schedule_tail
0.96 -0.0 0.94 perf-profile.calltrace.cycles-pp.alloc_pages_noprof.pte_alloc_one.__pte_alloc.copy_pte_range.copy_p4d_range
0.67 -0.0 0.66 perf-profile.calltrace.cycles-pp.__alloc_frozen_pages_noprof.alloc_pages_mpol.alloc_pages_noprof.__pmd_alloc.copy_p4d_range
1.07 +0.0 1.09 perf-profile.calltrace.cycles-pp.__pte_alloc.copy_pte_range.copy_p4d_range.copy_page_range.dup_mmap
2.04 +0.0 2.07 perf-profile.calltrace.cycles-pp.unlink_anon_vmas.free_pgtables.exit_mmap.__mmput.exit_mm
0.74 +0.0 0.77 perf-profile.calltrace.cycles-pp.on_each_cpu_cond_mask.flush_tlb_mm_range.dup_mmap.dup_mm.copy_process
0.63 +0.0 0.67 perf-profile.calltrace.cycles-pp.__pud_alloc.copy_p4d_range.copy_page_range.dup_mmap.dup_mm
0.76 +0.0 0.79 perf-profile.calltrace.cycles-pp.flush_tlb_mm_range.dup_mmap.dup_mm.copy_process.kernel_clone
0.73 +0.0 0.76 perf-profile.calltrace.cycles-pp.smp_call_function_many_cond.on_each_cpu_cond_mask.flush_tlb_mm_range.dup_mmap.dup_mm
1.10 +0.0 1.15 perf-profile.calltrace.cycles-pp.tear_down_vmas.exit_mmap.__mmput.exit_mm.do_exit
1.06 +0.1 1.12 perf-profile.calltrace.cycles-pp.kthread.ret_from_fork.ret_from_fork_asm
0.76 +0.1 0.84 perf-profile.calltrace.cycles-pp.kmem_cache_free.tear_down_vmas.exit_mmap.__mmput.exit_mm
0.63 +0.1 0.75 perf-profile.calltrace.cycles-pp.kmem_cache_free.unlink_anon_vmas.free_pgtables.exit_mmap.__mmput
0.72 +0.1 0.84 perf-profile.calltrace.cycles-pp.vm_area_dup.dup_mmap.dup_mm.copy_process.kernel_clone
0.64 +0.1 0.77 perf-profile.calltrace.cycles-pp.kmem_cache_alloc_noprof.vm_area_dup.dup_mmap.dup_mm.copy_process
4.77 ± 2% +0.4 5.18 ± 2% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.release_task.wait_task_zombie.__do_wait.do_wait
4.43 ± 2% +0.4 4.84 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.release_task.wait_task_zombie.__do_wait
0.20 ±141% +0.4 0.62 ± 11% perf-profile.calltrace.cycles-pp.queue_event.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events
0.20 ±141% +0.4 0.63 ± 11% perf-profile.calltrace.cycles-pp.ordered_events__queue.process_simple.reader__read_event.perf_session__process_events.record__finish_output
0.21 ±141% +0.4 0.64 ± 11% perf-profile.calltrace.cycles-pp.process_simple.reader__read_event.perf_session__process_events.record__finish_output.cmd_record
4.85 ± 2% +0.5 5.30 ± 2% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.copy_process.kernel_clone.__do_sys_clone.do_syscall_64
4.53 ± 2% +0.5 4.99 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.copy_process.kernel_clone.__do_sys_clone
5.04 ± 2% +0.5 5.53 ± 2% perf-profile.calltrace.cycles-pp.queued_write_lock_slowpath.exit_notify.do_exit.do_group_exit.__x64_sys_exit_group
4.71 ± 2% +0.5 5.21 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_write_lock_slowpath.exit_notify.do_exit.do_group_exit
0.00 +0.5 0.52 perf-profile.calltrace.cycles-pp.rcu_do_batch.rcu_core.handle_softirqs.run_ksoftirqd.smpboot_thread_fn
0.00 +0.5 0.52 perf-profile.calltrace.cycles-pp.rcu_core.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread
0.00 +0.5 0.52 perf-profile.calltrace.cycles-pp.__memcg_slab_free_hook.kmem_cache_free.tear_down_vmas.exit_mmap.__mmput
6.04 ± 2% +0.5 6.56 ± 2% perf-profile.calltrace.cycles-pp.wait_task_zombie.__do_wait.do_wait.kernel_wait4.do_syscall_64
0.00 +0.5 0.52 perf-profile.calltrace.cycles-pp.handle_softirqs.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork
0.00 +0.5 0.52 perf-profile.calltrace.cycles-pp.run_ksoftirqd.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
0.00 +0.5 0.53 perf-profile.calltrace.cycles-pp.__memcg_slab_free_hook.kmem_cache_free.unlink_anon_vmas.free_pgtables.exit_mmap
5.87 ± 2% +0.5 6.40 ± 2% perf-profile.calltrace.cycles-pp.release_task.wait_task_zombie.__do_wait.do_wait.kernel_wait4
0.00 +0.5 0.54 perf-profile.calltrace.cycles-pp.smpboot_thread_fn.kthread.ret_from_fork.ret_from_fork_asm
5.98 ± 2% +0.5 6.52 ± 2% perf-profile.calltrace.cycles-pp.exit_notify.do_exit.do_group_exit.__x64_sys_exit_group.x64_sys_call
8.83 ± 2% +0.8 9.60 ± 2% perf-profile.calltrace.cycles-pp.queued_read_lock_slowpath.__do_wait.do_wait.kernel_wait4.do_syscall_64
8.28 ± 2% +0.8 9.06 ± 2% perf-profile.calltrace.cycles-pp.native_queued_spin_lock_slowpath.queued_read_lock_slowpath.__do_wait.do_wait.kernel_wait4
17.04 ± 2% +1.2 18.22 perf-profile.calltrace.cycles-pp.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
16.99 ± 2% +1.2 18.17 perf-profile.calltrace.cycles-pp.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
15.10 ± 2% +1.3 16.39 ± 2% perf-profile.calltrace.cycles-pp.__do_wait.do_wait.kernel_wait4.do_syscall_64.entry_SYSCALL_64_after_hwframe
71.62 +1.3 72.94 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
71.65 +1.3 72.97 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
20.02 -1.1 18.89 perf-profile.children.cycles-pp.asm_exc_page_fault
18.66 -1.0 17.61 perf-profile.children.cycles-pp.exc_page_fault
18.57 -1.0 17.52 perf-profile.children.cycles-pp.do_user_addr_fault
16.63 ± 2% -1.0 15.67 ± 2% perf-profile.children.cycles-pp.handle_mm_fault
16.14 ± 2% -0.9 15.22 ± 2% perf-profile.children.cycles-pp.__handle_mm_fault
10.73 ± 3% -0.8 9.96 ± 3% perf-profile.children.cycles-pp.do_read_fault
10.75 ± 3% -0.8 9.98 ± 3% perf-profile.children.cycles-pp.do_fault
10.44 ± 3% -0.7 9.70 ± 3% perf-profile.children.cycles-pp.filemap_map_pages
3.64 -0.2 3.44 perf-profile.children.cycles-pp.__schedule
2.34 -0.2 2.14 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
3.01 -0.2 2.86 perf-profile.children.cycles-pp.do_idle
3.01 -0.1 2.86 perf-profile.children.cycles-pp.common_startup_64
3.01 -0.1 2.86 perf-profile.children.cycles-pp.cpu_startup_entry
2.96 -0.1 2.82 perf-profile.children.cycles-pp.start_secondary
2.83 -0.1 2.71 perf-profile.children.cycles-pp.mm_init
1.76 -0.1 1.64 ± 2% perf-profile.children.cycles-pp.set_pte_range
1.44 ± 2% -0.1 1.32 ± 2% perf-profile.children.cycles-pp.folio_add_file_rmap_ptes
1.96 -0.1 1.85 perf-profile.children.cycles-pp.schedule
2.22 -0.1 2.11 ± 2% perf-profile.children.cycles-pp.pcpu_alloc_noprof
2.17 -0.1 2.07 perf-profile.children.cycles-pp.cpuidle_idle_call
2.25 -0.1 2.15 perf-profile.children.cycles-pp.get_page_from_freelist
1.81 -0.1 1.71 perf-profile.children.cycles-pp.__mmdrop
2.20 -0.1 2.10 perf-profile.children.cycles-pp.finish_task_switch
1.98 -0.1 1.88 perf-profile.children.cycles-pp.cpuidle_enter_state
2.00 -0.1 1.90 perf-profile.children.cycles-pp.cpuidle_enter
1.91 -0.1 1.82 perf-profile.children.cycles-pp.acpi_idle_enter
1.65 -0.1 1.56 perf-profile.children.cycles-pp.select_task_rq_fair
1.87 -0.1 1.78 perf-profile.children.cycles-pp.__vma_start_write
1.89 -0.1 1.80 perf-profile.children.cycles-pp.pv_native_safe_halt
1.89 -0.1 1.81 perf-profile.children.cycles-pp.acpi_idle_do_entry
1.34 -0.1 1.25 perf-profile.children.cycles-pp.do_wp_page
1.22 -0.1 1.14 perf-profile.children.cycles-pp.asm_sysvec_call_function_single
1.89 -0.1 1.81 perf-profile.children.cycles-pp.acpi_safe_halt
3.56 -0.1 3.48 perf-profile.children.cycles-pp.alloc_pages_mpol
1.19 -0.1 1.11 perf-profile.children.cycles-pp.folio_batch_move_lru
2.09 -0.1 2.01 perf-profile.children.cycles-pp.schedule_tail
1.08 ± 2% -0.1 1.01 perf-profile.children.cycles-pp.__percpu_counter_init_many
1.10 -0.1 1.03 perf-profile.children.cycles-pp.its_return_thunk
3.45 -0.1 3.38 perf-profile.children.cycles-pp.__alloc_frozen_pages_noprof
0.64 -0.1 0.57 perf-profile.children.cycles-pp.__slab_free
0.88 -0.1 0.81 perf-profile.children.cycles-pp.__pcs_replace_empty_main
0.89 -0.1 0.82 perf-profile.children.cycles-pp.refill_objects
1.27 -0.1 1.20 perf-profile.children.cycles-pp.sched_balance_find_dst_group
0.89 -0.1 0.82 perf-profile.children.cycles-pp.__refill_objects_node
1.32 ± 2% -0.1 1.26 perf-profile.children.cycles-pp.__pick_next_task
1.25 ± 2% -0.1 1.18 perf-profile.children.cycles-pp.pick_next_task_fair
3.17 -0.1 3.11 perf-profile.children.cycles-pp.alloc_pages_noprof
1.19 -0.1 1.13 perf-profile.children.cycles-pp.update_sg_wakeup_stats
0.88 ± 2% -0.1 0.82 perf-profile.children.cycles-pp.lru_add_drain
1.03 -0.1 0.97 perf-profile.children.cycles-pp.memset_orig
0.87 -0.1 0.82 perf-profile.children.cycles-pp.lru_add_drain_cpu
1.00 ± 2% -0.1 0.94 ± 2% perf-profile.children.cycles-pp.__page_cache_release
0.86 ± 2% -0.1 0.80 perf-profile.children.cycles-pp.dequeue_task_fair
1.11 -0.1 1.06 perf-profile.children.cycles-pp.lock_vma_under_rcu
0.95 -0.1 0.89 ± 2% perf-profile.children.cycles-pp.__wp_page_copy_user
0.84 ± 2% -0.1 0.79 perf-profile.children.cycles-pp.dequeue_entities
0.68 -0.1 0.63 perf-profile.children.cycles-pp.free_frozen_page_commit
0.81 -0.1 0.76 perf-profile.children.cycles-pp.rmqueue
0.86 -0.0 0.81 perf-profile.children.cycles-pp.__percpu_counter_sum
0.67 ± 2% -0.0 0.62 perf-profile.children.cycles-pp.dequeue_entity
0.73 -0.0 0.68 perf-profile.children.cycles-pp.schedule_idle
0.92 -0.0 0.87 ± 2% perf-profile.children.cycles-pp.copy_mc_enhanced_fast_string
1.26 -0.0 1.21 perf-profile.children.cycles-pp.kernel_init_pages
0.89 -0.0 0.85 perf-profile.children.cycles-pp.native_irq_return_iret
0.40 -0.0 0.36 perf-profile.children.cycles-pp.__anon_vma_interval_tree_remove
0.99 -0.0 0.95 perf-profile.children.cycles-pp.__vma_start_exclude_readers
1.14 -0.0 1.10 perf-profile.children.cycles-pp.alloc_thread_stack_node
0.28 ± 7% -0.0 0.24 ± 7% perf-profile.children.cycles-pp.callchain_cursor_reset
2.11 -0.0 2.06 perf-profile.children.cycles-pp.wake_up_new_task
0.97 -0.0 0.93 perf-profile.children.cycles-pp.__put_user_4
0.48 -0.0 0.44 perf-profile.children.cycles-pp.try_charge_memcg
0.82 -0.0 0.78 perf-profile.children.cycles-pp.__vmalloc_node_noprof
2.06 -0.0 2.02 perf-profile.children.cycles-pp.copy_pte_range
0.13 -0.0 0.09 perf-profile.children.cycles-pp.trylock_stock
0.55 -0.0 0.51 perf-profile.children.cycles-pp.free_pcppages_bulk
0.55 -0.0 0.52 ± 2% perf-profile.children.cycles-pp.free_percpu
0.81 ± 2% -0.0 0.78 perf-profile.children.cycles-pp.sync_regs
0.82 -0.0 0.78 perf-profile.children.cycles-pp.__vmalloc_node_range_noprof
0.65 -0.0 0.61 perf-profile.children.cycles-pp.sched_move_task
0.61 -0.0 0.57 perf-profile.children.cycles-pp.sysvec_call_function_single
0.51 -0.0 0.48 perf-profile.children.cycles-pp.__sysvec_call_function_single
0.68 -0.0 0.64 perf-profile.children.cycles-pp.copy_present_ptes
0.51 -0.0 0.48 perf-profile.children.cycles-pp.delayed_vfree_work
0.73 -0.0 0.70 perf-profile.children.cycles-pp.enqueue_task
0.56 -0.0 0.52 perf-profile.children.cycles-pp.process_one_work
0.49 -0.0 0.46 perf-profile.children.cycles-pp.__flush_smp_call_function_queue
0.57 -0.0 0.54 perf-profile.children.cycles-pp.__wake_up_common
0.70 -0.0 0.66 perf-profile.children.cycles-pp.enqueue_task_fair
3.37 -0.0 3.33 perf-profile.children.cycles-pp.ret_from_fork_asm
0.67 -0.0 0.63 perf-profile.children.cycles-pp.do_notify_parent
0.17 ± 2% -0.0 0.14 perf-profile.children.cycles-pp.rcu_cblist_dequeue
0.49 -0.0 0.46 perf-profile.children.cycles-pp.vfree
0.57 -0.0 0.54 perf-profile.children.cycles-pp.__wake_up_sync_key
0.57 -0.0 0.54 perf-profile.children.cycles-pp.try_to_wake_up
0.70 -0.0 0.66 perf-profile.children.cycles-pp.pte_offset_map_lock
0.61 -0.0 0.58 perf-profile.children.cycles-pp.worker_thread
0.54 -0.0 0.51 perf-profile.children.cycles-pp.__rmqueue_pcplist
0.16 ± 2% -0.0 0.13 ± 3% perf-profile.children.cycles-pp.___free_pages
0.44 -0.0 0.41 perf-profile.children.cycles-pp.mas_find
0.20 ± 2% -0.0 0.17 ± 2% perf-profile.children.cycles-pp.mutex_spin_on_owner
0.48 ± 2% -0.0 0.45 perf-profile.children.cycles-pp.try_to_block_task
0.45 -0.0 0.42 perf-profile.children.cycles-pp.exit_to_user_mode_loop
0.38 -0.0 0.35 ± 2% perf-profile.children.cycles-pp.percpu_counter_destroy_many
0.23 ± 4% -0.0 0.20 ± 3% perf-profile.children.cycles-pp.__mem_cgroup_uncharge_folios
0.20 ± 4% -0.0 0.17 ± 4% perf-profile.children.cycles-pp.rwsem_spin_on_owner
0.47 -0.0 0.44 perf-profile.children.cycles-pp.__mod_memcg_state
0.57 -0.0 0.54 perf-profile.children.cycles-pp.ptep_clear_flush
0.37 -0.0 0.35 perf-profile.children.cycles-pp.__folio_batch_add_and_move
0.26 -0.0 0.24 perf-profile.children.cycles-pp.rseq_set_ids_get_csaddr
0.49 -0.0 0.47 perf-profile.children.cycles-pp.update_load_avg
0.53 -0.0 0.50 perf-profile.children.cycles-pp.enqueue_entity
0.28 ± 2% -0.0 0.26 perf-profile.children.cycles-pp.lru_gen_del_folio
0.55 -0.0 0.52 perf-profile.children.cycles-pp.__vmalloc_area_node
0.55 -0.0 0.52 perf-profile.children.cycles-pp.asm_sysvec_reschedule_ipi
0.46 -0.0 0.43 perf-profile.children.cycles-pp.vma_alloc_folio_noprof
1.67 -0.0 1.65 perf-profile.children.cycles-pp.dup_task_struct
0.35 -0.0 0.32 perf-profile.children.cycles-pp.arch_dup_task_struct
0.24 ± 3% -0.0 0.22 ± 2% perf-profile.children.cycles-pp.vma_interval_tree_remove
0.43 -0.0 0.41 perf-profile.children.cycles-pp.__perf_sw_event
0.33 -0.0 0.31 perf-profile.children.cycles-pp.mas_next_slot
0.36 -0.0 0.34 perf-profile.children.cycles-pp.ttwu_do_activate
0.36 -0.0 0.34 ± 3% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.38 ± 2% -0.0 0.36 perf-profile.children.cycles-pp.sched_change_begin
0.36 -0.0 0.34 perf-profile.children.cycles-pp.sched_ttwu_pending
0.42 -0.0 0.40 perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
0.26 -0.0 0.24 ± 2% perf-profile.children.cycles-pp.__rseq_handle_slowpath
0.34 -0.0 0.32 perf-profile.children.cycles-pp.mas_walk
0.27 -0.0 0.25 ± 3% perf-profile.children.cycles-pp.__get_vm_area_node
0.26 -0.0 0.24 perf-profile.children.cycles-pp.remove_vm_area
0.44 -0.0 0.42 perf-profile.children.cycles-pp.vm_area_alloc_pages
0.24 -0.0 0.22 ± 3% perf-profile.children.cycles-pp._raw_spin_trylock
0.27 -0.0 0.26 perf-profile.children.cycles-pp.perf_event_task
0.40 -0.0 0.39 perf-profile.children.cycles-pp.flush_tlb_func
0.29 -0.0 0.28 perf-profile.children.cycles-pp.handle_pte_fault
0.29 -0.0 0.28 perf-profile.children.cycles-pp.mas_store
0.31 -0.0 0.30 perf-profile.children.cycles-pp.lru_add
0.23 ± 2% -0.0 0.21 ± 2% perf-profile.children.cycles-pp._raw_spin_lock_irq
0.28 -0.0 0.26 perf-profile.children.cycles-pp.__pi_memcpy
0.21 ± 2% -0.0 0.19 ± 2% perf-profile.children.cycles-pp.free_unref_folios
0.07 ± 7% -0.0 0.05 ± 8% perf-profile.children.cycles-pp.map_symbol__copy
0.26 -0.0 0.24 perf-profile.children.cycles-pp.update_cfs_rq_load_avg
0.20 -0.0 0.19 ± 2% perf-profile.children.cycles-pp.lock_mm_and_find_vma
0.16 -0.0 0.15 ± 2% perf-profile.children.cycles-pp._raw_write_lock_irq
0.07 -0.0 0.06 ± 7% perf-profile.children.cycles-pp.put_cred_rcu
0.23 -0.0 0.22 perf-profile.children.cycles-pp.select_task_rq
0.19 -0.0 0.18 perf-profile.children.cycles-pp._find_next_or_bit
0.20 -0.0 0.19 perf-profile.children.cycles-pp.add_device_randomness
0.10 -0.0 0.09 perf-profile.children.cycles-pp.filp_close
0.10 -0.0 0.09 perf-profile.children.cycles-pp.free_swap_cache
0.13 -0.0 0.12 perf-profile.children.cycles-pp.___task_rq_lock
0.13 -0.0 0.12 perf-profile.children.cycles-pp.__mt_destroy
0.13 -0.0 0.12 perf-profile.children.cycles-pp.__x64_sys_rt_sigprocmask
0.14 -0.0 0.13 perf-profile.children.cycles-pp._find_next_and_bit
0.09 -0.0 0.08 perf-profile.children.cycles-pp._raw_spin_unlock
0.07 -0.0 0.06 perf-profile.children.cycles-pp.cgroup_task_dead
0.07 -0.0 0.06 perf-profile.children.cycles-pp.exit_fs
0.13 -0.0 0.12 perf-profile.children.cycles-pp.select_idle_sibling
0.12 -0.0 0.11 perf-profile.children.cycles-pp.get_free_pages_noprof
0.12 -0.0 0.11 perf-profile.children.cycles-pp.mod_node_page_state
0.06 -0.0 0.05 perf-profile.children.cycles-pp.set_task_cpu
0.21 -0.0 0.20 perf-profile.children.cycles-pp.memcpy_and_pad
0.07 +0.0 0.08 perf-profile.children.cycles-pp.memcg_charge_kernel_stack
0.11 ± 4% +0.0 0.13 ± 6% perf-profile.children.cycles-pp.pgd_free
0.17 ± 2% +0.0 0.18 ± 2% perf-profile.children.cycles-pp.exit_task_stack_account
0.12 +0.0 0.14 ± 3% perf-profile.children.cycles-pp.kmem_cache_alloc_node_noprof
1.07 +0.0 1.09 perf-profile.children.cycles-pp.__pte_alloc
1.30 +0.0 1.32 perf-profile.children.cycles-pp.flush_tlb_mm_range
1.03 +0.0 1.05 perf-profile.children.cycles-pp.pte_alloc_one
0.49 +0.0 0.50 perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
2.05 +0.0 2.08 perf-profile.children.cycles-pp.unlink_anon_vmas
0.64 +0.0 0.67 perf-profile.children.cycles-pp.__pud_alloc
0.75 +0.0 0.78 perf-profile.children.cycles-pp.on_each_cpu_cond_mask
0.74 +0.0 0.78 perf-profile.children.cycles-pp.smp_call_function_many_cond
1.12 +0.1 1.17 perf-profile.children.cycles-pp.__memcg_kmem_charge_page
1.11 +0.1 1.16 perf-profile.children.cycles-pp.tear_down_vmas
0.26 +0.1 0.31 ± 2% perf-profile.children.cycles-pp.__exit_signal
1.92 +0.1 1.97 perf-profile.children.cycles-pp._raw_spin_lock
1.06 +0.1 1.12 perf-profile.children.cycles-pp.kthread
0.27 +0.1 0.34 ± 2% perf-profile.children.cycles-pp.__put_task_struct
0.48 +0.1 0.54 perf-profile.children.cycles-pp.__account_obj_stock
0.10 +0.1 0.18 ± 2% perf-profile.children.cycles-pp.lru_gen_del_mm
0.10 ± 4% +0.1 0.20 ± 2% perf-profile.children.cycles-pp.lru_gen_add_mm
0.44 +0.1 0.54 perf-profile.children.cycles-pp.smpboot_thread_fn
0.43 +0.1 0.52 perf-profile.children.cycles-pp.run_ksoftirqd
0.07 ± 7% +0.1 0.18 ± 2% perf-profile.children.cycles-pp.put_pid
0.73 +0.1 0.86 perf-profile.children.cycles-pp.vm_area_dup
0.30 ± 2% +0.1 0.43 perf-profile.children.cycles-pp.folio_add_new_anon_rmap
0.84 +0.1 0.98 perf-profile.children.cycles-pp.__free_frozen_pages
0.23 ± 3% +0.1 0.37 ± 2% perf-profile.children.cycles-pp.__folio_mod_stat
0.46 ± 24% +0.2 0.64 ± 11% perf-profile.children.cycles-pp.process_simple
0.45 ± 24% +0.2 0.63 ± 11% perf-profile.children.cycles-pp.ordered_events__queue
0.45 ± 26% +0.2 0.62 ± 11% perf-profile.children.cycles-pp.queue_event
2.28 +0.2 2.46 perf-profile.children.cycles-pp.kmem_cache_alloc_noprof
0.15 ± 3% +0.2 0.35 ± 3% perf-profile.children.cycles-pp.__memcg_kmem_uncharge_page
0.00 +0.2 0.23 ± 6% perf-profile.children.cycles-pp.drain_obj_stock
0.59 ± 2% +0.3 0.87 ± 2% perf-profile.children.cycles-pp._raw_write_unlock_irq
1.13 +0.3 1.41 perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
0.13 +0.4 0.53 ± 3% perf-profile.children.cycles-pp.__refill_obj_stock
1.09 +0.4 1.53 perf-profile.children.cycles-pp.tlb_remove_table_rcu
6.04 ± 2% +0.5 6.56 ± 2% perf-profile.children.cycles-pp.wait_task_zombie
5.87 ± 2% +0.5 6.40 ± 2% perf-profile.children.cycles-pp.release_task
5.98 ± 2% +0.5 6.52 ± 2% perf-profile.children.cycles-pp.exit_notify
2.17 +0.6 2.77 perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
2.11 +0.6 2.71 perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
1.77 +0.6 2.37 perf-profile.children.cycles-pp.__irq_exit_rcu
2.18 +0.6 2.82 perf-profile.children.cycles-pp.kmem_cache_free
2.18 +0.7 2.88 perf-profile.children.cycles-pp.handle_softirqs
2.06 +0.7 2.76 perf-profile.children.cycles-pp.rcu_core
2.03 +0.7 2.74 perf-profile.children.cycles-pp.rcu_do_batch
1.21 +0.7 1.95 perf-profile.children.cycles-pp.__memcg_slab_free_hook
0.64 ± 2% +0.8 1.39 perf-profile.children.cycles-pp.lruvec_stat_mod_folio
8.83 ± 2% +0.8 9.60 ± 2% perf-profile.children.cycles-pp.queued_read_lock_slowpath
17.04 ± 2% +1.2 18.22 perf-profile.children.cycles-pp.kernel_wait4
16.99 ± 2% +1.2 18.17 perf-profile.children.cycles-pp.do_wait
15.10 ± 2% +1.3 16.39 ± 2% perf-profile.children.cycles-pp.__do_wait
71.84 +1.3 73.15 perf-profile.children.cycles-pp.do_syscall_64
71.88 +1.3 73.18 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
14.66 ± 2% +1.4 16.02 ± 2% perf-profile.children.cycles-pp.queued_write_lock_slowpath
23.80 ± 2% +2.1 25.86 ± 2% perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
6.60 ± 3% -0.5 6.10 ± 3% perf-profile.self.cycles-pp.next_uptodate_folio
1.53 ± 2% -0.1 1.38 ± 2% perf-profile.self.cycles-pp.filemap_map_pages
1.40 ± 2% -0.1 1.28 ± 2% perf-profile.self.cycles-pp.folio_add_file_rmap_ptes
1.19 -0.1 1.12 perf-profile.self.cycles-pp.pv_native_safe_halt
1.00 -0.1 0.93 perf-profile.self.cycles-pp.memset_orig
0.83 -0.1 0.77 ± 2% perf-profile.self.cycles-pp.__refill_objects_node
1.00 -0.1 0.95 perf-profile.self.cycles-pp.update_sg_wakeup_stats
0.42 -0.1 0.36 perf-profile.self.cycles-pp.try_charge_memcg
0.54 -0.1 0.49 perf-profile.self.cycles-pp.__slab_free
0.95 -0.1 0.90 perf-profile.self.cycles-pp.down_write
0.39 -0.1 0.34 perf-profile.self.cycles-pp.__anon_vma_interval_tree_remove
0.90 -0.1 0.85 perf-profile.self.cycles-pp.copy_mc_enhanced_fast_string
0.60 -0.0 0.55 perf-profile.self.cycles-pp.its_return_thunk
0.93 -0.0 0.88 perf-profile.self.cycles-pp.__vma_start_exclude_readers
1.23 -0.0 1.18 perf-profile.self.cycles-pp.kernel_init_pages
0.89 -0.0 0.85 perf-profile.self.cycles-pp.native_irq_return_iret
0.78 -0.0 0.74 perf-profile.self.cycles-pp.lock_vma_under_rcu
0.86 -0.0 0.81 ± 2% perf-profile.self.cycles-pp.__vma_start_write
0.81 -0.0 0.77 perf-profile.self.cycles-pp.sync_regs
0.65 -0.0 0.61 perf-profile.self.cycles-pp.copy_present_ptes
0.61 -0.0 0.57 perf-profile.self.cycles-pp.__percpu_counter_sum
0.17 ± 2% -0.0 0.14 perf-profile.self.cycles-pp.rcu_cblist_dequeue
0.91 -0.0 0.88 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
1.48 -0.0 1.45 perf-profile.self.cycles-pp._raw_spin_lock
0.41 -0.0 0.38 perf-profile.self.cycles-pp.do_wp_page
0.16 -0.0 0.13 ± 3% perf-profile.self.cycles-pp.___free_pages
0.43 -0.0 0.40 perf-profile.self.cycles-pp.__mod_memcg_state
0.33 -0.0 0.30 ± 2% perf-profile.self.cycles-pp.tlb_remove_table_rcu
0.19 ± 4% -0.0 0.17 ± 2% perf-profile.self.cycles-pp.mutex_spin_on_owner
0.52 -0.0 0.50 perf-profile.self.cycles-pp.anon_vma_fork
0.33 -0.0 0.30 perf-profile.self.cycles-pp.unlink_anon_vmas
0.24 -0.0 0.22 ± 5% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.21 ± 2% -0.0 0.18 ± 8% perf-profile.self.cycles-pp.vma_interval_tree_insert_after
0.19 ± 4% -0.0 0.17 ± 6% perf-profile.self.cycles-pp.rwsem_spin_on_owner
0.53 -0.0 0.51 perf-profile.self.cycles-pp.zap_pte_range
0.27 -0.0 0.25 perf-profile.self.cycles-pp.mas_next_slot
0.22 ± 3% -0.0 0.20 perf-profile.self.cycles-pp.lru_gen_del_folio
0.26 -0.0 0.24 ± 2% perf-profile.self.cycles-pp.__schedule
0.33 -0.0 0.32 perf-profile.self.cycles-pp.mas_walk
0.25 -0.0 0.23 perf-profile.self.cycles-pp.anon_vma_clone
0.22 ± 2% -0.0 0.21 ± 2% perf-profile.self.cycles-pp._raw_spin_trylock
0.27 -0.0 0.26 perf-profile.self.cycles-pp.__rmqueue_pcplist
0.27 -0.0 0.25 perf-profile.self.cycles-pp.__pi_memcpy
0.22 ± 2% -0.0 0.20 ± 2% perf-profile.self.cycles-pp.__put_user_4
0.18 ± 2% -0.0 0.16 ± 2% perf-profile.self.cycles-pp.page_counter_uncharge
0.26 -0.0 0.25 perf-profile.self.cycles-pp.do_user_addr_fault
0.14 -0.0 0.13 ± 3% perf-profile.self.cycles-pp.__task_pid_nr_ns
0.16 -0.0 0.15 ± 2% perf-profile.self.cycles-pp._raw_write_lock_irq
0.16 -0.0 0.15 ± 2% perf-profile.self.cycles-pp.mas_store
0.20 -0.0 0.19 perf-profile.self.cycles-pp._raw_spin_lock_irq
0.19 -0.0 0.18 perf-profile.self.cycles-pp.handle_mm_fault
0.10 -0.0 0.09 perf-profile.self.cycles-pp.error_entry
0.10 -0.0 0.09 perf-profile.self.cycles-pp.exit_to_user_mode_loop
0.10 -0.0 0.09 perf-profile.self.cycles-pp.prepare_creds
0.16 -0.0 0.15 perf-profile.self.cycles-pp.__perf_sw_event
0.07 -0.0 0.06 perf-profile.self.cycles-pp._raw_spin_unlock
0.13 -0.0 0.12 perf-profile.self.cycles-pp.anon_vma_interval_tree_insert
0.16 -0.0 0.15 perf-profile.self.cycles-pp.dup_fd
0.09 -0.0 0.08 perf-profile.self.cycles-pp.free_unref_folios
0.17 -0.0 0.16 perf-profile.self.cycles-pp.get_page_from_freelist
0.09 -0.0 0.08 perf-profile.self.cycles-pp.lru_add
0.16 -0.0 0.15 perf-profile.self.cycles-pp.lru_gen_add_folio
0.09 -0.0 0.08 perf-profile.self.cycles-pp.ptep_clear_flush
0.13 -0.0 0.12 perf-profile.self.cycles-pp.tear_down_vmas
0.13 -0.0 0.12 perf-profile.self.cycles-pp.update_curr
0.06 -0.0 0.05 perf-profile.self.cycles-pp.__percpu_counter_init_many
0.11 -0.0 0.10 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.11 -0.0 0.10 perf-profile.self.cycles-pp.enqueue_task_fair
0.08 -0.0 0.07 perf-profile.self.cycles-pp.exit_task_stack_account
0.08 -0.0 0.07 perf-profile.self.cycles-pp.flush_tlb_func
0.08 -0.0 0.07 perf-profile.self.cycles-pp.pidfs_add_pid
0.06 -0.0 0.05 perf-profile.self.cycles-pp.rseq_set_ids_get_csaddr
0.11 -0.0 0.10 perf-profile.self.cycles-pp.update_cfs_rq_load_avg
0.06 -0.0 0.05 perf-profile.self.cycles-pp.vm_area_init_from
0.15 -0.0 0.14 perf-profile.self.cycles-pp.acct_collect
0.23 ± 2% +0.0 0.26 perf-profile.self.cycles-pp.pcpu_alloc_noprof
0.68 +0.0 0.72 perf-profile.self.cycles-pp.smp_call_function_many_cond
0.07 +0.0 0.11 perf-profile.self.cycles-pp.__mmput
0.40 +0.1 0.46 perf-profile.self.cycles-pp.__account_obj_stock
0.00 +0.1 0.06 perf-profile.self.cycles-pp.lru_gen_add_mm
0.02 ±141% +0.1 0.09 perf-profile.self.cycles-pp.folio_lruvec_lock_irqsave
0.32 +0.1 0.44 perf-profile.self.cycles-pp.__memcg_kmem_charge_page
0.09 +0.2 0.26 ± 4% perf-profile.self.cycles-pp.__refill_obj_stock
0.44 ± 25% +0.2 0.62 ± 11% perf-profile.self.cycles-pp.queue_event
0.06 +0.2 0.27 ± 2% perf-profile.self.cycles-pp.__memcg_kmem_uncharge_page
0.00 +0.2 0.22 ± 4% perf-profile.self.cycles-pp.drain_obj_stock
0.75 +0.3 1.01 perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
0.87 +0.3 1.17 perf-profile.self.cycles-pp.__memcg_slab_free_hook
0.29 ± 5% +0.8 1.06 perf-profile.self.cycles-pp.lruvec_stat_mod_folio
23.65 ± 2% +2.0 25.66 ± 2% perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
Disclaimer:
Results have been estimated based on internal Intel analysis and are provided
for informational purposes only. Any difference in system hardware or software
design or configuration may affect actual performance.
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [PATCH 0/3] mm/zswap: Implement per-cgroup proactive writeback
From: Hao Jia @ 2026-05-12 11:23 UTC (permalink / raw)
To: Michal Koutný
Cc: akpm, tj, hannes, shakeel.butt, mhocko, yosry, nphamcs,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <agG-gNEclOVf-9WA@localhost.localdomain>
On 2026/5/11 19:39, Michal Koutný wrote:
> On Mon, May 11, 2026 at 06:51:46PM +0800, Hao Jia <jiahao.kernel@gmail.com> wrote:
>> From: Hao Jia <jiahao1@lixiang.com>
>>
>> Zswap currently writes back pages to backing swap devices reactively,
>> triggered either by memory pressure via the shrinker or by the pool
>> reaching its size limit. However, this reactive approach makes writeback
>> timing indeterminate and can disrupt latency-sensitive workloads when
>> eviction happens to coincide with a critical execution window.
>>
>> Furthermore, in certain scenarios, it is desirable to trigger writeback
>> in advance to free up memory. For example, users may want to prepare for
>> an upcoming memory-intensive workload by flushing cold memory to the
>> backing storage when the system is relatively idle.
>
> I can imagine the zswap writeout can come at the least possible
> moment...
>
>> To address these issues, this patch series introduces a per-cgroup
>> interface that allows users to proactively write back cold compressed
>> pages from zswap to the backing swap device.
>
> ...but I see this series is not only per-cgroup proactive reclaim but
> it's also age-based reclaim.
>
> The per-cg consumption and limits (and regular memory reclaim) are all
> measured in sizes. This age-based invocations don't seem commensurable
> (e.g. how would users in practice determine what is the desired input to
> here).
>
Thanks Michal — you are right. The series is both per-memcg *and*
age-based.
The interface carries a size budget, like memory.reclaim. The two
parameters play different roles:
"write back up to <max> bytes, chosen from entries whose residency
in zswap is at least <age>"
Size stays the unit of *amount*; age is just how we describe *which*
entries are eligible.
> Could you explain more reasoning behind this design?
>
Context on the use case:
Our deployment runs a userspace proactive reclaimer driven by the
system's runtime state (memory/CPU/IO pressure, refault rate, ...)
and workload-specific policy. It uses memory.reclaim to drive
reclaim, which compresses cold anon pages into zswap as the first
stage. For entries that then remain in zswap past a policy-defined
age threshold, the reclaimer wants to write them back to the backing
swap device at a moment of its own choosing, to further reclaim the
DRAM still held by the compressed data.
Why age is a reasonable selector at this stage:
Pages in zswap have already passed a first-stage coldness judgement
(otherwise they would not have been compressed). For second-level
offloading, the question is which of them are cold *enough*.
Time-in-zswap is a natural proxy for that. A swap-in invalidates the
corresponding zswap entry and resets the clock, so by construction
an entry that has sat in zswap for N seconds has not been faulted in
for at least N seconds. Residency in zswap is therefore a strong
signal that the entry is not about to refault.
In our deployment the userspace reclaimer starts from a conservative
threshold (the starting value depends on the workload) and adjusts it
through closed-loop feedback:
- on one side, the age distribution of zswap entries, to see
whether there is a meaningful population past the threshold;
- on the other side, the post-writeback refault rate and related
signals, to confirm that entries written back were in fact cold
enough.
Both <age> and max=<bytes> are tuned against these signals until the
realized writeback volume matches target. This is the same
control-loop style already used to drive the first-stage
memory.reclaim budget.
Thanks,
Hao
^ permalink raw reply
* Re: [PATCH v2 10/10] sched/eevdf: Move to a single runqueue
From: Peter Zijlstra @ 2026-05-12 11:09 UTC (permalink / raw)
To: K Prateek Nayak
Cc: mingo, longman, chenridong, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, qyousef
In-Reply-To: <133c4d08-5dfb-4f4f-83cb-f9652d4212ef@amd.com>
On Mon, May 11, 2026 at 09:51:57PM +0530, K Prateek Nayak wrote:
> Hello Peter,
>
> On 5/11/2026 5:01 PM, Peter Zijlstra wrote:
> > @@ -9291,34 +9206,25 @@ static void wakeup_preempt_fair(struct r
> > + se = pick_next_entity(rq, true);
> > + if (!se)
> > + goto again;
> >
> > p = task_of(se);
> > - if (unlikely(throttled))
> > + if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
> > task_throttle_setup_work(p);
>
> I think this bit should also be replicated in set_next_task() after
> account_cfs_rq_runtime() since any part of the hierarchy may get
> throttled as a result of failing to grab runtime.
>
> Also check_cfs_rq_runtime() only sees if the cfs_rq is throttled
> but the task can fail to run if it is on a throttled_hierarchy() too
> so that should be the correct check here.
>
> Something like below (only build tested on queue/sched/flat):
>
> diff --git a/kernel/sched/fair.c b/kernel/sched/fair.c
> index e54da4c6c945..950c072244b2 100644
> --- a/kernel/sched/fair.c
> +++ b/kernel/sched/fair.c
> @@ -9224,7 +9224,19 @@ struct task_struct *pick_task_fair(struct rq *rq, struct rq_flags *rf)
> goto again;
>
> p = task_of(se);
> - if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
> + /*
> + * For cases where prev is picked again after
> + * being throttled, entity_tick() would have
> + * already marked its hierarchy as throttled.
> + *
> + * Add throttle work here since
> + * put_prev_set_next_task() is skipped on
> + * same task's selection.
> + *
> + * For other case, set_next_task_fair() will
> + * handle adding the throttle work.
> + */
> + if (throttled_hierarchy(cfs_rq_of(se)))
> task_throttle_setup_work(p);
Ah, right, because we've not accumulated runtime, it doesn't make sense
to use check_cfs_rq_runtime() at pick time, all we need to do is check
if the task should be throttled.
However, since set_next_task_fair() will walk the entire hierarchy
anyway, we can remove it here entirely and fully rely on that.
> return p;
>
> @@ -13819,6 +13831,12 @@ static void set_next_task_fair(struct rq *rq, struct task_struct *p, bool first)
> if (on_rq)
> weight = __calc_prop_weight(cfs_rq, se, weight);
> }
> + /*
> + * Add throttle work if the bandwidth allocation above failed
> + * to grab any runtime and throttled the task's hierarchy.
> + */
> + if (throttled_hierarchy(task_cfs_rq(p)))
> + task_throttle_setup_work(p);
We already call into account_cfs_rq_runtime(); which basically does all
we need.
I think the distinction between account_cfs_rq_runtime() and
check_cfs_rq_runtime() no longer makes sense. We can throttle a cfs_rq
at any point now, since we no longer remove the cfs_rq, but rather we
make the tasks suspend themselves until the cfs_rq naturally dequeues
for being empty.
Something like so perhaps?
---
--- a/kernel/sched/fair.c
+++ b/kernel/sched/fair.c
@@ -488,7 +488,7 @@ static int se_is_idle(struct sched_entit
#endif /* !CONFIG_FAIR_GROUP_SCHED */
static __always_inline
-void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);
+bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec);
/**************************************************************
* Scheduling class tree data structure manipulation methods:
@@ -1420,12 +1420,22 @@ static void update_curr(struct cfs_rq *c
}
}
+static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
+static inline void task_throttle_setup_work(struct task_struct *p);
+
static void update_curr_fair(struct rq *rq)
{
struct sched_entity *se = &rq->donor->se;
+ bool throttled = false;
- for_each_sched_entity(se)
- update_curr(cfs_rq_of(se));
+ for_each_sched_entity(se) {
+ struct cfs_rq *cfs_rq = cfs_rq_of(se);
+ update_curr(cfs_rq);
+ throttled |= cfs_rq_throttled(cfs_rq);
+ }
+
+ if (throttled)
+ task_throttle_setup_work(rq->donor);
}
static inline void
@@ -5627,7 +5637,6 @@ place_entity(struct cfs_rq *cfs_rq, stru
}
static void check_enqueue_throttle(struct cfs_rq *cfs_rq);
-static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq);
static void
enqueue_entity(struct cfs_rq *cfs_rq, struct sched_entity *se, int flags)
@@ -5830,8 +5839,6 @@ pick_next_entity(struct rq *rq, bool pro
return se;
}
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq);
-
static void put_prev_entity(struct cfs_rq *cfs_rq, struct sched_entity *prev)
{
/*
@@ -5841,9 +5848,6 @@ static void put_prev_entity(struct cfs_r
if (prev->on_rq)
update_curr(cfs_rq);
- /* throttle cfs_rqs exceeding runtime */
- check_cfs_rq_runtime(cfs_rq);
-
if (prev->on_rq) {
update_stats_wait_start_fair(cfs_rq, prev);
/* in !on_rq case, update occurred at dequeue */
@@ -5976,44 +5980,29 @@ static int __assign_cfs_rq_runtime(struc
return cfs_rq->runtime_remaining > 0;
}
-/* returns 0 on failure to allocate runtime */
-static int assign_cfs_rq_runtime(struct cfs_rq *cfs_rq)
-{
- struct cfs_bandwidth *cfs_b = tg_cfs_bandwidth(cfs_rq->tg);
- int ret;
-
- raw_spin_lock(&cfs_b->lock);
- ret = __assign_cfs_rq_runtime(cfs_b, cfs_rq, sched_cfs_bandwidth_slice());
- raw_spin_unlock(&cfs_b->lock);
-
- return ret;
-}
+static bool throttle_cfs_rq(struct cfs_rq *cfs_rq);
-static void __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
+static bool __account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
{
/* dock delta_exec before expiring quota (as it could span periods) */
cfs_rq->runtime_remaining -= delta_exec;
if (likely(cfs_rq->runtime_remaining > 0))
- return;
+ return false;
if (cfs_rq->throttled)
- return;
- /*
- * if we're unable to extend our runtime we resched so that the active
- * hierarchy can be throttled
- */
- if (!assign_cfs_rq_runtime(cfs_rq) && likely(cfs_rq->h_curr))
- resched_curr(rq_of(cfs_rq));
+ return true;
+
+ return throttle_cfs_rq(cfs_rq);
}
static __always_inline
-void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
+bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec)
{
if (!cfs_bandwidth_used() || !cfs_rq->runtime_enabled)
- return;
+ return false;
- __account_cfs_rq_runtime(cfs_rq, delta_exec);
+ return __account_cfs_rq_runtime(cfs_rq, delta_exec);
}
static inline int cfs_rq_throttled(struct cfs_rq *cfs_rq)
@@ -6284,9 +6273,9 @@ static bool throttle_cfs_rq(struct cfs_r
* We have raced with bandwidth becoming available, and if we
* actually throttled the timer might not unthrottle us for an
* entire period. We additionally needed to make sure that any
- * subsequent check_cfs_rq_runtime calls agree not to throttle
- * us, as we may commit to do cfs put_prev+pick_next, so we ask
- * for 1ns of runtime rather than just check cfs_b.
+ * subsequent account_cfs_rq_runtime() calls agree not to
+ * throttle us, as we may commit to do cfs put_prev+pick_next,
+ * so we ask for 1ns of runtime rather than just check cfs_b.
*/
dequeue = 0;
} else {
@@ -6711,8 +6700,6 @@ static void check_enqueue_throttle(struc
/* update runtime allocation */
account_cfs_rq_runtime(cfs_rq, 0);
- if (cfs_rq->runtime_remaining <= 0)
- throttle_cfs_rq(cfs_rq);
}
static void sync_throttle(struct task_group *tg, int cpu)
@@ -6742,25 +6729,6 @@ static void sync_throttle(struct task_gr
cfs_rq->pelt_clock_throttled = 1;
}
-/* conditionally throttle active cfs_rq's from put_prev_entity() */
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq)
-{
- if (!cfs_bandwidth_used())
- return false;
-
- if (likely(!cfs_rq->runtime_enabled || cfs_rq->runtime_remaining > 0))
- return false;
-
- /*
- * it's possible for a throttled entity to be forced into a running
- * state (e.g. set_curr_task), in this case we're finished.
- */
- if (cfs_rq_throttled(cfs_rq))
- return true;
-
- return throttle_cfs_rq(cfs_rq);
-}
-
static enum hrtimer_restart sched_cfs_slack_timer(struct hrtimer *timer)
{
struct cfs_bandwidth *cfs_b =
@@ -7015,8 +6983,7 @@ static void sched_fair_update_stop_tick(
#else /* !CONFIG_CFS_BANDWIDTH: */
-static void account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) {}
-static bool check_cfs_rq_runtime(struct cfs_rq *cfs_rq) { return false; }
+static bool account_cfs_rq_runtime(struct cfs_rq *cfs_rq, u64 delta_exec) { return false; }
static void check_enqueue_throttle(struct cfs_rq *cfs_rq) {}
static inline void sync_throttle(struct task_group *tg, int cpu) {}
static __always_inline void return_cfs_rq_runtime(struct cfs_rq *cfs_rq) {}
@@ -9208,7 +9175,6 @@ struct task_struct *pick_task_fair(struc
{
struct cfs_rq *cfs_rq = &rq->cfs;
struct sched_entity *se;
- struct task_struct *p;
int new_tasks;
again:
@@ -9223,10 +9189,7 @@ struct task_struct *pick_task_fair(struc
if (!se)
goto again;
- p = task_of(se);
- if (unlikely(check_cfs_rq_runtime(cfs_rq_of(se))))
- task_throttle_setup_work(p);
- return p;
+ return task_of(se);
idle:
new_tasks = sched_balance_newidle(rq, rf);
@@ -13618,6 +13581,7 @@ static void task_tick_fair(struct rq *rq
{
struct sched_entity *se = &curr->se;
unsigned long weight = NICE_0_LOAD;
+ bool throttled = false;
struct cfs_rq *cfs_rq;
for_each_sched_entity(se) {
@@ -13625,8 +13589,13 @@ static void task_tick_fair(struct rq *rq
entity_tick(cfs_rq, se, queued);
weight = __calc_prop_weight(cfs_rq, se, weight);
+
+ throttled |= cfs_rq_throttled(cfs_rq);
}
+ if (throttled)
+ task_throttle_setup_work(curr);
+
se = &curr->se;
reweight_eevdf(cfs_rq, se, weight, se->on_rq);
@@ -13800,6 +13769,7 @@ static void set_next_task_fair(struct rq
struct cfs_rq *cfs_rq = &rq->cfs;
unsigned long weight = NICE_0_LOAD;
bool on_rq = se->on_rq;
+ bool throttled = false;
clear_buddies(cfs_rq, se);
@@ -13814,12 +13784,15 @@ static void set_next_task_fair(struct rq
set_next_entity(cfs_rq, se);
/* ensure bandwidth has been allocated on our new cfs_rq */
- account_cfs_rq_runtime(cfs_rq, 0);
+ throttled |= account_cfs_rq_runtime(cfs_rq, 0);
if (on_rq)
weight = __calc_prop_weight(cfs_rq, se, weight);
}
+ if (throttled)
+ task_throttle_setup_work(p);
+
se = &p->se;
cfs_rq->curr = se;
^ permalink raw reply
* Re: [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: Christian König @ 2026-05-12 10:14 UTC (permalink / raw)
To: Albert Esteve, Tejun Heo, Johannes Weiner, Michal Koutný,
Jonathan Corbet, Shuah Khan, Sumit Semwal, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Benjamin Gaignard, Brian Starkey, John Stultz, T.J. Mercier,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan
Cc: cgroups, linux-doc, linux-kernel, linux-media, dri-devel,
linaro-mm-sig, linux-mm, linux-security-module, selinux,
linux-kselftest, mripard, echanude
In-Reply-To: <20260512-v2_20230123_tjmercier_google_com-v1-2-6326701c3691@redhat.com>
On 5/12/26 11:10, Albert Esteve wrote:
> On embedded platforms a central process often allocates dma-buf
> memory on behalf of client applications. Without a way to
> attribute the charge to the requesting client's cgroup, the
> cost lands on the allocator, making per-cgroup memory limits
> ineffective for the actual consumers.
>
> Add charge_pid_fd to struct dma_heap_allocation_data. When set to
> a valid pidfd, DMA_HEAP_IOCTL_ALLOC resolves the target task's
> memcg and charges the buffer there via mem_cgroup_charge_dmabuf()
> inside dma_heap_buffer_alloc(). Without charge_pid_fd, and with
> the mem_accounting module parameter enabled, the buffer is charged
> to the allocator's own cgroup.
>
> Additionally, commit 3c227be90659 ("dma-buf: system_heap: account for
> system heap allocation in memcg") adds __GFP_ACCOUNT to system-heap
> page allocations. Keeping __GFP_ACCOUNT would charge the same pages
> twice (once to kmem, once to MEMCG_DMABUF), thus remove it and route
> all accounting through a single MEMCG_DMABUF path.
>
> Usage examples:
>
> 1. Central allocator charging to a client at allocation time.
> The allocator knows the client's PID (e.g., from binder's
> sender_pid) and uses pidfd to attribute the charge:
>
> pid_t client_pid = txn->sender_pid;
> int pidfd = pidfd_open(client_pid, 0);
>
> struct dma_heap_allocation_data alloc = {
> .len = buffer_size,
> .fd_flags = O_RDWR | O_CLOEXEC,
> .charge_pid_fd = pidfd,
> };
> ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
> close(pidfd);
> /* alloc.fd is now charged to client's cgroup */
>
> 2. Default allocation (no pidfd, mem_accounting=1).
> When charge_pid_fd is not set and the mem_accounting module
> parameter is enabled, the buffer is charged to the allocator's
> own cgroup:
>
> struct dma_heap_allocation_data alloc = {
> .len = buffer_size,
> .fd_flags = O_RDWR | O_CLOEXEC,
> };
> ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
> /* charged to current process's cgroup */
>
> Current limitations:
>
> - Single-owner model: a dma-buf carries one memcg charge regardless of
> how many processes share it. Means only the first owner (and exporter)
> of the shared buffer bears the charge.
> - Only memcg accounting supported. While this makes sense for system
> heap buffers, other heaps (e.g., CMA heaps) will require selectively
> charging also for the dmem controller.
Well that doesn't looks soo bad, it at least seems to tackle the problem at hand for Android and some of other embedded use cases.
I'm just not sure if this is future prove and will work for all use cases, e.g. cloud gaming, native context for automotive etc...
Essentially the problem boils down to two limitations:
1) a piece of memory can only be charged to one cgroup, the framework doesn't has a concept of charging shared memory to multiple groups
2) when memory references in the form of file descriptors are passed between applications we have no way of changing the accounting to a different cgroup
The passing of the memory reference already has a well defined uAPI and if we could solve those two limitations we not only solve the problem without introducing new uAPI (with potential new security risks) but also solve it for all other use cases which uses file descriptors as well as. E.g. memfd, accel and GPU drivers etc...
On the other hand it is really nice to finally see this tackled for at least DMA-buf heaps. On the GPU side I have seen just another try of a driver doing some kind of special driver specific accounting to solve this just a few weeks ago. And to be honest such single driver island approach have the tendency to break more often that they are working correctly.
Regards,
Christian.
>
> Signed-off-by: Albert Esteve <aesteve@redhat.com>
> ---
> Documentation/admin-guide/cgroup-v2.rst | 5 ++--
> drivers/dma-buf/dma-buf.c | 16 ++++---------
> drivers/dma-buf/dma-heap.c | 42 ++++++++++++++++++++++++++++++---
> drivers/dma-buf/heaps/system_heap.c | 2 --
> include/uapi/linux/dma-heap.h | 6 +++++
> 5 files changed, 53 insertions(+), 18 deletions(-)
>
> diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
> index 8bdbc2e866430..824d269531eb1 100644
> --- a/Documentation/admin-guide/cgroup-v2.rst
> +++ b/Documentation/admin-guide/cgroup-v2.rst
> @@ -1636,8 +1636,9 @@ The following nested keys are defined.
> structures.
>
> dmabuf (npn)
> - Amount of memory used for exported DMA buffers allocated by the cgroup.
> - Stays with the allocating cgroup regardless of how the buffer is shared.
> + Amount of memory used for exported DMA buffers allocated by or on
> + behalf of the cgroup. Stays with the allocating cgroup regardless
> + of how the buffer is shared.
>
> workingset_refault_anon
> Number of refaults of previously evicted anonymous pages.
> diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
> index ce02377f48908..23fb758b78297 100644
> --- a/drivers/dma-buf/dma-buf.c
> +++ b/drivers/dma-buf/dma-buf.c
> @@ -181,8 +181,11 @@ static void dma_buf_release(struct dentry *dentry)
> */
> BUG_ON(dmabuf->cb_in.active || dmabuf->cb_out.active);
>
> - mem_cgroup_uncharge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE);
> - mem_cgroup_put(dmabuf->memcg);
> + if (dmabuf->memcg) {
> + mem_cgroup_uncharge_dmabuf(dmabuf->memcg,
> + PAGE_ALIGN(dmabuf->size) / PAGE_SIZE);
> + mem_cgroup_put(dmabuf->memcg);
> + }
>
> dmabuf->ops->release(dmabuf);
>
> @@ -764,13 +767,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
> dmabuf->resv = resv;
> }
>
> - dmabuf->memcg = get_mem_cgroup_from_mm(current->mm);
> - if (!mem_cgroup_charge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE,
> - GFP_KERNEL)) {
> - ret = -ENOMEM;
> - goto err_memcg;
> - }
> -
> file->private_data = dmabuf;
> file->f_path.dentry->d_fsdata = dmabuf;
> dmabuf->file = file;
> @@ -781,8 +777,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
>
> return dmabuf;
>
> -err_memcg:
> - mem_cgroup_put(dmabuf->memcg);
> err_file:
> fput(file);
> err_module:
> diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c
> index ac5f8685a6494..ff6e259afcdc0 100644
> --- a/drivers/dma-buf/dma-heap.c
> +++ b/drivers/dma-buf/dma-heap.c
> @@ -7,13 +7,17 @@
> */
>
> #include <linux/cdev.h>
> +#include <linux/cgroup.h>
> #include <linux/device.h>
> #include <linux/dma-buf.h>
> #include <linux/dma-heap.h>
> +#include <linux/memcontrol.h>
> +#include <linux/sched/mm.h>
> #include <linux/err.h>
> #include <linux/export.h>
> #include <linux/list.h>
> #include <linux/nospec.h>
> +#include <linux/pidfd.h>
> #include <linux/syscalls.h>
> #include <linux/uaccess.h>
> #include <linux/xarray.h>
> @@ -55,10 +59,12 @@ MODULE_PARM_DESC(mem_accounting,
> "Enable cgroup-based memory accounting for dma-buf heap allocations (default=false).");
>
> static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
> - u32 fd_flags,
> - u64 heap_flags)
> + u32 fd_flags, u64 heap_flags,
> + struct mem_cgroup *charge_to)
> {
> struct dma_buf *dmabuf;
> + unsigned int nr_pages;
> + struct mem_cgroup *memcg = charge_to;
> int fd;
>
> /*
> @@ -73,6 +79,22 @@ static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
> if (IS_ERR(dmabuf))
> return PTR_ERR(dmabuf);
>
> + nr_pages = len / PAGE_SIZE;
> +
> + if (memcg)
> + css_get(&memcg->css);
> + else if (mem_accounting)
> + memcg = get_mem_cgroup_from_mm(current->mm);
> +
> + if (memcg) {
> + if (!mem_cgroup_charge_dmabuf(memcg, nr_pages, GFP_KERNEL)) {
> + mem_cgroup_put(memcg);
> + dma_buf_put(dmabuf);
> + return -ENOMEM;
> + }
> + dmabuf->memcg = memcg;
> + }
> +
> fd = dma_buf_fd(dmabuf, fd_flags);
> if (fd < 0) {
> dma_buf_put(dmabuf);
> @@ -102,6 +124,9 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
> {
> struct dma_heap_allocation_data *heap_allocation = data;
> struct dma_heap *heap = file->private_data;
> + struct mem_cgroup *memcg = NULL;
> + struct task_struct *task;
> + unsigned int pidfd_flags;
> int fd;
>
> if (heap_allocation->fd)
> @@ -113,9 +138,20 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
> if (heap_allocation->heap_flags & ~DMA_HEAP_VALID_HEAP_FLAGS)
> return -EINVAL;
>
> + if (heap_allocation->charge_pid_fd) {
> + task = pidfd_get_task(heap_allocation->charge_pid_fd, &pidfd_flags);
> + if (IS_ERR(task))
> + return PTR_ERR(task);
> +
> + memcg = get_mem_cgroup_from_mm(task->mm);
> + put_task_struct(task);
> + }
> +
> fd = dma_heap_buffer_alloc(heap, heap_allocation->len,
> heap_allocation->fd_flags,
> - heap_allocation->heap_flags);
> + heap_allocation->heap_flags,
> + memcg);
> + mem_cgroup_put(memcg);
> if (fd < 0)
> return fd;
>
> diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
> index 03c2b87cb1112..95d7688167b93 100644
> --- a/drivers/dma-buf/heaps/system_heap.c
> +++ b/drivers/dma-buf/heaps/system_heap.c
> @@ -385,8 +385,6 @@ static struct page *alloc_largest_available(unsigned long size,
> if (max_order < orders[i])
> continue;
> flags = order_flags[i];
> - if (mem_accounting)
> - flags |= __GFP_ACCOUNT;
> page = alloc_pages(flags, orders[i]);
> if (!page)
> continue;
> diff --git a/include/uapi/linux/dma-heap.h b/include/uapi/linux/dma-heap.h
> index a4cf716a49fa6..e02b0f8cbc6a1 100644
> --- a/include/uapi/linux/dma-heap.h
> +++ b/include/uapi/linux/dma-heap.h
> @@ -29,6 +29,10 @@
> * handle to the allocated dma-buf
> * @fd_flags: file descriptor flags used when allocating
> * @heap_flags: flags passed to heap
> + * @charge_pid_fd: optional pidfd of the process whose cgroup should be
> + * charged for this allocation; 0 means charge the calling
> + * process's cgroup
> + * @__padding: reserved, must be zero
> *
> * Provided by userspace as an argument to the ioctl
> */
> @@ -37,6 +41,8 @@ struct dma_heap_allocation_data {
> __u32 fd;
> __u32 fd_flags;
> __u64 heap_flags;
> + __u32 charge_pid_fd;
> + __u32 __padding;
> };
>
> #define DMA_HEAP_IOC_MAGIC 'H'
>
^ permalink raw reply
* Re: [PATCH v3 05/12] mm, swap: unify large folio allocation
From: Baolin Wang @ 2026-05-12 10:10 UTC (permalink / raw)
To: kasong, linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Qi Zheng, linux-kernel, cgroups,
Yosry Ahmed, Lorenzo Stoakes, Dev Jain, Lance Yang, Michal Hocko,
Michal Hocko, Suren Baghdasaryan, Axel Rasmussen
In-Reply-To: <20260421-swap-table-p4-v3-5-2f23759a76bc@tencent.com>
On 4/21/26 2:16 PM, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
>
> Now that direct large order allocation is supported in the swap cache,
> both anon and shmem can use it instead of implementing their own methods.
> This unifies the fallback and swap cache check, which also reduces the
> TOCTOU race window of swap cache state: previously, high order swapin
> required checking swap cache states first, then allocating and falling
> back separately. Now all these steps happen in the same compact loop.
>
> Order fallback and statistics are also unified, callers just need to
> check and pass the acceptable order bitmask.
>
> There is basically no behavior change. This only makes things more
> unified and prepares for later commits. Cgroup and zero map checks can
> also be moved into the compact loop, further reducing race windows and
> redundancy
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
> mm/memory.c | 77 ++++++------------------------
> mm/shmem.c | 94 +++++++++---------------------------
> mm/swap.h | 30 ++----------
> mm/swap_state.c | 145 ++++++++++----------------------------------------------
> mm/swapfile.c | 3 +-
> 5 files changed, 67 insertions(+), 282 deletions(-)
>
> diff --git a/mm/memory.c b/mm/memory.c
> index ea6568571131..404734a5bcff 100644
> --- a/mm/memory.c
> +++ b/mm/memory.c
> @@ -4593,26 +4593,6 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)
> return VM_FAULT_SIGBUS;
> }
>
> -static struct folio *__alloc_swap_folio(struct vm_fault *vmf)
> -{
> - struct vm_area_struct *vma = vmf->vma;
> - struct folio *folio;
> - softleaf_t entry;
> -
> - folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address);
> - if (!folio)
> - return NULL;
> -
> - entry = softleaf_from_pte(vmf->orig_pte);
> - if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
> - GFP_KERNEL, entry)) {
> - folio_put(folio);
> - return NULL;
> - }
> -
> - return folio;
> -}
> -
> #ifdef CONFIG_TRANSPARENT_HUGEPAGE
> /*
> * Check if the PTEs within a range are contiguous swap entries
> @@ -4642,8 +4622,6 @@ static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
> */
> if (unlikely(swap_zeromap_batch(entry, nr_pages, NULL) != nr_pages))
> return false;
> - if (unlikely(non_swapcache_batch(entry, nr_pages) != nr_pages))
> - return false;
>
> return true;
> }
> @@ -4671,16 +4649,14 @@ static inline unsigned long thp_swap_suitable_orders(pgoff_t swp_offset,
> return orders;
> }
>
> -static struct folio *alloc_swap_folio(struct vm_fault *vmf)
> +static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
> {
> struct vm_area_struct *vma = vmf->vma;
> unsigned long orders;
> - struct folio *folio;
> unsigned long addr;
> softleaf_t entry;
> spinlock_t *ptl;
> pte_t *pte;
> - gfp_t gfp;
> int order;
>
> /*
> @@ -4688,7 +4664,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
> * maintain the uffd semantics.
> */
> if (unlikely(userfaultfd_armed(vma)))
> - goto fallback;
> + return 0;
>
> /*
> * A large swapped out folio could be partially or fully in zswap. We
> @@ -4696,7 +4672,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
> * folio.
> */
> if (!zswap_never_enabled())
> - goto fallback;
> + return 0;
>
> entry = softleaf_from_pte(vmf->orig_pte);
> /*
> @@ -4710,12 +4686,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
> vmf->address, orders);
>
> if (!orders)
> - goto fallback;
> + return 0;
>
> pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
> vmf->address & PMD_MASK, &ptl);
> if (unlikely(!pte))
> - goto fallback;
> + return 0;
>
> /*
> * For do_swap_page, find the highest order where the aligned range is
> @@ -4731,29 +4707,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
>
> pte_unmap_unlock(pte, ptl);
>
> - /* Try allocating the highest of the remaining orders. */
> - gfp = vma_thp_gfp_mask(vma);
> - while (orders) {
> - addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
> - folio = vma_alloc_folio(gfp, order, vma, addr);
> - if (folio) {
> - if (!mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
> - gfp, entry))
> - return folio;
> - count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
> - folio_put(folio);
> - }
> - count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
> - order = next_order(&orders, order);
> - }
> -
> -fallback:
> - return __alloc_swap_folio(vmf);
> + return orders;
> }
> #else /* !CONFIG_TRANSPARENT_HUGEPAGE */
> -static struct folio *alloc_swap_folio(struct vm_fault *vmf)
> +static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
> {
> - return __alloc_swap_folio(vmf);
> + return 0;
> }
> #endif /* CONFIG_TRANSPARENT_HUGEPAGE */
>
> @@ -4859,21 +4818,13 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
> if (folio)
> swap_update_readahead(folio, vma, vmf->address);
> if (!folio) {
> - if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {
> - folio = alloc_swap_folio(vmf);
> - if (folio) {
> - /*
> - * folio is charged, so swapin can only fail due
> - * to raced swapin and return NULL.
> - */
> - swapcache = swapin_folio(entry, folio);
> - if (swapcache != folio)
> - folio_put(folio);
> - folio = swapcache;
> - }
> - } else {
> + /* Swapin bypasses readahead for SWP_SYNCHRONOUS_IO devices */
> + if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
> + folio = swapin_entry(entry, GFP_HIGHUSER_MOVABLE,
> + thp_swapin_suitable_orders(vmf),
> + vmf, NULL, 0);
> + else
> folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf);
> - }
>
> if (!folio) {
> /*
> diff --git a/mm/shmem.c b/mm/shmem.c
> index 5916acf594a8..17e3da11bb1d 100644
> --- a/mm/shmem.c
> +++ b/mm/shmem.c
> @@ -159,7 +159,7 @@ static unsigned long shmem_default_max_inodes(void)
>
> static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
> struct folio **foliop, enum sgp_type sgp, gfp_t gfp,
> - struct vm_area_struct *vma, vm_fault_t *fault_type);
> + struct vm_fault *vmf, vm_fault_t *fault_type);
>
> static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb)
> {
> @@ -2017,68 +2017,24 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf,
> }
>
> static struct folio *shmem_swap_alloc_folio(struct inode *inode,
> - struct vm_area_struct *vma, pgoff_t index,
> + struct vm_fault *vmf, pgoff_t index,
> swp_entry_t entry, int order, gfp_t gfp)
> {
> + pgoff_t ilx;
> + struct folio *folio;
> + struct mempolicy *mpol;
> + unsigned long orders = BIT(order);
> struct shmem_inode_info *info = SHMEM_I(inode);
> - struct folio *new, *swapcache;
> - int nr_pages = 1 << order;
> - gfp_t alloc_gfp = gfp;
> -
> - if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
> - if (WARN_ON_ONCE(order))
> - return ERR_PTR(-EINVAL);
> - } else if (order) {
> - /*
> - * If uffd is active for the vma, we need per-page fault
> - * fidelity to maintain the uffd semantics, then fallback
> - * to swapin order-0 folio, as well as for zswap case.
> - * Any existing sub folio in the swap cache also blocks
> - * mTHP swapin.
> - */
> - if ((vma && unlikely(userfaultfd_armed(vma))) ||
> - !zswap_never_enabled() ||
> - non_swapcache_batch(entry, nr_pages) != nr_pages)
> - goto fallback;
>
> - alloc_gfp = thp_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
> - }
> -retry:
> - new = shmem_alloc_folio(alloc_gfp, order, info, index);
> - if (!new) {
> - new = ERR_PTR(-ENOMEM);
> - goto fallback;
> - }
> + if ((vmf && unlikely(userfaultfd_armed(vmf->vma))) ||
> + !zswap_never_enabled())
> + orders = 0;
>
> - if (mem_cgroup_swapin_charge_folio(new, vma ? vma->vm_mm : NULL,
> - alloc_gfp, entry)) {
> - folio_put(new);
> - new = ERR_PTR(-ENOMEM);
> - goto fallback;
> - }
> + mpol = shmem_get_pgoff_policy(info, index, order, &ilx);
> + folio = swapin_entry(entry, gfp, orders, vmf, mpol, ilx);
> + mpol_cond_put(mpol);
>
> - swapcache = swapin_folio(entry, new);
> - if (swapcache != new) {
> - folio_put(new);
> - if (!swapcache) {
> - /*
> - * The new folio is charged already, swapin can
> - * only fail due to another raced swapin.
> - */
> - new = ERR_PTR(-EEXIST);
> - goto fallback;
> - }
> - }
> - return swapcache;
> -fallback:
> - /* Order 0 swapin failed, nothing to fallback to, abort */
> - if (!order)
> - return new;
> - entry.val += index - round_down(index, nr_pages);
> - alloc_gfp = gfp;
> - nr_pages = 1;
> - order = 0;
> - goto retry;
> + return folio;
> }
IIUC, in the __swap_cache_alloc() implementation in patch 4, when shmem
swapin falls back to order 0, it doesn't adjust the swap entry value
like here. Because the original swap entry may not correspond to the
swap entry for the order 0 index.
Of course, I haven't tested this yet, just pointing it out for you to
double check.
^ permalink raw reply
* Re: [PATCH v3 04/12] mm, swap: add support for stable large allocation in swap cache directly
From: Kairui Song @ 2026-05-12 9:55 UTC (permalink / raw)
To: Baolin Wang
Cc: linux-mm, Andrew Morton, David Hildenbrand, Zi Yan, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Qi Zheng, linux-kernel, cgroups,
Yosry Ahmed, Lorenzo Stoakes, Dev Jain, Lance Yang, Michal Hocko,
Michal Hocko, Suren Baghdasaryan, Axel Rasmussen
In-Reply-To: <19f31906-d8fb-489b-8e2a-c4414c99f338@linux.alibaba.com>
On Tue, May 12, 2026 at 5:49 PM Baolin Wang
<baolin.wang@linux.alibaba.com> wrote:
>
>
>
> On 4/21/26 2:16 PM, Kairui Song via B4 Relay wrote:
> > From: Kairui Song <kasong@tencent.com>
> >
> > +/**
> > + * swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.
> > + * @targ_entry: swap entry indicating the target slot
> > + * @gfp: memory allocation flags
> > + * @orders: allocation orders
> > + * @vmf: fault information
> > + * @mpol: NUMA memory allocation policy to be applied
> > + * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
> > + *
> > + * Allocate a folio in the swap cache for one swap slot, typically before
> > + * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
> > + * @targ_entry must have a non-zero swap count (swapped out).
> > + *
> > + * Context: Caller must protect the swap device with reference count or locks.
> > + * Return: Returns the folio if allocation succeeded and folio is added to
> > + * swap cache. Returns error code if allocation failed due to race.
> > + */
> > +struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
> > + unsigned long orders, struct vm_fault *vmf,
> > + struct mempolicy *mpol, pgoff_t ilx)
> > +{
> > + int order, err;
> > + struct folio *ret;
> > + struct swap_cluster_info *ci;
> > +
> > + /* Always allow order 0 so swap won't fail under pressure. */
> > + order = orders ? highest_order(orders |= BIT(0)) : 0;
>
> This seems a bit odd here. In THP/mTHP operations, it's usually the
> callers' responsibility to determine the allowable orders. So I think we
> should not implicitly set order 0 here. Instead, we should let callers
> explicitly set it. What do you think?
Totally agree. I hesitated between these two designs.
And Usama also needed this because some callers (PMD swapin) don't
want the fallback.
I'll let the caller explicitly pass in the allowable order in v4.
Thanks for the review!
^ permalink raw reply
* Re: [PATCH v3 04/12] mm, swap: add support for stable large allocation in swap cache directly
From: Baolin Wang @ 2026-05-12 9:48 UTC (permalink / raw)
To: kasong, linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Qi Zheng, linux-kernel, cgroups,
Yosry Ahmed, Lorenzo Stoakes, Dev Jain, Lance Yang, Michal Hocko,
Michal Hocko, Suren Baghdasaryan, Axel Rasmussen
In-Reply-To: <20260421-swap-table-p4-v3-4-2f23759a76bc@tencent.com>
On 4/21/26 2:16 PM, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
>
> To make it possible to allocate large folios directly in swap cache,
> provide a new infrastructure helper to handle the swap cache status
> check, allocation, and order fallback in the swap cache layer
>
> The new helper replaces the existing swap_cache_alloc_folio. Based on
> this, all the separate swap folio allocation that is being done by anon
> / shmem before is converted to use this helper directly, unifying folio
> allocation for anon, shmem, and readahead.
>
> This slightly consolidates how allocation is synchronized, making it
> more stable and less prone to errors. The slot-count and cache-conflict
> check is now always performed with the cluster lock held before
> allocation, and repeated under the same lock right before cache
> insertion. This double check produces a stable result compared to the
> previous anon and shmem mTHP allocation implementation, avoids the
> false-negative conflict checks that the lockless path can return — large
> allocations no longer have to be unwound because the range turned out to
> be occupied — and aborts early for already-freed slots, which helps
> ordinary swapin and especially readahead, with only a marginal increase
> in cluster-lock contention (the lock is very lightly contended and stays
> local in the first place). Hence, callers of swap_cache_alloc_folio() no
> longer need to check the swap slot count or swap cache status
> themselves.
>
> And now whoever first successfully allocates a folio in the swap cache
> will be the one who charges it and performs the swap-in. The race window
> of swapping is also reduced since the loop is much more compact.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
> mm/swap.h | 3 +-
> mm/swap_state.c | 222 +++++++++++++++++++++++++++++++++++++++++---------------
> mm/zswap.c | 2 +-
> 3 files changed, 165 insertions(+), 62 deletions(-)
>
> diff --git a/mm/swap.h b/mm/swap.h
> index ad8b17a93758..6774af10a943 100644
> --- a/mm/swap.h
> +++ b/mm/swap.h
> @@ -280,7 +280,8 @@ bool swap_cache_has_folio(swp_entry_t entry);
> struct folio *swap_cache_get_folio(swp_entry_t entry);
> void *swap_cache_get_shadow(swp_entry_t entry);
> void swap_cache_del_folio(struct folio *folio);
> -struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_flags,
> +struct folio *swap_cache_alloc_folio(swp_entry_t target_entry, gfp_t gfp_mask,
> + unsigned long orders, struct vm_fault *vmf,
> struct mempolicy *mpol, pgoff_t ilx);
> /* Below helpers require the caller to lock and pass in the swap cluster. */
> void __swap_cache_add_folio(struct swap_cluster_info *ci,
> diff --git a/mm/swap_state.c b/mm/swap_state.c
> index 3da285a891b2..f5c77f348bbd 100644
> --- a/mm/swap_state.c
> +++ b/mm/swap_state.c
> @@ -139,10 +139,10 @@ void *swap_cache_get_shadow(swp_entry_t entry)
>
> /**
> * __swap_cache_add_check - Check if a range is suitable for adding a folio.
> - * @ci: The locked swap cluster.
> - * @ci_off: Range start offset.
> - * @nr: Number of slots to check.
> - * @shadow: Returns the shadow value if one exists in the range.
> + * @ci: The locked swap cluster
> + * @targ_entry: The target swap entry to check, will be rounded down by @nr
> + * @nr: Number of slots to check, must be a power of 2
> + * @shadowp: Returns the shadow value if one exists in the range.
> *
> * Check if all slots covered by given range have a swap count >= 1.
> * Retrieves the shadow if there is one.
> @@ -150,22 +150,38 @@ void *swap_cache_get_shadow(swp_entry_t entry)
> * Context: Caller must lock the cluster.
> */
> static int __swap_cache_add_check(struct swap_cluster_info *ci,
> - unsigned int ci_off, unsigned int nr,
> - void **shadow)
> + swp_entry_t targ_entry,
> + unsigned long nr, void **shadowp)
> {
> - unsigned int ci_end = ci_off + nr;
> + unsigned int ci_off, ci_end;
> unsigned long old_tb;
>
> + /*
> + * If the target slot is not swapped out, return
> + * -EEXIST or -ENOENT. If the batch is not suitable, could be a
> + * race with concurrent free or cache add, return -EBUSY.
> + */
> if (unlikely(!ci->table))
> return -ENOENT;
> + ci_off = swp_cluster_offset(targ_entry);
> + old_tb = __swap_table_get(ci, ci_off);
> + if (swp_tb_is_folio(old_tb))
> + return -EEXIST;
> + if (!__swp_tb_get_count(old_tb))
> + return -ENOENT;
> + if (swp_tb_is_shadow(old_tb) && shadowp)
> + *shadowp = swp_tb_to_shadow(old_tb);
> +
> + if (nr == 1)
> + return 0;
> +
> + ci_off = round_down(ci_off, nr);
> + ci_end = ci_off + nr;
> do {
> old_tb = __swap_table_get(ci, ci_off);
> - if (unlikely(swp_tb_is_folio(old_tb)))
> - return -EEXIST;
> - if (unlikely(!__swp_tb_get_count(old_tb)))
> - return -ENOENT;
> - if (swp_tb_is_shadow(old_tb))
> - *shadow = swp_tb_to_shadow(old_tb);
> + if (unlikely(swp_tb_is_folio(old_tb) ||
> + !__swp_tb_get_count(old_tb)))
> + return -EBUSY;
> } while (++ci_off < ci_end);
>
> return 0;
> @@ -244,7 +260,7 @@ static int swap_cache_add_folio(struct folio *folio, swp_entry_t entry,
> si = __swap_entry_to_info(entry);
> ci = swap_cluster_lock(si, swp_offset(entry));
> ci_off = swp_cluster_offset(entry);
> - err = __swap_cache_add_check(ci, ci_off, nr_pages, &shadow);
> + err = __swap_cache_add_check(ci, entry, nr_pages, &shadow);
> if (err) {
> swap_cluster_unlock(ci);
> return err;
> @@ -399,6 +415,137 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,
> }
> }
>
> +/*
> + * Try to allocate a folio of given order in the swap cache.
> + *
> + * This helper resolves the potential races of swap allocation
> + * and prepares a folio to be used for swap IO. May return following
> + * value:
> + *
> + * -ENOMEM / -EBUSY: Order is too large or in conflict with sub slot,
> + * caller should shrink the order and retry
> + * -ENOENT / -EEXIST: Target swap entry is unavailable or cached, the caller
> + * should abort or try to use the cached folio instead
> + */
> +static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
> + swp_entry_t targ_entry, gfp_t gfp,
> + unsigned int order, struct vm_fault *vmf,
> + struct mempolicy *mpol, pgoff_t ilx)
> +{
> + int err;
> + swp_entry_t entry;
> + struct folio *folio;
> + void *shadow = NULL;
> + unsigned long address, nr_pages = 1 << order;
> + struct vm_area_struct *vma = vmf ? vmf->vma : NULL;
> +
> + entry.val = round_down(targ_entry.val, nr_pages);
> +
> + /* Check if the slot and range are available, skip allocation if not */
> + spin_lock(&ci->lock);
> + err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL);
> + spin_unlock(&ci->lock);
> + if (unlikely(err))
> + return ERR_PTR(err);
> +
> + /*
> + * Limit THP gfp. The limitation is a no-op for typical
> + * GFP_HIGHUSER_MOVABLE but matters for shmem.
> + */
> + if (order)
> + gfp = thp_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
> +
> + if (mpol || !vmf) {
> + folio = folio_alloc_mpol(gfp, order, mpol, ilx, numa_node_id());
> + } else {
> + address = round_down(vmf->address, PAGE_SIZE << order);
> + folio = vma_alloc_folio(gfp, order, vmf->vma, address);
> + }
> + if (unlikely(!folio))
> + return ERR_PTR(-ENOMEM);
> +
> + /* Double check the range is still not in conflict */
> + spin_lock(&ci->lock);
> + err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow);
> + if (unlikely(err)) {
> + spin_unlock(&ci->lock);
> + folio_put(folio);
> + return ERR_PTR(err);
> + }
> +
> + __folio_set_locked(folio);
> + __folio_set_swapbacked(folio);
> + __swap_cache_do_add_folio(ci, folio, entry);
> + spin_unlock(&ci->lock);
> +
> + if (mem_cgroup_swapin_charge_folio(folio, vmf ? vmf->vma->vm_mm : NULL,
> + gfp, entry)) {
> + spin_lock(&ci->lock);
> + __swap_cache_do_del_folio(ci, folio, entry, shadow);
> + spin_unlock(&ci->lock);
> + folio_unlock(folio);
> + /* nr_pages refs from swap cache, 1 from allocation */
> + folio_put_refs(folio, nr_pages + 1);
> + count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
> + return ERR_PTR(-ENOMEM);
> + }
> +
> + /* For memsw accounting, swap is uncharged when folio is added to swap cache */
> + memcg1_swapin(entry, 1 << order);
> + if (shadow)
> + workingset_refault(folio, shadow);
> +
> + node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
> + lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
> +
> + /* Caller will initiate read into locked new_folio */
> + folio_add_lru(folio);
> + return folio;
> +}
> +
> +/**
> + * swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.
> + * @targ_entry: swap entry indicating the target slot
> + * @gfp: memory allocation flags
> + * @orders: allocation orders
> + * @vmf: fault information
> + * @mpol: NUMA memory allocation policy to be applied
> + * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
> + *
> + * Allocate a folio in the swap cache for one swap slot, typically before
> + * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
> + * @targ_entry must have a non-zero swap count (swapped out).
> + *
> + * Context: Caller must protect the swap device with reference count or locks.
> + * Return: Returns the folio if allocation succeeded and folio is added to
> + * swap cache. Returns error code if allocation failed due to race.
> + */
> +struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
> + unsigned long orders, struct vm_fault *vmf,
> + struct mempolicy *mpol, pgoff_t ilx)
> +{
> + int order, err;
> + struct folio *ret;
> + struct swap_cluster_info *ci;
> +
> + /* Always allow order 0 so swap won't fail under pressure. */
> + order = orders ? highest_order(orders |= BIT(0)) : 0;
This seems a bit odd here. In THP/mTHP operations, it's usually the
callers' responsibility to determine the allowable orders. So I think we
should not implicitly set order 0 here. Instead, we should let callers
explicitly set it. What do you think?
diff --git a/mm/shmem.c b/mm/shmem.c
index f0da10054620..fb05daeab59a 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2023,7 +2023,8 @@ static struct folio *shmem_swap_alloc_folio(struct
inode *inode,
pgoff_t ilx;
struct folio *folio;
struct mempolicy *mpol;
- unsigned long orders = BIT(order);
+ /* Always allow order 0 so swap won't fail under pressure. */
+ unsigned long orders = BIT(order) | BIT(0);
struct shmem_inode_info *info = SHMEM_I(inode);
if ((vmf && unlikely(userfaultfd_armed(vmf->vma))) ||
^ permalink raw reply related
* Re: [PATCH v2 08/10] sched/fair: Add newidle balance to pick_task_fair()
From: Peter Zijlstra @ 2026-05-12 9:45 UTC (permalink / raw)
To: K Prateek Nayak
Cc: mingo, longman, chenridong, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, tj, hannes,
mkoutny, cgroups, linux-kernel, jstultz, qyousef
In-Reply-To: <8c4d8466-eefc-45a8-a93c-bb66d5e9db83@amd.com>
On Tue, May 12, 2026 at 11:07:13AM +0530, K Prateek Nayak wrote:
> Hello Peter,
>
> On 5/11/2026 5:01 PM, Peter Zijlstra wrote:
> > @@ -9245,6 +9247,14 @@ static struct task_struct *pick_task_fai
> > if (unlikely(throttled))
> > task_throttle_setup_work(p);
> > return p;
> > +
> > +idle:
> > + new_tasks = sched_balance_newidle(rq, rf);
> > + if (new_tasks < 0)
> > + return RETRY_TASK;
> > + if (new_tasks > 0)
> > + goto again;
> > + return NULL;
> > }
>
> For core scheduling will now trigger a newidle balance during the pick
> when core_cookie is reset to 0 which can cause tasks to migrate only
> for them to find they cannot run on the CPU since core-wide selection
> leads to a cookie mismatch and it is kept hanging there.
>
> Can we return early if sched_core_enabled() here or are the additional
> newidle balance okay?
This basically makes fair behave like every other class, so in that sense
this is probably okay. That said, fair is the most common case, so
perhaps.
Lets see if the people actually using this notice first though ;-)
^ permalink raw reply
* Re: [PATCH 2/3] mm/zswap: Implement proactive writeback
From: Hao Jia @ 2026-05-12 9:37 UTC (permalink / raw)
To: Nhat Pham
Cc: akpm, tj, hannes, shakeel.butt, mhocko, yosry, mkoutny,
chengming.zhou, muchun.song, roman.gushchin, cgroups, linux-mm,
linux-kernel, linux-doc, Hao Jia
In-Reply-To: <CAKEwX=PW2+EN41ANutv4cv+iM+JpwV5V+NSp5ukAt0M6fbHFLg@mail.gmail.com>
On 2026/5/12 03:54, Nhat Pham wrote:
> On Mon, May 11, 2026 at 3:52 AM Hao Jia <jiahao.kernel@gmail.com> wrote:
>> diff --git a/mm/zswap.c b/mm/zswap.c
>> index 19538d6f169a..1173ac6836fa 100644
>> --- a/mm/zswap.c
>> +++ b/mm/zswap.c
>> @@ -36,6 +36,7 @@
>> #include <linux/workqueue.h>
>> #include <linux/list_lru.h>
>> #include <linux/zsmalloc.h>
>> +#include <linux/timekeeping.h>
>>
>> #include "swap.h"
>> #include "internal.h"
>> @@ -160,6 +161,12 @@ struct zswap_pool {
>> char tfm_name[CRYPTO_MAX_ALG_NAME];
>> };
>>
>> +struct zswap_shrink_walk_arg {
>> + ktime_t cutoff_time;
>> + bool proactive;
>> + bool encountered_page_in_swapcache;
>> +};
>> +
>> /* Global LRU lists shared by all zswap pools. */
>> static struct list_lru zswap_list_lru;
>>
>> @@ -183,6 +190,7 @@ static struct shrinker *zswap_shrinker;
>> * handle - zsmalloc allocation handle that stores the compressed page data
>> * objcg - the obj_cgroup that the compressed memory is charged to
>> * lru - handle to the pool's lru used to evict pages.
>> + * store_time - Time when the entry was stored, for proactive writeback.
>> */
>> struct zswap_entry {
>> swp_entry_t swpentry;
>> @@ -192,6 +200,7 @@ struct zswap_entry {
>> unsigned long handle;
>> struct obj_cgroup *objcg;
>> struct list_head lru;
>> + ktime_t store_time;
>
> On the implementation side - will this blow up struct zswap_entry
> memory footprint? If so, can you guard this behind a CONFIG option, if
> we are to go this route?
Thanks for the review. I'll address this in v2.
Thanks,
Hao
^ permalink raw reply
* Re: [PATCH 2/3] mm/zswap: Implement proactive writeback
From: Hao Jia @ 2026-05-12 9:32 UTC (permalink / raw)
To: Yosry Ahmed, Nhat Pham
Cc: akpm, tj, hannes, shakeel.butt, mhocko, mkoutny, chengming.zhou,
muchun.song, roman.gushchin, cgroups, linux-mm, linux-kernel,
linux-doc, Hao Jia
In-Reply-To: <CAO9r8zNOPdpJuTmccvQ6ZAVS+tXxp-_ofA765DbnfaUZOPPO-g@mail.gmail.com>
On 2026/5/12 03:57, Yosry Ahmed wrote:
> On Mon, May 11, 2026 at 12:49 PM Nhat Pham <nphamcs@gmail.com> wrote:
>>
>> On Mon, May 11, 2026 at 3:52 AM Hao Jia <jiahao.kernel@gmail.com> wrote:
>>>
>>> From: Hao Jia <jiahao1@lixiang.com>
>>>
>>> Zswap currently writes back pages to backing swap devices reactively,
>>> triggered either by memory pressure via the shrinker or by the pool
>>> reaching its size limit. This reactive approach offers no precise
>>> control over when writeback happens, which can disturb latency-sensitive
>>> workloads, and it cannot direct writeback at a specific memory cgroup.
>>> However, there are scenarios where users might want to proactively
>>> write back cold pages from zswap to the backing swap device, for
>>> example, to free up memory for other applications or to prepare for
>>> upcoming memory-intensive workloads.
>>>
>>> Therefore, implement a proactive writeback mechanism for zswap by
>>> adding a new cgroup interface file memory.zswap.proactive_writeback
>>> within the memory controller.
>>
Thanks Nhat, Yosry — let me address both comments together.
>>
>> We already have memory.reclaim, no? Would that not work to create
>> headroom generally for your use case? Is there a reason why we are
>> treating zswap memory as special here?
>
Apologies for the lack of detailed explanation in the patch description,
which led to the confusion.
While we are already utilizing memory.reclaim, it does not fully address
our requirements.
Our deployment runs a userspace proactive reclaimer that drives
memory.reclaim based on the system's runtime state (memory/CPU/IO
pressure, refault rate, ...) and workload-specific
policy. That first stage compresses cold anon pages into zswap. Entries
that then remain in zswap past a policy-defined age threshold are
considered "twice cold", and the reclaimer wants
to write them back to the backing swap device at a moment of its own
choosing, to further reclaim the DRAM still held by the compressed data.
This is the "second-level offloading" pattern described in Meta's TMO
paper [1]. zswap proactive writeback is what this series introduces to
address that second-level offloading stage.
[1] https://www.pdl.cmu.edu/ftp/NVM/tmo_asplos22.pdf
> +1, why do we need to specifically proactively reclaim the compressed memory?
>
> Also, if we do need to minimize the compressed memory and force higher
> writeback rates, we can do so with memory.zswap.max, right?
Here are a few reasons why memory.zswap.max is not enough:
1. Writing memory.zswap.max itself does not trigger any writeback
immediately. For a memcg that has reached steady state (on which the
userspace reclaimer is no longer invoking
memory.reclaim), after enough time has passed, the reclaimer has no good
way to trigger proactive writeback for second-level offloading by
lowering memory.zswap.max, because in steady
state nothing drives the zswap_store() -> shrink_memcg() path. The
userspace reclaimer still has no control over when proactive writeback
happens.
2. memory.zswap.max currently triggers zswap writeback via zswap_store()
-> shrink_memcg(), and each over-limit event can write back at most
NR_NODES entries. If zswap residency is far
above memory.zswap.max, converging to the target size requires at least
O(over-limit pages / NR_NODES) zswap_store() events, with no batching —
proactive writeback therefore has
significant latency.
3. memory.zswap.max is a stateful interface. If the userspace reclaimer
crashes for any reason mid-operation, it may leave memory.zswap.max at
some set value, putting the application in a
persistently throttled bad state.
4. Once the userspace reclaimer has lowered memory.zswap.max, if the
workload is rapidly expanding and triggers memory reclaim via
memory.high / kswapd / etc., the actual amount written
back can exceed what was intended.
Thanks,
Hao
^ permalink raw reply
* Re: [PATCH v2 00/10] sched: Flatten the pick
From: Peter Zijlstra @ 2026-05-12 9:20 UTC (permalink / raw)
To: Vincent Guittot
Cc: mingo, longman, chenridong, juri.lelli, dietmar.eggemann, rostedt,
bsegall, mgorman, vschneid, tj, hannes, mkoutny, cgroups,
linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <CAKfTPtA2aBtuBffVV02VgsRRi5mRK0G5ununzuvJ7h7buygNxg@mail.gmail.com>
On Tue, May 12, 2026 at 10:42:33AM +0200, Vincent Guittot wrote:
>
> I haven't reviewed the patches yet but I ran some tests with it while
> testing sched latency related changes for short slice wakeup
> preemption. I have some large hackbench regressions with this series
> on HMP system with and without EAS. those figures are unexpected
> because the benchs run on root cfs
>
> One example with hackbench 8 groups thread pipe
> tip/sched/core tip/sched/core +this patchset +this patchset
> slice 2.8ms 16ms 2.8ms 16ms
> dragonboard rb5 with EAS
> 0,748(+/-4,6%) 0,621(+/-3.6%) +17% 1,915(+/-7.9%) -156%
> 0,689(+/- 9.1%) +8%
>
> radxa orion6 HMP without EAS
> 0,588(+/-5.8%) 0,677(+/-5.9%) -15% 1,505(+/-10%) -156%
> 1,071(+/-5.9%) -82%
>
> Increasing the slice partly removes regressions but tis is surprising
> because the bench runs at root cfs and I thought that results will not
> change in such a case
>
> I will review the patchset and try to get what is going wrong
Yeah, that is unexpected. Let me go have another look too.
^ permalink raw reply
* [PATCH RFC 5/5] selftests/dmabuf-heaps: Add dma-buf memcg accounting tests
From: Albert Esteve @ 2026-05-12 9:10 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Jonathan Corbet,
Shuah Khan, Sumit Semwal, Christian König, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Benjamin Gaignard, Brian Starkey, John Stultz, T.J. Mercier,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan
Cc: cgroups, linux-doc, linux-kernel, linux-media, dri-devel,
linaro-mm-sig, linux-mm, linux-security-module, selinux,
linux-kselftest, Albert Esteve, mripard, echanude
In-Reply-To: <20260512-v2_20230123_tjmercier_google_com-v1-0-6326701c3691@redhat.com>
Add tests for the new charge_pid_fd field in struct
dma_heap_allocation_data.
When the charge_pid_fd feature is absent (unpatched kernel),
the probe in pidfd_alloc_supported() detects this and the
tests are skipped gracefully.
Add vmtest.sh similar to other subsystem suites, to orchestrate
building the selftests (optionally with a freshly compiled kernel)
inside a virtme-ng VM, so the tests can be run without modifying
the host system. Add a config fragment with required Kconfig symbols.
Also add test_memcg_dmabuf() to the existing test_memcontrol suite
to verify end-to-end cross-cgroup accounting: a parent process opens
a pidfd for a child in a separate cgroup, allocates a dma-buf via
DMA_HEAP_IOCTL_ALLOC with that pidfd, and asserts that memory.stat
dmabuf in the child's cgroup reflects the allocation. If the dmabuf
key is missing (unpatched kernel) or /dev/dma_heap/system is absent,
the test is skipped.
Assisted-by: Claude:claude-sonnet-4-6 Cursor
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
tools/testing/selftests/cgroup/Makefile | 2 +-
tools/testing/selftests/cgroup/test_memcontrol.c | 143 +++++++++++++-
tools/testing/selftests/dmabuf-heaps/config | 1 +
tools/testing/selftests/dmabuf-heaps/dmabuf-heap.c | 126 ++++++++++++-
tools/testing/selftests/dmabuf-heaps/vmtest.sh | 205 +++++++++++++++++++++
5 files changed, 473 insertions(+), 4 deletions(-)
diff --git a/tools/testing/selftests/cgroup/Makefile b/tools/testing/selftests/cgroup/Makefile
index e01584c2189ac..9edfc9f1de5c4 100644
--- a/tools/testing/selftests/cgroup/Makefile
+++ b/tools/testing/selftests/cgroup/Makefile
@@ -1,5 +1,5 @@
# SPDX-License-Identifier: GPL-2.0
-CFLAGS += -Wall -pthread
+CFLAGS += -Wall -pthread $(KHDR_INCLUDES)
all: ${HELPER_PROGS}
diff --git a/tools/testing/selftests/cgroup/test_memcontrol.c b/tools/testing/selftests/cgroup/test_memcontrol.c
index b43da9bc20c49..b6a228407530f 100644
--- a/tools/testing/selftests/cgroup/test_memcontrol.c
+++ b/tools/testing/selftests/cgroup/test_memcontrol.c
@@ -19,9 +19,17 @@
#include <errno.h>
#include <sys/mman.h>
+#include <linux/dma-heap.h>
+#include <signal.h>
+#include <sys/ioctl.h>
+
+#include "../pidfd/pidfd.h"
#include "kselftest.h"
#include "cgroup_util.h"
+#define DMA_HEAP_SYSTEM "/dev/dma_heap/system"
+#define ONE_MEG (1024 * 1024)
+
#define MEMCG_SOCKSTAT_WAIT_RETRIES 30
static bool has_localevents;
@@ -1762,6 +1770,125 @@ static int test_memcg_inotify_delete_dir(const char *root)
return ret;
}
+static int memcg_dmabuf_child(const char *cgroup, void *arg)
+{
+ pause();
+ return 0;
+}
+
+/*
+ * This test allocates a dma-buf via DMA_HEAP_IOCTL_ALLOC with a pidfd
+ * pointing to a child process in a separate cgroup, then checks that
+ * memory.stat[dmabuf] in the child's cgroup rises by the allocation size
+ * and returns to zero after the buffer fd is closed.
+ */
+static int test_memcg_dmabuf(const char *root)
+{
+ char *parent = NULL, *child_cg = NULL;
+ int ret = KSFT_FAIL;
+ int heap_fd = -1, dmabuf_fd = -1, pidfd = -1;
+ pid_t child_pid;
+ int child_status;
+ long dmabuf_stat;
+ struct dma_heap_allocation_data alloc = {
+ .len = ONE_MEG,
+ .fd_flags = O_RDWR | O_CLOEXEC,
+ };
+
+ if (access(DMA_HEAP_SYSTEM, R_OK | W_OK)) {
+ ret = KSFT_SKIP;
+ goto cleanup;
+ }
+
+ parent = cg_name(root, "dmabuf_memcg_test");
+ if (!parent)
+ goto cleanup;
+
+ if (cg_create(parent))
+ goto cleanup_parent;
+
+ if (cg_write(parent, "cgroup.subtree_control", "+memory"))
+ goto cleanup_parent;
+
+ child_cg = cg_name(parent, "child");
+ if (!child_cg)
+ goto cleanup_parent;
+
+ if (cg_create(child_cg))
+ goto cleanup_parent;
+
+ child_pid = cg_run_nowait(child_cg, memcg_dmabuf_child, NULL);
+ if (child_pid < 0)
+ goto cleanup_child;
+
+ if (cg_wait_for_proc_count(child_cg, 1))
+ goto cleanup_kill;
+
+ pidfd = sys_pidfd_open(child_pid, 0);
+ if (pidfd < 0) {
+ ret = KSFT_SKIP;
+ goto cleanup_kill;
+ }
+
+ heap_fd = open(DMA_HEAP_SYSTEM, O_RDWR);
+ if (heap_fd < 0) {
+ ret = KSFT_SKIP;
+ goto cleanup_pidfd;
+ }
+
+ alloc.charge_pid_fd = (__u32)pidfd;
+ if (ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc) < 0)
+ goto cleanup_heap;
+ dmabuf_fd = (int)alloc.fd;
+
+ dmabuf_stat = cg_read_key_long(child_cg, "memory.stat", "dmabuf ");
+ if (dmabuf_stat == -1) {
+ ret = KSFT_SKIP;
+ goto cleanup_dmabuf;
+ }
+ if (dmabuf_stat != ONE_MEG)
+ dmabuf_stat = cg_read_key_long_poll(child_cg, "memory.stat",
+ "dmabuf ", ONE_MEG,
+ 15, 200000);
+ if (dmabuf_stat != ONE_MEG) {
+ fprintf(stderr, "Expected dmabuf stat %d, got %ld\n",
+ ONE_MEG, dmabuf_stat);
+ goto cleanup_dmabuf;
+ }
+
+ close(dmabuf_fd);
+ dmabuf_fd = -1;
+
+ dmabuf_stat = cg_read_key_long_poll(child_cg, "memory.stat",
+ "dmabuf ", 0, 15, 200000);
+ if (dmabuf_stat != 0) {
+ fprintf(stderr, "Expected dmabuf stat 0 after close, got %ld\n",
+ dmabuf_stat);
+ goto cleanup_heap;
+ }
+
+ ret = KSFT_PASS;
+
+cleanup_dmabuf:
+ if (dmabuf_fd >= 0)
+ close(dmabuf_fd);
+cleanup_heap:
+ close(heap_fd);
+cleanup_pidfd:
+ close(pidfd);
+cleanup_kill:
+ kill(child_pid, SIGTERM);
+ waitpid(child_pid, &child_status, 0);
+cleanup_child:
+ cg_destroy(child_cg);
+ free(child_cg);
+cleanup_parent:
+ cg_destroy(parent);
+ free(parent);
+cleanup:
+ return ret;
+}
+
#define T(x) { x, #x }
struct memcg_test {
int (*fn)(const char *root);
@@ -1783,16 +1910,26 @@ struct memcg_test {
T(test_memcg_oom_group_score_events),
T(test_memcg_inotify_delete_file),
T(test_memcg_inotify_delete_dir),
+ T(test_memcg_dmabuf),
};
#undef T
int main(int argc, char **argv)
{
char root[PATH_MAX];
- int i, proc_status;
+ int i, proc_status, plan;
+ const char *filter = NULL;
+
+ if (argc > 1)
+ filter = argv[1];
+
+ plan = 0;
+ for (i = 0; i < ARRAY_SIZE(tests); i++)
+ if (!filter || !strcmp(tests[i].name, filter))
+ plan++;
ksft_print_header();
- ksft_set_plan(ARRAY_SIZE(tests));
+ ksft_set_plan(plan);
if (cg_find_unified_root(root, sizeof(root), NULL))
ksft_exit_skip("cgroup v2 isn't mounted\n");
@@ -1818,6 +1955,8 @@ int main(int argc, char **argv)
has_localevents = proc_status;
for (i = 0; i < ARRAY_SIZE(tests); i++) {
+ if (filter && strcmp(tests[i].name, filter))
+ continue;
switch (tests[i].fn(root)) {
case KSFT_PASS:
ksft_test_result_pass("%s\n", tests[i].name);
diff --git a/tools/testing/selftests/dmabuf-heaps/config b/tools/testing/selftests/dmabuf-heaps/config
index be091f1cdfa04..94c8f33b71a28 100644
--- a/tools/testing/selftests/dmabuf-heaps/config
+++ b/tools/testing/selftests/dmabuf-heaps/config
@@ -1,3 +1,4 @@
+CONFIG_MEMCG=y
CONFIG_DMABUF_HEAPS=y
CONFIG_DMABUF_HEAPS_SYSTEM=y
CONFIG_DRM_VGEM=y
diff --git a/tools/testing/selftests/dmabuf-heaps/dmabuf-heap.c b/tools/testing/selftests/dmabuf-heaps/dmabuf-heap.c
index fc9694fc4e89e..904332b17698a 100644
--- a/tools/testing/selftests/dmabuf-heaps/dmabuf-heap.c
+++ b/tools/testing/selftests/dmabuf-heaps/dmabuf-heap.c
@@ -3,6 +3,7 @@
#include <dirent.h>
#include <errno.h>
#include <fcntl.h>
+#include <signal.h>
#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
@@ -10,11 +11,14 @@
#include <unistd.h>
#include <sys/ioctl.h>
#include <sys/mman.h>
+#include <sys/syscall.h>
#include <sys/types.h>
+#include <sys/wait.h>
#include <linux/dma-buf.h>
#include <linux/dma-heap.h>
#include <drm/drm.h>
+#include "../pidfd/pidfd.h"
#include "kselftest.h"
#define DEVPATH "/dev/dma_heap"
@@ -320,6 +324,8 @@ static int dmabuf_heap_alloc_newer(int fd, size_t len, unsigned int flags,
__u32 fd;
__u32 fd_flags;
__u64 heap_flags;
+ __u32 charge_pid_fd;
+ __u32 __padding;
__u64 garbage1;
__u64 garbage2;
__u64 garbage3;
@@ -328,6 +334,8 @@ static int dmabuf_heap_alloc_newer(int fd, size_t len, unsigned int flags,
.fd = 0,
.fd_flags = O_RDWR | O_CLOEXEC,
.heap_flags = flags,
+ .charge_pid_fd = 0,
+ .__padding = 0,
.garbage1 = 0xffffffff,
.garbage2 = 0x88888888,
.garbage3 = 0x11111111,
@@ -390,6 +398,120 @@ static void test_alloc_errors(char *heap_name)
close(heap_fd);
}
+static int dmabuf_heap_alloc_pidfd(int fd, size_t len, unsigned int heap_flags,
+ unsigned int charge_pid_fd, int *dmabuf_fd)
+{
+ struct dma_heap_allocation_data data = {
+ .len = len,
+ .fd = 0,
+ .fd_flags = O_RDWR | O_CLOEXEC,
+ .heap_flags = heap_flags,
+ .charge_pid_fd = charge_pid_fd,
+ };
+ int ret;
+
+ if (!dmabuf_fd)
+ return -EINVAL;
+
+ ret = ioctl(fd, DMA_HEAP_IOCTL_ALLOC, &data);
+ if (ret < 0)
+ return ret;
+ *dmabuf_fd = (int)data.fd;
+ return ret;
+}
+
+/*
+ * Probe whether the kernel honours charge_pid_fd in DMA_HEAP_IOCTL_ALLOC.
+ */
+static bool pidfd_alloc_supported(int heap_fd)
+{
+ int devnull_fd, dmabuf_fd = -1, ret;
+
+ devnull_fd = open("/dev/null", O_RDONLY);
+ if (devnull_fd < 0)
+ return false;
+
+ ret = dmabuf_heap_alloc_pidfd(heap_fd, ONE_MEG, 0, devnull_fd, &dmabuf_fd);
+ if (dmabuf_fd >= 0) {
+ close(dmabuf_fd);
+ dmabuf_fd = -1;
+ }
+ close(devnull_fd);
+ return ret < 0;
+}
+
+/*
+ * Test: allocate charging the calling process's own cgroup via a self pidfd.
+ */
+static void test_alloc_pidfd_self(char *heap_name)
+{
+ int heap_fd = -1, pidfd = -1, dmabuf_fd = -1, ret;
+
+ heap_fd = dmabuf_heap_open(heap_name);
+
+ if (!pidfd_alloc_supported(heap_fd)) {
+ ksft_test_result_skip("charge_pid_fd not supported by this kernel\n");
+ goto out;
+ }
+
+ pidfd = sys_pidfd_open(getpid(), 0);
+ if (pidfd < 0) {
+ ksft_test_result_skip("pidfd_open not available\n");
+ goto out;
+ }
+
+ ret = dmabuf_heap_alloc_pidfd(heap_fd, ONE_MEG, 0, pidfd, &dmabuf_fd);
+ ksft_test_result(!ret, "Allocation with self pidfd %d\n", ret);
+ if (dmabuf_fd >= 0)
+ close(dmabuf_fd);
+ close(pidfd);
+out:
+ close(heap_fd);
+}
+
+/*
+ * Test: allocate charging a child process's cgroup via a child pidfd.
+ */
+static void test_alloc_pidfd_child(char *heap_name)
+{
+ int heap_fd = -1, pidfd = -1, dmabuf_fd = -1;
+ pid_t child_pid;
+ int status, ret;
+
+ heap_fd = dmabuf_heap_open(heap_name);
+
+ if (!pidfd_alloc_supported(heap_fd)) {
+ ksft_test_result_skip("charge_pid_fd not supported by this kernel\n");
+ goto out;
+ }
+
+ child_pid = fork();
+ if (child_pid == 0) {
+ pause();
+ _exit(0);
+ }
+ if (child_pid < 0)
+ ksft_exit_fail_msg("fork failed: %s\n", strerror(errno));
+
+ pidfd = sys_pidfd_open(child_pid, 0);
+ if (pidfd < 0) {
+ kill(child_pid, SIGTERM);
+ waitpid(child_pid, &status, 0);
+ ksft_test_result_skip("pidfd_open for child failed\n");
+ goto out;
+ }
+
+ ret = dmabuf_heap_alloc_pidfd(heap_fd, ONE_MEG, 0, pidfd, &dmabuf_fd);
+ ksft_test_result(!ret, "Allocation with child pidfd %d\n", ret);
+ if (dmabuf_fd >= 0)
+ close(dmabuf_fd);
+ close(pidfd);
+ kill(child_pid, SIGTERM);
+ waitpid(child_pid, &status, 0);
+out:
+ close(heap_fd);
+}
+
static int numer_of_heaps(void)
{
DIR *d = opendir(DEVPATH);
@@ -420,7 +542,7 @@ int main(void)
return KSFT_SKIP;
}
- ksft_set_plan(11 * numer_of_heaps());
+ ksft_set_plan(13 * numer_of_heaps());
while ((dir = readdir(d))) {
if (!strncmp(dir->d_name, ".", 2))
@@ -435,6 +557,8 @@ int main(void)
test_alloc_zeroed(dir->d_name, ONE_MEG);
test_alloc_compat(dir->d_name);
test_alloc_errors(dir->d_name);
+ test_alloc_pidfd_self(dir->d_name);
+ test_alloc_pidfd_child(dir->d_name);
}
closedir(d);
diff --git a/tools/testing/selftests/dmabuf-heaps/vmtest.sh b/tools/testing/selftests/dmabuf-heaps/vmtest.sh
new file mode 100755
index 0000000000000..6f1a878384127
--- /dev/null
+++ b/tools/testing/selftests/dmabuf-heaps/vmtest.sh
@@ -0,0 +1,205 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# Copyright (c) 2026 Red Hat
+#
+# Dependencies:
+# * virtme-ng
+# * qemu (used by virtme-ng)
+
+readonly SCRIPT_DIR="$(cd -P -- "$(dirname -- "${BASH_SOURCE[0]}")" && pwd -P)"
+readonly KERNEL_CHECKOUT=$(realpath "${SCRIPT_DIR}"/../../../../)
+readonly CGROUP_DIR="${KERNEL_CHECKOUT}/tools/testing/selftests/cgroup"
+
+source "${SCRIPT_DIR}"/../kselftest/ktap_helpers.sh
+
+readonly DMABUF_HEAP_TEST="${SCRIPT_DIR}"/dmabuf-heap
+readonly MEMCONTROL_TEST="${CGROUP_DIR}"/test_memcontrol
+readonly TMP_DIR=$(mktemp -d /tmp/dmabuf-vmtest.XXXXXXXX)
+
+VERBOSE=false
+BUILD=false
+BUILD_HOST=""
+BUILD_HOST_PODMAN_CONTAINER_NAME=""
+
+usage() {
+ echo
+ echo "$0 [OPTIONS]"
+ echo
+ echo "Options"
+ echo " -b: build the kernel from the current source tree and use it for the VM"
+ echo " -H: hostname for remote build host (used with -b)"
+ echo " -p: podman container name for remote build host (used with -b)"
+ echo " Example: -H beefyserver -p vng"
+
+ echo " -v: enable verbose vng/qemu output"
+ echo
+
+ exit 1
+}
+
+die() {
+ echo "$*" >&2
+ exit "${KSFT_FAIL}"
+}
+
+cleanup() {
+ rm -rf "${TMP_DIR}"
+}
+
+check_deps() {
+ for dep in vng make; do
+ if [[ ! -x $(command -v "${dep}") ]]; then
+ echo -e "skip: dependency ${dep} not found!\n"
+ exit "${KSFT_SKIP}"
+ fi
+ done
+
+ if [[ ! -x "${DMABUF_HEAP_TEST}" ]]; then
+ printf "skip: %s not found!" "${DMABUF_HEAP_TEST}"
+ printf " Please build the kselftest dmabuf-heaps target (or use -b).\n"
+ exit "${KSFT_SKIP}"
+ fi
+
+ if [[ ! -x "${MEMCONTROL_TEST}" ]]; then
+ printf "skip: %s not found!" "${MEMCONTROL_TEST}"
+ printf " Please build the kselftest cgroup target (or use -b).\n"
+ exit "${KSFT_SKIP}"
+ fi
+}
+
+check_vng() {
+ local tested_versions=("1.36" "1.37")
+ local version
+ local ok=0
+
+ version="$(vng --version)"
+ for tv in "${tested_versions[@]}"; do
+ if [[ "${version}" == *"${tv}"* ]]; then
+ ok=1
+ break
+ fi
+ done
+
+ if [[ "${ok}" -eq 0 ]]; then
+ printf "warning: vng version '%s' has not been tested and may " "${version}" >&2
+ printf "not function properly.\n\tThe following versions have been tested: " >&2
+ echo "${tested_versions[@]}" >&2
+ fi
+}
+
+build_selftests() {
+ make -C "${KERNEL_CHECKOUT}" headers_install \
+ INSTALL_HDR_PATH="${TMP_DIR}/usr" -j"$(nproc)"
+
+ local khdr="-isystem ${TMP_DIR}/usr/include"
+
+ if ! make -C "${SCRIPT_DIR}" KHDR_INCLUDES="${khdr}" -j"$(nproc)"; then
+ die "failed to build dmabuf-heaps selftests"
+ fi
+
+ if ! make -C "${CGROUP_DIR}" KHDR_INCLUDES="${khdr}" \
+ "${MEMCONTROL_TEST}" -j"$(nproc)"; then
+ die "failed to build cgroup/test_memcontrol selftest"
+ fi
+}
+
+handle_build() {
+ if ! ${BUILD}; then
+ return
+ fi
+
+ if [[ ! -d "${KERNEL_CHECKOUT}" ]]; then
+ echo "-b requires vmtest.sh called from the kernel source tree" >&2
+ exit 1
+ fi
+
+ pushd "${KERNEL_CHECKOUT}" &>/dev/null
+
+ if ! vng --kconfig --config "${SCRIPT_DIR}/config"; then
+ die "failed to generate .config for kernel source tree (${KERNEL_CHECKOUT})"
+ fi
+
+ local vng_args=("-v" "--config" "${SCRIPT_DIR}/config" "--build")
+
+ if [[ -n "${BUILD_HOST}" ]]; then
+ vng_args+=("--build-host" "${BUILD_HOST}")
+ fi
+
+ if [[ -n "${BUILD_HOST_PODMAN_CONTAINER_NAME}" ]]; then
+ vng_args+=("--build-host-exec-prefix" \
+ "podman exec -ti ${BUILD_HOST_PODMAN_CONTAINER_NAME}")
+ fi
+
+ if ! vng "${vng_args[@]}"; then
+ die "failed to build kernel from source tree (${KERNEL_CHECKOUT})"
+ fi
+
+ build_selftests
+
+ popd &>/dev/null
+}
+
+make_runner() {
+ # virtme-ng shares the host filesystem, so TMP_DIR is accessible
+ # inside the VM at the same absolute path.
+ cat > "${TMP_DIR}/run_tests.sh" <<-EOF
+ #!/bin/sh
+ set -u
+ PASS=0; FAIL=0; SKIP=0; N=0
+
+ run() {
+ name="\$1"; shift
+ N=\$((N+1))
+ "\$@"; rc=\$?
+ if [ \$rc -eq 0 ]; then echo "ok \$N \$name"; PASS=\$((PASS+1))
+ elif [ \$rc -eq 4 ]; then echo "ok \$N \$name # SKIP"; SKIP=\$((SKIP+1))
+ else echo "not ok \$N \$name"; FAIL=\$((FAIL+1))
+ fi
+ }
+
+ run "dmabuf-heap charge_pid_fd ioctl" ${DMABUF_HEAP_TEST}
+ run "memcontrol dma-buf memcg" ${MEMCONTROL_TEST} test_memcg_dmabuf
+ echo "# PASS=\$PASS SKIP=\$SKIP FAIL=\$FAIL"
+ [ \$FAIL -eq 0 ]
+ EOF
+ chmod +x "${TMP_DIR}/run_tests.sh"
+}
+
+run_vm() {
+ local verbose_opt=""
+ local kernel_opt=""
+
+ ${VERBOSE} && verbose_opt="--verbose"
+
+ # If we are running from within the kernel source tree, use the kernel
+ # source tree as the kernel to boot, otherwise use the running kernel.
+ if [[ "$(realpath "$(pwd)")" == "${KERNEL_CHECKOUT}"* ]]; then
+ kernel_opt="${KERNEL_CHECKOUT}"
+ fi
+
+ vng --run ${kernel_opt} ${verbose_opt} --user root --memory 512M \
+ --exec "${TMP_DIR}/run_tests.sh"
+}
+
+while getopts :hvbH:p: o
+do
+ case $o in
+ v) VERBOSE=true;;
+ b) BUILD=true;;
+ H) BUILD_HOST=$OPTARG;;
+ p) BUILD_HOST_PODMAN_CONTAINER_NAME=$OPTARG;;
+ h|*) usage;;
+ esac
+done
+shift $((OPTIND-1))
+
+trap cleanup EXIT
+
+check_vng
+handle_build
+check_deps
+make_runner
+
+echo "Booting VM and running tests..."
+run_vm
--
2.53.0
^ permalink raw reply related
* [PATCH RFC 4/5] selinux: Restrict cross-cgroup dma-heap charging
From: Albert Esteve @ 2026-05-12 9:10 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Jonathan Corbet,
Shuah Khan, Sumit Semwal, Christian König, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Benjamin Gaignard, Brian Starkey, John Stultz, T.J. Mercier,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan
Cc: cgroups, linux-doc, linux-kernel, linux-media, dri-devel,
linaro-mm-sig, linux-mm, linux-security-module, selinux,
linux-kselftest, Albert Esteve, mripard, echanude
In-Reply-To: <20260512-v2_20230123_tjmercier_google_com-v1-0-6326701c3691@redhat.com>
The security_dma_heap_alloc() hook allows security modules
to control which processes may charge dma-buf allocations
to another process's cgroup via the charge_pid_fd field of
DMA_HEAP_IOCTL_ALLOC. Without a policy implementation, the
hook is a no-op and the restriction is not enforced.
On SELinux-managed systems any domain with access to a
dma-heap device node can therefore exhaust another cgroup's
memory budget without restriction.
Implement selinux_dma_heap_alloc() using avc_has_perm() with
a new dma_heap object class and a charge_to permission. Policy
authors can then grant cross-cgroup charging selectively,
for example:
allow allocator_app_t client_app_t:dma_heap charge_to;
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
security/selinux/hooks.c | 7 +++++++
security/selinux/include/classmap.h | 1 +
2 files changed, 8 insertions(+)
diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c
index 0f704380a8c81..ea1f410b9f619 100644
--- a/security/selinux/hooks.c
+++ b/security/selinux/hooks.c
@@ -2189,6 +2189,12 @@ static int selinux_capable(const struct cred *cred, struct user_namespace *ns,
return cred_has_capability(cred, cap, opts, ns == &init_user_ns);
}
+static int selinux_dma_heap_alloc(const struct cred *from, const struct cred *to)
+{
+ return avc_has_perm(cred_sid(from), cred_sid(to),
+ SECCLASS_DMA_HEAP, DMA_HEAP__CHARGE_TO, NULL);
+}
+
static int selinux_quotactl(int cmds, int type, int id, const struct super_block *sb)
{
const struct cred *cred = current_cred();
@@ -7541,6 +7547,7 @@ static struct security_hook_list selinux_hooks[] __ro_after_init = {
LSM_HOOK_INIT(capget, selinux_capget),
LSM_HOOK_INIT(capset, selinux_capset),
LSM_HOOK_INIT(capable, selinux_capable),
+ LSM_HOOK_INIT(dma_heap_alloc, selinux_dma_heap_alloc),
LSM_HOOK_INIT(quotactl, selinux_quotactl),
LSM_HOOK_INIT(quota_on, selinux_quota_on),
LSM_HOOK_INIT(syslog, selinux_syslog),
diff --git a/security/selinux/include/classmap.h b/security/selinux/include/classmap.h
index 90cb61b164256..d232f7808f6b8 100644
--- a/security/selinux/include/classmap.h
+++ b/security/selinux/include/classmap.h
@@ -181,6 +181,7 @@ const struct security_class_mapping secclass_map[] = {
{ "user_namespace", { "create", NULL } },
{ "memfd_file",
{ COMMON_FILE_PERMS, "execute_no_trans", "entrypoint", NULL } },
+ { "dma_heap", { "charge_to", NULL } },
/* last one */ { NULL, {} }
};
--
2.53.0
^ permalink raw reply related
* [PATCH RFC 3/5] security: dma-heap: Add dma_heap_alloc LSM hook
From: Albert Esteve @ 2026-05-12 9:10 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Jonathan Corbet,
Shuah Khan, Sumit Semwal, Christian König, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Benjamin Gaignard, Brian Starkey, John Stultz, T.J. Mercier,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan
Cc: cgroups, linux-doc, linux-kernel, linux-media, dri-devel,
linaro-mm-sig, linux-mm, linux-security-module, selinux,
linux-kselftest, Albert Esteve, mripard, echanude
In-Reply-To: <20260512-v2_20230123_tjmercier_google_com-v1-0-6326701c3691@redhat.com>
DMA_HEAP_IOCTL_ALLOC accepts a charge_pid_fd field that,
when set, causes the allocation to be charged to an arbitrary
process's cgroup rather than the caller's.
Without an access-control point, any process that holds a handle
to a dma-heap device node can charge unlimited memory to any other
process's cgroup, potentially exhausting that cgroup's limit and
triggering OOM kills independent of the victim's own activity or
privileges.
Add security_dma_heap_alloc(), called in dma_heap_ioctl_allocate()
when charge_pid_fd refers to another process. The hook receives
the credentials of the allocating process (from) and the credentials
of the process whose cgroup will be charged (to), giving security
modules a controlled enforcement point for cross-cgroup dma-buf
attribution policy.
When CONFIG_SECURITY is not set the hook compiles to an inline
returning 0, adding no overhead to the fast path.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
drivers/dma-buf/dma-heap.c | 12 +++++++++++-
include/linux/lsm_hook_defs.h | 1 +
include/linux/security.h | 7 +++++++
security/security.c | 16 ++++++++++++++++
4 files changed, 35 insertions(+), 1 deletion(-)
diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c
index ff6e259afcdc0..e8ffb1031955e 100644
--- a/drivers/dma-buf/dma-heap.c
+++ b/drivers/dma-buf/dma-heap.c
@@ -18,6 +18,7 @@
#include <linux/list.h>
#include <linux/nospec.h>
#include <linux/pidfd.h>
+#include <linux/security.h>
#include <linux/syscalls.h>
#include <linux/uaccess.h>
#include <linux/xarray.h>
@@ -122,12 +123,13 @@ static int dma_heap_open(struct inode *inode, struct file *file)
static long dma_heap_ioctl_allocate(struct file *file, void *data)
{
+ const struct cred *tcred;
struct dma_heap_allocation_data *heap_allocation = data;
struct dma_heap *heap = file->private_data;
struct mem_cgroup *memcg = NULL;
struct task_struct *task;
unsigned int pidfd_flags;
- int fd;
+ int fd, ret;
if (heap_allocation->fd)
return -EINVAL;
@@ -143,6 +145,14 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
if (IS_ERR(task))
return PTR_ERR(task);
+ tcred = get_task_cred(task);
+ ret = security_dma_heap_alloc(current_cred(), tcred);
+ put_cred(tcred);
+ if (ret) {
+ put_task_struct(task);
+ return ret;
+ }
+
memcg = get_mem_cgroup_from_mm(task->mm);
put_task_struct(task);
}
diff --git a/include/linux/lsm_hook_defs.h b/include/linux/lsm_hook_defs.h
index 2b8dfb35caed3..6a91656f97e1e 100644
--- a/include/linux/lsm_hook_defs.h
+++ b/include/linux/lsm_hook_defs.h
@@ -43,6 +43,7 @@ LSM_HOOK(int, 0, capset, struct cred *new, const struct cred *old,
const kernel_cap_t *permitted)
LSM_HOOK(int, 0, capable, const struct cred *cred, struct user_namespace *ns,
int cap, unsigned int opts)
+LSM_HOOK(int, 0, dma_heap_alloc, const struct cred *from, const struct cred *to)
LSM_HOOK(int, 0, quotactl, int cmds, int type, int id, const struct super_block *sb)
LSM_HOOK(int, 0, quota_on, struct dentry *dentry)
LSM_HOOK(int, 0, syslog, int type)
diff --git a/include/linux/security.h b/include/linux/security.h
index 41d7367cf4036..f1dad1eabe754 100644
--- a/include/linux/security.h
+++ b/include/linux/security.h
@@ -350,6 +350,7 @@ int security_capable(const struct cred *cred,
struct user_namespace *ns,
int cap,
unsigned int opts);
+int security_dma_heap_alloc(const struct cred *from, const struct cred *to);
int security_quotactl(int cmds, int type, int id, const struct super_block *sb);
int security_quota_on(struct dentry *dentry);
int security_syslog(int type);
@@ -701,6 +702,12 @@ static inline int security_capable(const struct cred *cred,
return cap_capable(cred, ns, cap, opts);
}
+static inline int security_dma_heap_alloc(const struct cred *from,
+ const struct cred *to)
+{
+ return 0;
+}
+
static inline int security_quotactl(int cmds, int type, int id,
const struct super_block *sb)
{
diff --git a/security/security.c b/security/security.c
index 4e999f0236516..4adacef73c507 100644
--- a/security/security.c
+++ b/security/security.c
@@ -660,6 +660,22 @@ int security_capable(const struct cred *cred,
return call_int_hook(capable, cred, ns, cap, opts);
}
+/**
+ * security_dma_heap_alloc() - Check if cross-cgroup dma-heap charging is allowed
+ * @from: credentials of the allocating process
+ * @to: credentials of the process to charge
+ *
+ * Check whether the process with credentials @from is allowed to allocate
+ * dma-heap memory and charge it to the cgroup of the process with credentials
+ * @to.
+ *
+ * Return: Returns 0 if permission is granted.
+ */
+int security_dma_heap_alloc(const struct cred *from, const struct cred *to)
+{
+ return call_int_hook(dma_heap_alloc, from, to);
+}
+
/**
* security_quotactl() - Check if a quotactl() syscall is allowed for this fs
* @cmds: commands
--
2.53.0
^ permalink raw reply related
* [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: Albert Esteve @ 2026-05-12 9:10 UTC (permalink / raw)
To: Tejun Heo, Johannes Weiner, Michal Koutný, Jonathan Corbet,
Shuah Khan, Sumit Semwal, Christian König, Michal Hocko,
Roman Gushchin, Shakeel Butt, Muchun Song, Andrew Morton,
Benjamin Gaignard, Brian Starkey, John Stultz, T.J. Mercier,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan
Cc: cgroups, linux-doc, linux-kernel, linux-media, dri-devel,
linaro-mm-sig, linux-mm, linux-security-module, selinux,
linux-kselftest, Albert Esteve, mripard, echanude
In-Reply-To: <20260512-v2_20230123_tjmercier_google_com-v1-0-6326701c3691@redhat.com>
On embedded platforms a central process often allocates dma-buf
memory on behalf of client applications. Without a way to
attribute the charge to the requesting client's cgroup, the
cost lands on the allocator, making per-cgroup memory limits
ineffective for the actual consumers.
Add charge_pid_fd to struct dma_heap_allocation_data. When set to
a valid pidfd, DMA_HEAP_IOCTL_ALLOC resolves the target task's
memcg and charges the buffer there via mem_cgroup_charge_dmabuf()
inside dma_heap_buffer_alloc(). Without charge_pid_fd, and with
the mem_accounting module parameter enabled, the buffer is charged
to the allocator's own cgroup.
Additionally, commit 3c227be90659 ("dma-buf: system_heap: account for
system heap allocation in memcg") adds __GFP_ACCOUNT to system-heap
page allocations. Keeping __GFP_ACCOUNT would charge the same pages
twice (once to kmem, once to MEMCG_DMABUF), thus remove it and route
all accounting through a single MEMCG_DMABUF path.
Usage examples:
1. Central allocator charging to a client at allocation time.
The allocator knows the client's PID (e.g., from binder's
sender_pid) and uses pidfd to attribute the charge:
pid_t client_pid = txn->sender_pid;
int pidfd = pidfd_open(client_pid, 0);
struct dma_heap_allocation_data alloc = {
.len = buffer_size,
.fd_flags = O_RDWR | O_CLOEXEC,
.charge_pid_fd = pidfd,
};
ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
close(pidfd);
/* alloc.fd is now charged to client's cgroup */
2. Default allocation (no pidfd, mem_accounting=1).
When charge_pid_fd is not set and the mem_accounting module
parameter is enabled, the buffer is charged to the allocator's
own cgroup:
struct dma_heap_allocation_data alloc = {
.len = buffer_size,
.fd_flags = O_RDWR | O_CLOEXEC,
};
ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
/* charged to current process's cgroup */
Current limitations:
- Single-owner model: a dma-buf carries one memcg charge regardless of
how many processes share it. Means only the first owner (and exporter)
of the shared buffer bears the charge.
- Only memcg accounting supported. While this makes sense for system
heap buffers, other heaps (e.g., CMA heaps) will require selectively
charging also for the dmem controller.
Signed-off-by: Albert Esteve <aesteve@redhat.com>
---
Documentation/admin-guide/cgroup-v2.rst | 5 ++--
drivers/dma-buf/dma-buf.c | 16 ++++---------
drivers/dma-buf/dma-heap.c | 42 ++++++++++++++++++++++++++++++---
drivers/dma-buf/heaps/system_heap.c | 2 --
include/uapi/linux/dma-heap.h | 6 +++++
5 files changed, 53 insertions(+), 18 deletions(-)
diff --git a/Documentation/admin-guide/cgroup-v2.rst b/Documentation/admin-guide/cgroup-v2.rst
index 8bdbc2e866430..824d269531eb1 100644
--- a/Documentation/admin-guide/cgroup-v2.rst
+++ b/Documentation/admin-guide/cgroup-v2.rst
@@ -1636,8 +1636,9 @@ The following nested keys are defined.
structures.
dmabuf (npn)
- Amount of memory used for exported DMA buffers allocated by the cgroup.
- Stays with the allocating cgroup regardless of how the buffer is shared.
+ Amount of memory used for exported DMA buffers allocated by or on
+ behalf of the cgroup. Stays with the allocating cgroup regardless
+ of how the buffer is shared.
workingset_refault_anon
Number of refaults of previously evicted anonymous pages.
diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c
index ce02377f48908..23fb758b78297 100644
--- a/drivers/dma-buf/dma-buf.c
+++ b/drivers/dma-buf/dma-buf.c
@@ -181,8 +181,11 @@ static void dma_buf_release(struct dentry *dentry)
*/
BUG_ON(dmabuf->cb_in.active || dmabuf->cb_out.active);
- mem_cgroup_uncharge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE);
- mem_cgroup_put(dmabuf->memcg);
+ if (dmabuf->memcg) {
+ mem_cgroup_uncharge_dmabuf(dmabuf->memcg,
+ PAGE_ALIGN(dmabuf->size) / PAGE_SIZE);
+ mem_cgroup_put(dmabuf->memcg);
+ }
dmabuf->ops->release(dmabuf);
@@ -764,13 +767,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
dmabuf->resv = resv;
}
- dmabuf->memcg = get_mem_cgroup_from_mm(current->mm);
- if (!mem_cgroup_charge_dmabuf(dmabuf->memcg, PAGE_ALIGN(dmabuf->size) / PAGE_SIZE,
- GFP_KERNEL)) {
- ret = -ENOMEM;
- goto err_memcg;
- }
-
file->private_data = dmabuf;
file->f_path.dentry->d_fsdata = dmabuf;
dmabuf->file = file;
@@ -781,8 +777,6 @@ struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info)
return dmabuf;
-err_memcg:
- mem_cgroup_put(dmabuf->memcg);
err_file:
fput(file);
err_module:
diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c
index ac5f8685a6494..ff6e259afcdc0 100644
--- a/drivers/dma-buf/dma-heap.c
+++ b/drivers/dma-buf/dma-heap.c
@@ -7,13 +7,17 @@
*/
#include <linux/cdev.h>
+#include <linux/cgroup.h>
#include <linux/device.h>
#include <linux/dma-buf.h>
#include <linux/dma-heap.h>
+#include <linux/memcontrol.h>
+#include <linux/sched/mm.h>
#include <linux/err.h>
#include <linux/export.h>
#include <linux/list.h>
#include <linux/nospec.h>
+#include <linux/pidfd.h>
#include <linux/syscalls.h>
#include <linux/uaccess.h>
#include <linux/xarray.h>
@@ -55,10 +59,12 @@ MODULE_PARM_DESC(mem_accounting,
"Enable cgroup-based memory accounting for dma-buf heap allocations (default=false).");
static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
- u32 fd_flags,
- u64 heap_flags)
+ u32 fd_flags, u64 heap_flags,
+ struct mem_cgroup *charge_to)
{
struct dma_buf *dmabuf;
+ unsigned int nr_pages;
+ struct mem_cgroup *memcg = charge_to;
int fd;
/*
@@ -73,6 +79,22 @@ static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len,
if (IS_ERR(dmabuf))
return PTR_ERR(dmabuf);
+ nr_pages = len / PAGE_SIZE;
+
+ if (memcg)
+ css_get(&memcg->css);
+ else if (mem_accounting)
+ memcg = get_mem_cgroup_from_mm(current->mm);
+
+ if (memcg) {
+ if (!mem_cgroup_charge_dmabuf(memcg, nr_pages, GFP_KERNEL)) {
+ mem_cgroup_put(memcg);
+ dma_buf_put(dmabuf);
+ return -ENOMEM;
+ }
+ dmabuf->memcg = memcg;
+ }
+
fd = dma_buf_fd(dmabuf, fd_flags);
if (fd < 0) {
dma_buf_put(dmabuf);
@@ -102,6 +124,9 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
{
struct dma_heap_allocation_data *heap_allocation = data;
struct dma_heap *heap = file->private_data;
+ struct mem_cgroup *memcg = NULL;
+ struct task_struct *task;
+ unsigned int pidfd_flags;
int fd;
if (heap_allocation->fd)
@@ -113,9 +138,20 @@ static long dma_heap_ioctl_allocate(struct file *file, void *data)
if (heap_allocation->heap_flags & ~DMA_HEAP_VALID_HEAP_FLAGS)
return -EINVAL;
+ if (heap_allocation->charge_pid_fd) {
+ task = pidfd_get_task(heap_allocation->charge_pid_fd, &pidfd_flags);
+ if (IS_ERR(task))
+ return PTR_ERR(task);
+
+ memcg = get_mem_cgroup_from_mm(task->mm);
+ put_task_struct(task);
+ }
+
fd = dma_heap_buffer_alloc(heap, heap_allocation->len,
heap_allocation->fd_flags,
- heap_allocation->heap_flags);
+ heap_allocation->heap_flags,
+ memcg);
+ mem_cgroup_put(memcg);
if (fd < 0)
return fd;
diff --git a/drivers/dma-buf/heaps/system_heap.c b/drivers/dma-buf/heaps/system_heap.c
index 03c2b87cb1112..95d7688167b93 100644
--- a/drivers/dma-buf/heaps/system_heap.c
+++ b/drivers/dma-buf/heaps/system_heap.c
@@ -385,8 +385,6 @@ static struct page *alloc_largest_available(unsigned long size,
if (max_order < orders[i])
continue;
flags = order_flags[i];
- if (mem_accounting)
- flags |= __GFP_ACCOUNT;
page = alloc_pages(flags, orders[i]);
if (!page)
continue;
diff --git a/include/uapi/linux/dma-heap.h b/include/uapi/linux/dma-heap.h
index a4cf716a49fa6..e02b0f8cbc6a1 100644
--- a/include/uapi/linux/dma-heap.h
+++ b/include/uapi/linux/dma-heap.h
@@ -29,6 +29,10 @@
* handle to the allocated dma-buf
* @fd_flags: file descriptor flags used when allocating
* @heap_flags: flags passed to heap
+ * @charge_pid_fd: optional pidfd of the process whose cgroup should be
+ * charged for this allocation; 0 means charge the calling
+ * process's cgroup
+ * @__padding: reserved, must be zero
*
* Provided by userspace as an argument to the ioctl
*/
@@ -37,6 +41,8 @@ struct dma_heap_allocation_data {
__u32 fd;
__u32 fd_flags;
__u64 heap_flags;
+ __u32 charge_pid_fd;
+ __u32 __padding;
};
#define DMA_HEAP_IOC_MAGIC 'H'
--
2.53.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox