* Re: [PATCH v2 00/10] sched: Flatten the pick
From: Peter Zijlstra @ 2026-05-18 7:14 UTC (permalink / raw)
To: Tejun Heo
Cc: mingo, longman, chenridong, juri.lelli, vincent.guittot,
dietmar.eggemann, rostedt, bsegall, mgorman, vschneid, hannes,
mkoutny, cgroups, linux-kernel, jstultz, kprateek.nayak, qyousef
In-Reply-To: <agN1QbsjFv2aXFhK@slm.duckdns.org>
On Tue, May 12, 2026 at 08:45:21AM -1000, Tejun Heo wrote:
> Hello, Peter.
>
> On Tue, May 12, 2026 at 10:10:00AM +0200, Peter Zijlstra wrote:
> ...
> > Anyway, this is why I've been looking at these alternative weight
> > schemes, to get the nominal fraction near 1 and make these problems go
> > away. It is both the numerical issues and the disparity between levels
> > (with root being at level 0 being the most obvious).
>
> I see. I think what bothers me is that I'm unsure what the weight config
> would mean when the shares are scaled by the number of active cpus in that
> cgroup.
Relative weight per active cpu :-), but yes, that is a somewhat more
difficult concept I suppose.
> Here's a simple example:
>
> - There are 256 cpus.
> - /cgroup-A has weight 100 and 128 active threads. No pinning.
> - /cgroup-B has weight 100 and 256 active thredas. No pinning.
>
> In the current code, assuming math holds up, cgroup-A and B would get about
> the same shares - ~128 CPUs each. However, if we scale the share by active
> CPUs in each cgroup, B's tasks would end up with the same weight as A's on
> CPUs that they end up competing on, which would lead to ~ 1:3 distribution.
> Is that the right reading of the code?
Indeed. So both A and B will get ~1024 weight per (active) CPU, such
that on the CPUs they contend they will get 1:1 and then B will get the
full CPU on the uncontested CPUs, resulting in a total of 1:3
distribution.
This can of course be compensated by increasing the relative
weight of A, if that is so desired. But the alternative view is that for
those 128 CPUs they overlap, A and B will get equal parts, it is just
that B consumes another 128 CPUs and will not have contention there.
So the current scheme will inflate the part of A to be double the weight
(of B), giving them 2 out of 3 parts on the contended CPUs, but then B
will still get complete / uncontested access to those extra 128 CPUs,
resulting in a 2:4 weight distribution.
Which also isn't as straight forward as one might think.
So perhaps 'weight on the CPUs you contest on' isn't as unintuitive as
it seems on first glance, its just different.
And it has tremendous advantages as outlined before; it is naturally
normalized -- the disparity between nesting levels goes away, and the
edge case of a single CPU active will be sane.
Eg. consider your example except now A will have 1 active thread. Then A
will get the full group weight (1024) on its one CPU, while B will get
(1024/256=8) on each CPU.
So for the one contended CPU A gets 256 out of 257 parts, while B gets
the full CPU for the remaining 255 CPUs, for a:
256 1 257
--- : --- + 255*--- = 256:65535 ~ 1:256
257 257 257
distribution. While with the new scheme it would be:
1 1 2
- : - + 255*- = 1:511
2 2 2
Which, realistically isn't all that different, except the old scheme has
this really large weight to deal with.
So from where I'm sitting, yes different, but it behaves better.
^ permalink raw reply
* [PATCH] blk-cgroup: defer blkcg css_put until blkg is unlinked from queue
From: Zizhi Wo @ 2026-05-18 1:09 UTC (permalink / raw)
To: axboe, tj, josef, yukuai, linux-block
Cc: cgroups, yangerkun, chengzhihao1, wozizhi
From: Zizhi Wo <wozizhi@huawei.com>
[BUG]
Our fuzz testing triggered a blkcg use-after-free issue:
BUG: KASAN: slab-use-after-free in _raw_spin_lock+0x75/0xe0
Call Trace:
...
blkcg_deactivate_policy+0x244/0x4d0
ioc_rqos_exit+0x44/0xe0
rq_qos_exit+0xba/0x120
__del_gendisk+0x50b/0x800
del_gendisk+0xff/0x190
...
[CAUSE]
process1 process2
cgroup_rmdir
...
css_killed_work_fn
offline_css
...
blkcg_destroy_blkgs
...
__blkg_release
css_put(&blkg->blkcg->css)
blkg_free
INIT_WORK(xxx, blkg_free_workfn)
schedule_work
css_put
...
blkcg_css_free
kfree(blkcg)--------blkcg has been freed!!!
====================================schedule_work
blkg_free_workfn
__del_gendisk
rq_qos_exit
ioc_rqos_exit
blkcg_deactivate_policy
mutex_lock(&q->blkcg_mutex)
spin_lock_irq(&q->queue_lock)
list_for_each_entry(blkg, xxx)
blkcg = blkg->blkcg
spin_lock(&blkcg->lock)-------UAF!!!
mutex_lock(&q->blkcg_mutex)
spin_lock_irq(&q->queue_lock)
/* Only then is the blkg removed from the list */
list_del_init(&blkg->q_node)
As a result, a blkg can still be reachable through q->blkg_list while
its ->blkcg has already been freed.
[Fix]
Fix this by deferring the blkcg css_put() until after the blkg has been
unlinked from q->blkg_list in blkg_free_workfn(). This ensures that the
blkcg outlives every blkg still reachable through q->blkg_list, so any
iterator holding q->queue_lock is guaranteed to observe a valid
blkg->blkcg.
Fixes: f1c006f1c685 ("blk-cgroup: synchronize pd_free_fn() from blkg_free_workfn() and blkcg_deactivate_policy()")
Signed-off-by: Zizhi Wo <wozizhi@huawei.com>
---
block/blk-cgroup.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c
index 554c87bb4a86..7b7677d022d0 100644
--- a/block/blk-cgroup.c
+++ b/block/blk-cgroup.c
@@ -132,10 +132,15 @@ static void blkg_free_workfn(struct work_struct *work)
if (blkg->parent)
blkg_put(blkg->parent);
spin_lock_irq(&q->queue_lock);
list_del_init(&blkg->q_node);
spin_unlock_irq(&q->queue_lock);
+ /*
+ * Release blkcg css ref only after blkg is removed from q->blkg_list,
+ * so concurrent iterators won't see a blkg with a freed blkcg.
+ */
+ css_put(&blkg->blkcg->css);
mutex_unlock(&q->blkcg_mutex);
blk_put_queue(q);
free_percpu(blkg->iostat_cpu);
percpu_ref_exit(&blkg->refcnt);
@@ -177,12 +182,10 @@ static void __blkg_release(struct rcu_head *rcu)
* blkg_stat_lock is for serializing blkg stat update
*/
for_each_possible_cpu(cpu)
__blkcg_rstat_flush(blkcg, cpu);
- /* release the blkcg and parent blkg refs this blkg has been holding */
- css_put(&blkg->blkcg->css);
blkg_free(blkg);
}
/*
* A group is RCU protected, but having an rcu lock does not mean that one
--
2.52.0
^ permalink raw reply related
* [PATCH v2] memcg: cache obj_stock by memcg, not by objcg pointer
From: Shakeel Butt @ 2026-05-17 19:43 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
Commit 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg
per-node type") split a memcg's single obj_cgroup into one per NUMA
node, but the per-CPU obj_stock_pcp still keys cached_objcg by
pointer. Cross-NUMA workloads now see a drain on every refill and a
miss on every consume that targets a sibling per-node objcg of the
same memcg, producing the 67.7% stress-ng switch-mq regression
reported by LKP.
stock->nr_bytes are fungible across per-node objcgs of one memcg.
Treat the cache as keyed by memcg in __consume_obj_stock() and
__refill_obj_stock() so siblings share the reserve. Compare via
READ_ONCE(objcg->memcg) directly: pointer-compare only, no deref, so
the rcu_read_lock contract on obj_cgroup_memcg() does not apply.
In the same-memcg refill path also fold the incoming objcg's
nr_charged_bytes into the stock; otherwise sub-page residue
accumulates on whichever sibling was cached at drain time and
obj_cgroup_release() silently drops it, leaking up to nr_node_ids *
(PAGE_SIZE - 1) bytes per memcg lifecycle from the page_counter.
This issue was reported by Sashiko.
Update the now-stale invariant comment on __account_obj_stock().
Qi Zheng built a specialized reproducer [1] for the corner case and
confirmed the fix.
Reported-by: kernel test robot <oliver.sang@intel.com>
Closes: https://lore.kernel.org/oe-lkp/202605121641.b6a60cb0-lkp@intel.com
Fixes: 01b9da291c49 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
Link: https://lore.kernel.org/19693be6-7132-446e-b3fc-b7e9f56e5949@linux.dev/ [1]
Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
Debugged-by: Qi Zheng <qi.zheng@linux.dev>
Tested-by: Qi Zheng <qi.zheng@linux.dev>
---
Changes since v1:
- Fix the rcu warning (Sashiko).
- Fix the page counter possible underflow warning (Sashiko).
mm/memcontrol.c | 25 ++++++++++++++++++++++---
1 file changed, 22 insertions(+), 3 deletions(-)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index d978e18b9b2d..e22ffa3b3319 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -3152,7 +3152,12 @@ static void unlock_stock(struct obj_stock_pcp *stock)
local_unlock(&obj_stock.lock);
}
-/* Call after __refill_obj_stock() to ensure stock->cached_objg == objcg */
+/*
+ * Call after __consume_obj_stock() / __refill_obj_stock(). The stock may be
+ * cached for a sibling per-node objcg of the same memcg; in that case the
+ * vmstat batching slot does not match objcg and we fall through to the
+ * direct path.
+ */
static void __account_obj_stock(struct obj_cgroup *objcg,
struct obj_stock_pcp *stock, int nr,
struct pglist_data *pgdat, enum node_stat_item idx)
@@ -3210,7 +3215,11 @@ static bool __consume_obj_stock(struct obj_cgroup *objcg,
struct obj_stock_pcp *stock,
unsigned int nr_bytes)
{
- if (objcg == READ_ONCE(stock->cached_objcg) &&
+ struct obj_cgroup *cached = READ_ONCE(stock->cached_objcg);
+
+ /* Sibling per-node objcgs share the reserve. */
+ if ((cached == objcg ||
+ (cached && READ_ONCE(cached->memcg) == READ_ONCE(objcg->memcg))) &&
stock->nr_bytes >= nr_bytes) {
stock->nr_bytes -= nr_bytes;
return true;
@@ -3318,6 +3327,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
unsigned int nr_bytes,
bool allow_uncharge)
{
+ struct obj_cgroup *cached;
unsigned int nr_pages = 0;
if (!stock) {
@@ -3327,7 +3337,11 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
goto out;
}
- if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
+ cached = READ_ONCE(stock->cached_objcg);
+ if (cached == objcg)
+ goto add_bytes;
+ /* Direct READ_ONCE due to just pointer comparison. */
+ if (!cached || READ_ONCE(cached->memcg) != READ_ONCE(objcg->memcg)) {
drain_obj_stock(stock);
obj_cgroup_get(objcg);
stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
@@ -3335,7 +3349,12 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
WRITE_ONCE(stock->cached_objcg, objcg);
allow_uncharge = true; /* Allow uncharge when objcg changes */
+ } else if (atomic_read(&objcg->nr_charged_bytes)) {
+ /* Fold sibling's stranded ncb into stock; else release leaks it. */
+ stock->nr_bytes += atomic_xchg(&objcg->nr_charged_bytes, 0);
+ allow_uncharge = true;
}
+add_bytes:
stock->nr_bytes += nr_bytes;
if (allow_uncharge && (stock->nr_bytes > PAGE_SIZE)) {
--
2.53.0-Meta
^ permalink raw reply related
* Re: [linus:master] [mm] 01b9da291c: stress-ng.switch.ops_per_sec 67.7% regression
From: Shakeel Butt @ 2026-05-17 19:38 UTC (permalink / raw)
To: Oliver Sang
Cc: Qi Zheng, oe-lkp, lkp, linux-kernel, Andrew Morton, David Carlier,
Allen Pais, Axel Rasmussen, Baoquan He, Chengming Zhou,
Chen Ridong, David Hildenbrand, Hamza Mahfooz, Harry Yoo,
Hugh Dickins, Imran Khan, Johannes Weiner, Kamalesh Babulal,
Lance Yang, Liam Howlett, Lorenzo Stoakes, Michal Hocko,
Michal Koutný, Mike Rapoport, Muchun Song, Muchun Song,
Nhat Pham, Roman Gushchin, Suren Baghdasaryan, Usama Arif,
Vlastimil Babka, Wei Xu, Yosry Ahmed, Yuanchu Xie, Zi Yan,
Usama Arif, cgroups, linux-mm
In-Reply-To: <agm61hMv08XnV8sI@xsang-OptiPlex-9020>
On Sun, May 17, 2026 at 08:55:50PM +0800, Oliver Sang wrote:
> hi, Shakeel, hi, Qi,
>
> On Fri, May 15, 2026 at 10:09:06AM -0700, Shakeel Butt wrote:
> > On Fri, May 15, 2026 at 03:37:22PM +0800, Qi Zheng wrote:
> > > Hi Shakeel,
> > >
> > > On 5/14/26 9:40 PM, Shakeel Butt wrote:
> > > > May 14, 2026 at 12:46 AM, "Qi Zheng" <qi.zheng@linux.dev mailto:qi.zheng@linux.dev?to=%22Qi%20Zheng%22%20%3Cqi.zheng%40linux.dev%3E > wrote:
> > > >
> > > >
> > > > >
> > > > > On 5/13/26 10:27 PM, Shakeel Butt wrote:
> > > > >
> > > > > >
> > > > > > On Wed, May 13, 2026 at 06:49:45AM -0700, Shakeel Butt wrote:
> > > > > >
> > > > > > >
> > > > > > > On Wed, May 13, 2026 at 10:10:34AM +0800, Qi Zheng wrote:
> > > > > > >
> > > > > > On 5/13/26 12:03 AM, Shakeel Butt wrote:
> > > > > > On Tue, May 12, 2026 at 08:56:52PM +0800, kernel test robot wrote:
> > > > > >
> > > > > > Hello,
> > > > > >
> > > > > > kernel test robot noticed a 67.7% regression of stress-ng.switch.ops_per_sec on:
> > > > > >
> > > > > > commit: 01b9da291c4969354807b52956f4aae1f41b4924 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> > > > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > > > > >
> > > > > > This is most probably due to shuffling of struct mem_cgroup and struct
> > > > > > mem_cgroup_per_node members.
> > > > > >
> > > > > > Another possibility is that after objcg was split into per-node, the
> > > > > > slab accounting fast path is still designed assuming only one current
> > > > > > objcg per CPU:
> > > > > >
> > > > > > struct obj_stock_pcp {
> > > > > > struct obj_cgroup *cached_objcg;
> > > > > > };
> > > > > >
> > > > > > So it's may cause the following thrashing:
> > > > > >
> > > > > > CPU stock cached = memcg/node0 objcg
> > > > > > free object tagged = memcg/node1 objcg
> > > > > > => __refill_obj_stock --> objcg mismatch
> > > > > > => drain_obj_stock()
> > > > > > => cache switches to node1 objcg
> > > > > >
> > > > > > next local allocation tagged = node0 objcg
> > > > > > => mismatch again
> > > > > > => drain_obj_stock()
> > > > > >
> > > > > > >
> > > > > > > Actually I think this is the issue, we have ping pong threads running on
> > > > > > > different nodes where though theu are in same cgroup but their current->obcg is
> > > > > > > for local node and thus this ping pong is thrashing the per-cpu objcg stock.
> > > > > > >
> > > > > > > The easier fix would be to compare objcg->memcg instead of just objcg during
> > > > > > > draining and caching. In addition we can add support for multiple objcg per-cpu
> > > > > > > stock caching.
> > > > > > >
> > > > > > Something like the following:
> > > > > > From d756abe831a905d6fe32bad9a984fc619dafb7e0 Mon Sep 17 00:00:00 2001
> > > > > > From: Shakeel Butt <shakeel.butt@linux.dev>
> > > > > > Date: Wed, 13 May 2026 07:24:55 -0700
> > > > > > Subject: [PATCH] mm/memcontrol: skip obj_stock drain when refilled objcg
> > > > > > shares memcg
> > > > > > Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> > > > > > ---
> > > > > > mm/memcontrol.c | 14 +++++++++++++-
> > > > > > 1 file changed, 13 insertions(+), 1 deletion(-)
> > > > > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > > > > > index d978e18b9b2d..01ed7a8e18ac 100644
> > > > > > --- a/mm/memcontrol.c
> > > > > > +++ b/mm/memcontrol.c
> > > > > > @@ -3318,6 +3318,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > > > > > unsigned int nr_bytes,
> > > > > > bool allow_uncharge)
> > > > > > {
> > > > > > + struct obj_cgroup *cached;
> > > > > > unsigned int nr_pages = 0;
> > > > > > > if (!stock) {
> > > > > > @@ -3327,7 +3328,18 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > > > > > goto out;
> > > > > > }
> > > > > > > - if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
> > > > > > + cached = READ_ONCE(stock->cached_objcg);
> > > > > > + if (cached != objcg &&
> > > > > > + (!cached || obj_cgroup_memcg(cached) != obj_cgroup_memcg(objcg))) {
> > > > > > drain_obj_stock(stock);
> > > > > > obj_cgroup_get(objcg);
> > > > > > stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
> > > > > >
> > > > > This change looks like it should be able to fix the ping-pong issue, but
> > > > > I stiil haven't reproduced the performance regression locally. I'll
> > > > > continue testing it.
> > > >
> > > > Same here, couldn't reproduce locally. It seems like we had to craft a scenario
> > > > where the pair pingpong threads get their current->objcg from different nodes.
> > > > I will try that.
> > >
> > > I still haven't been able to reproduce the LKP results locally, but I
> > > used an AI bot to generate a pingpong test case (pasted at the end) and
> > > automatically ran the test on a physical machine. The results are as
> > > follows:
> > >
> > > parent: 8285917d6f
> > > bad: 01b9da291c
> > > fix: 01b9da291c + stock patch
> > >
> > > | kernel | mq_ops/sec mean | vs parent | drain_obj_stock / round |
> > > |--------|-----------------|-----------|-------------------------|
> > > | parent | 9.743M | baseline | ~0 |
> > > | bad | 7.821M | -19.73% | ~11.16M |
> > > | fix | 9.274M | -4.81% | ~0 |
> > >
> > > Probing the drain_obj_stock() calls confirms that the fix restores the
> > > frequency to the parent's baseline.
> > >
> > > And it seems that besides __refill_obj_stock(), we should also modify
> > > __consume_obj_stock()?
> > >
> >
> > Thanks a lot Qi. I will send the formal patch and will add your Debugged-by if
> > you don't mind.
> >
>
> Tested-by: kernel test robot <oliver.sang@intel.com>
>
> we tested above patch, and it recovers the regression:
>
> =========================================================================================
> compiler/cpufreq_governor/kconfig/method/nr_threads/rootfs/tbox_group/test/testcase/testtime:
> gcc-14/performance/x86_64-rhel-9.4/mq/100%/debian-13-x86_64-20250902.cgz/lkp-spr-r02/switch/stress-ng/60s
>
> commit:
> 8285917d6f ("mm: memcontrol: prepare for reparenting non-hierarchical stats")
> 01b9da291c ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> 682fd4e9ff <--- above patch from Shakeel
>
> 8285917d6f383aef 01b9da291c4969354807b52956f 682fd4e9ffd4009805f81dd25ed
> ---------------- --------------------------- ---------------------------
> %stddev %change %stddev %change %stddev
> \ | \ | \
> 5849 +210.2% 18145 ± 3% +1.5% 5935 stress-ng.switch.nanosecs_per_context_switch_mq_method
> 2.296e+09 -67.7% 7.408e+08 ± 3% -1.4% 2.263e+09 stress-ng.switch.ops
> 38288993 -67.7% 12355813 ± 3% -1.4% 37739220 stress-ng.switch.ops_per_sec
>
>
> full compasison is as below [3]
>
> but there are two notes.
>
> #1 is that we noticed there is a fomal patch later from Shakeel in [1] which has
> more changes. not sure if this test is enough? do you want us to test [1]
> further?
Thanks Oliver, I will send a v2 soon, please test v2.
>
> #2: when we test above patch, we found the server easy to crash while running
> tests. we try to run up to 20 times, only 2 of them run successfully (above
> 37739220 is just the average data from these 2 runs, since the data is stable,
> we think maybe it's ok to report to you with this data).
> we also noticed for [1] there is a [syzbot ci] report in [2]. since we don't
> have serial output for our test server in this report which is for performance
> tests, we cannot say if other 18 runs failed due to similar reason. just FYI.
>
The syzbot report is simply a rcu warning which will be fixed in v2. Do you
have more details on the crash you are seeing? Is it page counter underflow
warning?
Thanks again for the help.
^ permalink raw reply
* Re: [PATCH] memcg: cache obj_stock by memcg, not by objcg pointer
From: Shakeel Butt @ 2026-05-17 19:34 UTC (permalink / raw)
To: Andrew Morton
Cc: Johannes Weiner, Michal Hocko, Roman Gushchin, Muchun Song,
Qi Zheng, Meta kernel team, linux-mm, cgroups, linux-kernel,
kernel test robot
In-Reply-To: <agdnTPXhuepuu7hG@linux.dev>
On Fri, May 15, 2026 at 11:42:39AM -0700, Shakeel Butt wrote:
[...]
> > Will sharing the reserve between per-node sibling objcgs without updating
> > stock->cached_objcg break the page multiple invariant in
> > obj_cgroup_release()?
> >
> > If an allocation for objcg_B consumes bytes originally funded by objcg_A,
> > and the stock is later drained, those borrowed bytes are flushed into
> > objcg_A->nr_charged_bytes.
> >
> > When obj_cgroup_release() is invoked, nr_charged_bytes will not be an
> > exact multiple of PAGE_SIZE. Will this trigger
> > WARN_ON_ONCE(nr_bytes & (PAGE_SIZE - 1)) and truncate the remainder,
> > permanently leaking the page charge from the memcg?
>
> This is actually a very good point and need more thought.
>
I think we can handle this simply by taking over objcg->nr_charged_bytes into
the stock.
^ permalink raw reply
* [PATCH v5 12/12] mm, swap: merge zeromap into swap table
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
By allocating one additional bit in the swap table entry's flags field
alongside the count, we can store the zeromap inline
For 64 bit systems, zeromap will store in the swap table, avoiding zeromap
allocation. It reduces the allocated memory. That is the happy path.
For certain 32-bit archs, there might not be enough bits in the swap
table to contain both PFN and flags. Therefore, conditionally let each
cluster have a zeromap field at build time, and use that instead.
If the swapfile cluster is not fully used, it will still save memory for
zeromap. The empty cluster does not allocate a zeromap. In the worst case,
all cluster are fully populated. We will use memory similar to the
previous zeromap implementation.
A few macros were moved to different headers for build time struct
definition.
Acked-by: Chris Li <chrisl@kernel.org>
Reviewed-by: Youngjun Park <youngjun.park@lge.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/swap.h | 1 -
mm/memory.c | 11 +----
mm/page_io.c | 61 +++++++++++++++++++++++----
mm/swap.h | 51 +++++++++--------------
mm/swap_state.c | 14 ++++---
mm/swap_table.h | 115 +++++++++++++++++++++++++++++++++++++--------------
mm/swapfile.c | 54 +++++++++++-------------
7 files changed, 191 insertions(+), 116 deletions(-)
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 203bbe23ba1f..6d72778e6cc3 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -253,7 +253,6 @@ struct swap_info_struct {
struct plist_node list; /* entry in swap_active_head */
signed char type; /* strange name for an index */
unsigned int max; /* size of this swap device */
- unsigned long *zeromap; /* kvmalloc'ed bitmap to track zero pages */
struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
struct list_head free_clusters; /* free clusters list */
struct list_head full_clusters; /* full clusters list */
diff --git a/mm/memory.c b/mm/memory.c
index da891bcce59c..7c020995eafc 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4611,13 +4611,11 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
/*
- * Check if the PTEs within a range are contiguous swap entries
- * and have consistent swapcache, zeromap.
+ * Check if the PTEs within a range are contiguous swap entries.
*/
static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
{
unsigned long addr;
- softleaf_t entry;
int idx;
pte_t pte;
@@ -4627,18 +4625,13 @@ static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
if (!pte_same(pte, pte_move_swp_offset(vmf->orig_pte, -idx)))
return false;
- entry = softleaf_from_pte(pte);
- if (swap_pte_batch(ptep, nr_pages, pte) != nr_pages)
- return false;
-
/*
* swap_read_folio() can't handle the case a large folio is hybridly
* from different backends. And they are likely corner cases. Similar
* things might be added once zswap support large folios.
*/
- if (unlikely(swap_zeromap_batch(entry, nr_pages, NULL) != nr_pages))
+ if (swap_pte_batch(ptep, nr_pages, pte) != nr_pages)
return false;
-
return true;
}
diff --git a/mm/page_io.c b/mm/page_io.c
index 7ed76592e20d..f2d8fe7fd057 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -26,6 +26,7 @@
#include <linux/delayacct.h>
#include <linux/zswap.h>
#include "swap.h"
+#include "swap_table.h"
static void __end_swap_bio_write(struct bio *bio)
{
@@ -204,15 +205,20 @@ static bool is_folio_zero_filled(struct folio *folio)
static void swap_zeromap_folio_set(struct folio *folio)
{
struct obj_cgroup *objcg = get_obj_cgroup_from_folio(folio);
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
int nr_pages = folio_nr_pages(folio);
+ struct swap_cluster_info *ci;
swp_entry_t entry;
unsigned int i;
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+
+ ci = swap_cluster_get_and_lock(folio);
for (i = 0; i < folio_nr_pages(folio); i++) {
entry = page_swap_entry(folio_page(folio, i));
- set_bit(swp_offset(entry), sis->zeromap);
+ __swap_table_set_zero(ci, swp_cluster_offset(entry));
}
+ swap_cluster_unlock(ci);
count_vm_events(SWPOUT_ZERO, nr_pages);
if (objcg) {
@@ -223,14 +229,19 @@ static void swap_zeromap_folio_set(struct folio *folio)
static void swap_zeromap_folio_clear(struct folio *folio)
{
- struct swap_info_struct *sis = __swap_entry_to_info(folio->swap);
+ struct swap_cluster_info *ci;
swp_entry_t entry;
unsigned int i;
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+
+ ci = swap_cluster_get_and_lock(folio);
for (i = 0; i < folio_nr_pages(folio); i++) {
entry = page_swap_entry(folio_page(folio, i));
- clear_bit(swp_offset(entry), sis->zeromap);
+ __swap_table_clear_zero(ci, swp_cluster_offset(entry));
}
+ swap_cluster_unlock(ci);
}
/*
@@ -255,10 +266,9 @@ int swap_writeout(struct folio *folio, struct swap_iocb **swap_plug)
}
/*
- * Use a bitmap (zeromap) to avoid doing IO for zero-filled pages.
- * The bits in zeromap are protected by the locked swapcache folio
- * and atomic updates are used to protect against read-modify-write
- * corruption due to other zero swap entries seeing concurrent updates.
+ * Use the swap table zero mark to avoid doing IO for zero-filled
+ * pages. The zero mark is protected by the cluster lock, which is
+ * acquired internally by swap_zeromap_folio_set/clear.
*/
if (is_folio_zero_filled(folio)) {
swap_zeromap_folio_set(folio);
@@ -509,19 +519,52 @@ static void sio_read_complete(struct kiocb *iocb, long ret)
mempool_free(sio, sio_pool);
}
+/*
+ * Return the count of contiguous swap entries that share the same
+ * zeromap status as the starting entry. If is_zerop is not NULL,
+ * it will return the zeromap status of the starting entry.
+ *
+ * Context: Caller must ensure the cluster containing the entries
+ * that will be checked won't be freed.
+ */
+static int swap_zeromap_batch(swp_entry_t entry, int max_nr,
+ bool *is_zerop)
+{
+ int i;
+ bool is_zero;
+ unsigned int ci_start = swp_cluster_offset(entry);
+ struct swap_cluster_info *ci = __swap_entry_to_cluster(entry);
+
+ VM_WARN_ON_ONCE(ci_start + max_nr > SWAPFILE_CLUSTER);
+
+ rcu_read_lock();
+ is_zero = __swap_table_test_zero(ci, ci_start);
+ for (i = 1; i < max_nr; i++)
+ if (is_zero != __swap_table_test_zero(ci, ci_start + i))
+ break;
+ rcu_read_unlock();
+ if (is_zerop)
+ *is_zerop = is_zero;
+
+ return i;
+}
+
static bool swap_read_folio_zeromap(struct folio *folio)
{
int nr_pages = folio_nr_pages(folio);
struct obj_cgroup *objcg;
bool is_zeromap;
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+
/*
* Swapping in a large folio that is partially in the zeromap is not
* currently handled. Return true without marking the folio uptodate so
* that an IO error is emitted (e.g. do_swap_page() will sigbus).
+ * Folio lock stabilizes the cluster and map, so the check is safe.
*/
if (WARN_ON_ONCE(swap_zeromap_batch(folio->swap, nr_pages,
- &is_zeromap) != nr_pages))
+ &is_zeromap) != nr_pages))
return true;
if (!is_zeromap)
diff --git a/mm/swap.h b/mm/swap.h
index 5b2f095fff6e..81c06aae7ccd 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -3,12 +3,29 @@
#define _MM_SWAP_H
#include <linux/atomic.h> /* for atomic_long_t */
+#include <linux/mm.h> /* for PAGE_SHIFT */
struct mempolicy;
struct swap_iocb;
struct swap_memcg_table;
extern int page_cluster;
+#if defined(MAX_POSSIBLE_PHYSMEM_BITS)
+#define SWAP_CACHE_PFN_BITS (MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT)
+#elif defined(MAX_PHYSMEM_BITS)
+#define SWAP_CACHE_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT)
+#else
+#define SWAP_CACHE_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT)
+#endif
+
+/* Swap table marker, 0x1 means shadow, 0x2 means PFN (SWP_TB_PFN_MARK) */
+#define SWAP_CACHE_PFN_MARK_BITS 2
+/* At least 2 bits are needed to distinguish SWP_TB_COUNT_MAX, 1 and 0 */
+#define SWAP_COUNT_MIN_BITS 2
+/* If there are enough bits besides PFN and marker, store zero flag inline */
+#define SWAP_TABLE_HAS_ZEROFLAG ((BITS_PER_LONG - SWAP_CACHE_PFN_MARK_BITS - \
+ SWAP_CACHE_PFN_BITS) > SWAP_COUNT_MIN_BITS)
+
#ifdef CONFIG_THP_SWAP
#define SWAPFILE_CLUSTER HPAGE_PMD_NR
#define swap_entry_order(order) (order)
@@ -41,6 +58,9 @@ struct swap_cluster_info {
unsigned int *extend_table; /* For large swap count, protected by ci->lock */
#ifdef CONFIG_MEMCG
struct swap_memcg_table *memcg_table; /* Swap table entries' cgroup record */
+#endif
+#if !SWAP_TABLE_HAS_ZEROFLAG
+ unsigned long *zero_bitmap;
#endif
struct list_head list;
};
@@ -314,31 +334,6 @@ static inline unsigned int folio_swap_flags(struct folio *folio)
return __swap_entry_to_info(folio->swap)->flags;
}
-/*
- * Return the count of contiguous swap entries that share the same
- * zeromap status as the starting entry. If is_zeromap is not NULL,
- * it will return the zeromap status of the starting entry.
- */
-static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr,
- bool *is_zeromap)
-{
- struct swap_info_struct *sis = __swap_entry_to_info(entry);
- unsigned long start = swp_offset(entry);
- unsigned long end = start + max_nr;
- bool first_bit;
-
- first_bit = test_bit(start, sis->zeromap);
- if (is_zeromap)
- *is_zeromap = first_bit;
-
- if (max_nr <= 1)
- return max_nr;
- if (first_bit)
- return find_next_zero_bit(sis->zeromap, end, start) - start;
- else
- return find_next_bit(sis->zeromap, end, start) - start;
-}
-
#else /* CONFIG_SWAP */
struct swap_iocb;
static inline struct swap_cluster_info *swap_cluster_lock(
@@ -476,11 +471,5 @@ static inline unsigned int folio_swap_flags(struct folio *folio)
{
return 0;
}
-
-static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr,
- bool *has_zeromap)
-{
- return 0;
-}
#endif /* CONFIG_SWAP */
#endif /* _MM_SWAP_H */
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 873cb3f26337..04f5ce992401 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -160,6 +160,7 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
{
unsigned int ci_off, ci_end;
unsigned long old_tb;
+ bool is_zero;
lockdep_assert_held(&ci->lock);
@@ -184,12 +185,14 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
if (nr == 1)
return 0;
+ is_zero = __swap_table_test_zero(ci, ci_off);
ci_off = round_down(ci_off, nr);
ci_end = ci_off + nr;
do {
old_tb = __swap_table_get(ci, ci_off);
if (unlikely(swp_tb_is_folio(old_tb) ||
!__swp_tb_get_count(old_tb) ||
+ is_zero != __swap_table_test_zero(ci, ci_off) ||
(memcg_id && *memcg_id != __swap_cgroup_get(ci, ci_off))))
return -EBUSY;
} while (++ci_off < ci_end);
@@ -213,7 +216,7 @@ static void __swap_cache_do_add_folio(struct swap_cluster_info *ci,
do {
old_tb = __swap_table_get(ci, ci_off);
VM_WARN_ON_ONCE(swp_tb_is_folio(old_tb));
- __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_count(old_tb)));
+ __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_flags(old_tb)));
} while (++ci_off < ci_end);
folio_ref_add(folio, nr_pages);
@@ -249,7 +252,6 @@ static void __swap_cache_do_del_folio(struct swap_cluster_info *ci,
struct folio *folio,
swp_entry_t entry, void *shadow)
{
- int count;
unsigned long old_tb;
struct swap_info_struct *si;
unsigned int ci_start, ci_off, ci_end;
@@ -269,13 +271,13 @@ static void __swap_cache_do_del_folio(struct swap_cluster_info *ci,
old_tb = __swap_table_get(ci, ci_off);
WARN_ON_ONCE(!swp_tb_is_folio(old_tb) ||
swp_tb_to_folio(old_tb) != folio);
- count = __swp_tb_get_count(old_tb);
- if (count)
+ if (__swp_tb_get_count(old_tb))
folio_swapped = true;
else
need_free = true;
/* If shadow is NULL, we set an empty shadow. */
- __swap_table_set(ci, ci_off, shadow_to_swp_tb(shadow, count));
+ __swap_table_set(ci, ci_off, shadow_to_swp_tb(shadow,
+ __swp_tb_get_flags(old_tb)));
} while (++ci_off < ci_end);
folio->swap.val = 0;
@@ -369,7 +371,7 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,
do {
old_tb = __swap_table_get(ci, ci_off);
WARN_ON_ONCE(!swp_tb_is_folio(old_tb) || swp_tb_to_folio(old_tb) != old);
- __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_count(old_tb)));
+ __swap_table_set(ci, ci_off, pfn_to_swp_tb(pfn, __swp_tb_get_flags(old_tb)));
} while (++ci_off < ci_end);
/*
diff --git a/mm/swap_table.h b/mm/swap_table.h
index b4e1100f8296..e6613e62f8d0 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -26,12 +26,14 @@ struct swap_memcg_table {
* Swap table entry type and bits layouts:
*
* NULL: |---------------- 0 ---------------| - Free slot
- * Shadow: | SWAP_COUNT |---- SHADOW_VAL ---|1| - Swapped out slot
- * PFN: | SWAP_COUNT |------ PFN -------|10| - Cached slot
+ * Shadow: |SWAP_COUNT|Z|---- SHADOW_VAL ---|1| - Swapped out slot
+ * PFN: |SWAP_COUNT|Z|------ PFN -------|10| - Cached slot
* Pointer: |----------- Pointer ----------|100| - (Unused)
* Bad: |------------- 1 -------------|1000| - Bad slot
*
- * SWAP_COUNT is `SWP_TB_COUNT_BITS` long, each entry is an atomic long.
+ * COUNT is `SWP_TB_COUNT_BITS` long, Z is the `SWP_TB_ZERO_FLAG` bit,
+ * and together they form the `SWP_TB_FLAGS_BITS` wide flags field.
+ * Each entry is an atomic long.
*
* Usages:
*
@@ -54,14 +56,6 @@ struct swap_memcg_table {
* - Bad: Swap slot is reserved, protects swap header or holes on swap devices.
*/
-#if defined(MAX_POSSIBLE_PHYSMEM_BITS)
-#define SWAP_CACHE_PFN_BITS (MAX_POSSIBLE_PHYSMEM_BITS - PAGE_SHIFT)
-#elif defined(MAX_PHYSMEM_BITS)
-#define SWAP_CACHE_PFN_BITS (MAX_PHYSMEM_BITS - PAGE_SHIFT)
-#else
-#define SWAP_CACHE_PFN_BITS (BITS_PER_LONG - PAGE_SHIFT)
-#endif
-
/* NULL Entry, all 0 */
#define SWP_TB_NULL 0UL
@@ -69,22 +63,26 @@ struct swap_memcg_table {
#define SWP_TB_SHADOW_MARK 0b1UL
/* Cached: PFN */
-#define SWP_TB_PFN_BITS (SWAP_CACHE_PFN_BITS + SWP_TB_PFN_MARK_BITS)
+#define SWP_TB_PFN_BITS (SWAP_CACHE_PFN_BITS + SWAP_CACHE_PFN_MARK_BITS)
#define SWP_TB_PFN_MARK 0b10UL
-#define SWP_TB_PFN_MARK_BITS 2
-#define SWP_TB_PFN_MARK_MASK (BIT(SWP_TB_PFN_MARK_BITS) - 1)
+#define SWP_TB_PFN_MARK_MASK (BIT(SWAP_CACHE_PFN_MARK_BITS) - 1)
-/* SWAP_COUNT part for PFN or shadow, the width can be shrunk or extended */
-#define SWP_TB_COUNT_BITS min(4, BITS_PER_LONG - SWP_TB_PFN_BITS)
+/* Flags: For PFN or shadow, contains SWAP_COUNT, width changes */
+#define SWP_TB_FLAGS_BITS min(5, BITS_PER_LONG - SWP_TB_PFN_BITS)
+#define SWP_TB_COUNT_BITS (SWP_TB_FLAGS_BITS - SWAP_TABLE_HAS_ZEROFLAG)
+#define SWP_TB_FLAGS_MASK (~((~0UL) >> SWP_TB_FLAGS_BITS))
#define SWP_TB_COUNT_MASK (~((~0UL) >> SWP_TB_COUNT_BITS))
+#define SWP_TB_FLAGS_SHIFT (BITS_PER_LONG - SWP_TB_FLAGS_BITS)
#define SWP_TB_COUNT_SHIFT (BITS_PER_LONG - SWP_TB_COUNT_BITS)
#define SWP_TB_COUNT_MAX ((1 << SWP_TB_COUNT_BITS) - 1)
+/* The first flag is zero bit (SWAP_TABLE_HAS_ZEROFLAG) */
+#define SWP_TB_ZERO_FLAG BIT(BITS_PER_LONG - SWP_TB_FLAGS_BITS)
/* Bad slot: ends with 0b1000 and rests of bits are all 1 */
#define SWP_TB_BAD ((~0UL) << 3)
/* Macro for shadow offset calculation */
-#define SWAP_COUNT_SHIFT SWP_TB_COUNT_BITS
+#define SWAP_COUNT_SHIFT SWP_TB_FLAGS_BITS
/*
* Helpers for casting one type of info into a swap table entry.
@@ -102,40 +100,47 @@ static inline unsigned long __count_to_swp_tb(unsigned char count)
* used (count > 0 && count < SWP_TB_COUNT_MAX), and
* overflow (count == SWP_TB_COUNT_MAX).
*/
- BUILD_BUG_ON(SWP_TB_COUNT_MAX < 2 || SWP_TB_COUNT_BITS < 2);
+ BUILD_BUG_ON(SWP_TB_COUNT_BITS < SWAP_COUNT_MIN_BITS);
VM_WARN_ON(count > SWP_TB_COUNT_MAX);
return ((unsigned long)count) << SWP_TB_COUNT_SHIFT;
}
-static inline unsigned long pfn_to_swp_tb(unsigned long pfn, unsigned int count)
+static inline unsigned long __flags_to_swp_tb(unsigned char flags)
+{
+ BUILD_BUG_ON(SWP_TB_FLAGS_BITS > BITS_PER_BYTE);
+ VM_WARN_ON(flags >> SWP_TB_FLAGS_BITS);
+ return ((unsigned long)flags) << SWP_TB_FLAGS_SHIFT;
+}
+
+static inline unsigned long pfn_to_swp_tb(unsigned long pfn, unsigned char flags)
{
unsigned long swp_tb;
BUILD_BUG_ON(sizeof(unsigned long) != sizeof(void *));
BUILD_BUG_ON(SWAP_CACHE_PFN_BITS >
- (BITS_PER_LONG - SWP_TB_PFN_MARK_BITS - SWP_TB_COUNT_BITS));
+ (BITS_PER_LONG - SWAP_CACHE_PFN_MARK_BITS - SWP_TB_FLAGS_BITS));
- swp_tb = (pfn << SWP_TB_PFN_MARK_BITS) | SWP_TB_PFN_MARK;
- VM_WARN_ON_ONCE(swp_tb & SWP_TB_COUNT_MASK);
+ swp_tb = (pfn << SWAP_CACHE_PFN_MARK_BITS) | SWP_TB_PFN_MARK;
+ VM_WARN_ON_ONCE(swp_tb & SWP_TB_FLAGS_MASK);
- return swp_tb | __count_to_swp_tb(count);
+ return swp_tb | __flags_to_swp_tb(flags);
}
-static inline unsigned long folio_to_swp_tb(struct folio *folio, unsigned int count)
+static inline unsigned long folio_to_swp_tb(struct folio *folio, unsigned char flags)
{
- return pfn_to_swp_tb(folio_pfn(folio), count);
+ return pfn_to_swp_tb(folio_pfn(folio), flags);
}
-static inline unsigned long shadow_to_swp_tb(void *shadow, unsigned int count)
+static inline unsigned long shadow_to_swp_tb(void *shadow, unsigned char flags)
{
BUILD_BUG_ON((BITS_PER_XA_VALUE + 1) !=
BITS_PER_BYTE * sizeof(unsigned long));
BUILD_BUG_ON((unsigned long)xa_mk_value(0) != SWP_TB_SHADOW_MARK);
VM_WARN_ON_ONCE(shadow && !xa_is_value(shadow));
- VM_WARN_ON_ONCE(shadow && ((unsigned long)shadow & SWP_TB_COUNT_MASK));
+ VM_WARN_ON_ONCE(shadow && ((unsigned long)shadow & SWP_TB_FLAGS_MASK));
- return (unsigned long)shadow | __count_to_swp_tb(count) | SWP_TB_SHADOW_MARK;
+ return (unsigned long)shadow | SWP_TB_SHADOW_MARK | __flags_to_swp_tb(flags);
}
/*
@@ -173,14 +178,14 @@ static inline bool swp_tb_is_countable(unsigned long swp_tb)
static inline struct folio *swp_tb_to_folio(unsigned long swp_tb)
{
VM_WARN_ON(!swp_tb_is_folio(swp_tb));
- return pfn_folio((swp_tb & ~SWP_TB_COUNT_MASK) >> SWP_TB_PFN_MARK_BITS);
+ return pfn_folio((swp_tb & ~SWP_TB_FLAGS_MASK) >> SWAP_CACHE_PFN_MARK_BITS);
}
static inline void *swp_tb_to_shadow(unsigned long swp_tb)
{
VM_WARN_ON(!swp_tb_is_shadow(swp_tb));
/* No shift needed, xa_value is stored as it is in the lower bits. */
- return (void *)(swp_tb & ~SWP_TB_COUNT_MASK);
+ return (void *)(swp_tb & ~SWP_TB_FLAGS_MASK);
}
static inline unsigned char __swp_tb_get_count(unsigned long swp_tb)
@@ -189,6 +194,12 @@ static inline unsigned char __swp_tb_get_count(unsigned long swp_tb)
return ((swp_tb & SWP_TB_COUNT_MASK) >> SWP_TB_COUNT_SHIFT);
}
+static inline unsigned char __swp_tb_get_flags(unsigned long swp_tb)
+{
+ VM_WARN_ON(!swp_tb_is_countable(swp_tb));
+ return ((swp_tb & SWP_TB_FLAGS_MASK) >> SWP_TB_FLAGS_SHIFT);
+}
+
static inline int swp_tb_get_count(unsigned long swp_tb)
{
if (swp_tb_is_countable(swp_tb))
@@ -253,6 +264,50 @@ static inline unsigned long swap_table_get(struct swap_cluster_info *ci,
return swp_tb;
}
+static inline void __swap_table_set_zero(struct swap_cluster_info *ci,
+ unsigned int ci_off)
+{
+#if SWAP_TABLE_HAS_ZEROFLAG
+ unsigned long swp_tb = __swap_table_get(ci, ci_off);
+
+ BUILD_BUG_ON(SWP_TB_ZERO_FLAG & ~SWP_TB_FLAGS_MASK);
+ VM_WARN_ON(!swp_tb_is_countable(swp_tb));
+ swp_tb |= SWP_TB_ZERO_FLAG;
+ __swap_table_set(ci, ci_off, swp_tb);
+#else
+ lockdep_assert_held(&ci->lock);
+ __set_bit(ci_off, ci->zero_bitmap);
+#endif
+}
+
+static inline bool __swap_table_test_zero(struct swap_cluster_info *ci,
+ unsigned int ci_off)
+{
+#if SWAP_TABLE_HAS_ZEROFLAG
+ unsigned long swp_tb = __swap_table_get(ci, ci_off);
+
+ VM_WARN_ON(!swp_tb_is_countable(swp_tb));
+ return !!(swp_tb & SWP_TB_ZERO_FLAG);
+#else
+ return test_bit(ci_off, ci->zero_bitmap);
+#endif
+}
+
+static inline void __swap_table_clear_zero(struct swap_cluster_info *ci,
+ unsigned int ci_off)
+{
+#if SWAP_TABLE_HAS_ZEROFLAG
+ unsigned long swp_tb = __swap_table_get(ci, ci_off);
+
+ VM_WARN_ON(!swp_tb_is_countable(swp_tb));
+ swp_tb &= ~SWP_TB_ZERO_FLAG;
+ __swap_table_set(ci, ci_off, swp_tb);
+#else
+ lockdep_assert_held(&ci->lock);
+ __clear_bit(ci_off, ci->zero_bitmap);
+#endif
+}
+
#ifdef CONFIG_MEMCG
static inline void __swap_cgroup_set(struct swap_cluster_info *ci,
unsigned int ci_off, unsigned long nr, unsigned short id)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 095e9c953e49..a9a1e477fec9 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -427,6 +427,11 @@ static void swap_cluster_free_table(struct swap_cluster_info *ci)
ci->memcg_table = NULL;
#endif
+#if !SWAP_TABLE_HAS_ZEROFLAG
+ kfree(ci->zero_bitmap);
+ ci->zero_bitmap = NULL;
+#endif
+
table = (struct swap_table *)rcu_access_pointer(ci->table);
if (!table)
return;
@@ -469,13 +474,21 @@ static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
VM_WARN_ON_ONCE(ci->memcg_table);
ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
if (!ci->memcg_table)
- ret = -ENOMEM;
+ goto err_free;
}
#endif
- if (ret)
- swap_cluster_free_table(ci);
- return ret;
+#if !SWAP_TABLE_HAS_ZEROFLAG
+ VM_WARN_ON_ONCE(ci->zero_bitmap);
+ ci->zero_bitmap = bitmap_zalloc(SWAPFILE_CLUSTER, gfp);
+ if (!ci->zero_bitmap)
+ goto err_free;
+#endif
+ return 0;
+
+err_free:
+ swap_cluster_free_table(ci);
+ return -ENOMEM;
}
/*
@@ -928,8 +941,8 @@ static bool __swap_cluster_alloc_entries(struct swap_info_struct *si,
order = 0;
nr_pages = 1;
swap_cluster_assert_empty(ci, ci_off, 1, false);
- /* Sets a fake shadow as placeholder */
- __swap_table_set(ci, ci_off, shadow_to_swp_tb(NULL, 1));
+ /* Fake shadow placeholder with no flag, hibernation does not use the zeromap */
+ __swap_table_set(ci, ci_off, __swp_tb_mk_count(shadow_to_swp_tb(NULL, 0), 1));
} else {
/* Allocation without folio is only possible with hibernation */
WARN_ON_ONCE(1);
@@ -1302,14 +1315,8 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
void (*swap_slot_free_notify)(struct block_device *, unsigned long);
unsigned int i;
- /*
- * Use atomic clear_bit operations only on zeromap instead of non-atomic
- * bitmap_clear to prevent adjacent bits corruption due to simultaneous writes.
- */
- for (i = 0; i < nr_entries; i++) {
- clear_bit(offset + i, si->zeromap);
+ for (i = 0; i < nr_entries; i++)
zswap_invalidate(swp_entry(si->type, offset + i));
- }
if (si->flags & SWP_BLKDEV)
swap_slot_free_notify =
@@ -1894,7 +1901,11 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
* ref, or after swap cache is dropped
*/
VM_WARN_ON(!swp_tb_is_shadow(old_tb) || __swp_tb_get_count(old_tb) > 1);
+
+ /* Resetting the slot to NULL also clears the inline flags. */
__swap_table_set(ci, ci_off, null_to_swp_tb());
+ if (!SWAP_TABLE_HAS_ZEROFLAG)
+ __swap_table_clear_zero(ci, ci_off);
/*
* Uncharge swap slots by memcg in batches. Consecutive
@@ -3088,7 +3099,6 @@ static void flush_percpu_swap_cluster(struct swap_info_struct *si)
SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
{
struct swap_info_struct *p = NULL;
- unsigned long *zeromap;
struct swap_cluster_info *cluster_info;
struct file *swap_file, *victim;
struct address_space *mapping;
@@ -3184,8 +3194,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
swap_file = p->swap_file;
p->swap_file = NULL;
- zeromap = p->zeromap;
- p->zeromap = NULL;
maxpages = p->max;
cluster_info = p->cluster_info;
p->max = 0;
@@ -3197,7 +3205,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
mutex_unlock(&swapon_mutex);
kfree(p->global_cluster);
p->global_cluster = NULL;
- kvfree(zeromap);
free_swap_cluster_info(cluster_info, maxpages);
inode = mapping->host;
@@ -3729,17 +3736,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
if (error)
goto bad_swap_unlock_inode;
- /*
- * Use kvmalloc_array instead of bitmap_zalloc as the allocation order might
- * be above MAX_PAGE_ORDER incase of a large swap file.
- */
- si->zeromap = kvmalloc_array(BITS_TO_LONGS(maxpages), sizeof(long),
- GFP_KERNEL | __GFP_ZERO);
- if (!si->zeromap) {
- error = -ENOMEM;
- goto bad_swap_unlock_inode;
- }
-
if (si->bdev && bdev_stable_writes(si->bdev))
si->flags |= SWP_STABLE_WRITES;
@@ -3841,8 +3837,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
destroy_swap_extents(si, swap_file);
free_swap_cluster_info(si->cluster_info, si->max);
si->cluster_info = NULL;
- kvfree(si->zeromap);
- si->zeromap = NULL;
/*
* Clear the SWP_USED flag after all resources are freed so
* alloc_swap_info can reuse this si safely.
--
2.54.0
^ permalink raw reply related
* [PATCH v5 07/12] mm, swap: support flexible batch freeing of slots in different memcgs
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
Instead of requiring the caller to ensure all slots are in the same
memcg, make the function handle different memcgs at once.
This is both a micro optimization and required for removing the memcg
lookup in the page table layer, so it can be unified at the swap layer.
We are not removing the memcg lookup in the page table in this commit.
It has to be done after the memcg lookup is deferred to the swap layer.
Acked-by: Chris Li <chrisl@kernel.org>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/swapfile.c | 33 +++++++++++++++++++++++++++++----
1 file changed, 29 insertions(+), 4 deletions(-)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 5c8bb15719bf..c9c80ba9252b 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1873,21 +1873,46 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
unsigned int ci_start, unsigned int nr_pages)
{
unsigned long old_tb;
+ unsigned int type = si->type;
+ unsigned short batch_id = 0, id_cur;
unsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;
- unsigned long offset = cluster_offset(si, ci) + ci_start;
+ unsigned long ci_head = cluster_offset(si, ci);
+ unsigned int batch_off = ci_off;
+ swp_entry_t entry;
VM_WARN_ON(ci->count < nr_pages);
ci->count -= nr_pages;
do {
old_tb = __swap_table_get(ci, ci_off);
- /* Release the last ref, or after swap cache is dropped */
+ /*
+ * Freeing is done after release of the last swap count
+ * ref, or after swap cache is dropped
+ */
VM_WARN_ON(!swp_tb_is_shadow(old_tb) || __swp_tb_get_count(old_tb) > 1);
__swap_table_set(ci, ci_off, null_to_swp_tb());
+
+ /*
+ * Uncharge swap slots by memcg in batches. Consecutive
+ * slots with the same cgroup id are uncharged together.
+ */
+ entry = swp_entry(type, ci_head + ci_off);
+ id_cur = lookup_swap_cgroup_id(entry);
+ if (batch_id != id_cur) {
+ if (batch_id)
+ mem_cgroup_uncharge_swap(swp_entry(type, ci_head + batch_off),
+ ci_off - batch_off);
+ batch_id = id_cur;
+ batch_off = ci_off;
+ }
} while (++ci_off < ci_end);
- mem_cgroup_uncharge_swap(swp_entry(si->type, offset), nr_pages);
- swap_range_free(si, offset, nr_pages);
+ if (batch_id) {
+ mem_cgroup_uncharge_swap(swp_entry(type, ci_head + batch_off),
+ ci_off - batch_off);
+ }
+
+ swap_range_free(si, ci_head + ci_start, nr_pages);
swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
if (!ci->count)
--
2.54.0
^ permalink raw reply related
* [PATCH v5 10/12] mm/memcg, swap: store cgroup id in cluster table directly
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
Drop the usage of the swap_cgroup_ctrl, and use the dynamic cluster
table instead.
The per-cluster memcg table is 1024 / 512 bytes on most archs, and does
not need RCU protection: the cgroup data is only read and written under
the cluster lock. That keeps things simple, lets the allocation use
plain kmalloc with immediate kfree (no deferred free), and keeps
fragmentation acceptable.
Acked-by: Chris Li <chrisl@kernel.org>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/memcontrol.h | 6 +++--
include/linux/swap.h | 8 +++---
mm/memcontrol-v1.c | 42 +++++++++++++++++++-----------
mm/memcontrol.c | 13 ++++++----
mm/swap.h | 4 +++
mm/swap_state.c | 6 ++---
mm/swap_table.h | 64 ++++++++++++++++++++++++++++++++++++++++++++++
mm/swapfile.c | 37 ++++++++++++++++++---------
mm/vmscan.c | 2 +-
9 files changed, 139 insertions(+), 43 deletions(-)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index a013f37f24aa..bf1a6e131eca 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -29,6 +29,7 @@ struct obj_cgroup;
struct page;
struct mm_struct;
struct kmem_cache;
+struct swap_cluster_info;
/* Cgroup-specific page state, on top of universal node page state */
enum memcg_stat_item {
@@ -1899,7 +1900,7 @@ static inline void mem_cgroup_exit_user_fault(void)
current->in_user_fault = 0;
}
-void __memcg1_swapout(struct folio *folio);
+void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci);
void memcg1_swapin(struct folio *folio);
#else /* CONFIG_MEMCG_V1 */
@@ -1929,7 +1930,8 @@ static inline void mem_cgroup_exit_user_fault(void)
{
}
-static inline void __memcg1_swapout(struct folio *folio)
+static inline void __memcg1_swapout(struct folio *folio,
+ struct swap_cluster_info *ci)
{
}
diff --git a/include/linux/swap.h b/include/linux/swap.h
index 6b3acdf9bdd4..203bbe23ba1f 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -584,12 +584,12 @@ static inline int mem_cgroup_try_charge_swap(struct folio *folio)
return __mem_cgroup_try_charge_swap(folio);
}
-extern void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages);
-static inline void mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)
+extern void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages);
+static inline void mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
{
if (mem_cgroup_disabled())
return;
- __mem_cgroup_uncharge_swap(entry, nr_pages);
+ __mem_cgroup_uncharge_swap(id, nr_pages);
}
extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
@@ -600,7 +600,7 @@ static inline int mem_cgroup_try_charge_swap(struct folio *folio)
return 0;
}
-static inline void mem_cgroup_uncharge_swap(swp_entry_t entry,
+static inline void mem_cgroup_uncharge_swap(unsigned short id,
unsigned int nr_pages)
{
}
diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
index 36c507d81dc5..494e7b9adc60 100644
--- a/mm/memcontrol-v1.c
+++ b/mm/memcontrol-v1.c
@@ -14,6 +14,7 @@
#include "internal.h"
#include "swap.h"
+#include "swap_table.h"
#include "memcontrol-v1.h"
/*
@@ -606,14 +607,15 @@ void memcg1_commit_charge(struct folio *folio, struct mem_cgroup *memcg)
/**
* __memcg1_swapout - transfer a memsw charge to swap
* @folio: folio whose memsw charge to transfer
+ * @ci: the locked swap cluster holding the swap entries
*
* Transfer the memsw charge of @folio to the swap entry stored in
* folio->swap.
*
- * Context: folio must be isolated, unmapped, locked and is just about
- * to be freed, and caller must disable IRQs.
+ * Context: folio must be isolated, unmapped, locked and is just about to
+ * be freed, and caller must disable IRQs and hold the swap cluster lock.
*/
-void __memcg1_swapout(struct folio *folio)
+void __memcg1_swapout(struct folio *folio, struct swap_cluster_info *ci)
{
struct mem_cgroup *memcg, *swap_memcg;
struct obj_cgroup *objcg;
@@ -646,7 +648,8 @@ void __memcg1_swapout(struct folio *folio)
swap_memcg = mem_cgroup_private_id_get_online(memcg, nr_entries);
mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries);
- swap_cgroup_record(folio, mem_cgroup_private_id(swap_memcg), folio->swap);
+ __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_entries,
+ mem_cgroup_private_id(swap_memcg));
folio_unqueue_deferred_split(folio);
folio->memcg_data = 0;
@@ -661,8 +664,7 @@ void __memcg1_swapout(struct folio *folio)
}
/*
- * Interrupts should be disabled here because the caller holds the
- * i_pages lock which is taken with interrupts-off. It is
+ * The caller must hold the swap cluster lock with IRQ off. It is
* important here to have the interrupts disabled because it is the
* only synchronisation we have for updating the per-CPU variables.
*/
@@ -677,7 +679,7 @@ void __memcg1_swapout(struct folio *folio)
}
/**
- * memcg1_swapin - uncharge swap slot
+ * memcg1_swapin - uncharge swap slot on swapin
* @folio: folio being swapped in
*
* Call this function after successfully adding the charged
@@ -687,6 +689,10 @@ void __memcg1_swapout(struct folio *folio)
*/
void memcg1_swapin(struct folio *folio)
{
+ struct swap_cluster_info *ci;
+ unsigned long nr_pages;
+ unsigned short id;
+
VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
@@ -702,14 +708,20 @@ void memcg1_swapin(struct folio *folio)
* correspond 1:1 to page and swap slot lifetimes: we charge the
* page to memory here, and uncharge swap when the slot is freed.
*/
- if (do_memsw_account()) {
- /*
- * The swap entry might not get freed for a long time,
- * let's not wait for it. The page already received a
- * memory+swap charge, drop the swap entry duplicate.
- */
- mem_cgroup_uncharge_swap(folio->swap, folio_nr_pages(folio));
- }
+ if (!do_memsw_account())
+ return;
+
+ /*
+ * The swap entry might not get freed for a long time,
+ * let's not wait for it. The page already received a
+ * memory+swap charge, drop the swap entry duplicate.
+ */
+ nr_pages = folio_nr_pages(folio);
+ ci = swap_cluster_get_and_lock(folio);
+ id = __swap_cgroup_clear(ci, swp_cluster_offset(folio->swap),
+ nr_pages);
+ swap_cluster_unlock(ci);
+ mem_cgroup_uncharge_swap(id, nr_pages);
}
void memcg1_uncharge_batch(struct mem_cgroup *memcg, unsigned long pgpgout,
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 4f940cf22ffe..b5c267a061a9 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -64,6 +64,7 @@
#include <linux/sched/isolation.h>
#include <linux/kmemleak.h>
#include "internal.h"
+#include "swap_table.h"
#include <net/sock.h>
#include <net/ip.h>
#include "slab.h"
@@ -5470,6 +5471,7 @@ int __init mem_cgroup_init(void)
int __mem_cgroup_try_charge_swap(struct folio *folio)
{
unsigned int nr_pages = folio_nr_pages(folio);
+ struct swap_cluster_info *ci;
struct page_counter *counter;
struct mem_cgroup *memcg;
struct obj_cgroup *objcg;
@@ -5503,22 +5505,23 @@ int __mem_cgroup_try_charge_swap(struct folio *folio)
}
mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
- swap_cgroup_record(folio, mem_cgroup_private_id(memcg), folio->swap);
+ ci = swap_cluster_get_and_lock(folio);
+ __swap_cgroup_set(ci, swp_cluster_offset(folio->swap), nr_pages,
+ mem_cgroup_private_id(memcg));
+ swap_cluster_unlock(ci);
return 0;
}
/**
* __mem_cgroup_uncharge_swap - uncharge swap space
- * @entry: swap entry to uncharge
+ * @id: cgroup id to uncharge
* @nr_pages: the amount of swap space to uncharge
*/
-void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages)
+void __mem_cgroup_uncharge_swap(unsigned short id, unsigned int nr_pages)
{
struct mem_cgroup *memcg;
- unsigned short id;
- id = swap_cgroup_clear(entry, nr_pages);
rcu_read_lock();
memcg = mem_cgroup_from_private_id(id);
if (memcg) {
diff --git a/mm/swap.h b/mm/swap.h
index 8e57e9431624..5b2f095fff6e 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -5,6 +5,7 @@
#include <linux/atomic.h> /* for atomic_long_t */
struct mempolicy;
struct swap_iocb;
+struct swap_memcg_table;
extern int page_cluster;
@@ -38,6 +39,9 @@ struct swap_cluster_info {
u8 order;
atomic_long_t __rcu *table; /* Swap table entries, see mm/swap_table.h */
unsigned int *extend_table; /* For large swap count, protected by ci->lock */
+#ifdef CONFIG_MEMCG
+ struct swap_memcg_table *memcg_table; /* Swap table entries' cgroup record */
+#endif
struct list_head list;
};
diff --git a/mm/swap_state.c b/mm/swap_state.c
index bdd949ae0044..873cb3f26337 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -179,21 +179,19 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
if (shadowp && swp_tb_is_shadow(old_tb))
*shadowp = swp_tb_to_shadow(old_tb);
if (memcg_id)
- *memcg_id = lookup_swap_cgroup_id(targ_entry);
+ *memcg_id = __swap_cgroup_get(ci, ci_off);
if (nr == 1)
return 0;
- targ_entry.val = round_down(targ_entry.val, nr);
ci_off = round_down(ci_off, nr);
ci_end = ci_off + nr;
do {
old_tb = __swap_table_get(ci, ci_off);
if (unlikely(swp_tb_is_folio(old_tb) ||
!__swp_tb_get_count(old_tb) ||
- (memcg_id && *memcg_id != lookup_swap_cgroup_id(targ_entry))))
+ (memcg_id && *memcg_id != __swap_cgroup_get(ci, ci_off))))
return -EBUSY;
- targ_entry.val++;
} while (++ci_off < ci_end);
return 0;
diff --git a/mm/swap_table.h b/mm/swap_table.h
index 8415ffbe2b9c..b4e1100f8296 100644
--- a/mm/swap_table.h
+++ b/mm/swap_table.h
@@ -11,6 +11,11 @@ struct swap_table {
atomic_long_t entries[SWAPFILE_CLUSTER];
};
+/* For storing memcg private id */
+struct swap_memcg_table {
+ unsigned short id[SWAPFILE_CLUSTER];
+};
+
#define SWP_TABLE_USE_PAGE (sizeof(struct swap_table) == PAGE_SIZE)
/*
@@ -247,4 +252,63 @@ static inline unsigned long swap_table_get(struct swap_cluster_info *ci,
return swp_tb;
}
+
+#ifdef CONFIG_MEMCG
+static inline void __swap_cgroup_set(struct swap_cluster_info *ci,
+ unsigned int ci_off, unsigned long nr, unsigned short id)
+{
+ lockdep_assert_held(&ci->lock);
+ VM_WARN_ON_ONCE(ci_off >= SWAPFILE_CLUSTER);
+ if (WARN_ON_ONCE(!ci->memcg_table))
+ return;
+ do {
+ ci->memcg_table->id[ci_off++] = id;
+ } while (--nr);
+}
+
+static inline unsigned short __swap_cgroup_get(struct swap_cluster_info *ci,
+ unsigned int ci_off)
+{
+ lockdep_assert_held(&ci->lock);
+ VM_WARN_ON_ONCE(ci_off >= SWAPFILE_CLUSTER);
+ if (unlikely(!ci->memcg_table))
+ return 0;
+ return ci->memcg_table->id[ci_off];
+}
+
+static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
+ unsigned int ci_off,
+ unsigned long nr)
+{
+ unsigned short old = __swap_cgroup_get(ci, ci_off);
+
+ if (!old)
+ return 0;
+ do {
+ VM_WARN_ON_ONCE(ci->memcg_table->id[ci_off] != old);
+ ci->memcg_table->id[ci_off++] = 0;
+ } while (--nr);
+
+ return old;
+}
+#else
+static inline void __swap_cgroup_set(struct swap_cluster_info *ci,
+ unsigned int ci_off, unsigned long nr, unsigned short id)
+{
+}
+
+static inline unsigned short __swap_cgroup_get(struct swap_cluster_info *ci,
+ unsigned int ci_off)
+{
+ return 0;
+}
+
+static inline unsigned short __swap_cgroup_clear(struct swap_cluster_info *ci,
+ unsigned int ci_off,
+ unsigned long nr)
+{
+ return 0;
+}
+#endif
+
#endif
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 7740ba99f87e..ae14d4049e4b 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -423,7 +423,12 @@ static void swap_cluster_free_table(struct swap_cluster_info *ci)
{
struct swap_table *table;
- table = (struct swap_table *)rcu_dereference_protected(ci->table, true);
+#ifdef CONFIG_MEMCG
+ kfree(ci->memcg_table);
+ ci->memcg_table = NULL;
+#endif
+
+ table = (struct swap_table *)rcu_access_pointer(ci->table);
if (!table)
return;
@@ -441,6 +446,7 @@ static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
{
struct swap_table *table = NULL;
struct folio *folio;
+ int ret = 0;
/* The cluster must be empty and not on any list during allocation. */
VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));
@@ -458,7 +464,19 @@ static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
return -ENOMEM;
rcu_assign_pointer(ci->table, table);
- return 0;
+
+#ifdef CONFIG_MEMCG
+ if (!mem_cgroup_disabled()) {
+ VM_WARN_ON_ONCE(ci->memcg_table);
+ ci->memcg_table = kzalloc_obj(*ci->memcg_table, gfp);
+ if (!ci->memcg_table)
+ ret = -ENOMEM;
+ }
+#endif
+ if (ret)
+ swap_cluster_free_table(ci);
+
+ return ret;
}
/*
@@ -483,6 +501,7 @@ static void swap_cluster_assert_empty(struct swap_cluster_info *ci,
bad_slots++;
else
WARN_ON_ONCE(!swp_tb_is_null(swp_tb));
+ WARN_ON_ONCE(__swap_cgroup_get(ci, ci_off));
} while (++ci_off < ci_end);
WARN_ON_ONCE(bad_slots != (swapoff ? ci->count : 0));
@@ -1861,12 +1880,10 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
unsigned int ci_start, unsigned int nr_pages)
{
unsigned long old_tb;
- unsigned int type = si->type;
unsigned short batch_id = 0, id_cur;
unsigned int ci_off = ci_start, ci_end = ci_start + nr_pages;
unsigned long ci_head = cluster_offset(si, ci);
unsigned int batch_off = ci_off;
- swp_entry_t entry;
VM_WARN_ON(ci->count < nr_pages);
@@ -1884,21 +1901,17 @@ void __swap_cluster_free_entries(struct swap_info_struct *si,
* Uncharge swap slots by memcg in batches. Consecutive
* slots with the same cgroup id are uncharged together.
*/
- entry = swp_entry(type, ci_head + ci_off);
- id_cur = lookup_swap_cgroup_id(entry);
+ id_cur = __swap_cgroup_clear(ci, ci_off, 1);
if (batch_id != id_cur) {
if (batch_id)
- mem_cgroup_uncharge_swap(swp_entry(type, ci_head + batch_off),
- ci_off - batch_off);
+ mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
batch_id = id_cur;
batch_off = ci_off;
}
} while (++ci_off < ci_end);
- if (batch_id) {
- mem_cgroup_uncharge_swap(swp_entry(type, ci_head + batch_off),
- ci_off - batch_off);
- }
+ if (batch_id)
+ mem_cgroup_uncharge_swap(batch_id, ci_off - batch_off);
swap_range_free(si, ci_head + ci_start, nr_pages);
swap_cluster_assert_empty(ci, ci_start, nr_pages, false);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 924c84326551..ca4533eba701 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -737,7 +737,7 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,
if (reclaimed && !mapping_exiting(mapping))
shadow = workingset_eviction(folio, target_memcg);
- __memcg1_swapout(folio);
+ __memcg1_swapout(folio, ci);
__swap_cache_del_folio(ci, folio, swap, shadow);
swap_cluster_unlock_irq(ci);
} else {
--
2.54.0
^ permalink raw reply related
* [PATCH v5 11/12] mm/memcg: remove no longer used swap cgroup array
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
Now all swap cgroup records are stored in the swap cluster directly,
the static array is no longer needed.
Acked-by: Chris Li <chrisl@kernel.org>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
MAINTAINERS | 1 -
include/linux/swap_cgroup.h | 47 ------------
mm/Makefile | 3 -
mm/internal.h | 1 -
mm/memcontrol-v1.c | 1 -
mm/memcontrol.c | 1 -
mm/swap_cgroup.c | 174 --------------------------------------------
mm/swapfile.c | 8 --
8 files changed, 236 deletions(-)
diff --git a/MAINTAINERS b/MAINTAINERS
index 0116eb99b708..9be179722d42 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6564,7 +6564,6 @@ F: mm/memcontrol.c
F: mm/memcontrol-v1.c
F: mm/memcontrol-v1.h
F: mm/page_counter.c
-F: mm/swap_cgroup.c
F: samples/cgroup/*
F: tools/testing/selftests/cgroup/memcg_protection.m
F: tools/testing/selftests/cgroup/test_hugetlb_memcg.c
diff --git a/include/linux/swap_cgroup.h b/include/linux/swap_cgroup.h
deleted file mode 100644
index 91cdf12190a0..000000000000
--- a/include/linux/swap_cgroup.h
+++ /dev/null
@@ -1,47 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef __LINUX_SWAP_CGROUP_H
-#define __LINUX_SWAP_CGROUP_H
-
-#include <linux/swap.h>
-
-#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
-
-extern void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent);
-extern unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents);
-extern unsigned short lookup_swap_cgroup_id(swp_entry_t ent);
-extern int swap_cgroup_swapon(int type, unsigned long max_pages);
-extern void swap_cgroup_swapoff(int type);
-
-#else
-
-static inline
-void swap_cgroup_record(struct folio *folio, unsigned short id, swp_entry_t ent)
-{
-}
-
-static inline
-unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents)
-{
- return 0;
-}
-
-static inline
-unsigned short lookup_swap_cgroup_id(swp_entry_t ent)
-{
- return 0;
-}
-
-static inline int
-swap_cgroup_swapon(int type, unsigned long max_pages)
-{
- return 0;
-}
-
-static inline void swap_cgroup_swapoff(int type)
-{
- return;
-}
-
-#endif
-
-#endif /* __LINUX_SWAP_CGROUP_H */
diff --git a/mm/Makefile b/mm/Makefile
index 8ad2ab08244e..eff9f9e7e061 100644
--- a/mm/Makefile
+++ b/mm/Makefile
@@ -103,9 +103,6 @@ obj-$(CONFIG_PAGE_COUNTER) += page_counter.o
obj-$(CONFIG_LIVEUPDATE_MEMFD) += memfd_luo.o
obj-$(CONFIG_MEMCG_V1) += memcontrol-v1.o
obj-$(CONFIG_MEMCG) += memcontrol.o vmpressure.o
-ifdef CONFIG_SWAP
-obj-$(CONFIG_MEMCG) += swap_cgroup.o
-endif
ifdef CONFIG_BPF_SYSCALL
obj-$(CONFIG_MEMCG) += bpf_memcontrol.o
endif
diff --git a/mm/internal.h b/mm/internal.h
index 9d2fec696bd6..7646ecb9d621 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -17,7 +17,6 @@
#include <linux/rmap.h>
#include <linux/swap.h>
#include <linux/leafops.h>
-#include <linux/swap_cgroup.h>
#include <linux/tracepoint-defs.h>
/* Internal core VMA manipulation functions. */
diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
index 494e7b9adc60..08be1a752c2e 100644
--- a/mm/memcontrol-v1.c
+++ b/mm/memcontrol-v1.c
@@ -5,7 +5,6 @@
#include <linux/mm_inline.h>
#include <linux/pagewalk.h>
#include <linux/backing-dev.h>
-#include <linux/swap_cgroup.h>
#include <linux/eventfd.h>
#include <linux/poll.h>
#include <linux/sort.h>
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index b5c267a061a9..039e9bc8971c 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -54,7 +54,6 @@
#include <linux/vmpressure.h>
#include <linux/memremap.h>
#include <linux/mm_inline.h>
-#include <linux/swap_cgroup.h>
#include <linux/cpu.h>
#include <linux/oom.h>
#include <linux/lockdep.h>
diff --git a/mm/swap_cgroup.c b/mm/swap_cgroup.c
deleted file mode 100644
index 95c38e54dd58..000000000000
--- a/mm/swap_cgroup.c
+++ /dev/null
@@ -1,174 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-#include <linux/swap_cgroup.h>
-#include <linux/vmalloc.h>
-#include <linux/mm.h>
-
-#include <linux/swapops.h> /* depends on mm.h include */
-
-static DEFINE_MUTEX(swap_cgroup_mutex);
-
-/* Pack two cgroup id (short) of two entries in one swap_cgroup (atomic_t) */
-#define ID_PER_SC (sizeof(struct swap_cgroup) / sizeof(unsigned short))
-#define ID_SHIFT (BITS_PER_TYPE(unsigned short))
-#define ID_MASK (BIT(ID_SHIFT) - 1)
-struct swap_cgroup {
- atomic_t ids;
-};
-
-struct swap_cgroup_ctrl {
- struct swap_cgroup *map;
-};
-
-static struct swap_cgroup_ctrl swap_cgroup_ctrl[MAX_SWAPFILES];
-
-static unsigned short __swap_cgroup_id_lookup(struct swap_cgroup *map,
- pgoff_t offset)
-{
- unsigned int shift = (offset % ID_PER_SC) * ID_SHIFT;
- unsigned int old_ids = atomic_read(&map[offset / ID_PER_SC].ids);
-
- BUILD_BUG_ON(!is_power_of_2(ID_PER_SC));
- BUILD_BUG_ON(sizeof(struct swap_cgroup) != sizeof(atomic_t));
-
- return (old_ids >> shift) & ID_MASK;
-}
-
-static unsigned short __swap_cgroup_id_xchg(struct swap_cgroup *map,
- pgoff_t offset,
- unsigned short new_id)
-{
- unsigned short old_id;
- struct swap_cgroup *sc = &map[offset / ID_PER_SC];
- unsigned int shift = (offset % ID_PER_SC) * ID_SHIFT;
- unsigned int new_ids, old_ids = atomic_read(&sc->ids);
-
- do {
- old_id = (old_ids >> shift) & ID_MASK;
- new_ids = (old_ids & ~(ID_MASK << shift));
- new_ids |= ((unsigned int)new_id) << shift;
- } while (!atomic_try_cmpxchg(&sc->ids, &old_ids, new_ids));
-
- return old_id;
-}
-
-/**
- * swap_cgroup_record - record mem_cgroup for a set of swap entries.
- * These entries must belong to one single folio, and that folio
- * must be being charged for swap space (swap out), and these
- * entries must not have been charged
- *
- * @folio: the folio that the swap entry belongs to
- * @id: mem_cgroup ID to be recorded
- * @ent: the first swap entry to be recorded
- */
-void swap_cgroup_record(struct folio *folio, unsigned short id,
- swp_entry_t ent)
-{
- unsigned int nr_ents = folio_nr_pages(folio);
- struct swap_cgroup *map;
- pgoff_t offset, end;
- unsigned short old;
-
- offset = swp_offset(ent);
- end = offset + nr_ents;
- map = swap_cgroup_ctrl[swp_type(ent)].map;
-
- do {
- old = __swap_cgroup_id_xchg(map, offset, id);
- VM_BUG_ON(old);
- } while (++offset != end);
-}
-
-/**
- * swap_cgroup_clear - clear mem_cgroup for a set of swap entries.
- * These entries must be being uncharged from swap. They either
- * belongs to one single folio in the swap cache (swap in for
- * cgroup v1), or no longer have any users (slot freeing).
- *
- * @ent: the first swap entry to be recorded into
- * @nr_ents: number of swap entries to be recorded
- *
- * Returns the existing old value.
- */
-unsigned short swap_cgroup_clear(swp_entry_t ent, unsigned int nr_ents)
-{
- pgoff_t offset, end;
- struct swap_cgroup *map;
- unsigned short old, iter = 0;
-
- offset = swp_offset(ent);
- end = offset + nr_ents;
- map = swap_cgroup_ctrl[swp_type(ent)].map;
-
- do {
- old = __swap_cgroup_id_xchg(map, offset, 0);
- if (!iter)
- iter = old;
- VM_BUG_ON(iter != old);
- } while (++offset != end);
-
- return old;
-}
-
-/**
- * lookup_swap_cgroup_id - lookup mem_cgroup id tied to swap entry
- * @ent: swap entry to be looked up.
- *
- * Returns ID of mem_cgroup at success. 0 at failure. (0 is invalid ID)
- */
-unsigned short lookup_swap_cgroup_id(swp_entry_t ent)
-{
- struct swap_cgroup_ctrl *ctrl;
-
- if (mem_cgroup_disabled())
- return 0;
-
- ctrl = &swap_cgroup_ctrl[swp_type(ent)];
- if (unlikely(!ctrl->map))
- return 0;
- return __swap_cgroup_id_lookup(ctrl->map, swp_offset(ent));
-}
-
-int swap_cgroup_swapon(int type, unsigned long max_pages)
-{
- struct swap_cgroup *map;
- struct swap_cgroup_ctrl *ctrl;
-
- if (mem_cgroup_disabled())
- return 0;
-
- BUILD_BUG_ON(sizeof(unsigned short) * ID_PER_SC !=
- sizeof(struct swap_cgroup));
- map = vzalloc(DIV_ROUND_UP(max_pages, ID_PER_SC) *
- sizeof(struct swap_cgroup));
- if (!map)
- goto nomem;
-
- ctrl = &swap_cgroup_ctrl[type];
- mutex_lock(&swap_cgroup_mutex);
- ctrl->map = map;
- mutex_unlock(&swap_cgroup_mutex);
-
- return 0;
-nomem:
- pr_info("couldn't allocate enough memory for swap_cgroup\n");
- pr_info("swap_cgroup can be disabled by swapaccount=0 boot option\n");
- return -ENOMEM;
-}
-
-void swap_cgroup_swapoff(int type)
-{
- struct swap_cgroup *map;
- struct swap_cgroup_ctrl *ctrl;
-
- if (mem_cgroup_disabled())
- return;
-
- mutex_lock(&swap_cgroup_mutex);
- ctrl = &swap_cgroup_ctrl[type];
- map = ctrl->map;
- ctrl->map = NULL;
- mutex_unlock(&swap_cgroup_mutex);
-
- vfree(map);
-}
diff --git a/mm/swapfile.c b/mm/swapfile.c
index ae14d4049e4b..095e9c953e49 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -45,7 +45,6 @@
#include <asm/tlbflush.h>
#include <linux/leafops.h>
-#include <linux/swap_cgroup.h>
#include "swap_table.h"
#include "internal.h"
#include "swap.h"
@@ -3200,8 +3199,6 @@ SYSCALL_DEFINE1(swapoff, const char __user *, specialfile)
p->global_cluster = NULL;
kvfree(zeromap);
free_swap_cluster_info(cluster_info, maxpages);
- /* Destroy swap account information */
- swap_cgroup_swapoff(p->type);
inode = mapping->host;
@@ -3732,10 +3729,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
if (error)
goto bad_swap_unlock_inode;
- error = swap_cgroup_swapon(si->type, maxpages);
- if (error)
- goto bad_swap_unlock_inode;
-
/*
* Use kvmalloc_array instead of bitmap_zalloc as the allocation order might
* be above MAX_PAGE_ORDER incase of a large swap file.
@@ -3846,7 +3839,6 @@ SYSCALL_DEFINE2(swapon, const char __user *, specialfile, int, swap_flags)
si->global_cluster = NULL;
inode = NULL;
destroy_swap_extents(si, swap_file);
- swap_cgroup_swapoff(si->type);
free_swap_cluster_info(si->cluster_info, si->max);
si->cluster_info = NULL;
kvfree(si->zeromap);
--
2.54.0
^ permalink raw reply related
* [PATCH v5 09/12] mm, swap: consolidate cluster allocation helpers
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
Swap cluster table management is spread across several narrow
helpers. As a result, the allocation and fallback sequences are
open-coded in multiple places.
A few more per-cluster tables will be added soon, so avoid
duplicating these sequences per table type. Fold the existing
pairs into cluster-oriented helpers, and rename for consistency.
No functional change, only a few sanity checks are slightly adjusted.
Acked-by: Chris Li <chrisl@kernel.org>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/swapfile.c | 110 ++++++++++++++++++++++++++--------------------------------
1 file changed, 49 insertions(+), 61 deletions(-)
diff --git a/mm/swapfile.c b/mm/swapfile.c
index c9c80ba9252b..7740ba99f87e 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -411,20 +411,7 @@ static inline unsigned int cluster_offset(struct swap_info_struct *si,
return cluster_index(si, ci) * SWAPFILE_CLUSTER;
}
-static struct swap_table *swap_table_alloc(gfp_t gfp)
-{
- struct folio *folio;
-
- if (!SWP_TABLE_USE_PAGE)
- return kmem_cache_zalloc(swap_table_cachep, gfp);
-
- folio = folio_alloc(gfp | __GFP_ZERO, 0);
- if (folio)
- return folio_address(folio);
- return NULL;
-}
-
-static void swap_table_free_folio_rcu_cb(struct rcu_head *head)
+static void swap_cluster_free_table_folio_rcu_cb(struct rcu_head *head)
{
struct folio *folio;
@@ -432,15 +419,46 @@ static void swap_table_free_folio_rcu_cb(struct rcu_head *head)
folio_put(folio);
}
-static void swap_table_free(struct swap_table *table)
+static void swap_cluster_free_table(struct swap_cluster_info *ci)
{
+ struct swap_table *table;
+
+ table = (struct swap_table *)rcu_dereference_protected(ci->table, true);
+ if (!table)
+ return;
+
+ rcu_assign_pointer(ci->table, NULL);
if (!SWP_TABLE_USE_PAGE) {
kmem_cache_free(swap_table_cachep, table);
return;
}
call_rcu(&(folio_page(virt_to_folio(table), 0)->rcu_head),
- swap_table_free_folio_rcu_cb);
+ swap_cluster_free_table_folio_rcu_cb);
+}
+
+static int swap_cluster_alloc_table(struct swap_cluster_info *ci, gfp_t gfp)
+{
+ struct swap_table *table = NULL;
+ struct folio *folio;
+
+ /* The cluster must be empty and not on any list during allocation. */
+ VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));
+ if (rcu_access_pointer(ci->table))
+ return 0;
+
+ if (SWP_TABLE_USE_PAGE) {
+ folio = folio_alloc(gfp | __GFP_ZERO, 0);
+ if (folio)
+ table = folio_address(folio);
+ } else {
+ table = kmem_cache_zalloc(swap_table_cachep, gfp);
+ }
+ if (!table)
+ return -ENOMEM;
+
+ rcu_assign_pointer(ci->table, table);
+ return 0;
}
/*
@@ -471,27 +489,15 @@ static void swap_cluster_assert_empty(struct swap_cluster_info *ci,
WARN_ON_ONCE(nr == SWAPFILE_CLUSTER && ci->extend_table);
}
-static void swap_cluster_free_table(struct swap_cluster_info *ci)
-{
- struct swap_table *table;
-
- /* Only empty cluster's table is allow to be freed */
- lockdep_assert_held(&ci->lock);
- table = (void *)rcu_dereference_protected(ci->table, true);
- rcu_assign_pointer(ci->table, NULL);
-
- swap_table_free(table);
-}
-
/*
* Allocate swap table for one cluster. Attempt an atomic allocation first,
* then fallback to sleeping allocation.
*/
static struct swap_cluster_info *
-swap_cluster_alloc_table(struct swap_info_struct *si,
+swap_cluster_populate(struct swap_info_struct *si,
struct swap_cluster_info *ci)
{
- struct swap_table *table;
+ int ret;
/*
* Only cluster isolation from the allocator does table allocation.
@@ -502,14 +508,9 @@ swap_cluster_alloc_table(struct swap_info_struct *si,
lockdep_assert_held(&si->global_cluster_lock);
lockdep_assert_held(&ci->lock);
- /* The cluster must be free and was just isolated from the free list. */
- VM_WARN_ON_ONCE(ci->flags || !cluster_is_empty(ci));
-
- table = swap_table_alloc(__GFP_HIGH | __GFP_NOMEMALLOC | __GFP_NOWARN);
- if (table) {
- rcu_assign_pointer(ci->table, table);
+ if (!swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
+ __GFP_NOWARN))
return ci;
- }
/*
* Try a sleep allocation. Each isolated free cluster may cause
@@ -521,7 +522,8 @@ swap_cluster_alloc_table(struct swap_info_struct *si,
spin_unlock(&si->global_cluster_lock);
local_unlock(&percpu_swap_cluster.lock);
- table = swap_table_alloc(__GFP_HIGH | __GFP_NOMEMALLOC | GFP_KERNEL);
+ ret = swap_cluster_alloc_table(ci, __GFP_HIGH | __GFP_NOMEMALLOC |
+ GFP_KERNEL);
/*
* Back to atomic context. We might have migrated to a new CPU with a
@@ -536,20 +538,11 @@ swap_cluster_alloc_table(struct swap_info_struct *si,
spin_lock(&si->global_cluster_lock);
spin_lock(&ci->lock);
- /* Nothing except this helper should touch a dangling empty cluster. */
- if (WARN_ON_ONCE(cluster_table_is_alloced(ci))) {
- if (table)
- swap_table_free(table);
- return ci;
- }
-
- if (!table) {
+ if (ret) {
move_cluster(si, ci, &si->free_clusters, CLUSTER_FLAG_FREE);
spin_unlock(&ci->lock);
return NULL;
}
-
- rcu_assign_pointer(ci->table, table);
return ci;
}
@@ -621,12 +614,11 @@ static struct swap_cluster_info *isolate_lock_cluster(
}
spin_unlock(&si->lock);
- if (found && !cluster_table_is_alloced(found)) {
- /* Only an empty free cluster's swap table can be freed. */
- VM_WARN_ON_ONCE(flags != CLUSTER_FLAG_FREE);
+ /* Cluster's table is freed when and only when it's on the free list. */
+ if (found && flags == CLUSTER_FLAG_FREE) {
VM_WARN_ON_ONCE(list != &si->free_clusters);
- VM_WARN_ON_ONCE(!cluster_is_empty(found));
- return swap_cluster_alloc_table(si, found);
+ VM_WARN_ON_ONCE(cluster_table_is_alloced(found));
+ return swap_cluster_populate(si, found);
}
return found;
@@ -769,7 +761,6 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
unsigned int ci_off = offset % SWAPFILE_CLUSTER;
unsigned long idx = offset / SWAPFILE_CLUSTER;
struct swap_cluster_info *ci;
- struct swap_table *table;
int ret = 0;
/* si->max may got shrunk by swap swap_activate() */
@@ -790,12 +781,9 @@ static int swap_cluster_setup_bad_slot(struct swap_info_struct *si,
}
ci = cluster_info + idx;
- if (!ci->table) {
- table = swap_table_alloc(GFP_KERNEL);
- if (!table)
- return -ENOMEM;
- rcu_assign_pointer(ci->table, table);
- }
+ /* Need to allocate swap table first for initial bad slot marking. */
+ if (!ci->count && swap_cluster_alloc_table(ci, GFP_KERNEL))
+ return -ENOMEM;
spin_lock(&ci->lock);
/* Check for duplicated bad swap slots. */
if (__swap_table_xchg(ci, ci_off, SWP_TB_BAD) != SWP_TB_NULL) {
@@ -3054,7 +3042,7 @@ static void free_swap_cluster_info(struct swap_cluster_info *cluster_info,
ci = cluster_info + i;
/* Cluster with bad marks count will have a remaining table */
spin_lock(&ci->lock);
- if (rcu_dereference_protected(ci->table, true)) {
+ if (cluster_table_is_alloced(ci)) {
swap_cluster_assert_empty(ci, 0, SWAPFILE_CLUSTER, true);
swap_cluster_free_table(ci);
}
--
2.54.0
^ permalink raw reply related
* [PATCH v5 08/12] mm, swap: delay and unify memcg lookup and charging for swapin
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
Instead of checking the cgroup private ID during page table walk in
swap_pte_batch(), move the memcg lookup into __swap_cache_add_check()
under the cluster lock.
The first pre-alloc check is speculative and skips the memcg check since
the post-alloc stable check ensures all slots covered by the folio
belong to the same memcg. It is very rare for contiguous and aligned
entries across a contiguous region of a page table of the same process
or shmem mapping to belong to different memcgs.
This also prepares for recording the memcg info in the cluster's table.
Also make the order check and fallback more compact.
There should be no user-observable behavior change.
Acked-by: Chris Li <chrisl@kernel.org>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/memcontrol.h | 6 +++---
mm/internal.h | 10 +---------
mm/memcontrol.c | 10 ++++------
mm/swap_state.c | 28 +++++++++++++++++++---------
4 files changed, 27 insertions(+), 27 deletions(-)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 7d08128de1fd..a013f37f24aa 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -646,8 +646,8 @@ static inline int mem_cgroup_charge(struct folio *folio, struct mm_struct *mm,
int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp);
-int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
- gfp_t gfp, swp_entry_t entry);
+int mem_cgroup_swapin_charge_folio(struct folio *folio, unsigned short id,
+ struct mm_struct *mm, gfp_t gfp);
void __mem_cgroup_uncharge(struct folio *folio);
@@ -1137,7 +1137,7 @@ static inline int mem_cgroup_charge_hugetlb(struct folio* folio, gfp_t gfp)
}
static inline int mem_cgroup_swapin_charge_folio(struct folio *folio,
- struct mm_struct *mm, gfp_t gfp, swp_entry_t entry)
+ unsigned short id, struct mm_struct *mm, gfp_t gfp)
{
return 0;
}
diff --git a/mm/internal.h b/mm/internal.h
index 5a2ddcf68e0b..9d2fec696bd6 100644
--- a/mm/internal.h
+++ b/mm/internal.h
@@ -451,24 +451,16 @@ static inline int swap_pte_batch(pte_t *start_ptep, int max_nr, pte_t pte)
{
pte_t expected_pte = pte_next_swp_offset(pte);
const pte_t *end_ptep = start_ptep + max_nr;
- const softleaf_t entry = softleaf_from_pte(pte);
pte_t *ptep = start_ptep + 1;
- unsigned short cgroup_id;
VM_WARN_ON(max_nr < 1);
- VM_WARN_ON(!softleaf_is_swap(entry));
+ VM_WARN_ON(!softleaf_is_swap(softleaf_from_pte(pte)));
- cgroup_id = lookup_swap_cgroup_id(entry);
while (ptep < end_ptep) {
- softleaf_t entry;
-
pte = ptep_get(ptep);
if (!pte_same(pte, expected_pte))
break;
- entry = softleaf_from_pte(pte);
- if (lookup_swap_cgroup_id(entry) != cgroup_id)
- break;
expected_pte = pte_next_swp_offset(expected_pte);
ptep++;
}
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index a28a68eed7ba..4f940cf22ffe 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5070,27 +5070,25 @@ int mem_cgroup_charge_hugetlb(struct folio *folio, gfp_t gfp)
/**
* mem_cgroup_swapin_charge_folio - Charge a newly allocated folio for swapin.
- * @folio: folio to charge.
+ * @folio: the folio to charge
+ * @id: memory cgroup id
* @mm: mm context of the victim
* @gfp: reclaim mode
- * @entry: swap entry for which the folio is allocated
*
* This function charges a folio allocated for swapin. Please call this before
* adding the folio to the swapcache.
*
* Returns 0 on success. Otherwise, an error code is returned.
*/
-int mem_cgroup_swapin_charge_folio(struct folio *folio, struct mm_struct *mm,
- gfp_t gfp, swp_entry_t entry)
+int mem_cgroup_swapin_charge_folio(struct folio *folio, unsigned short id,
+ struct mm_struct *mm, gfp_t gfp)
{
struct mem_cgroup *memcg;
- unsigned short id;
int ret;
if (mem_cgroup_disabled())
return 0;
- id = lookup_swap_cgroup_id(entry);
rcu_read_lock();
memcg = mem_cgroup_from_private_id(id);
if (!memcg || !css_tryget_online(&memcg->css))
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 7a80494fa37f..bdd949ae0044 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -142,17 +142,21 @@ void *swap_cache_get_shadow(swp_entry_t entry)
* @ci: The locked swap cluster
* @targ_entry: The target swap entry to check, will be rounded down by @nr
* @nr: Number of slots to check, must be a power of 2
- * @shadowp: Returns the shadow value if one exists in the range.
+ * @shadowp: Returns the shadow value if one exists in the range
+ * @memcg_id: Returns the memory cgroup id, NULL to ignore cgroup check
*
* Check if all slots covered by given range have a swap count >= 1.
- * Retrieves the shadow if there is one.
+ * Retrieves the shadow if there is one. If @memcg_id is not NULL, also
+ * checks if all slots belong to the same cgroup and return the cgroup
+ * private id.
*
* Context: Caller must lock the cluster.
* Return: 0 if success, error code if failed.
*/
static int __swap_cache_add_check(struct swap_cluster_info *ci,
swp_entry_t targ_entry,
- unsigned long nr, void **shadowp)
+ unsigned long nr, void **shadowp,
+ unsigned short *memcg_id)
{
unsigned int ci_off, ci_end;
unsigned long old_tb;
@@ -172,19 +176,24 @@ static int __swap_cache_add_check(struct swap_cluster_info *ci,
return -EEXIST;
if (!__swp_tb_get_count(old_tb))
return -ENOENT;
- if (swp_tb_is_shadow(old_tb) && shadowp)
+ if (shadowp && swp_tb_is_shadow(old_tb))
*shadowp = swp_tb_to_shadow(old_tb);
+ if (memcg_id)
+ *memcg_id = lookup_swap_cgroup_id(targ_entry);
if (nr == 1)
return 0;
+ targ_entry.val = round_down(targ_entry.val, nr);
ci_off = round_down(ci_off, nr);
ci_end = ci_off + nr;
do {
old_tb = __swap_table_get(ci, ci_off);
if (unlikely(swp_tb_is_folio(old_tb) ||
- !__swp_tb_get_count(old_tb)))
+ !__swp_tb_get_count(old_tb) ||
+ (memcg_id && *memcg_id != lookup_swap_cgroup_id(targ_entry))))
return -EBUSY;
+ targ_entry.val++;
} while (++ci_off < ci_end);
return 0;
@@ -400,6 +409,7 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
swp_entry_t entry;
struct folio *folio;
void *shadow = NULL;
+ unsigned short memcg_id;
unsigned long address, nr_pages = 1UL << order;
struct vm_area_struct *vma = vmf ? vmf->vma : NULL;
@@ -408,7 +418,7 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
/* Check if the slot and range are available, skip allocation if not */
spin_lock(&ci->lock);
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL);
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL, NULL);
spin_unlock(&ci->lock);
if (unlikely(err))
return ERR_PTR(err);
@@ -431,7 +441,7 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
/* Double check the range is still not in conflict */
spin_lock(&ci->lock);
- err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow);
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow, &memcg_id);
if (unlikely(err)) {
spin_unlock(&ci->lock);
folio_put(folio);
@@ -443,8 +453,8 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
__swap_cache_do_add_folio(ci, folio, entry);
spin_unlock(&ci->lock);
- if (mem_cgroup_swapin_charge_folio(folio, vmf ? vmf->vma->vm_mm : NULL,
- gfp, entry)) {
+ if (mem_cgroup_swapin_charge_folio(folio, memcg_id,
+ vmf ? vmf->vma->vm_mm : NULL, gfp)) {
spin_lock(&ci->lock);
__swap_cache_do_del_folio(ci, folio, entry, shadow);
spin_unlock(&ci->lock);
--
2.54.0
^ permalink raw reply related
* [PATCH v5 05/12] mm, swap: unify large folio allocation
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
Now that direct large order allocation is supported in the swap cache,
both anon and shmem can use it instead of implementing their own methods.
This unifies the fallback and swap cache check, which also reduces the
TOCTOU race window of swap cache state: previously, high order swapin
required checking swap cache states first, then allocating and falling
back separately. Now all these steps happen in the same compact loop.
Order fallback and statistics are also unified, callers just need to
check and pass the acceptable order bitmask.
There is basically no behavior change. This only makes things more
unified and prepares for later commits. Cgroup and zero map checks can
also be moved into the compact loop, further reducing race windows and
redundancy
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/memory.c | 80 +++++++------------------------
mm/shmem.c | 102 +++++++++++++---------------------------
mm/swap.h | 30 ++----------
mm/swap_state.c | 143 ++++++++++----------------------------------------------
mm/swapfile.c | 3 +-
5 files changed, 79 insertions(+), 279 deletions(-)
diff --git a/mm/memory.c b/mm/memory.c
index 0c9d9c2cbf0e..da891bcce59c 100644
--- a/mm/memory.c
+++ b/mm/memory.c
@@ -4609,26 +4609,6 @@ static vm_fault_t handle_pte_marker(struct vm_fault *vmf)
return VM_FAULT_SIGBUS;
}
-static struct folio *__alloc_swap_folio(struct vm_fault *vmf)
-{
- struct vm_area_struct *vma = vmf->vma;
- struct folio *folio;
- softleaf_t entry;
-
- folio = vma_alloc_folio(GFP_HIGHUSER_MOVABLE, 0, vma, vmf->address);
- if (!folio)
- return NULL;
-
- entry = softleaf_from_pte(vmf->orig_pte);
- if (mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
- GFP_KERNEL, entry)) {
- folio_put(folio);
- return NULL;
- }
-
- return folio;
-}
-
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
/*
* Check if the PTEs within a range are contiguous swap entries
@@ -4658,8 +4638,6 @@ static bool can_swapin_thp(struct vm_fault *vmf, pte_t *ptep, int nr_pages)
*/
if (unlikely(swap_zeromap_batch(entry, nr_pages, NULL) != nr_pages))
return false;
- if (unlikely(non_swapcache_batch(entry, nr_pages) != nr_pages))
- return false;
return true;
}
@@ -4687,16 +4665,14 @@ static inline unsigned long thp_swap_suitable_orders(pgoff_t swp_offset,
return orders;
}
-static struct folio *alloc_swap_folio(struct vm_fault *vmf)
+static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
{
struct vm_area_struct *vma = vmf->vma;
unsigned long orders;
- struct folio *folio;
unsigned long addr;
softleaf_t entry;
spinlock_t *ptl;
pte_t *pte;
- gfp_t gfp;
int order;
/*
@@ -4704,7 +4680,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
* maintain the uffd semantics.
*/
if (unlikely(userfaultfd_armed(vma)))
- goto fallback;
+ return 0;
/*
* A large swapped out folio could be partially or fully in zswap. We
@@ -4712,7 +4688,7 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
* folio.
*/
if (!zswap_never_enabled())
- goto fallback;
+ return 0;
entry = softleaf_from_pte(vmf->orig_pte);
/*
@@ -4726,12 +4702,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
vmf->address, orders);
if (!orders)
- goto fallback;
+ return 0;
pte = pte_offset_map_lock(vmf->vma->vm_mm, vmf->pmd,
vmf->address & PMD_MASK, &ptl);
if (unlikely(!pte))
- goto fallback;
+ return 0;
/*
* For do_swap_page, find the highest order where the aligned range is
@@ -4747,29 +4723,12 @@ static struct folio *alloc_swap_folio(struct vm_fault *vmf)
pte_unmap_unlock(pte, ptl);
- /* Try allocating the highest of the remaining orders. */
- gfp = vma_thp_gfp_mask(vma);
- while (orders) {
- addr = ALIGN_DOWN(vmf->address, PAGE_SIZE << order);
- folio = vma_alloc_folio(gfp, order, vma, addr);
- if (folio) {
- if (!mem_cgroup_swapin_charge_folio(folio, vma->vm_mm,
- gfp, entry))
- return folio;
- count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
- folio_put(folio);
- }
- count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
- order = next_order(&orders, order);
- }
-
-fallback:
- return __alloc_swap_folio(vmf);
+ return orders;
}
#else /* !CONFIG_TRANSPARENT_HUGEPAGE */
-static struct folio *alloc_swap_folio(struct vm_fault *vmf)
+static unsigned long thp_swapin_suitable_orders(struct vm_fault *vmf)
{
- return __alloc_swap_folio(vmf);
+ return 0;
}
#endif /* CONFIG_TRANSPARENT_HUGEPAGE */
@@ -4875,23 +4834,15 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
if (folio)
swap_update_readahead(folio, vma, vmf->address);
if (!folio) {
- if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {
- folio = alloc_swap_folio(vmf);
- if (folio) {
- /*
- * folio is charged, so swapin can only fail due
- * to raced swapin and return NULL.
- */
- swapcache = swapin_folio(entry, folio);
- if (swapcache != folio)
- folio_put(folio);
- folio = swapcache;
- }
- } else {
+ /* Swapin bypasses readahead for SWP_SYNCHRONOUS_IO devices */
+ if (data_race(si->flags & SWP_SYNCHRONOUS_IO))
+ folio = swapin_sync(entry, GFP_HIGHUSER_MOVABLE,
+ thp_swapin_suitable_orders(vmf) | BIT(0),
+ vmf, NULL, 0);
+ else
folio = swapin_readahead(entry, GFP_HIGHUSER_MOVABLE, vmf);
- }
- if (!folio) {
+ if (IS_ERR_OR_NULL(folio)) {
/*
* Back out if somebody else faulted in this pte
* while we released the pte lock.
@@ -4901,6 +4852,7 @@ vm_fault_t do_swap_page(struct vm_fault *vmf)
if (likely(vmf->pte &&
pte_same(ptep_get(vmf->pte), vmf->orig_pte)))
ret = VM_FAULT_OOM;
+ folio = NULL;
goto unlock;
}
diff --git a/mm/shmem.c b/mm/shmem.c
index 6edb23b41bac..77a3e28e5160 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -159,7 +159,7 @@ static unsigned long shmem_default_max_inodes(void)
static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
struct folio **foliop, enum sgp_type sgp, gfp_t gfp,
- struct vm_area_struct *vma, vm_fault_t *fault_type);
+ struct vm_fault *vmf, vm_fault_t *fault_type);
static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb)
{
@@ -2017,68 +2017,32 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf,
}
static struct folio *shmem_swap_alloc_folio(struct inode *inode,
- struct vm_area_struct *vma, pgoff_t index,
+ struct vm_fault *vmf, pgoff_t index,
swp_entry_t entry, int order, gfp_t gfp)
{
+ pgoff_t ilx;
+ struct folio *folio;
+ struct mempolicy *mpol;
struct shmem_inode_info *info = SHMEM_I(inode);
- struct folio *new, *swapcache;
- int nr_pages = 1 << order;
- gfp_t alloc_gfp = gfp;
-
- if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
- if (WARN_ON_ONCE(order))
- return ERR_PTR(-EINVAL);
- } else if (order) {
- /*
- * If uffd is active for the vma, we need per-page fault
- * fidelity to maintain the uffd semantics, then fallback
- * to swapin order-0 folio, as well as for zswap case.
- * Any existing sub folio in the swap cache also blocks
- * mTHP swapin.
- */
- if ((vma && unlikely(userfaultfd_armed(vma))) ||
- !zswap_never_enabled() ||
- non_swapcache_batch(entry, nr_pages) != nr_pages)
- goto fallback;
- alloc_gfp = thp_shmem_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
- }
-retry:
- new = shmem_alloc_folio(alloc_gfp, order, info, index);
- if (!new) {
- new = ERR_PTR(-ENOMEM);
- goto fallback;
- }
+ if ((vmf && unlikely(userfaultfd_armed(vmf->vma))) ||
+ !zswap_never_enabled())
+ order = 0;
- if (mem_cgroup_swapin_charge_folio(new, vma ? vma->vm_mm : NULL,
- alloc_gfp, entry)) {
- folio_put(new);
- new = ERR_PTR(-ENOMEM);
- goto fallback;
- }
+again:
+ mpol = shmem_get_pgoff_policy(info, index, order, &ilx);
+ folio = swapin_sync(entry, gfp, BIT(order), vmf, mpol, ilx);
+ mpol_cond_put(mpol);
- swapcache = swapin_folio(entry, new);
- if (swapcache != new) {
- folio_put(new);
- if (!swapcache) {
- /*
- * The new folio is charged already, swapin can
- * only fail due to another raced swapin.
- */
- new = ERR_PTR(-EEXIST);
- goto fallback;
- }
+ if (!IS_ERR(folio))
+ return folio;
+
+ if (order) {
+ order = 0;
+ goto again;
}
- return swapcache;
-fallback:
- /* Order 0 swapin failed, nothing to fallback to, abort */
- if (!order)
- return new;
- entry.val += index - round_down(index, nr_pages);
- alloc_gfp = gfp;
- nr_pages = 1;
- order = 0;
- goto retry;
+
+ return folio;
}
/*
@@ -2265,11 +2229,12 @@ static int shmem_split_large_entry(struct inode *inode, pgoff_t index,
*/
static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
struct folio **foliop, enum sgp_type sgp,
- gfp_t gfp, struct vm_area_struct *vma,
+ gfp_t gfp, struct vm_fault *vmf,
vm_fault_t *fault_type)
{
struct address_space *mapping = inode->i_mapping;
- struct mm_struct *fault_mm = vma ? vma->vm_mm : NULL;
+ struct vm_area_struct *vma = vmf ? vmf->vma : NULL;
+ struct mm_struct *fault_mm = vmf ? vmf->vma->vm_mm : NULL;
struct shmem_inode_info *info = SHMEM_I(inode);
swp_entry_t swap;
softleaf_t index_entry;
@@ -2310,20 +2275,19 @@ static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
if (!folio) {
if (data_race(si->flags & SWP_SYNCHRONOUS_IO)) {
/* Direct swapin skipping swap cache & readahead */
- folio = shmem_swap_alloc_folio(inode, vma, index,
- index_entry, order, gfp);
- if (IS_ERR(folio)) {
- error = PTR_ERR(folio);
- folio = NULL;
- goto failed;
- }
+ folio = shmem_swap_alloc_folio(inode, vmf, index,
+ swap, order, gfp);
} else {
/* Cached swapin only supports order 0 folio */
folio = shmem_swapin_cluster(swap, gfp, info, index);
- if (!folio) {
+ }
+ if (IS_ERR_OR_NULL(folio)) {
+ if (IS_ERR(folio))
+ error = PTR_ERR(folio);
+ else
error = -ENOMEM;
- goto failed;
- }
+ folio = NULL;
+ goto failed;
}
if (fault_type) {
*fault_type |= VM_FAULT_MAJOR;
@@ -2471,7 +2435,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
if (xa_is_value(folio)) {
error = shmem_swapin_folio(inode, index, &folio,
- sgp, gfp, vma, fault_type);
+ sgp, gfp, vmf, fault_type);
if (error == -EEXIST)
goto repeat;
diff --git a/mm/swap.h b/mm/swap.h
index 6774af10a943..8e57e9431624 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -300,7 +300,8 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t flag,
struct mempolicy *mpol, pgoff_t ilx);
struct folio *swapin_readahead(swp_entry_t entry, gfp_t flag,
struct vm_fault *vmf);
-struct folio *swapin_folio(swp_entry_t entry, struct folio *folio);
+struct folio *swapin_sync(swp_entry_t entry, gfp_t flag, unsigned long orders,
+ struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx);
void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,
unsigned long addr);
@@ -334,24 +335,6 @@ static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr,
return find_next_bit(sis->zeromap, end, start) - start;
}
-static inline int non_swapcache_batch(swp_entry_t entry, int max_nr)
-{
- int i;
-
- /*
- * While allocating a large folio and doing mTHP swapin, we need to
- * ensure all entries are not cached, otherwise, the mTHP folio will
- * be in conflict with the folio in swap cache.
- */
- for (i = 0; i < max_nr; i++) {
- if (swap_cache_has_folio(entry))
- return i;
- entry.val++;
- }
-
- return i;
-}
-
#else /* CONFIG_SWAP */
struct swap_iocb;
static inline struct swap_cluster_info *swap_cluster_lock(
@@ -433,7 +416,9 @@ static inline struct folio *swapin_readahead(swp_entry_t swp, gfp_t gfp_mask,
return NULL;
}
-static inline struct folio *swapin_folio(swp_entry_t entry, struct folio *folio)
+static inline struct folio *swapin_sync(
+ swp_entry_t entry, gfp_t flag, unsigned long orders,
+ struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx)
{
return NULL;
}
@@ -493,10 +478,5 @@ static inline int swap_zeromap_batch(swp_entry_t entry, int max_nr,
{
return 0;
}
-
-static inline int non_swapcache_batch(swp_entry_t entry, int max_nr)
-{
- return 0;
-}
#endif /* CONFIG_SWAP */
#endif /* _MM_SWAP_H */
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 0adb0565bbb1..98c8691826fb 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -238,43 +238,6 @@ void __swap_cache_add_folio(struct swap_cluster_info *ci,
lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
}
-/**
- * swap_cache_add_folio - Add a folio into the swap cache.
- * @folio: The folio to be added.
- * @entry: The swap entry corresponding to the folio.
- * @shadowp: If a shadow is found, return the shadow.
- *
- * Add a folio into the swap cache. Will return error if any slot is no
- * longer a valid swapped out slot or already occupied by another folio.
- *
- * Context: Caller must ensure @entry is valid and protect the swap device
- * with reference count or locks.
- */
-static int swap_cache_add_folio(struct folio *folio, swp_entry_t entry,
- void **shadowp)
-{
- int err;
- void *shadow = NULL;
- struct swap_info_struct *si;
- struct swap_cluster_info *ci;
- unsigned long nr_pages = folio_nr_pages(folio);
-
- si = __swap_entry_to_info(entry);
- ci = swap_cluster_lock(si, swp_offset(entry));
- err = __swap_cache_add_check(ci, entry, nr_pages, &shadow);
- if (err) {
- swap_cluster_unlock(ci);
- return err;
- }
-
- __swap_cache_add_folio(ci, folio, entry);
- swap_cluster_unlock(ci);
- if (shadowp)
- *shadowp = shadow;
-
- return 0;
-}
-
static void __swap_cache_do_del_folio(struct swap_cluster_info *ci,
struct folio *folio,
swp_entry_t entry, void *shadow)
@@ -650,51 +613,6 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,
}
}
-/**
- * __swap_cache_prepare_and_add - Prepare the folio and add it to swap cache.
- * @entry: swap entry to be bound to the folio.
- * @folio: folio to be added.
- * @gfp: memory allocation flags for charge, can be 0 if @charged if true.
- * @charged: if the folio is already charged.
- *
- * Update the swap_map and add folio as swap cache, typically before swapin.
- * All swap slots covered by the folio must have a non-zero swap count.
- *
- * Context: Caller must protect the swap device with reference count or locks.
- * Return: 0 if success, error code if failed.
- */
-static int __swap_cache_prepare_and_add(swp_entry_t entry,
- struct folio *folio,
- gfp_t gfp, bool charged)
-{
- void *shadow;
- int ret;
-
- __folio_set_locked(folio);
- __folio_set_swapbacked(folio);
-
- if (!charged && mem_cgroup_swapin_charge_folio(folio, NULL, gfp, entry)) {
- ret = -ENOMEM;
- goto failed;
- }
-
- ret = swap_cache_add_folio(folio, entry, &shadow);
- if (ret)
- goto failed;
-
- memcg1_swapin(entry, folio_nr_pages(folio));
- if (shadow)
- workingset_refault(folio, shadow);
-
- /* Caller will initiate read into locked folio */
- folio_add_lru(folio);
- return 0;
-
-failed:
- folio_unlock(folio);
- return ret;
-}
-
static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
struct mempolicy *mpol, pgoff_t ilx,
struct swap_iocb **plug, bool readahead)
@@ -705,7 +623,6 @@ static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
folio = swap_cache_get_folio(entry);
if (folio)
return folio;
-
folio = swap_cache_alloc_folio(entry, gfp, BIT(0), NULL, mpol, ilx);
} while (PTR_ERR(folio) == -EEXIST);
@@ -722,49 +639,37 @@ static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
}
/**
- * swapin_folio - swap-in one or multiple entries skipping readahead.
- * @entry: starting swap entry to swap in
- * @folio: a new allocated and charged folio
+ * swapin_sync - swap-in one or multiple entries skipping readahead.
+ * @entry: swap entry indicating the target slot
+ * @gfp: memory allocation flags
+ * @orders: allocation orders
+ * @vmf: fault information
+ * @mpol: NUMA memory allocation policy to be applied
+ * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
*
- * Reads @entry into @folio, @folio will be added to the swap cache.
- * If @folio is a large folio, the @entry will be rounded down to align
- * with the folio size.
+ * This allocates a folio suitable for given @orders, or returns the
+ * existing folio in the swap cache for @entry. This initiates the IO, too,
+ * if needed. @entry is rounded down if @orders allow large allocation.
*
- * Return: returns pointer to @folio on success. If folio is a large folio
- * and this raced with another swapin, NULL will be returned to allow fallback
- * to order 0. Else, if another folio was already added to the swap cache,
- * return that swap cache folio instead.
+ * Context: Caller must ensure @entry is valid and pin the swap device with refcount.
+ * Return: Returns the folio on success, error code if failed.
*/
-struct folio *swapin_folio(swp_entry_t entry, struct folio *folio)
+struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
+ struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx)
{
- int ret;
- struct folio *swapcache;
- pgoff_t offset = swp_offset(entry);
- unsigned long nr_pages = folio_nr_pages(folio);
-
- entry = swp_entry(swp_type(entry), round_down(offset, nr_pages));
- for (;;) {
- ret = __swap_cache_prepare_and_add(entry, folio, 0, true);
- if (!ret) {
- swap_read_folio(folio, NULL);
- break;
- }
+ struct folio *folio;
- /*
- * Large order allocation needs special handling on
- * race: if a smaller folio exists in cache, swapin needs
- * to fall back to order 0, and doing a swap cache lookup
- * might return a folio that is irrelevant to the faulting
- * entry because @entry is aligned down. Just return NULL.
- */
- if (ret != -EEXIST || nr_pages > 1)
- return NULL;
+ do {
+ folio = swap_cache_get_folio(entry);
+ if (folio)
+ return folio;
+ folio = swap_cache_alloc_folio(entry, gfp, orders, vmf, mpol, ilx);
+ } while (PTR_ERR(folio) == -EEXIST);
- swapcache = swap_cache_get_folio(entry);
- if (swapcache)
- return swapcache;
- }
+ if (IS_ERR(folio))
+ return folio;
+ swap_read_folio(folio, NULL);
return folio;
}
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 08309c1dafa3..4e5a54769e81 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1827,8 +1827,7 @@ void folio_put_swap(struct folio *folio, struct page *subpage)
* do_swap_page()
* ... swapoff+swapon
* swap_cache_alloc_folio()
- * swap_cache_add_folio()
- * // check swap_map
+ * // check swap_map
* // verify PTE not changed
*
* In __swap_duplicate(), the swap_map need to be checked before
--
2.54.0
^ permalink raw reply related
* [PATCH v5 06/12] mm/memcg, swap: tidy up cgroup v1 memsw swap helpers
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
The cgroup v1 swap helpers always operate on swap cache folios whose
swap entry is stable: the folio is locked and in the swap cache. There
is no need to pass the swap entry or page count as separate parameters
when they can be derived from the folio itself.
Simplify the redundant parameters and add sanity checks to document
the required preconditions.
Also rename memcg1_swapout to __memcg1_swapout to indicate it requires
special calling context: the folio must be isolated and dying, and the
call must be made with interrupts disabled.
No functional change.
Acked-by: Chris Li <chrisl@kernel.org>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/memcontrol.h | 8 ++++----
include/linux/swap.h | 10 ++++------
mm/huge_memory.c | 2 +-
mm/memcontrol-v1.c | 33 ++++++++++++++++++++-------------
mm/memcontrol.c | 9 ++++-----
mm/swap_state.c | 4 ++--
mm/swapfile.c | 2 +-
mm/vmscan.c | 2 +-
8 files changed, 37 insertions(+), 33 deletions(-)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index dc3fa687759b..7d08128de1fd 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1899,8 +1899,8 @@ static inline void mem_cgroup_exit_user_fault(void)
current->in_user_fault = 0;
}
-void memcg1_swapout(struct folio *folio, swp_entry_t entry);
-void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages);
+void __memcg1_swapout(struct folio *folio);
+void memcg1_swapin(struct folio *folio);
#else /* CONFIG_MEMCG_V1 */
static inline
@@ -1929,11 +1929,11 @@ static inline void mem_cgroup_exit_user_fault(void)
{
}
-static inline void memcg1_swapout(struct folio *folio, swp_entry_t entry)
+static inline void __memcg1_swapout(struct folio *folio)
{
}
-static inline void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages)
+static inline void memcg1_swapin(struct folio *folio)
{
}
diff --git a/include/linux/swap.h b/include/linux/swap.h
index aa89e1d30a77..6b3acdf9bdd4 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -576,13 +576,12 @@ static inline void folio_throttle_swaprate(struct folio *folio, gfp_t gfp)
#endif
#if defined(CONFIG_MEMCG) && defined(CONFIG_SWAP)
-int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry);
-static inline int mem_cgroup_try_charge_swap(struct folio *folio,
- swp_entry_t entry)
+int __mem_cgroup_try_charge_swap(struct folio *folio);
+static inline int mem_cgroup_try_charge_swap(struct folio *folio)
{
if (mem_cgroup_disabled())
return 0;
- return __mem_cgroup_try_charge_swap(folio, entry);
+ return __mem_cgroup_try_charge_swap(folio);
}
extern void __mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_pages);
@@ -596,8 +595,7 @@ static inline void mem_cgroup_uncharge_swap(swp_entry_t entry, unsigned int nr_p
extern long mem_cgroup_get_nr_swap_pages(struct mem_cgroup *memcg);
extern bool mem_cgroup_swap_full(struct folio *folio);
#else
-static inline int mem_cgroup_try_charge_swap(struct folio *folio,
- swp_entry_t entry)
+static inline int mem_cgroup_try_charge_swap(struct folio *folio)
{
return 0;
}
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index c565b2a651e0..42b86e8ab7c0 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -4430,7 +4430,7 @@ void deferred_split_folio(struct folio *folio, bool partially_mapped)
/*
* Exclude swapcache: originally to avoid a corrupt deferred split
- * queue. Nowadays that is fully prevented by memcg1_swapout();
+ * queue. Nowadays that is fully prevented by __memcg1_swapout();
* but if page reclaim is already handling the same folio, it is
* unnecessary to handle it again in the shrinker, so excluding
* swapcache here may still be a useful optimization.
diff --git a/mm/memcontrol-v1.c b/mm/memcontrol-v1.c
index 433bba9dfe71..36c507d81dc5 100644
--- a/mm/memcontrol-v1.c
+++ b/mm/memcontrol-v1.c
@@ -604,18 +604,23 @@ void memcg1_commit_charge(struct folio *folio, struct mem_cgroup *memcg)
}
/**
- * memcg1_swapout - transfer a memsw charge to swap
+ * __memcg1_swapout - transfer a memsw charge to swap
* @folio: folio whose memsw charge to transfer
- * @entry: swap entry to move the charge to
*
- * Transfer the memsw charge of @folio to @entry.
+ * Transfer the memsw charge of @folio to the swap entry stored in
+ * folio->swap.
+ *
+ * Context: folio must be isolated, unmapped, locked and is just about
+ * to be freed, and caller must disable IRQs.
*/
-void memcg1_swapout(struct folio *folio, swp_entry_t entry)
+void __memcg1_swapout(struct folio *folio)
{
struct mem_cgroup *memcg, *swap_memcg;
struct obj_cgroup *objcg;
unsigned int nr_entries;
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
VM_BUG_ON_FOLIO(folio_ref_count(folio), folio);
@@ -641,7 +646,7 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry)
swap_memcg = mem_cgroup_private_id_get_online(memcg, nr_entries);
mod_memcg_state(swap_memcg, MEMCG_SWAP, nr_entries);
- swap_cgroup_record(folio, mem_cgroup_private_id(swap_memcg), entry);
+ swap_cgroup_record(folio, mem_cgroup_private_id(swap_memcg), folio->swap);
folio_unqueue_deferred_split(folio);
folio->memcg_data = 0;
@@ -671,18 +676,20 @@ void memcg1_swapout(struct folio *folio, swp_entry_t entry)
obj_cgroup_put(objcg);
}
-/*
+/**
* memcg1_swapin - uncharge swap slot
- * @entry: the first swap entry for which the pages are charged
- * @nr_pages: number of pages which will be uncharged
+ * @folio: folio being swapped in
*
- * Call this function after successfully adding the charged page to swapcache.
+ * Call this function after successfully adding the charged
+ * folio to swapcache.
*
- * Note: This function assumes the page for which swap slot is being uncharged
- * is order 0 page.
+ * Context: The folio has to be in swap cache and locked.
*/
-void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages)
+void memcg1_swapin(struct folio *folio)
{
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_swapcache(folio), folio);
+ VM_WARN_ON_ONCE_FOLIO(!folio_test_locked(folio), folio);
+
/*
* Cgroup1's unified memory+swap counter has been charged with the
* new swapcache page, finish the transfer by uncharging the swap
@@ -701,7 +708,7 @@ void memcg1_swapin(swp_entry_t entry, unsigned int nr_pages)
* let's not wait for it. The page already received a
* memory+swap charge, drop the swap entry duplicate.
*/
- mem_cgroup_uncharge_swap(entry, nr_pages);
+ mem_cgroup_uncharge_swap(folio->swap, folio_nr_pages(folio));
}
}
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index d978e18b9b2d..a28a68eed7ba 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -5464,13 +5464,12 @@ int __init mem_cgroup_init(void)
/**
* __mem_cgroup_try_charge_swap - try charging swap space for a folio
* @folio: folio being added to swap
- * @entry: swap entry to charge
*
- * Try to charge @folio's memcg for the swap space at @entry.
+ * Try to charge @folio's memcg for the swap space at folio->swap.
*
* Returns 0 on success, -ENOMEM on failure.
*/
-int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry)
+int __mem_cgroup_try_charge_swap(struct folio *folio)
{
unsigned int nr_pages = folio_nr_pages(folio);
struct page_counter *counter;
@@ -5487,7 +5486,7 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry)
rcu_read_lock();
memcg = obj_cgroup_memcg(objcg);
- if (!entry.val) {
+ if (!folio_test_swapcache(folio)) {
memcg_memory_event(memcg, MEMCG_SWAP_FAIL);
rcu_read_unlock();
return 0;
@@ -5506,7 +5505,7 @@ int __mem_cgroup_try_charge_swap(struct folio *folio, swp_entry_t entry)
}
mod_memcg_state(memcg, MEMCG_SWAP, nr_pages);
- swap_cgroup_record(folio, mem_cgroup_private_id(memcg), entry);
+ swap_cgroup_record(folio, mem_cgroup_private_id(memcg), folio->swap);
return 0;
}
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 98c8691826fb..7a80494fa37f 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -455,8 +455,8 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
return ERR_PTR(-ENOMEM);
}
- /* For memsw accounting, swap is uncharged when folio is added to swap cache */
- memcg1_swapin(entry, 1 << order);
+ /* memsw uncharges swap when folio is added to swap cache */
+ memcg1_swapin(folio);
if (shadow)
workingset_refault(folio, shadow);
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 4e5a54769e81..5c8bb15719bf 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1731,7 +1731,7 @@ int folio_alloc_swap(struct folio *folio)
}
/* Need to call this even if allocation failed, for MEMCG_SWAP_FAIL. */
- if (unlikely(mem_cgroup_try_charge_swap(folio, folio->swap)))
+ if (unlikely(mem_cgroup_try_charge_swap(folio)))
swap_cache_del_folio(folio);
if (unlikely(!folio_test_swapcache(folio)))
diff --git a/mm/vmscan.c b/mm/vmscan.c
index b3e555561417..924c84326551 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -737,7 +737,7 @@ static int __remove_mapping(struct address_space *mapping, struct folio *folio,
if (reclaimed && !mapping_exiting(mapping))
shadow = workingset_eviction(folio, target_memcg);
- memcg1_swapout(folio, swap);
+ __memcg1_swapout(folio);
__swap_cache_del_folio(ci, folio, swap, shadow);
swap_cluster_unlock_irq(ci);
} else {
--
2.54.0
^ permalink raw reply related
* [PATCH v5 04/12] mm, swap: add support for stable large allocation in swap cache directly
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
To make it possible to allocate large folios directly in swap cache,
provide a new infrastructure helper to handle the swap cache status
check, allocation, and order fallback in the swap cache layer
The new helper replaces the existing swap_cache_alloc_folio. Based on
this, all the separate swap folio allocation that is being done by anon
/ shmem before is converted to use this helper directly, unifying folio
allocation for anon, shmem, and readahead.
This slightly consolidates how allocation is synchronized, making it
more stable and less prone to errors. The slot-count and cache-conflict
check is now always performed with the cluster lock held before
allocation, and repeated under the same lock right before cache
insertion. This double check produces a stable result compared to the
previous anon and shmem mTHP allocation implementation, avoids the
false-negative conflict checks that the lockless path can return — large
allocations no longer have to be unwound because the range turned out to
be occupied — and aborts early for already-freed slots, which helps
ordinary swapin and especially readahead, with only a marginal increase
in cluster-lock contention (the lock is very lightly contended and stays
local in the first place). Hence, callers of swap_cache_alloc_folio() no
longer need to check the swap slot count or swap cache status
themselves.
And now whoever first successfully allocates a folio in the swap cache
will be the one who charges it and performs the swap-in. The race window
of swapping is also reduced since the loop is much more compact.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/swap.h | 3 +-
mm/swap_state.c | 236 +++++++++++++++++++++++++++++++++++++++-----------------
mm/zswap.c | 2 +-
3 files changed, 170 insertions(+), 71 deletions(-)
diff --git a/mm/swap.h b/mm/swap.h
index ad8b17a93758..6774af10a943 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -280,7 +280,8 @@ bool swap_cache_has_folio(swp_entry_t entry);
struct folio *swap_cache_get_folio(swp_entry_t entry);
void *swap_cache_get_shadow(swp_entry_t entry);
void swap_cache_del_folio(struct folio *folio);
-struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_flags,
+struct folio *swap_cache_alloc_folio(swp_entry_t target_entry, gfp_t gfp_mask,
+ unsigned long orders, struct vm_fault *vmf,
struct mempolicy *mpol, pgoff_t ilx);
/* Below helpers require the caller to lock and pass in the swap cluster. */
void __swap_cache_add_folio(struct swap_cluster_info *ci,
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 89fa19ec13f6..0adb0565bbb1 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -139,10 +139,10 @@ void *swap_cache_get_shadow(swp_entry_t entry)
/**
* __swap_cache_add_check - Check if a range is suitable for adding a folio.
- * @ci: The locked swap cluster.
- * @ci_off: Range start offset.
- * @nr: Number of slots to check.
- * @shadow: Returns the shadow value if one exists in the range.
+ * @ci: The locked swap cluster
+ * @targ_entry: The target swap entry to check, will be rounded down by @nr
+ * @nr: Number of slots to check, must be a power of 2
+ * @shadowp: Returns the shadow value if one exists in the range.
*
* Check if all slots covered by given range have a swap count >= 1.
* Retrieves the shadow if there is one.
@@ -151,26 +151,40 @@ void *swap_cache_get_shadow(swp_entry_t entry)
* Return: 0 if success, error code if failed.
*/
static int __swap_cache_add_check(struct swap_cluster_info *ci,
- unsigned int ci_off, unsigned int nr,
- void **shadow)
+ swp_entry_t targ_entry,
+ unsigned long nr, void **shadowp)
{
- unsigned int ci_end = ci_off + nr;
+ unsigned int ci_off, ci_end;
unsigned long old_tb;
lockdep_assert_held(&ci->lock);
- if (WARN_ON_ONCE(ci_off >= SWAPFILE_CLUSTER))
- return -EINVAL;
+ /*
+ * If the target slot is not swapped out or already cached, return
+ * -ENOENT or -EEXIST. If the batch is not suitable, could be a
+ * race with concurrent free or cache add, return -EBUSY.
+ */
if (unlikely(!ci->table))
return -ENOENT;
+ ci_off = swp_cluster_offset(targ_entry);
+ old_tb = __swap_table_get(ci, ci_off);
+ if (swp_tb_is_folio(old_tb))
+ return -EEXIST;
+ if (!__swp_tb_get_count(old_tb))
+ return -ENOENT;
+ if (swp_tb_is_shadow(old_tb) && shadowp)
+ *shadowp = swp_tb_to_shadow(old_tb);
+
+ if (nr == 1)
+ return 0;
+
+ ci_off = round_down(ci_off, nr);
+ ci_end = ci_off + nr;
do {
old_tb = __swap_table_get(ci, ci_off);
- if (unlikely(swp_tb_is_folio(old_tb)))
- return -EEXIST;
- if (unlikely(!__swp_tb_get_count(old_tb)))
- return -ENOENT;
- if (swp_tb_is_shadow(old_tb))
- *shadow = swp_tb_to_shadow(old_tb);
+ if (unlikely(swp_tb_is_folio(old_tb) ||
+ !__swp_tb_get_count(old_tb)))
+ return -EBUSY;
} while (++ci_off < ci_end);
return 0;
@@ -241,15 +255,13 @@ static int swap_cache_add_folio(struct folio *folio, swp_entry_t entry,
{
int err;
void *shadow = NULL;
- unsigned int ci_off;
struct swap_info_struct *si;
struct swap_cluster_info *ci;
unsigned long nr_pages = folio_nr_pages(folio);
si = __swap_entry_to_info(entry);
ci = swap_cluster_lock(si, swp_offset(entry));
- ci_off = swp_cluster_offset(entry);
- err = __swap_cache_add_check(ci, ci_off, nr_pages, &shadow);
+ err = __swap_cache_add_check(ci, entry, nr_pages, &shadow);
if (err) {
swap_cluster_unlock(ci);
return err;
@@ -404,6 +416,142 @@ void __swap_cache_replace_folio(struct swap_cluster_info *ci,
}
}
+/*
+ * Try to allocate a folio of given order in the swap cache.
+ *
+ * This helper resolves the potential races of swap allocation
+ * and prepares a folio to be used for swap IO. May return following
+ * value:
+ *
+ * -ENOMEM / -EBUSY: Order is too large or in conflict with sub slot,
+ * caller should shrink the order and retry
+ * -ENOENT / -EEXIST: Target swap entry is unavailable or cached, the caller
+ * should abort or try to use the cached folio instead
+ */
+static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
+ swp_entry_t targ_entry, gfp_t gfp,
+ unsigned int order, struct vm_fault *vmf,
+ struct mempolicy *mpol, pgoff_t ilx)
+{
+ int err;
+ swp_entry_t entry;
+ struct folio *folio;
+ void *shadow = NULL;
+ unsigned long address, nr_pages = 1UL << order;
+ struct vm_area_struct *vma = vmf ? vmf->vma : NULL;
+
+ VM_WARN_ON_ONCE(nr_pages > SWAPFILE_CLUSTER);
+ entry.val = round_down(targ_entry.val, nr_pages);
+
+ /* Check if the slot and range are available, skip allocation if not */
+ spin_lock(&ci->lock);
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, NULL);
+ spin_unlock(&ci->lock);
+ if (unlikely(err))
+ return ERR_PTR(err);
+
+ /*
+ * Limit THP gfp. The limitation is a no-op for typical
+ * GFP_HIGHUSER_MOVABLE but matters for shmem.
+ */
+ if (order)
+ gfp = thp_shmem_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
+
+ if (mpol || !vmf) {
+ folio = folio_alloc_mpol(gfp, order, mpol, ilx, numa_node_id());
+ } else {
+ address = round_down(vmf->address, PAGE_SIZE << order);
+ folio = vma_alloc_folio(gfp, order, vmf->vma, address);
+ }
+ if (unlikely(!folio))
+ return ERR_PTR(-ENOMEM);
+
+ /* Double check the range is still not in conflict */
+ spin_lock(&ci->lock);
+ err = __swap_cache_add_check(ci, targ_entry, nr_pages, &shadow);
+ if (unlikely(err)) {
+ spin_unlock(&ci->lock);
+ folio_put(folio);
+ return ERR_PTR(err);
+ }
+
+ __folio_set_locked(folio);
+ __folio_set_swapbacked(folio);
+ __swap_cache_do_add_folio(ci, folio, entry);
+ spin_unlock(&ci->lock);
+
+ if (mem_cgroup_swapin_charge_folio(folio, vmf ? vmf->vma->vm_mm : NULL,
+ gfp, entry)) {
+ spin_lock(&ci->lock);
+ __swap_cache_do_del_folio(ci, folio, entry, shadow);
+ spin_unlock(&ci->lock);
+ folio_unlock(folio);
+ /* nr_pages refs from swap cache, 1 from allocation */
+ folio_put_refs(folio, nr_pages + 1);
+ count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK_CHARGE);
+ return ERR_PTR(-ENOMEM);
+ }
+
+ /* For memsw accounting, swap is uncharged when folio is added to swap cache */
+ memcg1_swapin(entry, 1 << order);
+ if (shadow)
+ workingset_refault(folio, shadow);
+
+ node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
+ lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
+
+ /* Caller will initiate read into locked new_folio */
+ folio_add_lru(folio);
+ return folio;
+}
+
+/**
+ * swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.
+ * @targ_entry: swap entry indicating the target slot
+ * @gfp: memory allocation flags
+ * @orders: allocation orders, must be non zero
+ * @vmf: fault information
+ * @mpol: NUMA memory allocation policy to be applied
+ * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
+ *
+ * Allocate a folio in the swap cache for one swap slot, typically before
+ * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
+ * @targ_entry must have a non-zero swap count (swapped out).
+ *
+ * Context: Caller must protect the swap device with reference count or locks.
+ * Return: Returns the folio if allocation succeeded and folio is in the swap
+ * cache. Returns error code if failed due to race, OOM or invalid arguments.
+ */
+struct folio *swap_cache_alloc_folio(swp_entry_t targ_entry, gfp_t gfp,
+ unsigned long orders, struct vm_fault *vmf,
+ struct mempolicy *mpol, pgoff_t ilx)
+{
+ int order, err;
+ struct folio *ret;
+ struct swap_cluster_info *ci;
+
+ ci = __swap_entry_to_cluster(targ_entry);
+ order = highest_order(orders);
+
+ /* orders must be non-zero, and must not exceed cluster size. */
+ if (WARN_ON_ONCE(!orders || (1UL << order) > SWAPFILE_CLUSTER))
+ return ERR_PTR(-EINVAL);
+
+ do {
+ ret = __swap_cache_alloc(ci, targ_entry, gfp, order,
+ vmf, mpol, ilx);
+ if (!IS_ERR(ret))
+ break;
+ err = PTR_ERR(ret);
+ if (!order || (err && err != -EBUSY && err != -ENOMEM))
+ break;
+ count_mthp_stat(order, MTHP_STAT_SWPIN_FALLBACK);
+ order = next_order(&orders, order);
+ } while (orders);
+
+ return ret;
+}
+
/*
* If we are the only user, then try to free up the swap cache.
*
@@ -547,68 +695,18 @@ static int __swap_cache_prepare_and_add(swp_entry_t entry,
return ret;
}
-/**
- * swap_cache_alloc_folio - Allocate folio for swapped out slot in swap cache.
- * @entry: the swapped out swap entry to be binded to the folio.
- * @gfp_mask: memory allocation flags
- * @mpol: NUMA memory allocation policy to be applied
- * @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
- *
- * Allocate a folio in the swap cache for one swap slot, typically before
- * doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
- * @entry must have a non-zero swap count (swapped out).
- * Currently only supports order 0.
- *
- * Context: Caller must protect the swap device with reference count or locks.
- * Return: Returns the folio if allocation succeeded and folio is added to
- * swap cache. Returns error code if allocation failed due to race or OOM.
- */
-struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_mask,
- struct mempolicy *mpol, pgoff_t ilx)
-{
- int err;
- struct folio *folio;
-
- /* Allocate a new folio to be added into the swap cache. */
- folio = folio_alloc_mpol(gfp_mask, 0, mpol, ilx, numa_node_id());
- if (!folio)
- return ERR_PTR(-ENOMEM);
-
- /*
- * Try to add the new folio to the swap cache. It returns
- * -EEXIST if the entry is already cached.
- */
- err = __swap_cache_prepare_and_add(entry, folio, gfp_mask, false);
- if (err) {
- folio_put(folio);
- return ERR_PTR(err);
- }
-
- return folio;
-}
-
static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
struct mempolicy *mpol, pgoff_t ilx,
struct swap_iocb **plug, bool readahead)
{
- struct swap_info_struct *si = __swap_entry_to_info(entry);
struct folio *folio;
- /* Check the swap cache again for readahead path. */
- folio = swap_cache_get_folio(entry);
- if (folio)
- return folio;
-
- /* Skip allocation for unused and bad swap slot for readahead. */
- if (!swap_entry_swapped(si, entry))
- return NULL;
-
do {
folio = swap_cache_get_folio(entry);
if (folio)
return folio;
- folio = swap_cache_alloc_folio(entry, gfp, mpol, ilx);
+ folio = swap_cache_alloc_folio(entry, gfp, BIT(0), NULL, mpol, ilx);
} while (PTR_ERR(folio) == -EEXIST);
if (IS_ERR_OR_NULL(folio))
diff --git a/mm/zswap.c b/mm/zswap.c
index e27f6e96f003..761cd699e0a3 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1000,7 +1000,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
return -EEXIST;
mpol = get_task_policy(current);
- folio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, mpol,
+ folio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, BIT(0), NULL, mpol,
NO_INTERLEAVE_INDEX);
put_swap_device(si);
--
2.54.0
^ permalink raw reply related
* [PATCH v5 01/12] mm, swap: simplify swap cache allocation helper
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
Instead of trying to return the existing folio if the entry is already
cached in swap_cache_alloc_folio, simply return an error pointer if the
allocation failed, and drop the output argument that indicates what kind
of folio is actually returned.
And a proper wrapper swap_cache_read_folio that decouples and handles
the actual requirement - read in the folio, or return the already read
folio in cache. This is what async swapin and readahead actually
required.
As for zswap swap out, the caller just needs to abort if the allocation
fails because the entry is gone or already cached, so removing
simplifies the return argument, making it cleaner.
No feature change.
Acked-by: Chris Li <chrisl@kernel.org>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/swap.h | 3 +-
mm/swap_state.c | 180 +++++++++++++++++++++++++++++---------------------------
mm/zswap.c | 23 +++-----
3 files changed, 103 insertions(+), 103 deletions(-)
diff --git a/mm/swap.h b/mm/swap.h
index a77016f2423b..ad8b17a93758 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -281,8 +281,7 @@ struct folio *swap_cache_get_folio(swp_entry_t entry);
void *swap_cache_get_shadow(swp_entry_t entry);
void swap_cache_del_folio(struct folio *folio);
struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_flags,
- struct mempolicy *mpol, pgoff_t ilx,
- bool *alloced);
+ struct mempolicy *mpol, pgoff_t ilx);
/* Below helpers require the caller to lock and pass in the swap cluster. */
void __swap_cache_add_folio(struct swap_cluster_info *ci,
struct folio *folio, swp_entry_t entry);
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 1415a5c54a43..3bba82f6dc79 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -459,54 +459,38 @@ void swap_update_readahead(struct folio *folio, struct vm_area_struct *vma,
* All swap slots covered by the folio must have a non-zero swap count.
*
* Context: Caller must protect the swap device with reference count or locks.
- * Return: Returns the folio being added on success. Returns the existing folio
- * if @entry is already cached. Returns NULL if raced with swapin or swapoff.
+ * Return: 0 if success, error code if failed.
*/
-static struct folio *__swap_cache_prepare_and_add(swp_entry_t entry,
- struct folio *folio,
- gfp_t gfp, bool charged)
+static int __swap_cache_prepare_and_add(swp_entry_t entry,
+ struct folio *folio,
+ gfp_t gfp, bool charged)
{
- struct folio *swapcache = NULL;
void *shadow;
int ret;
__folio_set_locked(folio);
__folio_set_swapbacked(folio);
- if (!charged && mem_cgroup_swapin_charge_folio(folio, NULL, gfp, entry))
+ if (!charged && mem_cgroup_swapin_charge_folio(folio, NULL, gfp, entry)) {
+ ret = -ENOMEM;
goto failed;
-
- for (;;) {
- ret = swap_cache_add_folio(folio, entry, &shadow);
- if (!ret)
- break;
-
- /*
- * Large order allocation needs special handling on
- * race: if a smaller folio exists in cache, swapin needs
- * to fallback to order 0, and doing a swap cache lookup
- * might return a folio that is irrelevant to the faulting
- * entry because @entry is aligned down. Just return NULL.
- */
- if (ret != -EEXIST || folio_test_large(folio))
- goto failed;
-
- swapcache = swap_cache_get_folio(entry);
- if (swapcache)
- goto failed;
}
+ ret = swap_cache_add_folio(folio, entry, &shadow);
+ if (ret)
+ goto failed;
+
memcg1_swapin(entry, folio_nr_pages(folio));
if (shadow)
workingset_refault(folio, shadow);
/* Caller will initiate read into locked folio */
folio_add_lru(folio);
- return folio;
+ return 0;
failed:
folio_unlock(folio);
- return swapcache;
+ return ret;
}
/**
@@ -515,7 +499,6 @@ static struct folio *__swap_cache_prepare_and_add(swp_entry_t entry,
* @gfp_mask: memory allocation flags
* @mpol: NUMA memory allocation policy to be applied
* @ilx: NUMA interleave index, for use only when MPOL_INTERLEAVE
- * @new_page_allocated: sets true if allocation happened, false otherwise
*
* Allocate a folio in the swap cache for one swap slot, typically before
* doing IO (e.g. swap in or zswap writeback). The swap slot indicated by
@@ -523,18 +506,40 @@ static struct folio *__swap_cache_prepare_and_add(swp_entry_t entry,
* Currently only supports order 0.
*
* Context: Caller must protect the swap device with reference count or locks.
- * Return: Returns the existing folio if @entry is cached already. Returns
- * NULL if failed due to -ENOMEM or @entry have a swap count < 1.
+ * Return: Returns the folio if allocation succeeded and folio is added to
+ * swap cache. Returns error code if allocation failed due to race or OOM.
*/
struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_mask,
- struct mempolicy *mpol, pgoff_t ilx,
- bool *new_page_allocated)
+ struct mempolicy *mpol, pgoff_t ilx)
+{
+ int err;
+ struct folio *folio;
+
+ /* Allocate a new folio to be added into the swap cache. */
+ folio = folio_alloc_mpol(gfp_mask, 0, mpol, ilx, numa_node_id());
+ if (!folio)
+ return ERR_PTR(-ENOMEM);
+
+ /*
+ * Try to add the new folio to the swap cache. It returns
+ * -EEXIST if the entry is already cached.
+ */
+ err = __swap_cache_prepare_and_add(entry, folio, gfp_mask, false);
+ if (err) {
+ folio_put(folio);
+ return ERR_PTR(err);
+ }
+
+ return folio;
+}
+
+static struct folio *swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
+ struct mempolicy *mpol, pgoff_t ilx,
+ struct swap_iocb **plug, bool readahead)
{
struct swap_info_struct *si = __swap_entry_to_info(entry);
struct folio *folio;
- struct folio *result = NULL;
- *new_page_allocated = false;
/* Check the swap cache again for readahead path. */
folio = swap_cache_get_folio(entry);
if (folio)
@@ -544,17 +549,24 @@ struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_mask,
if (!swap_entry_swapped(si, entry))
return NULL;
- /* Allocate a new folio to be added into the swap cache. */
- folio = folio_alloc_mpol(gfp_mask, 0, mpol, ilx, numa_node_id());
- if (!folio)
+ do {
+ folio = swap_cache_get_folio(entry);
+ if (folio)
+ return folio;
+
+ folio = swap_cache_alloc_folio(entry, gfp, mpol, ilx);
+ } while (PTR_ERR(folio) == -EEXIST);
+
+ if (IS_ERR_OR_NULL(folio))
return NULL;
- /* Try add the new folio, returns existing folio or NULL on failure. */
- result = __swap_cache_prepare_and_add(entry, folio, gfp_mask, false);
- if (result == folio)
- *new_page_allocated = true;
- else
- folio_put(folio);
- return result;
+
+ swap_read_folio(folio, plug);
+ if (readahead) {
+ folio_set_readahead(folio);
+ count_vm_event(SWAP_RA);
+ }
+
+ return folio;
}
/**
@@ -573,15 +585,35 @@ struct folio *swap_cache_alloc_folio(swp_entry_t entry, gfp_t gfp_mask,
*/
struct folio *swapin_folio(swp_entry_t entry, struct folio *folio)
{
+ int ret;
struct folio *swapcache;
pgoff_t offset = swp_offset(entry);
unsigned long nr_pages = folio_nr_pages(folio);
entry = swp_entry(swp_type(entry), round_down(offset, nr_pages));
- swapcache = __swap_cache_prepare_and_add(entry, folio, 0, true);
- if (swapcache == folio)
- swap_read_folio(folio, NULL);
- return swapcache;
+ for (;;) {
+ ret = __swap_cache_prepare_and_add(entry, folio, 0, true);
+ if (!ret) {
+ swap_read_folio(folio, NULL);
+ break;
+ }
+
+ /*
+ * Large order allocation needs special handling on
+ * race: if a smaller folio exists in cache, swapin needs
+ * to fall back to order 0, and doing a swap cache lookup
+ * might return a folio that is irrelevant to the faulting
+ * entry because @entry is aligned down. Just return NULL.
+ */
+ if (ret != -EEXIST || nr_pages > 1)
+ return NULL;
+
+ swapcache = swap_cache_get_folio(entry);
+ if (swapcache)
+ return swapcache;
+ }
+
+ return folio;
}
/*
@@ -595,7 +627,6 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
struct swap_iocb **plug)
{
struct swap_info_struct *si;
- bool page_allocated;
struct mempolicy *mpol;
pgoff_t ilx;
struct folio *folio;
@@ -605,13 +636,9 @@ struct folio *read_swap_cache_async(swp_entry_t entry, gfp_t gfp_mask,
return NULL;
mpol = get_vma_policy(vma, addr, 0, &ilx);
- folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,
- &page_allocated);
+ folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx, plug, false);
mpol_cond_put(mpol);
- if (page_allocated)
- swap_read_folio(folio, plug);
-
put_swap_device(si);
return folio;
}
@@ -696,7 +723,7 @@ static unsigned long swapin_nr_pages(unsigned long offset)
* are fairly likely to have been swapped out from the same node.
*/
struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
- struct mempolicy *mpol, pgoff_t ilx)
+ struct mempolicy *mpol, pgoff_t ilx)
{
struct folio *folio;
unsigned long entry_offset = swp_offset(entry);
@@ -706,7 +733,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
struct swap_info_struct *si = __swap_entry_to_info(entry);
struct blk_plug plug;
struct swap_iocb *splug = NULL;
- bool page_allocated;
+ swp_entry_t ra_entry;
mask = swapin_nr_pages(offset) - 1;
if (!mask)
@@ -723,18 +750,11 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
blk_start_plug(&plug);
for (offset = start_offset; offset <= end_offset ; offset++) {
/* Ok, do the async read-ahead now */
- folio = swap_cache_alloc_folio(
- swp_entry(swp_type(entry), offset), gfp_mask, mpol, ilx,
- &page_allocated);
+ ra_entry = swp_entry(swp_type(entry), offset);
+ folio = swap_cache_read_folio(ra_entry, gfp_mask, mpol, ilx,
+ &splug, offset != entry_offset);
if (!folio)
continue;
- if (page_allocated) {
- swap_read_folio(folio, &splug);
- if (offset != entry_offset) {
- folio_set_readahead(folio);
- count_vm_event(SWAP_RA);
- }
- }
folio_put(folio);
}
blk_finish_plug(&plug);
@@ -742,11 +762,7 @@ struct folio *swap_cluster_readahead(swp_entry_t entry, gfp_t gfp_mask,
lru_add_drain(); /* Push any new pages onto the LRU now */
skip:
/* The page was likely read above, so no need for plugging here */
- folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,
- &page_allocated);
- if (unlikely(page_allocated))
- swap_read_folio(folio, NULL);
- return folio;
+ return swap_cache_read_folio(entry, gfp_mask, mpol, ilx, NULL, false);
}
static int swap_vma_ra_win(struct vm_fault *vmf, unsigned long *start,
@@ -812,8 +828,7 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
pte_t *pte = NULL, pentry;
int win;
unsigned long start, end, addr;
- pgoff_t ilx;
- bool page_allocated;
+ pgoff_t ilx = targ_ilx;
win = swap_vma_ra_win(vmf, &start, &end);
if (win == 1)
@@ -847,19 +862,12 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
if (!si)
continue;
}
- folio = swap_cache_alloc_folio(entry, gfp_mask, mpol, ilx,
- &page_allocated);
+ folio = swap_cache_read_folio(entry, gfp_mask, mpol, ilx,
+ &splug, addr != vmf->address);
if (si)
put_swap_device(si);
if (!folio)
continue;
- if (page_allocated) {
- swap_read_folio(folio, &splug);
- if (addr != vmf->address) {
- folio_set_readahead(folio);
- count_vm_event(SWAP_RA);
- }
- }
folio_put(folio);
}
if (pte)
@@ -869,10 +877,8 @@ static struct folio *swap_vma_readahead(swp_entry_t targ_entry, gfp_t gfp_mask,
lru_add_drain();
skip:
/* The folio was likely read above, so no need for plugging here */
- folio = swap_cache_alloc_folio(targ_entry, gfp_mask, mpol, targ_ilx,
- &page_allocated);
- if (unlikely(page_allocated))
- swap_read_folio(folio, NULL);
+ folio = swap_cache_read_folio(targ_entry, gfp_mask, mpol, targ_ilx,
+ NULL, false);
return folio;
}
diff --git a/mm/zswap.c b/mm/zswap.c
index 4b5149173b0e..e27f6e96f003 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -991,7 +991,6 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
pgoff_t offset = swp_offset(swpentry);
struct folio *folio;
struct mempolicy *mpol;
- bool folio_was_allocated;
struct swap_info_struct *si;
int ret = 0;
@@ -1002,22 +1001,18 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
mpol = get_task_policy(current);
folio = swap_cache_alloc_folio(swpentry, GFP_KERNEL, mpol,
- NO_INTERLEAVE_INDEX, &folio_was_allocated);
+ NO_INTERLEAVE_INDEX);
put_swap_device(si);
- if (!folio)
- return -ENOMEM;
/*
- * Found an existing folio, we raced with swapin or concurrent
- * shrinker. We generally writeback cold folios from zswap, and
- * swapin means the folio just became hot, so skip this folio.
- * For unlikely concurrent shrinker case, it will be unlinked
- * and freed when invalidated by the concurrent shrinker anyway.
+ * Swap cache allocation might fail due to OOM, or the entry
+ * may already be cached due to concurrent swapin or have been
+ * freed. If already cached, a concurrent swapin made the folio
+ * hot, so skip it. For the unlikely concurrent shrinker case,
+ * it will be unlinked and freed when invalidated anyway.
*/
- if (!folio_was_allocated) {
- ret = -EEXIST;
- goto out;
- }
+ if (IS_ERR(folio))
+ return PTR_ERR(folio);
/*
* folio is locked, and the swapcache is now secured against
@@ -1057,7 +1052,7 @@ static int zswap_writeback_entry(struct zswap_entry *entry,
__swap_writepage(folio, NULL);
out:
- if (ret && ret != -EEXIST) {
+ if (ret) {
swap_cache_del_folio(folio);
folio_unlock(folio);
}
--
2.54.0
^ permalink raw reply related
* [PATCH v5 03/12] mm/huge_memory: move THP gfp limit helper into header
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
Shmem has some special requirements for THP GFP and has to limit it in
certain zones or provide a more lenient fallback.
We'll use this helper for generic swap THP allocation, which needs to
support shmem. For a typical GFP_HIGHUSER_MOVABLE swap-in, this helper
is basically a no-op. But it's necessary for certain shmem users, mostly
drivers.
No feature change.
Acked-by: Chris Li <chrisl@kernel.org>
Reviewed-by: Zi Yan <ziy@nvidia.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/huge_mm.h | 30 ++++++++++++++++++++++++++++++
mm/shmem.c | 30 +++---------------------------
2 files changed, 33 insertions(+), 27 deletions(-)
diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h
index 127f9e1e7604..edece3e26985 100644
--- a/include/linux/huge_mm.h
+++ b/include/linux/huge_mm.h
@@ -242,6 +242,31 @@ static inline bool thp_vma_suitable_order(struct vm_area_struct *vma,
return true;
}
+/*
+ * Make sure huge_gfp is always more limited than limit_gfp.
+ * Some shmem users want THP allocation to be done less aggressively
+ * and only in certain zone.
+ */
+static inline gfp_t thp_shmem_limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
+{
+ gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM;
+ gfp_t denyflags = __GFP_NOWARN | __GFP_NORETRY;
+ gfp_t zoneflags = limit_gfp & GFP_ZONEMASK;
+ gfp_t result = huge_gfp & ~(allowflags | GFP_ZONEMASK);
+
+ /* Allow allocations only from the originally specified zones. */
+ result |= zoneflags;
+
+ /*
+ * Minimize the result gfp by taking the union with the deny flags,
+ * and the intersection of the allow flags.
+ */
+ result |= (limit_gfp & denyflags);
+ result |= (huge_gfp & limit_gfp) & allowflags;
+
+ return result;
+}
+
/*
* Filter the bitfield of input orders to the ones suitable for use in the vma.
* See thp_vma_suitable_order().
@@ -565,6 +590,11 @@ static inline bool thp_vma_suitable_order(struct vm_area_struct *vma,
return false;
}
+static inline gfp_t thp_shmem_limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
+{
+ return huge_gfp;
+}
+
static inline unsigned long thp_vma_suitable_orders(struct vm_area_struct *vma,
unsigned long addr, unsigned long orders)
{
diff --git a/mm/shmem.c b/mm/shmem.c
index bab3529af23c..6edb23b41bac 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -1791,30 +1791,6 @@ static struct folio *shmem_swapin_cluster(swp_entry_t swap, gfp_t gfp,
return folio;
}
-/*
- * Make sure huge_gfp is always more limited than limit_gfp.
- * Some of the flags set permissions, while others set limitations.
- */
-static gfp_t limit_gfp_mask(gfp_t huge_gfp, gfp_t limit_gfp)
-{
- gfp_t allowflags = __GFP_IO | __GFP_FS | __GFP_RECLAIM;
- gfp_t denyflags = __GFP_NOWARN | __GFP_NORETRY;
- gfp_t zoneflags = limit_gfp & GFP_ZONEMASK;
- gfp_t result = huge_gfp & ~(allowflags | GFP_ZONEMASK);
-
- /* Allow allocations only from the originally specified zones. */
- result |= zoneflags;
-
- /*
- * Minimize the result gfp by taking the union with the deny flags,
- * and the intersection of the allow flags.
- */
- result |= (limit_gfp & denyflags);
- result |= (huge_gfp & limit_gfp) & allowflags;
-
- return result;
-}
-
#ifdef CONFIG_TRANSPARENT_HUGEPAGE
bool shmem_hpage_pmd_enabled(void)
{
@@ -2065,7 +2041,7 @@ static struct folio *shmem_swap_alloc_folio(struct inode *inode,
non_swapcache_batch(entry, nr_pages) != nr_pages)
goto fallback;
- alloc_gfp = limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
+ alloc_gfp = thp_shmem_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
}
retry:
new = shmem_alloc_folio(alloc_gfp, order, info, index);
@@ -2141,7 +2117,7 @@ static int shmem_replace_folio(struct folio **foliop, gfp_t gfp,
if (nr_pages > 1) {
gfp_t huge_gfp = vma_thp_gfp_mask(vma);
- gfp = limit_gfp_mask(huge_gfp, gfp);
+ gfp = thp_shmem_limit_gfp_mask(huge_gfp, gfp);
}
#endif
@@ -2548,7 +2524,7 @@ static int shmem_get_folio_gfp(struct inode *inode, pgoff_t index,
gfp_t huge_gfp;
huge_gfp = vma_thp_gfp_mask(vma);
- huge_gfp = limit_gfp_mask(huge_gfp, gfp);
+ huge_gfp = thp_shmem_limit_gfp_mask(huge_gfp, gfp);
folio = shmem_alloc_and_add_folio(vmf, huge_gfp,
inode, index, fault_mm, orders);
if (!IS_ERR(folio)) {
--
2.54.0
^ permalink raw reply related
* [PATCH v5 02/12] mm, swap: move common swap cache operations into standalone helpers
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260517-swap-table-p4-v5-0-88ae43e064c7@tencent.com>
From: Kairui Song <kasong@tencent.com>
Move a few swap cache checking, adding, and deletion operations
into standalone helpers to be used later. And while at it, add
proper kernel doc.
No feature or behavior change.
Acked-by: Chris Li <chrisl@kernel.org>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/swap_state.c | 146 ++++++++++++++++++++++++++++++++++++++------------------
1 file changed, 100 insertions(+), 46 deletions(-)
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 3bba82f6dc79..89fa19ec13f6 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -137,8 +137,47 @@ void *swap_cache_get_shadow(swp_entry_t entry)
return NULL;
}
-void __swap_cache_add_folio(struct swap_cluster_info *ci,
- struct folio *folio, swp_entry_t entry)
+/**
+ * __swap_cache_add_check - Check if a range is suitable for adding a folio.
+ * @ci: The locked swap cluster.
+ * @ci_off: Range start offset.
+ * @nr: Number of slots to check.
+ * @shadow: Returns the shadow value if one exists in the range.
+ *
+ * Check if all slots covered by given range have a swap count >= 1.
+ * Retrieves the shadow if there is one.
+ *
+ * Context: Caller must lock the cluster.
+ * Return: 0 if success, error code if failed.
+ */
+static int __swap_cache_add_check(struct swap_cluster_info *ci,
+ unsigned int ci_off, unsigned int nr,
+ void **shadow)
+{
+ unsigned int ci_end = ci_off + nr;
+ unsigned long old_tb;
+
+ lockdep_assert_held(&ci->lock);
+ if (WARN_ON_ONCE(ci_off >= SWAPFILE_CLUSTER))
+ return -EINVAL;
+
+ if (unlikely(!ci->table))
+ return -ENOENT;
+ do {
+ old_tb = __swap_table_get(ci, ci_off);
+ if (unlikely(swp_tb_is_folio(old_tb)))
+ return -EEXIST;
+ if (unlikely(!__swp_tb_get_count(old_tb)))
+ return -ENOENT;
+ if (swp_tb_is_shadow(old_tb))
+ *shadow = swp_tb_to_shadow(old_tb);
+ } while (++ci_off < ci_end);
+
+ return 0;
+}
+
+static void __swap_cache_do_add_folio(struct swap_cluster_info *ci,
+ struct folio *folio, swp_entry_t entry)
{
unsigned int ci_off = swp_cluster_offset(entry), ci_end;
unsigned long nr_pages = folio_nr_pages(folio);
@@ -159,7 +198,28 @@ void __swap_cache_add_folio(struct swap_cluster_info *ci,
folio_ref_add(folio, nr_pages);
folio_set_swapcache(folio);
folio->swap = entry;
+}
+
+/**
+ * __swap_cache_add_folio - Add a folio to the swap cache and update stats.
+ * @ci: The locked swap cluster.
+ * @folio: The folio to be added.
+ * @entry: The swap entry corresponding to the folio.
+ *
+ * Unconditionally add a folio to the swap cache. The caller must ensure
+ * all slots are usable and have no conflicts. This assigns entry to
+ * @folio->swap, increases folio refcount by the number of pages, and
+ * updates swap cache stats.
+ *
+ * Context: Caller must ensure the folio is locked and lock the cluster
+ * that holds the entries.
+ */
+void __swap_cache_add_folio(struct swap_cluster_info *ci,
+ struct folio *folio, swp_entry_t entry)
+{
+ unsigned long nr_pages = folio_nr_pages(folio);
+ __swap_cache_do_add_folio(ci, folio, entry);
node_stat_mod_folio(folio, NR_FILE_PAGES, nr_pages);
lruvec_stat_mod_folio(folio, NR_SWAPCACHE, nr_pages);
}
@@ -168,9 +228,11 @@ void __swap_cache_add_folio(struct swap_cluster_info *ci,
* swap_cache_add_folio - Add a folio into the swap cache.
* @folio: The folio to be added.
* @entry: The swap entry corresponding to the folio.
- * @gfp: gfp_mask for XArray node allocation.
* @shadowp: If a shadow is found, return the shadow.
*
+ * Add a folio into the swap cache. Will return error if any slot is no
+ * longer a valid swapped out slot or already occupied by another folio.
+ *
* Context: Caller must ensure @entry is valid and protect the swap device
* with reference count or locks.
*/
@@ -179,60 +241,31 @@ static int swap_cache_add_folio(struct folio *folio, swp_entry_t entry,
{
int err;
void *shadow = NULL;
- unsigned long old_tb;
+ unsigned int ci_off;
struct swap_info_struct *si;
struct swap_cluster_info *ci;
- unsigned int ci_start, ci_off, ci_end;
unsigned long nr_pages = folio_nr_pages(folio);
si = __swap_entry_to_info(entry);
- ci_start = swp_cluster_offset(entry);
- ci_end = ci_start + nr_pages;
- ci_off = ci_start;
ci = swap_cluster_lock(si, swp_offset(entry));
- if (unlikely(!ci->table)) {
- err = -ENOENT;
- goto failed;
+ ci_off = swp_cluster_offset(entry);
+ err = __swap_cache_add_check(ci, ci_off, nr_pages, &shadow);
+ if (err) {
+ swap_cluster_unlock(ci);
+ return err;
}
- do {
- old_tb = __swap_table_get(ci, ci_off);
- if (unlikely(swp_tb_is_folio(old_tb))) {
- err = -EEXIST;
- goto failed;
- }
- if (unlikely(!__swp_tb_get_count(old_tb))) {
- err = -ENOENT;
- goto failed;
- }
- if (swp_tb_is_shadow(old_tb))
- shadow = swp_tb_to_shadow(old_tb);
- } while (++ci_off < ci_end);
+
__swap_cache_add_folio(ci, folio, entry);
swap_cluster_unlock(ci);
if (shadowp)
*shadowp = shadow;
- return 0;
-failed:
- swap_cluster_unlock(ci);
- return err;
+ return 0;
}
-/**
- * __swap_cache_del_folio - Removes a folio from the swap cache.
- * @ci: The locked swap cluster.
- * @folio: The folio.
- * @entry: The first swap entry that the folio corresponds to.
- * @shadow: shadow value to be filled in the swap cache.
- *
- * Removes a folio from the swap cache and fills a shadow in place.
- * This won't put the folio's refcount. The caller has to do that.
- *
- * Context: Caller must ensure the folio is locked and in the swap cache
- * using the index of @entry, and lock the cluster that holds the entries.
- */
-void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio,
- swp_entry_t entry, void *shadow)
+static void __swap_cache_do_del_folio(struct swap_cluster_info *ci,
+ struct folio *folio,
+ swp_entry_t entry, void *shadow)
{
int count;
unsigned long old_tb;
@@ -259,14 +292,12 @@ void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio,
folio_swapped = true;
else
need_free = true;
- /* If shadow is NULL, we sets an empty shadow. */
+ /* If shadow is NULL, we set an empty shadow. */
__swap_table_set(ci, ci_off, shadow_to_swp_tb(shadow, count));
} while (++ci_off < ci_end);
folio->swap.val = 0;
folio_clear_swapcache(folio);
- node_stat_mod_folio(folio, NR_FILE_PAGES, -nr_pages);
- lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr_pages);
if (!folio_swapped) {
__swap_cluster_free_entries(si, ci, ci_start, nr_pages);
@@ -279,6 +310,29 @@ void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio,
}
}
+/**
+ * __swap_cache_del_folio - Removes a folio from the swap cache.
+ * @ci: The locked swap cluster.
+ * @folio: The folio.
+ * @entry: The first swap entry that the folio corresponds to.
+ * @shadow: shadow value to be filled in the swap cache.
+ *
+ * Removes a folio from the swap cache and fills a shadow in place.
+ * This won't put the folio's refcount. The caller has to do that.
+ *
+ * Context: Caller must ensure the folio is locked and in the swap cache
+ * using the index of @entry, and lock the cluster that holds the entries.
+ */
+void __swap_cache_del_folio(struct swap_cluster_info *ci, struct folio *folio,
+ swp_entry_t entry, void *shadow)
+{
+ unsigned long nr_pages = folio_nr_pages(folio);
+
+ __swap_cache_do_del_folio(ci, folio, entry, shadow);
+ node_stat_mod_folio(folio, NR_FILE_PAGES, -nr_pages);
+ lruvec_stat_mod_folio(folio, NR_SWAPCACHE, -nr_pages);
+}
+
/**
* swap_cache_del_folio - Removes a folio from the swap cache.
* @folio: The folio.
--
2.54.0
^ permalink raw reply related
* [PATCH v5 00/12] mm, swap: swap table phase IV: unify allocation and reduce static metadata
From: Kairui Song via B4 Relay @ 2026-05-17 15:39 UTC (permalink / raw)
To: linux-mm
Cc: Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang, Barry Song,
Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham, Baoquan He,
Johannes Weiner, Youngjun Park, Chengming Zhou, Roman Gushchin,
Shakeel Butt, Muchun Song, Usama Arif, linux-kernel, cgroups,
Kairui Song, Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
From: Kairui Song <kasong@tencent.com>
This series unifies the allocation and charging of anon and shmem swap
in folios, provides better synchronization, consolidates the metadata
management, hence dropping the static array and map, and improves the
performance. The static metadata overhead is now close to zero, and
workload performance is slightly improved.
For example, mounting a 1TB swap device saves about 512MB of memory:
Before:
free -m
total used free shared buff/cache available
Mem: 1464 805 346 1 382 658
Swap: 1048575 0 1048575
After:
free -m
total used free shared buff/cache available
Mem: 1464 277 899 1 356 1187
Swap: 1048575 0 1048575
Memory usage is ~512M lower, and we now have a close to 0 static
overhead. It was about 2 bytes per slot before, now roughly 0.09375
bytes per slot (48 bytes ci info per cluster, which is 512 slots).
Performance test is also looking good, testing Redis in a 2G VM using
6G ZRAM as swap:
valkey-server --maxmemory 2560M
redis-benchmark -r 3000000 -n 3000000 -d 1024 -c 12 -P 32 -t get
Before: 3385017.283654 RPS
After: 3433309.307292 RPS (1.42% better)
Testing with build kernel under global pressure on a 48c96t system,
limiting the total memory to 8G, using 12G ZRAM, 24 test runs,
enabling THP:
make -j96, using defconfig
Before: user time 2904.59s system time 4773.99s
After: user time 2909.38s system time 4641.55s (2.77% better)
Testing with usemem on a 32c machine using 48G brd ramdisk and 16G
RAM, 12 test run:
usemem --init-time -O -y -x -n 48 1G
Before: Throughput (Sum): 6482.58 MB/s Free Latency: 371371.67us
After: Throughput (Sum): 6539.28 MB/s Free Latency: 363059.88us
Seems similar, or slightly better.
This series also reduces memory thrashing, I no longer see any:
"Huh VM_FAULT_OOM leaked out to the #PF handler. Retrying PF", it was
shown several times during stress testing before this series when under
great pressure:
Before: grep -Ri VM_FAULT_OOM <test logs> | wc -l => 18
After: grep -Ri VM_FAULT_OOM <test logs> | wc -l => 0
Signed-off-by: Kairui Song <kasong@tencent.com>
---
Changes in v5:
- Fix error with !CONFIG_TRANSPARENT_HUGEPAGE:
https://lore.kernel.org/linux-mm/agcdxIFQ8QBI9R6z@KASONG-MC4/
The actual fix applied is different since the posted one forgot to
check `orders` as loop breaking condition.
- Improve mem policy interleave in patch 5:
https://lore.kernel.org/linux-mm/CAMgjq7AqKskE5UVivTEdPzmTa09_aapWZM7JeSshhmf-4GYbZw@mail.gmail.com/
- Retest is still looking good.
- Link to v4: https://patch.msgid.link/20260515-swap-table-p4-v4-0-f1b49e845a8d@tencent.com
Changes in v4:
- Rebased on latest mm-unstable and re-test, benchmark results are
basically the same so mostly kept unchanged. Changes in v4 are code
style and very minor behavior change.
- Improve a few commit messages, rename a few variables as suggested by
[ Chris Li ].
- Rename thp_limit_gfp_mask to thp_shmem_limit_gfp_mask as suggested by
[ Zi Yan ].
- Cleanup a few allocation and code style issue [ YoungJun Park ]
- Remove the forced fallback in swap_cache_alloc_folio, the caller will
pass in the exact orders to be used. [ Baolin Wang ]
- Rename swapin_entry to swapin_sync, it's only used by synchronization
devices at this moment and describes what it does better
[ David Hildenbrand ]
- Link to v3: https://patch.msgid.link/20260421-swap-table-p4-v3-0-2f23759a76bc@tencent.com
Changes in v3:
- This is based on mm-unstable, also applies to mm-new, and has no
conflict with YoungJun's tier series, and only trivial conflict with
Baoquan's swapops due to filename change.
- Fix zero map build issue on 32 bit archs [ YoungJun Park ]
- Cleanup memcg table allocation helpers [ YoungJun Park ]
- Fix WARN for non NUMA build:
https://lore.kernel.org/linux-mm/CAMgjq7ANih7u7SJB8uWcQHS8XRJySNRc3ti9V-SVey0nGE3gLQ@mail.gmail.com/
- Improve of commit messages.
- Re-test several tests, the conclusion is the same as v2.
- Link to v2: https://patch.msgid.link/20260417-swap-table-p4-v2-0-17f5d1015428@tencent.com
Changes in v2:
- Drop the RFC prefix and also the RFC part.
- Now there is zero change to cgroup or refault tracking, RFC v1 changed
some cgroup behavior. To archive that v2 use a standalone memcg_table
for each cluster. It can be dropped or better optimized later if we
have a better solution. The performance gain is partly cancelled
compared to RFC v1 since we now need an extra allocation for free cluster
isolation and peak memory usage is 2 bytes higher. But still looking
good. That table size is accetable (1024 bytes), no RCU needed, and
fits for kmalloc. Even if we keep it as it is in the future,
it's still accetable.
- Link to v1: https://lore.kernel.org/r/20260220-swap-table-p4-v1-0-104795d19815@tencent.com
---
Kairui Song (12):
mm, swap: simplify swap cache allocation helper
mm, swap: move common swap cache operations into standalone helpers
mm/huge_memory: move THP gfp limit helper into header
mm, swap: add support for stable large allocation in swap cache directly
mm, swap: unify large folio allocation
mm/memcg, swap: tidy up cgroup v1 memsw swap helpers
mm, swap: support flexible batch freeing of slots in different memcgs
mm, swap: delay and unify memcg lookup and charging for swapin
mm, swap: consolidate cluster allocation helpers
mm/memcg, swap: store cgroup id in cluster table directly
mm/memcg: remove no longer used swap cgroup array
mm, swap: merge zeromap into swap table
MAINTAINERS | 1 -
include/linux/huge_mm.h | 30 +++
include/linux/memcontrol.h | 16 +-
include/linux/swap.h | 19 +-
include/linux/swap_cgroup.h | 47 ----
mm/Makefile | 3 -
mm/huge_memory.c | 2 +-
mm/internal.h | 11 +-
mm/memcontrol-v1.c | 66 ++++--
mm/memcontrol.c | 31 ++-
mm/memory.c | 91 ++------
mm/page_io.c | 61 +++++-
mm/shmem.c | 130 +++--------
mm/swap.h | 91 +++-----
mm/swap_cgroup.c | 174 ---------------
mm/swap_state.c | 523 +++++++++++++++++++++++++-------------------
mm/swap_table.h | 179 ++++++++++++---
mm/swapfile.c | 215 +++++++++---------
mm/vmscan.c | 2 +-
mm/zswap.c | 25 +--
20 files changed, 814 insertions(+), 903 deletions(-)
---
base-commit: 444fc9435e57157fcf30fc99aee44997f3458641
change-id: 20260111-swap-table-p4-98ee92baa7c4
Best regards,
--
Kairui Song <kasong@tencent.com>
^ permalink raw reply
* Re: [linus:master] [mm] 01b9da291c: stress-ng.switch.ops_per_sec 67.7% regression
From: Oliver Sang @ 2026-05-17 12:55 UTC (permalink / raw)
To: Shakeel Butt
Cc: Qi Zheng, oe-lkp, lkp, linux-kernel, Andrew Morton, David Carlier,
Allen Pais, Axel Rasmussen, Baoquan He, Chengming Zhou,
Chen Ridong, David Hildenbrand, Hamza Mahfooz, Harry Yoo,
Hugh Dickins, Imran Khan, Johannes Weiner, Kamalesh Babulal,
Lance Yang, Liam Howlett, Lorenzo Stoakes, Michal Hocko,
Michal Koutný, Mike Rapoport, Muchun Song, Muchun Song,
Nhat Pham, Roman Gushchin, Suren Baghdasaryan, Usama Arif,
Vlastimil Babka, Wei Xu, Yosry Ahmed, Yuanchu Xie, Zi Yan,
Usama Arif, cgroups, linux-mm, oliver.sang
In-Reply-To: <agdS5rIhcjIBVSog@linux.dev>
hi, Shakeel, hi, Qi,
On Fri, May 15, 2026 at 10:09:06AM -0700, Shakeel Butt wrote:
> On Fri, May 15, 2026 at 03:37:22PM +0800, Qi Zheng wrote:
> > Hi Shakeel,
> >
> > On 5/14/26 9:40 PM, Shakeel Butt wrote:
> > > May 14, 2026 at 12:46 AM, "Qi Zheng" <qi.zheng@linux.dev mailto:qi.zheng@linux.dev?to=%22Qi%20Zheng%22%20%3Cqi.zheng%40linux.dev%3E > wrote:
> > >
> > >
> > > >
> > > > On 5/13/26 10:27 PM, Shakeel Butt wrote:
> > > >
> > > > >
> > > > > On Wed, May 13, 2026 at 06:49:45AM -0700, Shakeel Butt wrote:
> > > > >
> > > > > >
> > > > > > On Wed, May 13, 2026 at 10:10:34AM +0800, Qi Zheng wrote:
> > > > > >
> > > > > On 5/13/26 12:03 AM, Shakeel Butt wrote:
> > > > > On Tue, May 12, 2026 at 08:56:52PM +0800, kernel test robot wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > kernel test robot noticed a 67.7% regression of stress-ng.switch.ops_per_sec on:
> > > > >
> > > > > commit: 01b9da291c4969354807b52956f4aae1f41b4924 ("mm: memcontrol: convert objcg to be per-memcg per-node type")
> > > > > https://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git master
> > > > >
> > > > > This is most probably due to shuffling of struct mem_cgroup and struct
> > > > > mem_cgroup_per_node members.
> > > > >
> > > > > Another possibility is that after objcg was split into per-node, the
> > > > > slab accounting fast path is still designed assuming only one current
> > > > > objcg per CPU:
> > > > >
> > > > > struct obj_stock_pcp {
> > > > > struct obj_cgroup *cached_objcg;
> > > > > };
> > > > >
> > > > > So it's may cause the following thrashing:
> > > > >
> > > > > CPU stock cached = memcg/node0 objcg
> > > > > free object tagged = memcg/node1 objcg
> > > > > => __refill_obj_stock --> objcg mismatch
> > > > > => drain_obj_stock()
> > > > > => cache switches to node1 objcg
> > > > >
> > > > > next local allocation tagged = node0 objcg
> > > > > => mismatch again
> > > > > => drain_obj_stock()
> > > > >
> > > > > >
> > > > > > Actually I think this is the issue, we have ping pong threads running on
> > > > > > different nodes where though theu are in same cgroup but their current->obcg is
> > > > > > for local node and thus this ping pong is thrashing the per-cpu objcg stock.
> > > > > >
> > > > > > The easier fix would be to compare objcg->memcg instead of just objcg during
> > > > > > draining and caching. In addition we can add support for multiple objcg per-cpu
> > > > > > stock caching.
> > > > > >
> > > > > Something like the following:
> > > > > From d756abe831a905d6fe32bad9a984fc619dafb7e0 Mon Sep 17 00:00:00 2001
> > > > > From: Shakeel Butt <shakeel.butt@linux.dev>
> > > > > Date: Wed, 13 May 2026 07:24:55 -0700
> > > > > Subject: [PATCH] mm/memcontrol: skip obj_stock drain when refilled objcg
> > > > > shares memcg
> > > > > Signed-off-by: Shakeel Butt <shakeel.butt@linux.dev>
> > > > > ---
> > > > > mm/memcontrol.c | 14 +++++++++++++-
> > > > > 1 file changed, 13 insertions(+), 1 deletion(-)
> > > > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > > > > index d978e18b9b2d..01ed7a8e18ac 100644
> > > > > --- a/mm/memcontrol.c
> > > > > +++ b/mm/memcontrol.c
> > > > > @@ -3318,6 +3318,7 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > > > > unsigned int nr_bytes,
> > > > > bool allow_uncharge)
> > > > > {
> > > > > + struct obj_cgroup *cached;
> > > > > unsigned int nr_pages = 0;
> > > > > > if (!stock) {
> > > > > @@ -3327,7 +3328,18 @@ static void __refill_obj_stock(struct obj_cgroup *objcg,
> > > > > goto out;
> > > > > }
> > > > > > - if (READ_ONCE(stock->cached_objcg) != objcg) { /* reset if necessary */
> > > > > + cached = READ_ONCE(stock->cached_objcg);
> > > > > + if (cached != objcg &&
> > > > > + (!cached || obj_cgroup_memcg(cached) != obj_cgroup_memcg(objcg))) {
> > > > > drain_obj_stock(stock);
> > > > > obj_cgroup_get(objcg);
> > > > > stock->nr_bytes = atomic_read(&objcg->nr_charged_bytes)
> > > > >
> > > > This change looks like it should be able to fix the ping-pong issue, but
> > > > I stiil haven't reproduced the performance regression locally. I'll
> > > > continue testing it.
> > >
> > > Same here, couldn't reproduce locally. It seems like we had to craft a scenario
> > > where the pair pingpong threads get their current->objcg from different nodes.
> > > I will try that.
> >
> > I still haven't been able to reproduce the LKP results locally, but I
> > used an AI bot to generate a pingpong test case (pasted at the end) and
> > automatically ran the test on a physical machine. The results are as
> > follows:
> >
> > parent: 8285917d6f
> > bad: 01b9da291c
> > fix: 01b9da291c + stock patch
> >
> > | kernel | mq_ops/sec mean | vs parent | drain_obj_stock / round |
> > |--------|-----------------|-----------|-------------------------|
> > | parent | 9.743M | baseline | ~0 |
> > | bad | 7.821M | -19.73% | ~11.16M |
> > | fix | 9.274M | -4.81% | ~0 |
> >
> > Probing the drain_obj_stock() calls confirms that the fix restores the
> > frequency to the parent's baseline.
> >
> > And it seems that besides __refill_obj_stock(), we should also modify
> > __consume_obj_stock()?
> >
>
> Thanks a lot Qi. I will send the formal patch and will add your Debugged-by if
> you don't mind.
>
Tested-by: kernel test robot <oliver.sang@intel.com>
we tested above patch, and it recovers the regression:
=========================================================================================
compiler/cpufreq_governor/kconfig/method/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-14/performance/x86_64-rhel-9.4/mq/100%/debian-13-x86_64-20250902.cgz/lkp-spr-r02/switch/stress-ng/60s
commit:
8285917d6f ("mm: memcontrol: prepare for reparenting non-hierarchical stats")
01b9da291c ("mm: memcontrol: convert objcg to be per-memcg per-node type")
682fd4e9ff <--- above patch from Shakeel
8285917d6f383aef 01b9da291c4969354807b52956f 682fd4e9ffd4009805f81dd25ed
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
5849 +210.2% 18145 ± 3% +1.5% 5935 stress-ng.switch.nanosecs_per_context_switch_mq_method
2.296e+09 -67.7% 7.408e+08 ± 3% -1.4% 2.263e+09 stress-ng.switch.ops
38288993 -67.7% 12355813 ± 3% -1.4% 37739220 stress-ng.switch.ops_per_sec
full compasison is as below [3]
but there are two notes.
#1 is that we noticed there is a fomal patch later from Shakeel in [1] which has
more changes. not sure if this test is enough? do you want us to test [1]
further?
#2: when we test above patch, we found the server easy to crash while running
tests. we try to run up to 20 times, only 2 of them run successfully (above
37739220 is just the average data from these 2 runs, since the data is stable,
we think maybe it's ok to report to you with this data).
we also noticed for [1] there is a [syzbot ci] report in [2]. since we don't
have serial output for our test server in this report which is for performance
tests, we cannot say if other 18 runs failed due to similar reason. just FYI.
[1] https://lore.kernel.org/all/20260515171953.2224503-1-shakeel.butt@linux.dev/
[2] https://lore.kernel.org/all/6a081599.170a0220.4530d.0003.GAE@google.com/
[3]
=========================================================================================
compiler/cpufreq_governor/kconfig/method/nr_threads/rootfs/tbox_group/test/testcase/testtime:
gcc-14/performance/x86_64-rhel-9.4/mq/100%/debian-13-x86_64-20250902.cgz/lkp-spr-r02/switch/stress-ng/60s
commit:
8285917d6f ("mm: memcontrol: prepare for reparenting non-hierarchical stats")
01b9da291c ("mm: memcontrol: convert objcg to be per-memcg per-node type")
682fd4e9ff <--- above patch from Shakeel
8285917d6f383aef 01b9da291c4969354807b52956f 682fd4e9ffd4009805f81dd25ed
---------------- --------------------------- ---------------------------
%stddev %change %stddev %change %stddev
\ | \ | \
5849 +210.2% 18145 ± 3% +1.5% 5935 stress-ng.switch.nanosecs_per_context_switch_mq_method
2.296e+09 -67.7% 7.408e+08 ± 3% -1.4% 2.263e+09 stress-ng.switch.ops
38288993 -67.7% 12355813 ± 3% -1.4% 37739220 stress-ng.switch.ops_per_sec
93416932 -68.6% 29310048 ± 3% +0.1% 93506345 stress-ng.time.involuntary_context_switches
15845 +11.0% 17584 +0.3% 15894 stress-ng.time.percent_of_cpu_this_job_got
8556 +18.2% 10115 +0.5% 8597 stress-ng.time.system_time
963.36 -53.5% 447.72 ± 3% -1.3% 950.84 stress-ng.time.user_time
1.518e+09 -69.7% 4.607e+08 ± 2% -1.5% 1.496e+09 stress-ng.time.voluntary_context_switches
1124 ± 17% +34.3% 1509 ± 8% -7.7% 1037 ± 14% perf-c2c.HITM.remote
2.55e+09 -12.3% 2.236e+09 ± 3% -2.3% 2.491e+09 cpuidle..time
8.29e+08 -71.8% 2.337e+08 ± 2% -1.8% 8.143e+08 cpuidle..usage
14184409 ± 2% -16.4% 11860068 +0.5% 14255389 ± 2% vmstat.memory.cache
39204964 -69.7% 11868752 ± 2% -1.2% 38715052 vmstat.system.cs
1808848 -38.5% 1111830 -0.8% 1793867 vmstat.system.in
115757 ± 52% +8.5% 125625 ± 44% -62.0% 44014 ± 32% numa-numastat.node0.other_node
4102960 ± 5% -19.0% 3324393 ± 4% -1.0% 4063575 ± 2% numa-numastat.node1.local_node
4218983 ± 3% -18.7% 3430325 ± 3% +7.3% 4526769 ± 3% numa-numastat.node1.numa_hit
116023 ± 52% -8.7% 105932 ± 52% +62.0% 187903 ± 7% numa-numastat.node1.other_node
93416932 -68.6% 29310048 ± 3% +0.1% 93506345 time.involuntary_context_switches
15845 +11.0% 17584 +0.3% 15894 time.percent_of_cpu_this_job_got
8556 +18.2% 10115 +0.5% 8597 time.system_time
963.36 -53.5% 447.72 ± 3% -1.3% 950.84 time.user_time
1.518e+09 -69.7% 4.607e+08 ± 2% -1.5% 1.496e+09 time.voluntary_context_switches
22.48 -4.4 18.08 -0.2 22.29 mpstat.cpu.all.idle%
1.13 -0.4 0.73 -0.0 1.12 mpstat.cpu.all.irq%
0.10 -0.0 0.09 -0.0 0.10 mpstat.cpu.all.soft%
67.98 +9.1 77.06 +0.3 68.29 mpstat.cpu.all.sys%
8.32 -4.3 4.04 ± 2% -0.1 8.21 mpstat.cpu.all.usr%
17.33 ± 2% +15.4% 20.00 ± 4% +3.8% 18.00 mpstat.max_utilization.seconds
10552401 ± 4% -23.3% 8092823 +0.3% 10586331 ± 6% numa-meminfo.node1.Active
10552392 ± 4% -23.3% 8092820 +0.3% 10586323 ± 6% numa-meminfo.node1.Active(anon)
12454155 ± 15% -34.9% 8106052 -3.6% 12008629 ± 16% numa-meminfo.node1.FilePages
559046 ± 8% -19.2% 451929 ± 2% -2.4% 545736 ± 10% numa-meminfo.node1.Mapped
14688311 ± 13% -30.0% 10285394 ± 2% -4.7% 14004338 ± 15% numa-meminfo.node1.MemUsed
10028979 ± 3% -22.4% 7783864 +0.8% 10109957 ± 3% numa-meminfo.node1.Shmem
10939677 ± 2% -21.0% 8641166 +0.5% 10995134 ± 2% meminfo.Active
10939661 ± 2% -21.0% 8641149 +0.5% 10995117 ± 2% meminfo.Active(anon)
13917673 ± 2% -16.4% 11633722 +0.4% 13973077 ± 2% meminfo.Cached
14400924 ± 2% -16.0% 12102150 +0.6% 14482134 ± 2% meminfo.Committed_AS
8394752 ± 5% +16.7% 9796949 ± 8% +10.5% 9274368 ± 16% meminfo.DirectMap2M
617671 -12.0% 543559 -1.5% 608569 meminfo.Mapped
18364992 -12.5% 16065468 +0.3% 18426986 meminfo.Memused
10124702 ± 2% -22.6% 7839682 +0.6% 10181038 ± 3% meminfo.Shmem
18393665 -12.5% 16100473 +0.4% 18458236 meminfo.max_used_kB
115757 ± 52% +8.5% 125625 ± 44% -62.0% 44014 ± 32% numa-vmstat.node0.numa_other
2638537 ± 4% -23.3% 2022639 +0.3% 2647116 ± 6% numa-vmstat.node1.nr_active_anon
3113944 ± 15% -34.9% 2025946 -3.6% 3002667 ± 16% numa-vmstat.node1.nr_file_pages
139848 ± 9% -19.3% 112912 ± 2% -2.4% 136548 ± 10% numa-vmstat.node1.nr_mapped
2507650 ± 3% -22.4% 1945399 +0.8% 2527999 ± 3% numa-vmstat.node1.nr_shmem
2638531 ± 4% -23.3% 2022634 +0.3% 2647111 ± 6% numa-vmstat.node1.nr_zone_active_anon
4219206 ± 3% -18.7% 3430093 ± 3% +7.3% 4527044 ± 3% numa-vmstat.node1.numa_hit
4103183 ± 4% -19.0% 3324161 ± 4% -1.0% 4063850 ± 2% numa-vmstat.node1.numa_local
116023 ± 52% -8.7% 105932 ± 52% +62.0% 187903 ± 7% numa-vmstat.node1.numa_other
10.59 -7.6 2.97 ± 8% -0.0 10.58 turbostat.C1%
0.85 ± 3% +9.1 9.96 ± 2% +0.1 0.90 turbostat.C1E%
1.29 ± 6% +19.4% 1.54 ± 2% +7.8% 1.39 turbostat.CPU%c1
48.67 ± 2% -15.1% 41.33 ± 3% -14.7% 41.50 turbostat.CoreTmp
0.56 -60.7% 0.22 ± 3% +1.8% 0.57 turbostat.IPC
1.153e+08 -38.7% 70680365 -1.1% 1.14e+08 turbostat.IRQ
10242404 -14.8% 8723704 -0.2% 10218332 turbostat.NMI
88.65 -84.0 4.67 ± 33% -2.4 86.25 ± 2% turbostat.PKG_%
3.82 -3.8 0.04 ± 10% -0.1 3.68 turbostat.POLL%
48.67 ± 2% -13.7% 42.00 ± 3% -12.7% 42.50 turbostat.PkgTmp
683.77 -13.1% 594.00 -0.1% 683.24 turbostat.PkgWatt
18.74 -3.3% 18.13 -1.0% 18.54 turbostat.RAMWatt
2735312 ± 2% -21.0% 2160742 +0.5% 2749182 ± 2% proc-vmstat.nr_active_anon
204708 -1.6% 201435 -0.1% 204401 proc-vmstat.nr_anon_pages
3479812 ± 2% -16.4% 2908863 +0.4% 3493660 ± 2% proc-vmstat.nr_file_pages
154477 -12.0% 135959 -1.5% 152200 proc-vmstat.nr_mapped
2531568 ± 2% -22.6% 1960353 +0.6% 2545651 ± 3% proc-vmstat.nr_shmem
42010 -3.5% 40543 +0.0% 42030 proc-vmstat.nr_slab_reclaimable
2735312 ± 2% -21.0% 2160742 +0.5% 2749182 ± 2% proc-vmstat.nr_zone_active_anon
210167 ± 5% -11.5% 185950 ± 11% +7.6% 226044 ± 3% proc-vmstat.numa_hint_faults
198174 ± 6% -7.8% 182781 ± 11% +12.5% 222958 ± 4% proc-vmstat.numa_hint_faults_local
4730338 ± 2% -18.2% 3871343 +6.3% 5030657 ± 3% proc-vmstat.numa_hit
4498551 ± 2% -19.1% 3639783 +0.6% 4523449 ± 2% proc-vmstat.numa_local
22013 ± 74% -11.9% 19394 ± 79% -88.0% 2632 ± 2% proc-vmstat.numa_pages_migrated
4808959 ± 2% -17.8% 3954157 +0.3% 4821123 ± 2% proc-vmstat.pgalloc_normal
806619 -5.1% 765525 ± 2% +1.1% 815270 proc-vmstat.pgfault
467932 ± 3% -0.3% 466368 ± 2% -5.2% 443691 proc-vmstat.pgfree
22013 ± 74% -11.9% 19394 ± 79% -88.0% 2632 ± 2% proc-vmstat.pgmigrate_success
34098 ± 3% -14.8% 29054 -8.8% 31100 ± 4% proc-vmstat.pgreuse
0.11 +59.9% 0.17 ± 3% -3.8% 0.10 ± 2% perf-stat.i.MPKI
6.653e+10 -61.7% 2.546e+10 ± 2% +1.7% 6.767e+10 perf-stat.i.branch-instructions
0.76 +0.1 0.89 +0.0 0.78 perf-stat.i.branch-miss-rate%
4.685e+08 -59.7% 1.888e+08 ± 2% +4.8% 4.911e+08 perf-stat.i.branch-misses
1.12 +0.6 1.76 ± 3% +0.0 1.12 perf-stat.i.cache-miss-rate%
35553724 ± 3% -40.4% 21188697 -0.9% 35218701 perf-stat.i.cache-misses
4.194e+09 -68.3% 1.331e+09 ± 2% -1.3% 4.139e+09 perf-stat.i.cache-references
40710745 -69.6% 12395879 ± 2% -1.5% 40082554 perf-stat.i.context-switches
1.84 +189.1% 5.31 ± 2% -1.7% 1.81 perf-stat.i.cpi
5.965e+11 -2.0% 5.848e+11 -0.1% 5.962e+11 perf-stat.i.cpu-cycles
8813175 -64.5% 3125097 ± 2% -2.0% 8632551 perf-stat.i.cpu-migrations
24447 ± 3% +68.5% 41184 ± 2% +2.6% 25087 perf-stat.i.cycles-between-cache-misses
3.374e+11 -61.8% 1.287e+11 ± 2% +1.5% 3.426e+11 perf-stat.i.instructions
0.57 -60.8% 0.22 ± 2% +1.6% 0.58 perf-stat.i.ipc
0.01 ±141% +185.0% 0.04 ± 46% +280.6% 0.05 ± 20% perf-stat.i.major-faults
221.10 -68.6% 69.32 ± 2% -1.6% 217.48 perf-stat.i.metric.K/sec
11782 ± 3% -6.1% 11068 ± 3% -1.0% 11660 perf-stat.i.minor-faults
11782 ± 3% -6.1% 11068 ± 3% -1.0% 11660 perf-stat.i.page-faults
0.10 ± 2% +59.2% 0.17 ± 3% -3.0% 0.10 perf-stat.overall.MPKI
0.71 +0.0 0.75 +0.0 0.73 perf-stat.overall.branch-miss-rate%
0.83 ± 3% +0.7 1.56 ± 3% -0.0 0.82 perf-stat.overall.cache-miss-rate%
1.78 +162.2% 4.67 ± 2% -1.4% 1.76 perf-stat.overall.cpi
17181 ± 3% +64.6% 28283 +1.6% 17452 perf-stat.overall.cycles-between-cache-misses
0.56 -61.8% 0.21 ± 2% +1.4% 0.57 perf-stat.overall.ipc
6.388e+10 -62.3% 2.409e+10 ± 2% +1.6% 6.492e+10 perf-stat.ps.branch-instructions
4.538e+08 -60.0% 1.817e+08 ± 2% +5.0% 4.764e+08 perf-stat.ps.branch-misses
33674051 ± 3% -40.1% 20155290 -1.6% 33143601 perf-stat.ps.cache-misses
4.077e+09 -68.2% 1.296e+09 ± 2% -1.1% 4.032e+09 perf-stat.ps.cache-references
39570629 -69.5% 12072702 ± 2% -1.4% 39036286 perf-stat.ps.context-switches
5.78e+11 -1.4% 5.7e+11 +0.1% 5.784e+11 perf-stat.ps.cpu-cycles
8584979 -64.5% 3051930 ± 2% -1.8% 8430431 perf-stat.ps.cpu-migrations
3.243e+11 -62.4% 1.22e+11 ± 2% +1.5% 3.291e+11 perf-stat.ps.instructions
0.01 ±141% +195.1% 0.03 ± 41% +270.3% 0.04 ± 20% perf-stat.ps.major-faults
11022 ± 4% -6.5% 10300 ± 3% -0.1% 11015 perf-stat.ps.minor-faults
11022 ± 4% -6.5% 10300 ± 3% -0.1% 11015 perf-stat.ps.page-faults
1.941e+13 -61.9% 7.405e+12 ± 3% +2.5% 1.989e+13 perf-stat.total.instructions
18451 +9.9% 20272 +2.2% 18858 ± 4% sched_debug.cfs_rq:/.avg_vruntime.avg
5869 ± 4% -7.4% 5437 ± 5% +10.3% 6472 ± 19% sched_debug.cfs_rq:/.avg_vruntime.stddev
0.68 ± 2% -12.9% 0.59 ± 3% +0.6% 0.68 sched_debug.cfs_rq:/.h_nr_queued.stddev
0.62 ± 6% -12.9% 0.54 ± 2% +0.4% 0.62 sched_debug.cfs_rq:/.h_nr_runnable.stddev
8469 +12.7% 9544 ± 2% +3.0% 8727 ± 4% sched_debug.cfs_rq:/.left_deadline.stddev
8467 +12.7% 9542 ± 2% +3.1% 8727 ± 4% sched_debug.cfs_rq:/.left_vruntime.stddev
3513124 ± 25% -30.0% 2459550 ± 10% -2.7% 3419965 ± 22% sched_debug.cfs_rq:/.load.max
588329 ± 5% -11.2% 522578 ± 5% +4.1% 612170 sched_debug.cfs_rq:/.load.stddev
50699 ± 17% -19.8% 40655 ± 7% +20.0% 60854 ± 36% sched_debug.cfs_rq:/.load_avg.max
0.68 ± 2% -12.9% 0.59 ± 3% +0.9% 0.68 sched_debug.cfs_rq:/.nr_queued.stddev
38.80 ± 32% +108.5% 80.90 ± 16% +276.0% 145.88 ± 64% sched_debug.cfs_rq:/.removed.load_avg.avg
857.83 ± 12% +61.0% 1381 ± 12% +2561.3% 22829 ± 96% sched_debug.cfs_rq:/.removed.load_avg.max
152.02 ± 18% +57.2% 239.02 ± 11% +955.1% 1604 ± 88% sched_debug.cfs_rq:/.removed.load_avg.stddev
26.08 ± 28% +143.0% 63.37 ± 14% +23.5% 32.20 ± 9% sched_debug.cfs_rq:/.removed.runnable_avg.avg
547.00 ± 13% +88.7% 1032 ± 12% +6.4% 582.00 sched_debug.cfs_rq:/.removed.runnable_avg.max
94.86 ± 17% +84.3% 174.82 ± 9% +19.2% 113.04 ± 2% sched_debug.cfs_rq:/.removed.runnable_avg.stddev
9.09 ± 52% +253.3% 32.11 ± 17% +53.1% 13.91 ± 6% sched_debug.cfs_rq:/.removed.util_avg.avg
275.17 ± 3% +130.3% 633.67 ± 9% +0.0% 275.25 sched_debug.cfs_rq:/.removed.util_avg.max
44.90 ± 30% +126.4% 101.66 ± 11% +30.9% 58.78 ± 2% sched_debug.cfs_rq:/.removed.util_avg.stddev
8467 +12.7% 9542 ± 2% +3.1% 8727 ± 4% sched_debug.cfs_rq:/.right_vruntime.stddev
659.63 ± 3% +13.0% 745.47 +0.6% 663.51 sched_debug.cfs_rq:/.runnable_avg.avg
271.34 ± 2% +31.2% 355.98 ± 3% +0.8% 273.61 sched_debug.cfs_rq:/.runnable_avg.stddev
0.00 ± 26% +110.4% 0.00 ± 45% +3.3% 0.00 ± 21% sched_debug.cfs_rq:/.spread.avg
0.01 ± 13% +174.3% 0.02 ± 25% -36.7% 0.00 sched_debug.cfs_rq:/.spread.max
0.00 ± 7% +146.2% 0.00 ± 27% -13.2% 0.00 ± 8% sched_debug.cfs_rq:/.spread.stddev
431.00 +14.5% 493.62 -1.9% 422.73 sched_debug.cfs_rq:/.util_avg.avg
1061 ± 3% +26.4% 1341 ± 3% -3.5% 1024 sched_debug.cfs_rq:/.util_avg.max
151.53 ± 5% +50.1% 227.46 ± 2% -8.0% 139.45 sched_debug.cfs_rq:/.util_avg.stddev
206.96 +17.5% 243.18 ± 3% +10.4% 228.53 sched_debug.cfs_rq:/.util_est.avg
18451 +9.9% 20272 +2.2% 18858 ± 4% sched_debug.cfs_rq:/.zero_vruntime.avg
5869 ± 4% -7.4% 5437 ± 5% +10.3% 6472 ± 19% sched_debug.cfs_rq:/.zero_vruntime.stddev
460133 ± 2% +2.0% 469231 +5.0% 483144 ± 5% sched_debug.cpu.avg_idle.avg
2345 +33.6% 3133 ± 5% +887.0% 23149 ± 88% sched_debug.cpu.avg_idle.min
13.18 ± 2% +39.8% 18.42 ± 6% +5.1% 13.85 ± 5% sched_debug.cpu.clock.stddev
3961 +14.6% 4541 +0.7% 3989 sched_debug.cpu.curr->pid.avg
3213 -15.4% 2718 -5.7% 3030 sched_debug.cpu.curr->pid.stddev
0.00 ± 29% +157.3% 0.00 ± 35% -9.3% 0.00 ± 26% sched_debug.cpu.next_balance.stddev
0.70 -15.8% 0.59 ± 3% -2.7% 0.68 sched_debug.cpu.nr_running.stddev
5474800 -69.7% 1660250 ± 2% -1.5% 5392540 sched_debug.cpu.nr_switches.avg
5648642 -65.5% 1946319 ± 5% -1.3% 5576589 sched_debug.cpu.nr_switches.max
2229198 ± 8% -67.1% 734011 ± 20% +11.1% 2476939 ± 5% sched_debug.cpu.nr_switches.min
297592 ± 6% -25.9% 220513 ± 18% -5.6% 281029 ± 8% sched_debug.cpu.nr_switches.stddev
27.83 ± 30% -24.0% 21.17 ± 17% +33.8% 37.25 ± 14% sched_debug.cpu.nr_uninterruptible.max
23.75 -10.9 12.88 -0.3 23.42 perf-profile.calltrace.cycles-pp.common_startup_64
23.65 -10.8 12.82 -0.3 23.32 perf-profile.calltrace.cycles-pp.start_secondary.common_startup_64
23.62 -10.8 12.81 -0.3 23.28 perf-profile.calltrace.cycles-pp.cpu_startup_entry.start_secondary.common_startup_64
23.51 -10.8 12.76 -0.3 23.18 perf-profile.calltrace.cycles-pp.do_idle.cpu_startup_entry.start_secondary.common_startup_64
12.93 -7.0 5.94 ± 4% -0.1 12.82 perf-profile.calltrace.cycles-pp.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
12.78 -6.9 5.89 ± 4% -0.1 12.66 perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
11.30 -5.2 6.07 ± 3% -0.1 11.22 perf-profile.calltrace.cycles-pp.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
11.17 -5.2 6.02 ± 3% -0.1 11.10 perf-profile.calltrace.cycles-pp.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
9.89 -4.8 5.08 ± 4% -0.1 9.76 perf-profile.calltrace.cycles-pp.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q
4.52 -4.5 0.00 -0.1 4.39 perf-profile.calltrace.cycles-pp.poll_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
12.41 -4.4 7.96 -0.2 12.20 perf-profile.calltrace.cycles-pp.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary.common_startup_64
9.19 -4.4 4.77 ± 4% -0.1 9.06 perf-profile.calltrace.cycles-pp.select_idle_cpu.select_idle_sibling.select_task_rq_fair.select_task_rq.try_to_wake_up
11.29 -4.0 7.29 -0.2 11.11 perf-profile.calltrace.cycles-pp.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry
11.38 -4.0 7.40 -0.2 11.20 perf-profile.calltrace.cycles-pp.cpuidle_enter.cpuidle_idle_call.do_idle.cpu_startup_entry.start_secondary
8.04 -3.9 4.13 ± 4% -0.1 7.93 perf-profile.calltrace.cycles-pp.select_idle_core.select_idle_cpu.select_idle_sibling.select_task_rq_fair.select_task_rq
6.91 -3.8 3.08 ± 4% -0.1 6.80 perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive
6.76 -3.8 3.00 ± 4% -0.1 6.66 perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedreceive
8.71 -3.7 5.00 -0.0 8.67 perf-profile.calltrace.cycles-pp.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
5.71 -3.4 2.26 ± 4% -0.1 5.64 perf-profile.calltrace.cycles-pp.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary.common_startup_64
8.12 -3.4 4.72 -0.0 8.08 perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
7.76 -3.2 4.55 -0.0 7.72 perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend.__x64_sys_mq_timedsend
8.39 -3.1 5.26 -0.0 8.35 perf-profile.calltrace.cycles-pp.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
7.47 -3.1 4.35 -0.0 7.44 perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedsend
4.92 -2.9 2.00 ± 4% -0.1 4.86 perf-profile.calltrace.cycles-pp.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry.start_secondary
7.79 -2.8 4.97 -0.0 7.74 perf-profile.calltrace.cycles-pp.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
7.48 -2.7 4.79 -0.0 7.44 perf-profile.calltrace.cycles-pp.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive.__x64_sys_mq_timedreceive
7.17 -2.6 4.60 -0.0 7.13 perf-profile.calltrace.cycles-pp.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep.do_mq_timedreceive
5.54 -2.3 3.24 ± 4% -0.1 5.48 perf-profile.calltrace.cycles-pp.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend
5.41 -2.2 3.16 ± 4% -0.1 5.35 perf-profile.calltrace.cycles-pp.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q.do_mq_timedsend
3.88 -2.2 1.67 ± 4% -0.1 3.82 perf-profile.calltrace.cycles-pp.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle.cpu_startup_entry
4.04 -2.2 1.88 -0.0 3.99 perf-profile.calltrace.cycles-pp.schedule_idle.do_idle.cpu_startup_entry.start_secondary.common_startup_64
3.44 -2.2 1.28 ± 3% -0.1 3.38 perf-profile.calltrace.cycles-pp.store_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.84 -2.0 1.80 -0.0 3.79 perf-profile.calltrace.cycles-pp.__schedule.schedule_idle.do_idle.cpu_startup_entry.start_secondary
4.83 -2.0 2.85 -0.0 4.80 perf-profile.calltrace.cycles-pp.try_to_block_task.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep
4.71 -1.9 2.78 -0.0 4.68 perf-profile.calltrace.cycles-pp.dequeue_task_fair.try_to_block_task.__schedule.schedule.schedule_hrtimeout_range_clock
1.84 -1.8 0.00 -0.0 1.81 perf-profile.calltrace.cycles-pp.wake_affine.select_task_rq_fair.select_task_rq.try_to_wake_up.wake_up_q
4.39 -1.8 2.58 -0.0 4.36 perf-profile.calltrace.cycles-pp.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule.schedule
2.37 -1.5 0.84 ± 3% -0.0 2.34 perf-profile.calltrace.cycles-pp._copy_to_user.store_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
2.66 -1.4 1.26 ± 4% -0.0 2.62 perf-profile.calltrace.cycles-pp.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue.do_idle
2.73 -1.4 1.33 ± 4% -0.0 2.70 perf-profile.calltrace.cycles-pp.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
3.50 -1.3 2.17 -0.0 3.49 perf-profile.calltrace.cycles-pp.dequeue_entity.dequeue_entities.dequeue_task_fair.try_to_block_task.__schedule
2.37 -1.3 1.06 ± 2% -0.0 2.32 perf-profile.calltrace.cycles-pp.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.23 -1.2 0.00 -0.0 1.20 perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule_idle.do_idle.cpu_startup_entry
1.15 -1.2 0.00 -0.0 1.13 perf-profile.calltrace.cycles-pp.task_h_load.wake_affine.select_task_rq_fair.select_task_rq.try_to_wake_up
1.15 -1.1 0.00 -0.0 1.12 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule_idle.do_idle
4.10 -1.1 3.03 +0.0 4.10 perf-profile.calltrace.cycles-pp.__pick_next_task.__schedule.schedule.schedule_hrtimeout_range_clock.wq_sleep
2.20 -1.1 1.13 ± 5% -0.0 2.16 perf-profile.calltrace.cycles-pp.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue.flush_smp_call_function_queue
2.20 -1.1 1.15 ± 4% -0.0 2.18 perf-profile.calltrace.cycles-pp.switch_fpu_return.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.05 -1.0 0.00 +0.0 1.06 perf-profile.calltrace.cycles-pp.set_next_task_idle.__pick_next_task.__schedule.schedule.schedule_hrtimeout_range_clock
1.21 -1.0 0.18 ±141% -0.0 1.20 perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend
1.20 -1.0 0.17 ±141% -0.0 1.19 perf-profile.calltrace.cycles-pp.do_perf_trace_sched_wakeup_template.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedsend
1.74 -1.0 0.73 ± 3% +0.0 1.77 perf-profile.calltrace.cycles-pp.msg_get.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.99 -1.0 0.00 +0.0 1.00 perf-profile.calltrace.cycles-pp.schedule.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.98 -1.0 0.00 +0.0 1.00 perf-profile.calltrace.cycles-pp.__schedule.schedule.exit_to_user_mode_loop.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.32 -1.0 0.37 ± 70% -0.0 1.32 perf-profile.calltrace.cycles-pp.do_perf_trace_sched_wakeup_template.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedreceive
1.93 -0.9 0.99 ± 4% -0.0 1.89 perf-profile.calltrace.cycles-pp.enqueue_task_fair.enqueue_task.ttwu_do_activate.sched_ttwu_pending.__flush_smp_call_function_queue
1.34 -0.8 0.54 ± 5% -0.0 1.34 perf-profile.calltrace.cycles-pp.perf_trace_sched_wakeup_template.try_to_wake_up.wake_up_q.do_mq_timedreceive.__x64_sys_mq_timedreceive
1.52 -0.8 0.73 ± 4% -0.0 1.50 perf-profile.calltrace.cycles-pp.enqueue_entity.enqueue_task_fair.enqueue_task.ttwu_do_activate.sched_ttwu_pending
0.74 -0.7 0.00 -0.0 0.73 perf-profile.calltrace.cycles-pp.__smp_call_single_queue.ttwu_queue_wakelist.try_to_wake_up.wake_up_q.do_mq_timedsend
1.05 -0.7 0.34 ± 70% -0.0 1.05 perf-profile.calltrace.cycles-pp.__switch_to
0.71 -0.7 0.00 -0.0 0.68 perf-profile.calltrace.cycles-pp.msg_insert.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
1.57 -0.7 0.90 ± 6% -0.0 1.56 perf-profile.calltrace.cycles-pp.restore_fpregs_from_fpstate.switch_fpu_return.arch_exit_to_user_mode_prepare.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.67 -0.7 0.00 -0.0 0.64 perf-profile.calltrace.cycles-pp.__wake_up.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.66 -0.7 0.00 -0.0 0.65 perf-profile.calltrace.cycles-pp.update_load_avg.enqueue_entity.enqueue_task_fair.enqueue_task.ttwu_do_activate
0.65 -0.6 0.00 -0.0 0.64 perf-profile.calltrace.cycles-pp.___task_rq_lock.try_to_wake_up.wake_up_q.do_mq_timedsend.__x64_sys_mq_timedsend
0.60 -0.6 0.00 -0.0 0.58 perf-profile.calltrace.cycles-pp._raw_spin_lock_irqsave.__wake_up.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
0.56 -0.6 0.00 -0.0 0.55 perf-profile.calltrace.cycles-pp._raw_spin_lock.raw_spin_rq_lock_nested.___task_rq_lock.try_to_wake_up.wake_up_q
0.54 -0.5 0.00 -0.0 0.53 perf-profile.calltrace.cycles-pp.os_xsave
0.52 -0.5 0.00 -0.0 0.51 perf-profile.calltrace.cycles-pp.__switch_to_asm
2.56 -0.3 2.28 -0.0 2.54 perf-profile.calltrace.cycles-pp.pick_next_task_fair.__pick_next_task.__schedule.schedule.schedule_hrtimeout_range_clock
0.00 +0.0 0.00 +0.5 0.50 perf-profile.calltrace.cycles-pp.__check_object_size.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
5.87 +0.9 6.78 -0.0 5.86 perf-profile.calltrace.cycles-pp.intel_idle.cpuidle_enter_state.cpuidle_enter.cpuidle_idle_call.do_idle
0.00 +0.9 0.91 ± 30% +0.0 0.00 perf-profile.calltrace.cycles-pp.sched_balance_newidle.pick_next_task_fair.__pick_next_task.__schedule.schedule
35.80 +3.5 39.29 +0.1 35.92 perf-profile.calltrace.cycles-pp.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
35.59 +3.6 39.21 +0.1 35.70 perf-profile.calltrace.cycles-pp.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.00 +4.4 4.37 ± 3% +0.0 0.00 perf-profile.calltrace.cycles-pp.drain_obj_stock.__refill_obj_stock.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg
0.00 +4.4 4.39 ± 4% +0.0 0.00 perf-profile.calltrace.cycles-pp.drain_obj_stock.__refill_obj_stock.__memcg_slab_free_hook.kfree.free_msg
0.00 +8.0 8.01 ± 4% +0.0 0.00 perf-profile.calltrace.cycles-pp.__refill_obj_stock.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg.do_mq_timedsend
0.00 +8.3 8.29 ± 4% +0.0 0.00 perf-profile.calltrace.cycles-pp.__refill_obj_stock.__memcg_slab_free_hook.kfree.free_msg.do_mq_timedreceive
28.51 +13.5 41.99 +0.3 28.81 perf-profile.calltrace.cycles-pp.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
28.23 +13.5 41.71 +0.3 28.53 perf-profile.calltrace.cycles-pp.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
70.69 +13.7 84.35 +0.3 71.02 perf-profile.calltrace.cycles-pp.entry_SYSCALL_64_after_hwframe
70.44 +13.8 84.26 +0.3 70.77 perf-profile.calltrace.cycles-pp.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.99 +20.2 23.24 ± 2% +0.4 3.35 perf-profile.calltrace.cycles-pp.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64.entry_SYSCALL_64_after_hwframe
2.79 +20.4 23.15 ± 2% +0.4 3.16 perf-profile.calltrace.cycles-pp.kfree.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive.do_syscall_64
2.43 +20.6 22.98 ± 2% +0.4 2.80 perf-profile.calltrace.cycles-pp.__memcg_slab_free_hook.kfree.free_msg.do_mq_timedreceive.__x64_sys_mq_timedreceive
2.26 +26.0 28.23 ± 2% +0.6 2.90 perf-profile.calltrace.cycles-pp.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64.entry_SYSCALL_64_after_hwframe
0.99 +26.8 27.80 ± 2% +0.6 1.60 perf-profile.calltrace.cycles-pp.__kmalloc_node_noprof.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend.do_syscall_64
0.65 +27.0 27.62 ± 2% +0.6 1.24 perf-profile.calltrace.cycles-pp.__memcg_slab_post_alloc_hook.__kmalloc_node_noprof.load_msg.do_mq_timedsend.__x64_sys_mq_timedsend
24.25 -12.2 12.02 ± 4% -0.2 24.05 perf-profile.children.cycles-pp.wake_up_q
24.00 -12.1 11.93 ± 4% -0.2 23.81 perf-profile.children.cycles-pp.try_to_wake_up
23.75 -10.9 12.88 -0.3 23.42 perf-profile.children.cycles-pp.common_startup_64
23.75 -10.9 12.88 -0.3 23.42 perf-profile.children.cycles-pp.cpu_startup_entry
23.68 -10.8 12.85 -0.3 23.34 perf-profile.children.cycles-pp.do_idle
23.65 -10.8 12.82 -0.3 23.32 perf-profile.children.cycles-pp.start_secondary
19.65 -8.3 11.33 -0.1 19.54 perf-profile.children.cycles-pp.__schedule
17.14 -6.9 10.28 -0.1 17.06 perf-profile.children.cycles-pp.wq_sleep
16.24 -6.4 9.82 -0.1 16.18 perf-profile.children.cycles-pp.schedule
15.92 -6.2 9.69 -0.1 15.84 perf-profile.children.cycles-pp.schedule_hrtimeout_range_clock
12.46 -6.1 6.32 ± 4% -0.2 12.30 perf-profile.children.cycles-pp.select_task_rq
12.19 -6.0 6.17 ± 4% -0.2 12.03 perf-profile.children.cycles-pp.select_task_rq_fair
9.91 -4.8 5.09 ± 4% -0.1 9.78 perf-profile.children.cycles-pp.select_idle_sibling
4.57 -4.6 0.02 ±141% -0.1 4.44 perf-profile.children.cycles-pp.poll_idle
9.27 -4.5 4.80 ± 4% -0.1 9.14 perf-profile.children.cycles-pp.select_idle_cpu
12.49 -4.5 8.03 -0.2 12.28 perf-profile.children.cycles-pp.cpuidle_idle_call
11.41 -4.0 7.39 -0.2 11.23 perf-profile.children.cycles-pp.cpuidle_enter_state
11.44 -4.0 7.43 -0.2 11.26 perf-profile.children.cycles-pp.cpuidle_enter
8.17 -4.0 4.18 ± 4% -0.1 8.06 perf-profile.children.cycles-pp.select_idle_core
5.76 -3.5 2.28 ± 4% -0.1 5.70 perf-profile.children.cycles-pp.flush_smp_call_function_queue
5.47 -3.1 2.35 ± 4% -0.1 5.40 perf-profile.children.cycles-pp.__flush_smp_call_function_queue
4.30 -2.4 1.94 ± 4% -0.1 4.24 perf-profile.children.cycles-pp.sched_ttwu_pending
4.08 -2.2 1.90 -0.0 4.03 perf-profile.children.cycles-pp.schedule_idle
3.47 -2.2 1.30 ± 2% -0.1 3.42 perf-profile.children.cycles-pp.store_msg
5.87 -2.1 3.78 -0.0 5.86 perf-profile.children.cycles-pp.__pick_next_task
4.84 -2.0 2.86 -0.0 4.81 perf-profile.children.cycles-pp.try_to_block_task
4.73 -1.9 2.79 -0.0 4.69 perf-profile.children.cycles-pp.dequeue_entities
4.73 -1.9 2.82 -0.0 4.70 perf-profile.children.cycles-pp.dequeue_task_fair
4.04 -1.8 2.26 -0.0 4.00 perf-profile.children.cycles-pp._raw_spin_lock
3.72 -1.7 2.03 ± 4% -0.0 3.70 perf-profile.children.cycles-pp.enqueue_task
2.40 -1.6 0.85 ± 3% -0.0 2.37 perf-profile.children.cycles-pp._copy_to_user
3.39 -1.5 1.86 ± 3% -0.0 3.36 perf-profile.children.cycles-pp.ttwu_do_activate
2.56 -1.5 1.04 ± 5% -0.0 2.54 perf-profile.children.cycles-pp.perf_trace_sched_wakeup_template
2.54 -1.5 1.03 ± 5% -0.0 2.52 perf-profile.children.cycles-pp.do_perf_trace_sched_wakeup_template
2.76 -1.4 1.34 ± 4% -0.0 2.73 perf-profile.children.cycles-pp.arch_exit_to_user_mode_prepare
3.75 -1.4 2.34 -0.0 3.74 perf-profile.children.cycles-pp.dequeue_entity
2.32 -1.4 0.95 ± 4% -0.0 2.30 perf-profile.children.cycles-pp.ttwu_queue_wakelist
2.72 -1.3 1.38 ± 2% -0.0 2.72 perf-profile.children.cycles-pp.update_curr
2.38 -1.3 1.06 ± 2% -0.1 2.32 perf-profile.children.cycles-pp.exit_to_user_mode_loop
4.24 -1.2 2.99 -0.0 4.21 perf-profile.children.cycles-pp.pick_next_task_fair
2.21 -1.1 1.15 ± 5% -0.0 2.19 perf-profile.children.cycles-pp.switch_fpu_return
1.76 -1.0 0.73 ± 3% +0.0 1.80 perf-profile.children.cycles-pp.msg_get
1.67 -1.0 0.65 ± 3% -0.1 1.54 perf-profile.children.cycles-pp._raw_spin_lock_irqsave
2.47 -1.0 1.47 ± 3% -0.0 2.44 perf-profile.children.cycles-pp.enqueue_task_fair
2.29 -1.0 1.32 -0.0 2.25 perf-profile.children.cycles-pp.update_load_avg
2.39 -1.0 1.42 -0.0 2.37 perf-profile.children.cycles-pp.raw_spin_rq_lock_nested
1.60 -1.0 0.64 -0.0 1.59 perf-profile.children.cycles-pp.switch_mm_irqs_off
1.51 -0.9 0.57 ± 5% -0.0 1.49 perf-profile.children.cycles-pp.__smp_call_single_queue
1.84 -0.9 0.93 ± 6% -0.0 1.82 perf-profile.children.cycles-pp.wake_affine
1.49 -0.9 0.59 +0.0 1.49 perf-profile.children.cycles-pp.__check_object_size
1.61 -0.9 0.72 ± 5% +0.0 1.62 perf-profile.children.cycles-pp.update_rq_clock_task
1.73 -0.9 0.87 ± 2% -0.0 1.73 perf-profile.children.cycles-pp.update_se
1.51 -0.9 0.65 ± 2% +0.0 1.53 perf-profile.children.cycles-pp.wakeup_preempt
2.00 -0.8 1.15 ± 3% -0.0 1.97 perf-profile.children.cycles-pp.enqueue_entity
1.26 -0.8 0.42 ± 3% -0.1 1.14 perf-profile.children.cycles-pp.__wake_up
1.31 -0.7 0.58 ± 5% -0.0 1.30 perf-profile.children.cycles-pp.set_task_cpu
1.15 -0.7 0.45 ± 3% -0.0 1.10 perf-profile.children.cycles-pp.msg_insert
1.04 -0.7 0.35 ± 5% -0.0 1.04 perf-profile.children.cycles-pp.perf_trace_buf_alloc
1.57 -0.7 0.90 ± 5% -0.0 1.56 perf-profile.children.cycles-pp.restore_fpregs_from_fpstate
1.00 -0.7 0.34 ± 7% -0.0 1.00 perf-profile.children.cycles-pp.perf_swevent_get_recursion_context
0.95 -0.7 0.29 ± 4% -0.0 0.94 perf-profile.children.cycles-pp.llist_reverse_order
1.14 -0.6 0.50 ± 5% -0.0 1.13 perf-profile.children.cycles-pp.migrate_task_rq_fair
1.11 -0.6 0.51 -0.0 1.10 perf-profile.children.cycles-pp.set_next_entity
0.95 -0.6 0.39 ± 2% +0.0 0.96 perf-profile.children.cycles-pp.__update_idle_core
1.04 -0.6 0.49 ± 2% -0.0 1.03 perf-profile.children.cycles-pp.pick_task_fair
1.15 -0.6 0.60 ± 7% -0.0 1.14 perf-profile.children.cycles-pp.task_h_load
0.84 -0.5 0.30 ± 5% -0.0 0.84 perf-profile.children.cycles-pp.call_function_single_prep_ipi
1.06 -0.5 0.52 +0.0 1.07 perf-profile.children.cycles-pp.set_next_task_idle
1.04 -0.5 0.51 -0.0 1.03 perf-profile.children.cycles-pp._find_next_bit
1.38 -0.5 0.85 -0.0 1.36 perf-profile.children.cycles-pp.update_cfs_rq_load_avg
1.28 -0.5 0.75 -0.0 1.27 perf-profile.children.cycles-pp.__switch_to
0.85 -0.5 0.34 ± 3% +0.0 0.86 perf-profile.children.cycles-pp.cpuacct_charge
0.77 ± 4% -0.5 0.25 -0.0 0.73 perf-profile.children.cycles-pp.__bitmap_andnot
0.88 -0.5 0.41 ± 6% +0.0 0.89 perf-profile.children.cycles-pp.update_entity_lag
0.94 -0.5 0.48 -0.0 0.92 perf-profile.children.cycles-pp.prepare_task_switch
0.74 -0.4 0.31 ± 2% +0.0 0.76 perf-profile.children.cycles-pp.check_heap_object
0.75 -0.4 0.32 ± 5% +0.0 0.75 perf-profile.children.cycles-pp.requeue_delayed_entity
0.80 -0.4 0.40 -0.0 0.80 perf-profile.children.cycles-pp.wakeup_preempt_fair
0.55 -0.4 0.16 ± 2% -0.0 0.54 perf-profile.children.cycles-pp.native_sched_clock
0.57 -0.4 0.18 ± 4% -0.0 0.56 perf-profile.children.cycles-pp.entry_SYSRETQ_unsafe_stack
0.55 -0.4 0.17 -0.0 0.54 perf-profile.children.cycles-pp.os_xsave
0.58 -0.4 0.20 -0.0 0.57 perf-profile.children.cycles-pp.sched_clock_cpu
1.39 -0.4 1.01 -0.0 1.36 perf-profile.children.cycles-pp.native_queued_spin_lock_slowpath
1.18 -0.4 0.81 -0.0 1.18 perf-profile.children.cycles-pp.___task_rq_lock
0.51 ± 3% -0.4 0.14 ± 3% -0.0 0.50 perf-profile.children.cycles-pp._copy_from_user
0.56 -0.4 0.19 ± 2% -0.0 0.54 perf-profile.children.cycles-pp.update_rq_clock
0.51 -0.3 0.17 ± 2% -0.0 0.50 perf-profile.children.cycles-pp.sched_clock
0.56 -0.3 0.23 ± 3% +0.0 0.58 perf-profile.children.cycles-pp.simple_inode_init_ts
0.89 -0.3 0.56 +0.0 0.89 perf-profile.children.cycles-pp.put_prev_entity
0.53 ± 3% -0.3 0.21 ± 2% +0.0 0.53 perf-profile.children.cycles-pp.__put_user_4
0.68 ± 12% -0.3 0.36 ± 4% +0.1 0.76 perf-profile.children.cycles-pp.stress_switch_mq
0.52 -0.3 0.20 -0.0 0.50 perf-profile.children.cycles-pp.set_next_task_fair
0.55 -0.3 0.24 ± 5% -0.0 0.54 perf-profile.children.cycles-pp.__switch_to_asm
0.51 -0.3 0.21 ± 3% +0.0 0.52 perf-profile.children.cycles-pp.inode_set_ctime_current
0.56 -0.3 0.26 ± 3% +0.0 0.57 perf-profile.children.cycles-pp.fdget
0.52 -0.3 0.23 ± 3% -0.0 0.51 perf-profile.children.cycles-pp.mm_cid_switch_to
0.54 -0.3 0.25 ± 3% -0.0 0.54 perf-profile.children.cycles-pp.remove_entity_load_avg
0.79 -0.3 0.51 -0.0 0.78 perf-profile.children.cycles-pp.asm_sysvec_call_function_single
0.37 -0.3 0.11 ± 7% +0.0 0.37 perf-profile.children.cycles-pp.__resched_curr
0.57 -0.2 0.32 -0.0 0.56 perf-profile.children.cycles-pp.__update_load_avg_cfs_rq
0.35 -0.2 0.12 ± 4% -0.0 0.35 perf-profile.children.cycles-pp.avg_vruntime
0.62 -0.2 0.39 ± 2% -0.0 0.62 perf-profile.children.cycles-pp.sysvec_call_function_single
0.34 -0.2 0.11 -0.0 0.33 perf-profile.children.cycles-pp.entry_SYSCALL_64
0.58 -0.2 0.36 +0.0 0.59 perf-profile.children.cycles-pp.__enqueue_entity
0.46 -0.2 0.24 -0.0 0.45 perf-profile.children.cycles-pp.__pick_eevdf
0.52 -0.2 0.30 ± 3% +0.0 0.52 perf-profile.children.cycles-pp.perf_tp_event
0.35 -0.2 0.14 ± 3% -0.0 0.35 perf-profile.children.cycles-pp.__wrgsbase_inactive
0.56 -0.2 0.36 -0.0 0.56 perf-profile.children.cycles-pp.__sysvec_call_function_single
0.33 -0.2 0.13 ± 3% -0.0 0.32 perf-profile.children.cycles-pp.__update_load_avg_se
0.31 ± 2% -0.2 0.12 ± 4% +0.0 0.33 perf-profile.children.cycles-pp.__virt_addr_valid
0.58 -0.2 0.40 -0.0 0.58 perf-profile.children.cycles-pp.menu_select
0.32 ± 5% -0.2 0.13 -0.0 0.29 ± 3% perf-profile.children.cycles-pp.__check_heap_object
0.31 -0.2 0.13 ± 3% -0.0 0.30 perf-profile.children.cycles-pp.place_entity
0.32 -0.2 0.14 -0.0 0.32 perf-profile.children.cycles-pp.tick_nohz_idle_enter
0.36 -0.2 0.18 ± 2% -0.0 0.36 perf-profile.children.cycles-pp.perf_trace_sched_stat_runtime
0.29 -0.2 0.11 -0.0 0.28 perf-profile.children.cycles-pp.syscall_return_via_sysret
0.33 -0.2 0.16 ± 3% -0.0 0.32 perf-profile.children.cycles-pp.do_perf_trace_sched_stat_runtime
0.35 -0.2 0.18 ± 2% +0.0 0.36 perf-profile.children.cycles-pp.ktime_get
0.25 -0.2 0.08 ± 5% -0.0 0.25 perf-profile.children.cycles-pp.entry_SYSCALL_64_safe_stack
0.37 -0.2 0.20 ± 4% -0.0 0.36 perf-profile.children.cycles-pp.___perf_sw_event
0.23 -0.2 0.07 ± 6% -0.0 0.22 ± 2% perf-profile.children.cycles-pp.tick_nohz_idle_exit
0.22 -0.2 0.07 -0.0 0.22 ± 2% perf-profile.children.cycles-pp.read_tsc
0.21 -0.1 0.06 ± 7% -0.0 0.20 ± 2% perf-profile.children.cycles-pp.__rdgsbase_inactive
0.25 -0.1 0.11 ± 7% -0.0 0.25 perf-profile.children.cycles-pp.strnlen
0.16 -0.1 0.02 ±141% -0.0 0.15 perf-profile.children.cycles-pp.ct_idle_exit
0.32 -0.1 0.18 -0.0 0.31 perf-profile.children.cycles-pp.__dequeue_entity
0.23 ± 2% -0.1 0.10 ± 4% +0.0 0.24 ± 2% perf-profile.children.cycles-pp.check_stack_object
0.53 -0.1 0.40 ± 2% -0.0 0.52 perf-profile.children.cycles-pp.sysvec_apic_timer_interrupt
0.23 ± 2% -0.1 0.09 ± 5% -0.0 0.22 ± 2% perf-profile.children.cycles-pp._raw_spin_unlock_irqrestore
0.30 ± 3% -0.1 0.17 ± 4% -0.0 0.30 perf-profile.children.cycles-pp.tick_nohz_handler
0.20 -0.1 0.07 +0.4 0.57 ± 3% perf-profile.children.cycles-pp.__account_obj_stock
0.13 -0.1 0.00 -0.0 0.12 perf-profile.children.cycles-pp.ct_kernel_enter
0.13 -0.1 0.00 -0.0 0.12 perf-profile.children.cycles-pp.ct_kernel_exit_state
0.41 -0.1 0.28 ± 2% -0.0 0.40 perf-profile.children.cycles-pp.hrtimer_interrupt
0.31 ± 2% -0.1 0.18 ± 2% -0.0 0.30 perf-profile.children.cycles-pp.__hrtimer_run_queues
0.41 ± 2% -0.1 0.29 ± 3% -0.0 0.41 perf-profile.children.cycles-pp.__sysvec_apic_timer_interrupt
0.28 ± 2% -0.1 0.16 ± 6% -0.0 0.27 perf-profile.children.cycles-pp.update_process_times
0.33 -0.1 0.21 ± 6% +0.0 0.33 perf-profile.children.cycles-pp.attach_entity_load_avg
0.12 ± 3% -0.1 0.00 -0.0 0.08 perf-profile.children.cycles-pp.__do_notify
0.19 ± 2% -0.1 0.07 ± 7% -0.0 0.18 perf-profile.children.cycles-pp.wake_q_add_safe
0.23 ± 2% -0.1 0.11 -0.0 0.22 perf-profile.children.cycles-pp.__kmalloc_cache_noprof
0.18 -0.1 0.07 ± 6% -0.0 0.18 perf-profile.children.cycles-pp.nohz_run_idle_balance
0.56 -0.1 0.46 -0.0 0.56 perf-profile.children.cycles-pp.asm_sysvec_apic_timer_interrupt
0.15 -0.1 0.05 +0.0 0.15 perf-profile.children.cycles-pp._raw_spin_unlock
0.17 -0.1 0.08 -0.0 0.16 perf-profile.children.cycles-pp.security_msg_msg_free
0.15 -0.1 0.06 +0.0 0.16 perf-profile.children.cycles-pp.inode_set_ctime_to_ts
0.08 ± 5% -0.1 0.00 +0.0 0.11 perf-profile.children.cycles-pp.trylock_stock
0.16 ± 2% -0.1 0.08 ± 5% -0.0 0.16 perf-profile.children.cycles-pp.dl_server_update
0.13 -0.1 0.06 ± 8% +0.0 0.13 perf-profile.children.cycles-pp.timestamp_truncate
0.15 ± 3% -0.1 0.08 +0.0 0.17 perf-profile.children.cycles-pp.perf_trace_buf_update
0.13 ± 3% -0.1 0.06 +0.0 0.13 perf-profile.children.cycles-pp.ktime_get_coarse_real_ts64_mg
0.13 ± 3% -0.1 0.06 +0.0 0.13 perf-profile.children.cycles-pp.migrate_disable_switch
0.12 ± 6% -0.1 0.05 ± 8% -0.0 0.12 ± 4% perf-profile.children.cycles-pp.__cgroup_account_cputime
0.11 ± 4% -0.1 0.05 -0.0 0.11 perf-profile.children.cycles-pp.cpuidle_governor_latency_req
0.06 -0.1 0.00 +0.0 0.07 perf-profile.children.cycles-pp.raw_spin_rq_unlock
0.14 -0.1 0.08 ± 5% +0.0 0.14 perf-profile.children.cycles-pp.update_curr_dl_se
0.06 ± 16% -0.1 0.00 +0.1 0.12 ± 4% perf-profile.children.cycles-pp.css_rstat_updated
0.10 -0.1 0.05 -0.0 0.10 perf-profile.children.cycles-pp.ct_kernel_exit
0.11 -0.1 0.06 +0.0 0.11 perf-profile.children.cycles-pp.tracing_gen_ctx_irq_test
0.10 ± 4% -0.0 0.05 +0.0 0.10 perf-profile.children.cycles-pp.__rb_insert_augmented
0.10 -0.0 0.06 ± 8% -0.0 0.10 perf-profile.children.cycles-pp.rest_init
0.10 -0.0 0.06 ± 8% -0.0 0.10 perf-profile.children.cycles-pp.start_kernel
0.10 -0.0 0.06 ± 8% -0.0 0.10 perf-profile.children.cycles-pp.x86_64_start_kernel
0.10 -0.0 0.06 ± 8% -0.0 0.10 perf-profile.children.cycles-pp.x86_64_start_reservations
0.08 ± 10% -0.0 0.04 ± 71% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.mq_timedreceive
0.15 -0.0 0.11 ± 4% +0.0 0.15 perf-profile.children.cycles-pp.vruntime_eligible
0.13 -0.0 0.09 +0.0 0.13 perf-profile.children.cycles-pp.put_prev_task_fair
0.09 -0.0 0.05 ± 8% -0.0 0.09 perf-profile.children.cycles-pp.native_irq_return_iret
0.08 -0.0 0.05 +0.0 0.08 perf-profile.children.cycles-pp.choose_new_asid
0.13 -0.0 0.11 +0.0 0.13 perf-profile.children.cycles-pp.__irq_exit_rcu
0.07 -0.0 0.05 +0.0 0.07 perf-profile.children.cycles-pp.__set_next_task_fair
0.76 -0.0 0.74 -0.0 0.74 perf-profile.children.cycles-pp.finish_task_switch
0.09 -0.0 0.07 ± 6% -0.0 0.09 perf-profile.children.cycles-pp.propagate_entity_load_avg
0.10 -0.0 0.09 -0.0 0.10 perf-profile.children.cycles-pp.handle_softirqs
0.07 -0.0 0.06 +0.0 0.08 perf-profile.children.cycles-pp.clockevents_program_event
0.00 +0.0 0.00 +0.1 0.14 ± 3% perf-profile.children.cycles-pp.__mod_memcg_state
0.00 +0.0 0.00 +0.1 0.14 ± 3% perf-profile.children.cycles-pp.try_charge_memcg
0.00 +0.0 0.00 +0.3 0.32 ± 4% perf-profile.children.cycles-pp.__mod_memcg_lruvec_state
0.07 +0.0 0.08 +0.0 0.07 perf-profile.children.cycles-pp.perf_swevent_event
0.48 +0.0 0.49 +0.0 0.48 perf-profile.children.cycles-pp.process_simple
0.05 +0.0 0.06 ± 7% -0.0 0.05 perf-profile.children.cycles-pp.sched_update_worker
0.05 +0.0 0.07 -0.0 0.05 perf-profile.children.cycles-pp.arch_cpu_idle_enter
0.07 ± 11% +0.0 0.09 ± 5% +0.0 0.12 ± 4% perf-profile.children.cycles-pp.mq_timedsend
0.15 ± 3% +0.0 0.20 ± 4% -0.0 0.15 perf-profile.children.cycles-pp.x64_sys_call
0.00 +0.1 0.05 +0.0 0.00 perf-profile.children.cycles-pp.__sched_balance_update_blocked_averages
0.00 +0.1 0.05 +0.0 0.00 perf-profile.children.cycles-pp.update_cfs_group
0.00 +0.1 0.06 ± 23% +0.0 0.00 perf-profile.children.cycles-pp.generic_perform_write
0.00 +0.1 0.06 ± 7% +0.0 0.00 perf-profile.children.cycles-pp.detach_tasks
0.00 +0.1 0.06 ± 29% +0.0 0.00 perf-profile.children.cycles-pp.shmem_file_write_iter
0.00 +0.1 0.06 ± 29% +0.0 0.00 perf-profile.children.cycles-pp.vfs_write
0.00 +0.1 0.07 ± 25% +0.0 0.00 perf-profile.children.cycles-pp.ksys_write
0.00 +0.1 0.08 ± 30% +0.0 0.00 perf-profile.children.cycles-pp.record__pushfn
0.04 ± 71% +0.1 0.12 ± 35% -0.0 0.03 ±100% perf-profile.children.cycles-pp.perf_mmap__push
0.54 ± 2% +0.1 0.62 ± 7% -0.0 0.54 ± 2% perf-profile.children.cycles-pp.cmd_record
0.04 ± 70% +0.1 0.13 ± 32% -0.0 0.03 ±100% perf-profile.children.cycles-pp.record__mmap_read_evlist
0.04 ± 71% +0.1 0.13 ± 35% -0.0 0.04 ±100% perf-profile.children.cycles-pp.handle_internal_command
0.04 ± 71% +0.1 0.13 ± 35% -0.0 0.04 ±100% perf-profile.children.cycles-pp.main
0.04 ± 71% +0.1 0.13 ± 35% -0.0 0.04 ±100% perf-profile.children.cycles-pp.run_builtin
0.10 ± 4% +0.1 0.20 ± 4% -0.0 0.10 perf-profile.children.cycles-pp.do_perf_trace_sched_switch
0.00 +0.1 0.10 ± 4% +0.0 0.00 perf-profile.children.cycles-pp.ct_idle_enter
0.13 ± 3% +0.2 0.29 ± 4% -0.0 0.13 perf-profile.children.cycles-pp.perf_trace_sched_switch
0.21 ± 6% +0.6 0.78 ± 3% -0.0 0.20 perf-profile.children.cycles-pp.update_sg_lb_stats
0.22 ± 6% +0.6 0.81 ± 4% -0.0 0.21 perf-profile.children.cycles-pp.update_sd_lb_stats
0.22 ± 6% +0.6 0.81 ± 3% -0.0 0.21 perf-profile.children.cycles-pp.sched_balance_find_src_group
0.40 ± 3% +0.7 1.08 ± 4% -0.0 0.40 perf-profile.children.cycles-pp.sched_balance_newidle
0.27 ± 6% +0.7 0.97 ± 4% -0.0 0.26 perf-profile.children.cycles-pp.sched_balance_rq
5.90 +0.9 6.81 -0.0 5.88 perf-profile.children.cycles-pp.intel_idle
35.82 +3.5 39.30 +0.1 35.93 perf-profile.children.cycles-pp.__x64_sys_mq_timedreceive
35.70 +3.5 39.25 +0.1 35.82 perf-profile.children.cycles-pp.do_mq_timedreceive
0.00 +8.8 8.78 ± 3% +0.0 0.00 perf-profile.children.cycles-pp.drain_obj_stock
28.34 +13.4 41.76 +0.3 28.64 perf-profile.children.cycles-pp.do_mq_timedsend
28.52 +13.5 41.99 +0.3 28.82 perf-profile.children.cycles-pp.__x64_sys_mq_timedsend
70.76 +13.7 84.45 +0.3 71.10 perf-profile.children.cycles-pp.entry_SYSCALL_64_after_hwframe
70.56 +13.8 84.39 +0.3 70.90 perf-profile.children.cycles-pp.do_syscall_64
0.08 +16.2 16.31 ± 4% +0.2 0.27 ± 3% perf-profile.children.cycles-pp.__refill_obj_stock
3.22 +20.1 23.33 ± 2% +0.4 3.62 perf-profile.children.cycles-pp.kfree
3.01 +20.2 23.25 ± 2% +0.4 3.38 perf-profile.children.cycles-pp.free_msg
2.46 +20.5 23.00 ± 2% +0.4 2.83 perf-profile.children.cycles-pp.__memcg_slab_free_hook
2.31 +25.9 28.25 ± 2% +0.6 2.94 perf-profile.children.cycles-pp.load_msg
1.00 +26.8 27.81 ± 2% +0.6 1.62 perf-profile.children.cycles-pp.__kmalloc_node_noprof
0.68 +27.0 27.65 ± 2% +0.6 1.29 perf-profile.children.cycles-pp.__memcg_slab_post_alloc_hook
4.41 -4.4 0.02 ±141% -0.1 4.28 perf-profile.self.cycles-pp.poll_idle
6.79 -3.2 3.63 ± 5% -0.1 6.70 perf-profile.self.cycles-pp.select_idle_core
3.07 -1.7 1.33 ± 4% -0.0 3.06 perf-profile.self.cycles-pp.do_mq_timedreceive
2.90 -1.6 1.31 ± 3% -0.0 2.88 perf-profile.self.cycles-pp.__schedule
2.37 -1.5 0.84 ± 3% -0.0 2.34 perf-profile.self.cycles-pp._copy_to_user
2.73 -1.5 1.22 ± 3% -0.0 2.70 perf-profile.self.cycles-pp.do_mq_timedsend
2.63 -1.4 1.25 ± 3% -0.0 2.59 perf-profile.self.cycles-pp._raw_spin_lock
1.64 -1.0 0.64 ± 3% -0.1 1.52 perf-profile.self.cycles-pp._raw_spin_lock_irqsave
1.46 -0.9 0.55 ± 2% -0.0 1.45 perf-profile.self.cycles-pp.switch_mm_irqs_off
1.54 -0.9 0.67 ± 6% +0.0 1.54 perf-profile.self.cycles-pp.update_rq_clock_task
1.48 -0.9 0.62 ± 3% +0.0 1.49 perf-profile.self.cycles-pp.msg_get
1.36 -0.8 0.58 ± 3% -0.1 1.30 perf-profile.self.cycles-pp.exit_to_user_mode_loop
1.11 -0.7 0.44 ± 4% -0.0 1.08 perf-profile.self.cycles-pp.msg_insert
1.56 -0.7 0.90 ± 6% -0.0 1.56 perf-profile.self.cycles-pp.restore_fpregs_from_fpstate
0.95 -0.7 0.29 ± 5% -0.0 0.94 perf-profile.self.cycles-pp.llist_reverse_order
1.00 -0.7 0.34 ± 7% -0.0 0.99 perf-profile.self.cycles-pp.perf_swevent_get_recursion_context
1.19 -0.6 0.59 ± 3% -0.0 1.18 perf-profile.self.cycles-pp.wq_sleep
0.92 -0.6 0.36 ± 6% -0.0 0.92 perf-profile.self.cycles-pp.do_perf_trace_sched_wakeup_template
0.90 -0.6 0.34 ± 2% +0.0 0.92 perf-profile.self.cycles-pp.__update_idle_core
1.15 -0.5 0.60 ± 7% -0.0 1.14 perf-profile.self.cycles-pp.task_h_load
0.83 -0.5 0.30 ± 5% -0.0 0.83 perf-profile.self.cycles-pp.call_function_single_prep_ipi
0.97 -0.5 0.44 ± 2% -0.0 0.94 perf-profile.self.cycles-pp.dequeue_entities
0.84 -0.5 0.34 ± 3% +0.0 0.86 perf-profile.self.cycles-pp.cpuacct_charge
0.96 -0.5 0.47 ± 4% +0.0 0.96 perf-profile.self.cycles-pp.do_syscall_64
1.23 -0.5 0.74 -0.0 1.22 perf-profile.self.cycles-pp.__switch_to
0.73 ± 4% -0.5 0.24 ± 3% -0.0 0.70 perf-profile.self.cycles-pp.__bitmap_andnot
0.69 -0.5 0.20 ± 4% +0.0 0.69 perf-profile.self.cycles-pp.flush_smp_call_function_queue
0.94 -0.5 0.47 +0.0 0.94 ± 2% perf-profile.self.cycles-pp._find_next_bit
0.77 -0.4 0.36 ± 6% +0.0 0.77 perf-profile.self.cycles-pp.update_entity_lag
0.76 -0.4 0.36 ± 3% -0.0 0.74 perf-profile.self.cycles-pp.update_load_avg
0.67 -0.4 0.27 ± 5% -0.0 0.66 perf-profile.self.cycles-pp.__smp_call_single_queue
0.70 -0.4 0.31 ± 2% +0.0 0.70 perf-profile.self.cycles-pp.pick_next_task_fair
0.55 -0.4 0.17 ± 2% -0.0 0.54 perf-profile.self.cycles-pp.os_xsave
0.56 -0.4 0.18 ± 4% -0.0 0.54 perf-profile.self.cycles-pp.entry_SYSRETQ_unsafe_stack
0.63 -0.4 0.25 -0.0 0.62 perf-profile.self.cycles-pp.switch_fpu_return
1.38 -0.4 1.01 -0.0 1.36 perf-profile.self.cycles-pp.native_queued_spin_lock_slowpath
0.69 -0.4 0.33 ± 5% -0.0 0.68 perf-profile.self.cycles-pp.wake_affine
0.53 -0.4 0.16 -0.0 0.51 perf-profile.self.cycles-pp.native_sched_clock
0.67 -0.4 0.30 ± 2% +0.0 0.70 perf-profile.self.cycles-pp.kfree
0.75 -0.4 0.39 ± 3% +0.0 0.76 perf-profile.self.cycles-pp.update_curr
0.67 -0.4 0.31 ± 5% -0.0 0.66 perf-profile.self.cycles-pp.ttwu_queue_wakelist
0.53 -0.4 0.18 ± 2% -0.0 0.52 perf-profile.self.cycles-pp.arch_exit_to_user_mode_prepare
0.49 ± 2% -0.4 0.14 ± 3% -0.0 0.48 perf-profile.self.cycles-pp._copy_from_user
0.63 -0.3 0.28 ± 3% -0.0 0.62 perf-profile.self.cycles-pp.schedule_hrtimeout_range_clock
0.58 -0.3 0.23 ± 4% -0.0 0.57 perf-profile.self.cycles-pp.select_idle_sibling
0.58 -0.3 0.24 ± 7% -0.0 0.56 perf-profile.self.cycles-pp.migrate_task_rq_fair
0.64 -0.3 0.31 -0.0 0.63 perf-profile.self.cycles-pp.prepare_task_switch
0.82 -0.3 0.49 ± 3% -0.0 0.82 perf-profile.self.cycles-pp.select_idle_cpu
0.52 ± 3% -0.3 0.21 ± 2% +0.0 0.52 perf-profile.self.cycles-pp.__put_user_4
0.63 ± 11% -0.3 0.32 ± 5% +0.0 0.64 perf-profile.self.cycles-pp.stress_switch_mq
0.54 -0.3 0.24 ± 5% -0.0 0.53 perf-profile.self.cycles-pp.__switch_to_asm
0.54 -0.3 0.25 ± 3% +0.0 0.55 perf-profile.self.cycles-pp.fdget
0.51 -0.3 0.22 ± 4% -0.0 0.50 perf-profile.self.cycles-pp.mm_cid_switch_to
0.44 -0.3 0.15 ± 6% -0.0 0.44 perf-profile.self.cycles-pp.select_task_rq_fair
0.43 -0.3 0.15 ± 5% -0.0 0.42 perf-profile.self.cycles-pp.sched_ttwu_pending
0.49 -0.3 0.23 ± 6% +0.0 0.50 perf-profile.self.cycles-pp.enqueue_task
0.59 -0.2 0.34 ± 2% -0.0 0.58 perf-profile.self.cycles-pp.try_to_wake_up
0.73 -0.2 0.48 -0.0 0.72 perf-profile.self.cycles-pp.update_cfs_rq_load_avg
0.35 -0.2 0.11 ± 4% +0.0 0.35 perf-profile.self.cycles-pp.__resched_curr
0.40 -0.2 0.16 ± 2% -0.0 0.40 perf-profile.self.cycles-pp.__pick_next_task
0.36 -0.2 0.12 ± 3% -0.0 0.35 perf-profile.self.cycles-pp.cpuidle_idle_call
0.34 -0.2 0.11 -0.0 0.33 perf-profile.self.cycles-pp.entry_SYSCALL_64
0.57 -0.2 0.36 +0.0 0.58 perf-profile.self.cycles-pp.__enqueue_entity
0.51 -0.2 0.31 -0.0 0.50 perf-profile.self.cycles-pp.__update_load_avg_cfs_rq
0.34 -0.2 0.14 ± 5% +0.0 0.36 perf-profile.self.cycles-pp.wakeup_preempt
0.34 -0.2 0.14 ± 3% -0.0 0.34 perf-profile.self.cycles-pp.__wrgsbase_inactive
0.39 -0.2 0.20 ± 2% -0.0 0.39 perf-profile.self.cycles-pp.do_idle
0.31 ± 5% -0.2 0.11 -0.0 0.28 ± 5% perf-profile.self.cycles-pp.__check_heap_object
0.37 -0.2 0.17 ± 2% +0.0 0.37 perf-profile.self.cycles-pp.check_heap_object
0.51 -0.2 0.32 +0.0 0.51 perf-profile.self.cycles-pp.schedule
0.36 -0.2 0.18 ± 2% -0.0 0.36 perf-profile.self.cycles-pp.__pick_eevdf
0.61 -0.2 0.43 +0.0 0.63 perf-profile.self.cycles-pp.dequeue_entity
0.29 ± 2% -0.2 0.11 ± 4% +0.0 0.31 perf-profile.self.cycles-pp.__virt_addr_valid
0.30 -0.2 0.12 -0.0 0.29 perf-profile.self.cycles-pp.__update_load_avg_se
0.27 -0.2 0.10 ± 4% -0.0 0.27 perf-profile.self.cycles-pp.syscall_return_via_sysret
0.25 -0.2 0.08 ± 5% -0.0 0.24 ± 2% perf-profile.self.cycles-pp.__check_object_size
0.26 -0.2 0.11 ± 4% -0.0 0.25 perf-profile.self.cycles-pp.place_entity
0.45 -0.2 0.30 ± 3% -0.0 0.44 perf-profile.self.cycles-pp.enqueue_entity
0.44 -0.2 0.29 ± 5% -0.0 0.44 perf-profile.self.cycles-pp.enqueue_task_fair
0.24 -0.2 0.09 +0.0 0.24 perf-profile.self.cycles-pp.wake_up_q
0.22 ± 2% -0.1 0.07 -0.0 0.21 perf-profile.self.cycles-pp.entry_SYSCALL_64_after_hwframe
0.24 -0.1 0.09 ± 5% +0.0 0.24 perf-profile.self.cycles-pp.___perf_sw_event
0.32 -0.1 0.18 ± 2% +0.0 0.32 perf-profile.self.cycles-pp.dequeue_task_fair
0.21 ± 2% -0.1 0.06 ± 7% -0.0 0.20 perf-profile.self.cycles-pp.read_tsc
0.20 -0.1 0.06 -0.0 0.20 ± 2% perf-profile.self.cycles-pp.__rdgsbase_inactive
0.36 -0.1 0.22 ± 4% -0.0 0.35 perf-profile.self.cycles-pp.perf_tp_event
0.24 -0.1 0.11 ± 4% -0.0 0.24 perf-profile.self.cycles-pp.strnlen
0.28 -0.1 0.14 +0.0 0.28 perf-profile.self.cycles-pp.__kmalloc_node_noprof
0.21 -0.1 0.08 ± 6% +0.0 0.23 perf-profile.self.cycles-pp.load_msg
0.33 -0.1 0.20 ± 4% -0.0 0.32 perf-profile.self.cycles-pp.attach_entity_load_avg
0.12 ± 3% -0.1 0.00 -0.0 0.08 ± 6% perf-profile.self.cycles-pp.__do_notify
0.41 -0.1 0.29 -0.0 0.40 perf-profile.self.cycles-pp.update_se
0.44 -0.1 0.33 -0.0 0.44 perf-profile.self.cycles-pp.menu_select
0.27 -0.1 0.15 ± 6% -0.0 0.26 perf-profile.self.cycles-pp.select_task_rq
0.17 ± 2% -0.1 0.06 ± 8% -0.0 0.17 perf-profile.self.cycles-pp.entry_SYSCALL_64_safe_stack
0.23 ± 2% -0.1 0.11 ± 4% -0.0 0.22 perf-profile.self.cycles-pp.__flush_smp_call_function_queue
0.19 -0.1 0.08 ± 6% -0.0 0.18 ± 2% perf-profile.self.cycles-pp.update_rq_clock
0.18 ± 2% -0.1 0.06 ± 7% -0.0 0.17 perf-profile.self.cycles-pp.wake_q_add_safe
0.20 ± 2% -0.1 0.09 +0.0 0.21 perf-profile.self.cycles-pp.check_stack_object
0.18 -0.1 0.07 +0.1 0.26 perf-profile.self.cycles-pp.__account_obj_stock
0.21 ± 2% -0.1 0.10 ± 4% -0.0 0.20 perf-profile.self.cycles-pp.__kmalloc_cache_noprof
0.24 -0.1 0.14 -0.0 0.24 ± 2% perf-profile.self.cycles-pp.__dequeue_entity
0.19 -0.1 0.10 ± 4% -0.0 0.19 perf-profile.self.cycles-pp.pick_task_fair
0.14 ± 3% -0.1 0.05 +0.0 0.14 ± 3% perf-profile.self.cycles-pp.inode_set_ctime_current
0.16 ± 3% -0.1 0.06 ± 7% +0.0 0.16 perf-profile.self.cycles-pp.nohz_run_idle_balance
0.15 -0.1 0.06 +0.0 0.15 perf-profile.self.cycles-pp.avg_vruntime
0.16 -0.1 0.07 +0.0 0.16 perf-profile.self.cycles-pp.schedule_idle
0.08 -0.1 0.00 -0.0 0.07 perf-profile.self.cycles-pp.security_msg_msg_free
0.07 -0.1 0.00 -0.0 0.06 perf-profile.self.cycles-pp.ct_kernel_enter
0.07 -0.1 0.00 +0.0 0.08 ± 5% perf-profile.self.cycles-pp.trylock_stock
0.15 -0.1 0.08 ± 5% -0.0 0.14 ± 3% perf-profile.self.cycles-pp.dl_server_update
0.15 -0.1 0.08 ± 5% +0.0 0.15 perf-profile.self.cycles-pp.raw_spin_rq_lock_nested
0.12 -0.1 0.05 ± 8% +0.0 0.12 perf-profile.self.cycles-pp.migrate_disable_switch
0.12 ± 4% -0.1 0.05 +0.0 0.12 perf-profile.self.cycles-pp.__x64_sys_mq_timedreceive
0.13 -0.1 0.07 ± 7% -0.0 0.12 ± 4% perf-profile.self.cycles-pp.___task_rq_lock
0.09 ± 5% -0.1 0.03 ± 70% +0.0 0.11 perf-profile.self.cycles-pp.inode_set_ctime_to_ts
0.22 -0.1 0.16 -0.0 0.21 perf-profile.self.cycles-pp.cpuidle_enter_state
0.12 -0.1 0.06 -0.0 0.12 ± 4% perf-profile.self.cycles-pp.store_msg
0.12 -0.1 0.06 -0.0 0.12 ± 4% perf-profile.self.cycles-pp.wakeup_preempt_fair
0.11 -0.1 0.05 +0.0 0.11 perf-profile.self.cycles-pp.timestamp_truncate
0.11 ± 4% -0.1 0.05 ± 8% +0.0 0.12 perf-profile.self.cycles-pp.ktime_get_coarse_real_ts64_mg
0.12 ± 4% -0.1 0.06 ± 7% -0.0 0.11 perf-profile.self.cycles-pp.do_perf_trace_sched_stat_runtime
0.11 ± 4% -0.1 0.06 ± 8% -0.0 0.10 ± 4% perf-profile.self.cycles-pp.tracing_gen_ctx_irq_test
0.07 ± 11% -0.1 0.02 ±141% +0.0 0.11 ± 9% perf-profile.self.cycles-pp.mq_timedreceive
0.13 ± 3% -0.0 0.08 +0.0 0.13 perf-profile.self.cycles-pp.update_curr_dl_se
0.15 -0.0 0.11 ± 4% +0.0 0.16 ± 3% perf-profile.self.cycles-pp.sched_balance_newidle
0.09 -0.0 0.05 ± 8% -0.0 0.09 perf-profile.self.cycles-pp.native_irq_return_iret
0.14 ± 3% -0.0 0.10 ± 4% -0.0 0.13 perf-profile.self.cycles-pp.vruntime_eligible
0.14 ± 3% -0.0 0.11 +0.0 0.14 ± 3% perf-profile.self.cycles-pp.ktime_get
0.07 ± 7% -0.0 0.03 ± 70% -0.0 0.06 perf-profile.self.cycles-pp.__set_next_task_fair
0.15 ± 3% -0.0 0.13 +0.0 0.16 perf-profile.self.cycles-pp.put_prev_entity
0.02 ±141% -0.0 0.00 +0.1 0.09 perf-profile.self.cycles-pp.css_rstat_updated
0.02 ±141% -0.0 0.00 +0.0 0.06 ± 9% perf-profile.self.cycles-pp.perf_trace_buf_update
0.19 -0.0 0.18 ± 2% -0.0 0.18 perf-profile.self.cycles-pp.__x64_sys_mq_timedsend
0.00 +0.0 0.00 +0.1 0.11 perf-profile.self.cycles-pp.__mod_memcg_state
0.00 +0.0 0.00 +0.1 0.12 ± 4% perf-profile.self.cycles-pp.try_charge_memcg
0.00 +0.0 0.00 +0.3 0.26 ± 3% perf-profile.self.cycles-pp.__mod_memcg_lruvec_state
0.11 ± 4% +0.0 0.12 ± 3% +0.0 0.11 perf-profile.self.cycles-pp.set_next_task_idle
0.06 +0.0 0.08 ± 6% +0.0 0.06 perf-profile.self.cycles-pp.perf_swevent_event
0.07 ± 7% +0.0 0.09 ± 10% +0.0 0.11 perf-profile.self.cycles-pp.mq_timedsend
0.00 +0.1 0.05 +0.0 0.00 perf-profile.self.cycles-pp.ct_idle_enter
0.00 +0.1 0.05 +0.0 0.00 perf-profile.self.cycles-pp.perf_trace_sched_switch
0.00 +0.1 0.06 +0.0 0.00 perf-profile.self.cycles-pp.sched_update_worker
0.12 +0.1 0.18 ± 6% +0.0 0.12 perf-profile.self.cycles-pp.x64_sys_call
0.38 ± 2% +0.1 0.46 -0.0 0.37 perf-profile.self.cycles-pp.finish_task_switch
0.08 ± 5% +0.1 0.20 ± 6% -0.0 0.08 perf-profile.self.cycles-pp.do_perf_trace_sched_switch
0.18 ± 5% +0.5 0.71 ± 3% -0.0 0.18 ± 2% perf-profile.self.cycles-pp.update_sg_lb_stats
5.89 +0.9 6.81 -0.0 5.88 perf-profile.self.cycles-pp.intel_idle
0.07 +7.4 7.48 ± 4% +0.1 0.22 ± 6% perf-profile.self.cycles-pp.__refill_obj_stock
0.00 +8.7 8.70 ± 3% +0.0 0.00 perf-profile.self.cycles-pp.drain_obj_stock
2.25 +12.4 14.61 +0.0 2.26 perf-profile.self.cycles-pp.__memcg_slab_free_hook
0.53 +19.0 19.48 +0.1 0.67 perf-profile.self.cycles-pp.__memcg_slab_post_alloc_hook
^ permalink raw reply
* Re: [PATCH v4 05/12] mm, swap: unify large folio allocation
From: Kairui Song @ 2026-05-16 18:26 UTC (permalink / raw)
To: kasong
Cc: linux-mm, Andrew Morton, David Hildenbrand, Zi Yan, Baolin Wang,
Barry Song, Hugh Dickins, Chris Li, Kemeng Shi, Nhat Pham,
Baoquan He, Johannes Weiner, Youngjun Park, Chengming Zhou,
Roman Gushchin, Shakeel Butt, Muchun Song, linux-kernel, cgroups,
Lorenzo Stoakes, Yosry Ahmed, Qi Zheng
In-Reply-To: <20260515-swap-table-p4-v4-5-f1b49e845a8d@tencent.com>
On Fri, May 15, 2026 at 6:11 PM Kairui Song via B4 Relay
<devnull+kasong.tencent.com@kernel.org> wrote:
>
> From: Kairui Song <kasong@tencent.com>
>
> Now that direct large order allocation is supported in the swap cache,
> both anon and shmem can use it instead of implementing their own methods.
> This unifies the fallback and swap cache check, which also reduces the
> TOCTOU race window of swap cache state: previously, high order swapin
> required checking swap cache states first, then allocating and falling
> back separately. Now all these steps happen in the same compact loop.
>
> Order fallback and statistics are also unified, callers just need to
> check and pass the acceptable order bitmask.
>
> There is basically no behavior change. This only makes things more
> unified and prepares for later commits. Cgroup and zero map checks can
> also be moved into the compact loop, further reducing race windows and
> redundancy
>
> Acked-by: Chris Li <chrisl@kernel.org>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
> mm/memory.c | 77 ++++++------------------------
> mm/shmem.c | 95 ++++++++++---------------------------
> mm/swap.h | 30 ++----------
> mm/swap_state.c | 143 ++++++++++----------------------------------------------
> mm/swapfile.c | 3 +-
> 5 files changed, 68 insertions(+), 280 deletions(-)
>
> diff --git a/mm/shmem.c b/mm/shmem.c
> index 6edb23b41bac..e3edc0c20e34 100644
> --- a/mm/shmem.c
> +++ b/mm/shmem.c
> @@ -159,7 +159,7 @@ static unsigned long shmem_default_max_inodes(void)
>
> static int shmem_swapin_folio(struct inode *inode, pgoff_t index,
> struct folio **foliop, enum sgp_type sgp, gfp_t gfp,
> - struct vm_area_struct *vma, vm_fault_t *fault_type);
> + struct vm_fault *vmf, vm_fault_t *fault_type);
>
> static inline struct shmem_sb_info *SHMEM_SB(struct super_block *sb)
> {
> @@ -2017,68 +2017,25 @@ static struct folio *shmem_alloc_and_add_folio(struct vm_fault *vmf,
> }
>
> static struct folio *shmem_swap_alloc_folio(struct inode *inode,
> - struct vm_area_struct *vma, pgoff_t index,
> + struct vm_fault *vmf, pgoff_t index,
> swp_entry_t entry, int order, gfp_t gfp)
> {
> + pgoff_t ilx;
> + struct folio *folio;
> + struct mempolicy *mpol;
> + /* Always allow order 0 so swap won't fail under pressure. */
> + unsigned long orders = BIT(order) | BIT(0);
> struct shmem_inode_info *info = SHMEM_I(inode);
> - struct folio *new, *swapcache;
> - int nr_pages = 1 << order;
> - gfp_t alloc_gfp = gfp;
> -
> - if (!IS_ENABLED(CONFIG_TRANSPARENT_HUGEPAGE)) {
> - if (WARN_ON_ONCE(order))
> - return ERR_PTR(-EINVAL);
> - } else if (order) {
> - /*
> - * If uffd is active for the vma, we need per-page fault
> - * fidelity to maintain the uffd semantics, then fallback
> - * to swapin order-0 folio, as well as for zswap case.
> - * Any existing sub folio in the swap cache also blocks
> - * mTHP swapin.
> - */
> - if ((vma && unlikely(userfaultfd_armed(vma))) ||
> - !zswap_never_enabled() ||
> - non_swapcache_batch(entry, nr_pages) != nr_pages)
> - goto fallback;
>
> - alloc_gfp = thp_shmem_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp);
> - }
> -retry:
> - new = shmem_alloc_folio(alloc_gfp, order, info, index);
> - if (!new) {
> - new = ERR_PTR(-ENOMEM);
> - goto fallback;
> - }
> + if ((vmf && unlikely(userfaultfd_armed(vmf->vma))) ||
> + !zswap_never_enabled())
> + orders = BIT(0);
>
> - if (mem_cgroup_swapin_charge_folio(new, vma ? vma->vm_mm : NULL,
> - alloc_gfp, entry)) {
> - folio_put(new);
> - new = ERR_PTR(-ENOMEM);
> - goto fallback;
> - }
> + mpol = shmem_get_pgoff_policy(info, index, order, &ilx);
> + folio = swapin_sync(entry, gfp, orders, vmf, mpol, ilx);
> + mpol_cond_put(mpol);
>
> - swapcache = swapin_folio(entry, new);
> - if (swapcache != new) {
> - folio_put(new);
> - if (!swapcache) {
> - /*
> - * The new folio is charged already, swapin can
> - * only fail due to another raced swapin.
> - */
> - new = ERR_PTR(-EEXIST);
> - goto fallback;
> - }
> - }
> - return swapcache;
> -fallback:
> - /* Order 0 swapin failed, nothing to fallback to, abort */
> - if (!order)
> - return new;
> - entry.val += index - round_down(index, nr_pages);
> - alloc_gfp = gfp;
> - nr_pages = 1;
> - order = 0;
> - goto retry;
> + return folio;
> }
>
Sashiko reported a problem on this:
When shmem_swap_alloc_folio() computes the interleave index (ilx) for
MPOL_INTERLEAVE and MPOL_WEIGHTED_INTERLEAVE NUMA policies, it passes the
original large swap entry order to shmem_get_pgoff_policy().
If the allocation falls back to smaller orders (like order-0) inside
swap_cache_alloc_folio(), will this ilx be reused for all those fallback
allocations?
Since the calculation of ilx incorporates the original order, reusing the
same interleave index for all 512 fallback pages of a 2MB swap entry might
force them all onto the exact same NUMA node. Does this defeat the intended
page-by-page interleaving policy and potentially cause memory bandwidth
bottlenecks?
===
I initialially thought this is trivial. ilx is already somewhat broken
if we are doing fallback. shmem_get_pgoff_policy() computes ilx =
i_ino + (index >> order). The shift makes sense of all folios are in
the same order: an unshifted ilx = i_ino + index would give index %
nnodes == 0 for every folio on power-of-2 node counts for THP so
shifting by order will ensure interleave is still effective.
However, once a file is backed with mixed-order folios due to fallback
or case, the shift becomes order-dependent and the ilx mapping is no
longer monotonic. The calculated interleave is skewed already.
It deserves a separate look as that's a pre-exist seperate problem. I
think I better not change that, as it might cause confusion. The
problem can be solved (or ignored?) later as it's not critical, the
ilx is just a hint anyway. For now I'll just squash the following
change to keep the behavior identical to before (hoist the fallback to
shmem just like before):
diff --git a/mm/shmem.c b/mm/shmem.c
index e3edc0c20e34..4427661ab2ee 100644
--- a/mm/shmem.c
+++ b/mm/shmem.c
@@ -2023,19 +2023,26 @@ static struct folio
*shmem_swap_alloc_folio(struct inode *inode,
pgoff_t ilx;
struct folio *folio;
struct mempolicy *mpol;
- /* Always allow order 0 so swap won't fail under pressure. */
- unsigned long orders = BIT(order) | BIT(0);
struct shmem_inode_info *info = SHMEM_I(inode);
if ((vmf && unlikely(userfaultfd_armed(vmf->vma))) ||
!zswap_never_enabled())
- orders = BIT(0);
+ order = 0;
+again:
mpol = shmem_get_pgoff_policy(info, index, order, &ilx);
- folio = swapin_sync(entry, gfp, orders, vmf, mpol, ilx);
+ folio = swapin_sync(entry, gfp, BIT(order), vmf, mpol, ilx);
mpol_cond_put(mpol);
- return folio;
+ if (!IS_ERR(folio))
+ return folio;
+
+ if (order) {
+ order = 0;
+ goto again;
+ }
+
+ return NULL;
}
/*
diff --git a/mm/swap_state.c b/mm/swap_state.c
index 946ec4ae9ae1..ce4e8c39ed12 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -652,7 +652,7 @@ static struct folio
*swap_cache_read_folio(swp_entry_t entry, gfp_t gfp,
* if needed. @entry is rounded down if @orders allow large allocation.
*
* Context: Caller must ensure @entry is valid and pin the swap
device with refcount.
- * Return: Returns the folio on success, NULL if failed.
+ * Return: Returns the folio on success, error code if failed.
*/
struct folio *swapin_sync(swp_entry_t entry, gfp_t gfp, unsigned long orders,
struct vm_fault *vmf, struct mempolicy *mpol, pgoff_t ilx)
@@ -667,7 +667,7 @@ struct folio *swapin_sync(swp_entry_t entry, gfp_t
gfp, unsigned long orders,
} while (IS_ERR(folio) && PTR_ERR(folio) == -EEXIST);
if (IS_ERR(folio))
- return NULL;
+ return folio;
swap_read_folio(folio, NULL);
return folio;
^ permalink raw reply related
* Re: [PATCH v2 2/3] security: Expand task_setscheduler LSM hook to include CPU affinity mask
From: Aaron Tomlin @ 2026-05-16 13:36 UTC (permalink / raw)
To: Paul Moore
Cc: tsbogend, jmorris, serge, mingo, peterz, juri.lelli,
vincent.guittot, stephen.smalley.work, casey, longman, tj, hannes,
mkoutny, chenridong, dietmar.eggemann, rostedt, bsegall, mgorman,
vschneid, kprateek.nayak, omosnace, kees, neelx, sean, chjohnst,
steve, mproche, nick.lange, cgroups, linux-mips, linux-fsdevel,
linux-security-module, selinux, linux-kernel
In-Reply-To: <CAHC9VhSWrJc=aE1Sg4xfv1ZMmh=JqZFLWGeG2SnzOFqXxcUbtQ@mail.gmail.com>
On Thu, May 14, 2026 at 04:15:15PM -0400, Paul Moore wrote:
> > However, I suspect the MIPS-related patch will need to remain coupled with
> > this feature patch. Because the first patch fundamentally alters the
> > signature of the security_task_setscheduler() hook, the MIPS FPU affinity
> > code must be updated concurrently to accommodate the new parameter.
>
> I generally dislike when bug fixes depend on new functionality; it's
> backwards in my opinion. I would much rather see the MIPS bug fix
> patch submitted as a standalone patch and then have the LSM hook
> modification patch come separately, perhaps with a note that it
> depends on the bug fix patch.
Hi Paul,
That is a fair point, and I completely agree with your philosophy.
I will decouple them accordingly. I will submit the MIPS FPU affinity fix
as a standalone patch first so it can be routed, reviewed, and potentially
backported independently.
Once that is out, I will submit the LSM hook modification and the rest of
this feature series separately, rebased on top of the MIPS fix, and will
include a clear note regarding the dependency.
Thanks for the guidance.
Kind regards,
--
Aaron Tomlin
^ permalink raw reply
* [tj-cgroup:for-7.2] BUILD SUCCESS 1dffd95575eb05bc7ec20ec096ce73be4c5d1ed5
From: kernel test robot @ 2026-05-16 11:42 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-7.2
branch HEAD: 1dffd95575eb05bc7ec20ec096ce73be4c5d1ed5 cgroup: Defer kill_css_finish() in cgroup_apply_control_disable()
elapsed time: 1079m
configs tested: 291
configs skipped: 5
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-15.2.0
alpha allyesconfig gcc-15.2.0
alpha defconfig gcc-15.2.0
arc allmodconfig clang-16
arc allmodconfig gcc-15.2.0
arc allnoconfig gcc-15.2.0
arc allyesconfig clang-23
arc allyesconfig gcc-15.2.0
arc defconfig gcc-15.2.0
arc randconfig-001 gcc-8.5.0
arc randconfig-001-20260516 gcc-14.3.0
arc randconfig-001-20260516 gcc-8.5.0
arc randconfig-002 gcc-8.5.0
arc randconfig-002-20260516 gcc-13.4.0
arc randconfig-002-20260516 gcc-8.5.0
arm allnoconfig clang-23
arm allnoconfig gcc-15.2.0
arm allyesconfig clang-16
arm allyesconfig gcc-15.2.0
arm defconfig clang-23
arm defconfig gcc-15.2.0
arm randconfig-001 gcc-8.5.0
arm randconfig-001-20260516 gcc-13.4.0
arm randconfig-001-20260516 gcc-8.5.0
arm randconfig-002 gcc-8.5.0
arm randconfig-002-20260516 gcc-8.5.0
arm randconfig-003 gcc-8.5.0
arm randconfig-003-20260516 clang-19
arm randconfig-003-20260516 gcc-8.5.0
arm randconfig-004 gcc-8.5.0
arm randconfig-004-20260516 clang-23
arm randconfig-004-20260516 gcc-8.5.0
arm64 allmodconfig clang-19
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-15.2.0
arm64 defconfig gcc-15.2.0
arm64 randconfig-001-20260516 gcc-13.4.0
arm64 randconfig-001-20260516 gcc-9.5.0
arm64 randconfig-002-20260516 clang-23
arm64 randconfig-002-20260516 gcc-9.5.0
arm64 randconfig-003-20260516 gcc-9.5.0
arm64 randconfig-004-20260516 clang-23
arm64 randconfig-004-20260516 gcc-9.5.0
csky allmodconfig gcc-15.2.0
csky allnoconfig gcc-15.2.0
csky defconfig gcc-15.2.0
csky randconfig-001-20260516 gcc-15.2.0
csky randconfig-001-20260516 gcc-9.5.0
csky randconfig-002-20260516 gcc-10.5.0
csky randconfig-002-20260516 gcc-9.5.0
hexagon allmodconfig clang-17
hexagon allmodconfig gcc-15.2.0
hexagon allnoconfig clang-23
hexagon allnoconfig gcc-15.2.0
hexagon defconfig clang-23
hexagon defconfig gcc-15.2.0
hexagon randconfig-001-20260516 clang-23
hexagon randconfig-002-20260516 clang-16
i386 allmodconfig clang-20
i386 allmodconfig gcc-14
i386 allnoconfig gcc-14
i386 allnoconfig gcc-15.2.0
i386 allyesconfig clang-20
i386 allyesconfig gcc-14
i386 buildonly-randconfig-001 clang-20
i386 buildonly-randconfig-001-20260516 clang-20
i386 buildonly-randconfig-002 clang-20
i386 buildonly-randconfig-002-20260516 clang-20
i386 buildonly-randconfig-003 clang-20
i386 buildonly-randconfig-003-20260516 clang-20
i386 buildonly-randconfig-004 clang-20
i386 buildonly-randconfig-004-20260516 clang-20
i386 buildonly-randconfig-004-20260516 gcc-14
i386 buildonly-randconfig-005 clang-20
i386 buildonly-randconfig-005-20260516 clang-20
i386 buildonly-randconfig-006 clang-20
i386 buildonly-randconfig-006-20260516 clang-20
i386 defconfig clang-20
i386 defconfig gcc-15.2.0
i386 randconfig-001 clang-20
i386 randconfig-001-20260516 clang-20
i386 randconfig-002 clang-20
i386 randconfig-002-20260516 clang-20
i386 randconfig-003 clang-20
i386 randconfig-003-20260516 clang-20
i386 randconfig-004 clang-20
i386 randconfig-004-20260516 clang-20
i386 randconfig-005 clang-20
i386 randconfig-005-20260516 clang-20
i386 randconfig-006 clang-20
i386 randconfig-006-20260516 clang-20
i386 randconfig-007 clang-20
i386 randconfig-007-20260516 clang-20
i386 randconfig-011-20260516 gcc-14
i386 randconfig-012-20260516 clang-20
i386 randconfig-012-20260516 gcc-14
i386 randconfig-013-20260516 gcc-14
i386 randconfig-014-20260516 gcc-14
i386 randconfig-015-20260516 clang-20
i386 randconfig-015-20260516 gcc-14
i386 randconfig-016-20260516 gcc-14
i386 randconfig-017-20260516 gcc-14
loongarch allmodconfig clang-19
loongarch allmodconfig clang-23
loongarch allnoconfig clang-23
loongarch allnoconfig gcc-15.2.0
loongarch defconfig clang-19
loongarch randconfig-001-20260516 gcc-15.2.0
loongarch randconfig-002-20260516 clang-18
m68k allmodconfig gcc-15.2.0
m68k allnoconfig gcc-15.2.0
m68k allyesconfig clang-16
m68k allyesconfig gcc-15.2.0
m68k defconfig clang-19
m68k defconfig gcc-15.2.0
microblaze allnoconfig gcc-15.2.0
microblaze allyesconfig gcc-15.2.0
microblaze defconfig clang-19
microblaze defconfig gcc-15.2.0
mips allmodconfig gcc-15.2.0
mips allnoconfig gcc-15.2.0
mips allyesconfig gcc-15.2.0
nios2 allmodconfig clang-23
nios2 allmodconfig gcc-11.5.0
nios2 allnoconfig clang-23
nios2 allnoconfig gcc-11.5.0
nios2 defconfig clang-19
nios2 defconfig gcc-11.5.0
nios2 randconfig-001-20260516 gcc-10.5.0
nios2 randconfig-002-20260516 gcc-11.5.0
openrisc allmodconfig clang-23
openrisc allmodconfig gcc-15.2.0
openrisc allnoconfig clang-23
openrisc allnoconfig gcc-15.2.0
openrisc defconfig gcc-15.2.0
parisc allmodconfig gcc-15.2.0
parisc allnoconfig clang-23
parisc allnoconfig gcc-15.2.0
parisc allyesconfig clang-19
parisc allyesconfig gcc-15.2.0
parisc defconfig gcc-15.2.0
parisc randconfig-001 gcc-13.4.0
parisc randconfig-001-20260516 gcc-10.5.0
parisc randconfig-001-20260516 gcc-12.5.0
parisc randconfig-002 gcc-8.5.0
parisc randconfig-002-20260516 gcc-11.5.0
parisc randconfig-002-20260516 gcc-12.5.0
parisc64 defconfig clang-19
parisc64 defconfig gcc-15.2.0
powerpc allmodconfig gcc-15.2.0
powerpc allnoconfig clang-23
powerpc allnoconfig gcc-15.2.0
powerpc randconfig-001 gcc-13.4.0
powerpc randconfig-001-20260516 clang-23
powerpc randconfig-001-20260516 gcc-12.5.0
powerpc randconfig-002 gcc-10.5.0
powerpc randconfig-002-20260516 gcc-12.5.0
powerpc64 randconfig-001 clang-17
powerpc64 randconfig-001-20260516 clang-17
powerpc64 randconfig-001-20260516 gcc-12.5.0
powerpc64 randconfig-002 clang-23
powerpc64 randconfig-002-20260516 clang-23
powerpc64 randconfig-002-20260516 gcc-12.5.0
riscv allmodconfig clang-23
riscv allnoconfig clang-23
riscv allnoconfig gcc-15.2.0
riscv allyesconfig clang-16
riscv defconfig clang-23
riscv defconfig gcc-15.2.0
riscv randconfig-001-20260516 gcc-15.2.0
riscv randconfig-001-20260516 gcc-8.5.0
riscv randconfig-002-20260516 clang-16
riscv randconfig-002-20260516 gcc-15.2.0
s390 allmodconfig clang-18
s390 allmodconfig clang-19
s390 allnoconfig clang-23
s390 allyesconfig gcc-15.2.0
s390 defconfig clang-23
s390 defconfig gcc-15.2.0
s390 randconfig-001-20260516 clang-23
s390 randconfig-001-20260516 gcc-15.2.0
s390 randconfig-002-20260516 clang-23
s390 randconfig-002-20260516 gcc-15.2.0
sh allmodconfig gcc-15.2.0
sh allnoconfig clang-23
sh allnoconfig gcc-15.2.0
sh allyesconfig clang-19
sh allyesconfig gcc-15.2.0
sh defconfig gcc-14
sh defconfig gcc-15.2.0
sh edosk7705_defconfig gcc-15.2.0
sh randconfig-001-20260516 gcc-12.5.0
sh randconfig-001-20260516 gcc-15.2.0
sh randconfig-002-20260516 gcc-15.2.0
sparc allnoconfig clang-23
sparc allnoconfig gcc-15.2.0
sparc defconfig gcc-15.2.0
sparc randconfig-001-20260516 gcc-8.5.0
sparc randconfig-002-20260516 gcc-15.2.0
sparc randconfig-002-20260516 gcc-8.5.0
sparc64 allmodconfig clang-23
sparc64 defconfig clang-20
sparc64 defconfig gcc-14
sparc64 randconfig-001-20260516 gcc-10.5.0
sparc64 randconfig-001-20260516 gcc-8.5.0
sparc64 randconfig-002-20260516 clang-20
sparc64 randconfig-002-20260516 gcc-8.5.0
um allmodconfig clang-19
um allnoconfig clang-23
um allyesconfig gcc-14
um allyesconfig gcc-15.2.0
um defconfig clang-23
um defconfig gcc-14
um i386_defconfig gcc-14
um randconfig-001-20260516 clang-16
um randconfig-001-20260516 gcc-8.5.0
um randconfig-002-20260516 clang-18
um randconfig-002-20260516 gcc-8.5.0
um x86_64_defconfig clang-23
um x86_64_defconfig gcc-14
x86_64 allmodconfig clang-20
x86_64 allnoconfig clang-20
x86_64 allnoconfig clang-23
x86_64 allyesconfig clang-20
x86_64 buildonly-randconfig-001 gcc-12
x86_64 buildonly-randconfig-001-20260516 clang-20
x86_64 buildonly-randconfig-001-20260516 gcc-14
x86_64 buildonly-randconfig-002 clang-20
x86_64 buildonly-randconfig-002-20260516 gcc-14
x86_64 buildonly-randconfig-003 gcc-14
x86_64 buildonly-randconfig-003-20260516 gcc-13
x86_64 buildonly-randconfig-003-20260516 gcc-14
x86_64 buildonly-randconfig-004 gcc-14
x86_64 buildonly-randconfig-004-20260516 gcc-14
x86_64 buildonly-randconfig-005 gcc-14
x86_64 buildonly-randconfig-005-20260516 gcc-14
x86_64 buildonly-randconfig-006 clang-20
x86_64 buildonly-randconfig-006-20260516 gcc-14
x86_64 defconfig gcc-14
x86_64 kexec clang-20
x86_64 randconfig-001-20260516 clang-20
x86_64 randconfig-001-20260516 gcc-14
x86_64 randconfig-002-20260516 clang-20
x86_64 randconfig-002-20260516 gcc-14
x86_64 randconfig-003-20260516 gcc-14
x86_64 randconfig-004-20260516 gcc-14
x86_64 randconfig-005-20260516 gcc-14
x86_64 randconfig-006-20260516 gcc-14
x86_64 randconfig-011 clang-20
x86_64 randconfig-011-20260516 clang-20
x86_64 randconfig-011-20260516 gcc-14
x86_64 randconfig-012 clang-20
x86_64 randconfig-012-20260516 clang-20
x86_64 randconfig-013 clang-20
x86_64 randconfig-013-20260516 clang-20
x86_64 randconfig-014 clang-20
x86_64 randconfig-014-20260516 clang-20
x86_64 randconfig-014-20260516 gcc-14
x86_64 randconfig-015 clang-20
x86_64 randconfig-015-20260516 clang-20
x86_64 randconfig-016 clang-20
x86_64 randconfig-016-20260516 clang-20
x86_64 randconfig-016-20260516 gcc-14
x86_64 randconfig-071 gcc-14
x86_64 randconfig-071-20260516 gcc-14
x86_64 randconfig-072 gcc-14
x86_64 randconfig-072-20260516 gcc-14
x86_64 randconfig-073 clang-20
x86_64 randconfig-073-20260516 clang-20
x86_64 randconfig-073-20260516 gcc-14
x86_64 randconfig-074 gcc-14
x86_64 randconfig-074-20260516 clang-20
x86_64 randconfig-074-20260516 gcc-14
x86_64 randconfig-075 gcc-13
x86_64 randconfig-075-20260516 gcc-14
x86_64 randconfig-076 clang-20
x86_64 randconfig-076-20260516 clang-20
x86_64 randconfig-076-20260516 gcc-14
x86_64 rhel-9.4 clang-20
x86_64 rhel-9.4-bpf gcc-14
x86_64 rhel-9.4-func clang-20
x86_64 rhel-9.4-kselftests clang-20
x86_64 rhel-9.4-kunit gcc-14
x86_64 rhel-9.4-ltp gcc-14
x86_64 rhel-9.4-rust clang-20
xtensa allnoconfig clang-23
xtensa allnoconfig gcc-15.2.0
xtensa allyesconfig clang-23
xtensa randconfig-001-20260516 gcc-13.4.0
xtensa randconfig-001-20260516 gcc-8.5.0
xtensa randconfig-002-20260516 gcc-8.5.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* [tj-cgroup:for-next] BUILD SUCCESS 6c79fb30f5cd939f22959bd9b54d7f30c713a759
From: kernel test robot @ 2026-05-16 11:39 UTC (permalink / raw)
To: Tejun Heo; +Cc: cgroups
tree/branch: https://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git for-next
branch HEAD: 6c79fb30f5cd939f22959bd9b54d7f30c713a759 Merge branch 'for-7.2' into for-next
elapsed time: 1075m
configs tested: 281
configs skipped: 5
The following configs have been built successfully.
More configs may be tested in the coming days.
tested configs:
alpha allnoconfig gcc-15.2.0
alpha allyesconfig gcc-15.2.0
alpha defconfig gcc-15.2.0
arc allmodconfig clang-16
arc allmodconfig gcc-15.2.0
arc allnoconfig gcc-15.2.0
arc allyesconfig clang-23
arc allyesconfig gcc-15.2.0
arc defconfig gcc-15.2.0
arc randconfig-001 gcc-8.5.0
arc randconfig-001-20260516 gcc-14.3.0
arc randconfig-001-20260516 gcc-8.5.0
arc randconfig-002 gcc-8.5.0
arc randconfig-002-20260516 gcc-13.4.0
arc randconfig-002-20260516 gcc-8.5.0
arm allnoconfig clang-23
arm allnoconfig gcc-15.2.0
arm allyesconfig clang-16
arm allyesconfig gcc-15.2.0
arm defconfig clang-23
arm defconfig gcc-15.2.0
arm randconfig-001 gcc-8.5.0
arm randconfig-001-20260516 gcc-13.4.0
arm randconfig-001-20260516 gcc-8.5.0
arm randconfig-002 gcc-8.5.0
arm randconfig-002-20260516 gcc-8.5.0
arm randconfig-003 gcc-8.5.0
arm randconfig-003-20260516 clang-19
arm randconfig-003-20260516 gcc-8.5.0
arm randconfig-004 gcc-8.5.0
arm randconfig-004-20260516 clang-23
arm randconfig-004-20260516 gcc-8.5.0
arm64 allmodconfig clang-19
arm64 allmodconfig clang-23
arm64 allnoconfig gcc-15.2.0
arm64 defconfig gcc-15.2.0
arm64 randconfig-001 gcc-8.5.0
arm64 randconfig-001-20260516 gcc-13.4.0
arm64 randconfig-001-20260516 gcc-9.5.0
arm64 randconfig-002 gcc-14.3.0
arm64 randconfig-002-20260516 clang-23
arm64 randconfig-002-20260516 gcc-9.5.0
arm64 randconfig-003 clang-23
arm64 randconfig-003-20260516 gcc-9.5.0
arm64 randconfig-004 clang-23
arm64 randconfig-004-20260516 clang-23
arm64 randconfig-004-20260516 gcc-9.5.0
csky allmodconfig gcc-15.2.0
csky allnoconfig gcc-15.2.0
csky defconfig gcc-15.2.0
csky randconfig-001 gcc-10.5.0
csky randconfig-001-20260516 gcc-15.2.0
csky randconfig-001-20260516 gcc-9.5.0
csky randconfig-002 gcc-10.5.0
csky randconfig-002-20260516 gcc-10.5.0
csky randconfig-002-20260516 gcc-9.5.0
hexagon allmodconfig clang-17
hexagon allmodconfig gcc-15.2.0
hexagon allnoconfig clang-23
hexagon allnoconfig gcc-15.2.0
hexagon defconfig clang-23
hexagon defconfig gcc-15.2.0
hexagon randconfig-001-20260516 clang-23
hexagon randconfig-002-20260516 clang-16
i386 allmodconfig clang-20
i386 allmodconfig gcc-14
i386 allnoconfig gcc-14
i386 allnoconfig gcc-15.2.0
i386 allyesconfig clang-20
i386 allyesconfig gcc-14
i386 buildonly-randconfig-001 clang-20
i386 buildonly-randconfig-001-20260516 clang-20
i386 buildonly-randconfig-002 clang-20
i386 buildonly-randconfig-002-20260516 clang-20
i386 buildonly-randconfig-003 clang-20
i386 buildonly-randconfig-003-20260516 clang-20
i386 buildonly-randconfig-004 clang-20
i386 buildonly-randconfig-004-20260516 clang-20
i386 buildonly-randconfig-004-20260516 gcc-14
i386 buildonly-randconfig-005 clang-20
i386 buildonly-randconfig-005-20260516 clang-20
i386 buildonly-randconfig-006 clang-20
i386 buildonly-randconfig-006-20260516 clang-20
i386 defconfig clang-20
i386 defconfig gcc-15.2.0
i386 randconfig-001 clang-20
i386 randconfig-001-20260516 clang-20
i386 randconfig-002 clang-20
i386 randconfig-002-20260516 clang-20
i386 randconfig-003 clang-20
i386 randconfig-003-20260516 clang-20
i386 randconfig-004 clang-20
i386 randconfig-004-20260516 clang-20
i386 randconfig-005 clang-20
i386 randconfig-005-20260516 clang-20
i386 randconfig-006 clang-20
i386 randconfig-006-20260516 clang-20
i386 randconfig-007 clang-20
i386 randconfig-007-20260516 clang-20
i386 randconfig-011-20260516 gcc-14
i386 randconfig-012-20260516 clang-20
i386 randconfig-012-20260516 gcc-14
i386 randconfig-013-20260516 gcc-14
i386 randconfig-014-20260516 gcc-14
i386 randconfig-015-20260516 clang-20
i386 randconfig-015-20260516 gcc-14
i386 randconfig-016-20260516 gcc-14
i386 randconfig-017-20260516 gcc-14
loongarch allmodconfig clang-19
loongarch allmodconfig clang-23
loongarch allnoconfig clang-23
loongarch allnoconfig gcc-15.2.0
loongarch defconfig clang-19
loongarch randconfig-001-20260516 gcc-15.2.0
loongarch randconfig-002-20260516 clang-18
m68k allmodconfig gcc-15.2.0
m68k allnoconfig gcc-15.2.0
m68k allyesconfig clang-16
m68k allyesconfig gcc-15.2.0
m68k defconfig clang-19
m68k defconfig gcc-15.2.0
microblaze allnoconfig gcc-15.2.0
microblaze allyesconfig gcc-15.2.0
microblaze defconfig clang-19
microblaze defconfig gcc-15.2.0
mips allmodconfig gcc-15.2.0
mips allnoconfig gcc-15.2.0
mips allyesconfig gcc-15.2.0
nios2 allmodconfig clang-23
nios2 allmodconfig gcc-11.5.0
nios2 allnoconfig clang-23
nios2 allnoconfig gcc-11.5.0
nios2 defconfig clang-19
nios2 defconfig gcc-11.5.0
nios2 randconfig-001-20260516 gcc-10.5.0
nios2 randconfig-002-20260516 gcc-11.5.0
openrisc allmodconfig clang-23
openrisc allmodconfig gcc-15.2.0
openrisc allnoconfig clang-23
openrisc allnoconfig gcc-15.2.0
openrisc defconfig gcc-15.2.0
parisc allmodconfig gcc-15.2.0
parisc allnoconfig clang-23
parisc allnoconfig gcc-15.2.0
parisc allyesconfig clang-19
parisc allyesconfig gcc-15.2.0
parisc defconfig gcc-15.2.0
parisc randconfig-001 gcc-13.4.0
parisc randconfig-001-20260516 gcc-10.5.0
parisc randconfig-001-20260516 gcc-12.5.0
parisc randconfig-002 gcc-8.5.0
parisc randconfig-002-20260516 gcc-11.5.0
parisc randconfig-002-20260516 gcc-12.5.0
parisc64 defconfig clang-19
parisc64 defconfig gcc-15.2.0
powerpc allmodconfig gcc-15.2.0
powerpc allnoconfig clang-23
powerpc allnoconfig gcc-15.2.0
powerpc randconfig-001 gcc-13.4.0
powerpc randconfig-001-20260516 clang-23
powerpc randconfig-001-20260516 gcc-12.5.0
powerpc randconfig-002 gcc-10.5.0
powerpc randconfig-002-20260516 gcc-12.5.0
powerpc64 randconfig-001 clang-17
powerpc64 randconfig-001-20260516 clang-17
powerpc64 randconfig-001-20260516 gcc-12.5.0
powerpc64 randconfig-002 clang-23
powerpc64 randconfig-002-20260516 clang-23
powerpc64 randconfig-002-20260516 gcc-12.5.0
riscv allmodconfig clang-23
riscv allnoconfig clang-23
riscv allnoconfig gcc-15.2.0
riscv allyesconfig clang-16
riscv defconfig clang-23
riscv defconfig gcc-15.2.0
riscv randconfig-001-20260516 gcc-15.2.0
riscv randconfig-001-20260516 gcc-8.5.0
riscv randconfig-002-20260516 clang-16
riscv randconfig-002-20260516 gcc-15.2.0
s390 allmodconfig clang-18
s390 allmodconfig clang-19
s390 allnoconfig clang-23
s390 allyesconfig gcc-15.2.0
s390 defconfig clang-23
s390 defconfig gcc-15.2.0
s390 randconfig-001-20260516 clang-23
s390 randconfig-001-20260516 gcc-15.2.0
s390 randconfig-002-20260516 clang-23
s390 randconfig-002-20260516 gcc-15.2.0
sh allmodconfig gcc-15.2.0
sh allnoconfig clang-23
sh allnoconfig gcc-15.2.0
sh allyesconfig clang-19
sh allyesconfig gcc-15.2.0
sh defconfig gcc-14
sh edosk7705_defconfig gcc-15.2.0
sh randconfig-001-20260516 gcc-12.5.0
sh randconfig-001-20260516 gcc-15.2.0
sh randconfig-002-20260516 gcc-15.2.0
sparc allnoconfig clang-23
sparc allnoconfig gcc-15.2.0
sparc defconfig gcc-15.2.0
sparc randconfig-001-20260516 gcc-8.5.0
sparc randconfig-002-20260516 gcc-15.2.0
sparc randconfig-002-20260516 gcc-8.5.0
sparc64 allmodconfig clang-23
sparc64 defconfig gcc-14
sparc64 randconfig-001-20260516 gcc-10.5.0
sparc64 randconfig-001-20260516 gcc-8.5.0
sparc64 randconfig-002-20260516 clang-20
sparc64 randconfig-002-20260516 gcc-8.5.0
um allmodconfig clang-19
um allnoconfig clang-23
um allyesconfig gcc-14
um allyesconfig gcc-15.2.0
um defconfig gcc-14
um i386_defconfig gcc-14
um randconfig-001-20260516 clang-16
um randconfig-001-20260516 gcc-8.5.0
um randconfig-002-20260516 clang-18
um randconfig-002-20260516 gcc-8.5.0
um x86_64_defconfig gcc-14
x86_64 allmodconfig clang-20
x86_64 allnoconfig clang-20
x86_64 allnoconfig clang-23
x86_64 allyesconfig clang-20
x86_64 buildonly-randconfig-001-20260516 clang-20
x86_64 buildonly-randconfig-001-20260516 gcc-14
x86_64 buildonly-randconfig-002-20260516 gcc-14
x86_64 buildonly-randconfig-003-20260516 gcc-13
x86_64 buildonly-randconfig-003-20260516 gcc-14
x86_64 buildonly-randconfig-004-20260516 gcc-14
x86_64 buildonly-randconfig-005-20260516 gcc-14
x86_64 buildonly-randconfig-006-20260516 gcc-14
x86_64 defconfig gcc-14
x86_64 kexec clang-20
x86_64 randconfig-001-20260516 clang-20
x86_64 randconfig-001-20260516 gcc-14
x86_64 randconfig-002-20260516 clang-20
x86_64 randconfig-002-20260516 gcc-14
x86_64 randconfig-003-20260516 gcc-14
x86_64 randconfig-004-20260516 gcc-14
x86_64 randconfig-005-20260516 gcc-14
x86_64 randconfig-006-20260516 gcc-14
x86_64 randconfig-011 clang-20
x86_64 randconfig-011-20260516 clang-20
x86_64 randconfig-011-20260516 gcc-14
x86_64 randconfig-012 clang-20
x86_64 randconfig-012-20260516 clang-20
x86_64 randconfig-013 clang-20
x86_64 randconfig-013-20260516 clang-20
x86_64 randconfig-014 clang-20
x86_64 randconfig-014-20260516 clang-20
x86_64 randconfig-014-20260516 gcc-14
x86_64 randconfig-015 clang-20
x86_64 randconfig-015-20260516 clang-20
x86_64 randconfig-016 clang-20
x86_64 randconfig-016-20260516 clang-20
x86_64 randconfig-016-20260516 gcc-14
x86_64 randconfig-071-20260516 gcc-14
x86_64 randconfig-072-20260516 gcc-14
x86_64 randconfig-073-20260516 clang-20
x86_64 randconfig-073-20260516 gcc-14
x86_64 randconfig-074-20260516 clang-20
x86_64 randconfig-074-20260516 gcc-14
x86_64 randconfig-075-20260516 gcc-14
x86_64 randconfig-076-20260516 clang-20
x86_64 randconfig-076-20260516 gcc-14
x86_64 rhel-9.4 clang-20
x86_64 rhel-9.4-bpf gcc-14
x86_64 rhel-9.4-func clang-20
x86_64 rhel-9.4-kselftests clang-20
x86_64 rhel-9.4-kunit gcc-14
x86_64 rhel-9.4-ltp gcc-14
x86_64 rhel-9.4-rust clang-20
xtensa allnoconfig clang-23
xtensa allnoconfig gcc-15.2.0
xtensa allyesconfig clang-23
xtensa randconfig-001-20260516 gcc-13.4.0
xtensa randconfig-001-20260516 gcc-8.5.0
xtensa randconfig-002-20260516 gcc-8.5.0
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [Linaro-mm-sig] Re: [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: Barry Song @ 2026-05-16 9:19 UTC (permalink / raw)
To: T.J. Mercier
Cc: Albert Esteve, Christian König, Tejun Heo, Johannes Weiner,
Michal Koutný, Jonathan Corbet, Shuah Khan, Sumit Semwal,
Michal Hocko, Roman Gushchin, Shakeel Butt, Muchun Song,
Andrew Morton, Benjamin Gaignard, Brian Starkey, John Stultz,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan, cgroups, linux-doc,
linux-kernel, linux-media, dri-, linaro-mm-sig, linux-mm,
linux-security-module, selinux, linux-kselftest, mripard,
echanude
In-Reply-To: <CABdmKX3DhejYBis9htLDnzPrG7vuF3R3URLVNEbnyd61SSsx=g@mail.gmail.com>
On Thu, May 14, 2026 at 12:35 AM T.J. Mercier <tjmercier@google.com> wrote:
[...]
> > > I have a question about this part. Albert I guess you are interested
> > > only in accounting dmabuf-heap allocations, or do you expect to add
> > > __GFP_ACCOUNT or mem_cgroup_charge_dmabuf calls to other
> > > non-dmabuf-heap exporters?
> >
> > We're scoping this to dma-buf heaps for now. CMA heaps and the dmem
> > controller are on the radar for follow-up/parallel work (there will be
> > dragons and will surely need discussion). For DRM and V4L2 the
> > long-term intent is migration to heaps, which would make direct
> > accounting on those paths unnecessary.
>
> Ah I see. GEM buffers exported to dmabufs are what I had in mind. I
> guess this would only leave the odd non-DRM driver with the need to
> add their own accounting calls, which I don't expect would be a big
> problem.
>
sounds like we still have a long way to go to correctly account for
various v4l2, drm, GEM, CMA, etc. In patch 1, the charging is done in
dma_buf_export(), so I guess it covers all dma-buf types except
dma_heap, but the problem is that it has no remote charging support at
all?
> > udmabufs are already
> > memcg-charged, so adding a separate MEMCG_DMABUF would double count.
> > Are there any other exporters you had in mind that would benefit from
> > this approach?
> >
Thanks
Barry
^ permalink raw reply
* Re: [PATCH RFC 2/5] dma-heap: charge dma-buf memory via explicit memcg
From: Barry Song @ 2026-05-16 8:39 UTC (permalink / raw)
To: T.J. Mercier
Cc: Christian König, Albert Esteve, Tejun Heo, Johannes Weiner,
Michal Koutný, Jonathan Corbet, Shuah Khan, Sumit Semwal,
Michal Hocko, Roman Gushchin, Shakeel Butt, Muchun Song,
Andrew Morton, Benjamin Gaignard, Brian Starkey, John Stultz,
Christian Brauner, Paul Moore, James Morris, Serge E. Hallyn,
Stephen Smalley, Ondrej Mosnacek, Shuah Khan, cgroups, linux-doc,
linux-kernel, linux-media, dri-devel, linaro-mm-sig, linux-mm,
linux-security-module, selinux, linux-kselftest, mripard,
echanude
In-Reply-To: <CABdmKX2uwZ12kYJYPJGfWxuMBOJS=64b1GRj72tfB5D=NKM22w@mail.gmail.com>
On Wed, May 13, 2026 at 2:54 AM T.J. Mercier <tjmercier@google.com> wrote:
>
> On Tue, May 12, 2026 at 3:14 AM Christian König
> <christian.koenig@amd.com> wrote:
> >
> > On 5/12/26 11:10, Albert Esteve wrote:
> > > On embedded platforms a central process often allocates dma-buf
> > > memory on behalf of client applications. Without a way to
> > > attribute the charge to the requesting client's cgroup, the
> > > cost lands on the allocator, making per-cgroup memory limits
> > > ineffective for the actual consumers.
> > >
> > > Add charge_pid_fd to struct dma_heap_allocation_data. When set to
> > > a valid pidfd, DMA_HEAP_IOCTL_ALLOC resolves the target task's
> > > memcg and charges the buffer there via mem_cgroup_charge_dmabuf()
> > > inside dma_heap_buffer_alloc(). Without charge_pid_fd, and with
> > > the mem_accounting module parameter enabled, the buffer is charged
> > > to the allocator's own cgroup.
> > >
> > > Additionally, commit 3c227be90659 ("dma-buf: system_heap: account for
> > > system heap allocation in memcg") adds __GFP_ACCOUNT to system-heap
> > > page allocations. Keeping __GFP_ACCOUNT would charge the same pages
> > > twice (once to kmem, once to MEMCG_DMABUF), thus remove it and route
> > > all accounting through a single MEMCG_DMABUF path.
> > >
> > > Usage examples:
> > >
> > > 1. Central allocator charging to a client at allocation time.
> > > The allocator knows the client's PID (e.g., from binder's
> > > sender_pid) and uses pidfd to attribute the charge:
> > >
> > > pid_t client_pid = txn->sender_pid;
> > > int pidfd = pidfd_open(client_pid, 0);
> > >
> > > struct dma_heap_allocation_data alloc = {
> > > .len = buffer_size,
> > > .fd_flags = O_RDWR | O_CLOEXEC,
> > > .charge_pid_fd = pidfd,
> > > };
> > > ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
> > > close(pidfd);
> > > /* alloc.fd is now charged to client's cgroup */
> > >
> > > 2. Default allocation (no pidfd, mem_accounting=1).
> > > When charge_pid_fd is not set and the mem_accounting module
> > > parameter is enabled, the buffer is charged to the allocator's
> > > own cgroup:
> > >
> > > struct dma_heap_allocation_data alloc = {
> > > .len = buffer_size,
> > > .fd_flags = O_RDWR | O_CLOEXEC,
> > > };
> > > ioctl(heap_fd, DMA_HEAP_IOCTL_ALLOC, &alloc);
> > > /* charged to current process's cgroup */
> > >
> > > Current limitations:
> > >
> > > - Single-owner model: a dma-buf carries one memcg charge regardless of
> > > how many processes share it. Means only the first owner (and exporter)
> > > of the shared buffer bears the charge.
> > > - Only memcg accounting supported. While this makes sense for system
> > > heap buffers, other heaps (e.g., CMA heaps) will require selectively
> > > charging also for the dmem controller.
> >
> > Well that doesn't looks soo bad, it at least seems to tackle the problem at hand for Android and some of other embedded use cases.
>
> Yeah I think this might work. I know of 3 cases, and it trivially
> solves the first two. The third requires some work on our end to
> extend our userspace interfaces to include the pidfd but it seems
> doable. I'm checking with our graphics folks.
>
> 1) Direct allocation from user (e.g. app -> allocation ioctl on
> /dev/dma_heap/foo)
> No changes required to userspace. mem_accounting=1 charges the app.
>
> 2) Single hop remote allocation (e.g. app -> AHardwareBuffer_allocate
> -> gralloc)
> gralloc has the caller's pid as described in the commit message. Open
> a pidfd and pass it in the dma_heap_allocation_data.
>
> 3) Double hop remote allocation (e.g. app -> dequeueBuffer ->
> SurfaceFlinger -> gralloc)
> In this case gralloc knows SurfaceFlinger's pid, but not the app's. So
> we need to add the app's pidfd to the SurfaceFlinger -> gralloc
> interface, or transfer the memcg charge from SurfaceFlinger to the app
> after the allocation.
> It'd be nice to avoid the charge transfer option entirely, but if we
> need it that doesn't seem so bad in this case because it's a bulk
> charge for the entire dmabuf rather than per-page. So the exporter
> doesn't need to get involved (we wouldn't need a new dma_buf_op) and
> we wouldn't have to worry about looping and locking for each page.
>
Hi T.J.,
Your description of the three different cases sounds very interesting.
It helps me understand how difficult it can be to correctly charge
dma-buf in the current user scenarios.
I’m wondering where I can find Android userspace code that transfers
the PID of RPC callers. Do we have any existing sample code in Android
for this?
> > I'm just not sure if this is future prove and will work for all use cases, e.g. cloud gaming, native context for automotive etc...
Thanks
Barry
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox