* [PATCH v4 00/17] mm/mglru: frequency guided promotion (MGLRU-FG) and flag cleanup
@ 2026-10-15 18:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
Hi all, this cover letter is going to be very long, as it includes a
systematic test and analysis. And moving this series out of RFC phase
as almost all readings are positive, tested on at least five different
platforms (desktop, large server, Android, x86_64 and arm64 boxes)
covering typical benchmarks, database, real applications, and has
good test reports from other individual testers. It's also pretty clear
about the next steps, and this blocks or breaks nothing AFAIK, which
should be a pure gain, improving the performance or reducing IO of many
workloads by ~10-40%.
Still following the idea proposed at LSF/MM/BPF [1], based on current
mm-new. Different workloads get a minor or dramatic improvement,
while reducing the page flags usage by 1 bit. The gain here is mostly
from a real improvement of LRU's ability to distinguish the hotter
workingset. Tested across multiple servers of different archs, desktops,
and Android.
It also fixes several long-standing issues including under-accounted PSI
and poor workingset tracking (especially for page cache).
Test results (CLRU means classical LRU):
Build kernel test:
==================
Running make -j48 in a 3G memcg, using ZRAM as swap (256G, lzo-rle) and
holding the kernel and build output on a NVMe drive, 3 runs of 5 swappiness
configurations (15 builds per kernel) [2]; the patched version is better
than mainline at every swappiness value, measuring the total on average:
real sys pgpgin pswpin pswpout refault_file refault_anon
CLRU 1m39s 14m59s 21.47M 2.81M 11.47M 2.40M 2.87M
Before 1m32s 9m18s 12.57M 2.10M 9.05M 476k 2.07M
After 1m32s 9m01s 9.99M 1.62M 8.40M 369k 1.60M
delta -0.4s -17s(-3%) -20.5% -22.4% -7.1% -22.4% -22.6%
Same -j48 test, disk swap instead of ZRAM (SSD-backed; the kernel and build
output stay on NVMe). The regression reported for earlier revisions is gone:
it is now a small net win, with every I/O counter improved, at the cost of
~1% more system time spent on the extra promotion work:
real sys pgpgin pswpin pswpout refault_file refault_anon
CLRU 4m22s 21m54s 36.80M 3.00M 10.56M 6.03M 3.05M
Before 2m51s 10m12s 10.04M 1.55M 5.02M 416k 1.02M
After 2m48s 10m18s 8.95M 1.41M 4.93M 341k 931k
delta -4s(-2%) +6s(+1%) -10.9% -8.8% -1.8% -18.1% -9.0%
FIO cached read with zipf
=========================
Using an NVMe disk, in a 16G cgroup, total file size 40G, this measures
the LRU's theoretical ability to distinguish the hotter portion, 3 test
runs per config:
Using read()
------------
command: numactl --membind=0 --cpunodebind=0 \
fio --name=fg --numjobs=16 --nrfiles=1 \
--filename_format="/var/lib/kmultibench/realdk.\$jobnum.img" \
--size=2560M \
--buffered=1 --ioengine=sync --rw=randread \
--random_distribution=zipf:$ZIPF --bs=4k --time_based \
--ramp_time=45s --runtime=600s --group_reporting
READING: Avg IOPS (higher is better)
zipf:θ CLRU | MGLRU before (vs CLRU) | FG (vs CLRU) | Imprv. |
0.8 397,572| 396,046 ( -0.4%) | 449,714 (+13.1%) | +13.6% |
0.9 605,862| 602,201 ( -0.6%) | 692,524 (+14.3%) | +15.0% |
1.1 2,092,280| 2,122,415 ( +1.4%) | 2,457,564 (+17.5%) | +15.8% |
1.2 4,444,665| 4,513,839 ( +1.6%) | 5,066,901 (+14.0%) | +12.3% |
READING: refault/read (lower is better)
zipf:θ CLRU | MGLRU before (vs CLRU) | FG (vs CLRU) | Imprv. |
0.8 0.20108 | 0.20074 (-0.2%) | 0.17660 (-12.2%) | +12.0% |
0.9 0.13356 | 0.13186 (-1.3%) | 0.11522 (-13.7%) | +12.6% |
1.1 0.03641 | 0.03586 (-1.5%) | 0.03065 (-15.8%) | +14.5% |
1.2 0.01479 | 0.01466 (-0.8%) | 0.01230 (-16.9%) | +16.1% |
For typical read() following zipf distributions, MGLRU was basically the
same as CLRU before or slightly worse in certain cases, now it's ~12-16%
better and leading. Worth noting we also observed in certain systems MGLRU
was worse than CLRU before in v2 - v3, which is now also optimized into a
gain.
Using mmap
----------
command: numactl --membind=0 --cpunodebind=0 \
fio --name=fg --numjobs=16 --nrfiles=1 \
--filename_format="/var/lib/kmultibench/realdk.\$jobnum.img" \
--size=2560M \
--ioengine=mmap --rw=randread --fadvise_hint=0 \
--random_distribution=zipf:$ZIPF --bs=4k --time_based \
--ramp_time=45s --runtime=600s --group_reporting
READING: Avg IOPS (higher is better)
zipf:θ CLRU | MGLRU before (vs CLRU) | FG (vs CLRU) | Imprv. |
0.8 1,521,315| 3,610,516 (+137.3%) | 3,980,874 (+161.7%) | +10.3% |
0.9 926,158| 3,856,644 (+316.4%) | 4,423,859 (+377.7%) | +14.7% |
1.1 2,303,035| 2,225,641 ( -3.4%) | 2,715,460 (+17.9%) | +22.0% |
1.2 5,015,274| 4,811,580 ( -4.1%) | 5,357,325 ( +6.8%) | +11.3% |
READING: refault/read (lower is better)
zipf:θ CLRU | MGLRU before (vs CLRU) | FG (vs CLRU) | Imprv. |
0.8 0.05403| 0.02009 (-62.8%) | 0.01805 (-66.6%) | +10.2% |
0.9 0.08937| 0.01899 (-78.8%) | 0.01638 (-81.7%) | +13.8% |
1.1 0.03549| 0.03644 ( +2.7%) | 0.02925 (-17.6%) | +19.7% |
1.2 0.01529| 0.01569 ( +2.6%) | 0.01375 (-10.1%) | +12.4% |
For typical mmap read of file following zipf distributions, MGLRU was
behaving much better for theta < 1 and with a slight regression for
theta > 1. And FG greatly improved both cases, no more regression in any
case with an improvement ~10-20%.
Worth noting, without fadvise_hint=0, we will see a regression like this:
zipf:θ CLRU | MGLRU before (vs CLRU) | FG (vs CLRU) | Imprv. |
1.1 1,873,230| 1,641,981 (-12.3%) | 1,648,997 (-12.0%) | +0.4% |
That's because MGLRU deliberately disables hotness tracking for MADV_RANDOM
areas and meanwhile for CLRU refault distance is still taking effect. I
don't think this is an issue or blocker as that's exactly what the
userspace asked and we also have plan to unify or improve shadow or
refault distance usage [9], [10].
Tighter cache
-------------
command: following the mmap test, adjusting cgroup limit and zipf theta, 2
tests per case:
READING: Avg IOPS (higher is better)
memcg/θ CLRU | MGLRU before (vs CLRU) | FG (vs CLRU) | Imprv. |
16G/1.4 20,563,861| 20,408,851 ( -0.8%) | 21,314,656 ( +3.7%) | +4.4% |
8G/1.4 14,606,000| 13,650,929 ( -6.5%) | 14,603,348 ( 0.0%) | +7.0% |
4G/1.4 10,102,227| 9,392,019 ( -7.0%) | 9,992,258 ( -1.1%) | +6.4% |
12G/1.2 3,688,008| 3,502,117 ( -5.0%) | 3,795,627 ( +2.9%) | +8.4% |
8G/1.2 2,726,922| 2,511,410 ( -7.9%) | 2,735,389 ( +0.3%) | +8.9% |
4G/1.2 1,906,548| 1,672,370 (-12.3%) | 1,785,545 ( -6.3%)*| +6.8% |
12G/1.1 1,618,437| 1,520,102 ( -6.1%) | 1,641,287 ( +1.4%) | +8.0% |
8G/1.1 1,209,058| 1,100,118 ( -9.0%) | 1,196,796 ( -1.0%) | +8.8% |
6G/1.1 1,024,652| 910,669 (-11.1%) | 1,030,723 ( +0.6%) | +13.2% |
READING: refault per read (lower is better)
memcg:θ CLRU | MGLRU before (vs CLRU) | FG (vs CLRU) | Imprv. |
16G/1.4 0.00205 | 0.00210 ( +2.4%) | 0.00180 (-12.2%) | +14.3% |
8G/1.4 0.00392 | 0.00430 ( +9.7%) | 0.00379 ( -3.3%) | +11.9% |
4G/1.4 0.00645 | 0.00722 (+11.9%) | 0.00650 ( +0.8%) | +10.0% |
12G/1.2 0.02169 | 0.02245 ( +3.5%) | 0.02055 ( -5.3%) | +8.5% |
8G/1.2 0.02944 | 0.03209 ( +9.0%) | 0.02875 ( -2.3%) | +10.4% |
4G/1.2 0.04459 | 0.04897 ( +9.8%) | 0.04388 ( -1.6%)*| +10.4% |
12G/1.1 0.05159 | 0.05361 ( +3.9%) | 0.04907 ( -4.9%) | +8.5% |
8G/1.1 0.06830 | 0.07387 ( +8.2%) | 0.06708 ( -1.8%) | +9.2% |
6G/1.1 0.08064 | 0.08816 ( +9.3%) | 0.08046 ( -0.2%) | +8.7% |
It can be clearly seen that MGLRU was worse than CLRU before under tight
cache usage, and with FG all cases improved. And the missing rate measured
by refault per read shows FG is better than unpatched MGLRU and CLRU in
almost every configuration.
If we re-run the only obvious regression, the 4G/1.2 case (*), with perf,
the bottleneck is the lruvec lock, with the contention concentrated in
try_to_shrink_lruvec() where the lazy promotion work happens:
command:
<same test command above, with 4G memcg and zipf:1.2, and below tracing>
perf lock contention -ab -- sleep 355
READING (runtime=300s, 4G memcg, zipf:1.2):
config:θ IOPS | mean clat | sys% | lruvec lock total wait |
CLRU 1,868k| 8195 ns| 6.01%| ~10 s |
MGLRU before 1,677k| 9173 ns| 7.25%| ~75 s |
MGLRU after 1,837k| 8488 ns| 7.33%| ~78 s |
The wait column is a per-run total. MGLRU now promotes more folios either
due to PID protection or FG promotion while holding the LRU lock. And
FG achieves comparable or even better performance by reducing actual IO
already, which is a net improvement compared to unpatched MGLRU. The
remaining gap over CLRU is inherited lock cost, which can be reduced
by batching and prefetching during scanning as demonstrated before [13]
(and interestingly, you can see the benchmark kind of matches the
improvement that I measured at that time).
MongoDB YCSB workloadb [3]
==========================
With recordcount:20000000 operationcount:6000000, threads:48,
in a 16G memcg, 3 runs:
CLRU: 97,856.45 ops/s
MGLRU Before: 85,702.39 ops/s (-12.4% vs CLRU)
MGLRU After: 98,569.77 ops/s ( +0.7% vs CLRU) +15.0% vs Before
MGLRU was doing worse than CLRU before, now it's doing better.
Chromium & Node.js test [4]
===========================
Using ZRAM as swap, on a 48c96t machine with 128G memory, 64 workers, run
for 1 hour:
Total requests:
CLRU: 63822
MGLRU Before: 130650
MGLRU After: 228864 (+75.1%)
Worth noting there seems to be some reclaim balance jitter for this test
but MGLRU is far better already and now leading even more.
LevelDB Scan/Get
================
I also retested the LevelDB benchmark from the cache_ext paper [5].
Interestingly, mainline MGLRU already beats CLRU on this one after a
recent change in lru_gen_folio_seq that bumps new folios with refs == 1
to the second-oldest generation as discussed. That change accidentally
gave random reads a higher hotness level while making sequential reads
much colder as described before [14]. However it's unreliable and
incorrect because it depends on LRU drain timing, and hurts some workloads
as reported, or where the sequential part is actually hotter (any workload
involving hotter large file and many small cold files will be affected)
which is reproduced below.
This series improves it while covering ordinary workloads too and should
fix regressions:
LevelDB Scan / Get, Throughput Total:
CLRU: 4708.3 ops/s
MGLRU: 5026.9 ops/s (faster than CLRU, but hurts other workloads)
MGLRU After: 5179.2 ops/s (even better in all cases, and no regression)
The hot-sequential and cold-random workload can be easily reproduced with
SQLite and grep. SQLite continuously scans and looks up a small hot
portion of a DB file, while grep iterates over a set of small files much
larger than RAM [6]:
SQLite scan & lookup time: Grep iterate time:
CLRU: 14.51ms 13281.37ms
MGLRU mainline: 567.05ms 13694.47ms
MGLRU After this series: 10.58ms 12930.43ms
The grep cold portion is larger than RAM and accessed only once per
iteration, so there's no promotion of any of it. CLRU handles
this reasonably; mainline MGLRU has a clear regression; MGLRU-FG now
not only recovers but is able to catch some hot parts from the cold grep
workload. This test is somewhat subjective, but the signal is clear.
Android (For reference)
=======================
Testing on Android is quite difficult: lack of mainline support, and it is
very noisy with real apps to get a stable result. I did an informal
backport of the latest MGLRU-FG patches onto the 6.1 GKI tree of a
Pixel 9 Pro (16 GB RAM, 8 GB ZRAM, Android 17), preserving the frozen
kABI layouts so the implementation is limited (see the code here [15]).
The "before" and "after" kernels share the same base and differ only by
the FG series: the "before" tree has MGLRU aligned to the current upstream
code, so that unrelated backport deltas cancel out. (And BTW that alignment
also improved the performance by a lot, which matches the report from
previous MGLRU reclaim optimization series [12]).
The workload is a churn loop driven over adb: 34 common apps and 20 Chrome
tabs are launched and cycled, with a brief scroll per app during the cold
build of each iteration, 16 iterations per run, about one hour per run. The
loop exhausts memory in every iteration (zram peaks at 6.9-7.8 GB of 8 GB,
MemFree down to 130-335 MB in every run). The two kernels are run in an
interleaved rotation and compared both on matched per-iteration samples
(same period, same iteration index) and on paired run totals, which
cancels most of the run-to-run drift. I've repeated the test for several
days and times getting similar results.
The memory-management stack in Android is built around the non-FG MGLRU
behaviour, and the 6.1 base lacks some of the upstream infrastructure. Even
so, FG holds its own, it lowers reclaim traffic on both the file and the anon
side:
Before After Delta 95% CI Avg Improv.
-----------------------------------------------------------------------
workingset_refault_anon 1.43M 1.25M [-23%, -3%] +12%
workingset_refault_file 1.54M 1.39M [-17%, +2%] +10%
pgscan_anon 4.04M 3.56M [-20%, -5%] +12%
pgsteal_anon 1.64M 1.45M [-23%, -1%] +12%
pgscan_file 7.49M 6.52M [-17%, -5%] +13%
pgsteal_file 6.55M 5.63M [-18%, -7%] +14%
pswpout 1.82M 1.62M [-22%, -1%] +11%
pgpgin 23.56M 20.33M [-18%, -7%] +14%
pgmajfault 1.61M 1.42M [-21%, -4%] +12%
pgpgout 3.39M 3.48M [ +0%, +5%] -3%
"Before" is the aligned tree without the FG series, "After" adds FG. The
numbers are trimmed means over the 12 interleaved runs per kernel: for each
counter's avg, the highest and lowest run are dropped and the middle ten
averaged. Each counter is the vmstat delta of one full run. The 95% CI is
a bootstrap over the 12 rounds of the paired run-total ratio (After vs
Before).
The environment is noisy, but given the fact that every counter except
pgpgout is improved by ~10% on average and the 95% CI reading is also
looking good, and we have been seeing positive readings since v2 of this
series, I think we can safely say that this series reduced IO by about
10% for the same Android workload and has a very positive effect on it.
I also ran the Android Jank test from Zicheng [7] on this device (chrome
scroll, FrameTimeline), plus a 34-app keepalive run. No obvious difference
was observed: the run-mean chrome scroll fps of every kernel sits in a
100.6-103.6 band on a 120 Hz panel, and the jank ratios are all below 8%
per scroll round with no consistent ordering across kernels. The keepalive
test does not discriminate on this build: 31-32 of 34 apps stay alive
with every kernel. On the previous build a run kept slightly more apps
alive with FG (average 4.8 vs 4.5, peak 23 vs 20), that could be noise
or a slight improvement. Both kernels saturate all 8 cores with no obvious
CPU usage difference (99.8-99.9% utilization, ~518s busy per 65s window).
I also did a test on another Android phone with a 5.15 kernel (Xperia 1 V,
code here [16]), which has all apps in one global memcg (this Pixel 9 Pro
has each app in a separate memcg). The result looks much better there,
either due to the memcg layout or the non-reclaiming anon shadow (the 6.1
kernel reclaims anon shadow). But in either case, the performance is
a positive reading.
So in summary: with this revision we get lower file refaults, lower
pgpgin, and lower anon reclaim and swap traffic.
Redis / Valkey
==============
Valkey in a 1.5G ARM64 VM using 8G ZRAM as swap:
valkey-server --maxmemory 2560M
redis-benchmark -r 3000000 -n 3000000 -d 1024 -c 12 -P 32 -t get
With BGSAVE running Without BGSAVE running
Before: 308356.47 RPS 431468.44 TPS
After: 336300.34 RPS (+9.0%) 442151.81 TPS (+2.4%)
Looking good with some improvement, tested 3 times. Note on that VM we have
$ cat /sys/kernel/mm/lru_gen/enabled
0x0001
So page table walker is not available, but still working well.
Others
======
Additionally, PSI, smaps, and readahead should all benefit from better
accuracy since this series unifies the flag usage between classical
LRU and MGLRU.
Other tests such as MySQL are looking fine, with no regressions. There is
also community test report [8].
Refault distance is not included yet, so MGLRU may respond more slowly to
workingset shifts. That can be added later, as previously demonstrated
[9], [10].
Extra note about future development: this series is actually highly
compatible with ideas like workingset reporting [11]. The "gen climbing
folio" design may appear to conflict with workingset reporting's idea,
but it doesn't. The solution is simple and straightforward: once we can
extend the generation number to a larger value (e.g. 64 or 128), the
refs-driven promotion can stop at a lower gen (e.g. oldest_gen + 16),
leaving the remaining newer generations as perfectly time-gap-separated
bins.
The tier count is not fixed either; we'll need to find a way to tune it
if tiers go beyond 4 though, but that shouldn't be hard, not a blocker
either.
More details are in the individual commit messages. LLM is used to help
improve the comments and tests; all results are double checked.
Link: https://lore.kernel.org/linux-mm/CAMgjq7BoekNjg-Ra3C8M7=8=75su38w=HD782T5E_cxyeCeH_g@mail.gmail.com/ [1]
Link: https://lore.kernel.org/linux-mm/CAGsJ_4xre-x0e+qNVm=KLFnO1dbPkPX5RuecqwvTZu-vS+o8yQ@mail.gmail.com/ [2]
Link: https://github.com/brianfrankcooper/YCSB/blob/master/workloads/workloadb [3]
Link: https://lore.kernel.org/all/20221220214923.1229538-1-yuzhao@google.com/ [4]
Link: https://dl.acm.org/doi/10.1145/3731569.3764820 [5]
Link: https://github.com/ryncsn/emm-test-project/tree/master/sqlite-grep [6]
Link: https://github.com/purplewall1206/android-perf-bench [7]
Link: https://lore.kernel.org/linux-mm/92DCEFFD13221261+20260828102344.1537874-1-zhaozhengzhuo@uniontech.com/ [8]
Link: https://lwn.net/Articles/945266/ [9]
Link: https://lore.kernel.org/linux-mm/20260502-mglru-fg-v1-0-913619b014d9@tencent.com/ [10]
Link: https://lwn.net/Articles/976985/ [11]
Link: https://lore.kernel.org/linux-mm/20260417025123.2971253-1-wxy2009nrrr@163.com/ [12]
Link: https://lore.kernel.org/linux-mm/20240123184552.59758-1-ryncsn@gmail.com/ [13]
Link: https://lore.kernel.org/all/CAMgjq7Agqr1FLcG3HqT_M_1s8cHc21qQKTM-JS67-3gfQ6azPQ@mail.gmail.com/ [14]
Link: https://github.com/ryncsn/linux/tree/kasong/android/android14-6.1-mglru-fg-v4c-20261010 [15]
Link: https://github.com/ryncsn/linux/tree/silsrc/android/android13-5.15-yodo [16]
Assisted-by: LLM
Signed-off-by: Kairui Song <kasong@tencent.com>
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
---
Changes in v4:
- Defer folios at the tail of their LRU instead of min_seq only:
https://lore.kernel.org/linux-mm/CAMgjq7AOhDkuZZnWrGDguMvNtarZ7s0Ouzs1xYi2O4AkPEPbTQ@mail.gmail.com/
- Re-read max_seq inside the folio_inc_lru_refs_walk() cmpxchg loop, and
apply the reparent fix before the lockless change.
- Tidy up a few code shuffles.
- Use generic refs-based helper for smaps, DAMON, khugepaged
and madvise with better kdocs.
- Add community Tested-by tags for unchanged parts.
- Link to v3: https://patch.msgid.link/20261003-mglru-fg-v3-0-cbd4546a5bd9@tencent.com
Changes in v3:
- Micro-optimization for page flags operations: if the flags are
unchanged after calculation, skip the cmpxchg.
- Avoid touching folio_activate even in the cold path. I tried multiple
ways for that lockless promotion, PG_lru in the earlier RFC, and
previous folio_activate, the new speculative barrier + retry seem the
best solution.
- Apply tier cap for MADV_PAGEOUT.
- Only promote folio during page table walk or rmap if the folio
is in the min gen. May be worth trying to enlarge the range to
(max_seq - MIN_NR_GENS) in the next version.
- Fix a potential folio leak caused by reparenting.
- Retest shows great results especially for the Android case.
- Link to v2: https://patch.msgid.link/20260911-mglru-fg-v2-0-f26e5cb26da7@tencent.com
Changes in v2:
- Rebased; dropped v1 03/05/06 (already upstream), folded v1 01 and 07
into patches 01 and 04; new patches 06, 07, 10, 12, 13.
- Make folio_test_workingset() itself arbitrate, dropping the parallel
folio_is_* helpers; convert the last raw PageWorkingset() user
(erofs). (Johannes)
- Use LRU_REF_MAPPED/LRU_REF_EXEC flags instead of is_fault/is_exec
booleans. (Barry)
- Fix syzbot "WARNING in folio_inc_lru_refs" on off-LRU folios.
- Account active/inactive per folio from refs, not the gen window:
/proc/vmstat and memory.stat no longer jump on aging or reverse on
swapless machines.
- Make folio_inc_lru_refs() lockless; add folio_inc_lru_refs_fast()
for the gup fast paths.
- Convert DAMON and khugepaged to the refs-based operations.
- Drop the lru_size WARN_ON_ONCE() and lockdep_assert_held(): the
counter is lockless now, so transient negatives are expected.
- Link to v1: https://patch.msgid.link/20260804-mglru-fg-v1-0-4d8dad39dad6@tencent.com
---
Kairui Song (17):
mm/memcontrol: allow update of LRU statistic without holding LRU lock
mm/mglru: make generation page counters atomic
mm/memcg: add folio-based lruvec live helper
mm/mglru: frequency guided workingset promotion (MGLRU-FG)
mm/mglru: make folio lru referenced times count a generic API
mm/mglru: move add/del LRU size accounting out of lru_gen_update_size()
mm/mglru, gup: mark folios referenced via a fast helper
mm/mglru, smaps: convert to LRU refs based operations for MGLRU
mm/damon: convert to LRU refs based operations for MGLRU
mm/khugepaged: convert to LRU refs based operations for MGLRU
mm/madvise: adapt for LRU refs based operations for MGLRU
mm/huge_memory: mark folio as accessed more accurately on split
mm/mglru: folio LRU refs based active/inactive number accounting
mm/mglru: reparent folios from all generations
mm/mglru: make folio_inc_lru_refs lruvec lockless
mm/mglru: make folio_test_workingset() work based on folio LRU refs
Documentation/mm: multi-gen LRU: update for frequency guided promotion
Documentation/admin-guide/mm/multigen_lru.rst | 4 +-
Documentation/mm/multigen_lru.rst | 57 +--
fs/btrfs/compression.c | 1 +
fs/erofs/zdata.c | 3 +-
fs/proc/task_mmu.c | 6 +-
include/linux/memcontrol.h | 39 ++
include/linux/mm_inline.h | 354 ++++++++++++------
include/linux/mmzone.h | 169 ++++++---
include/linux/page-flags.h | 2 -
kernel/bounds.c | 2 +-
mm/damon/paddr.c | 3 +-
mm/filemap.c | 1 +
mm/folio.c | 85 +----
mm/gup.c | 6 +-
mm/huge_memory.c | 8 +-
mm/khugepaged.c | 4 +-
mm/madvise.c | 57 ++-
mm/migrate.c | 2 -
mm/page_io.c | 1 +
mm/vmscan.c | 520 +++++++++++++++++---------
mm/workingset.c | 45 ++-
21 files changed, 875 insertions(+), 494 deletions(-)
---
base-commit: 71a9b024444eba184335b67867897714e4da143c
change-id: 20260722-mglru-fg-3a2c8574725b
Best regards,
--
Kairui Song <kasong@tencent.com>
^ permalink raw reply [flat|nested] 59+ messages in thread* [PATCH v4 01/17] mm/memcontrol: allow update of LRU statistic without holding LRU lock
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:15 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
To enable moving file pages in folio_mark_accessed directly and lazily
for MGLRU, allow updating the LRU statistic atomically without holding a
lock. It may cause temporary counter underflow, which should be fine as
we still follow final consistency of the counter, and it only serves as
a estimated metrics, a little inaccuracy has no visible effect.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index ab69b9930893..597f013c8e04 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -47,11 +47,10 @@ static __always_inline void __update_lru_size(struct lruvec *lruvec,
{
struct pglist_data *pgdat = lruvec_pgdat(lruvec);
- lockdep_assert_held(&lruvec->lru_lock);
WARN_ON_ONCE(nr_pages != (int)nr_pages);
mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
- __mod_zone_page_state(&pgdat->node_zones[zid],
+ mod_zone_page_state(&pgdat->node_zones[zid],
NR_ZONE_LRU_BASE + lru, nr_pages);
}
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 01/17] mm/memcontrol: allow update of LRU statistic without holding LRU lock
@ 2026-10-15 18:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
To enable moving file pages in folio_mark_accessed directly and lazily
for MGLRU, allow updating the LRU statistic atomically without holding a
lock. It may cause temporary counter underflow, which should be fine as
we still follow final consistency of the counter, and it only serves as
a estimated metrics, a little inaccuracy has no visible effect.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index ab69b9930893..597f013c8e04 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -47,11 +47,10 @@ static __always_inline void __update_lru_size(struct lruvec *lruvec,
{
struct pglist_data *pgdat = lruvec_pgdat(lruvec);
- lockdep_assert_held(&lruvec->lru_lock);
WARN_ON_ONCE(nr_pages != (int)nr_pages);
mod_lruvec_state(lruvec, NR_LRU_BASE + lru, nr_pages);
- __mod_zone_page_state(&pgdat->node_zones[zid],
+ mod_zone_page_state(&pgdat->node_zones[zid],
NR_ZONE_LRU_BASE + lru, nr_pages);
}
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 01/17] mm/memcontrol: allow update of LRU statistic without holding LRU lock
2026-10-15 18:15 ` Kairui Song
(?)
@ 2026-10-15 18:32 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:32 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> To enable moving file pages in folio_mark_accessed directly and lazily
> for MGLRU, allow updating the LRU statistic atomically without holding a
> lock. It may cause temporary counter underflow, which should be fine as
> we still follow final consistency of the counter, and it only serves as
> a estimated metrics, a little inaccuracy has no visible effect.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=1
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 02/17] mm/mglru: make generation page counters atomic
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:15 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
No feature change, convert them to atomic so we can update them without
holding the LRU lock. There is no risk of overflow. The reader always
compares and uses zero instead if the counter values are negative. It
follows final consistency.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 6 ++----
include/linux/mmzone.h | 2 +-
mm/vmscan.c | 27 ++++++++++++---------------
3 files changed, 15 insertions(+), 20 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 597f013c8e04..f52f02e8e5be 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -265,11 +265,9 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
VM_WARN_ON_ONCE(old_gen == -1 && new_gen == -1);
if (old_gen >= 0)
- WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
- lrugen->nr_pages[old_gen][type][zone] - delta);
+ atomic_long_sub(delta, &lrugen->nr_pages[old_gen][type][zone]);
if (new_gen >= 0)
- WRITE_ONCE(lrugen->nr_pages[new_gen][type][zone],
- lrugen->nr_pages[new_gen][type][zone] + delta);
+ atomic_long_add(delta, &lrugen->nr_pages[new_gen][type][zone]);
/* addition */
if (old_gen < 0) {
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 3b96d6c7123b..09ce82fa841a 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -554,7 +554,7 @@ struct lru_gen_folio {
/* the multi-gen LRU lists, lazily sorted on eviction */
struct list_head folios[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
/* the multi-gen LRU sizes, eventually consistent */
- long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
+ atomic_long_t nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
/* the exponential moving average of refaulted */
unsigned long avg_refaulted[ANON_AND_FILE][MAX_NR_TIERS];
/* the exponential moving average of evicted+protected */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index daa7e37c2236..cab3ba14d20c 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3495,8 +3495,7 @@ static void reset_batch_size(struct lru_gen_mm_walk *walk)
continue;
walk->nr_pages[gen][type][zone] = 0;
- WRITE_ONCE(lrugen->nr_pages[gen][type][zone],
- lrugen->nr_pages[gen][type][zone] + delta);
+ atomic_long_add(delta, &lrugen->nr_pages[gen][type][zone]);
if (lru_gen_is_active(lruvec, gen))
lru += LRU_ACTIVE;
@@ -4116,11 +4115,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
break;
}
flush_lru_batch(head, &batch_end, target_list);
-
- WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
- lrugen->nr_pages[old_gen][type][zone] - delta);
- WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
- lrugen->nr_pages[target_gen][type][zone] + delta);
+ atomic_long_sub(delta, &lrugen->nr_pages[old_gen][type][zone]);
+ atomic_long_add(delta, &lrugen->nr_pages[target_gen][type][zone]);
if (!remaining)
return false;
}
@@ -4225,8 +4221,8 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
for (type = 0; type < ANON_AND_FILE; type++) {
for (zone = 0; zone < MAX_NR_ZONES; zone++) {
enum lru_list lru = type * LRU_INACTIVE_FILE;
- long delta = lrugen->nr_pages[prev][type][zone] -
- lrugen->nr_pages[next][type][zone];
+ long delta = atomic_long_read(&lrugen->nr_pages[prev][type][zone]) -
+ atomic_long_read(&lrugen->nr_pages[next][type][zone]);
if (!delta)
continue;
@@ -4344,7 +4340,8 @@ static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness
for (seq = min_seq[type]; seq <= max_seq; seq++) {
gen = lru_gen_from_seq(seq);
for (zone = 0; zone < MAX_NR_ZONES; zone++)
- total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+ total += max(atomic_long_read(&lrugen->nr_pages[gen][type][zone]),
+ 0L);
}
}
@@ -4764,7 +4761,7 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
for (i = 0; i < get_nr_gens(child_lruvec, type); i++) {
int gen = lru_gen_from_seq(child_lrugen->max_seq - i);
- long nr_pages = child_lrugen->nr_pages[gen][type][zone];
+ long nr_pages = atomic_long_read(&child_lrugen->nr_pages[gen][type][zone]);
int child_lru_active = lru_gen_is_active(child_lruvec, gen) ? LRU_ACTIVE : 0;
int parent_lru_active = lru_gen_is_active(parent_lruvec, gen) ? LRU_ACTIVE : 0;
@@ -4772,9 +4769,8 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
list_splice_tail_init(&child_lrugen->folios[gen][type][zone],
&parent_lrugen->folios[gen][type][zone]);
- WRITE_ONCE(child_lrugen->nr_pages[gen][type][zone], 0);
- WRITE_ONCE(parent_lrugen->nr_pages[gen][type][zone],
- parent_lrugen->nr_pages[gen][type][zone] + nr_pages);
+ atomic_long_set(&child_lrugen->nr_pages[gen][type][zone], 0);
+ atomic_long_add(nr_pages, &parent_lrugen->nr_pages[gen][type][zone]);
if (lru_gen_is_active(child_lruvec, gen) != lru_gen_is_active(parent_lruvec, gen)) {
__update_lru_size(child_lruvec, lru + child_lru_active, zone, -nr_pages);
@@ -5859,7 +5855,8 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)
char mark = full && seq < min_seq[type] ? 'x' : ' ';
for (zone = 0; zone < MAX_NR_ZONES; zone++)
- size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+ size += max(atomic_long_read(&lrugen->nr_pages[gen][type][zone]),
+ 0L);
seq_printf(m, " %10lu%c", size, mark);
}
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 02/17] mm/mglru: make generation page counters atomic
@ 2026-10-15 18:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
No feature change, convert them to atomic so we can update them without
holding the LRU lock. There is no risk of overflow. The reader always
compares and uses zero instead if the counter values are negative. It
follows final consistency.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 6 ++----
include/linux/mmzone.h | 2 +-
mm/vmscan.c | 27 ++++++++++++---------------
3 files changed, 15 insertions(+), 20 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 597f013c8e04..f52f02e8e5be 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -265,11 +265,9 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
VM_WARN_ON_ONCE(old_gen == -1 && new_gen == -1);
if (old_gen >= 0)
- WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
- lrugen->nr_pages[old_gen][type][zone] - delta);
+ atomic_long_sub(delta, &lrugen->nr_pages[old_gen][type][zone]);
if (new_gen >= 0)
- WRITE_ONCE(lrugen->nr_pages[new_gen][type][zone],
- lrugen->nr_pages[new_gen][type][zone] + delta);
+ atomic_long_add(delta, &lrugen->nr_pages[new_gen][type][zone]);
/* addition */
if (old_gen < 0) {
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 3b96d6c7123b..09ce82fa841a 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -554,7 +554,7 @@ struct lru_gen_folio {
/* the multi-gen LRU lists, lazily sorted on eviction */
struct list_head folios[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
/* the multi-gen LRU sizes, eventually consistent */
- long nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
+ atomic_long_t nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
/* the exponential moving average of refaulted */
unsigned long avg_refaulted[ANON_AND_FILE][MAX_NR_TIERS];
/* the exponential moving average of evicted+protected */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index daa7e37c2236..cab3ba14d20c 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -3495,8 +3495,7 @@ static void reset_batch_size(struct lru_gen_mm_walk *walk)
continue;
walk->nr_pages[gen][type][zone] = 0;
- WRITE_ONCE(lrugen->nr_pages[gen][type][zone],
- lrugen->nr_pages[gen][type][zone] + delta);
+ atomic_long_add(delta, &lrugen->nr_pages[gen][type][zone]);
if (lru_gen_is_active(lruvec, gen))
lru += LRU_ACTIVE;
@@ -4116,11 +4115,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
break;
}
flush_lru_batch(head, &batch_end, target_list);
-
- WRITE_ONCE(lrugen->nr_pages[old_gen][type][zone],
- lrugen->nr_pages[old_gen][type][zone] - delta);
- WRITE_ONCE(lrugen->nr_pages[target_gen][type][zone],
- lrugen->nr_pages[target_gen][type][zone] + delta);
+ atomic_long_sub(delta, &lrugen->nr_pages[old_gen][type][zone]);
+ atomic_long_add(delta, &lrugen->nr_pages[target_gen][type][zone]);
if (!remaining)
return false;
}
@@ -4225,8 +4221,8 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
for (type = 0; type < ANON_AND_FILE; type++) {
for (zone = 0; zone < MAX_NR_ZONES; zone++) {
enum lru_list lru = type * LRU_INACTIVE_FILE;
- long delta = lrugen->nr_pages[prev][type][zone] -
- lrugen->nr_pages[next][type][zone];
+ long delta = atomic_long_read(&lrugen->nr_pages[prev][type][zone]) -
+ atomic_long_read(&lrugen->nr_pages[next][type][zone]);
if (!delta)
continue;
@@ -4344,7 +4340,8 @@ static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness
for (seq = min_seq[type]; seq <= max_seq; seq++) {
gen = lru_gen_from_seq(seq);
for (zone = 0; zone < MAX_NR_ZONES; zone++)
- total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+ total += max(atomic_long_read(&lrugen->nr_pages[gen][type][zone]),
+ 0L);
}
}
@@ -4764,7 +4761,7 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
for (i = 0; i < get_nr_gens(child_lruvec, type); i++) {
int gen = lru_gen_from_seq(child_lrugen->max_seq - i);
- long nr_pages = child_lrugen->nr_pages[gen][type][zone];
+ long nr_pages = atomic_long_read(&child_lrugen->nr_pages[gen][type][zone]);
int child_lru_active = lru_gen_is_active(child_lruvec, gen) ? LRU_ACTIVE : 0;
int parent_lru_active = lru_gen_is_active(parent_lruvec, gen) ? LRU_ACTIVE : 0;
@@ -4772,9 +4769,8 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
list_splice_tail_init(&child_lrugen->folios[gen][type][zone],
&parent_lrugen->folios[gen][type][zone]);
- WRITE_ONCE(child_lrugen->nr_pages[gen][type][zone], 0);
- WRITE_ONCE(parent_lrugen->nr_pages[gen][type][zone],
- parent_lrugen->nr_pages[gen][type][zone] + nr_pages);
+ atomic_long_set(&child_lrugen->nr_pages[gen][type][zone], 0);
+ atomic_long_add(nr_pages, &parent_lrugen->nr_pages[gen][type][zone]);
if (lru_gen_is_active(child_lruvec, gen) != lru_gen_is_active(parent_lruvec, gen)) {
__update_lru_size(child_lruvec, lru + child_lru_active, zone, -nr_pages);
@@ -5859,7 +5855,8 @@ static int lru_gen_seq_show(struct seq_file *m, void *v)
char mark = full && seq < min_seq[type] ? 'x' : ' ';
for (zone = 0; zone < MAX_NR_ZONES; zone++)
- size += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
+ size += max(atomic_long_read(&lrugen->nr_pages[gen][type][zone]),
+ 0L);
seq_printf(m, " %10lu%c", size, mark);
}
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 02/17] mm/mglru: make generation page counters atomic
2026-10-15 18:15 ` Kairui Song
(?)
@ 2026-10-15 18:37 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:37 UTC (permalink / raw)
To: Kairui Song; +Cc: damon, tj, hannes, mkoutny
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [High] mm: mglru: lost LRU page counter updates in __lru_gen_reparent_memcg()
--
commit 82d0af39bfdf4a479c7e716d6991b3cf9cc6e2b4
Author: Kairui Song <kasong@tencent.com>
mm/mglru: make generation page counters atomic
This commit converts the multi-gen LRU generation page counters to atomic
longs. This allows the counters to be updated without holding the LRU lock,
relying on eventual consistency.
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index daa7e37c2236a..cab3ba14d20c2 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
[ ... ]
> @@ -4764,7 +4761,7 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
>
> for (i = 0; i < get_nr_gens(child_lruvec, type); i++) {
> int gen = lru_gen_from_seq(child_lrugen->max_seq - i);
> - long nr_pages = child_lrugen->nr_pages[gen][type][zone];
> + long nr_pages = atomic_long_read(&child_lrugen->nr_pages[gen][type][zone]);
> int child_lru_active = lru_gen_is_active(child_lruvec, gen) ? LRU_ACTIVE : 0;
> int parent_lru_active = lru_gen_is_active(parent_lruvec, gen) ? LRU_ACTIVE : 0;
>
> @@ -4772,9 +4769,8 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
> list_splice_tail_init(&child_lrugen->folios[gen][type][zone],
> &parent_lrugen->folios[gen][type][zone]);
>
> - WRITE_ONCE(child_lrugen->nr_pages[gen][type][zone], 0);
> - WRITE_ONCE(parent_lrugen->nr_pages[gen][type][zone],
> - parent_lrugen->nr_pages[gen][type][zone] + nr_pages);
> + atomic_long_set(&child_lrugen->nr_pages[gen][type][zone], 0);
> + atomic_long_add(nr_pages, &parent_lrugen->nr_pages[gen][type][zone]);
[Severity: High]
Does the non-atomic extraction and clearing of the child's counter create
a read-modify-write window that wipes out lockless updates from
folio_inc_lru_refs()?
Because __lru_gen_reparent_memcg() reads the child's counter, executes a
list_splice, and then unconditionally overwrites the counter via
atomic_long_set(0), any concurrent lockless counter additions or
subtractions that land during this window appear to be completely wiped
out and never propagated to the parent memcg.
Could we replace the read and zeroing with atomic_long_xchg(&..., 0) after
the list splice? This would atomically extract and clear the counter,
ensuring updates during the window are passed to the parent.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=2
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 03/17] mm/memcg: add folio-based lruvec live helper
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:15 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
Add a helper that resolves a stable lruvec for a folio under RCU
without taking the lruvec lock. It takes a folio directly so the
lruvec lookup happens inside the RCU read-side critical section,
which a lruvec-based interface cannot guarantee.
No functional change.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/memcontrol.h | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 74110a324f9e..829ab776ab44 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1535,6 +1535,45 @@ static inline void lruvec_lock_irq(struct lruvec *lruvec)
spin_lock_irq(&lruvec->lru_lock);
}
+/**
+ * folio_lruvec_live_get - get a live lruvec for a folio under RCU
+ * @folio: the folio
+ *
+ * Computes @folio's lruvec and walks up to the nearest live ancestor
+ * if the folio's memcg is dying. Paired with folio_lruvec_live_put().
+ * The result may be stale: RCU keeps it alive but does not pin @folio
+ * to it. That is fine as the counters are fixed up on reparenting.
+ *
+ * Return: the live lruvec, with rcu_read_lock held.
+ */
+static inline struct lruvec *folio_lruvec_live_get(struct folio *folio)
+{
+#ifdef CONFIG_MEMCG
+ struct lruvec *lruvec;
+ struct pglist_data *pgdat;
+ struct mem_cgroup *memcg;
+
+ rcu_read_lock();
+ lruvec = folio_lruvec(folio);
+ pgdat = lruvec_pgdat(lruvec);
+ memcg = lruvec_memcg(lruvec);
+ while (unlikely(memcg && css_is_dying(&memcg->css))) {
+ memcg = parent_mem_cgroup(memcg);
+ lruvec = mem_cgroup_lruvec(memcg, pgdat);
+ }
+ return lruvec;
+#else
+ return folio_lruvec(folio);
+#endif
+}
+
+static inline void folio_lruvec_live_put(struct lruvec *lruvec)
+{
+#ifdef CONFIG_MEMCG
+ rcu_read_unlock();
+#endif
+}
+
static inline struct lruvec *lruvec_live_lock_irq(struct lruvec *lruvec)
{
#ifdef CONFIG_MEMCG
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 03/17] mm/memcg: add folio-based lruvec live helper
@ 2026-10-15 18:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
Add a helper that resolves a stable lruvec for a folio under RCU
without taking the lruvec lock. It takes a folio directly so the
lruvec lookup happens inside the RCU read-side critical section,
which a lruvec-based interface cannot guarantee.
No functional change.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/memcontrol.h | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 74110a324f9e..829ab776ab44 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1535,6 +1535,45 @@ static inline void lruvec_lock_irq(struct lruvec *lruvec)
spin_lock_irq(&lruvec->lru_lock);
}
+/**
+ * folio_lruvec_live_get - get a live lruvec for a folio under RCU
+ * @folio: the folio
+ *
+ * Computes @folio's lruvec and walks up to the nearest live ancestor
+ * if the folio's memcg is dying. Paired with folio_lruvec_live_put().
+ * The result may be stale: RCU keeps it alive but does not pin @folio
+ * to it. That is fine as the counters are fixed up on reparenting.
+ *
+ * Return: the live lruvec, with rcu_read_lock held.
+ */
+static inline struct lruvec *folio_lruvec_live_get(struct folio *folio)
+{
+#ifdef CONFIG_MEMCG
+ struct lruvec *lruvec;
+ struct pglist_data *pgdat;
+ struct mem_cgroup *memcg;
+
+ rcu_read_lock();
+ lruvec = folio_lruvec(folio);
+ pgdat = lruvec_pgdat(lruvec);
+ memcg = lruvec_memcg(lruvec);
+ while (unlikely(memcg && css_is_dying(&memcg->css))) {
+ memcg = parent_mem_cgroup(memcg);
+ lruvec = mem_cgroup_lruvec(memcg, pgdat);
+ }
+ return lruvec;
+#else
+ return folio_lruvec(folio);
+#endif
+}
+
+static inline void folio_lruvec_live_put(struct lruvec *lruvec)
+{
+#ifdef CONFIG_MEMCG
+ rcu_read_unlock();
+#endif
+}
+
static inline struct lruvec *lruvec_live_lock_irq(struct lruvec *lruvec)
{
#ifdef CONFIG_MEMCG
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 03/17] mm/memcg: add folio-based lruvec live helper
2026-10-15 18:15 ` Kairui Song
(?)
@ 2026-10-15 18:31 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:31 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> Add a helper that resolves a stable lruvec for a folio under RCU
> without taking the lruvec lock. It takes a folio directly so the
> lruvec lookup happens inside the RCU read-side critical section,
> which a lruvec-based interface cannot guarantee.
>
> No functional change.
>
> Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=3
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v4 03/17] mm/memcg: add folio-based lruvec live helper
2026-10-15 18:15 ` Kairui Song
(?)
(?)
@ 2026-10-15 19:11 ` Harry Yoo
2026-10-16 6:15 ` Kairui Song
-1 siblings, 1 reply; 59+ messages in thread
From: Harry Yoo @ 2026-10-15 19:11 UTC (permalink / raw)
To: kasong
Cc: linux-mm, Liam R. Howlett, SJ Park, Andrew Morton,
Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang, Chris Li,
Baoquan He, Nico Pache, Usama Arif, Michal Hocko, Roman Gushchin,
Shakeel Butt, David Hildenbrand, Lorenzo Stoakes, Barry Song,
Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon
On Fri, Oct 16, 2026 at 02:15:52AM +0800, Kairui Song via B4 Relay wrote:
> From: Kairui Song <kasong@tencent.com>
>
> Add a helper that resolves a stable lruvec for a folio under RCU
> without taking the lruvec lock. It takes a folio directly so the
> lruvec lookup happens inside the RCU read-side critical section,
> which a lruvec-based interface cannot guarantee.
>
> No functional change.
>
> Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
This new helper is doing the the same thing as what
get_non_dying_memcg_start/end() do.
> include/linux/memcontrol.h | 39 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 39 insertions(+)
>
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index 74110a324f9e..829ab776ab44 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -1535,6 +1535,45 @@ static inline void lruvec_lock_irq(struct lruvec *lruvec)
> spin_lock_irq(&lruvec->lru_lock);
> }
>
> +/**
> + * folio_lruvec_live_get - get a live lruvec for a folio under RCU
> + * @folio: the folio
> + *
> + * Computes @folio's lruvec and walks up to the nearest live ancestor
> + * if the folio's memcg is dying. Paired with folio_lruvec_live_put().
> + * The result may be stale: RCU keeps it alive but does not pin @folio
> + * to it. That is fine as the counters are fixed up on reparenting.
> + *
> + * Return: the live lruvec, with rcu_read_lock held.
> + */
> +static inline struct lruvec *folio_lruvec_live_get(struct folio *folio)
> +{
> +#ifdef CONFIG_MEMCG
> + struct lruvec *lruvec;
> + struct pglist_data *pgdat;
> + struct mem_cgroup *memcg;
> +
> + rcu_read_lock();
> + lruvec = folio_lruvec(folio);
> + pgdat = lruvec_pgdat(lruvec);
> + memcg = lruvec_memcg(lruvec);
> + while (unlikely(memcg && css_is_dying(&memcg->css))) {
> + memcg = parent_mem_cgroup(memcg);
> + lruvec = mem_cgroup_lruvec(memcg, pgdat);
> + }
> + return lruvec;
> +#else
> + return folio_lruvec(folio);
> +#endif
> +}
> +
> +static inline void folio_lruvec_live_put(struct lruvec *lruvec)
> +{
> +#ifdef CONFIG_MEMCG
> + rcu_read_unlock();
> +#endif
> +}
> +
> static inline struct lruvec *lruvec_live_lock_irq(struct lruvec *lruvec)
> {
> #ifdef CONFIG_MEMCG
>
> --
> 2.56.0
--
Cheers,
Harry / Hyeonggon
^ permalink raw reply [flat|nested] 59+ messages in thread* Re: [PATCH v4 03/17] mm/memcg: add folio-based lruvec live helper
2026-10-15 19:11 ` Harry Yoo
@ 2026-10-16 6:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-16 6:15 UTC (permalink / raw)
To: Harry Yoo
Cc: linux-mm, Liam R. Howlett, SJ Park, Andrew Morton,
Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang, Chris Li,
Baoquan He, Nico Pache, Usama Arif, Michal Hocko, Roman Gushchin,
Shakeel Butt, David Hildenbrand, Lorenzo Stoakes, Barry Song,
Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Xiang Liu,
Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan, linux-kernel,
cgroups, damon
On Fri, Oct 16, 2026 at 3:11 AM Harry Yoo <harry@kernel.org> wrote:
>
> On Fri, Oct 16, 2026 at 02:15:52AM +0800, Kairui Song via B4 Relay wrote:
> > From: Kairui Song <kasong@tencent.com>
> >
> > Add a helper that resolves a stable lruvec for a folio under RCU
> > without taking the lruvec lock. It takes a folio directly so the
> > lruvec lookup happens inside the RCU read-side critical section,
> > which a lruvec-based interface cannot guarantee.
> >
> > No functional change.
> >
> > Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> > Signed-off-by: Kairui Song <kasong@tencent.com>
> > ---
>
> This new helper is doing the the same thing as what
> get_non_dying_memcg_start/end() do.
>
Hi Harry, thanks for letting me know, I wasn't aware this helper
exists, this series started before the re-planting work, so I missed
it.
But it's a bit different though, get_non_dying_memcg_start assumes the
caller holds a reference to memcg, while folio_lruvec_live_get only
assumes the caller holds an reference to the folio.
I'll check if I can deduplicate it though.
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v4 03/17] mm/memcg: add folio-based lruvec live helper
2026-10-15 18:15 ` Kairui Song
` (2 preceding siblings ...)
(?)
@ 2026-10-16 7:39 ` Muchun Song
2026-10-16 7:54 ` Kairui Song
-1 siblings, 1 reply; 59+ messages in thread
From: Muchun Song @ 2026-10-16 7:39 UTC (permalink / raw)
To: kasong
Cc: linux-mm, Liam R. Howlett, SJ Park, Andrew Morton,
Johannes Weiner, Qi Zheng, Ying Huang, Chris Li, Baoquan He,
Nico Pache, Usama Arif, Michal Hocko, Roman Gushchin,
Shakeel Butt, David Hildenbrand, Lorenzo Stoakes, Barry Song,
Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon
> On Oct 16, 2026, at 02:15, Kairui Song via B4 Relay <devnull+kasong.tencent.com@kernel.org> wrote:
>
> From: Kairui Song <kasong@tencent.com>
>
> Add a helper that resolves a stable lruvec for a folio under RCU
> without taking the lruvec lock. It takes a folio directly so the
> lruvec lookup happens inside the RCU read-side critical section,
> which a lruvec-based interface cannot guarantee.
>
> No functional change.
>
> Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
> ---
> include/linux/memcontrol.h | 39 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 39 insertions(+)
>
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index 74110a324f9e..829ab776ab44 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -1535,6 +1535,45 @@ static inline void lruvec_lock_irq(struct lruvec *lruvec)
> spin_lock_irq(&lruvec->lru_lock);
> }
>
> +/**
> + * folio_lruvec_live_get - get a live lruvec for a folio under RCU
> + * @folio: the folio
> + *
> + * Computes @folio's lruvec and walks up to the nearest live ancestor
> + * if the folio's memcg is dying. Paired with folio_lruvec_live_put().
> + * The result may be stale: RCU keeps it alive but does not pin @folio
> + * to it. That is fine as the counters are fixed up on reparenting.
> + *
> + * Return: the live lruvec, with rcu_read_lock held.
> + */
> +static inline struct lruvec *folio_lruvec_live_get(struct folio *folio)
> +{
> +#ifdef CONFIG_MEMCG
> + struct lruvec *lruvec;
> + struct pglist_data *pgdat;
> + struct mem_cgroup *memcg;
> +
> + rcu_read_lock();
> + lruvec = folio_lruvec(folio);
> + pgdat = lruvec_pgdat(lruvec);
> + memcg = lruvec_memcg(lruvec);
> + while (unlikely(memcg && css_is_dying(&memcg->css))) {
> + memcg = parent_mem_cgroup(memcg);
> + lruvec = mem_cgroup_lruvec(memcg, pgdat);
> + }
> + return lruvec;
Hi Kairui,
Actually, we don't have any means to guarantee that the memcg
to which the returned lruvec belongs will not become dying
after the return, right?
> +#else
> + return folio_lruvec(folio);
Then, I'm wondering why folio_lruvec() doesn't apply to the
memcg scenario.
Internally, folio_lruvec() obtains the memcg through
obj_cgroup_memcg(). So when the memcg becomes dying, it should
be reparented to the parent memcg within a short time.
Likewise, folio_lruvec() may also return an lruvec whose memcg
may also be in a short-lived dying state.
Since the results of the two are similar, why not just use
folio_lruvec() directly? Or is there something I misunderstood?
Thanks,
Muchun
> +#endif
> +}
> +
> +static inline void folio_lruvec_live_put(struct lruvec *lruvec)
> +{
> +#ifdef CONFIG_MEMCG
> + rcu_read_unlock();
> +#endif
> +}
> +
> static inline struct lruvec *lruvec_live_lock_irq(struct lruvec *lruvec)
> {
> #ifdef CONFIG_MEMCG
>
> --
> 2.56.0
>
>
^ permalink raw reply [flat|nested] 59+ messages in thread* Re: [PATCH v4 03/17] mm/memcg: add folio-based lruvec live helper
2026-10-16 7:39 ` Muchun Song
@ 2026-10-16 7:54 ` Kairui Song
2026-10-16 9:09 ` Muchun Song
0 siblings, 1 reply; 59+ messages in thread
From: Kairui Song @ 2026-10-16 7:54 UTC (permalink / raw)
To: Muchun Song
Cc: linux-mm, Liam R. Howlett, SJ Park, Andrew Morton,
Johannes Weiner, Qi Zheng, Ying Huang, Chris Li, Baoquan He,
Nico Pache, Usama Arif, Michal Hocko, Roman Gushchin,
Shakeel Butt, David Hildenbrand, Lorenzo Stoakes, Barry Song,
Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Xiang Liu,
Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan, linux-kernel,
cgroups, damon
On Fri, Oct 16, 2026 at 3:39 PM Muchun Song <muchun.song@linux.dev> wrote:
>
> > On Oct 16, 2026, at 02:15, Kairui Song via B4 Relay <devnull+kasong.tencent.com@kernel.org> wrote:
> >
> > From: Kairui Song <kasong@tencent.com>
> >
> > Add a helper that resolves a stable lruvec for a folio under RCU
> > without taking the lruvec lock. It takes a folio directly so the
> > lruvec lookup happens inside the RCU read-side critical section,
> > which a lruvec-based interface cannot guarantee.
> >
> > No functional change.
> >
> > Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> > Signed-off-by: Kairui Song <kasong@tencent.com>
> > ---
> > include/linux/memcontrol.h | 39 +++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 39 insertions(+)
> >
> > diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> > index 74110a324f9e..829ab776ab44 100644
> > --- a/include/linux/memcontrol.h
> > +++ b/include/linux/memcontrol.h
> > @@ -1535,6 +1535,45 @@ static inline void lruvec_lock_irq(struct lruvec *lruvec)
> > spin_lock_irq(&lruvec->lru_lock);
> > }
> >
> > +/**
> > + * folio_lruvec_live_get - get a live lruvec for a folio under RCU
> > + * @folio: the folio
> > + *
> > + * Computes @folio's lruvec and walks up to the nearest live ancestor
> > + * if the folio's memcg is dying. Paired with folio_lruvec_live_put().
> > + * The result may be stale: RCU keeps it alive but does not pin @folio
> > + * to it. That is fine as the counters are fixed up on reparenting.
> > + *
> > + * Return: the live lruvec, with rcu_read_lock held.
> > + */
> > +static inline struct lruvec *folio_lruvec_live_get(struct folio *folio)
> > +{
> > +#ifdef CONFIG_MEMCG
> > + struct lruvec *lruvec;
> > + struct pglist_data *pgdat;
> > + struct mem_cgroup *memcg;
> > +
> > + rcu_read_lock();
> > + lruvec = folio_lruvec(folio);
> > + pgdat = lruvec_pgdat(lruvec);
> > + memcg = lruvec_memcg(lruvec);
> > + while (unlikely(memcg && css_is_dying(&memcg->css))) {
> > + memcg = parent_mem_cgroup(memcg);
> > + lruvec = mem_cgroup_lruvec(memcg, pgdat);
> > + }
> > + return lruvec;
>
> Hi Kairui,
>
Hi Muchun, thanks for the review.
> Actually, we don't have any means to guarantee that the memcg
> to which the returned lruvec belongs will not become dying
> after the return, right?
Yeah but Qi once mentioned that RCU prevents reparenting, right? Even
if the memcg is dead after return, things happended to it
(non-hieratical counter reparenting, ones in __lru_gen_reparent_memcg)
will only occurs after RCU unlock so we can apply counter update to it
without grabbing an ref, just hold the RCU.
> > +#else
> > + return folio_lruvec(folio);
>
> Then, I'm wondering why folio_lruvec() doesn't apply to the
> memcg scenario.
>
> Internally, folio_lruvec() obtains the memcg through
> obj_cgroup_memcg(). So when the memcg becomes dying, it should
> be reparented to the parent memcg within a short time.
I wasn't sure if grabbing RCU can ensure an already dead cgroup has
not been reparented yet. If we apply the counter update to a already
reparented lruvec, then the counter is leaked. This new helper is just
a folio version of lruvec_live_lock_irq.
I got the idea from commit de4660898b7a, which says:
"""
.... check CSS_DYING under RCU before
flushing the pending batch. A non-dying memcg keeps the original lruvec
stable against RCU-delayed offlining; a dying memcg redirects the deltas
to the first non-dying ancestor.
"""
Applying counter update to a dead but not reparented lruvec is fine though.
^ permalink raw reply [flat|nested] 59+ messages in thread* Re: [PATCH v4 03/17] mm/memcg: add folio-based lruvec live helper
2026-10-16 7:54 ` Kairui Song
@ 2026-10-16 9:09 ` Muchun Song
0 siblings, 0 replies; 59+ messages in thread
From: Muchun Song @ 2026-10-16 9:09 UTC (permalink / raw)
To: Kairui Song
Cc: linux-mm, Liam R. Howlett, SJ Park, Andrew Morton,
Johannes Weiner, Qi Zheng, Ying Huang, Chris Li, Baoquan He,
Nico Pache, Usama Arif, Michal Hocko, Roman Gushchin,
Shakeel Butt, David Hildenbrand, Lorenzo Stoakes, Barry Song,
Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Xiang Liu,
Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan, linux-kernel,
cgroups, damon
> On Oct 16, 2026, at 15:54, Kairui Song <ryncsn@gmail.com> wrote:
>
> On Fri, Oct 16, 2026 at 3:39 PM Muchun Song <muchun.song@linux.dev> wrote:
>>
>>> On Oct 16, 2026, at 02:15, Kairui Song via B4 Relay <devnull+kasong.tencent.com@kernel.org> wrote:
>>>
>>> From: Kairui Song <kasong@tencent.com>
>>>
>>> Add a helper that resolves a stable lruvec for a folio under RCU
>>> without taking the lruvec lock. It takes a folio directly so the
>>> lruvec lookup happens inside the RCU read-side critical section,
>>> which a lruvec-based interface cannot guarantee.
>>>
>>> No functional change.
>>>
>>> Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
>>> Signed-off-by: Kairui Song <kasong@tencent.com>
>>> ---
>>> include/linux/memcontrol.h | 39 +++++++++++++++++++++++++++++++++++++++
>>> 1 file changed, 39 insertions(+)
>>>
>>> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
>>> index 74110a324f9e..829ab776ab44 100644
>>> --- a/include/linux/memcontrol.h
>>> +++ b/include/linux/memcontrol.h
>>> @@ -1535,6 +1535,45 @@ static inline void lruvec_lock_irq(struct lruvec *lruvec)
>>> spin_lock_irq(&lruvec->lru_lock);
>>> }
>>>
>>> +/**
>>> + * folio_lruvec_live_get - get a live lruvec for a folio under RCU
>>> + * @folio: the folio
>>> + *
>>> + * Computes @folio's lruvec and walks up to the nearest live ancestor
>>> + * if the folio's memcg is dying. Paired with folio_lruvec_live_put().
>>> + * The result may be stale: RCU keeps it alive but does not pin @folio
>>> + * to it. That is fine as the counters are fixed up on reparenting.
>>> + *
>>> + * Return: the live lruvec, with rcu_read_lock held.
>>> + */
>>> +static inline struct lruvec *folio_lruvec_live_get(struct folio *folio)
>>> +{
>>> +#ifdef CONFIG_MEMCG
>>> + struct lruvec *lruvec;
>>> + struct pglist_data *pgdat;
>>> + struct mem_cgroup *memcg;
>>> +
>>> + rcu_read_lock();
>>> + lruvec = folio_lruvec(folio);
>>> + pgdat = lruvec_pgdat(lruvec);
>>> + memcg = lruvec_memcg(lruvec);
>>> + while (unlikely(memcg && css_is_dying(&memcg->css))) {
>>> + memcg = parent_mem_cgroup(memcg);
>>> + lruvec = mem_cgroup_lruvec(memcg, pgdat);
>>> + }
>>> + return lruvec;
>>
>> Hi Kairui,
>>
>
> Hi Muchun, thanks for the review.
>
>> Actually, we don't have any means to guarantee that the memcg
>> to which the returned lruvec belongs will not become dying
>> after the return, right?
>
> Yeah but Qi once mentioned that RCU prevents reparenting, right? Even
> if the memcg is dead after return, things happended to it
> (non-hieratical counter reparenting, ones in __lru_gen_reparent_memcg)
> will only occurs after RCU unlock so we can apply counter update to it
> without grabbing an ref, just hold the RCU.
Thanks for the info. I checked the code and confirmed that, aside from
keeping the memcg from being freed, RCU also ensures the memcg doesn't get
reparented. If your scenario requires mutual exclusion with the reparenting
operation, RCU can handle that.
Thanks,
Muchun
>
>>> +#else
>>> + return folio_lruvec(folio);
>>
>> Then, I'm wondering why folio_lruvec() doesn't apply to the
>> memcg scenario.
>>
>> Internally, folio_lruvec() obtains the memcg through
>> obj_cgroup_memcg(). So when the memcg becomes dying, it should
>> be reparented to the parent memcg within a short time.
>
> I wasn't sure if grabbing RCU can ensure an already dead cgroup has
> not been reparented yet. If we apply the counter update to a already
> reparented lruvec, then the counter is leaked. This new helper is just
> a folio version of lruvec_live_lock_irq.
>
> I got the idea from commit de4660898b7a, which says:
>
> """
> .... check CSS_DYING under RCU before
> flushing the pending batch. A non-dying memcg keeps the original lruvec
> stable against RCU-delayed offlining; a dying memcg redirects the deltas
> to the first non-dying ancestor.
> """
>
> Applying counter update to a dead but not reparented lruvec is fine though.
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 04/17] mm/mglru: frequency guided workingset promotion (MGLRU-FG)
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:15 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
Complement MGLRU's eviction-time tier-PID protection with access-time
frequency-guided promotion. Introduce a unified set of helpers built based
on referenced (access) count of a folio.
Each access increments a folio's referenced count stored in folio flags
(refs), refs still maps to a logarithmic tier just like before, but with
more formal bit definitions, a few special thresholds are introduced:
LRU_REFS_REFERENCED (1), LRU_REFS_WORKINGSET (2), LRU_REFS_PROTECTED (3),
and LRU_REFS_MAX (7). When refs reaches a certain threshold, the folio is
promoted proactively instead of waiting for the PID controller to kick in.
Also simplify MGLRU's usage of PG_workingset and PG_referenced: they
become the low two bits of the refs count, with the higher bits
provided by LRU_REFS_MASK. This reduces MGLRU's original refs count
bit usage by one, since only one extra bit is now needed to record a
max referenced count of 7, and makes MGLRU's refs accounting more
accurate.
This doesn't affect classical LRU in any way, and it addresses several
shortcomings of MGLRU's old tier-only cache protection model:
- Long feedback loop: protection only activated after enough re-faults,
by which time the hot folios are already evicted, or no longer hot.
- Limited tier resolution: once referenced count exceeded the bits
limit (8 previously), MGLRU could no longer distinguish hotter folios as
they are capped by the tier. And what's worse, PG_workingset forces
a folio to stay on tier 3.
- Eviction hotness reversion: because PID protection activates upon
eviction and always targets the LRU tail, it tends to protect cold tail
folios at the expense of hotter head folios. Once the tail folios
consume the PID protection budget, head folios lose their protection.
- Additionally, the PID cannot distinguish the access time of folios
that share the same reference count, and there are only 4 tiers.
To achieve a frequency-guided framework, this commit introduces and reworks
the LRU_REFS related helpers and definitions; most of the work is done by
the helpers below, and their inline comments describe the details.
- folio_inc_lru_refs(): Called on any cache access (folio_mark_accessed)
or page table access. This is the main helper: it promotes folios
according to their access frequency.
A first access only defers eviction for a folio at the tail of its
LRU, i.e., in one of the MIN_NR_GENS oldest generations of its own
type (see lru_gen_at_tail()), and a workingset folio accessed through
a file descriptor is deferred from the same tail. Those are the
generations the eviction scans next, so the deferral moves the folio
out of its way for one generation.
Promotion is lazy: the gen bits and size counters are updated eagerly,
while the list move is deferred to the next isolation. NOTE: For now,
the lruvec lock is unconditionally taken on every on-list access to
block concurrent aging; a lockless fast path will be implemented very
soon in a following commit.
- folio_inc_lru_refs_walk(): Used by the PTE walk path during aging and
by the batched look-around, both without the LRU lock, so max_seq may
be stale. A first page table access only advances a folio at the tail
of its LRU, i.e., in one of the MIN_NR_GENS oldest generations of its
own type (second chance), a second access promotes it to the newest
generation; this also performs lazy promotion.
- folio_inc_lru_refs_isolated(): Used by the rmap check before
eviction. The folio is isolated and hence this doesn't perform
promotion by itself; the folio will be added back to the right gen
upon return according to the access frequency. This path also has a
higher promotion bias.
The eviction-time folio_inc_gen() still handles PID protection, but the
protection ratio is softer than before since proactive promotion is
mostly good enough already. The PID gain factors are relaxed from
(2:3) to (1:2) and the setpoint now spans the cumulative mass of the
tiers below the candidate instead of tier 0 alone. folio_inc_gen()
caps refs at WORKINGSET so the folio retains enough history to stay
above the cold tier. Tier 1 is the fallback tier for PID, and tier 2
is the fallback tier for frequency-guided promotion. The forced
protection for full-refs folios is removed, obsoleted by the proactive
promotion.
Refaults are now activated purely according to access frequency:
the old fault bias applied in folio_add_lru() is simplified, since a
folio's access history is a more consistent signal than the context of
the faulting task. Page table access is still considered a slightly
stronger signal.
Consequently the placement hint for newly added file folios also uses
LRU_REFS_WORKINGSET instead of PG_referenced, so a folio accessed only
once now starts in the oldest generation rather than the second oldest.
The previous one-generation grace was an artifact of the fault bias which
is well described in the cover letter of this series, and it depended
on LRU drain timing.
This also redefines PG_workingset and PG_referenced as the low two
bits of the refs count, eliminating the old restriction where
LRU_REFS_MASK was only valid when PG_referenced was set, and allows
all paths to use the same encoding consistently. Following this idea,
a workingset folio is now defined as refs >= LRU_REFS_WORKINGSET (2),
matching the active/inactive LRU's definition and giving in-kernel
consumers (PSI, readahead) consistent behavior on MGLRU, fixing the
longstanding issue that these users don't work well with MGLRU.
PG_workingset and PG_referenced are no longer independent flags under
MGLRU. Adjusting existing raw folio_test_*() callers to the new
semantics is left as follow-ups.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 96 ++++++++------
include/linux/mmzone.h | 145 +++++++++++++++------
kernel/bounds.c | 2 +-
mm/folio.c | 50 +-------
mm/vmscan.c | 320 +++++++++++++++++++++++++++++++---------------
mm/workingset.c | 45 ++++---
6 files changed, 409 insertions(+), 249 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index f52f02e8e5be..74d1b7a04ca1 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -144,12 +144,13 @@ static inline int lru_hist_from_seq(unsigned long seq)
return seq % NR_HIST_GENS;
}
-static inline int lru_tier_from_refs(int refs, bool workingset)
+static inline int lru_tier_from_refs(unsigned int refs)
{
- VM_WARN_ON_ONCE(refs > BIT(LRU_REFS_WIDTH));
-
- /* see the comment on MAX_NR_TIERS */
- return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs);
+ BUILD_BUG_ON(fls(LRU_REFS_MAX - 1) > MAX_NR_TIERS - 1);
+ VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
+ if (refs < LRU_REFS_WORKINGSET)
+ return 0;
+ return fls(refs - 1);
}
/**
@@ -187,21 +188,24 @@ static inline int lru_get_gen_flags(unsigned long flags)
* @flags: pointer to the folio flags
* @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
*
- * For MGLRU, PG_referenced holds the first ref, and the extra bits hold the
- * remaining refs. For classical LRU the extra bits are not used, so it can
- * also be seen as the refs count never exceeds 1. In both cases, refs == 1
- * means PG_referenced is set and the extra bits are zero, and refs == 0 means
- * PG_referenced and the extra bits are all unset.
+ * For MGLRU, PG_referenced, PG_workingset are used as the lower two bits of
+ * refs counter, and extra bits hold the remaining higher bits. For classical
+ * LRU the extra bits are not used, and the two flags has no direct
+ * relationship with each other, but this helper can still be used to sync
+ * them. For both cases, refs == 0 means these two flags and the extra bits
+ * are all unset. And refs == 1 / 2 / 3 means PG_referenced and PG_workingset
+ * are set in an bit order way, which is more meaningful for MGLRU though.
*/
static inline void lru_set_refs_flags(unsigned long *flags, unsigned int refs)
{
VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
- BUILD_BUG_ON(LRU_REFS_MAX != (LRU_REFS_MASK >> LRU_REFS_PGOFF) + 1);
-
+ BUILD_BUG_ON(LRU_REFS_MASK & (BIT(PG_referenced) | BIT(PG_workingset)));
*flags &= ~LRU_REFS_FLAGS;
- if (!refs)
- return;
- *flags |= (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF));
+ if (refs & BIT(0))
+ *flags |= BIT(PG_referenced);
+ if (refs & BIT(1))
+ *flags |= BIT(PG_workingset);
+ *flags |= ((unsigned long)refs >> 2) << LRU_REFS_PGOFF;
}
/**
@@ -212,13 +216,13 @@ static inline void lru_set_refs_flags(unsigned long *flags, unsigned int refs)
*/
static inline int lru_get_refs_flags(unsigned long flags)
{
- if (!(flags & BIT(PG_referenced)))
- return 0;
- /*
- * Return the total number of accesses including PG_referenced. Also see
- * the comment on LRU_REFS_FLAGS.
- */
- return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
+ int refs;
+
+ /* Return the total number of accesses. See the comment above MAX_NR_TIERS. */
+ refs = (flags & BIT(PG_referenced)) ? BIT(0) : 0;
+ refs += (flags & BIT(PG_workingset)) ? BIT(1) : 0;
+ refs += ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) << 2;
+ return refs;
}
static inline int folio_lru_refs(const struct folio *folio)
@@ -236,14 +240,29 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
}
+void folio_inc_lru_refs(struct folio *folio, unsigned int flags);
+
static inline int folio_lru_gen(const struct folio *folio)
{
return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
}
+static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
+{
+ unsigned long min_seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
+
+ /*
+ * Consider a folio is at the tail of its LRU when it is within
+ * MIN_NR_GENS of the oldest generation of its own type
+ */
+ BUILD_BUG_ON(MIN_NR_GENS != 2);
+ return gen == lru_gen_from_seq(min_seq) ||
+ gen == lru_gen_from_seq(min_seq + 1);
+}
+
static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
{
- unsigned long max_seq = lruvec->lrugen.max_seq;
+ unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
@@ -300,23 +319,24 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
bool reclaiming)
{
int gen;
+ int refs = folio_lru_refs(folio);
int type = folio_is_file_lru(folio);
const struct lru_gen_folio *lrugen = &lruvec->lrugen;
/*
- * +-----------------------------------+-----------------------------------+
- * | Accessed through page tables and | Accessed through file descriptors |
- * | promoted by folio_update_gen() | and protected by folio_inc_gen() |
- * +-----------------------------------+-----------------------------------+
- * | PG_active (set while isolated) | |
- * +-----------------+-----------------+-----------------+-----------------+
- * | PG_workingset | PG_referenced | PG_workingset | LRU_REFS_FLAGS |
- * +-----------------------------------+-----------------------------------+
- * |<---------- MIN_NR_GENS ---------->| |
- * |<---------------------------- MAX_NR_GENS ---------------------------->|
+ * +------------------------------------------+------------------------------------------+
+ * | Accessed through page tables and | Accessed through file descriptors |
+ * | promoted by folio_inc_lru_refs_walk() | protected by folio_inc_lru_refs/inc_gen |
+ * +------------------------------------------+------------------------------------------+
+ * | PG_active (set at isolation or refault) | |
+ * +--------------------+---------------------+--------------------+---------------------+
+ * | LRU_REFS_MAX | LRU_REFS_WORKINGSET | LRU_REFS_MAX | LRU_REFS_WORKINGSET |
+ * +------------------------------------------+------------------------------------------+
+ * |<-------------- MIN_NR_GENS ------------->| |
+ * |<----------------------------------- MAX_NR_GENS ----------------------------------->|
*/
if (folio_test_active(folio))
- gen = MIN_NR_GENS - folio_test_workingset(folio);
+ gen = MIN_NR_GENS - (refs >= LRU_REFS_WORKINGSET);
else if (reclaiming)
gen = MAX_NR_GENS;
else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) ||
@@ -324,7 +344,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
(folio_test_dirty(folio) || folio_test_writeback(folio))))
gen = MIN_NR_GENS;
else
- gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
+ gen = MAX_NR_GENS - (refs >= LRU_REFS_WORKINGSET);
return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
}
@@ -338,6 +358,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
int zone = folio_zonenum(folio);
struct lru_gen_folio *lrugen = &lruvec->lrugen;
+ BUILD_BUG_ON(BIT(LRU_GEN_WIDTH - 1) != MAX_NR_GENS);
VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
if (folio_test_unevictable(folio) || !lrugen->enabled)
@@ -392,7 +413,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
*/
static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
{
- BUILD_BUG_ON(LRU_REFS_MASK & BIT(PG_referenced));
folio_set_lru_refs(new, folio_lru_refs(old));
}
#else /* !CONFIG_LRU_GEN */
@@ -422,6 +442,10 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
return false;
}
+static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
+{
+}
+
static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
{
if (folio_test_referenced(old))
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 09ce82fa841a..b5a3769fdcab 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -453,57 +453,122 @@ enum lruvec_flags {
#define MAX_NR_GENS 4U
/*
- * Each generation is divided into multiple tiers. A folio accessed N times
- * through file descriptors is in tier order_base_2(N). A folio in the first
- * tier (N=0,1) is marked by PG_referenced unless it was faulted in through page
- * tables or read ahead. A folio in the last tier (MAX_NR_TIERS-1) is marked by
- * PG_workingset. A folio in any other tier (1<N<5) between the first and last
- * is marked by additional bits of LRU_REFS_WIDTH in folio->flags.
+ * Each generation is divided into multiple tiers. A folio's referenced
+ * count maps to a tier as shown below:
*
- * In contrast to moving across generations which requires the LRU lock, moving
- * across tiers only involves atomic operations on folio->flags and therefore
- * has a negligible cost in the buffered access path. In the eviction path,
- * comparisons of refaulted/(evicted+protected) from the first tier and the rest
- * infer whether folios accessed multiple times through file descriptors are
- * statistically hot and thus worth protecting.
+ * MGLRU (frequency guidance)
+ * Refs Tier |- Refs: how many times (at least) a folio has been referenced.
+ * 0 0 |- Mostly cold pages, readahead, etc. [1]
+ * 1 0 |= LRU_REFS_REFERENCED: Used at least once. [2]
+ * -WORKINGSET-+|- Pages beyond are workingset and never fall below this floor. [3]
+ * 2 1<-+|= LRU_REFS_WORKINGSET: Classical workingset, accessed twice, protected. [4]
+ * 3 2 |- LRU_REFS_PROTECTED: Protected workingset, promoted pages capped at here. [5]
+ * 4 2 |
+ * 5 3 |- The tier here is MAX_NR_TIERS - 1
+ * 6 3 |
+ * 7 3 |= LRU_REFS_MAX: Promotion candidate. [6]
+ * -PROMOTION->-/
*
- * MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice the
- * number of categories of the active/inactive LRU when keeping track of
- * accesses through file descriptors. This uses MAX_NR_TIERS-2 spare bits in
- * folio->flags, masked by LRU_REFS_MASK.
+ * Ideally each tier holds folios of similar access patterns: lower tiers
+ * are less important and evicted faster. A page's reference count and
+ * tier are capped when it changes generation, preventing it from
+ * dominating the new generation based on old-generation access history.
+ * Generation ordering already ensures a newer-gen page is hotter than an
+ * older-gen one regardless of tier.
+ *
+ * Refs tracks accesses from two sources: page table (lazily collected by
+ * the page table aging walk or rmap eviction lookup) and file descriptors
+ * (by folio_mark_accessed). Page table accesses are weighted heavier
+ * because the accessed bit is sticky (undercounts repeated accesses),
+ * passively collected, and page faults are generally more important as
+ * userspace does not expect a memory access to block on reclaim. Both
+ * access types increment refs by one; the result is capped at
+ * LRU_REFS_PROTECTED on promotion or deferral, or LRU_REFS_MAX otherwise.
+ *
+ * 1. Tier is fls(N-1) for N > 1, 0 otherwise. Folios with zero
+ * accesses (refs == 0) are generally cold, e.g. readahead folios.
+ *
+ * Freshly allocated folios start with refs == 0; faulted and mapped
+ * folios have their page table access bit set, so the first page table
+ * access check always sets LRU_REFS_REFERENCED. That is the second
+ * chance described above MIN_NR_GENS: it moves the folio one
+ * generation forward if it is at the tail of its LRU, i.e., in one
+ * of the two oldest generations of its type.
+ *
+ * 2. Folios accessed once stay on tier 0: one-time usage does not
+ * qualify for protection. A second access advances the folio,
+ * aligning with classical LRU's use-twice threshold. A second page
+ * table access promotes to the latest gen; file access only defers
+ * eviction from the tail gens of its type.
+ *
+ * 3. Folios accessed at least twice are considered workingset. This
+ * mostly aligns with classical LRU: at least one I/O is saved by
+ * keeping them in memory. Folios at or above this level never fall
+ * below tier 1 (the workingset floor), so tier 0 stays a clean tier
+ * for cold cache while tier 1 serves as the fallback line for
+ * actually reused or historically hot folios.
+ *
+ * Folios refaulted through a page fault at refs 1 will enter the second
+ * newest gen, so faulting will be protected better.
+ *
+ * 4. Starting from tier 1, PID protection sacrifices lower tiers to
+ * protect higher tiers by comparing refault rates for long-term
+ * accuracy, and caps higher refs to this value. Since PID protection
+ * bypasses page table lookup and clearing, when a further eviction
+ * attempt occurs after PID loosens, the folio's page table access is
+ * rechecked and a referenced folio is put back with refs capped to
+ * this value as well. This also gives folios a fair opportunity to be
+ * promoted by file access again.
+ *
+ * Folios refaulted through a page fault at tier 1 or above are activated
+ * and enter the newest gen. Non fault page will enter second oldest gen,
+ * driving aging and workingset shifting.
+ *
+ * 5. Pages beyond the ordinary workingset tier form new tiers for the
+ * PID controller to protect differently. Folios at or above this
+ * level are capped at LRU_REFS_PROTECTED on promotion or deferral,
+ * and at LRU_REFS_WORKINGSET under PID protection in the oldest
+ * generation, where they represent a historical workingset.
+ *
+ * 6. Folios that reach LRU_REFS_MAX are advanced to the next generation
+ * on further access, with refs capped to LRU_REFS_PROTECTED. This
+ * gives them a fair start for advancement to an even newer generation
+ * while keeping hot folios distinguishable.
+ *
+ * Tiering uses PG_referenced and PG_workingset as the lower two bits,
+ * and the bits masked by LRU_REFS_MASK as the higher bits, so the refs
+ * count ranges from 0 to LRU_REFS_MAX. A folio is on the workingset
+ * tier once accessed at least twice, which is more consistent with the
+ * classical LRU.
+ *
+ * A folio's referenced count never goes backwards except upon gen
+ * increase as described above, or when explicitly reset by
+ * lru_gen_clear_refs(). Refault of a reclaimed folio restores
+ * its referenced count, capped at LRU_REFS_PROTECTED, which aligns with
+ * promotion. Page table refaults of previous workingset folios send
+ * them to the latest gen, driving aging faster.
+ *
+ * MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice
+ * the number of categories of the active/inactive LRU.
*/
#define MAX_NR_TIERS 4U
#define LRU_TIER_MIN 0U
#define LRU_TIER_MAX (MAX_NR_TIERS - 1)
+/* Access source flags for folio_inc_lru_refs() */
+#define LRU_REF_MAPPED 0x1U
+#define LRU_REF_EXEC 0x2U
+
+#define LRU_REFS_REFERENCED 0x1
+#define LRU_REFS_WORKINGSET 0x2
+#define LRU_REFS_PROTECTED 0x3
+
#ifndef __GENERATING_BOUNDS_H
#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
-#define LRU_REFS_MAX BIT(LRU_REFS_WIDTH)
-
-/*
- * For folios accessed multiple times through file descriptors,
- * lru_gen_inc_refs() sets additional bits of LRU_REFS_WIDTH in folio->flags
- * after PG_referenced, then PG_workingset after LRU_REFS_WIDTH. After all its
- * bits are set, i.e., LRU_REFS_FLAGS|BIT(PG_workingset), a folio is lazily
- * promoted into the second oldest generation in the eviction path. And when
- * folio_inc_gen() does that, it clears LRU_REFS_FLAGS so that
- * lru_gen_inc_refs() can start over. Note that for this case, LRU_REFS_MASK is
- * only valid when PG_referenced is set.
- *
- * For folios accessed multiple times through page tables, folio_update_gen()
- * from a page table walk or lru_gen_set_refs() from a rmap walk sets
- * PG_referenced after the accessed bit is cleared for the first time.
- * Thereafter, those two paths set PG_workingset and promote folios to the
- * youngest generation. Like folio_inc_gen(), folio_update_gen() also clears
- * PG_referenced. Note that for this case, LRU_REFS_MASK is not used.
- *
- * For both cases above, after PG_workingset is set on a folio, it remains until
- * this folio is either reclaimed, or "deactivated" by lru_gen_clear_refs(). It
- * can be set again if lru_gen_test_recent() returns true upon a refault.
- */
-#define LRU_REFS_FLAGS (LRU_REFS_MASK | BIT(PG_referenced))
+#define LRU_REFS_FLAGS (LRU_REFS_MASK | BIT(PG_referenced) | BIT(PG_workingset))
+#define LRU_REFS_MAX (BIT(LRU_REFS_WIDTH + 2) - 1)
struct lruvec;
struct page_vma_mapped_walk;
diff --git a/kernel/bounds.c b/kernel/bounds.c
index 02b619eb6106..06a034713b5d 100644
--- a/kernel/bounds.c
+++ b/kernel/bounds.c
@@ -25,7 +25,7 @@ int main(void)
DEFINE(SPINLOCK_SIZE, sizeof(spinlock_t));
#ifdef CONFIG_LRU_GEN
DEFINE(LRU_GEN_WIDTH, order_base_2(MAX_NR_GENS + 1));
- DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 2);
+ DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 3);
#else
DEFINE(LRU_GEN_WIDTH, 0);
DEFINE(__LRU_REFS_WIDTH, 0);
diff --git a/mm/folio.c b/mm/folio.c
index 35e242b48870..e200efab814f 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -273,7 +273,6 @@ static void lru_activate(struct lruvec *lruvec, struct folio *folio)
if (folio_test_active(folio) || folio_test_unevictable(folio))
return;
-
lruvec_del_folio(lruvec, folio);
folio_set_active(folio);
lruvec_add_folio(lruvec, folio);
@@ -352,32 +351,6 @@ static void __lru_cache_activate_folio(struct folio *folio)
#ifdef CONFIG_LRU_GEN
-static void lru_gen_inc_refs(struct folio *folio)
-{
- unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
- int refs;
-
- if (folio_test_unevictable(folio))
- return;
-
- /* see the comment on LRU_REFS_FLAGS */
- if (!folio_lru_refs(folio)) {
- folio_set_lru_refs(folio, 1);
- return;
- }
-
- do {
- new_flags = old_flags;
- refs = lru_get_refs_flags(old_flags);
- if (refs == LRU_REFS_MAX) {
- if (!folio_test_workingset(folio))
- folio_set_workingset(folio);
- return;
- }
- lru_set_refs_flags(&new_flags, refs + 1);
- } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
-}
-
static bool lru_gen_clear_refs(struct folio *folio)
{
int gen = folio_lru_gen(folio);
@@ -388,7 +361,6 @@ static bool lru_gen_clear_refs(struct folio *folio)
return true;
folio_set_lru_refs(folio, 0);
- folio_clear_workingset(folio);
rcu_read_lock();
seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
@@ -399,10 +371,6 @@ static bool lru_gen_clear_refs(struct folio *folio)
#else /* !CONFIG_LRU_GEN */
-static void lru_gen_inc_refs(struct folio *folio)
-{
-}
-
static bool lru_gen_clear_refs(struct folio *folio)
{
return false;
@@ -428,7 +396,8 @@ void folio_mark_accessed(struct folio *folio)
if (folio_test_dropbehind(folio))
return;
if (lru_gen_enabled()) {
- lru_gen_inc_refs(folio);
+ if (!folio_test_unevictable(folio))
+ folio_inc_lru_refs(folio, 0);
return;
}
@@ -474,21 +443,6 @@ void folio_add_lru(struct folio *folio)
folio_test_unevictable(folio), folio);
VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
- /*
- * For refaulted workingset folios, set PG_active so they
- * can be added to active generations.
- * For prefaulted file folios, folio_mark_accessed() sets
- * PG_referenced so lru_gen_folio_seq() places them into
- * the second oldest generation.
- */
- if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
- lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
- if (folio_test_workingset(folio))
- folio_set_active(folio);
- else if (!folio_test_referenced(folio))
- folio_mark_accessed(folio);
- }
-
folio_batch_add_and_move(folio, lru_add);
}
EXPORT_SYMBOL(folio_add_lru);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index cab3ba14d20c..d0f0e5a25d87 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -931,38 +931,186 @@ enum folio_references {
};
#ifdef CONFIG_LRU_GEN
+/******************************************************************************
+ * Referenced count feedback
+ ******************************************************************************/
+
/*
- * Only used on a mapped folio in the eviction (rmap walk) path, where promotion
- * needs to be done by taking the folio off the LRU list and then adding it back
- * with PG_active set. In contrast, the aging (page table walk) path uses
- * folio_update_gen().
+ * The folio_inc_lru_refs{_*} helpers below collect the referenced info
+ * (hotness) from other parts, including the page table walker, the rmap walk
+ * upon eviction, the rmap lookaround, and file descriptors
+ * (folio_mark_accessed).
+ *
+ * Page table accesses use refs from either source. The first reference
+ * advances tail-gen folios by one generation; a second promotes to the
+ * newest. Executable file folios promote on their first access to avoid
+ * I/O thrashing. Isolated rmap accesses use evict_folios() for putback.
+ *
+ * Promotion changes folio->flags; sort_folio() or inc_min_seq() moves the
+ * folio later. Young PTEs found during unmapping use folio_mark_accessed()
+ * and follow the file descriptor rules below.
+ *
+ * File descriptor accesses do not promote. They only defer eviction from
+ * the tail gens of their type, and only once the folio is a workingset
+ * folio (LRU_REFS_WORKINGSET), leaving the rest to PID protection. Page
+ * table accesses are treated more generously because the accessed bit is
+ * sticky (it under-counts repeated accesses) and because a page fault is
+ * more costly than file descriptor I/O.
+ *
+ * PID protection operates on tier > 0 folios. The one proactive promotion
+ * outside of it and the page table path is the overflow case where the
+ * referenced count exceeds LRU_REFS_MAX, which means the folio is hotter
+ * than everything else in its generation.
+ *
+ * Whenever a folio changes generation here its referenced count is capped at
+ * LRU_REFS_PROTECTED, so it starts at or below the protected tier regardless
+ * of its old-generation access history. PID protection (folio_inc_gen) caps
+ * at LRU_REFS_WORKINGSET independently.
*/
-static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
-{
- /* see the comment on LRU_REFS_FLAGS */
- if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
- /* Activate file-backed executable folios after first usage. */
- if (is_exec_file_folio(folio, vma_flags)) {
- folio_set_workingset(folio);
- folio_set_lru_refs(folio, 0);
- return true;
+
+/*
+ * Update the folio's lru refs indicator. The caller doesn't need to hold
+ * the folio lock, isolate the folio, or hold the lruvec lock. Used by both
+ * cache access (flags == 0) and page table access (LRU_REF_MAPPED,
+ * optionally with LRU_REF_EXEC).
+ */
+void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
+{
+ int max_gen;
+ int type, refs, old_gen, gen;
+ unsigned long new_flags, old_flags, max_seq;
+ struct lru_gen_folio *lrugen;
+ struct lruvec *lruvec = NULL;
+
+ type = folio_is_file_lru(folio);
+ old_flags = READ_ONCE(*folio_flags(folio, 0));
+ do {
+ new_flags = old_flags;
+ old_gen = lru_get_gen_flags(old_flags);
+ refs = lru_get_refs_flags(old_flags) + 1;
+ gen = old_gen;
+ if (old_gen < 0)
+ goto out;
+ /*
+ * Lock the lruvec if the folio is on-list. We are already
+ * doing lazy promotion so in theory we don't need this,
+ * but for now, concurrent aging would still corrupt the
+ * size counters. This is a temporary limitation and
+ * will be lifted very soon, so the lock here is not a
+ * performance concern.
+ */
+ if (!lruvec) {
+ lruvec = lruvec_live_lock_irq(folio_lruvec(folio));
+ lrugen = &lruvec->lrugen;
}
+ max_seq = READ_ONCE(lrugen->max_seq);
+ max_gen = lru_gen_from_seq(max_seq);
+ if (old_gen == max_gen)
+ goto out;
- folio_set_lru_refs(folio, 1);
- return false;
- }
+ if (flags & (LRU_REF_MAPPED | LRU_REF_EXEC)) {
+ /* Promote second page table access or executable */
+ if (refs > LRU_REFS_REFERENCED || flags & LRU_REF_EXEC)
+ gen = max_gen;
+ /* First access only defers eviction of a tail gen */
+ else if (lru_gen_at_tail(lruvec, type, old_gen))
+ gen = (old_gen + 1) % MAX_NR_GENS;
+ refs = min(refs, LRU_REFS_PROTECTED);
+ } else if (refs > LRU_REFS_MAX) {
+ /* LRU refs counting overflow, bump the gen */
+ gen = (old_gen + 1) % MAX_NR_GENS;
+ refs = LRU_REFS_PROTECTED;
+ } else if (lru_gen_at_tail(lruvec, type, old_gen) && refs >= LRU_REFS_WORKINGSET) {
+ /* Defer eviction of just accessed workingset */
+ gen = (old_gen + 1) % MAX_NR_GENS;
+ refs = min(refs, LRU_REFS_PROTECTED);
+ }
+out:
+ refs = min(refs, LRU_REFS_MAX);
+ lru_set_refs_flags(&new_flags, refs);
+ if (gen != old_gen)
+ lru_set_gen_flags(&new_flags, gen);
+ if (new_flags == old_flags)
+ break;
+ } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
- /* Promote on second access */
- if (folio_lru_refs(folio) > 1) {
- folio_set_workingset(folio);
- folio_set_lru_refs(folio, 0);
- } else {
- folio_mark_accessed(folio);
- }
- return true;
+ if (gen != old_gen)
+ lru_gen_update_size(lruvec, folio, old_gen, gen);
+ if (lruvec)
+ lruvec_unlock_irq(lruvec);
+}
+
+/*
+ * Update the folio's lru refs indicator during a page table walk or the
+ * look-around. max_seq can be stale as neither holds the LRU lock.
+ *
+ * Returns the old generation and stores the new generation in @new_gen if
+ * the folio is on the LRU and not in the newest generation, or -1 otherwise.
+ */
+static int folio_inc_lru_refs_walk(struct folio *folio, struct lruvec *lruvec,
+ const vma_flags_t *vma_flags,
+ int *new_gen, int *type)
+{
+ unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+ int refs, gen, max_gen, ret;
+ unsigned long max_seq;
+
+ do {
+ max_seq = READ_ONCE(lruvec->lrugen.max_seq);
+ max_gen = lru_gen_from_seq(max_seq);
+ gen = lru_get_gen_flags(old_flags);
+ refs = lru_get_refs_flags(old_flags) + 1;
+ *type = folio_flags_is_file_lru(&old_flags);
+ new_flags = old_flags;
+
+ if (gen >= 0 && gen != max_gen) {
+ ret = gen;
+ /* Promote second page table access or executable */
+ if (refs > LRU_REFS_REFERENCED || is_exec_file_folio(folio, vma_flags))
+ *new_gen = max_gen;
+ /* First access only defers eviction of a tail gen */
+ else if (lru_gen_at_tail(lruvec, *type, gen))
+ *new_gen = (gen + 1) % MAX_NR_GENS;
+ else
+ *new_gen = gen;
+ lru_set_gen_flags(&new_flags, *new_gen);
+ lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_PROTECTED));
+ } else {
+ ret = -1;
+ lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_MAX));
+ }
+ if (new_flags == old_flags)
+ break;
+ } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+
+ return ret;
+}
+
+/*
+ * Update the lru refs indicator of an isolated folio, only used on
+ * mapped folios upon the final eviction.
+ *
+ * Increments the refs count (capped at LRU_REFS_PROTECTED, evict_folios()
+ * then caps it at LRU_REFS_WORKINGSET). Returns true if the caller should
+ * activate the folio (second access or executable), false to put it back
+ * for a second chance.
+ */
+static bool folio_inc_lru_refs_isolated(struct folio *folio, const vma_flags_t *vma_flags)
+{
+ unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+ int refs;
+
+ do {
+ new_flags = old_flags;
+ refs = lru_get_refs_flags(old_flags) + 1;
+ lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_PROTECTED));
+ } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+
+ /* Promote second page table access or executable */
+ return refs > LRU_REFS_REFERENCED || is_exec_file_folio(folio, vma_flags);
}
#else
-static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
+static bool folio_inc_lru_refs_isolated(struct folio *folio, const vma_flags_t *vma_flags)
{
return false;
}
@@ -997,7 +1145,8 @@ static enum folio_references folio_check_references(struct folio *folio,
if (!referenced_ptes)
return FOLIOREF_RECLAIM;
- return lru_gen_set_refs(folio, &vma_flags) ? FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
+ return folio_inc_lru_refs_isolated(folio, &vma_flags) ?
+ FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
}
referenced_folio = folio_test_clear_referenced(folio);
@@ -3298,9 +3447,9 @@ static bool iterate_mm_list_nowalk(struct lruvec *lruvec, unsigned long seq)
* P term over the generations previously evicted, using the smoothing factor
* 1/2; the D term isn't supported.
*
- * The setpoint (SP) is always the first tier of one type; the process variable
- * (PV) is either any tier of the other type or any other tier of the same
- * type.
+ * To select a type, the setpoint (SP) is all tiers of one type and the process
+ * variable (PV) is all tiers of the other type. To select a tier to protect,
+ * the SP is the tiers below it and the PV is the tier itself.
*
* The error is the difference between the SP and the PV; the correction is to
* turn off protection when SP>PV or turn on protection when SP<PV.
@@ -3388,50 +3537,15 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
* the aging
******************************************************************************/
-/* promote pages accessed through page tables */
-static int folio_update_gen(struct folio *folio, int new_gen, int *type,
- const vma_flags_t *vma_flags)
-{
- unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
- int old_gen;
-
- /*
- * See the comment on LRU_REFS_FLAGS, and activate file-backed
- * executable folios after first usage to avoid typical IO
- * thrashing from reclaiming.
- */
- if (!folio_test_referenced(folio) && !folio_test_workingset(folio) &&
- !is_exec_file_folio(folio, vma_flags)) {
- folio_set_lru_refs(folio, 1);
- return -1;
- }
-
- do {
- old_gen = lru_get_gen_flags(old_flags);
- new_flags = old_flags;
-
- /* lru_gen_del_folio() has isolated this page? */
- if (old_gen < 0)
- break;
-
- lru_set_gen_flags(&new_flags, new_gen);
- lru_set_refs_flags(&new_flags, 0);
- new_flags |= BIT(PG_workingset);
- } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
-
- *type = folio_flags_is_file_lru(&old_flags);
- return old_gen;
-}
-
static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
{
unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
- int new_gen;
+ int refs, new_gen;
do {
new_gen = lru_get_gen_flags(old_flags);
- /* folio_update_gen() has promoted this page? */
+ /* folio_inc_lru_refs() has promoted this page? */
if (new_gen >= 0 && new_gen != old_gen) {
if (increased)
*increased = false;
@@ -3440,9 +3554,9 @@ static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
new_flags = old_flags;
new_gen = (old_gen + 1) % MAX_NR_GENS;
-
+ refs = lru_get_refs_flags(old_flags);
lru_set_gen_flags(&new_flags, new_gen);
- lru_set_refs_flags(&new_flags, 0);
+ lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_WORKINGSET));
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
if (increased)
@@ -3450,17 +3564,21 @@ static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
return new_gen;
}
-/* protect pages accessed multiple times through file descriptors */
+/*
+ * Force bump a folio's generation. Used for PID protection or to skip a
+ * folio from a zone ineligible for the current reclaim.
+ */
static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
{
+ bool gen_increased;
int type = folio_is_file_lru(folio);
struct lru_gen_folio *lrugen = &lruvec->lrugen;
int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
- bool gen_increased;
new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
if (gen_increased)
lru_gen_update_size(lruvec, folio, old_gen, new_gen);
+
return new_gen;
}
@@ -3655,25 +3773,25 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struc
struct lruvec *lruvec, struct folio *folio, bool dirty)
{
int new_gen, old_gen, type;
+ unsigned int flags = LRU_REF_MAPPED;
if (!folio)
return;
- new_gen = lru_gen_from_seq(READ_ONCE(lruvec->lrugen.max_seq));
-
if (dirty && !folio_test_dirty(folio) &&
!(folio_test_anon(folio) && folio_test_swapbacked(folio) &&
!folio_test_swapcache(folio)))
folio_mark_dirty(folio);
if (walk) {
- old_gen = folio_update_gen(folio, new_gen, &type, &vma->flags);
+ old_gen = folio_inc_lru_refs_walk(folio, lruvec, &vma->flags,
+ &new_gen, &type);
if (old_gen >= 0 && old_gen != new_gen)
update_batch_size(walk, folio, old_gen, new_gen, type);
- } else if (lru_gen_set_refs(folio, &vma->flags)) {
- old_gen = folio_lru_gen(folio);
- if (old_gen >= 0 && old_gen != new_gen)
- folio_activate(folio);
+ } else {
+ if (is_exec_file_folio(folio, &vma->flags))
+ flags |= LRU_REF_EXEC;
+ folio_inc_lru_refs(folio, flags);
}
}
@@ -4080,7 +4198,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
struct folio *folio = list_entry(pos, struct folio, lru);
long nr_pages = folio_nr_pages(folio);
int refs = folio_lru_refs(folio);
- bool workingset = folio_test_workingset(folio);
+ int tier = lru_tier_from_refs(refs);
bool gen_increased;
VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
@@ -4100,13 +4218,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
delta += nr_pages;
batch_end = &folio->lru;
- /* don't count the workingset being lazily promoted */
- if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
- int tier = lru_tier_from_refs(refs, workingset);
-
- WRITE_ONCE(lrugen->protected[hist][type][tier],
- lrugen->protected[hist][type][tier] + nr_pages);
- }
+ WRITE_ONCE(lrugen->protected[hist][type][tier],
+ lrugen->protected[hist][type][tier] + nr_pages);
} else {
flush_lru_batch(head, &batch_end, target_list);
list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
@@ -4821,8 +4934,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
int refs = folio_lru_refs(folio);
- bool workingset = folio_test_workingset(folio);
- int tier = lru_tier_from_refs(refs, workingset);
+ int tier = lru_tier_from_refs(refs);
struct lru_gen_folio *lrugen = &lruvec->lrugen;
VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio);
@@ -4838,17 +4950,15 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c
}
/* protected */
- if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1) {
+ if (tier > tier_idx) {
+ int hist = lru_hist_from_seq(lrugen->min_seq[type]);
+
gen = folio_inc_gen(lruvec, folio);
list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
- /* don't count the workingset being lazily promoted */
- if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
- int hist = lru_hist_from_seq(lrugen->min_seq[type]);
+ WRITE_ONCE(lrugen->protected[hist][type][tier],
+ lrugen->protected[hist][type][tier] + delta);
- WRITE_ONCE(lrugen->protected[hist][type][tier],
- lrugen->protected[hist][type][tier] + delta);
- }
return true;
}
@@ -4876,10 +4986,6 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
return false;
}
- /* see the comment on LRU_REFS_FLAGS */
- if (!folio_test_referenced(folio))
- folio_set_lru_refs(folio, 0);
-
success = lru_gen_del_folio(lruvec, folio, true);
VM_WARN_ON_ONCE_FOLIO(!success, folio);
@@ -4967,13 +5073,14 @@ static int get_tier_idx(struct lruvec *lruvec, int type)
struct ctrl_pos sp, pv;
/*
- * To leave a margin for fluctuations, use a larger gain factor (2:3).
- * This value is chosen because any other tier would have at least twice
- * as many refaults as the first tier.
+ * To leave a margin for fluctuations, use a larger gain factor (1:2).
+ * Stop at the first tier whose refault rate is clearly worse than
+ * that of the cumulative mass of the tiers below it; the PID
+ * protects the tiers above it.
*/
- read_ctrl_pos(lruvec, type, LRU_TIER_MIN, LRU_TIER_MIN, 2, &sp);
for (tier = LRU_TIER_MIN + 1; tier <= LRU_TIER_MAX; tier++) {
- read_ctrl_pos(lruvec, type, tier, tier, 3, &pv);
+ read_ctrl_pos(lruvec, type, LRU_TIER_MIN, tier - 1, 1, &sp);
+ read_ctrl_pos(lruvec, type, tier, tier, 2, &pv);
if (!positive_ctrl_err(&sp, &pv))
break;
}
@@ -5115,12 +5222,13 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
}
/*
- * See the comments on LRU_REFS_FLAGS.
- *
* The rejected folios are never added to the oldest generation,
* so this effectively promotes them by at least one generation.
+ * PG_active is the only placement hint here, so a folio kept on
+ * its first page table access lands in the second newest one.
+ * See "Referenced count feedback" above.
*/
- folio_set_lru_refs(folio, 0);
+ folio_set_lru_refs(folio, min(folio_lru_refs(folio), LRU_REFS_WORKINGSET));
if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type])
folio_set_active(folio);
}
diff --git a/mm/workingset.c b/mm/workingset.c
index 1504f91cdca5..16cdd88e275e 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -188,6 +188,12 @@
#define EVICTION_SHIFT_ANON (EVICTION_SHIFT + SWAP_COUNT_SHIFT)
#define EVICTION_MASK (~0UL >> EVICTION_SHIFT)
#define EVICTION_MASK_ANON (~0UL >> EVICTION_SHIFT_ANON)
+/*
+ * LRU refs uses LRU_REFS_WIDTH + 2 bits, the 2 bits being PG_workingset
+ * and PG_referenced. The lowest bit is recorded in the workingset field
+ * of the shadow entry (to reuse pack_shadow()).
+ */
+#define LRU_REFS_BITS ((LRU_REFS_WIDTH + 2) - 1)
/*
* Eviction timestamps need to be able to cover the full range of
@@ -242,13 +248,12 @@ static void *lru_gen_eviction(struct folio *folio)
int type = folio_is_file_lru(folio);
int delta = folio_nr_pages(folio);
int refs = folio_lru_refs(folio);
- bool workingset = folio_test_workingset(folio);
- int tier = lru_tier_from_refs(refs, workingset);
+ int tier = lru_tier_from_refs(refs);
struct mem_cgroup *memcg;
struct pglist_data *pgdat = folio_pgdat(folio);
unsigned short memcg_id;
- BUILD_BUG_ON(LRU_GEN_WIDTH + LRU_REFS_WIDTH >
+ BUILD_BUG_ON(LRU_GEN_WIDTH + LRU_REFS_BITS >
BITS_PER_LONG - max(EVICTION_SHIFT, EVICTION_SHIFT_ANON));
rcu_read_lock();
@@ -256,14 +261,14 @@ static void *lru_gen_eviction(struct folio *folio)
lruvec = mem_cgroup_lruvec(memcg, pgdat);
lrugen = &lruvec->lrugen;
min_seq = READ_ONCE(lrugen->min_seq[type]);
- token = (min_seq << LRU_REFS_WIDTH) | max(refs - 1, 0);
+ token = (min_seq << LRU_REFS_BITS) | refs >> 1;
hist = lru_hist_from_seq(min_seq);
atomic_long_add(delta, &lrugen->evicted[hist][type][tier]);
memcg_id = mem_cgroup_private_id(memcg);
rcu_read_unlock();
- return pack_shadow(memcg_id, pgdat, token, workingset, type);
+ return pack_shadow(memcg_id, pgdat, token, refs & 1, type);
}
/*
@@ -284,9 +289,9 @@ static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec,
*lruvec = mem_cgroup_lruvec(memcg, pgdat);
max_seq = READ_ONCE((*lruvec)->lrugen.max_seq);
- max_seq &= (file ? EVICTION_MASK : EVICTION_MASK_ANON) >> LRU_REFS_WIDTH;
+ max_seq &= (file ? EVICTION_MASK : EVICTION_MASK_ANON) >> LRU_REFS_BITS;
- return abs_diff(max_seq, *token >> LRU_REFS_WIDTH) < MAX_NR_GENS;
+ return abs_diff(max_seq, *token >> LRU_REFS_BITS) < MAX_NR_GENS;
}
static void lru_gen_refault(struct folio *folio, void *shadow)
@@ -314,22 +319,26 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
lrugen = &lruvec->lrugen;
hist = lru_hist_from_seq(READ_ONCE(lrugen->min_seq[type]));
- refs = (token & (BIT(LRU_REFS_WIDTH) - 1)) + 1;
- tier = lru_tier_from_refs(refs, workingset);
+ refs = ((token & (BIT(LRU_REFS_BITS) - 1)) << 1) + workingset;
+ tier = lru_tier_from_refs(refs);
atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
- if (workingset) {
- /*
- * see folio_add_lru(), where folio_set_active() is
- * called for workingset folios
- */
- if (lru_gen_in_fault())
+ /*
+ * Activate a fault-driven refault folio, which would have been
+ * promoted had it stayed in memory.
+ */
+ if (refs >= LRU_REFS_REFERENCED) {
+ if (lru_gen_in_fault()) {
+ folio_set_active(folio);
mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
- folio_set_workingset(folio);
+ }
+ /* Refault is also promotion, cap the refs like folio_inc_lru_refs */
+ folio_set_lru_refs(folio, min(refs, LRU_REFS_PROTECTED));
+ }
+
+ if (refs >= LRU_REFS_WORKINGSET)
mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + type, delta);
- } else
- set_mask_bits(&folio->flags.f, LRU_REFS_MASK, (refs - 1UL) << LRU_REFS_PGOFF);
unlock:
rcu_read_unlock();
}
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 04/17] mm/mglru: frequency guided workingset promotion (MGLRU-FG)
@ 2026-10-15 18:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
Complement MGLRU's eviction-time tier-PID protection with access-time
frequency-guided promotion. Introduce a unified set of helpers built based
on referenced (access) count of a folio.
Each access increments a folio's referenced count stored in folio flags
(refs), refs still maps to a logarithmic tier just like before, but with
more formal bit definitions, a few special thresholds are introduced:
LRU_REFS_REFERENCED (1), LRU_REFS_WORKINGSET (2), LRU_REFS_PROTECTED (3),
and LRU_REFS_MAX (7). When refs reaches a certain threshold, the folio is
promoted proactively instead of waiting for the PID controller to kick in.
Also simplify MGLRU's usage of PG_workingset and PG_referenced: they
become the low two bits of the refs count, with the higher bits
provided by LRU_REFS_MASK. This reduces MGLRU's original refs count
bit usage by one, since only one extra bit is now needed to record a
max referenced count of 7, and makes MGLRU's refs accounting more
accurate.
This doesn't affect classical LRU in any way, and it addresses several
shortcomings of MGLRU's old tier-only cache protection model:
- Long feedback loop: protection only activated after enough re-faults,
by which time the hot folios are already evicted, or no longer hot.
- Limited tier resolution: once referenced count exceeded the bits
limit (8 previously), MGLRU could no longer distinguish hotter folios as
they are capped by the tier. And what's worse, PG_workingset forces
a folio to stay on tier 3.
- Eviction hotness reversion: because PID protection activates upon
eviction and always targets the LRU tail, it tends to protect cold tail
folios at the expense of hotter head folios. Once the tail folios
consume the PID protection budget, head folios lose their protection.
- Additionally, the PID cannot distinguish the access time of folios
that share the same reference count, and there are only 4 tiers.
To achieve a frequency-guided framework, this commit introduces and reworks
the LRU_REFS related helpers and definitions; most of the work is done by
the helpers below, and their inline comments describe the details.
- folio_inc_lru_refs(): Called on any cache access (folio_mark_accessed)
or page table access. This is the main helper: it promotes folios
according to their access frequency.
A first access only defers eviction for a folio at the tail of its
LRU, i.e., in one of the MIN_NR_GENS oldest generations of its own
type (see lru_gen_at_tail()), and a workingset folio accessed through
a file descriptor is deferred from the same tail. Those are the
generations the eviction scans next, so the deferral moves the folio
out of its way for one generation.
Promotion is lazy: the gen bits and size counters are updated eagerly,
while the list move is deferred to the next isolation. NOTE: For now,
the lruvec lock is unconditionally taken on every on-list access to
block concurrent aging; a lockless fast path will be implemented very
soon in a following commit.
- folio_inc_lru_refs_walk(): Used by the PTE walk path during aging and
by the batched look-around, both without the LRU lock, so max_seq may
be stale. A first page table access only advances a folio at the tail
of its LRU, i.e., in one of the MIN_NR_GENS oldest generations of its
own type (second chance), a second access promotes it to the newest
generation; this also performs lazy promotion.
- folio_inc_lru_refs_isolated(): Used by the rmap check before
eviction. The folio is isolated and hence this doesn't perform
promotion by itself; the folio will be added back to the right gen
upon return according to the access frequency. This path also has a
higher promotion bias.
The eviction-time folio_inc_gen() still handles PID protection, but the
protection ratio is softer than before since proactive promotion is
mostly good enough already. The PID gain factors are relaxed from
(2:3) to (1:2) and the setpoint now spans the cumulative mass of the
tiers below the candidate instead of tier 0 alone. folio_inc_gen()
caps refs at WORKINGSET so the folio retains enough history to stay
above the cold tier. Tier 1 is the fallback tier for PID, and tier 2
is the fallback tier for frequency-guided promotion. The forced
protection for full-refs folios is removed, obsoleted by the proactive
promotion.
Refaults are now activated purely according to access frequency:
the old fault bias applied in folio_add_lru() is simplified, since a
folio's access history is a more consistent signal than the context of
the faulting task. Page table access is still considered a slightly
stronger signal.
Consequently the placement hint for newly added file folios also uses
LRU_REFS_WORKINGSET instead of PG_referenced, so a folio accessed only
once now starts in the oldest generation rather than the second oldest.
The previous one-generation grace was an artifact of the fault bias which
is well described in the cover letter of this series, and it depended
on LRU drain timing.
This also redefines PG_workingset and PG_referenced as the low two
bits of the refs count, eliminating the old restriction where
LRU_REFS_MASK was only valid when PG_referenced was set, and allows
all paths to use the same encoding consistently. Following this idea,
a workingset folio is now defined as refs >= LRU_REFS_WORKINGSET (2),
matching the active/inactive LRU's definition and giving in-kernel
consumers (PSI, readahead) consistent behavior on MGLRU, fixing the
longstanding issue that these users don't work well with MGLRU.
PG_workingset and PG_referenced are no longer independent flags under
MGLRU. Adjusting existing raw folio_test_*() callers to the new
semantics is left as follow-ups.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 96 ++++++++------
include/linux/mmzone.h | 145 +++++++++++++++------
kernel/bounds.c | 2 +-
mm/folio.c | 50 +-------
mm/vmscan.c | 320 +++++++++++++++++++++++++++++++---------------
mm/workingset.c | 45 ++++---
6 files changed, 409 insertions(+), 249 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index f52f02e8e5be..74d1b7a04ca1 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -144,12 +144,13 @@ static inline int lru_hist_from_seq(unsigned long seq)
return seq % NR_HIST_GENS;
}
-static inline int lru_tier_from_refs(int refs, bool workingset)
+static inline int lru_tier_from_refs(unsigned int refs)
{
- VM_WARN_ON_ONCE(refs > BIT(LRU_REFS_WIDTH));
-
- /* see the comment on MAX_NR_TIERS */
- return workingset ? MAX_NR_TIERS - 1 : order_base_2(refs);
+ BUILD_BUG_ON(fls(LRU_REFS_MAX - 1) > MAX_NR_TIERS - 1);
+ VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
+ if (refs < LRU_REFS_WORKINGSET)
+ return 0;
+ return fls(refs - 1);
}
/**
@@ -187,21 +188,24 @@ static inline int lru_get_gen_flags(unsigned long flags)
* @flags: pointer to the folio flags
* @refs: referenced / access count number, between 0 and LRU_REFS_MAX, inclusive.
*
- * For MGLRU, PG_referenced holds the first ref, and the extra bits hold the
- * remaining refs. For classical LRU the extra bits are not used, so it can
- * also be seen as the refs count never exceeds 1. In both cases, refs == 1
- * means PG_referenced is set and the extra bits are zero, and refs == 0 means
- * PG_referenced and the extra bits are all unset.
+ * For MGLRU, PG_referenced, PG_workingset are used as the lower two bits of
+ * refs counter, and extra bits hold the remaining higher bits. For classical
+ * LRU the extra bits are not used, and the two flags has no direct
+ * relationship with each other, but this helper can still be used to sync
+ * them. For both cases, refs == 0 means these two flags and the extra bits
+ * are all unset. And refs == 1 / 2 / 3 means PG_referenced and PG_workingset
+ * are set in an bit order way, which is more meaningful for MGLRU though.
*/
static inline void lru_set_refs_flags(unsigned long *flags, unsigned int refs)
{
VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
- BUILD_BUG_ON(LRU_REFS_MAX != (LRU_REFS_MASK >> LRU_REFS_PGOFF) + 1);
-
+ BUILD_BUG_ON(LRU_REFS_MASK & (BIT(PG_referenced) | BIT(PG_workingset)));
*flags &= ~LRU_REFS_FLAGS;
- if (!refs)
- return;
- *flags |= (BIT(PG_referenced) | ((refs - 1UL) << LRU_REFS_PGOFF));
+ if (refs & BIT(0))
+ *flags |= BIT(PG_referenced);
+ if (refs & BIT(1))
+ *flags |= BIT(PG_workingset);
+ *flags |= ((unsigned long)refs >> 2) << LRU_REFS_PGOFF;
}
/**
@@ -212,13 +216,13 @@ static inline void lru_set_refs_flags(unsigned long *flags, unsigned int refs)
*/
static inline int lru_get_refs_flags(unsigned long flags)
{
- if (!(flags & BIT(PG_referenced)))
- return 0;
- /*
- * Return the total number of accesses including PG_referenced. Also see
- * the comment on LRU_REFS_FLAGS.
- */
- return ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) + 1;
+ int refs;
+
+ /* Return the total number of accesses. See the comment above MAX_NR_TIERS. */
+ refs = (flags & BIT(PG_referenced)) ? BIT(0) : 0;
+ refs += (flags & BIT(PG_workingset)) ? BIT(1) : 0;
+ refs += ((flags & LRU_REFS_MASK) >> LRU_REFS_PGOFF) << 2;
+ return refs;
}
static inline int folio_lru_refs(const struct folio *folio)
@@ -236,14 +240,29 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
}
+void folio_inc_lru_refs(struct folio *folio, unsigned int flags);
+
static inline int folio_lru_gen(const struct folio *folio)
{
return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
}
+static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
+{
+ unsigned long min_seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
+
+ /*
+ * Consider a folio is at the tail of its LRU when it is within
+ * MIN_NR_GENS of the oldest generation of its own type
+ */
+ BUILD_BUG_ON(MIN_NR_GENS != 2);
+ return gen == lru_gen_from_seq(min_seq) ||
+ gen == lru_gen_from_seq(min_seq + 1);
+}
+
static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
{
- unsigned long max_seq = lruvec->lrugen.max_seq;
+ unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
@@ -300,23 +319,24 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
bool reclaiming)
{
int gen;
+ int refs = folio_lru_refs(folio);
int type = folio_is_file_lru(folio);
const struct lru_gen_folio *lrugen = &lruvec->lrugen;
/*
- * +-----------------------------------+-----------------------------------+
- * | Accessed through page tables and | Accessed through file descriptors |
- * | promoted by folio_update_gen() | and protected by folio_inc_gen() |
- * +-----------------------------------+-----------------------------------+
- * | PG_active (set while isolated) | |
- * +-----------------+-----------------+-----------------+-----------------+
- * | PG_workingset | PG_referenced | PG_workingset | LRU_REFS_FLAGS |
- * +-----------------------------------+-----------------------------------+
- * |<---------- MIN_NR_GENS ---------->| |
- * |<---------------------------- MAX_NR_GENS ---------------------------->|
+ * +------------------------------------------+------------------------------------------+
+ * | Accessed through page tables and | Accessed through file descriptors |
+ * | promoted by folio_inc_lru_refs_walk() | protected by folio_inc_lru_refs/inc_gen |
+ * +------------------------------------------+------------------------------------------+
+ * | PG_active (set at isolation or refault) | |
+ * +--------------------+---------------------+--------------------+---------------------+
+ * | LRU_REFS_MAX | LRU_REFS_WORKINGSET | LRU_REFS_MAX | LRU_REFS_WORKINGSET |
+ * +------------------------------------------+------------------------------------------+
+ * |<-------------- MIN_NR_GENS ------------->| |
+ * |<----------------------------------- MAX_NR_GENS ----------------------------------->|
*/
if (folio_test_active(folio))
- gen = MIN_NR_GENS - folio_test_workingset(folio);
+ gen = MIN_NR_GENS - (refs >= LRU_REFS_WORKINGSET);
else if (reclaiming)
gen = MAX_NR_GENS;
else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) ||
@@ -324,7 +344,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
(folio_test_dirty(folio) || folio_test_writeback(folio))))
gen = MIN_NR_GENS;
else
- gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
+ gen = MAX_NR_GENS - (refs >= LRU_REFS_WORKINGSET);
return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
}
@@ -338,6 +358,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
int zone = folio_zonenum(folio);
struct lru_gen_folio *lrugen = &lruvec->lrugen;
+ BUILD_BUG_ON(BIT(LRU_GEN_WIDTH - 1) != MAX_NR_GENS);
VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
if (folio_test_unevictable(folio) || !lrugen->enabled)
@@ -392,7 +413,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
*/
static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
{
- BUILD_BUG_ON(LRU_REFS_MASK & BIT(PG_referenced));
folio_set_lru_refs(new, folio_lru_refs(old));
}
#else /* !CONFIG_LRU_GEN */
@@ -422,6 +442,10 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
return false;
}
+static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
+{
+}
+
static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
{
if (folio_test_referenced(old))
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 09ce82fa841a..b5a3769fdcab 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -453,57 +453,122 @@ enum lruvec_flags {
#define MAX_NR_GENS 4U
/*
- * Each generation is divided into multiple tiers. A folio accessed N times
- * through file descriptors is in tier order_base_2(N). A folio in the first
- * tier (N=0,1) is marked by PG_referenced unless it was faulted in through page
- * tables or read ahead. A folio in the last tier (MAX_NR_TIERS-1) is marked by
- * PG_workingset. A folio in any other tier (1<N<5) between the first and last
- * is marked by additional bits of LRU_REFS_WIDTH in folio->flags.
+ * Each generation is divided into multiple tiers. A folio's referenced
+ * count maps to a tier as shown below:
*
- * In contrast to moving across generations which requires the LRU lock, moving
- * across tiers only involves atomic operations on folio->flags and therefore
- * has a negligible cost in the buffered access path. In the eviction path,
- * comparisons of refaulted/(evicted+protected) from the first tier and the rest
- * infer whether folios accessed multiple times through file descriptors are
- * statistically hot and thus worth protecting.
+ * MGLRU (frequency guidance)
+ * Refs Tier |- Refs: how many times (at least) a folio has been referenced.
+ * 0 0 |- Mostly cold pages, readahead, etc. [1]
+ * 1 0 |= LRU_REFS_REFERENCED: Used at least once. [2]
+ * -WORKINGSET-+|- Pages beyond are workingset and never fall below this floor. [3]
+ * 2 1<-+|= LRU_REFS_WORKINGSET: Classical workingset, accessed twice, protected. [4]
+ * 3 2 |- LRU_REFS_PROTECTED: Protected workingset, promoted pages capped at here. [5]
+ * 4 2 |
+ * 5 3 |- The tier here is MAX_NR_TIERS - 1
+ * 6 3 |
+ * 7 3 |= LRU_REFS_MAX: Promotion candidate. [6]
+ * -PROMOTION->-/
*
- * MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice the
- * number of categories of the active/inactive LRU when keeping track of
- * accesses through file descriptors. This uses MAX_NR_TIERS-2 spare bits in
- * folio->flags, masked by LRU_REFS_MASK.
+ * Ideally each tier holds folios of similar access patterns: lower tiers
+ * are less important and evicted faster. A page's reference count and
+ * tier are capped when it changes generation, preventing it from
+ * dominating the new generation based on old-generation access history.
+ * Generation ordering already ensures a newer-gen page is hotter than an
+ * older-gen one regardless of tier.
+ *
+ * Refs tracks accesses from two sources: page table (lazily collected by
+ * the page table aging walk or rmap eviction lookup) and file descriptors
+ * (by folio_mark_accessed). Page table accesses are weighted heavier
+ * because the accessed bit is sticky (undercounts repeated accesses),
+ * passively collected, and page faults are generally more important as
+ * userspace does not expect a memory access to block on reclaim. Both
+ * access types increment refs by one; the result is capped at
+ * LRU_REFS_PROTECTED on promotion or deferral, or LRU_REFS_MAX otherwise.
+ *
+ * 1. Tier is fls(N-1) for N > 1, 0 otherwise. Folios with zero
+ * accesses (refs == 0) are generally cold, e.g. readahead folios.
+ *
+ * Freshly allocated folios start with refs == 0; faulted and mapped
+ * folios have their page table access bit set, so the first page table
+ * access check always sets LRU_REFS_REFERENCED. That is the second
+ * chance described above MIN_NR_GENS: it moves the folio one
+ * generation forward if it is at the tail of its LRU, i.e., in one
+ * of the two oldest generations of its type.
+ *
+ * 2. Folios accessed once stay on tier 0: one-time usage does not
+ * qualify for protection. A second access advances the folio,
+ * aligning with classical LRU's use-twice threshold. A second page
+ * table access promotes to the latest gen; file access only defers
+ * eviction from the tail gens of its type.
+ *
+ * 3. Folios accessed at least twice are considered workingset. This
+ * mostly aligns with classical LRU: at least one I/O is saved by
+ * keeping them in memory. Folios at or above this level never fall
+ * below tier 1 (the workingset floor), so tier 0 stays a clean tier
+ * for cold cache while tier 1 serves as the fallback line for
+ * actually reused or historically hot folios.
+ *
+ * Folios refaulted through a page fault at refs 1 will enter the second
+ * newest gen, so faulting will be protected better.
+ *
+ * 4. Starting from tier 1, PID protection sacrifices lower tiers to
+ * protect higher tiers by comparing refault rates for long-term
+ * accuracy, and caps higher refs to this value. Since PID protection
+ * bypasses page table lookup and clearing, when a further eviction
+ * attempt occurs after PID loosens, the folio's page table access is
+ * rechecked and a referenced folio is put back with refs capped to
+ * this value as well. This also gives folios a fair opportunity to be
+ * promoted by file access again.
+ *
+ * Folios refaulted through a page fault at tier 1 or above are activated
+ * and enter the newest gen. Non fault page will enter second oldest gen,
+ * driving aging and workingset shifting.
+ *
+ * 5. Pages beyond the ordinary workingset tier form new tiers for the
+ * PID controller to protect differently. Folios at or above this
+ * level are capped at LRU_REFS_PROTECTED on promotion or deferral,
+ * and at LRU_REFS_WORKINGSET under PID protection in the oldest
+ * generation, where they represent a historical workingset.
+ *
+ * 6. Folios that reach LRU_REFS_MAX are advanced to the next generation
+ * on further access, with refs capped to LRU_REFS_PROTECTED. This
+ * gives them a fair start for advancement to an even newer generation
+ * while keeping hot folios distinguishable.
+ *
+ * Tiering uses PG_referenced and PG_workingset as the lower two bits,
+ * and the bits masked by LRU_REFS_MASK as the higher bits, so the refs
+ * count ranges from 0 to LRU_REFS_MAX. A folio is on the workingset
+ * tier once accessed at least twice, which is more consistent with the
+ * classical LRU.
+ *
+ * A folio's referenced count never goes backwards except upon gen
+ * increase as described above, or when explicitly reset by
+ * lru_gen_clear_refs(). Refault of a reclaimed folio restores
+ * its referenced count, capped at LRU_REFS_PROTECTED, which aligns with
+ * promotion. Page table refaults of previous workingset folios send
+ * them to the latest gen, driving aging faster.
+ *
+ * MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice
+ * the number of categories of the active/inactive LRU.
*/
#define MAX_NR_TIERS 4U
#define LRU_TIER_MIN 0U
#define LRU_TIER_MAX (MAX_NR_TIERS - 1)
+/* Access source flags for folio_inc_lru_refs() */
+#define LRU_REF_MAPPED 0x1U
+#define LRU_REF_EXEC 0x2U
+
+#define LRU_REFS_REFERENCED 0x1
+#define LRU_REFS_WORKINGSET 0x2
+#define LRU_REFS_PROTECTED 0x3
+
#ifndef __GENERATING_BOUNDS_H
#define LRU_GEN_MASK ((BIT(LRU_GEN_WIDTH) - 1) << LRU_GEN_PGOFF)
#define LRU_REFS_MASK ((BIT(LRU_REFS_WIDTH) - 1) << LRU_REFS_PGOFF)
-#define LRU_REFS_MAX BIT(LRU_REFS_WIDTH)
-
-/*
- * For folios accessed multiple times through file descriptors,
- * lru_gen_inc_refs() sets additional bits of LRU_REFS_WIDTH in folio->flags
- * after PG_referenced, then PG_workingset after LRU_REFS_WIDTH. After all its
- * bits are set, i.e., LRU_REFS_FLAGS|BIT(PG_workingset), a folio is lazily
- * promoted into the second oldest generation in the eviction path. And when
- * folio_inc_gen() does that, it clears LRU_REFS_FLAGS so that
- * lru_gen_inc_refs() can start over. Note that for this case, LRU_REFS_MASK is
- * only valid when PG_referenced is set.
- *
- * For folios accessed multiple times through page tables, folio_update_gen()
- * from a page table walk or lru_gen_set_refs() from a rmap walk sets
- * PG_referenced after the accessed bit is cleared for the first time.
- * Thereafter, those two paths set PG_workingset and promote folios to the
- * youngest generation. Like folio_inc_gen(), folio_update_gen() also clears
- * PG_referenced. Note that for this case, LRU_REFS_MASK is not used.
- *
- * For both cases above, after PG_workingset is set on a folio, it remains until
- * this folio is either reclaimed, or "deactivated" by lru_gen_clear_refs(). It
- * can be set again if lru_gen_test_recent() returns true upon a refault.
- */
-#define LRU_REFS_FLAGS (LRU_REFS_MASK | BIT(PG_referenced))
+#define LRU_REFS_FLAGS (LRU_REFS_MASK | BIT(PG_referenced) | BIT(PG_workingset))
+#define LRU_REFS_MAX (BIT(LRU_REFS_WIDTH + 2) - 1)
struct lruvec;
struct page_vma_mapped_walk;
diff --git a/kernel/bounds.c b/kernel/bounds.c
index 02b619eb6106..06a034713b5d 100644
--- a/kernel/bounds.c
+++ b/kernel/bounds.c
@@ -25,7 +25,7 @@ int main(void)
DEFINE(SPINLOCK_SIZE, sizeof(spinlock_t));
#ifdef CONFIG_LRU_GEN
DEFINE(LRU_GEN_WIDTH, order_base_2(MAX_NR_GENS + 1));
- DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 2);
+ DEFINE(__LRU_REFS_WIDTH, MAX_NR_TIERS - 3);
#else
DEFINE(LRU_GEN_WIDTH, 0);
DEFINE(__LRU_REFS_WIDTH, 0);
diff --git a/mm/folio.c b/mm/folio.c
index 35e242b48870..e200efab814f 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -273,7 +273,6 @@ static void lru_activate(struct lruvec *lruvec, struct folio *folio)
if (folio_test_active(folio) || folio_test_unevictable(folio))
return;
-
lruvec_del_folio(lruvec, folio);
folio_set_active(folio);
lruvec_add_folio(lruvec, folio);
@@ -352,32 +351,6 @@ static void __lru_cache_activate_folio(struct folio *folio)
#ifdef CONFIG_LRU_GEN
-static void lru_gen_inc_refs(struct folio *folio)
-{
- unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
- int refs;
-
- if (folio_test_unevictable(folio))
- return;
-
- /* see the comment on LRU_REFS_FLAGS */
- if (!folio_lru_refs(folio)) {
- folio_set_lru_refs(folio, 1);
- return;
- }
-
- do {
- new_flags = old_flags;
- refs = lru_get_refs_flags(old_flags);
- if (refs == LRU_REFS_MAX) {
- if (!folio_test_workingset(folio))
- folio_set_workingset(folio);
- return;
- }
- lru_set_refs_flags(&new_flags, refs + 1);
- } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
-}
-
static bool lru_gen_clear_refs(struct folio *folio)
{
int gen = folio_lru_gen(folio);
@@ -388,7 +361,6 @@ static bool lru_gen_clear_refs(struct folio *folio)
return true;
folio_set_lru_refs(folio, 0);
- folio_clear_workingset(folio);
rcu_read_lock();
seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
@@ -399,10 +371,6 @@ static bool lru_gen_clear_refs(struct folio *folio)
#else /* !CONFIG_LRU_GEN */
-static void lru_gen_inc_refs(struct folio *folio)
-{
-}
-
static bool lru_gen_clear_refs(struct folio *folio)
{
return false;
@@ -428,7 +396,8 @@ void folio_mark_accessed(struct folio *folio)
if (folio_test_dropbehind(folio))
return;
if (lru_gen_enabled()) {
- lru_gen_inc_refs(folio);
+ if (!folio_test_unevictable(folio))
+ folio_inc_lru_refs(folio, 0);
return;
}
@@ -474,21 +443,6 @@ void folio_add_lru(struct folio *folio)
folio_test_unevictable(folio), folio);
VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
- /*
- * For refaulted workingset folios, set PG_active so they
- * can be added to active generations.
- * For prefaulted file folios, folio_mark_accessed() sets
- * PG_referenced so lru_gen_folio_seq() places them into
- * the second oldest generation.
- */
- if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
- lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
- if (folio_test_workingset(folio))
- folio_set_active(folio);
- else if (!folio_test_referenced(folio))
- folio_mark_accessed(folio);
- }
-
folio_batch_add_and_move(folio, lru_add);
}
EXPORT_SYMBOL(folio_add_lru);
diff --git a/mm/vmscan.c b/mm/vmscan.c
index cab3ba14d20c..d0f0e5a25d87 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -931,38 +931,186 @@ enum folio_references {
};
#ifdef CONFIG_LRU_GEN
+/******************************************************************************
+ * Referenced count feedback
+ ******************************************************************************/
+
/*
- * Only used on a mapped folio in the eviction (rmap walk) path, where promotion
- * needs to be done by taking the folio off the LRU list and then adding it back
- * with PG_active set. In contrast, the aging (page table walk) path uses
- * folio_update_gen().
+ * The folio_inc_lru_refs{_*} helpers below collect the referenced info
+ * (hotness) from other parts, including the page table walker, the rmap walk
+ * upon eviction, the rmap lookaround, and file descriptors
+ * (folio_mark_accessed).
+ *
+ * Page table accesses use refs from either source. The first reference
+ * advances tail-gen folios by one generation; a second promotes to the
+ * newest. Executable file folios promote on their first access to avoid
+ * I/O thrashing. Isolated rmap accesses use evict_folios() for putback.
+ *
+ * Promotion changes folio->flags; sort_folio() or inc_min_seq() moves the
+ * folio later. Young PTEs found during unmapping use folio_mark_accessed()
+ * and follow the file descriptor rules below.
+ *
+ * File descriptor accesses do not promote. They only defer eviction from
+ * the tail gens of their type, and only once the folio is a workingset
+ * folio (LRU_REFS_WORKINGSET), leaving the rest to PID protection. Page
+ * table accesses are treated more generously because the accessed bit is
+ * sticky (it under-counts repeated accesses) and because a page fault is
+ * more costly than file descriptor I/O.
+ *
+ * PID protection operates on tier > 0 folios. The one proactive promotion
+ * outside of it and the page table path is the overflow case where the
+ * referenced count exceeds LRU_REFS_MAX, which means the folio is hotter
+ * than everything else in its generation.
+ *
+ * Whenever a folio changes generation here its referenced count is capped at
+ * LRU_REFS_PROTECTED, so it starts at or below the protected tier regardless
+ * of its old-generation access history. PID protection (folio_inc_gen) caps
+ * at LRU_REFS_WORKINGSET independently.
*/
-static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
-{
- /* see the comment on LRU_REFS_FLAGS */
- if (!folio_test_referenced(folio) && !folio_test_workingset(folio)) {
- /* Activate file-backed executable folios after first usage. */
- if (is_exec_file_folio(folio, vma_flags)) {
- folio_set_workingset(folio);
- folio_set_lru_refs(folio, 0);
- return true;
+
+/*
+ * Update the folio's lru refs indicator. The caller doesn't need to hold
+ * the folio lock, isolate the folio, or hold the lruvec lock. Used by both
+ * cache access (flags == 0) and page table access (LRU_REF_MAPPED,
+ * optionally with LRU_REF_EXEC).
+ */
+void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
+{
+ int max_gen;
+ int type, refs, old_gen, gen;
+ unsigned long new_flags, old_flags, max_seq;
+ struct lru_gen_folio *lrugen;
+ struct lruvec *lruvec = NULL;
+
+ type = folio_is_file_lru(folio);
+ old_flags = READ_ONCE(*folio_flags(folio, 0));
+ do {
+ new_flags = old_flags;
+ old_gen = lru_get_gen_flags(old_flags);
+ refs = lru_get_refs_flags(old_flags) + 1;
+ gen = old_gen;
+ if (old_gen < 0)
+ goto out;
+ /*
+ * Lock the lruvec if the folio is on-list. We are already
+ * doing lazy promotion so in theory we don't need this,
+ * but for now, concurrent aging would still corrupt the
+ * size counters. This is a temporary limitation and
+ * will be lifted very soon, so the lock here is not a
+ * performance concern.
+ */
+ if (!lruvec) {
+ lruvec = lruvec_live_lock_irq(folio_lruvec(folio));
+ lrugen = &lruvec->lrugen;
}
+ max_seq = READ_ONCE(lrugen->max_seq);
+ max_gen = lru_gen_from_seq(max_seq);
+ if (old_gen == max_gen)
+ goto out;
- folio_set_lru_refs(folio, 1);
- return false;
- }
+ if (flags & (LRU_REF_MAPPED | LRU_REF_EXEC)) {
+ /* Promote second page table access or executable */
+ if (refs > LRU_REFS_REFERENCED || flags & LRU_REF_EXEC)
+ gen = max_gen;
+ /* First access only defers eviction of a tail gen */
+ else if (lru_gen_at_tail(lruvec, type, old_gen))
+ gen = (old_gen + 1) % MAX_NR_GENS;
+ refs = min(refs, LRU_REFS_PROTECTED);
+ } else if (refs > LRU_REFS_MAX) {
+ /* LRU refs counting overflow, bump the gen */
+ gen = (old_gen + 1) % MAX_NR_GENS;
+ refs = LRU_REFS_PROTECTED;
+ } else if (lru_gen_at_tail(lruvec, type, old_gen) && refs >= LRU_REFS_WORKINGSET) {
+ /* Defer eviction of just accessed workingset */
+ gen = (old_gen + 1) % MAX_NR_GENS;
+ refs = min(refs, LRU_REFS_PROTECTED);
+ }
+out:
+ refs = min(refs, LRU_REFS_MAX);
+ lru_set_refs_flags(&new_flags, refs);
+ if (gen != old_gen)
+ lru_set_gen_flags(&new_flags, gen);
+ if (new_flags == old_flags)
+ break;
+ } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
- /* Promote on second access */
- if (folio_lru_refs(folio) > 1) {
- folio_set_workingset(folio);
- folio_set_lru_refs(folio, 0);
- } else {
- folio_mark_accessed(folio);
- }
- return true;
+ if (gen != old_gen)
+ lru_gen_update_size(lruvec, folio, old_gen, gen);
+ if (lruvec)
+ lruvec_unlock_irq(lruvec);
+}
+
+/*
+ * Update the folio's lru refs indicator during a page table walk or the
+ * look-around. max_seq can be stale as neither holds the LRU lock.
+ *
+ * Returns the old generation and stores the new generation in @new_gen if
+ * the folio is on the LRU and not in the newest generation, or -1 otherwise.
+ */
+static int folio_inc_lru_refs_walk(struct folio *folio, struct lruvec *lruvec,
+ const vma_flags_t *vma_flags,
+ int *new_gen, int *type)
+{
+ unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+ int refs, gen, max_gen, ret;
+ unsigned long max_seq;
+
+ do {
+ max_seq = READ_ONCE(lruvec->lrugen.max_seq);
+ max_gen = lru_gen_from_seq(max_seq);
+ gen = lru_get_gen_flags(old_flags);
+ refs = lru_get_refs_flags(old_flags) + 1;
+ *type = folio_flags_is_file_lru(&old_flags);
+ new_flags = old_flags;
+
+ if (gen >= 0 && gen != max_gen) {
+ ret = gen;
+ /* Promote second page table access or executable */
+ if (refs > LRU_REFS_REFERENCED || is_exec_file_folio(folio, vma_flags))
+ *new_gen = max_gen;
+ /* First access only defers eviction of a tail gen */
+ else if (lru_gen_at_tail(lruvec, *type, gen))
+ *new_gen = (gen + 1) % MAX_NR_GENS;
+ else
+ *new_gen = gen;
+ lru_set_gen_flags(&new_flags, *new_gen);
+ lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_PROTECTED));
+ } else {
+ ret = -1;
+ lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_MAX));
+ }
+ if (new_flags == old_flags)
+ break;
+ } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+
+ return ret;
+}
+
+/*
+ * Update the lru refs indicator of an isolated folio, only used on
+ * mapped folios upon the final eviction.
+ *
+ * Increments the refs count (capped at LRU_REFS_PROTECTED, evict_folios()
+ * then caps it at LRU_REFS_WORKINGSET). Returns true if the caller should
+ * activate the folio (second access or executable), false to put it back
+ * for a second chance.
+ */
+static bool folio_inc_lru_refs_isolated(struct folio *folio, const vma_flags_t *vma_flags)
+{
+ unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
+ int refs;
+
+ do {
+ new_flags = old_flags;
+ refs = lru_get_refs_flags(old_flags) + 1;
+ lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_PROTECTED));
+ } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+
+ /* Promote second page table access or executable */
+ return refs > LRU_REFS_REFERENCED || is_exec_file_folio(folio, vma_flags);
}
#else
-static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
+static bool folio_inc_lru_refs_isolated(struct folio *folio, const vma_flags_t *vma_flags)
{
return false;
}
@@ -997,7 +1145,8 @@ static enum folio_references folio_check_references(struct folio *folio,
if (!referenced_ptes)
return FOLIOREF_RECLAIM;
- return lru_gen_set_refs(folio, &vma_flags) ? FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
+ return folio_inc_lru_refs_isolated(folio, &vma_flags) ?
+ FOLIOREF_ACTIVATE : FOLIOREF_KEEP;
}
referenced_folio = folio_test_clear_referenced(folio);
@@ -3298,9 +3447,9 @@ static bool iterate_mm_list_nowalk(struct lruvec *lruvec, unsigned long seq)
* P term over the generations previously evicted, using the smoothing factor
* 1/2; the D term isn't supported.
*
- * The setpoint (SP) is always the first tier of one type; the process variable
- * (PV) is either any tier of the other type or any other tier of the same
- * type.
+ * To select a type, the setpoint (SP) is all tiers of one type and the process
+ * variable (PV) is all tiers of the other type. To select a tier to protect,
+ * the SP is the tiers below it and the PV is the tier itself.
*
* The error is the difference between the SP and the PV; the correction is to
* turn off protection when SP>PV or turn on protection when SP<PV.
@@ -3388,50 +3537,15 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
* the aging
******************************************************************************/
-/* promote pages accessed through page tables */
-static int folio_update_gen(struct folio *folio, int new_gen, int *type,
- const vma_flags_t *vma_flags)
-{
- unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
- int old_gen;
-
- /*
- * See the comment on LRU_REFS_FLAGS, and activate file-backed
- * executable folios after first usage to avoid typical IO
- * thrashing from reclaiming.
- */
- if (!folio_test_referenced(folio) && !folio_test_workingset(folio) &&
- !is_exec_file_folio(folio, vma_flags)) {
- folio_set_lru_refs(folio, 1);
- return -1;
- }
-
- do {
- old_gen = lru_get_gen_flags(old_flags);
- new_flags = old_flags;
-
- /* lru_gen_del_folio() has isolated this page? */
- if (old_gen < 0)
- break;
-
- lru_set_gen_flags(&new_flags, new_gen);
- lru_set_refs_flags(&new_flags, 0);
- new_flags |= BIT(PG_workingset);
- } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
-
- *type = folio_flags_is_file_lru(&old_flags);
- return old_gen;
-}
-
static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
{
unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
- int new_gen;
+ int refs, new_gen;
do {
new_gen = lru_get_gen_flags(old_flags);
- /* folio_update_gen() has promoted this page? */
+ /* folio_inc_lru_refs() has promoted this page? */
if (new_gen >= 0 && new_gen != old_gen) {
if (increased)
*increased = false;
@@ -3440,9 +3554,9 @@ static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
new_flags = old_flags;
new_gen = (old_gen + 1) % MAX_NR_GENS;
-
+ refs = lru_get_refs_flags(old_flags);
lru_set_gen_flags(&new_flags, new_gen);
- lru_set_refs_flags(&new_flags, 0);
+ lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_WORKINGSET));
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
if (increased)
@@ -3450,17 +3564,21 @@ static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
return new_gen;
}
-/* protect pages accessed multiple times through file descriptors */
+/*
+ * Force bump a folio's generation. Used for PID protection or to skip a
+ * folio from a zone ineligible for the current reclaim.
+ */
static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
{
+ bool gen_increased;
int type = folio_is_file_lru(folio);
struct lru_gen_folio *lrugen = &lruvec->lrugen;
int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
- bool gen_increased;
new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
if (gen_increased)
lru_gen_update_size(lruvec, folio, old_gen, new_gen);
+
return new_gen;
}
@@ -3655,25 +3773,25 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struc
struct lruvec *lruvec, struct folio *folio, bool dirty)
{
int new_gen, old_gen, type;
+ unsigned int flags = LRU_REF_MAPPED;
if (!folio)
return;
- new_gen = lru_gen_from_seq(READ_ONCE(lruvec->lrugen.max_seq));
-
if (dirty && !folio_test_dirty(folio) &&
!(folio_test_anon(folio) && folio_test_swapbacked(folio) &&
!folio_test_swapcache(folio)))
folio_mark_dirty(folio);
if (walk) {
- old_gen = folio_update_gen(folio, new_gen, &type, &vma->flags);
+ old_gen = folio_inc_lru_refs_walk(folio, lruvec, &vma->flags,
+ &new_gen, &type);
if (old_gen >= 0 && old_gen != new_gen)
update_batch_size(walk, folio, old_gen, new_gen, type);
- } else if (lru_gen_set_refs(folio, &vma->flags)) {
- old_gen = folio_lru_gen(folio);
- if (old_gen >= 0 && old_gen != new_gen)
- folio_activate(folio);
+ } else {
+ if (is_exec_file_folio(folio, &vma->flags))
+ flags |= LRU_REF_EXEC;
+ folio_inc_lru_refs(folio, flags);
}
}
@@ -4080,7 +4198,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
struct folio *folio = list_entry(pos, struct folio, lru);
long nr_pages = folio_nr_pages(folio);
int refs = folio_lru_refs(folio);
- bool workingset = folio_test_workingset(folio);
+ int tier = lru_tier_from_refs(refs);
bool gen_increased;
VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
@@ -4100,13 +4218,8 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
delta += nr_pages;
batch_end = &folio->lru;
- /* don't count the workingset being lazily promoted */
- if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
- int tier = lru_tier_from_refs(refs, workingset);
-
- WRITE_ONCE(lrugen->protected[hist][type][tier],
- lrugen->protected[hist][type][tier] + nr_pages);
- }
+ WRITE_ONCE(lrugen->protected[hist][type][tier],
+ lrugen->protected[hist][type][tier] + nr_pages);
} else {
flush_lru_batch(head, &batch_end, target_list);
list_move(&folio->lru, &lrugen->folios[new_gen][type][zone]);
@@ -4821,8 +4934,7 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
int refs = folio_lru_refs(folio);
- bool workingset = folio_test_workingset(folio);
- int tier = lru_tier_from_refs(refs, workingset);
+ int tier = lru_tier_from_refs(refs);
struct lru_gen_folio *lrugen = &lruvec->lrugen;
VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio);
@@ -4838,17 +4950,15 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c
}
/* protected */
- if (tier > tier_idx || refs + workingset == BIT(LRU_REFS_WIDTH) + 1) {
+ if (tier > tier_idx) {
+ int hist = lru_hist_from_seq(lrugen->min_seq[type]);
+
gen = folio_inc_gen(lruvec, folio);
list_move(&folio->lru, &lrugen->folios[gen][type][zone]);
- /* don't count the workingset being lazily promoted */
- if (refs + workingset != BIT(LRU_REFS_WIDTH) + 1) {
- int hist = lru_hist_from_seq(lrugen->min_seq[type]);
+ WRITE_ONCE(lrugen->protected[hist][type][tier],
+ lrugen->protected[hist][type][tier] + delta);
- WRITE_ONCE(lrugen->protected[hist][type][tier],
- lrugen->protected[hist][type][tier] + delta);
- }
return true;
}
@@ -4876,10 +4986,6 @@ static bool isolate_folio(struct lruvec *lruvec, struct folio *folio, struct sca
return false;
}
- /* see the comment on LRU_REFS_FLAGS */
- if (!folio_test_referenced(folio))
- folio_set_lru_refs(folio, 0);
-
success = lru_gen_del_folio(lruvec, folio, true);
VM_WARN_ON_ONCE_FOLIO(!success, folio);
@@ -4967,13 +5073,14 @@ static int get_tier_idx(struct lruvec *lruvec, int type)
struct ctrl_pos sp, pv;
/*
- * To leave a margin for fluctuations, use a larger gain factor (2:3).
- * This value is chosen because any other tier would have at least twice
- * as many refaults as the first tier.
+ * To leave a margin for fluctuations, use a larger gain factor (1:2).
+ * Stop at the first tier whose refault rate is clearly worse than
+ * that of the cumulative mass of the tiers below it; the PID
+ * protects the tiers above it.
*/
- read_ctrl_pos(lruvec, type, LRU_TIER_MIN, LRU_TIER_MIN, 2, &sp);
for (tier = LRU_TIER_MIN + 1; tier <= LRU_TIER_MAX; tier++) {
- read_ctrl_pos(lruvec, type, tier, tier, 3, &pv);
+ read_ctrl_pos(lruvec, type, LRU_TIER_MIN, tier - 1, 1, &sp);
+ read_ctrl_pos(lruvec, type, tier, tier, 2, &pv);
if (!positive_ctrl_err(&sp, &pv))
break;
}
@@ -5115,12 +5222,13 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
}
/*
- * See the comments on LRU_REFS_FLAGS.
- *
* The rejected folios are never added to the oldest generation,
* so this effectively promotes them by at least one generation.
+ * PG_active is the only placement hint here, so a folio kept on
+ * its first page table access lands in the second newest one.
+ * See "Referenced count feedback" above.
*/
- folio_set_lru_refs(folio, 0);
+ folio_set_lru_refs(folio, min(folio_lru_refs(folio), LRU_REFS_WORKINGSET));
if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type])
folio_set_active(folio);
}
diff --git a/mm/workingset.c b/mm/workingset.c
index 1504f91cdca5..16cdd88e275e 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -188,6 +188,12 @@
#define EVICTION_SHIFT_ANON (EVICTION_SHIFT + SWAP_COUNT_SHIFT)
#define EVICTION_MASK (~0UL >> EVICTION_SHIFT)
#define EVICTION_MASK_ANON (~0UL >> EVICTION_SHIFT_ANON)
+/*
+ * LRU refs uses LRU_REFS_WIDTH + 2 bits, the 2 bits being PG_workingset
+ * and PG_referenced. The lowest bit is recorded in the workingset field
+ * of the shadow entry (to reuse pack_shadow()).
+ */
+#define LRU_REFS_BITS ((LRU_REFS_WIDTH + 2) - 1)
/*
* Eviction timestamps need to be able to cover the full range of
@@ -242,13 +248,12 @@ static void *lru_gen_eviction(struct folio *folio)
int type = folio_is_file_lru(folio);
int delta = folio_nr_pages(folio);
int refs = folio_lru_refs(folio);
- bool workingset = folio_test_workingset(folio);
- int tier = lru_tier_from_refs(refs, workingset);
+ int tier = lru_tier_from_refs(refs);
struct mem_cgroup *memcg;
struct pglist_data *pgdat = folio_pgdat(folio);
unsigned short memcg_id;
- BUILD_BUG_ON(LRU_GEN_WIDTH + LRU_REFS_WIDTH >
+ BUILD_BUG_ON(LRU_GEN_WIDTH + LRU_REFS_BITS >
BITS_PER_LONG - max(EVICTION_SHIFT, EVICTION_SHIFT_ANON));
rcu_read_lock();
@@ -256,14 +261,14 @@ static void *lru_gen_eviction(struct folio *folio)
lruvec = mem_cgroup_lruvec(memcg, pgdat);
lrugen = &lruvec->lrugen;
min_seq = READ_ONCE(lrugen->min_seq[type]);
- token = (min_seq << LRU_REFS_WIDTH) | max(refs - 1, 0);
+ token = (min_seq << LRU_REFS_BITS) | refs >> 1;
hist = lru_hist_from_seq(min_seq);
atomic_long_add(delta, &lrugen->evicted[hist][type][tier]);
memcg_id = mem_cgroup_private_id(memcg);
rcu_read_unlock();
- return pack_shadow(memcg_id, pgdat, token, workingset, type);
+ return pack_shadow(memcg_id, pgdat, token, refs & 1, type);
}
/*
@@ -284,9 +289,9 @@ static bool lru_gen_test_recent(void *shadow, struct lruvec **lruvec,
*lruvec = mem_cgroup_lruvec(memcg, pgdat);
max_seq = READ_ONCE((*lruvec)->lrugen.max_seq);
- max_seq &= (file ? EVICTION_MASK : EVICTION_MASK_ANON) >> LRU_REFS_WIDTH;
+ max_seq &= (file ? EVICTION_MASK : EVICTION_MASK_ANON) >> LRU_REFS_BITS;
- return abs_diff(max_seq, *token >> LRU_REFS_WIDTH) < MAX_NR_GENS;
+ return abs_diff(max_seq, *token >> LRU_REFS_BITS) < MAX_NR_GENS;
}
static void lru_gen_refault(struct folio *folio, void *shadow)
@@ -314,22 +319,26 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
lrugen = &lruvec->lrugen;
hist = lru_hist_from_seq(READ_ONCE(lrugen->min_seq[type]));
- refs = (token & (BIT(LRU_REFS_WIDTH) - 1)) + 1;
- tier = lru_tier_from_refs(refs, workingset);
+ refs = ((token & (BIT(LRU_REFS_BITS) - 1)) << 1) + workingset;
+ tier = lru_tier_from_refs(refs);
atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
- if (workingset) {
- /*
- * see folio_add_lru(), where folio_set_active() is
- * called for workingset folios
- */
- if (lru_gen_in_fault())
+ /*
+ * Activate a fault-driven refault folio, which would have been
+ * promoted had it stayed in memory.
+ */
+ if (refs >= LRU_REFS_REFERENCED) {
+ if (lru_gen_in_fault()) {
+ folio_set_active(folio);
mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
- folio_set_workingset(folio);
+ }
+ /* Refault is also promotion, cap the refs like folio_inc_lru_refs */
+ folio_set_lru_refs(folio, min(refs, LRU_REFS_PROTECTED));
+ }
+
+ if (refs >= LRU_REFS_WORKINGSET)
mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + type, delta);
- } else
- set_mask_bits(&folio->flags.f, LRU_REFS_MASK, (refs - 1UL) << LRU_REFS_PGOFF);
unlock:
rcu_read_unlock();
}
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 04/17] mm/mglru: frequency guided workingset promotion (MGLRU-FG)
2026-10-15 18:15 ` Kairui Song
(?)
@ 2026-10-15 18:41 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:41 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> Complement MGLRU's eviction-time tier-PID protection with access-time
> frequency-guided promotion. Introduce a unified set of helpers built based
> on referenced (access) count of a folio.
>
> Each access increments a folio's referenced count stored in folio flags
> (refs), refs still maps to a logarithmic tier just like before, but with
> more formal bit definitions, a few special thresholds are introduced:
> LRU_REFS_REFERENCED (1), LRU_REFS_WORKINGSET (2), LRU_REFS_PROTECTED (3),
> and LRU_REFS_MAX (7). When refs reaches a certain threshold, the folio is
> promoted proactively instead of waiting for the PID controller to kick in.
>
> Also simplify MGLRU's usage of PG_workingset and PG_referenced: they
> become the low two bits of the refs count, with the higher bits
> [ ... ]
> PG_workingset and PG_referenced are no longer independent flags under
> MGLRU. Adjusting existing raw folio_test_*() callers to the new
> semantics is left as follow-ups.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=4
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 05/17] mm/mglru: make folio lru referenced times count a generic API
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:15 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
Pure code shuffle, no behavior change.
Make the folio LRU refs helpers available regardless of CONFIG_LRU_GEN.
Without MGLRU the refs bits are never set, so they are inert. Move the
lru_gen_* helpers into the CONFIG_LRU_GEN section. This prepares for
unifying the API for checking folio referenced and workingset status.
folio_migrate_lru_refs() now migrates the complete refs count,
including PG_workingset (bit 1 of the encoding), bitwise identical to
the direct copy dropped from folio_migrate_flags().
lru_set_refs_flags() skips the nonexistent mask bits when
LRU_REFS_WIDTH collapses to 0, keeping the generic helpers inert for
!CONFIG_LRU_GEN builds.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 55 ++++++++++++++++++++++++-----------------------
mm/migrate.c | 2 --
2 files changed, 28 insertions(+), 29 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 74d1b7a04ca1..cfcbe4a9aa85 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -183,6 +183,12 @@ static inline int lru_get_gen_flags(unsigned long flags)
return gen;
}
+static inline int folio_lru_gen(const struct folio *folio)
+{
+ return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
+}
+#endif
+
/**
* lru_set_refs_flags - Set the LRU referenced count to folio flags.
* @flags: pointer to the folio flags
@@ -205,7 +211,8 @@ static inline void lru_set_refs_flags(unsigned long *flags, unsigned int refs)
*flags |= BIT(PG_referenced);
if (refs & BIT(1))
*flags |= BIT(PG_workingset);
- *flags |= ((unsigned long)refs >> 2) << LRU_REFS_PGOFF;
+ if (LRU_REFS_WIDTH)
+ *flags |= ((unsigned long)refs >> 2) << LRU_REFS_PGOFF;
}
/**
@@ -240,13 +247,31 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
}
+#ifdef CONFIG_LRU_GEN
void folio_inc_lru_refs(struct folio *folio, unsigned int flags);
+#else
+static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
+{
+ /* Should not be called with !CONFIG_LRU_GEN */
+ WARN_ON_ONCE(1);
+}
+#endif
-static inline int folio_lru_gen(const struct folio *folio)
+/**
+ * folio_migrate_lru_refs - copy the reference state to a new folio
+ * @new: the destination folio
+ * @old: the source folio
+ *
+ * Transfer the reference state to @new during migration: the MGLRU
+ * refs count, or PG_referenced and PG_workingset (bits 0-1 of the
+ * encoding) for the active/inactive LRU.
+ */
+static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
{
- return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
+ folio_set_lru_refs(new, folio_lru_refs(old));
}
+#ifdef CONFIG_LRU_GEN
static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
{
unsigned long min_seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
@@ -401,20 +426,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
return true;
}
-
-/**
- * folio_migrate_lru_refs - copy the reference state to a new folio
- * @new: the destination folio
- * @old: the source folio
- *
- * Transfer the reference state to @new during migration: the MGLRU
- * refs count, including PG_referenced, or just PG_referenced for the
- * active/inactive LRU.
- */
-static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
-{
- folio_set_lru_refs(new, folio_lru_refs(old));
-}
#else /* !CONFIG_LRU_GEN */
static inline bool lru_gen_enabled(void)
@@ -441,16 +452,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
{
return false;
}
-
-static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
-{
-}
-
-static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
-{
- if (folio_test_referenced(old))
- folio_set_referenced(new);
-}
#endif /* CONFIG_LRU_GEN */
static __always_inline
diff --git a/mm/migrate.c b/mm/migrate.c
index 7bdcdb57652f..5eefd8547f70 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -786,8 +786,6 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
folio_set_active(newfolio);
} else if (folio_test_clear_unevictable(folio))
folio_set_unevictable(newfolio);
- if (folio_test_workingset(folio))
- folio_set_workingset(newfolio);
if (folio_test_checked(folio))
folio_set_checked(newfolio);
/*
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 05/17] mm/mglru: make folio lru referenced times count a generic API
@ 2026-10-15 18:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
Pure code shuffle, no behavior change.
Make the folio LRU refs helpers available regardless of CONFIG_LRU_GEN.
Without MGLRU the refs bits are never set, so they are inert. Move the
lru_gen_* helpers into the CONFIG_LRU_GEN section. This prepares for
unifying the API for checking folio referenced and workingset status.
folio_migrate_lru_refs() now migrates the complete refs count,
including PG_workingset (bit 1 of the encoding), bitwise identical to
the direct copy dropped from folio_migrate_flags().
lru_set_refs_flags() skips the nonexistent mask bits when
LRU_REFS_WIDTH collapses to 0, keeping the generic helpers inert for
!CONFIG_LRU_GEN builds.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 55 ++++++++++++++++++++++++-----------------------
mm/migrate.c | 2 --
2 files changed, 28 insertions(+), 29 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 74d1b7a04ca1..cfcbe4a9aa85 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -183,6 +183,12 @@ static inline int lru_get_gen_flags(unsigned long flags)
return gen;
}
+static inline int folio_lru_gen(const struct folio *folio)
+{
+ return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
+}
+#endif
+
/**
* lru_set_refs_flags - Set the LRU referenced count to folio flags.
* @flags: pointer to the folio flags
@@ -205,7 +211,8 @@ static inline void lru_set_refs_flags(unsigned long *flags, unsigned int refs)
*flags |= BIT(PG_referenced);
if (refs & BIT(1))
*flags |= BIT(PG_workingset);
- *flags |= ((unsigned long)refs >> 2) << LRU_REFS_PGOFF;
+ if (LRU_REFS_WIDTH)
+ *flags |= ((unsigned long)refs >> 2) << LRU_REFS_PGOFF;
}
/**
@@ -240,13 +247,31 @@ static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
}
+#ifdef CONFIG_LRU_GEN
void folio_inc_lru_refs(struct folio *folio, unsigned int flags);
+#else
+static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
+{
+ /* Should not be called with !CONFIG_LRU_GEN */
+ WARN_ON_ONCE(1);
+}
+#endif
-static inline int folio_lru_gen(const struct folio *folio)
+/**
+ * folio_migrate_lru_refs - copy the reference state to a new folio
+ * @new: the destination folio
+ * @old: the source folio
+ *
+ * Transfer the reference state to @new during migration: the MGLRU
+ * refs count, or PG_referenced and PG_workingset (bits 0-1 of the
+ * encoding) for the active/inactive LRU.
+ */
+static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
{
- return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
+ folio_set_lru_refs(new, folio_lru_refs(old));
}
+#ifdef CONFIG_LRU_GEN
static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
{
unsigned long min_seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
@@ -401,20 +426,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
return true;
}
-
-/**
- * folio_migrate_lru_refs - copy the reference state to a new folio
- * @new: the destination folio
- * @old: the source folio
- *
- * Transfer the reference state to @new during migration: the MGLRU
- * refs count, including PG_referenced, or just PG_referenced for the
- * active/inactive LRU.
- */
-static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
-{
- folio_set_lru_refs(new, folio_lru_refs(old));
-}
#else /* !CONFIG_LRU_GEN */
static inline bool lru_gen_enabled(void)
@@ -441,16 +452,6 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
{
return false;
}
-
-static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
-{
-}
-
-static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
-{
- if (folio_test_referenced(old))
- folio_set_referenced(new);
-}
#endif /* CONFIG_LRU_GEN */
static __always_inline
diff --git a/mm/migrate.c b/mm/migrate.c
index 7bdcdb57652f..5eefd8547f70 100644
--- a/mm/migrate.c
+++ b/mm/migrate.c
@@ -786,8 +786,6 @@ void folio_migrate_flags(struct folio *newfolio, struct folio *folio)
folio_set_active(newfolio);
} else if (folio_test_clear_unevictable(folio))
folio_set_unevictable(newfolio);
- if (folio_test_workingset(folio))
- folio_set_workingset(newfolio);
if (folio_test_checked(folio))
folio_set_checked(newfolio);
/*
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 05/17] mm/mglru: make folio lru referenced times count a generic API
2026-10-15 18:15 ` Kairui Song
(?)
@ 2026-10-15 18:30 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:30 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> Pure code shuffle, no behavior change.
>
> Make the folio LRU refs helpers available regardless of CONFIG_LRU_GEN.
> Without MGLRU the refs bits are never set, so they are inert. Move the
> lru_gen_* helpers into the CONFIG_LRU_GEN section. This prepares for
> unifying the API for checking folio referenced and workingset status.
>
> folio_migrate_lru_refs() now migrates the complete refs count,
> including PG_workingset (bit 1 of the encoding), bitwise identical to
> the direct copy dropped from folio_migrate_flags().
>
> lru_set_refs_flags() skips the nonexistent mask bits when
> LRU_REFS_WIDTH collapses to 0, keeping the generic helpers inert for
> !CONFIG_LRU_GEN builds.
>
> Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=5
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 06/17] mm/mglru: move add/del LRU size accounting out of lru_gen_update_size()
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:15 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
Pure code shuffle, no behavior change. lru_gen_update_size() now only
updates the per-generation counters plus the promotion move; the
addition and deletion active/inactive accounting move to
lru_gen_add_folio() and lru_gen_del_folio(), which know the generation
and can apply the active-window test directly.
The unevictable case is untouched: unevictable folios never enter
lru_gen_add_folio() and keep being accounted by lruvec_add_folio() as
before.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index cfcbe4a9aa85..65415eeac802 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -313,21 +313,9 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
if (new_gen >= 0)
atomic_long_add(delta, &lrugen->nr_pages[new_gen][type][zone]);
- /* addition */
- if (old_gen < 0) {
- if (lru_gen_is_active(lruvec, new_gen))
- lru += LRU_ACTIVE;
- __update_lru_size(lruvec, lru, zone, delta);
+ /* return now if not a promotion */
+ if (old_gen < 0 || new_gen < 0)
return;
- }
-
- /* deletion */
- if (new_gen < 0) {
- if (lru_gen_is_active(lruvec, old_gen))
- lru += LRU_ACTIVE;
- __update_lru_size(lruvec, lru, zone, -delta);
- return;
- }
/* promotion */
if (!lru_gen_is_active(lruvec, old_gen) && lru_gen_is_active(lruvec, new_gen)) {
@@ -381,6 +369,8 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
int gen = folio_lru_gen(folio);
int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
+ int delta = folio_nr_pages(folio);
+ enum lru_list lru = type * LRU_INACTIVE_FILE;
struct lru_gen_folio *lrugen = &lruvec->lrugen;
BUILD_BUG_ON(BIT(LRU_GEN_WIDTH - 1) != MAX_NR_GENS);
@@ -396,6 +386,10 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
lru_gen_update_size(lruvec, folio, -1, gen);
+ if (lru_gen_is_active(lruvec, gen))
+ lru += LRU_ACTIVE;
+ __update_lru_size(lruvec, lru, zone, delta);
+
/* for folio_rotate_reclaimable() */
if (reclaiming)
list_add_tail(&folio->lru, &lrugen->folios[gen][type][zone]);
@@ -409,6 +403,9 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
{
unsigned long flags;
int gen = folio_lru_gen(folio);
+ int zone = folio_zonenum(folio);
+ int delta = folio_nr_pages(folio);
+ enum lru_list lru = folio_is_file_lru(folio) * LRU_INACTIVE_FILE;
if (gen < 0)
return false;
@@ -422,6 +419,9 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
lru_gen_update_size(lruvec, folio, gen, -1);
+ if (lru_gen_is_active(lruvec, gen))
+ lru += LRU_ACTIVE;
+ __update_lru_size(lruvec, lru, zone, -delta);
list_del(&folio->lru);
return true;
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 06/17] mm/mglru: move add/del LRU size accounting out of lru_gen_update_size()
@ 2026-10-15 18:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
Pure code shuffle, no behavior change. lru_gen_update_size() now only
updates the per-generation counters plus the promotion move; the
addition and deletion active/inactive accounting move to
lru_gen_add_folio() and lru_gen_del_folio(), which know the generation
and can apply the active-window test directly.
The unevictable case is untouched: unevictable folios never enter
lru_gen_add_folio() and keep being accounted by lruvec_add_folio() as
before.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 28 ++++++++++++++--------------
1 file changed, 14 insertions(+), 14 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index cfcbe4a9aa85..65415eeac802 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -313,21 +313,9 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
if (new_gen >= 0)
atomic_long_add(delta, &lrugen->nr_pages[new_gen][type][zone]);
- /* addition */
- if (old_gen < 0) {
- if (lru_gen_is_active(lruvec, new_gen))
- lru += LRU_ACTIVE;
- __update_lru_size(lruvec, lru, zone, delta);
+ /* return now if not a promotion */
+ if (old_gen < 0 || new_gen < 0)
return;
- }
-
- /* deletion */
- if (new_gen < 0) {
- if (lru_gen_is_active(lruvec, old_gen))
- lru += LRU_ACTIVE;
- __update_lru_size(lruvec, lru, zone, -delta);
- return;
- }
/* promotion */
if (!lru_gen_is_active(lruvec, old_gen) && lru_gen_is_active(lruvec, new_gen)) {
@@ -381,6 +369,8 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
int gen = folio_lru_gen(folio);
int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
+ int delta = folio_nr_pages(folio);
+ enum lru_list lru = type * LRU_INACTIVE_FILE;
struct lru_gen_folio *lrugen = &lruvec->lrugen;
BUILD_BUG_ON(BIT(LRU_GEN_WIDTH - 1) != MAX_NR_GENS);
@@ -396,6 +386,10 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
lru_gen_update_size(lruvec, folio, -1, gen);
+ if (lru_gen_is_active(lruvec, gen))
+ lru += LRU_ACTIVE;
+ __update_lru_size(lruvec, lru, zone, delta);
+
/* for folio_rotate_reclaimable() */
if (reclaiming)
list_add_tail(&folio->lru, &lrugen->folios[gen][type][zone]);
@@ -409,6 +403,9 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
{
unsigned long flags;
int gen = folio_lru_gen(folio);
+ int zone = folio_zonenum(folio);
+ int delta = folio_nr_pages(folio);
+ enum lru_list lru = folio_is_file_lru(folio) * LRU_INACTIVE_FILE;
if (gen < 0)
return false;
@@ -422,6 +419,9 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
lru_gen_update_size(lruvec, folio, gen, -1);
+ if (lru_gen_is_active(lruvec, gen))
+ lru += LRU_ACTIVE;
+ __update_lru_size(lruvec, lru, zone, -delta);
list_del(&folio->lru);
return true;
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 06/17] mm/mglru: move add/del LRU size accounting out of lru_gen_update_size()
2026-10-15 18:15 ` Kairui Song
(?)
@ 2026-10-15 18:28 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:28 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> Pure code shuffle, no behavior change. lru_gen_update_size() now only
> updates the per-generation counters plus the promotion move; the
> addition and deletion active/inactive accounting move to
> lru_gen_add_folio() and lru_gen_del_folio(), which know the generation
> and can apply the active-window test directly.
>
> The unevictable case is untouched: unevictable folios never enter
> lru_gen_add_folio() and keep being accounted by lruvec_add_folio() as
> before.
>
> Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=6
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 07/17] mm/mglru, gup: mark folios referenced via a fast helper
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:15 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
Introduce folio_inc_lru_refs_fast(): raise refs from 0 to
LRU_REFS_REFERENCED via cmpxchg, or just set PG_referenced for the
classical LRU as before. No counter update, promotion, and guarteens
no other complex operations in following commits.
Convert the gup fast-path sites. No behavior change for classical LRU.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 25 +++++++++++++++++++++++++
mm/gup.c | 6 +++---
2 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 65415eeac802..effa13c63632 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -454,6 +454,31 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
}
#endif /* CONFIG_LRU_GEN */
+/**
+ * folio_inc_lru_refs_fast - Bump folio refs count without promotion.
+ * @folio: the folio
+ *
+ * Raise refs from 0 to LRU_REFS_REFERENCED and leave hotter folios
+ * untouched. For the classical LRU, a plain PG_referenced set.
+ */
+static __always_inline void folio_inc_lru_refs_fast(struct folio *folio)
+{
+ unsigned long new_flags, old_flags;
+
+ if (!lru_gen_enabled()) {
+ folio_set_referenced(folio);
+ return;
+ }
+
+ old_flags = READ_ONCE(*folio_flags(folio, 0));
+ do {
+ if (lru_get_refs_flags(old_flags))
+ break;
+ new_flags = old_flags;
+ lru_set_refs_flags(&new_flags, LRU_REFS_REFERENCED);
+ } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+}
+
static __always_inline
void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio)
{
diff --git a/mm/gup.c b/mm/gup.c
index f166acf794e3..b3f563d41192 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2896,7 +2896,7 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
gup_put_folio(folio, 1, flags);
goto pte_unmap;
}
- folio_set_referenced(folio);
+ folio_inc_lru_refs_fast(folio);
pages[nr_pages] = page;
nr_pages++;
} while (ptep++, addr += PAGE_SIZE, addr != end);
@@ -2965,7 +2965,7 @@ static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
for (i = 0; i < nr_pages; i++)
*(pages++) = page++;
- folio_set_referenced(folio);
+ folio_inc_lru_refs_fast(folio);
return nr_pages;
}
@@ -3007,7 +3007,7 @@ static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
for (i = 0; i < nr_pages; i++)
*(pages++) = page++;
- folio_set_referenced(folio);
+ folio_inc_lru_refs_fast(folio);
return nr_pages;
}
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 07/17] mm/mglru, gup: mark folios referenced via a fast helper
@ 2026-10-15 18:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
Introduce folio_inc_lru_refs_fast(): raise refs from 0 to
LRU_REFS_REFERENCED via cmpxchg, or just set PG_referenced for the
classical LRU as before. No counter update, promotion, and guarteens
no other complex operations in following commits.
Convert the gup fast-path sites. No behavior change for classical LRU.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 25 +++++++++++++++++++++++++
mm/gup.c | 6 +++---
2 files changed, 28 insertions(+), 3 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 65415eeac802..effa13c63632 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -454,6 +454,31 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
}
#endif /* CONFIG_LRU_GEN */
+/**
+ * folio_inc_lru_refs_fast - Bump folio refs count without promotion.
+ * @folio: the folio
+ *
+ * Raise refs from 0 to LRU_REFS_REFERENCED and leave hotter folios
+ * untouched. For the classical LRU, a plain PG_referenced set.
+ */
+static __always_inline void folio_inc_lru_refs_fast(struct folio *folio)
+{
+ unsigned long new_flags, old_flags;
+
+ if (!lru_gen_enabled()) {
+ folio_set_referenced(folio);
+ return;
+ }
+
+ old_flags = READ_ONCE(*folio_flags(folio, 0));
+ do {
+ if (lru_get_refs_flags(old_flags))
+ break;
+ new_flags = old_flags;
+ lru_set_refs_flags(&new_flags, LRU_REFS_REFERENCED);
+ } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+}
+
static __always_inline
void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio)
{
diff --git a/mm/gup.c b/mm/gup.c
index f166acf794e3..b3f563d41192 100644
--- a/mm/gup.c
+++ b/mm/gup.c
@@ -2896,7 +2896,7 @@ static unsigned long gup_fast_pte_range(pmd_t pmd, pmd_t *pmdp,
gup_put_folio(folio, 1, flags);
goto pte_unmap;
}
- folio_set_referenced(folio);
+ folio_inc_lru_refs_fast(folio);
pages[nr_pages] = page;
nr_pages++;
} while (ptep++, addr += PAGE_SIZE, addr != end);
@@ -2965,7 +2965,7 @@ static unsigned long gup_fast_pmd_leaf(pmd_t orig, pmd_t *pmdp,
for (i = 0; i < nr_pages; i++)
*(pages++) = page++;
- folio_set_referenced(folio);
+ folio_inc_lru_refs_fast(folio);
return nr_pages;
}
@@ -3007,7 +3007,7 @@ static unsigned long gup_fast_pud_leaf(pud_t orig, pud_t *pudp,
for (i = 0; i < nr_pages; i++)
*(pages++) = page++;
- folio_set_referenced(folio);
+ folio_inc_lru_refs_fast(folio);
return nr_pages;
}
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 07/17] mm/mglru, gup: mark folios referenced via a fast helper
2026-10-15 18:15 ` Kairui Song
(?)
@ 2026-10-15 18:27 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:27 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> Introduce folio_inc_lru_refs_fast(): raise refs from 0 to
> LRU_REFS_REFERENCED via cmpxchg, or just set PG_referenced for the
> classical LRU as before. No counter update, promotion, and guarteens
> no other complex operations in following commits.
>
> Convert the gup fast-path sites. No behavior change for classical LRU.
>
> Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=7
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 08/17] mm/mglru, smaps: convert to LRU refs based operations for MGLRU
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:15 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
PG_referenced is only the lowest bit of the folio LRU refs count
(LRU_REFS_FLAGS) now, and MGLRU never kept it stable: promotion used to
clear it, and now it toggles as the count grows. But with the LRU
referenced count as a stable API now, we can use two helpers to provide
in-kernel users a stable and sane reading of the referenced status of
an LRU managed folio.
Add folio_check_lru_refs() and folio_clear_lru_refs(), which use the refs
count when MGLRU is enabled and PG_referenced otherwise, and switch the
smaps paths over to them. The lru_gen_enabled() test stays inside the
helpers, so its static key is not splattered over the few call sites.
Convert smaps as the first user of them. MGLRU now reports every folio
with a nonzero count, so the reading no longer flickers and clearing no
longer results in an ambiguous status.
The classical LRU case is unchanged.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
fs/proc/task_mmu.c | 6 +++---
include/linux/mm_inline.h | 31 +++++++++++++++++++++++++++++++
2 files changed, 34 insertions(+), 3 deletions(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 81c464139f8f..55f3a35fd3af 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -888,7 +888,7 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
mss->resident += size;
/* Accumulate the size in pages that have been accessed. */
- if (young || folio_test_young(folio) || folio_test_referenced(folio))
+ if (young || folio_test_young(folio) || folio_check_lru_refs(folio))
mss->referenced += size;
/*
@@ -1685,7 +1685,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,
/* Clear accessed and referenced bits. */
pmdp_test_and_clear_young(vma, addr, pmd);
folio_test_clear_young(folio);
- folio_clear_referenced(folio);
+ folio_clear_lru_refs(folio);
out:
spin_unlock(ptl);
return 0;
@@ -1714,7 +1714,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,
/* Clear accessed and referenced bits. */
ptep_test_and_clear_young(vma, addr, pte);
folio_test_clear_young(folio);
- folio_clear_referenced(folio);
+ folio_clear_lru_refs(folio);
}
pte_unmap_unlock(pte - 1, ptl);
cond_resched();
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index effa13c63632..6c9510002754 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -479,6 +479,37 @@ static __always_inline void folio_inc_lru_refs_fast(struct folio *folio)
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
}
+/**
+ * folio_check_lru_refs - Test if an LRU managed folio has been referenced.
+ * @folio: The folio to test.
+ *
+ * Returns true if the MGLRU refs count is nonzero, or PG_referenced is set,
+ * which is also correct for the non-MGLRU case. Kept as a separate helper,
+ * to avoid splattering the lru_gen_enabled() static key.
+ */
+static inline bool folio_check_lru_refs(const struct folio *folio)
+{
+ if (lru_gen_enabled())
+ return folio_lru_refs(folio) > 0;
+ return folio_test_referenced(folio);
+}
+
+/**
+ * folio_clear_lru_refs - Clear the referenced state of an LRU managed folio.
+ * @folio: The folio to clear.
+ *
+ * Reset the MGLRU refs count to zero, or clear PG_referenced, leaving
+ * PG_workingset alone for the non-MGLRU case. Kept as a separate helper
+ * just like folio_check_lru_refs().
+ */
+static inline void folio_clear_lru_refs(struct folio *folio)
+{
+ if (lru_gen_enabled())
+ folio_set_lru_refs(folio, 0);
+ else
+ folio_clear_referenced(folio);
+}
+
static __always_inline
void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio)
{
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 08/17] mm/mglru, smaps: convert to LRU refs based operations for MGLRU
@ 2026-10-15 18:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
PG_referenced is only the lowest bit of the folio LRU refs count
(LRU_REFS_FLAGS) now, and MGLRU never kept it stable: promotion used to
clear it, and now it toggles as the count grows. But with the LRU
referenced count as a stable API now, we can use two helpers to provide
in-kernel users a stable and sane reading of the referenced status of
an LRU managed folio.
Add folio_check_lru_refs() and folio_clear_lru_refs(), which use the refs
count when MGLRU is enabled and PG_referenced otherwise, and switch the
smaps paths over to them. The lru_gen_enabled() test stays inside the
helpers, so its static key is not splattered over the few call sites.
Convert smaps as the first user of them. MGLRU now reports every folio
with a nonzero count, so the reading no longer flickers and clearing no
longer results in an ambiguous status.
The classical LRU case is unchanged.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
fs/proc/task_mmu.c | 6 +++---
include/linux/mm_inline.h | 31 +++++++++++++++++++++++++++++++
2 files changed, 34 insertions(+), 3 deletions(-)
diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c
index 81c464139f8f..55f3a35fd3af 100644
--- a/fs/proc/task_mmu.c
+++ b/fs/proc/task_mmu.c
@@ -888,7 +888,7 @@ static void smaps_account(struct mem_size_stats *mss, struct page *page,
mss->resident += size;
/* Accumulate the size in pages that have been accessed. */
- if (young || folio_test_young(folio) || folio_test_referenced(folio))
+ if (young || folio_test_young(folio) || folio_check_lru_refs(folio))
mss->referenced += size;
/*
@@ -1685,7 +1685,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,
/* Clear accessed and referenced bits. */
pmdp_test_and_clear_young(vma, addr, pmd);
folio_test_clear_young(folio);
- folio_clear_referenced(folio);
+ folio_clear_lru_refs(folio);
out:
spin_unlock(ptl);
return 0;
@@ -1714,7 +1714,7 @@ static int clear_refs_pte_range(pmd_t *pmd, unsigned long addr,
/* Clear accessed and referenced bits. */
ptep_test_and_clear_young(vma, addr, pte);
folio_test_clear_young(folio);
- folio_clear_referenced(folio);
+ folio_clear_lru_refs(folio);
}
pte_unmap_unlock(pte - 1, ptl);
cond_resched();
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index effa13c63632..6c9510002754 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -479,6 +479,37 @@ static __always_inline void folio_inc_lru_refs_fast(struct folio *folio)
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
}
+/**
+ * folio_check_lru_refs - Test if an LRU managed folio has been referenced.
+ * @folio: The folio to test.
+ *
+ * Returns true if the MGLRU refs count is nonzero, or PG_referenced is set,
+ * which is also correct for the non-MGLRU case. Kept as a separate helper,
+ * to avoid splattering the lru_gen_enabled() static key.
+ */
+static inline bool folio_check_lru_refs(const struct folio *folio)
+{
+ if (lru_gen_enabled())
+ return folio_lru_refs(folio) > 0;
+ return folio_test_referenced(folio);
+}
+
+/**
+ * folio_clear_lru_refs - Clear the referenced state of an LRU managed folio.
+ * @folio: The folio to clear.
+ *
+ * Reset the MGLRU refs count to zero, or clear PG_referenced, leaving
+ * PG_workingset alone for the non-MGLRU case. Kept as a separate helper
+ * just like folio_check_lru_refs().
+ */
+static inline void folio_clear_lru_refs(struct folio *folio)
+{
+ if (lru_gen_enabled())
+ folio_set_lru_refs(folio, 0);
+ else
+ folio_clear_referenced(folio);
+}
+
static __always_inline
void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio)
{
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 08/17] mm/mglru, smaps: convert to LRU refs based operations for MGLRU
2026-10-15 18:15 ` Kairui Song
(?)
@ 2026-10-15 18:29 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:29 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> PG_referenced is only the lowest bit of the folio LRU refs count
> (LRU_REFS_FLAGS) now, and MGLRU never kept it stable: promotion used to
> clear it, and now it toggles as the count grows. But with the LRU
> referenced count as a stable API now, we can use two helpers to provide
> in-kernel users a stable and sane reading of the referenced status of
> an LRU managed folio.
>
> Add folio_check_lru_refs() and folio_clear_lru_refs(), which use the refs
> count when MGLRU is enabled and PG_referenced otherwise, and switch the
> smaps paths over to them. The lru_gen_enabled() test stays inside the
> helpers, so its static key is not splattered over the few call sites.
>
> Convert smaps as the first user of them. MGLRU now reports every folio
> with a nonzero count, so the reading no longer flickers and clearing no
> longer results in an ambiguous status.
>
> The classical LRU case is unchanged.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=8
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 09/17] mm/damon: convert to LRU refs based operations for MGLRU
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:15 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
damon_pa_pageout() cleared PG_referenced directly, which under MGLRU
may corrupts the folio's refs count in later commits. That flag is now
bit 0 of the count (LRU_REFS_FLAGS).
Reset the count instead under MGLRU.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/damon/paddr.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index 2cfdc356b415..6412ff2396b3 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -279,7 +279,8 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
else
*sz_filter_passed += folio_size(folio) / addr_unit;
- folio_clear_referenced(folio);
+ /* The regions is cold, better drop the hotness info */
+ folio_clear_lru_refs(folio);
folio_test_clear_young(folio);
if (!folio_isolate_lru(folio))
goto put_folio;
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [PATCH v4 09/17] mm/damon: convert to LRU refs based operations for MGLRU
@ 2026-10-15 18:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
damon_pa_pageout() cleared PG_referenced directly, which under MGLRU
may corrupts the folio's refs count in later commits. That flag is now
bit 0 of the count (LRU_REFS_FLAGS).
Reset the count instead under MGLRU.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/damon/paddr.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/damon/paddr.c b/mm/damon/paddr.c
index 2cfdc356b415..6412ff2396b3 100644
--- a/mm/damon/paddr.c
+++ b/mm/damon/paddr.c
@@ -279,7 +279,8 @@ static unsigned long damon_pa_pageout(struct damon_region *r,
else
*sz_filter_passed += folio_size(folio) / addr_unit;
- folio_clear_referenced(folio);
+ /* The regions is cold, better drop the hotness info */
+ folio_clear_lru_refs(folio);
folio_test_clear_young(folio);
if (!folio_isolate_lru(folio))
goto put_folio;
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread
* Re: [PATCH v4 09/17] mm/damon: convert to LRU refs based operations for MGLRU
2026-10-15 18:15 ` Kairui Song
(?)
@ 2026-10-15 18:27 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:27 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> damon_pa_pageout() cleared PG_referenced directly, which under MGLRU
> may corrupts the folio's refs count in later commits. That flag is now
> bit 0 of the count (LRU_REFS_FLAGS).
>
> Reset the count instead under MGLRU.
>
> Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=9
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 10/17] mm/khugepaged: convert to LRU refs based operations for MGLRU
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:15 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
For MGLRU, PG_referenced is only the lowest bit of the folio LRU refs
count, so folios with a higher refs count no longer have the bit set.
folio_pte_referenced() testing the raw bit misses hot folios at
refs >= 2, under-counting the referenced folios of a candidate range
and aborting otherwise good collapses with SCAN_LACK_REFERENCED_PAGE.
Test the refs count directly when MGLRU is enabled, mirroring the
smaps conversion. The classical LRU keeps the plain bit test.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/khugepaged.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index ef5cd6fda34f..5332b4c51fec 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -579,7 +579,9 @@ static bool folio_pte_referenced(struct folio *folio,
struct vm_area_struct *vma, unsigned long addr, pte_t pteval)
{
/* The folio was referenced previously ... */
- if (folio_test_young(folio) || folio_test_referenced(folio))
+ if (folio_test_young(folio))
+ return true;
+ if (folio_check_lru_refs(folio))
return true;
/* ... or the PTE mapping was recently used */
return pte_young(pteval) || mmu_notifier_test_young(vma->vm_mm, addr);
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 10/17] mm/khugepaged: convert to LRU refs based operations for MGLRU
@ 2026-10-15 18:15 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:15 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
For MGLRU, PG_referenced is only the lowest bit of the folio LRU refs
count, so folios with a higher refs count no longer have the bit set.
folio_pte_referenced() testing the raw bit misses hot folios at
refs >= 2, under-counting the referenced folios of a candidate range
and aborting otherwise good collapses with SCAN_LACK_REFERENCED_PAGE.
Test the refs count directly when MGLRU is enabled, mirroring the
smaps conversion. The classical LRU keeps the plain bit test.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/khugepaged.c | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/mm/khugepaged.c b/mm/khugepaged.c
index ef5cd6fda34f..5332b4c51fec 100644
--- a/mm/khugepaged.c
+++ b/mm/khugepaged.c
@@ -579,7 +579,9 @@ static bool folio_pte_referenced(struct folio *folio,
struct vm_area_struct *vma, unsigned long addr, pte_t pteval)
{
/* The folio was referenced previously ... */
- if (folio_test_young(folio) || folio_test_referenced(folio))
+ if (folio_test_young(folio))
+ return true;
+ if (folio_check_lru_refs(folio))
return true;
/* ... or the PTE mapping was recently used */
return pte_young(pteval) || mmu_notifier_test_young(vma->vm_mm, addr);
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 10/17] mm/khugepaged: convert to LRU refs based operations for MGLRU
2026-10-15 18:15 ` Kairui Song
(?)
@ 2026-10-15 18:29 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:29 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> For MGLRU, PG_referenced is only the lowest bit of the folio LRU refs
> count, so folios with a higher refs count no longer have the bit set.
> folio_pte_referenced() testing the raw bit misses hot folios at
> refs >= 2, under-counting the referenced folios of a candidate range
> and aborting otherwise good collapses with SCAN_LACK_REFERENCED_PAGE.
>
> Test the refs count directly when MGLRU is enabled, mirroring the
> smaps conversion. The classical LRU keeps the plain bit test.
>
> Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=10
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 11/17] mm/madvise: adapt for LRU refs based operations for MGLRU
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:16 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
For the active/inactive LRU, madvise clears PG_referenced so that one more
access is not enough to reactivate a folio, and keeps PG_workingset on a
folio demoted out of the active list so its refault is still accounted as
a workingset refault (PSI).
MGLRU keeps that history in the folio's refs count instead, which now
lives in PG_referenced, PG_workingset and LRU_REFS_MASK, so writing to
those bits directly corrupts the count. The two hints want different
things:
- MADV_COLD resets the count in folio_deactivate(), which also moves the
folio to the oldest generation.
- MADV_PAGEOUT caps the count at LRU_REFS_WORKINGSET before handing the
folio to reclaim_pages(). The hint marks the folio as cold, so the
eviction shadow records the capped count and does not inflate the
eviction/refault counters of the hot tiers that drive aging, while
folios at or above the workingset threshold keep their PSI
classification.
PG_young is cleared for both LRUs as before, so the referenced walk does
not count stale history as a recent access.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/madvise.c | 56 ++++++++++++++++++++++++++++++++++++++++++--------------
1 file changed, 42 insertions(+), 14 deletions(-)
diff --git a/mm/madvise.c b/mm/madvise.c
index 75b409e97dbf..2b4a8b2cb477 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -360,6 +360,44 @@ static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end,
FPB_MERGE_YOUNG_DIRTY);
}
+static void madvise_cold_prep_folio(struct folio *folio)
+{
+ /*
+ * VM couldn't reclaim the folio unless we clear PG_young.
+ * As a side effect, it makes confuse idle-page tracking
+ * because they will miss recent referenced history.
+ */
+ folio_test_clear_young(folio);
+
+ /*
+ * For the active/inactive LRU, a folio demoted out of the active
+ * list should have PG_workingset so its refault is still accounted
+ * as a workingset refault, and the referenced bit always needs to
+ * be cleared even for inactive folios. Nothing to do for MGLRU as
+ * folio_deactivate() always calls lru_gen_clear_refs().
+ */
+ if (!lru_gen_enabled()) {
+ folio_clear_referenced(folio);
+ if (folio_test_active(folio))
+ folio_set_workingset(folio);
+ }
+}
+
+static void madvise_pageout_prep_folio(struct folio *folio)
+{
+ int refs = folio_lru_refs(folio);
+
+ /*
+ * For MGLRU, drop the count below LRU_REFS_WORKINGSET and cap
+ * the rest at it: MADV is
+ * suggesting the folio is actually cold, so the eviction shadow
+ * should not record its hotness in the high tier counters, but
+ * the workingset classification is preserved for PSI.
+ */
+ if (lru_gen_enabled())
+ folio_set_lru_refs(folio, refs >= LRU_REFS_WORKINGSET ? LRU_REFS_WORKINGSET : 0);
+}
+
static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
unsigned long addr, unsigned long end,
struct mm_walk *walk)
@@ -434,12 +472,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
}
- folio_clear_referenced(folio);
- folio_test_clear_young(folio);
- if (folio_test_active(folio))
- folio_set_workingset(folio);
+ madvise_cold_prep_folio(folio);
if (pageout) {
if (folio_isolate_lru(folio)) {
+ madvise_pageout_prep_folio(folio);
if (folio_test_unevictable(folio))
folio_putback_lru(folio);
else
@@ -544,18 +580,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
tlb_remove_tlb_entries(tlb, pte, nr, addr);
}
- /*
- * We are deactivating a folio for accelerating reclaiming.
- * VM couldn't reclaim the folio unless we clear PG_young.
- * As a side effect, it makes confuse idle-page tracking
- * because they will miss recent referenced history.
- */
- folio_clear_referenced(folio);
- folio_test_clear_young(folio);
- if (folio_test_active(folio))
- folio_set_workingset(folio);
+ madvise_cold_prep_folio(folio);
if (pageout) {
if (folio_isolate_lru(folio)) {
+ madvise_pageout_prep_folio(folio);
if (folio_test_unevictable(folio))
folio_putback_lru(folio);
else
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 11/17] mm/madvise: adapt for LRU refs based operations for MGLRU
@ 2026-10-15 18:16 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
For the active/inactive LRU, madvise clears PG_referenced so that one more
access is not enough to reactivate a folio, and keeps PG_workingset on a
folio demoted out of the active list so its refault is still accounted as
a workingset refault (PSI).
MGLRU keeps that history in the folio's refs count instead, which now
lives in PG_referenced, PG_workingset and LRU_REFS_MASK, so writing to
those bits directly corrupts the count. The two hints want different
things:
- MADV_COLD resets the count in folio_deactivate(), which also moves the
folio to the oldest generation.
- MADV_PAGEOUT caps the count at LRU_REFS_WORKINGSET before handing the
folio to reclaim_pages(). The hint marks the folio as cold, so the
eviction shadow records the capped count and does not inflate the
eviction/refault counters of the hot tiers that drive aging, while
folios at or above the workingset threshold keep their PSI
classification.
PG_young is cleared for both LRUs as before, so the referenced walk does
not count stale history as a recent access.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/madvise.c | 56 ++++++++++++++++++++++++++++++++++++++++++--------------
1 file changed, 42 insertions(+), 14 deletions(-)
diff --git a/mm/madvise.c b/mm/madvise.c
index 75b409e97dbf..2b4a8b2cb477 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -360,6 +360,44 @@ static inline int madvise_folio_pte_batch(unsigned long addr, unsigned long end,
FPB_MERGE_YOUNG_DIRTY);
}
+static void madvise_cold_prep_folio(struct folio *folio)
+{
+ /*
+ * VM couldn't reclaim the folio unless we clear PG_young.
+ * As a side effect, it makes confuse idle-page tracking
+ * because they will miss recent referenced history.
+ */
+ folio_test_clear_young(folio);
+
+ /*
+ * For the active/inactive LRU, a folio demoted out of the active
+ * list should have PG_workingset so its refault is still accounted
+ * as a workingset refault, and the referenced bit always needs to
+ * be cleared even for inactive folios. Nothing to do for MGLRU as
+ * folio_deactivate() always calls lru_gen_clear_refs().
+ */
+ if (!lru_gen_enabled()) {
+ folio_clear_referenced(folio);
+ if (folio_test_active(folio))
+ folio_set_workingset(folio);
+ }
+}
+
+static void madvise_pageout_prep_folio(struct folio *folio)
+{
+ int refs = folio_lru_refs(folio);
+
+ /*
+ * For MGLRU, drop the count below LRU_REFS_WORKINGSET and cap
+ * the rest at it: MADV is
+ * suggesting the folio is actually cold, so the eviction shadow
+ * should not record its hotness in the high tier counters, but
+ * the workingset classification is preserved for PSI.
+ */
+ if (lru_gen_enabled())
+ folio_set_lru_refs(folio, refs >= LRU_REFS_WORKINGSET ? LRU_REFS_WORKINGSET : 0);
+}
+
static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
unsigned long addr, unsigned long end,
struct mm_walk *walk)
@@ -434,12 +472,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
tlb_remove_pmd_tlb_entry(tlb, pmd, addr);
}
- folio_clear_referenced(folio);
- folio_test_clear_young(folio);
- if (folio_test_active(folio))
- folio_set_workingset(folio);
+ madvise_cold_prep_folio(folio);
if (pageout) {
if (folio_isolate_lru(folio)) {
+ madvise_pageout_prep_folio(folio);
if (folio_test_unevictable(folio))
folio_putback_lru(folio);
else
@@ -544,18 +580,10 @@ static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
tlb_remove_tlb_entries(tlb, pte, nr, addr);
}
- /*
- * We are deactivating a folio for accelerating reclaiming.
- * VM couldn't reclaim the folio unless we clear PG_young.
- * As a side effect, it makes confuse idle-page tracking
- * because they will miss recent referenced history.
- */
- folio_clear_referenced(folio);
- folio_test_clear_young(folio);
- if (folio_test_active(folio))
- folio_set_workingset(folio);
+ madvise_cold_prep_folio(folio);
if (pageout) {
if (folio_isolate_lru(folio)) {
+ madvise_pageout_prep_folio(folio);
if (folio_test_unevictable(folio))
folio_putback_lru(folio);
else
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 11/17] mm/madvise: adapt for LRU refs based operations for MGLRU
2026-10-15 18:16 ` Kairui Song
(?)
@ 2026-10-15 18:25 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:25 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> For the active/inactive LRU, madvise clears PG_referenced so that one more
> access is not enough to reactivate a folio, and keeps PG_workingset on a
> folio demoted out of the active list so its refault is still accounted as
> a workingset refault (PSI).
>
> MGLRU keeps that history in the folio's refs count instead, which now
> lives in PG_referenced, PG_workingset and LRU_REFS_MASK, so writing to
> those bits directly corrupts the count. The two hints want different
> things:
>
> - MADV_COLD resets the count in folio_deactivate(), which also moves the
> folio to the oldest generation.
> - MADV_PAGEOUT caps the count at LRU_REFS_WORKINGSET before handing the
> folio to reclaim_pages(). The hint marks the folio as cold, so the
> eviction shadow records the capped count and does not inflate the
> eviction/refault counters of the hot tiers that drive aging, while
> folios at or above the workingset threshold keep their PSI
> classification.
>
> PG_young is cleared for both LRUs as before, so the referenced walk does
> not count stale history as a recent access.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=11
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 12/17] mm/huge_memory: mark folio as accessed more accurately on split
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:16 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
The behavior of updating the folio's access info isn't consistent for
huge mapping splitting or ordinary unmapping. The page table's young
flag has to be translated into folio's access info.
Right now it only check and set folio's referenced flag, which isn't
enough since folio flags update on access have its rules. Ordinary
unmapping (zapping) calls folio_mark_accessed(), and it also checks
if the VMA has recency to avoid false updates.
So first just use the right helper here to be more consistent.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index bd44d58e8684..1e2b6448c010 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3093,8 +3093,8 @@ static void __split_huge_pud_locked(struct vm_area_struct *vma, pud_t *pud,
if (!folio_test_dirty(folio) && pud_dirty(old_pud))
folio_mark_dirty(folio);
- if (!folio_test_referenced(folio) && pud_young(old_pud))
- folio_set_referenced(folio);
+ if (pud_young(old_pud) && vma_has_recency(vma))
+ folio_mark_accessed(folio);
folio_remove_rmap_pud(folio, page, vma);
add_mm_counter(vma->vm_mm, mm_counter_file(folio),
-HPAGE_PUD_NR);
@@ -3216,8 +3216,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
folio = page_folio(page);
if (!folio_test_dirty(folio) && pmd_dirty(old_pmd))
folio_mark_dirty(folio);
- if (!folio_test_referenced(folio) && pmd_young(old_pmd))
- folio_set_referenced(folio);
+ if (pmd_young(old_pmd) && vma_has_recency(vma))
+ folio_mark_accessed(folio);
folio_remove_rmap_pmd(folio, page, vma);
add_mm_counter(mm, mm_counter_file(folio), -HPAGE_PMD_NR);
folio_put(folio);
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [PATCH v4 12/17] mm/huge_memory: mark folio as accessed more accurately on split
@ 2026-10-15 18:16 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
The behavior of updating the folio's access info isn't consistent for
huge mapping splitting or ordinary unmapping. The page table's young
flag has to be translated into folio's access info.
Right now it only check and set folio's referenced flag, which isn't
enough since folio flags update on access have its rules. Ordinary
unmapping (zapping) calls folio_mark_accessed(), and it also checks
if the VMA has recency to avoid false updates.
So first just use the right helper here to be more consistent.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/huge_memory.c | 8 ++++----
1 file changed, 4 insertions(+), 4 deletions(-)
diff --git a/mm/huge_memory.c b/mm/huge_memory.c
index bd44d58e8684..1e2b6448c010 100644
--- a/mm/huge_memory.c
+++ b/mm/huge_memory.c
@@ -3093,8 +3093,8 @@ static void __split_huge_pud_locked(struct vm_area_struct *vma, pud_t *pud,
if (!folio_test_dirty(folio) && pud_dirty(old_pud))
folio_mark_dirty(folio);
- if (!folio_test_referenced(folio) && pud_young(old_pud))
- folio_set_referenced(folio);
+ if (pud_young(old_pud) && vma_has_recency(vma))
+ folio_mark_accessed(folio);
folio_remove_rmap_pud(folio, page, vma);
add_mm_counter(vma->vm_mm, mm_counter_file(folio),
-HPAGE_PUD_NR);
@@ -3216,8 +3216,8 @@ static void __split_huge_pmd_locked(struct vm_area_struct *vma, pmd_t *pmd,
folio = page_folio(page);
if (!folio_test_dirty(folio) && pmd_dirty(old_pmd))
folio_mark_dirty(folio);
- if (!folio_test_referenced(folio) && pmd_young(old_pmd))
- folio_set_referenced(folio);
+ if (pmd_young(old_pmd) && vma_has_recency(vma))
+ folio_mark_accessed(folio);
folio_remove_rmap_pmd(folio, page, vma);
add_mm_counter(mm, mm_counter_file(folio), -HPAGE_PMD_NR);
folio_put(folio);
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread
* Re: [PATCH v4 12/17] mm/huge_memory: mark folio as accessed more accurately on split
2026-10-15 18:16 ` Kairui Song
(?)
@ 2026-10-15 18:36 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:36 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> The behavior of updating the folio's access info isn't consistent for
> huge mapping splitting or ordinary unmapping. The page table's young
> flag has to be translated into folio's access info.
>
> Right now it only check and set folio's referenced flag, which isn't
> enough since folio flags update on access have its rules. Ordinary
> unmapping (zapping) calls folio_mark_accessed(), and it also checks
> if the VMA has recency to avoid false updates.
>
> So first just use the right helper here to be more consistent.
>
> Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=12
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 13/17] mm/mglru: folio LRU refs based active/inactive number accounting
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:16 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
Currently the active/inactive LRU sizes exported through /proc/vmstat
and memory.stat are derived from the generation window: folios in the
two newest generations are accounted as active, the rest as inactive.
This has always caused many problems:
- Unmapped file folios stick to the oldest gens, so active files are
always under-reported, unlike classical LRU.
- The reading is jumpy: on aging, the second-newest gen, previously
active, becomes inactive all at once, causing a huge drop of active
folios for no reason from time to time.
- The reading is reversed on swapless machines, and the active anon
number is an oscillating random number. When swappiness is 0 (no
swap or swap is full), MGLRU ages anon by moving the gens directly
without walk or touching any folio, to reduce the overhead. So all
inactive anon folios will suddenly count as active, or vice versa,
which breaks many metric reading components, and the reading itself
doesn't make any sense in any way.
With frequency guided promotion, hotness is already tracked per folio by
its referenced count, so we can do folio level accounting instead: a
folio is active once its refs reach LRU_REFS_ACTIVATED (currently
LRU_REFS_PROTECTED).
On access, folios are accounted to the active part. On forced aging or
PID protecting, folios are reset to LRU_REFS_WORKINGSET, dropping them
to the inactive part. The reading is now smooth and consistent, and
much closer to classical LRU. Note that on swapless machines anon refs
are never reset by aging (the shortcut skips the per-folio walk), so
active anon grows monotonically, but the reading is stable and still
much more meaningful than the previous oscillating random number.
The accounting is now per folio and lruvec independent, so no size
adjustment is needed when reparenting a dying memcg: the hierarchical
stats keep the counts in the parent automatically.
folio_migrate_lru_refs() also gains off-LRU VM_WARN_ON_ONCE checks and
__folio_set_lru_refs() warns when the folio is still on an LRU list.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
Documentation/admin-guide/mm/multigen_lru.rst | 4 +-
include/linux/mm_inline.h | 115 +++++++-------
include/linux/mmzone.h | 23 ++-
mm/folio.c | 35 +----
mm/madvise.c | 5 +-
mm/vmscan.c | 211 ++++++++++++++++----------
mm/workingset.c | 2 +-
7 files changed, 220 insertions(+), 175 deletions(-)
diff --git a/Documentation/admin-guide/mm/multigen_lru.rst b/Documentation/admin-guide/mm/multigen_lru.rst
index 9cb54b4ff5d9..808d2e97b3f3 100644
--- a/Documentation/admin-guide/mm/multigen_lru.rst
+++ b/Documentation/admin-guide/mm/multigen_lru.rst
@@ -149,8 +149,8 @@ generations less than or equal to ``min_gen_nr``.
``- memcg_id node_id min_gen_nr [swappiness [nr_to_reclaim]]``
``min_gen_nr`` should be less than ``max_gen_nr-1``, since
-``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged (equivalent to
-the active list) and therefore cannot be evicted. ``swappiness``
+``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged and therefore
+cannot be evicted. ``swappiness``
overrides the default value in ``/proc/sys/vm/swappiness`` and the valid
range is [0-200, max], with max being exclusively used for the reclamation
of anonymous memory. ``nr_to_reclaim`` limits the number of pages to evict.
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 6c9510002754..41c2e9150c70 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -105,6 +105,22 @@ static __always_inline enum lru_list folio_lru_list(const struct folio *folio)
return lru;
}
+/**
+ * lru_get_gen_flags - Return the LRU generation number from folio flags.
+ * @flags: folio flags
+ *
+ * Returns: A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns
+ * -1 if the flags indicate the folio is off the list (e.g., isolated).
+ */
+static inline int lru_get_gen_flags(unsigned long flags)
+{
+ int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+
+ /* Exclude the legal -1 from the unsigned MAX_NR_GENS comparison */
+ VM_WARN_ON_ONCE(gen != -1 && gen >= MAX_NR_GENS);
+ return gen;
+}
+
#ifdef CONFIG_LRU_GEN
static inline bool lru_gen_switching(void)
@@ -167,22 +183,6 @@ static inline void lru_set_gen_flags(unsigned long *flags, int gen)
*flags |= (gen + 1UL) << LRU_GEN_PGOFF;
}
-/**
- * lru_get_gen_flags - Return the LRU generation number from folio flags.
- * @flags: folio flags
- *
- * Returns: A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns
- * -1 if the flags indicate the folio is off the list (e.g., isolated).
- */
-static inline int lru_get_gen_flags(unsigned long flags)
-{
- int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
-
- /* Exclude the legal -1 from the unsigned MAX_NR_GENS comparison */
- VM_WARN_ON_ONCE(gen != -1 && gen >= MAX_NR_GENS);
- return gen;
-}
-
static inline int folio_lru_gen(const struct folio *folio)
{
return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
@@ -237,24 +237,39 @@ static inline int folio_lru_refs(const struct folio *folio)
return lru_get_refs_flags(READ_ONCE(*const_folio_flags(folio, 0)));
}
-static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
+/**
+ * __folio_set_lru_refs - Set a folio's LRU refs.
+ * @folio: the folio
+ * @refs: the new referenced count (0 .. LRU_REFS_MAX)
+ *
+ * Set the folio's LRU refs. The folio must be off the LRU list (e.g.,
+ * isolated), or use folio_inc_lru_refs or folio_reset_lru_refs instead.
+ */
+static inline void __folio_set_lru_refs(struct folio *folio, unsigned int refs)
{
unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
do {
new_flags = old_flags;
+ VM_WARN_ON_ONCE(lru_get_gen_flags(old_flags) != -1);
lru_set_refs_flags(&new_flags, refs);
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
}
#ifdef CONFIG_LRU_GEN
void folio_inc_lru_refs(struct folio *folio, unsigned int flags);
+bool folio_reset_lru_refs(struct folio *folio);
#else
static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
{
/* Should not be called with !CONFIG_LRU_GEN */
WARN_ON_ONCE(1);
}
+
+static inline bool folio_reset_lru_refs(struct folio *folio)
+{
+ return false;
+}
#endif
/**
@@ -268,10 +283,18 @@ static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
*/
static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
{
- folio_set_lru_refs(new, folio_lru_refs(old));
+ VM_WARN_ON_ONCE_FOLIO(folio_test_lru(old), old);
+ VM_WARN_ON_ONCE_FOLIO(folio_test_lru(new), new);
+ __folio_set_lru_refs(new, folio_lru_refs(old));
}
#ifdef CONFIG_LRU_GEN
+static inline bool lru_refs_is_active(unsigned int refs)
+{
+ VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
+ return refs >= LRU_REFS_ACTIVATED;
+}
+
static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
{
unsigned long min_seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
@@ -285,23 +308,12 @@ static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
gen == lru_gen_from_seq(min_seq + 1);
}
-static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
-{
- unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
-
- VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
-
- /* see the comment on MIN_NR_GENS */
- return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1);
-}
-
static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *folio,
int old_gen, int new_gen)
{
int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
- enum lru_list lru = type * LRU_INACTIVE_FILE;
struct lru_gen_folio *lrugen = &lruvec->lrugen;
VM_WARN_ON_ONCE(old_gen != -1 && old_gen >= MAX_NR_GENS);
@@ -312,19 +324,6 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
atomic_long_sub(delta, &lrugen->nr_pages[old_gen][type][zone]);
if (new_gen >= 0)
atomic_long_add(delta, &lrugen->nr_pages[new_gen][type][zone]);
-
- /* return now if not a promotion */
- if (old_gen < 0 || new_gen < 0)
- return;
-
- /* promotion */
- if (!lru_gen_is_active(lruvec, old_gen) && lru_gen_is_active(lruvec, new_gen)) {
- __update_lru_size(lruvec, lru, zone, -delta);
- __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, delta);
- }
-
- /* demotion requires isolation, e.g., lru_deactivate_fn() */
- VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen));
}
static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
@@ -366,7 +365,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
{
unsigned long seq;
unsigned long flags;
- int gen = folio_lru_gen(folio);
+ int gen, refs;
int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
@@ -374,7 +373,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
struct lru_gen_folio *lrugen = &lruvec->lrugen;
BUILD_BUG_ON(BIT(LRU_GEN_WIDTH - 1) != MAX_NR_GENS);
- VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
+ VM_WARN_ON_ONCE_FOLIO(folio_lru_gen(folio) != -1, folio);
if (folio_test_unevictable(folio) || !lrugen->enabled)
return false;
@@ -383,10 +382,12 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
gen = lru_gen_from_seq(seq);
flags = (gen + 1UL) << LRU_GEN_PGOFF;
/* see the comment on MIN_NR_GENS about PG_active */
- set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
+ flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
+ /* use the refs from the atomic snapshot to avoid raced update */
+ refs = lru_get_refs_flags(flags);
lru_gen_update_size(lruvec, folio, -1, gen);
- if (lru_gen_is_active(lruvec, gen))
+ if (lru_refs_is_active(refs))
lru += LRU_ACTIVE;
__update_lru_size(lruvec, lru, zone, delta);
@@ -402,24 +403,31 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming)
{
unsigned long flags;
- int gen = folio_lru_gen(folio);
+ int gen, refs;
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
enum lru_list lru = folio_is_file_lru(folio) * LRU_INACTIVE_FILE;
+ unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
+ flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, 0);
+ gen = lru_get_gen_flags(flags);
+ refs = lru_get_refs_flags(flags);
if (gen < 0)
return false;
VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
- /* for folio_migrate_flags() */
- flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
- flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
- gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+ /*
+ * See the comment in lru_gen_folio_seq. For migration, compaction,
+ * or any other isolation of a hot folio, try best to retain its gen
+ * info. Ideally we would keep the full gen info.
+ */
+ if (!reclaiming && ((max_seq - gen) % MAX_NR_GENS) < MIN_NR_GENS)
+ folio_set_active(folio);
lru_gen_update_size(lruvec, folio, gen, -1);
- if (lru_gen_is_active(lruvec, gen))
+ if (lru_refs_is_active(refs))
lru += LRU_ACTIVE;
__update_lru_size(lruvec, lru, zone, -delta);
list_del(&folio->lru);
@@ -470,6 +478,7 @@ static __always_inline void folio_inc_lru_refs_fast(struct folio *folio)
return;
}
+ BUILD_BUG_ON(LRU_REFS_REFERENCED >= LRU_REFS_ACTIVATED);
old_flags = READ_ONCE(*folio_flags(folio, 0));
do {
if (lru_get_refs_flags(old_flags))
@@ -505,7 +514,7 @@ static inline bool folio_check_lru_refs(const struct folio *folio)
static inline void folio_clear_lru_refs(struct folio *folio)
{
if (lru_gen_enabled())
- folio_set_lru_refs(folio, 0);
+ folio_reset_lru_refs(folio);
else
folio_clear_referenced(folio);
}
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index b5a3769fdcab..dacf066b499e 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -434,15 +434,15 @@ enum lruvec_flags {
* least twice before handing this folio over to the eviction. The first check
* clears the accessed bit from the initial fault; the second check makes sure
* this folio hasn't been used since then. This process, AKA second chance,
- * requires a minimum of two generations, hence MIN_NR_GENS. And to maintain ABI
- * compatibility with the active/inactive LRU, e.g., /proc/vmstat, these two
- * generations are considered active; the rest of generations, if they exist,
- * are considered inactive. See lru_gen_is_active().
+ * requires a minimum of two generations, hence MIN_NR_GENS.
+ *
+ * Active/inactive is per-folio based on the referenced count;
+ * see the comment above MAX_NR_TIERS.
*
* PG_active is always cleared while a folio is on one of lrugen->folios[] so
- * that the sliding window needs not to worry about it. And it's set again when
- * a folio considered active is isolated for non-reclaiming purposes, e.g.,
- * migration. See lru_gen_add_folio() and lru_gen_del_folio().
+ * that the sliding window needs not to worry about it. It is set on a
+ * fault-driven refault (see lru_gen_refault), or isolation (see
+ * lru_gen_del_folio).
*
* MAX_NR_GENS is set to 4 so that the multi-gen LRU can support twice the
* number of categories of the active/inactive LRU when keeping track of
@@ -543,11 +543,15 @@ enum lruvec_flags {
*
* A folio's referenced count never goes backwards except upon gen
* increase as described above, or when explicitly reset by
- * lru_gen_clear_refs(). Refault of a reclaimed folio restores
+ * folio_reset_lru_refs(). Refault of a reclaimed folio restores
* its referenced count, capped at LRU_REFS_PROTECTED, which aligns with
* promotion. Page table refaults of previous workingset folios send
* them to the latest gen, driving aging faster.
*
+ * For the active/inactive LRU ABI (/proc/vmstat), a folio is active
+ * if its referenced count reaches LRU_REFS_ACTIVATED. The threshold
+ * currently set to LRU_REFS_PROTECTED, which is tier 2.
+ *
* MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice
* the number of categories of the active/inactive LRU.
*/
@@ -562,6 +566,7 @@ enum lruvec_flags {
#define LRU_REFS_REFERENCED 0x1
#define LRU_REFS_WORKINGSET 0x2
#define LRU_REFS_PROTECTED 0x3
+#define LRU_REFS_ACTIVATED LRU_REFS_PROTECTED
#ifndef __GENERATING_BOUNDS_H
@@ -670,6 +675,8 @@ struct lru_gen_mm_walk {
unsigned long seq;
/* the next address within an mm to scan */
unsigned long next_addr;
+ /* to batch activated pages */
+ int nr_activated[ANON_AND_FILE][MAX_NR_ZONES];
/* to batch promoted pages */
int nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
/* to batch the mm stats */
diff --git a/mm/folio.c b/mm/folio.c
index e200efab814f..9b9a5fe568c9 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -349,35 +349,6 @@ static void __lru_cache_activate_folio(struct folio *folio)
local_unlock(&cpu_fbatches.lock);
}
-#ifdef CONFIG_LRU_GEN
-
-static bool lru_gen_clear_refs(struct folio *folio)
-{
- int gen = folio_lru_gen(folio);
- int type = folio_is_file_lru(folio);
- unsigned long seq;
-
- if (gen < 0)
- return true;
-
- folio_set_lru_refs(folio, 0);
-
- rcu_read_lock();
- seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
- rcu_read_unlock();
- /* whether can do without shuffling under the LRU lock */
- return gen == lru_gen_from_seq(seq);
-}
-
-#else /* !CONFIG_LRU_GEN */
-
-static bool lru_gen_clear_refs(struct folio *folio)
-{
- return false;
-}
-
-#endif /* CONFIG_LRU_GEN */
-
/**
* folio_mark_accessed - Mark a folio as having seen activity.
* @folio: The folio to mark.
@@ -554,7 +525,7 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
lruvec_del_folio(lruvec, folio);
folio_clear_active(folio);
if (lru_gen_enabled())
- lru_gen_clear_refs(folio);
+ __folio_set_lru_refs(folio, 0);
else
folio_clear_referenced(folio);
/*
@@ -627,7 +598,7 @@ void deactivate_file_folio(struct folio *folio)
if (folio_test_unevictable(folio) || !folio_test_lru(folio))
return;
- if (lru_gen_enabled() && lru_gen_clear_refs(folio))
+ if (lru_gen_enabled() && !folio_reset_lru_refs(folio))
return;
folio_batch_add_and_move(folio, lru_deactivate_file);
@@ -646,7 +617,7 @@ void folio_deactivate(struct folio *folio)
if (folio_test_unevictable(folio) || !folio_test_lru(folio))
return;
- if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(folio))
+ if (lru_gen_enabled() ? !folio_reset_lru_refs(folio) : !folio_test_active(folio))
return;
folio_batch_add_and_move(folio, lru_deactivate);
diff --git a/mm/madvise.c b/mm/madvise.c
index 2b4a8b2cb477..184611fdd6f9 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -374,7 +374,8 @@ static void madvise_cold_prep_folio(struct folio *folio)
* list should have PG_workingset so its refault is still accounted
* as a workingset refault, and the referenced bit always needs to
* be cleared even for inactive folios. Nothing to do for MGLRU as
- * folio_deactivate() always calls lru_gen_clear_refs().
+ * folio_deactivate() clears the refs count through
+ * folio_reset_lru_refs().
*/
if (!lru_gen_enabled()) {
folio_clear_referenced(folio);
@@ -395,7 +396,7 @@ static void madvise_pageout_prep_folio(struct folio *folio)
* the workingset classification is preserved for PSI.
*/
if (lru_gen_enabled())
- folio_set_lru_refs(folio, refs >= LRU_REFS_WORKINGSET ? LRU_REFS_WORKINGSET : 0);
+ __folio_set_lru_refs(folio, refs >= LRU_REFS_WORKINGSET ? LRU_REFS_WORKINGSET : 0);
}
static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
diff --git a/mm/vmscan.c b/mm/vmscan.c
index d0f0e5a25d87..617f69c1d9f9 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -977,17 +977,19 @@ enum folio_references {
void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
{
int max_gen;
- int type, refs, old_gen, gen;
+ int file, old_refs, refs, old_gen, gen;
unsigned long new_flags, old_flags, max_seq;
+ long nr_pages = folio_nr_pages(folio);
struct lru_gen_folio *lrugen;
struct lruvec *lruvec = NULL;
- type = folio_is_file_lru(folio);
old_flags = READ_ONCE(*folio_flags(folio, 0));
do {
new_flags = old_flags;
old_gen = lru_get_gen_flags(old_flags);
- refs = lru_get_refs_flags(old_flags) + 1;
+ old_refs = lru_get_refs_flags(old_flags);
+ file = folio_flags_is_file_lru(&old_flags);
+ refs = old_refs + 1;
gen = old_gen;
if (old_gen < 0)
goto out;
@@ -1013,14 +1015,14 @@ void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
if (refs > LRU_REFS_REFERENCED || flags & LRU_REF_EXEC)
gen = max_gen;
/* First access only defers eviction of a tail gen */
- else if (lru_gen_at_tail(lruvec, type, old_gen))
+ else if (lru_gen_at_tail(lruvec, file, old_gen))
gen = (old_gen + 1) % MAX_NR_GENS;
refs = min(refs, LRU_REFS_PROTECTED);
} else if (refs > LRU_REFS_MAX) {
/* LRU refs counting overflow, bump the gen */
gen = (old_gen + 1) % MAX_NR_GENS;
refs = LRU_REFS_PROTECTED;
- } else if (lru_gen_at_tail(lruvec, type, old_gen) && refs >= LRU_REFS_WORKINGSET) {
+ } else if (lru_gen_at_tail(lruvec, file, old_gen) && refs >= LRU_REFS_WORKINGSET) {
/* Defer eviction of just accessed workingset */
gen = (old_gen + 1) % MAX_NR_GENS;
refs = min(refs, LRU_REFS_PROTECTED);
@@ -1036,54 +1038,108 @@ void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
if (gen != old_gen)
lru_gen_update_size(lruvec, folio, old_gen, gen);
+ if (lru_refs_is_active(old_refs) != lru_refs_is_active(refs) && old_gen >= 0) {
+ enum lru_list lru = file * LRU_INACTIVE_FILE;
+
+ __update_lru_size(lruvec, lru + lru_refs_is_active(old_refs),
+ folio_zonenum(folio), -nr_pages);
+ __update_lru_size(lruvec, lru + lru_refs_is_active(refs),
+ folio_zonenum(folio), nr_pages);
+ }
if (lruvec)
lruvec_unlock_irq(lruvec);
}
+/*
+ * Reset the folio's lru refs indicator. The caller doesn't need to hold
+ * the folio lock, isolate the folio, or hold the lruvec lock.
+ */
+bool folio_reset_lru_refs(struct folio *folio)
+{
+ int type, gen, refs;
+ unsigned long seq, new_flags, old_flags;
+ long nr_pages = folio_nr_pages(folio);
+ struct lruvec *lruvec;
+ bool reset = false;
+
+ old_flags = READ_ONCE(*folio_flags(folio, 0));
+ do {
+ new_flags = old_flags;
+ refs = lru_get_refs_flags(old_flags);
+ type = folio_flags_is_file_lru(&old_flags);
+ gen = lru_get_gen_flags(old_flags);
+ if (!refs)
+ break;
+ lru_set_refs_flags(&new_flags, 0);
+ } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+
+ if (gen >= 0) {
+ lruvec = folio_lruvec_live_get(folio);
+
+ /* If the folio is on list, caller might also want to demote it */
+ seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
+ reset = (gen != lru_gen_from_seq(seq));
+ if (lru_refs_is_active(refs)) {
+ __update_lru_size(lruvec, type * LRU_FILE + LRU_ACTIVE,
+ folio_zonenum(folio), -nr_pages);
+ __update_lru_size(lruvec, type * LRU_FILE,
+ folio_zonenum(folio), nr_pages);
+ }
+
+ folio_lruvec_live_put(lruvec);
+ }
+
+ return reset;
+}
+
/*
* Update the folio's lru refs indicator during a page table walk or the
* look-around. max_seq can be stale as neither holds the LRU lock.
*
- * Returns the old generation and stores the new generation in @new_gen if
- * the folio is on the LRU and not in the newest generation, or -1 otherwise.
+ * Returns the old generation, or -1 if the folio is off the LRU list.
+ * Old LRU refs, and updated gen and LRU refs info by the successful
+ * cmpxchg are all stored by returning arguments.
*/
static int folio_inc_lru_refs_walk(struct folio *folio, struct lruvec *lruvec,
- const vma_flags_t *vma_flags,
- int *new_gen, int *type)
+ const vma_flags_t *vma_flags, int *new_gen,
+ int *old_refs, int *new_refs, int *type)
{
unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
- int refs, gen, max_gen, ret;
+ int refs, old_gen, max_gen;
unsigned long max_seq;
do {
max_seq = READ_ONCE(lruvec->lrugen.max_seq);
max_gen = lru_gen_from_seq(max_seq);
- gen = lru_get_gen_flags(old_flags);
+ old_gen = lru_get_gen_flags(old_flags);
refs = lru_get_refs_flags(old_flags) + 1;
*type = folio_flags_is_file_lru(&old_flags);
new_flags = old_flags;
- if (gen >= 0 && gen != max_gen) {
- ret = gen;
+ if (old_gen >= 0 && old_gen != max_gen) {
+ *new_refs = min(refs, LRU_REFS_PROTECTED);
/* Promote second page table access or executable */
if (refs > LRU_REFS_REFERENCED || is_exec_file_folio(folio, vma_flags))
*new_gen = max_gen;
/* First access only defers eviction of a tail gen */
- else if (lru_gen_at_tail(lruvec, *type, gen))
- *new_gen = (gen + 1) % MAX_NR_GENS;
+ else if (lru_gen_at_tail(lruvec, *type, old_gen))
+ *new_gen = (old_gen + 1) % MAX_NR_GENS;
else
- *new_gen = gen;
+ *new_gen = old_gen;
lru_set_gen_flags(&new_flags, *new_gen);
- lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_PROTECTED));
+ lru_set_refs_flags(&new_flags, *new_refs);
} else {
- ret = -1;
- lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_MAX));
+ *new_gen = old_gen;
+ *new_refs = min(refs, LRU_REFS_MAX);
+ lru_set_refs_flags(&new_flags, *new_refs);
}
if (new_flags == old_flags)
break;
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
- return ret;
+ *old_refs = lru_get_refs_flags(old_flags);
+
+ return old_gen;
}
/*
@@ -3537,10 +3593,11 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
* the aging
******************************************************************************/
-static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
+static int __folio_inc_gen(struct lruvec *lruvec, struct folio *folio,
+ int old_gen, bool *increased)
{
unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
- int refs, new_gen;
+ int file, refs, old_refs, new_gen;
do {
new_gen = lru_get_gen_flags(old_flags);
@@ -3554,11 +3611,24 @@ static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
new_flags = old_flags;
new_gen = (old_gen + 1) % MAX_NR_GENS;
- refs = lru_get_refs_flags(old_flags);
+ old_refs = lru_get_refs_flags(old_flags);
+ file = folio_flags_is_file_lru(&old_flags);
+ refs = min(old_refs, LRU_REFS_WORKINGSET);
lru_set_gen_flags(&new_flags, new_gen);
- lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_WORKINGSET));
+ lru_set_refs_flags(&new_flags, refs);
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+ /* Refs capped below PROTECTED: demote if previously active. */
+ if (lru_refs_is_active(old_refs) != lru_refs_is_active(refs)) {
+ enum lru_list lru = file * LRU_INACTIVE_FILE;
+ int nr_pages = folio_nr_pages(folio);
+
+ __update_lru_size(lruvec, lru + lru_refs_is_active(old_refs),
+ folio_zonenum(folio), -nr_pages);
+ __update_lru_size(lruvec, lru + lru_refs_is_active(refs),
+ folio_zonenum(folio), nr_pages);
+ }
+
if (increased)
*increased = true;
return new_gen;
@@ -3575,7 +3645,7 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
struct lru_gen_folio *lrugen = &lruvec->lrugen;
int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
- new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
+ new_gen = __folio_inc_gen(lruvec, folio, old_gen, &gen_increased);
if (gen_increased)
lru_gen_update_size(lruvec, folio, old_gen, new_gen);
@@ -3583,18 +3653,26 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
}
static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *folio,
- int old_gen, int new_gen, int type)
+ int old_gen, int new_gen, int old_refs, int new_refs, int type)
{
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
- VM_WARN_ON_ONCE(old_gen >= MAX_NR_GENS);
- VM_WARN_ON_ONCE(new_gen >= MAX_NR_GENS);
-
- walk->batched++;
+ /* gen counter update */
+ if (old_gen != new_gen) {
+ VM_WARN_ON_ONCE(old_gen >= MAX_NR_GENS);
+ VM_WARN_ON_ONCE(new_gen >= MAX_NR_GENS);
+ walk->batched++;
+ walk->nr_pages[old_gen][type][zone] -= delta;
+ walk->nr_pages[new_gen][type][zone] += delta;
+ }
- walk->nr_pages[old_gen][type][zone] -= delta;
- walk->nr_pages[new_gen][type][zone] += delta;
+ /* active/inactive counter update */
+ if (lru_refs_is_active(old_refs) != lru_refs_is_active(new_refs)) {
+ walk->nr_activated[type][zone] +=
+ lru_refs_is_active(new_refs) ? delta : -delta;
+ walk->batched++;
+ }
}
static void reset_batch_size(struct lru_gen_mm_walk *walk)
@@ -3606,7 +3684,6 @@ static void reset_batch_size(struct lru_gen_mm_walk *walk)
walk->batched = 0;
for_each_gen_type_zone(gen, type, zone) {
- enum lru_list lru = type * LRU_INACTIVE_FILE;
int delta = walk->nr_pages[gen][type][zone];
if (!delta)
@@ -3614,10 +3691,21 @@ static void reset_batch_size(struct lru_gen_mm_walk *walk)
walk->nr_pages[gen][type][zone] = 0;
atomic_long_add(delta, &lrugen->nr_pages[gen][type][zone]);
+ }
- if (lru_gen_is_active(lruvec, gen))
- lru += LRU_ACTIVE;
- __update_lru_size(lruvec, lru, zone, delta);
+ /* apply batched active/inactive updates */
+ for (type = 0; type < ANON_AND_FILE; type++) {
+ for (zone = 0; zone < MAX_NR_ZONES; zone++) {
+ enum lru_list lru = type * LRU_INACTIVE_FILE;
+ int delta = walk->nr_activated[type][zone];
+
+ if (delta) {
+ __update_lru_size(lruvec, lru + LRU_ACTIVE,
+ zone, delta);
+ __update_lru_size(lruvec, lru, zone, -delta);
+ walk->nr_activated[type][zone] = 0;
+ }
+ }
}
lruvec_unlock_irq(lruvec);
@@ -3772,7 +3860,7 @@ static bool suitable_to_scan(int total, int young)
static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma,
struct lruvec *lruvec, struct folio *folio, bool dirty)
{
- int new_gen, old_gen, type;
+ int new_gen, old_gen, old_refs, new_refs, type;
unsigned int flags = LRU_REF_MAPPED;
if (!folio)
@@ -3785,9 +3873,10 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struc
if (walk) {
old_gen = folio_inc_lru_refs_walk(folio, lruvec, &vma->flags,
- &new_gen, &type);
- if (old_gen >= 0 && old_gen != new_gen)
- update_batch_size(walk, folio, old_gen, new_gen, type);
+ &new_gen, &old_refs, &new_refs, &type);
+ if (old_gen >= 0)
+ update_batch_size(walk, folio, old_gen, new_gen,
+ old_refs, new_refs, type);
} else {
if (is_exec_file_folio(folio, &vma->flags))
flags |= LRU_REF_EXEC;
@@ -4146,6 +4235,7 @@ static void clear_mm_walk(void)
VM_WARN_ON_ONCE(walk && memchr_inv(walk->nr_pages, 0, sizeof(walk->nr_pages)));
VM_WARN_ON_ONCE(walk && memchr_inv(walk->mm_stats, 0, sizeof(walk->mm_stats)));
+ VM_WARN_ON_ONCE(walk && memchr_inv(walk->nr_activated, 0, sizeof(walk->nr_activated)));
current->reclaim_state->mm_walk = NULL;
@@ -4184,8 +4274,6 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
goto done;
VM_WARN_ON_ONCE(get_nr_gens(lruvec, type) != MAX_NR_GENS);
- VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) !=
- lru_gen_is_active(lruvec, target_gen));
/* prevent cold/hot inversion if the type is evictable */
for (zone = 0; zone < MAX_NR_ZONES; zone++) {
struct list_head *target_list = &lrugen->folios[target_gen][type][zone];
@@ -4208,7 +4296,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
prefetchw_next_lru_folio(folio, head);
pos = pos->next;
- new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
+ new_gen = __folio_inc_gen(lruvec, folio, old_gen, &gen_increased);
/*
* If gen_increased is false, this is a promotion. Put folios
* at the head of the promoted gen. Otherwise, put them at
@@ -4295,8 +4383,7 @@ static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)
static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness)
{
bool success;
- int prev, next;
- int type, zone;
+ int type, next;
struct lru_gen_folio *lrugen = &lruvec->lrugen;
restart:
if (seq < READ_ONCE(lrugen->max_seq))
@@ -4322,32 +4409,10 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
goto restart;
}
- /*
- * Update the active/inactive LRU sizes for compatibility. Both sides of
- * the current max_seq need to be covered, since max_seq+1 can overlap
- * with min_seq[LRU_GEN_ANON] if swapping is constrained. And if they do
- * overlap, cold/hot inversion happens.
- */
- prev = lru_gen_from_seq(lrugen->max_seq - 1);
- next = lru_gen_from_seq(lrugen->max_seq + 1);
-
- for (type = 0; type < ANON_AND_FILE; type++) {
- for (zone = 0; zone < MAX_NR_ZONES; zone++) {
- enum lru_list lru = type * LRU_INACTIVE_FILE;
- long delta = atomic_long_read(&lrugen->nr_pages[prev][type][zone]) -
- atomic_long_read(&lrugen->nr_pages[next][type][zone]);
-
- if (!delta)
- continue;
-
- __update_lru_size(lruvec, lru, zone, delta);
- __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, -delta);
- }
- }
-
for (type = 0; type < ANON_AND_FILE; type++)
reset_ctrl_pos(lruvec, type, false);
+ next = lru_gen_from_seq(lrugen->max_seq + 1);
WRITE_ONCE(lrugen->timestamps[next], jiffies);
/* make sure preceding modifications appear */
smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1);
@@ -4866,7 +4931,6 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
int zone, int type)
{
struct lru_gen_folio *child_lrugen, *parent_lrugen;
- enum lru_list lru = type * LRU_INACTIVE_FILE;
int i;
child_lrugen = &child_lruvec->lrugen;
@@ -4875,8 +4939,6 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
for (i = 0; i < get_nr_gens(child_lruvec, type); i++) {
int gen = lru_gen_from_seq(child_lrugen->max_seq - i);
long nr_pages = atomic_long_read(&child_lrugen->nr_pages[gen][type][zone]);
- int child_lru_active = lru_gen_is_active(child_lruvec, gen) ? LRU_ACTIVE : 0;
- int parent_lru_active = lru_gen_is_active(parent_lruvec, gen) ? LRU_ACTIVE : 0;
/* Assuming that child pages are colder than parent pages */
list_splice_tail_init(&child_lrugen->folios[gen][type][zone],
@@ -4884,11 +4946,6 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
atomic_long_set(&child_lrugen->nr_pages[gen][type][zone], 0);
atomic_long_add(nr_pages, &parent_lrugen->nr_pages[gen][type][zone]);
-
- if (lru_gen_is_active(child_lruvec, gen) != lru_gen_is_active(parent_lruvec, gen)) {
- __update_lru_size(child_lruvec, lru + child_lru_active, zone, -nr_pages);
- __update_lru_size(parent_lruvec, lru + parent_lru_active, zone, nr_pages);
- }
}
}
@@ -5228,7 +5285,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
* its first page table access lands in the second newest one.
* See "Referenced count feedback" above.
*/
- folio_set_lru_refs(folio, min(folio_lru_refs(folio), LRU_REFS_WORKINGSET));
+ __folio_set_lru_refs(folio, min(folio_lru_refs(folio), LRU_REFS_WORKINGSET));
if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type])
folio_set_active(folio);
}
diff --git a/mm/workingset.c b/mm/workingset.c
index 16cdd88e275e..7681af535ca7 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -334,7 +334,7 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
}
/* Refault is also promotion, cap the refs like folio_inc_lru_refs */
- folio_set_lru_refs(folio, min(refs, LRU_REFS_PROTECTED));
+ __folio_set_lru_refs(folio, min(refs, LRU_REFS_PROTECTED));
}
if (refs >= LRU_REFS_WORKINGSET)
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 13/17] mm/mglru: folio LRU refs based active/inactive number accounting
@ 2026-10-15 18:16 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
Currently the active/inactive LRU sizes exported through /proc/vmstat
and memory.stat are derived from the generation window: folios in the
two newest generations are accounted as active, the rest as inactive.
This has always caused many problems:
- Unmapped file folios stick to the oldest gens, so active files are
always under-reported, unlike classical LRU.
- The reading is jumpy: on aging, the second-newest gen, previously
active, becomes inactive all at once, causing a huge drop of active
folios for no reason from time to time.
- The reading is reversed on swapless machines, and the active anon
number is an oscillating random number. When swappiness is 0 (no
swap or swap is full), MGLRU ages anon by moving the gens directly
without walk or touching any folio, to reduce the overhead. So all
inactive anon folios will suddenly count as active, or vice versa,
which breaks many metric reading components, and the reading itself
doesn't make any sense in any way.
With frequency guided promotion, hotness is already tracked per folio by
its referenced count, so we can do folio level accounting instead: a
folio is active once its refs reach LRU_REFS_ACTIVATED (currently
LRU_REFS_PROTECTED).
On access, folios are accounted to the active part. On forced aging or
PID protecting, folios are reset to LRU_REFS_WORKINGSET, dropping them
to the inactive part. The reading is now smooth and consistent, and
much closer to classical LRU. Note that on swapless machines anon refs
are never reset by aging (the shortcut skips the per-folio walk), so
active anon grows monotonically, but the reading is stable and still
much more meaningful than the previous oscillating random number.
The accounting is now per folio and lruvec independent, so no size
adjustment is needed when reparenting a dying memcg: the hierarchical
stats keep the counts in the parent automatically.
folio_migrate_lru_refs() also gains off-LRU VM_WARN_ON_ONCE checks and
__folio_set_lru_refs() warns when the folio is still on an LRU list.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
Documentation/admin-guide/mm/multigen_lru.rst | 4 +-
include/linux/mm_inline.h | 115 +++++++-------
include/linux/mmzone.h | 23 ++-
mm/folio.c | 35 +----
mm/madvise.c | 5 +-
mm/vmscan.c | 211 ++++++++++++++++----------
mm/workingset.c | 2 +-
7 files changed, 220 insertions(+), 175 deletions(-)
diff --git a/Documentation/admin-guide/mm/multigen_lru.rst b/Documentation/admin-guide/mm/multigen_lru.rst
index 9cb54b4ff5d9..808d2e97b3f3 100644
--- a/Documentation/admin-guide/mm/multigen_lru.rst
+++ b/Documentation/admin-guide/mm/multigen_lru.rst
@@ -149,8 +149,8 @@ generations less than or equal to ``min_gen_nr``.
``- memcg_id node_id min_gen_nr [swappiness [nr_to_reclaim]]``
``min_gen_nr`` should be less than ``max_gen_nr-1``, since
-``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged (equivalent to
-the active list) and therefore cannot be evicted. ``swappiness``
+``max_gen_nr`` and ``max_gen_nr-1`` are not fully aged and therefore
+cannot be evicted. ``swappiness``
overrides the default value in ``/proc/sys/vm/swappiness`` and the valid
range is [0-200, max], with max being exclusively used for the reclamation
of anonymous memory. ``nr_to_reclaim`` limits the number of pages to evict.
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 6c9510002754..41c2e9150c70 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -105,6 +105,22 @@ static __always_inline enum lru_list folio_lru_list(const struct folio *folio)
return lru;
}
+/**
+ * lru_get_gen_flags - Return the LRU generation number from folio flags.
+ * @flags: folio flags
+ *
+ * Returns: A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns
+ * -1 if the flags indicate the folio is off the list (e.g., isolated).
+ */
+static inline int lru_get_gen_flags(unsigned long flags)
+{
+ int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+
+ /* Exclude the legal -1 from the unsigned MAX_NR_GENS comparison */
+ VM_WARN_ON_ONCE(gen != -1 && gen >= MAX_NR_GENS);
+ return gen;
+}
+
#ifdef CONFIG_LRU_GEN
static inline bool lru_gen_switching(void)
@@ -167,22 +183,6 @@ static inline void lru_set_gen_flags(unsigned long *flags, int gen)
*flags |= (gen + 1UL) << LRU_GEN_PGOFF;
}
-/**
- * lru_get_gen_flags - Return the LRU generation number from folio flags.
- * @flags: folio flags
- *
- * Returns: A number between 0 and (MAX_NR_GENS - 1), inclusive. Returns
- * -1 if the flags indicate the folio is off the list (e.g., isolated).
- */
-static inline int lru_get_gen_flags(unsigned long flags)
-{
- int gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
-
- /* Exclude the legal -1 from the unsigned MAX_NR_GENS comparison */
- VM_WARN_ON_ONCE(gen != -1 && gen >= MAX_NR_GENS);
- return gen;
-}
-
static inline int folio_lru_gen(const struct folio *folio)
{
return lru_get_gen_flags(READ_ONCE(*const_folio_flags(folio, 0)));
@@ -237,24 +237,39 @@ static inline int folio_lru_refs(const struct folio *folio)
return lru_get_refs_flags(READ_ONCE(*const_folio_flags(folio, 0)));
}
-static inline void folio_set_lru_refs(struct folio *folio, unsigned int refs)
+/**
+ * __folio_set_lru_refs - Set a folio's LRU refs.
+ * @folio: the folio
+ * @refs: the new referenced count (0 .. LRU_REFS_MAX)
+ *
+ * Set the folio's LRU refs. The folio must be off the LRU list (e.g.,
+ * isolated), or use folio_inc_lru_refs or folio_reset_lru_refs instead.
+ */
+static inline void __folio_set_lru_refs(struct folio *folio, unsigned int refs)
{
unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
do {
new_flags = old_flags;
+ VM_WARN_ON_ONCE(lru_get_gen_flags(old_flags) != -1);
lru_set_refs_flags(&new_flags, refs);
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
}
#ifdef CONFIG_LRU_GEN
void folio_inc_lru_refs(struct folio *folio, unsigned int flags);
+bool folio_reset_lru_refs(struct folio *folio);
#else
static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
{
/* Should not be called with !CONFIG_LRU_GEN */
WARN_ON_ONCE(1);
}
+
+static inline bool folio_reset_lru_refs(struct folio *folio)
+{
+ return false;
+}
#endif
/**
@@ -268,10 +283,18 @@ static inline void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
*/
static inline void folio_migrate_lru_refs(struct folio *new, const struct folio *old)
{
- folio_set_lru_refs(new, folio_lru_refs(old));
+ VM_WARN_ON_ONCE_FOLIO(folio_test_lru(old), old);
+ VM_WARN_ON_ONCE_FOLIO(folio_test_lru(new), new);
+ __folio_set_lru_refs(new, folio_lru_refs(old));
}
#ifdef CONFIG_LRU_GEN
+static inline bool lru_refs_is_active(unsigned int refs)
+{
+ VM_WARN_ON_ONCE(refs > LRU_REFS_MAX);
+ return refs >= LRU_REFS_ACTIVATED;
+}
+
static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
{
unsigned long min_seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
@@ -285,23 +308,12 @@ static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
gen == lru_gen_from_seq(min_seq + 1);
}
-static inline bool lru_gen_is_active(const struct lruvec *lruvec, int gen)
-{
- unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
-
- VM_WARN_ON_ONCE(gen >= MAX_NR_GENS);
-
- /* see the comment on MIN_NR_GENS */
- return gen == lru_gen_from_seq(max_seq) || gen == lru_gen_from_seq(max_seq - 1);
-}
-
static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *folio,
int old_gen, int new_gen)
{
int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
- enum lru_list lru = type * LRU_INACTIVE_FILE;
struct lru_gen_folio *lrugen = &lruvec->lrugen;
VM_WARN_ON_ONCE(old_gen != -1 && old_gen >= MAX_NR_GENS);
@@ -312,19 +324,6 @@ static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *foli
atomic_long_sub(delta, &lrugen->nr_pages[old_gen][type][zone]);
if (new_gen >= 0)
atomic_long_add(delta, &lrugen->nr_pages[new_gen][type][zone]);
-
- /* return now if not a promotion */
- if (old_gen < 0 || new_gen < 0)
- return;
-
- /* promotion */
- if (!lru_gen_is_active(lruvec, old_gen) && lru_gen_is_active(lruvec, new_gen)) {
- __update_lru_size(lruvec, lru, zone, -delta);
- __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, delta);
- }
-
- /* demotion requires isolation, e.g., lru_deactivate_fn() */
- VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) && !lru_gen_is_active(lruvec, new_gen));
}
static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
@@ -366,7 +365,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
{
unsigned long seq;
unsigned long flags;
- int gen = folio_lru_gen(folio);
+ int gen, refs;
int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
@@ -374,7 +373,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
struct lru_gen_folio *lrugen = &lruvec->lrugen;
BUILD_BUG_ON(BIT(LRU_GEN_WIDTH - 1) != MAX_NR_GENS);
- VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
+ VM_WARN_ON_ONCE_FOLIO(folio_lru_gen(folio) != -1, folio);
if (folio_test_unevictable(folio) || !lrugen->enabled)
return false;
@@ -383,10 +382,12 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
gen = lru_gen_from_seq(seq);
flags = (gen + 1UL) << LRU_GEN_PGOFF;
/* see the comment on MIN_NR_GENS about PG_active */
- set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
+ flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
+ /* use the refs from the atomic snapshot to avoid raced update */
+ refs = lru_get_refs_flags(flags);
lru_gen_update_size(lruvec, folio, -1, gen);
- if (lru_gen_is_active(lruvec, gen))
+ if (lru_refs_is_active(refs))
lru += LRU_ACTIVE;
__update_lru_size(lruvec, lru, zone, delta);
@@ -402,24 +403,31 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming)
{
unsigned long flags;
- int gen = folio_lru_gen(folio);
+ int gen, refs;
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
enum lru_list lru = folio_is_file_lru(folio) * LRU_INACTIVE_FILE;
+ unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
+ flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, 0);
+ gen = lru_get_gen_flags(flags);
+ refs = lru_get_refs_flags(flags);
if (gen < 0)
return false;
VM_WARN_ON_ONCE_FOLIO(folio_test_active(folio), folio);
VM_WARN_ON_ONCE_FOLIO(folio_test_unevictable(folio), folio);
- /* for folio_migrate_flags() */
- flags = !reclaiming && lru_gen_is_active(lruvec, gen) ? BIT(PG_active) : 0;
- flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, flags);
- gen = ((flags & LRU_GEN_MASK) >> LRU_GEN_PGOFF) - 1;
+ /*
+ * See the comment in lru_gen_folio_seq. For migration, compaction,
+ * or any other isolation of a hot folio, try best to retain its gen
+ * info. Ideally we would keep the full gen info.
+ */
+ if (!reclaiming && ((max_seq - gen) % MAX_NR_GENS) < MIN_NR_GENS)
+ folio_set_active(folio);
lru_gen_update_size(lruvec, folio, gen, -1);
- if (lru_gen_is_active(lruvec, gen))
+ if (lru_refs_is_active(refs))
lru += LRU_ACTIVE;
__update_lru_size(lruvec, lru, zone, -delta);
list_del(&folio->lru);
@@ -470,6 +478,7 @@ static __always_inline void folio_inc_lru_refs_fast(struct folio *folio)
return;
}
+ BUILD_BUG_ON(LRU_REFS_REFERENCED >= LRU_REFS_ACTIVATED);
old_flags = READ_ONCE(*folio_flags(folio, 0));
do {
if (lru_get_refs_flags(old_flags))
@@ -505,7 +514,7 @@ static inline bool folio_check_lru_refs(const struct folio *folio)
static inline void folio_clear_lru_refs(struct folio *folio)
{
if (lru_gen_enabled())
- folio_set_lru_refs(folio, 0);
+ folio_reset_lru_refs(folio);
else
folio_clear_referenced(folio);
}
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index b5a3769fdcab..dacf066b499e 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -434,15 +434,15 @@ enum lruvec_flags {
* least twice before handing this folio over to the eviction. The first check
* clears the accessed bit from the initial fault; the second check makes sure
* this folio hasn't been used since then. This process, AKA second chance,
- * requires a minimum of two generations, hence MIN_NR_GENS. And to maintain ABI
- * compatibility with the active/inactive LRU, e.g., /proc/vmstat, these two
- * generations are considered active; the rest of generations, if they exist,
- * are considered inactive. See lru_gen_is_active().
+ * requires a minimum of two generations, hence MIN_NR_GENS.
+ *
+ * Active/inactive is per-folio based on the referenced count;
+ * see the comment above MAX_NR_TIERS.
*
* PG_active is always cleared while a folio is on one of lrugen->folios[] so
- * that the sliding window needs not to worry about it. And it's set again when
- * a folio considered active is isolated for non-reclaiming purposes, e.g.,
- * migration. See lru_gen_add_folio() and lru_gen_del_folio().
+ * that the sliding window needs not to worry about it. It is set on a
+ * fault-driven refault (see lru_gen_refault), or isolation (see
+ * lru_gen_del_folio).
*
* MAX_NR_GENS is set to 4 so that the multi-gen LRU can support twice the
* number of categories of the active/inactive LRU when keeping track of
@@ -543,11 +543,15 @@ enum lruvec_flags {
*
* A folio's referenced count never goes backwards except upon gen
* increase as described above, or when explicitly reset by
- * lru_gen_clear_refs(). Refault of a reclaimed folio restores
+ * folio_reset_lru_refs(). Refault of a reclaimed folio restores
* its referenced count, capped at LRU_REFS_PROTECTED, which aligns with
* promotion. Page table refaults of previous workingset folios send
* them to the latest gen, driving aging faster.
*
+ * For the active/inactive LRU ABI (/proc/vmstat), a folio is active
+ * if its referenced count reaches LRU_REFS_ACTIVATED. The threshold
+ * currently set to LRU_REFS_PROTECTED, which is tier 2.
+ *
* MAX_NR_TIERS is set to 4 so that the multi-gen LRU can support twice
* the number of categories of the active/inactive LRU.
*/
@@ -562,6 +566,7 @@ enum lruvec_flags {
#define LRU_REFS_REFERENCED 0x1
#define LRU_REFS_WORKINGSET 0x2
#define LRU_REFS_PROTECTED 0x3
+#define LRU_REFS_ACTIVATED LRU_REFS_PROTECTED
#ifndef __GENERATING_BOUNDS_H
@@ -670,6 +675,8 @@ struct lru_gen_mm_walk {
unsigned long seq;
/* the next address within an mm to scan */
unsigned long next_addr;
+ /* to batch activated pages */
+ int nr_activated[ANON_AND_FILE][MAX_NR_ZONES];
/* to batch promoted pages */
int nr_pages[MAX_NR_GENS][ANON_AND_FILE][MAX_NR_ZONES];
/* to batch the mm stats */
diff --git a/mm/folio.c b/mm/folio.c
index e200efab814f..9b9a5fe568c9 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -349,35 +349,6 @@ static void __lru_cache_activate_folio(struct folio *folio)
local_unlock(&cpu_fbatches.lock);
}
-#ifdef CONFIG_LRU_GEN
-
-static bool lru_gen_clear_refs(struct folio *folio)
-{
- int gen = folio_lru_gen(folio);
- int type = folio_is_file_lru(folio);
- unsigned long seq;
-
- if (gen < 0)
- return true;
-
- folio_set_lru_refs(folio, 0);
-
- rcu_read_lock();
- seq = READ_ONCE(folio_lruvec(folio)->lrugen.min_seq[type]);
- rcu_read_unlock();
- /* whether can do without shuffling under the LRU lock */
- return gen == lru_gen_from_seq(seq);
-}
-
-#else /* !CONFIG_LRU_GEN */
-
-static bool lru_gen_clear_refs(struct folio *folio)
-{
- return false;
-}
-
-#endif /* CONFIG_LRU_GEN */
-
/**
* folio_mark_accessed - Mark a folio as having seen activity.
* @folio: The folio to mark.
@@ -554,7 +525,7 @@ static void lru_lazyfree(struct lruvec *lruvec, struct folio *folio)
lruvec_del_folio(lruvec, folio);
folio_clear_active(folio);
if (lru_gen_enabled())
- lru_gen_clear_refs(folio);
+ __folio_set_lru_refs(folio, 0);
else
folio_clear_referenced(folio);
/*
@@ -627,7 +598,7 @@ void deactivate_file_folio(struct folio *folio)
if (folio_test_unevictable(folio) || !folio_test_lru(folio))
return;
- if (lru_gen_enabled() && lru_gen_clear_refs(folio))
+ if (lru_gen_enabled() && !folio_reset_lru_refs(folio))
return;
folio_batch_add_and_move(folio, lru_deactivate_file);
@@ -646,7 +617,7 @@ void folio_deactivate(struct folio *folio)
if (folio_test_unevictable(folio) || !folio_test_lru(folio))
return;
- if (lru_gen_enabled() ? lru_gen_clear_refs(folio) : !folio_test_active(folio))
+ if (lru_gen_enabled() ? !folio_reset_lru_refs(folio) : !folio_test_active(folio))
return;
folio_batch_add_and_move(folio, lru_deactivate);
diff --git a/mm/madvise.c b/mm/madvise.c
index 2b4a8b2cb477..184611fdd6f9 100644
--- a/mm/madvise.c
+++ b/mm/madvise.c
@@ -374,7 +374,8 @@ static void madvise_cold_prep_folio(struct folio *folio)
* list should have PG_workingset so its refault is still accounted
* as a workingset refault, and the referenced bit always needs to
* be cleared even for inactive folios. Nothing to do for MGLRU as
- * folio_deactivate() always calls lru_gen_clear_refs().
+ * folio_deactivate() clears the refs count through
+ * folio_reset_lru_refs().
*/
if (!lru_gen_enabled()) {
folio_clear_referenced(folio);
@@ -395,7 +396,7 @@ static void madvise_pageout_prep_folio(struct folio *folio)
* the workingset classification is preserved for PSI.
*/
if (lru_gen_enabled())
- folio_set_lru_refs(folio, refs >= LRU_REFS_WORKINGSET ? LRU_REFS_WORKINGSET : 0);
+ __folio_set_lru_refs(folio, refs >= LRU_REFS_WORKINGSET ? LRU_REFS_WORKINGSET : 0);
}
static int madvise_cold_or_pageout_pte_range(pmd_t *pmd,
diff --git a/mm/vmscan.c b/mm/vmscan.c
index d0f0e5a25d87..617f69c1d9f9 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -977,17 +977,19 @@ enum folio_references {
void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
{
int max_gen;
- int type, refs, old_gen, gen;
+ int file, old_refs, refs, old_gen, gen;
unsigned long new_flags, old_flags, max_seq;
+ long nr_pages = folio_nr_pages(folio);
struct lru_gen_folio *lrugen;
struct lruvec *lruvec = NULL;
- type = folio_is_file_lru(folio);
old_flags = READ_ONCE(*folio_flags(folio, 0));
do {
new_flags = old_flags;
old_gen = lru_get_gen_flags(old_flags);
- refs = lru_get_refs_flags(old_flags) + 1;
+ old_refs = lru_get_refs_flags(old_flags);
+ file = folio_flags_is_file_lru(&old_flags);
+ refs = old_refs + 1;
gen = old_gen;
if (old_gen < 0)
goto out;
@@ -1013,14 +1015,14 @@ void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
if (refs > LRU_REFS_REFERENCED || flags & LRU_REF_EXEC)
gen = max_gen;
/* First access only defers eviction of a tail gen */
- else if (lru_gen_at_tail(lruvec, type, old_gen))
+ else if (lru_gen_at_tail(lruvec, file, old_gen))
gen = (old_gen + 1) % MAX_NR_GENS;
refs = min(refs, LRU_REFS_PROTECTED);
} else if (refs > LRU_REFS_MAX) {
/* LRU refs counting overflow, bump the gen */
gen = (old_gen + 1) % MAX_NR_GENS;
refs = LRU_REFS_PROTECTED;
- } else if (lru_gen_at_tail(lruvec, type, old_gen) && refs >= LRU_REFS_WORKINGSET) {
+ } else if (lru_gen_at_tail(lruvec, file, old_gen) && refs >= LRU_REFS_WORKINGSET) {
/* Defer eviction of just accessed workingset */
gen = (old_gen + 1) % MAX_NR_GENS;
refs = min(refs, LRU_REFS_PROTECTED);
@@ -1036,54 +1038,108 @@ void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
if (gen != old_gen)
lru_gen_update_size(lruvec, folio, old_gen, gen);
+ if (lru_refs_is_active(old_refs) != lru_refs_is_active(refs) && old_gen >= 0) {
+ enum lru_list lru = file * LRU_INACTIVE_FILE;
+
+ __update_lru_size(lruvec, lru + lru_refs_is_active(old_refs),
+ folio_zonenum(folio), -nr_pages);
+ __update_lru_size(lruvec, lru + lru_refs_is_active(refs),
+ folio_zonenum(folio), nr_pages);
+ }
if (lruvec)
lruvec_unlock_irq(lruvec);
}
+/*
+ * Reset the folio's lru refs indicator. The caller doesn't need to hold
+ * the folio lock, isolate the folio, or hold the lruvec lock.
+ */
+bool folio_reset_lru_refs(struct folio *folio)
+{
+ int type, gen, refs;
+ unsigned long seq, new_flags, old_flags;
+ long nr_pages = folio_nr_pages(folio);
+ struct lruvec *lruvec;
+ bool reset = false;
+
+ old_flags = READ_ONCE(*folio_flags(folio, 0));
+ do {
+ new_flags = old_flags;
+ refs = lru_get_refs_flags(old_flags);
+ type = folio_flags_is_file_lru(&old_flags);
+ gen = lru_get_gen_flags(old_flags);
+ if (!refs)
+ break;
+ lru_set_refs_flags(&new_flags, 0);
+ } while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+
+ if (gen >= 0) {
+ lruvec = folio_lruvec_live_get(folio);
+
+ /* If the folio is on list, caller might also want to demote it */
+ seq = READ_ONCE(lruvec->lrugen.min_seq[type]);
+ reset = (gen != lru_gen_from_seq(seq));
+ if (lru_refs_is_active(refs)) {
+ __update_lru_size(lruvec, type * LRU_FILE + LRU_ACTIVE,
+ folio_zonenum(folio), -nr_pages);
+ __update_lru_size(lruvec, type * LRU_FILE,
+ folio_zonenum(folio), nr_pages);
+ }
+
+ folio_lruvec_live_put(lruvec);
+ }
+
+ return reset;
+}
+
/*
* Update the folio's lru refs indicator during a page table walk or the
* look-around. max_seq can be stale as neither holds the LRU lock.
*
- * Returns the old generation and stores the new generation in @new_gen if
- * the folio is on the LRU and not in the newest generation, or -1 otherwise.
+ * Returns the old generation, or -1 if the folio is off the LRU list.
+ * Old LRU refs, and updated gen and LRU refs info by the successful
+ * cmpxchg are all stored by returning arguments.
*/
static int folio_inc_lru_refs_walk(struct folio *folio, struct lruvec *lruvec,
- const vma_flags_t *vma_flags,
- int *new_gen, int *type)
+ const vma_flags_t *vma_flags, int *new_gen,
+ int *old_refs, int *new_refs, int *type)
{
unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
- int refs, gen, max_gen, ret;
+ int refs, old_gen, max_gen;
unsigned long max_seq;
do {
max_seq = READ_ONCE(lruvec->lrugen.max_seq);
max_gen = lru_gen_from_seq(max_seq);
- gen = lru_get_gen_flags(old_flags);
+ old_gen = lru_get_gen_flags(old_flags);
refs = lru_get_refs_flags(old_flags) + 1;
*type = folio_flags_is_file_lru(&old_flags);
new_flags = old_flags;
- if (gen >= 0 && gen != max_gen) {
- ret = gen;
+ if (old_gen >= 0 && old_gen != max_gen) {
+ *new_refs = min(refs, LRU_REFS_PROTECTED);
/* Promote second page table access or executable */
if (refs > LRU_REFS_REFERENCED || is_exec_file_folio(folio, vma_flags))
*new_gen = max_gen;
/* First access only defers eviction of a tail gen */
- else if (lru_gen_at_tail(lruvec, *type, gen))
- *new_gen = (gen + 1) % MAX_NR_GENS;
+ else if (lru_gen_at_tail(lruvec, *type, old_gen))
+ *new_gen = (old_gen + 1) % MAX_NR_GENS;
else
- *new_gen = gen;
+ *new_gen = old_gen;
lru_set_gen_flags(&new_flags, *new_gen);
- lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_PROTECTED));
+ lru_set_refs_flags(&new_flags, *new_refs);
} else {
- ret = -1;
- lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_MAX));
+ *new_gen = old_gen;
+ *new_refs = min(refs, LRU_REFS_MAX);
+ lru_set_refs_flags(&new_flags, *new_refs);
}
if (new_flags == old_flags)
break;
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
- return ret;
+ *old_refs = lru_get_refs_flags(old_flags);
+
+ return old_gen;
}
/*
@@ -3537,10 +3593,11 @@ static bool positive_ctrl_err(struct ctrl_pos *sp, struct ctrl_pos *pv)
* the aging
******************************************************************************/
-static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
+static int __folio_inc_gen(struct lruvec *lruvec, struct folio *folio,
+ int old_gen, bool *increased)
{
unsigned long new_flags, old_flags = READ_ONCE(*folio_flags(folio, 0));
- int refs, new_gen;
+ int file, refs, old_refs, new_gen;
do {
new_gen = lru_get_gen_flags(old_flags);
@@ -3554,11 +3611,24 @@ static int __folio_inc_gen(struct folio *folio, int old_gen, bool *increased)
new_flags = old_flags;
new_gen = (old_gen + 1) % MAX_NR_GENS;
- refs = lru_get_refs_flags(old_flags);
+ old_refs = lru_get_refs_flags(old_flags);
+ file = folio_flags_is_file_lru(&old_flags);
+ refs = min(old_refs, LRU_REFS_WORKINGSET);
lru_set_gen_flags(&new_flags, new_gen);
- lru_set_refs_flags(&new_flags, min(refs, LRU_REFS_WORKINGSET));
+ lru_set_refs_flags(&new_flags, refs);
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
+ /* Refs capped below PROTECTED: demote if previously active. */
+ if (lru_refs_is_active(old_refs) != lru_refs_is_active(refs)) {
+ enum lru_list lru = file * LRU_INACTIVE_FILE;
+ int nr_pages = folio_nr_pages(folio);
+
+ __update_lru_size(lruvec, lru + lru_refs_is_active(old_refs),
+ folio_zonenum(folio), -nr_pages);
+ __update_lru_size(lruvec, lru + lru_refs_is_active(refs),
+ folio_zonenum(folio), nr_pages);
+ }
+
if (increased)
*increased = true;
return new_gen;
@@ -3575,7 +3645,7 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
struct lru_gen_folio *lrugen = &lruvec->lrugen;
int new_gen, old_gen = lru_gen_from_seq(lrugen->min_seq[type]);
- new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
+ new_gen = __folio_inc_gen(lruvec, folio, old_gen, &gen_increased);
if (gen_increased)
lru_gen_update_size(lruvec, folio, old_gen, new_gen);
@@ -3583,18 +3653,26 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
}
static void update_batch_size(struct lru_gen_mm_walk *walk, struct folio *folio,
- int old_gen, int new_gen, int type)
+ int old_gen, int new_gen, int old_refs, int new_refs, int type)
{
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
- VM_WARN_ON_ONCE(old_gen >= MAX_NR_GENS);
- VM_WARN_ON_ONCE(new_gen >= MAX_NR_GENS);
-
- walk->batched++;
+ /* gen counter update */
+ if (old_gen != new_gen) {
+ VM_WARN_ON_ONCE(old_gen >= MAX_NR_GENS);
+ VM_WARN_ON_ONCE(new_gen >= MAX_NR_GENS);
+ walk->batched++;
+ walk->nr_pages[old_gen][type][zone] -= delta;
+ walk->nr_pages[new_gen][type][zone] += delta;
+ }
- walk->nr_pages[old_gen][type][zone] -= delta;
- walk->nr_pages[new_gen][type][zone] += delta;
+ /* active/inactive counter update */
+ if (lru_refs_is_active(old_refs) != lru_refs_is_active(new_refs)) {
+ walk->nr_activated[type][zone] +=
+ lru_refs_is_active(new_refs) ? delta : -delta;
+ walk->batched++;
+ }
}
static void reset_batch_size(struct lru_gen_mm_walk *walk)
@@ -3606,7 +3684,6 @@ static void reset_batch_size(struct lru_gen_mm_walk *walk)
walk->batched = 0;
for_each_gen_type_zone(gen, type, zone) {
- enum lru_list lru = type * LRU_INACTIVE_FILE;
int delta = walk->nr_pages[gen][type][zone];
if (!delta)
@@ -3614,10 +3691,21 @@ static void reset_batch_size(struct lru_gen_mm_walk *walk)
walk->nr_pages[gen][type][zone] = 0;
atomic_long_add(delta, &lrugen->nr_pages[gen][type][zone]);
+ }
- if (lru_gen_is_active(lruvec, gen))
- lru += LRU_ACTIVE;
- __update_lru_size(lruvec, lru, zone, delta);
+ /* apply batched active/inactive updates */
+ for (type = 0; type < ANON_AND_FILE; type++) {
+ for (zone = 0; zone < MAX_NR_ZONES; zone++) {
+ enum lru_list lru = type * LRU_INACTIVE_FILE;
+ int delta = walk->nr_activated[type][zone];
+
+ if (delta) {
+ __update_lru_size(lruvec, lru + LRU_ACTIVE,
+ zone, delta);
+ __update_lru_size(lruvec, lru, zone, -delta);
+ walk->nr_activated[type][zone] = 0;
+ }
+ }
}
lruvec_unlock_irq(lruvec);
@@ -3772,7 +3860,7 @@ static bool suitable_to_scan(int total, int young)
static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struct *vma,
struct lruvec *lruvec, struct folio *folio, bool dirty)
{
- int new_gen, old_gen, type;
+ int new_gen, old_gen, old_refs, new_refs, type;
unsigned int flags = LRU_REF_MAPPED;
if (!folio)
@@ -3785,9 +3873,10 @@ static void walk_update_folio(struct lru_gen_mm_walk *walk, struct vm_area_struc
if (walk) {
old_gen = folio_inc_lru_refs_walk(folio, lruvec, &vma->flags,
- &new_gen, &type);
- if (old_gen >= 0 && old_gen != new_gen)
- update_batch_size(walk, folio, old_gen, new_gen, type);
+ &new_gen, &old_refs, &new_refs, &type);
+ if (old_gen >= 0)
+ update_batch_size(walk, folio, old_gen, new_gen,
+ old_refs, new_refs, type);
} else {
if (is_exec_file_folio(folio, &vma->flags))
flags |= LRU_REF_EXEC;
@@ -4146,6 +4235,7 @@ static void clear_mm_walk(void)
VM_WARN_ON_ONCE(walk && memchr_inv(walk->nr_pages, 0, sizeof(walk->nr_pages)));
VM_WARN_ON_ONCE(walk && memchr_inv(walk->mm_stats, 0, sizeof(walk->mm_stats)));
+ VM_WARN_ON_ONCE(walk && memchr_inv(walk->nr_activated, 0, sizeof(walk->nr_activated)));
current->reclaim_state->mm_walk = NULL;
@@ -4184,8 +4274,6 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
goto done;
VM_WARN_ON_ONCE(get_nr_gens(lruvec, type) != MAX_NR_GENS);
- VM_WARN_ON_ONCE(lru_gen_is_active(lruvec, old_gen) !=
- lru_gen_is_active(lruvec, target_gen));
/* prevent cold/hot inversion if the type is evictable */
for (zone = 0; zone < MAX_NR_ZONES; zone++) {
struct list_head *target_list = &lrugen->folios[target_gen][type][zone];
@@ -4208,7 +4296,7 @@ static bool inc_min_seq(struct lruvec *lruvec, int type, int swappiness)
prefetchw_next_lru_folio(folio, head);
pos = pos->next;
- new_gen = __folio_inc_gen(folio, old_gen, &gen_increased);
+ new_gen = __folio_inc_gen(lruvec, folio, old_gen, &gen_increased);
/*
* If gen_increased is false, this is a promotion. Put folios
* at the head of the promoted gen. Otherwise, put them at
@@ -4295,8 +4383,7 @@ static void try_to_inc_min_seq(struct lruvec *lruvec, int swappiness)
static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness)
{
bool success;
- int prev, next;
- int type, zone;
+ int type, next;
struct lru_gen_folio *lrugen = &lruvec->lrugen;
restart:
if (seq < READ_ONCE(lrugen->max_seq))
@@ -4322,32 +4409,10 @@ static bool inc_max_seq(struct lruvec *lruvec, unsigned long seq, int swappiness
goto restart;
}
- /*
- * Update the active/inactive LRU sizes for compatibility. Both sides of
- * the current max_seq need to be covered, since max_seq+1 can overlap
- * with min_seq[LRU_GEN_ANON] if swapping is constrained. And if they do
- * overlap, cold/hot inversion happens.
- */
- prev = lru_gen_from_seq(lrugen->max_seq - 1);
- next = lru_gen_from_seq(lrugen->max_seq + 1);
-
- for (type = 0; type < ANON_AND_FILE; type++) {
- for (zone = 0; zone < MAX_NR_ZONES; zone++) {
- enum lru_list lru = type * LRU_INACTIVE_FILE;
- long delta = atomic_long_read(&lrugen->nr_pages[prev][type][zone]) -
- atomic_long_read(&lrugen->nr_pages[next][type][zone]);
-
- if (!delta)
- continue;
-
- __update_lru_size(lruvec, lru, zone, delta);
- __update_lru_size(lruvec, lru + LRU_ACTIVE, zone, -delta);
- }
- }
-
for (type = 0; type < ANON_AND_FILE; type++)
reset_ctrl_pos(lruvec, type, false);
+ next = lru_gen_from_seq(lrugen->max_seq + 1);
WRITE_ONCE(lrugen->timestamps[next], jiffies);
/* make sure preceding modifications appear */
smp_store_release(&lrugen->max_seq, lrugen->max_seq + 1);
@@ -4866,7 +4931,6 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
int zone, int type)
{
struct lru_gen_folio *child_lrugen, *parent_lrugen;
- enum lru_list lru = type * LRU_INACTIVE_FILE;
int i;
child_lrugen = &child_lruvec->lrugen;
@@ -4875,8 +4939,6 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
for (i = 0; i < get_nr_gens(child_lruvec, type); i++) {
int gen = lru_gen_from_seq(child_lrugen->max_seq - i);
long nr_pages = atomic_long_read(&child_lrugen->nr_pages[gen][type][zone]);
- int child_lru_active = lru_gen_is_active(child_lruvec, gen) ? LRU_ACTIVE : 0;
- int parent_lru_active = lru_gen_is_active(parent_lruvec, gen) ? LRU_ACTIVE : 0;
/* Assuming that child pages are colder than parent pages */
list_splice_tail_init(&child_lrugen->folios[gen][type][zone],
@@ -4884,11 +4946,6 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
atomic_long_set(&child_lrugen->nr_pages[gen][type][zone], 0);
atomic_long_add(nr_pages, &parent_lrugen->nr_pages[gen][type][zone]);
-
- if (lru_gen_is_active(child_lruvec, gen) != lru_gen_is_active(parent_lruvec, gen)) {
- __update_lru_size(child_lruvec, lru + child_lru_active, zone, -nr_pages);
- __update_lru_size(parent_lruvec, lru + parent_lru_active, zone, nr_pages);
- }
}
}
@@ -5228,7 +5285,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
* its first page table access lands in the second newest one.
* See "Referenced count feedback" above.
*/
- folio_set_lru_refs(folio, min(folio_lru_refs(folio), LRU_REFS_WORKINGSET));
+ __folio_set_lru_refs(folio, min(folio_lru_refs(folio), LRU_REFS_WORKINGSET));
if (lru_gen_folio_seq(lruvec, folio, false) == min_seq[type])
folio_set_active(folio);
}
diff --git a/mm/workingset.c b/mm/workingset.c
index 16cdd88e275e..7681af535ca7 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -334,7 +334,7 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
}
/* Refault is also promotion, cap the refs like folio_inc_lru_refs */
- folio_set_lru_refs(folio, min(refs, LRU_REFS_PROTECTED));
+ __folio_set_lru_refs(folio, min(refs, LRU_REFS_PROTECTED));
}
if (refs >= LRU_REFS_WORKINGSET)
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 13/17] mm/mglru: folio LRU refs based active/inactive number accounting
2026-10-15 18:16 ` Kairui Song
(?)
@ 2026-10-15 18:34 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:34 UTC (permalink / raw)
To: Kairui Song; +Cc: damon, tj, hannes, mkoutny
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] mm: MGLRU tier bits (LRU_REFS_FLAGS) not cleared when updating LRU_GEN_MASK
--
commit 97ee7d2a683879e87f8e211218beb97d5cbbba1b
Author: Kairui Song <kasong@tencent.com>
mm/mglru: folio LRU refs based active/inactive number accounting
This commit updates the active/inactive LRU size tracking to use
per-folio referenced counts instead of the generation window. Folios
are now accounted as active once their references reach LRU_REFS_ACTIVATED.
> diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
> index 6c9510002754b..41c2e9150c709 100644
> --- a/include/linux/mm_inline.h
> +++ b/include/linux/mm_inline.h
[ ... ]
> @@ -374,7 +373,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
> struct lru_gen_folio *lrugen = &lruvec->lrugen;
>
> BUILD_BUG_ON(BIT(LRU_GEN_WIDTH - 1) != MAX_NR_GENS);
> - VM_WARN_ON_ONCE_FOLIO(gen != -1, folio);
> + VM_WARN_ON_ONCE_FOLIO(folio_lru_gen(folio) != -1, folio);
>
> if (folio_test_unevictable(folio) || !lrugen->enabled)
> return false;
> @@ -383,10 +382,12 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
> gen = lru_gen_from_seq(seq);
> flags = (gen + 1UL) << LRU_GEN_PGOFF;
> /* see the comment on MIN_NR_GENS about PG_active */
> - set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
> + flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK | BIT(PG_active), flags);
[Severity: Medium]
Should LRU_REFS_FLAGS be included in the mask cleared by set_mask_bits() here?
When a folio is added to a generation and LRU_GEN_MASK is modified, leaving
stale tier bits (LRU_REFS_FLAGS) intact can allow them to persist. This
could inadvertently inflate access counts and distort future eviction tracking.
> + /* use the refs from the atomic snapshot to avoid raced update */
> + refs = lru_get_refs_flags(flags);
>
> lru_gen_update_size(lruvec, folio, -1, gen);
[ ... ]
> @@ -402,24 +403,31 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
> static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio, bool reclaiming)
> {
> unsigned long flags;
> - int gen = folio_lru_gen(folio);
> + int gen, refs;
> int zone = folio_zonenum(folio);
> int delta = folio_nr_pages(folio);
> enum lru_list lru = folio_is_file_lru(folio) * LRU_INACTIVE_FILE;
> + unsigned long max_seq = READ_ONCE(lruvec->lrugen.max_seq);
>
> + flags = set_mask_bits(folio_flags(folio, 0), LRU_GEN_MASK, 0);
[Severity: Medium]
Could this mask also be missing the LRU_REFS_FLAGS bit clear?
Similar to the addition path above, when isolating a folio and clearing
LRU_GEN_MASK, the tier tracking bits might also need to be explicitly
cleared so they are not improperly carried over.
> + gen = lru_get_gen_flags(flags);
> + refs = lru_get_refs_flags(flags);
> if (gen < 0)
> return false;
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=13
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 14/17] mm/mglru: reparent folios from all generations
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:16 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
A lockless promotion racing with aging can reuse a generation index,
see folio_inc_lru_refs(). The retry repairs the folio flags, but
sort_folio() may have already moved the folio to the list of a
generation outside the [min_seq, max_seq] window.
__lru_gen_reparent_memcg() only splices the lists inside the child's
window, so such a folio would be left on a list of the dying memcg.
Splice the lists and move the counters of all MAX_NR_GENS generations
instead. Those outside the window are empty in the common case.
Assisted-by: LLM
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/vmscan.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 617f69c1d9f9..a9c224377234 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4936,7 +4936,8 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
child_lrugen = &child_lruvec->lrugen;
parent_lrugen = &parent_lruvec->lrugen;
- for (i = 0; i < get_nr_gens(child_lruvec, type); i++) {
+ /* Lockless promotion may leave folios outside the child's window */
+ for (i = 0; i < MAX_NR_GENS; i++) {
int gen = lru_gen_from_seq(child_lrugen->max_seq - i);
long nr_pages = atomic_long_read(&child_lrugen->nr_pages[gen][type][zone]);
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 14/17] mm/mglru: reparent folios from all generations
@ 2026-10-15 18:16 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
A lockless promotion racing with aging can reuse a generation index,
see folio_inc_lru_refs(). The retry repairs the folio flags, but
sort_folio() may have already moved the folio to the list of a
generation outside the [min_seq, max_seq] window.
__lru_gen_reparent_memcg() only splices the lists inside the child's
window, so such a folio would be left on a list of the dying memcg.
Splice the lists and move the counters of all MAX_NR_GENS generations
instead. Those outside the window are empty in the common case.
Assisted-by: LLM
Signed-off-by: Kairui Song <kasong@tencent.com>
---
mm/vmscan.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/mm/vmscan.c b/mm/vmscan.c
index 617f69c1d9f9..a9c224377234 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4936,7 +4936,8 @@ static void __lru_gen_reparent_memcg(struct lruvec *child_lruvec, struct lruvec
child_lrugen = &child_lruvec->lrugen;
parent_lrugen = &parent_lruvec->lrugen;
- for (i = 0; i < get_nr_gens(child_lruvec, type); i++) {
+ /* Lockless promotion may leave folios outside the child's window */
+ for (i = 0; i < MAX_NR_GENS; i++) {
int gen = lru_gen_from_seq(child_lrugen->max_seq - i);
long nr_pages = atomic_long_read(&child_lrugen->nr_pages[gen][type][zone]);
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 14/17] mm/mglru: reparent folios from all generations
2026-10-15 18:16 ` Kairui Song
(?)
@ 2026-10-15 18:27 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:27 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> A lockless promotion racing with aging can reuse a generation index,
> see folio_inc_lru_refs(). The retry repairs the folio flags, but
> sort_folio() may have already moved the folio to the list of a
> generation outside the [min_seq, max_seq] window.
>
> __lru_gen_reparent_memcg() only splices the lists inside the child's
> window, so such a folio would be left on a list of the dying memcg.
> Splice the lists and move the counters of all MAX_NR_GENS generations
> instead. Those outside the window are empty in the common case.
>
> Assisted-by: LLM
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=14
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 15/17] mm/mglru: make folio_inc_lru_refs lruvec lockless
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:16 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
The lruvec spinlock in folio_inc_lru_refs was only taken to keep
concurrent aging from corrupting the size counters. This is no
longer needed: the per-generation counters are atomic, and the
active/inactive ABI counters are updated per folio and linearized by
the CAS on the folio flags, so concurrent aging can no longer corrupt
them. The same argument already allowed folio_reset_lru_refs() to
drop the lock.
Use folio_lruvec_live_get()/folio_lruvec_live_put() to hold the RCU
read lock around the lruvec lookup, and drop the spinlock entirely.
Without the lock, aging can advance max_seq while a promotion is in
flight. Generations are indexed by seq % MAX_NR_GENS, so the folio
flags can return to the value read earlier and the cmpxchg succeeds
with a target gen computed from a stale max_seq. Recheck max_seq after
a gen-changing cmpxchg: if it moved, account the committed update and
retry with LRU_REF_FORCE, which promotes the folio to the newest gen.
The retry may count the access twice, which is acceptable and avoids
folio_activate(), whose LRU lock causes latency jitter. smp_rmb()
orders the flags read, including the one returned by a failed
cmpxchg(), before the max_seq read.
Also simplify the post-CAS accounting guards: gen is only assigned
non-negative values after the old_gen < 0 early exit, so gen != old_gen
implies gen >= 0, and lru_gen_update_size() needs no explicit guard.
The active/inactive ABI update keeps its gen >= 0 check because an
off-LRU folio has no lruvec to update.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 10 ++++-----
include/linux/mmzone.h | 3 ++-
mm/vmscan.c | 53 ++++++++++++++++++++++++++++++-----------------
3 files changed, 41 insertions(+), 25 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 41c2e9150c70..fd77dbfdaf70 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -308,10 +308,9 @@ static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
gen == lru_gen_from_seq(min_seq + 1);
}
-static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *folio,
- int old_gen, int new_gen)
+static inline void lru_gen_update_size(struct lruvec *lruvec, int type,
+ struct folio *folio, int old_gen, int new_gen)
{
- int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
struct lru_gen_folio *lrugen = &lruvec->lrugen;
@@ -386,7 +385,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
/* use the refs from the atomic snapshot to avoid raced update */
refs = lru_get_refs_flags(flags);
- lru_gen_update_size(lruvec, folio, -1, gen);
+ lru_gen_update_size(lruvec, type, folio, -1, gen);
if (lru_refs_is_active(refs))
lru += LRU_ACTIVE;
__update_lru_size(lruvec, lru, zone, delta);
@@ -404,6 +403,7 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
{
unsigned long flags;
int gen, refs;
+ int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
enum lru_list lru = folio_is_file_lru(folio) * LRU_INACTIVE_FILE;
@@ -426,7 +426,7 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
if (!reclaiming && ((max_seq - gen) % MAX_NR_GENS) < MIN_NR_GENS)
folio_set_active(folio);
- lru_gen_update_size(lruvec, folio, gen, -1);
+ lru_gen_update_size(lruvec, type, folio, gen, -1);
if (lru_refs_is_active(refs))
lru += LRU_ACTIVE;
__update_lru_size(lruvec, lru, zone, -delta);
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index dacf066b499e..f50449d10242 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -559,9 +559,10 @@ enum lruvec_flags {
#define LRU_TIER_MIN 0U
#define LRU_TIER_MAX (MAX_NR_TIERS - 1)
-/* Access source flags for folio_inc_lru_refs() */
+/* Flags for folio_inc_lru_refs() */
#define LRU_REF_MAPPED 0x1U
#define LRU_REF_EXEC 0x2U
+#define LRU_REF_FORCE 0x4U
#define LRU_REFS_REFERENCED 0x1
#define LRU_REFS_WORKINGSET 0x2
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a9c224377234..e604b0893847 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -966,6 +966,9 @@ enum folio_references {
* LRU_REFS_PROTECTED, so it starts at or below the protected tier regardless
* of its old-generation access history. PID protection (folio_inc_gen) caps
* at LRU_REFS_WORKINGSET independently.
+ *
+ * folio_inc_lru_refs() retries with LRU_REF_FORCE if the aging recycled the
+ * gen index it was about to store: newest gen and an extra reference.
*/
/*
@@ -982,37 +985,35 @@ void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
long nr_pages = folio_nr_pages(folio);
struct lru_gen_folio *lrugen;
struct lruvec *lruvec = NULL;
+ bool aged = false;
+retry:
old_flags = READ_ONCE(*folio_flags(folio, 0));
do {
new_flags = old_flags;
old_gen = lru_get_gen_flags(old_flags);
old_refs = lru_get_refs_flags(old_flags);
- file = folio_flags_is_file_lru(&old_flags);
refs = old_refs + 1;
gen = old_gen;
+ file = folio_flags_is_file_lru(&old_flags);
if (old_gen < 0)
goto out;
- /*
- * Lock the lruvec if the folio is on-list. We are already
- * doing lazy promotion so in theory we don't need this,
- * but for now, concurrent aging would still corrupt the
- * size counters. This is a temporary limitation and
- * will be lifted very soon, so the lock here is not a
- * performance concern.
- */
if (!lruvec) {
- lruvec = lruvec_live_lock_irq(folio_lruvec(folio));
+ lruvec = folio_lruvec_live_get(folio);
lrugen = &lruvec->lrugen;
}
+ /*
+ * Read the flags before max_seq, paired with the store of a
+ * gen, which samples max_seq first.
+ */
+ smp_rmb();
max_seq = READ_ONCE(lrugen->max_seq);
max_gen = lru_gen_from_seq(max_seq);
if (old_gen == max_gen)
goto out;
-
- if (flags & (LRU_REF_MAPPED | LRU_REF_EXEC)) {
- /* Promote second page table access or executable */
- if (refs > LRU_REFS_REFERENCED || flags & LRU_REF_EXEC)
+ if (flags & (LRU_REF_MAPPED | LRU_REF_EXEC | LRU_REF_FORCE)) {
+ if (refs > LRU_REFS_REFERENCED ||
+ flags & (LRU_REF_EXEC | LRU_REF_FORCE))
gen = max_gen;
/* First access only defers eviction of a tail gen */
else if (lru_gen_at_tail(lruvec, file, old_gen))
@@ -1036,9 +1037,19 @@ void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
break;
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
- if (gen != old_gen)
- lru_gen_update_size(lruvec, folio, old_gen, gen);
- if (lru_refs_is_active(old_refs) != lru_refs_is_active(refs) && old_gen >= 0) {
+ if (gen != old_gen) {
+ /*
+ * Gen-index reuse can fool cmpxchg(). Retry locklessly, accepting
+ * an extra reference to avoid folio_activate() latency.
+ */
+ if (unlikely(READ_ONCE(lrugen->max_seq) != max_seq)) {
+ flags = LRU_REF_FORCE;
+ aged = true;
+ }
+ lru_gen_update_size(lruvec, file, folio, old_gen, gen);
+ }
+
+ if (lru_refs_is_active(old_refs) != lru_refs_is_active(refs) && gen >= 0) {
enum lru_list lru = file * LRU_INACTIVE_FILE;
__update_lru_size(lruvec, lru + lru_refs_is_active(old_refs),
@@ -1046,8 +1057,12 @@ void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
__update_lru_size(lruvec, lru + lru_refs_is_active(refs),
folio_zonenum(folio), nr_pages);
}
+ if (aged) {
+ aged = false;
+ goto retry;
+ }
if (lruvec)
- lruvec_unlock_irq(lruvec);
+ folio_lruvec_live_put(lruvec);
}
/*
@@ -3647,7 +3662,7 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
new_gen = __folio_inc_gen(lruvec, folio, old_gen, &gen_increased);
if (gen_increased)
- lru_gen_update_size(lruvec, folio, old_gen, new_gen);
+ lru_gen_update_size(lruvec, type, folio, old_gen, new_gen);
return new_gen;
}
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 15/17] mm/mglru: make folio_inc_lru_refs lruvec lockless
@ 2026-10-15 18:16 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
The lruvec spinlock in folio_inc_lru_refs was only taken to keep
concurrent aging from corrupting the size counters. This is no
longer needed: the per-generation counters are atomic, and the
active/inactive ABI counters are updated per folio and linearized by
the CAS on the folio flags, so concurrent aging can no longer corrupt
them. The same argument already allowed folio_reset_lru_refs() to
drop the lock.
Use folio_lruvec_live_get()/folio_lruvec_live_put() to hold the RCU
read lock around the lruvec lookup, and drop the spinlock entirely.
Without the lock, aging can advance max_seq while a promotion is in
flight. Generations are indexed by seq % MAX_NR_GENS, so the folio
flags can return to the value read earlier and the cmpxchg succeeds
with a target gen computed from a stale max_seq. Recheck max_seq after
a gen-changing cmpxchg: if it moved, account the committed update and
retry with LRU_REF_FORCE, which promotes the folio to the newest gen.
The retry may count the access twice, which is acceptable and avoids
folio_activate(), whose LRU lock causes latency jitter. smp_rmb()
orders the flags read, including the one returned by a failed
cmpxchg(), before the max_seq read.
Also simplify the post-CAS accounting guards: gen is only assigned
non-negative values after the old_gen < 0 early exit, so gen != old_gen
implies gen >= 0, and lru_gen_update_size() needs no explicit guard.
The active/inactive ABI update keeps its gen >= 0 check because an
off-LRU folio has no lruvec to update.
Signed-off-by: Kairui Song <kasong@tencent.com>
---
include/linux/mm_inline.h | 10 ++++-----
include/linux/mmzone.h | 3 ++-
mm/vmscan.c | 53 ++++++++++++++++++++++++++++++-----------------
3 files changed, 41 insertions(+), 25 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index 41c2e9150c70..fd77dbfdaf70 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -308,10 +308,9 @@ static inline bool lru_gen_at_tail(struct lruvec *lruvec, int type, int gen)
gen == lru_gen_from_seq(min_seq + 1);
}
-static inline void lru_gen_update_size(struct lruvec *lruvec, struct folio *folio,
- int old_gen, int new_gen)
+static inline void lru_gen_update_size(struct lruvec *lruvec, int type,
+ struct folio *folio, int old_gen, int new_gen)
{
- int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
struct lru_gen_folio *lrugen = &lruvec->lrugen;
@@ -386,7 +385,7 @@ static inline bool lru_gen_add_folio(struct lruvec *lruvec, struct folio *folio,
/* use the refs from the atomic snapshot to avoid raced update */
refs = lru_get_refs_flags(flags);
- lru_gen_update_size(lruvec, folio, -1, gen);
+ lru_gen_update_size(lruvec, type, folio, -1, gen);
if (lru_refs_is_active(refs))
lru += LRU_ACTIVE;
__update_lru_size(lruvec, lru, zone, delta);
@@ -404,6 +403,7 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
{
unsigned long flags;
int gen, refs;
+ int type = folio_is_file_lru(folio);
int zone = folio_zonenum(folio);
int delta = folio_nr_pages(folio);
enum lru_list lru = folio_is_file_lru(folio) * LRU_INACTIVE_FILE;
@@ -426,7 +426,7 @@ static inline bool lru_gen_del_folio(struct lruvec *lruvec, struct folio *folio,
if (!reclaiming && ((max_seq - gen) % MAX_NR_GENS) < MIN_NR_GENS)
folio_set_active(folio);
- lru_gen_update_size(lruvec, folio, gen, -1);
+ lru_gen_update_size(lruvec, type, folio, gen, -1);
if (lru_refs_is_active(refs))
lru += LRU_ACTIVE;
__update_lru_size(lruvec, lru, zone, -delta);
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index dacf066b499e..f50449d10242 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -559,9 +559,10 @@ enum lruvec_flags {
#define LRU_TIER_MIN 0U
#define LRU_TIER_MAX (MAX_NR_TIERS - 1)
-/* Access source flags for folio_inc_lru_refs() */
+/* Flags for folio_inc_lru_refs() */
#define LRU_REF_MAPPED 0x1U
#define LRU_REF_EXEC 0x2U
+#define LRU_REF_FORCE 0x4U
#define LRU_REFS_REFERENCED 0x1
#define LRU_REFS_WORKINGSET 0x2
diff --git a/mm/vmscan.c b/mm/vmscan.c
index a9c224377234..e604b0893847 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -966,6 +966,9 @@ enum folio_references {
* LRU_REFS_PROTECTED, so it starts at or below the protected tier regardless
* of its old-generation access history. PID protection (folio_inc_gen) caps
* at LRU_REFS_WORKINGSET independently.
+ *
+ * folio_inc_lru_refs() retries with LRU_REF_FORCE if the aging recycled the
+ * gen index it was about to store: newest gen and an extra reference.
*/
/*
@@ -982,37 +985,35 @@ void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
long nr_pages = folio_nr_pages(folio);
struct lru_gen_folio *lrugen;
struct lruvec *lruvec = NULL;
+ bool aged = false;
+retry:
old_flags = READ_ONCE(*folio_flags(folio, 0));
do {
new_flags = old_flags;
old_gen = lru_get_gen_flags(old_flags);
old_refs = lru_get_refs_flags(old_flags);
- file = folio_flags_is_file_lru(&old_flags);
refs = old_refs + 1;
gen = old_gen;
+ file = folio_flags_is_file_lru(&old_flags);
if (old_gen < 0)
goto out;
- /*
- * Lock the lruvec if the folio is on-list. We are already
- * doing lazy promotion so in theory we don't need this,
- * but for now, concurrent aging would still corrupt the
- * size counters. This is a temporary limitation and
- * will be lifted very soon, so the lock here is not a
- * performance concern.
- */
if (!lruvec) {
- lruvec = lruvec_live_lock_irq(folio_lruvec(folio));
+ lruvec = folio_lruvec_live_get(folio);
lrugen = &lruvec->lrugen;
}
+ /*
+ * Read the flags before max_seq, paired with the store of a
+ * gen, which samples max_seq first.
+ */
+ smp_rmb();
max_seq = READ_ONCE(lrugen->max_seq);
max_gen = lru_gen_from_seq(max_seq);
if (old_gen == max_gen)
goto out;
-
- if (flags & (LRU_REF_MAPPED | LRU_REF_EXEC)) {
- /* Promote second page table access or executable */
- if (refs > LRU_REFS_REFERENCED || flags & LRU_REF_EXEC)
+ if (flags & (LRU_REF_MAPPED | LRU_REF_EXEC | LRU_REF_FORCE)) {
+ if (refs > LRU_REFS_REFERENCED ||
+ flags & (LRU_REF_EXEC | LRU_REF_FORCE))
gen = max_gen;
/* First access only defers eviction of a tail gen */
else if (lru_gen_at_tail(lruvec, file, old_gen))
@@ -1036,9 +1037,19 @@ void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
break;
} while (!try_cmpxchg(folio_flags(folio, 0), &old_flags, new_flags));
- if (gen != old_gen)
- lru_gen_update_size(lruvec, folio, old_gen, gen);
- if (lru_refs_is_active(old_refs) != lru_refs_is_active(refs) && old_gen >= 0) {
+ if (gen != old_gen) {
+ /*
+ * Gen-index reuse can fool cmpxchg(). Retry locklessly, accepting
+ * an extra reference to avoid folio_activate() latency.
+ */
+ if (unlikely(READ_ONCE(lrugen->max_seq) != max_seq)) {
+ flags = LRU_REF_FORCE;
+ aged = true;
+ }
+ lru_gen_update_size(lruvec, file, folio, old_gen, gen);
+ }
+
+ if (lru_refs_is_active(old_refs) != lru_refs_is_active(refs) && gen >= 0) {
enum lru_list lru = file * LRU_INACTIVE_FILE;
__update_lru_size(lruvec, lru + lru_refs_is_active(old_refs),
@@ -1046,8 +1057,12 @@ void folio_inc_lru_refs(struct folio *folio, unsigned int flags)
__update_lru_size(lruvec, lru + lru_refs_is_active(refs),
folio_zonenum(folio), nr_pages);
}
+ if (aged) {
+ aged = false;
+ goto retry;
+ }
if (lruvec)
- lruvec_unlock_irq(lruvec);
+ folio_lruvec_live_put(lruvec);
}
/*
@@ -3647,7 +3662,7 @@ static int folio_inc_gen(struct lruvec *lruvec, struct folio *folio)
new_gen = __folio_inc_gen(lruvec, folio, old_gen, &gen_increased);
if (gen_increased)
- lru_gen_update_size(lruvec, folio, old_gen, new_gen);
+ lru_gen_update_size(lruvec, type, folio, old_gen, new_gen);
return new_gen;
}
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 15/17] mm/mglru: make folio_inc_lru_refs lruvec lockless
2026-10-15 18:16 ` Kairui Song
(?)
@ 2026-10-15 18:29 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:29 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> The lruvec spinlock in folio_inc_lru_refs was only taken to keep
> concurrent aging from corrupting the size counters. This is no
> longer needed: the per-generation counters are atomic, and the
> active/inactive ABI counters are updated per folio and linearized by
> the CAS on the folio flags, so concurrent aging can no longer corrupt
> them. The same argument already allowed folio_reset_lru_refs() to
> drop the lock.
>
> Use folio_lruvec_live_get()/folio_lruvec_live_put() to hold the RCU
> read lock around the lruvec lookup, and drop the spinlock entirely.
>
> Without the lock, aging can advance max_seq while a promotion is in
> flight. Generations are indexed by seq % MAX_NR_GENS, so the folio
> [ ... ]
> implies gen >= 0, and lru_gen_update_size() needs no explicit guard.
> The active/inactive ABI update keeps its gen >= 0 check because an
> off-LRU folio has no lruvec to update.
>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=15
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 16/17] mm/mglru: make folio_test_workingset() work based on folio LRU refs
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:16 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
folio_test_workingset() currently tests the PG_workingset bit, which
is only the second lowest access bit of the folio LRU refs count now.
Folios with a higher refs count carry it in the LRU_REFS_MASK field
and no longer have PG_workingset set, so the bit test misses them.
folio_set_workingset() currently sets the PG_workingset bit with a
plain set_bit(), which also over-promotes folios at a higher refs
count: e.g. a folio at refs 4 is bumped to refs 6, advancing it
toward LRU_REFS_MAX and an unintended generation promotion on the
next access.
Move the test into mm_inline.h as an inline helper based on
folio_lru_refs(), checking refs >= LRU_REFS_WORKINGSET. The set
stays a plain PG_workingset bit operation reserved for the
classical LRU: under MGLRU the refs count is maintained by
folio_inc_lru_refs(), and now triggers a debug WARN when called
while MGLRU is fully on (the switching window is exempt, as the
classical paths legitimately run alongside MGLRU then). The one
PageWorkingset() user in erofs is converted to the folio helper.
Under the classical LRU the refs count is not maintained, so the
test falls back to the PG_workingset bit test, which is the old
behavior.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
fs/btrfs/compression.c | 1 +
fs/erofs/zdata.c | 3 ++-
include/linux/mm_inline.h | 29 +++++++++++++++++++++++++++++
include/linux/page-flags.h | 2 --
mm/filemap.c | 1 +
mm/page_io.c | 1 +
6 files changed, 34 insertions(+), 3 deletions(-)
diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
index c62b5148d5ac..57d24413265e 100644
--- a/fs/btrfs/compression.c
+++ b/fs/btrfs/compression.c
@@ -8,6 +8,7 @@
#include <linux/file.h>
#include <linux/fs.h>
#include <linux/pagemap.h>
+#include <linux/mm_inline.h>
#include <linux/folio_batch.h>
#include <linux/highmem.h>
#include <linux/kthread.h>
diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c
index e981e371d6c2..34ee41e1ccdc 100644
--- a/fs/erofs/zdata.c
+++ b/fs/erofs/zdata.c
@@ -6,6 +6,7 @@
*/
#include "compress.h"
#include <linux/psi.h>
+#include <linux/mm_inline.h>
#include <linux/cpuhotplug.h>
#include <trace/events/erofs.h>
@@ -1713,7 +1714,7 @@ static void z_erofs_submit_queue(struct z_erofs_frontend *f,
DBG_BUGON(bvec.bv_len < sb->s_blocksize);
}
- if (unlikely(PageWorkingset(bvec.bv_page)) &&
+ if (unlikely(folio_test_workingset(page_folio(bvec.bv_page))) &&
!memstall) {
psi_memstall_enter(&pflags);
memstall = 1;
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index fd77dbfdaf70..ae50f9880da8 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -288,6 +288,23 @@ static inline void folio_migrate_lru_refs(struct folio *new, const struct folio
__folio_set_lru_refs(new, folio_lru_refs(old));
}
+/**
+ * folio_test_workingset - Test if a folio is in the workingset.
+ * @folio: the folio
+ *
+ * A folio is workingset when its LRU refs count reaches
+ * LRU_REFS_WORKINGSET. Under the classical LRU the extra refs counter
+ * are always zero so this is just testing the PG_workingset bit.
+ * NOTE: folio_set_workingset() must not be used under MGLRU, as the
+ * folio refs are tracked by folio_inc_lru_refs() with rules.
+ *
+ * Return: true if the folio is workingset.
+ */
+static __always_inline bool folio_test_workingset(const struct folio *folio)
+{
+ return folio_lru_refs(folio) >= LRU_REFS_WORKINGSET;
+}
+
#ifdef CONFIG_LRU_GEN
static inline bool lru_refs_is_active(unsigned int refs)
{
@@ -519,6 +536,18 @@ static inline void folio_clear_lru_refs(struct folio *folio)
folio_clear_referenced(folio);
}
+/*
+ * For the classical LRU only: with MGLRU the PG_workingset bit is
+ * part of the folio refs count maintained by folio_inc_lru_refs(),
+ * and a raw set would corrupt it. The switching window is exempt
+ * because the classical paths legitimately run alongside MGLRU then.
+ */
+static __always_inline void folio_set_workingset(struct folio *folio)
+{
+ VM_WARN_ON_ONCE(lru_gen_enabled() && !lru_gen_switching());
+ set_bit(PG_workingset, folio_flags(folio, FOLIO_HEAD_PAGE));
+}
+
static __always_inline
void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio)
{
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index b0ddc652e76c..f6625dc042da 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -549,8 +549,6 @@ PAGEFLAG(LRU, lru, PF_HEAD) __CLEARPAGEFLAG(LRU, lru, PF_HEAD)
FOLIO_FLAG(active, FOLIO_HEAD_PAGE)
__FOLIO_CLEAR_FLAG(active, FOLIO_HEAD_PAGE)
FOLIO_TEST_CLEAR_FLAG(active, FOLIO_HEAD_PAGE)
-PAGEFLAG(Workingset, workingset, PF_HEAD)
- TESTCLEARFLAG(Workingset, workingset, PF_HEAD)
PAGEFLAG(Checked, checked, PF_NO_COMPOUND) /* Used by some filesystems */
/* Xen */
diff --git a/mm/filemap.c b/mm/filemap.c
index b74bc1e5015c..49e827058e14 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -21,6 +21,7 @@
#include <linux/gfp.h>
#include <linux/mm.h>
#include <linux/swap.h>
+#include <linux/mm_inline.h>
#include <linux/leafops.h>
#include <linux/syscalls.h>
#include <linux/mman.h>
diff --git a/mm/page_io.c b/mm/page_io.c
index c6824fcd483e..540ae8caa932 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -16,6 +16,7 @@
#include <linux/gfp.h>
#include <linux/pagemap.h>
#include <linux/swap.h>
+#include <linux/mm_inline.h>
#include <linux/bio.h>
#include <linux/swapops.h>
#include <linux/writeback.h>
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* [PATCH v4 16/17] mm/mglru: make folio_test_workingset() work based on folio LRU refs
@ 2026-10-15 18:16 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
folio_test_workingset() currently tests the PG_workingset bit, which
is only the second lowest access bit of the folio LRU refs count now.
Folios with a higher refs count carry it in the LRU_REFS_MASK field
and no longer have PG_workingset set, so the bit test misses them.
folio_set_workingset() currently sets the PG_workingset bit with a
plain set_bit(), which also over-promotes folios at a higher refs
count: e.g. a folio at refs 4 is bumped to refs 6, advancing it
toward LRU_REFS_MAX and an unintended generation promotion on the
next access.
Move the test into mm_inline.h as an inline helper based on
folio_lru_refs(), checking refs >= LRU_REFS_WORKINGSET. The set
stays a plain PG_workingset bit operation reserved for the
classical LRU: under MGLRU the refs count is maintained by
folio_inc_lru_refs(), and now triggers a debug WARN when called
while MGLRU is fully on (the switching window is exempt, as the
classical paths legitimately run alongside MGLRU then). The one
PageWorkingset() user in erofs is converted to the folio helper.
Under the classical LRU the refs count is not maintained, so the
test falls back to the PG_workingset bit test, which is the old
behavior.
Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
Signed-off-by: Kairui Song <kasong@tencent.com>
---
fs/btrfs/compression.c | 1 +
fs/erofs/zdata.c | 3 ++-
include/linux/mm_inline.h | 29 +++++++++++++++++++++++++++++
include/linux/page-flags.h | 2 --
mm/filemap.c | 1 +
mm/page_io.c | 1 +
6 files changed, 34 insertions(+), 3 deletions(-)
diff --git a/fs/btrfs/compression.c b/fs/btrfs/compression.c
index c62b5148d5ac..57d24413265e 100644
--- a/fs/btrfs/compression.c
+++ b/fs/btrfs/compression.c
@@ -8,6 +8,7 @@
#include <linux/file.h>
#include <linux/fs.h>
#include <linux/pagemap.h>
+#include <linux/mm_inline.h>
#include <linux/folio_batch.h>
#include <linux/highmem.h>
#include <linux/kthread.h>
diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c
index e981e371d6c2..34ee41e1ccdc 100644
--- a/fs/erofs/zdata.c
+++ b/fs/erofs/zdata.c
@@ -6,6 +6,7 @@
*/
#include "compress.h"
#include <linux/psi.h>
+#include <linux/mm_inline.h>
#include <linux/cpuhotplug.h>
#include <trace/events/erofs.h>
@@ -1713,7 +1714,7 @@ static void z_erofs_submit_queue(struct z_erofs_frontend *f,
DBG_BUGON(bvec.bv_len < sb->s_blocksize);
}
- if (unlikely(PageWorkingset(bvec.bv_page)) &&
+ if (unlikely(folio_test_workingset(page_folio(bvec.bv_page))) &&
!memstall) {
psi_memstall_enter(&pflags);
memstall = 1;
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index fd77dbfdaf70..ae50f9880da8 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -288,6 +288,23 @@ static inline void folio_migrate_lru_refs(struct folio *new, const struct folio
__folio_set_lru_refs(new, folio_lru_refs(old));
}
+/**
+ * folio_test_workingset - Test if a folio is in the workingset.
+ * @folio: the folio
+ *
+ * A folio is workingset when its LRU refs count reaches
+ * LRU_REFS_WORKINGSET. Under the classical LRU the extra refs counter
+ * are always zero so this is just testing the PG_workingset bit.
+ * NOTE: folio_set_workingset() must not be used under MGLRU, as the
+ * folio refs are tracked by folio_inc_lru_refs() with rules.
+ *
+ * Return: true if the folio is workingset.
+ */
+static __always_inline bool folio_test_workingset(const struct folio *folio)
+{
+ return folio_lru_refs(folio) >= LRU_REFS_WORKINGSET;
+}
+
#ifdef CONFIG_LRU_GEN
static inline bool lru_refs_is_active(unsigned int refs)
{
@@ -519,6 +536,18 @@ static inline void folio_clear_lru_refs(struct folio *folio)
folio_clear_referenced(folio);
}
+/*
+ * For the classical LRU only: with MGLRU the PG_workingset bit is
+ * part of the folio refs count maintained by folio_inc_lru_refs(),
+ * and a raw set would corrupt it. The switching window is exempt
+ * because the classical paths legitimately run alongside MGLRU then.
+ */
+static __always_inline void folio_set_workingset(struct folio *folio)
+{
+ VM_WARN_ON_ONCE(lru_gen_enabled() && !lru_gen_switching());
+ set_bit(PG_workingset, folio_flags(folio, FOLIO_HEAD_PAGE));
+}
+
static __always_inline
void lruvec_add_folio(struct lruvec *lruvec, struct folio *folio)
{
diff --git a/include/linux/page-flags.h b/include/linux/page-flags.h
index b0ddc652e76c..f6625dc042da 100644
--- a/include/linux/page-flags.h
+++ b/include/linux/page-flags.h
@@ -549,8 +549,6 @@ PAGEFLAG(LRU, lru, PF_HEAD) __CLEARPAGEFLAG(LRU, lru, PF_HEAD)
FOLIO_FLAG(active, FOLIO_HEAD_PAGE)
__FOLIO_CLEAR_FLAG(active, FOLIO_HEAD_PAGE)
FOLIO_TEST_CLEAR_FLAG(active, FOLIO_HEAD_PAGE)
-PAGEFLAG(Workingset, workingset, PF_HEAD)
- TESTCLEARFLAG(Workingset, workingset, PF_HEAD)
PAGEFLAG(Checked, checked, PF_NO_COMPOUND) /* Used by some filesystems */
/* Xen */
diff --git a/mm/filemap.c b/mm/filemap.c
index b74bc1e5015c..49e827058e14 100644
--- a/mm/filemap.c
+++ b/mm/filemap.c
@@ -21,6 +21,7 @@
#include <linux/gfp.h>
#include <linux/mm.h>
#include <linux/swap.h>
+#include <linux/mm_inline.h>
#include <linux/leafops.h>
#include <linux/syscalls.h>
#include <linux/mman.h>
diff --git a/mm/page_io.c b/mm/page_io.c
index c6824fcd483e..540ae8caa932 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -16,6 +16,7 @@
#include <linux/gfp.h>
#include <linux/pagemap.h>
#include <linux/swap.h>
+#include <linux/mm_inline.h>
#include <linux/bio.h>
#include <linux/swapops.h>
#include <linux/writeback.h>
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread* Re: [PATCH v4 16/17] mm/mglru: make folio_test_workingset() work based on folio LRU refs
2026-10-15 18:16 ` Kairui Song
(?)
@ 2026-10-15 18:35 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:35 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> folio_test_workingset() currently tests the PG_workingset bit, which
> is only the second lowest access bit of the folio LRU refs count now.
> Folios with a higher refs count carry it in the LRU_REFS_MASK field
> and no longer have PG_workingset set, so the bit test misses them.
>
> folio_set_workingset() currently sets the PG_workingset bit with a
> plain set_bit(), which also over-promotes folios at a higher refs
> count: e.g. a folio at refs 4 is bumped to refs 6, advancing it
> toward LRU_REFS_MAX and an unintended generation promotion on the
> next access.
>
> Move the test into mm_inline.h as an inline helper based on
> folio_lru_refs(), checking refs >= LRU_REFS_WORKINGSET. The set
> stays a plain PG_workingset bit operation reserved for the
> classical LRU: under MGLRU the refs count is maintained by
> folio_inc_lru_refs(), and now triggers a debug WARN when called
> while MGLRU is fully on (the switching window is exempt, as the
> classical paths legitimately run alongside MGLRU then). The one
> PageWorkingset() user in erofs is converted to the folio helper.
>
> Under the classical LRU the refs count is not maintained, so the
> test falls back to the PG_workingset bit test, which is the old
> behavior.
>
> Tested-by: zhaozhengzhuo <zhaozhengzhuo@uniontech.com>
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=16
^ permalink raw reply [flat|nested] 59+ messages in thread
* [PATCH v4 17/17] Documentation/mm: multi-gen LRU: update for frequency guided promotion
2026-10-15 18:15 ` Kairui Song
@ 2026-10-15 18:16 ` Kairui Song
-1 siblings, 0 replies; 59+ messages in thread
From: Kairui Song via B4 Relay @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
From: Kairui Song <kasong@tencent.com>
The design doc still describes the old tier and promotion model:
- Tiers are no longer order_base_2() of the file descriptor accesses.
A folio's tier is derived from its referenced count, which counts
accesses from both page tables and file descriptors.
- A young PTE no longer always promotes a folio to the newest
generation. The first access only moves a folio one generation
forward if it is at the tail of its LRU, i.e., in one of the two
oldest generations of its type, a second access promotes it, and
executable file folios are promoted on their first access.
- The eviction no longer uses the first tier as the only baseline.
The type is selected by comparing all tiers of each type, and a tier
is protected by comparing it against the tiers below it.
Update the aging and eviction sections accordingly.
Assisted-by: LLM
Signed-off-by: Kairui Song <kasong@tencent.com>
---
Documentation/mm/multigen_lru.rst | 57 ++++++++++++++++++++++-----------------
1 file changed, 33 insertions(+), 24 deletions(-)
diff --git a/Documentation/mm/multigen_lru.rst b/Documentation/mm/multigen_lru.rst
index 52ed5092022f..ec8744c1f42c 100644
--- a/Documentation/mm/multigen_lru.rst
+++ b/Documentation/mm/multigen_lru.rst
@@ -95,16 +95,19 @@ at most ``MAX_NR_GENS`` generations. The gen counter stores a value
within ``[1, MAX_NR_GENS]`` while a page is on one of
``lrugen->folios[]``; otherwise it stores zero.
-Each generation is divided into multiple tiers. A page accessed ``N``
-times through file descriptors is in tier ``order_base_2(N)``. Unlike
-generations, tiers do not have dedicated ``lrugen->folios[]``. In
-contrast to moving across generations, which requires the LRU lock,
-moving across tiers only involves atomic operations on
-``folio->flags`` and therefore has a negligible cost. A feedback loop
-modeled after the PID controller monitors refaults over all the tiers
-from anon and file types and decides which tiers from which types to
-evict or protect. The desired effect is to balance refault percentages
-between anon and file types proportional to the swappiness level.
+Each generation is divided into multiple tiers. A page's tier is
+derived from its referenced count, which counts the accesses collected
+from both channels above and is capped at ``LRU_REFS_MAX``: pages
+accessed at most once are in the lowest tier, and the tiers above hold
+pages accessed at least twice. Unlike generations, tiers do not have
+dedicated ``lrugen->folios[]``. In contrast to moving across
+generations, which requires the LRU lock, updating the referenced
+count only involves atomic operations on ``folio->flags`` and
+therefore has a negligible cost. A feedback loop modeled after the PID
+controller monitors refaults over all the tiers from anon and file
+types and decides which tiers from which types to evict or protect.
+The desired effect is to balance refault percentages between anon and
+file types proportional to the swappiness level.
There are two conceptually independent procedures: the aging and the
eviction. They form a closed-loop system, i.e., the page reclaim.
@@ -122,25 +125,31 @@ and calls ``walk_page_range()`` with each ``mm_struct`` on this list
to scan PTEs, and after each iteration, it increments ``max_seq``. For
the latter, when the eviction walks the rmap and finds a young PTE,
the aging scans the adjacent PTEs. For both, on finding a young PTE,
-the aging clears the accessed bit and updates the gen counter of the
-page mapped by this PTE to ``(max_seq%MAX_NR_GENS)+1``.
+the aging clears the accessed bit and bumps the referenced count of
+the page mapped by this PTE. A second access promotes the page by
+updating its gen counter to ``(max_seq%MAX_NR_GENS)+1``; the first
+access only moves a page one generation forward if it is at the tail
+of its LRU, i.e., in one of the two oldest generations of its type,
+so that the next time it becomes the oldest one, its accessed bit
+reflects whether it has been used again. Executable file pages are
+promoted on their first access, since reclaiming them causes typical
+I/O thrashing.
Eviction
--------
The eviction consumes old generations. Given an ``lruvec``, it
increments ``min_seq`` when ``lrugen->folios[]`` indexed by
-``min_seq%MAX_NR_GENS`` becomes empty. To select a type and a tier to
-evict from, it first compares ``min_seq[]`` to select the older type.
-If both types are equally old, it selects the one whose first tier has
-a lower refault percentage. The first tier contains single-use
-unmapped clean pages, which are the best bet. The eviction sorts a
-page according to its gen counter if the aging has found this page
-accessed through page tables and updated its gen counter. It also
-moves a page to the next generation, i.e., ``min_seq+1``, if this page
-was accessed multiple times through file descriptors and the feedback
-loop has detected outlying refaults from the tier this page is in. To
-this end, the feedback loop uses the first tier as the baseline, for
-the reason stated earlier.
+``min_seq%MAX_NR_GENS`` becomes empty. To select a type to evict from,
+it first compares ``min_seq[]`` to select the older type. If both
+types are equally old, it compares the refault percentages of all the
+tiers of each type proportional to the swappiness level. To select a
+tier to protect, it compares the refault percentage of each tier
+against that of the tiers below it. The lowest tier contains pages
+accessed at most once, which are the best bet. The eviction sorts a
+page according to its gen counter if the aging or an access has
+updated its gen counter. It also moves a page to the next generation,
+i.e., ``min_seq+1``, if the feedback loop has detected outlying
+refaults from the tier this page is in.
Working set protection
----------------------
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [PATCH v4 17/17] Documentation/mm: multi-gen LRU: update for frequency guided promotion
@ 2026-10-15 18:16 ` Kairui Song
0 siblings, 0 replies; 59+ messages in thread
From: Kairui Song @ 2026-10-15 18:16 UTC (permalink / raw)
To: linux-mm, Liam R. Howlett, SJ Park
Cc: Andrew Morton, Johannes Weiner, Muchun Song, Qi Zheng, Ying Huang,
Chris Li, Baoquan He, Nico Pache, Usama Arif, Michal Hocko,
Roman Gushchin, Shakeel Butt, David Hildenbrand, Lorenzo Stoakes,
Barry Song, Axel Rasmussen, Yuanchu Xie, Wei Xu, Vlastimil Babka,
Suren Baghdasaryan, Kemeng Shi, Nhat Pham, Youngjun Park, Zi Yan,
Gregory Price, Matthew Wilcox (Oracle), Baolin Wang, Ryan Roberts,
Dev Jain, Lance Yang, Hugh Dickins, David Rientjes, Yu Zhao,
Vernon Yang, Zicheng Wang, Chen Ridong, Tal Zussman, Kairui Song,
Xiang Liu, Ehab Ababneh, zhaozhengzhuo, Lian Wang, Kunwu Chan,
linux-kernel, cgroups, damon, Kairui Song
The design doc still describes the old tier and promotion model:
- Tiers are no longer order_base_2() of the file descriptor accesses.
A folio's tier is derived from its referenced count, which counts
accesses from both page tables and file descriptors.
- A young PTE no longer always promotes a folio to the newest
generation. The first access only moves a folio one generation
forward if it is at the tail of its LRU, i.e., in one of the two
oldest generations of its type, a second access promotes it, and
executable file folios are promoted on their first access.
- The eviction no longer uses the first tier as the only baseline.
The type is selected by comparing all tiers of each type, and a tier
is protected by comparing it against the tiers below it.
Update the aging and eviction sections accordingly.
Assisted-by: LLM
Signed-off-by: Kairui Song <kasong@tencent.com>
---
Documentation/mm/multigen_lru.rst | 57 ++++++++++++++++++++++-----------------
1 file changed, 33 insertions(+), 24 deletions(-)
diff --git a/Documentation/mm/multigen_lru.rst b/Documentation/mm/multigen_lru.rst
index 52ed5092022f..ec8744c1f42c 100644
--- a/Documentation/mm/multigen_lru.rst
+++ b/Documentation/mm/multigen_lru.rst
@@ -95,16 +95,19 @@ at most ``MAX_NR_GENS`` generations. The gen counter stores a value
within ``[1, MAX_NR_GENS]`` while a page is on one of
``lrugen->folios[]``; otherwise it stores zero.
-Each generation is divided into multiple tiers. A page accessed ``N``
-times through file descriptors is in tier ``order_base_2(N)``. Unlike
-generations, tiers do not have dedicated ``lrugen->folios[]``. In
-contrast to moving across generations, which requires the LRU lock,
-moving across tiers only involves atomic operations on
-``folio->flags`` and therefore has a negligible cost. A feedback loop
-modeled after the PID controller monitors refaults over all the tiers
-from anon and file types and decides which tiers from which types to
-evict or protect. The desired effect is to balance refault percentages
-between anon and file types proportional to the swappiness level.
+Each generation is divided into multiple tiers. A page's tier is
+derived from its referenced count, which counts the accesses collected
+from both channels above and is capped at ``LRU_REFS_MAX``: pages
+accessed at most once are in the lowest tier, and the tiers above hold
+pages accessed at least twice. Unlike generations, tiers do not have
+dedicated ``lrugen->folios[]``. In contrast to moving across
+generations, which requires the LRU lock, updating the referenced
+count only involves atomic operations on ``folio->flags`` and
+therefore has a negligible cost. A feedback loop modeled after the PID
+controller monitors refaults over all the tiers from anon and file
+types and decides which tiers from which types to evict or protect.
+The desired effect is to balance refault percentages between anon and
+file types proportional to the swappiness level.
There are two conceptually independent procedures: the aging and the
eviction. They form a closed-loop system, i.e., the page reclaim.
@@ -122,25 +125,31 @@ and calls ``walk_page_range()`` with each ``mm_struct`` on this list
to scan PTEs, and after each iteration, it increments ``max_seq``. For
the latter, when the eviction walks the rmap and finds a young PTE,
the aging scans the adjacent PTEs. For both, on finding a young PTE,
-the aging clears the accessed bit and updates the gen counter of the
-page mapped by this PTE to ``(max_seq%MAX_NR_GENS)+1``.
+the aging clears the accessed bit and bumps the referenced count of
+the page mapped by this PTE. A second access promotes the page by
+updating its gen counter to ``(max_seq%MAX_NR_GENS)+1``; the first
+access only moves a page one generation forward if it is at the tail
+of its LRU, i.e., in one of the two oldest generations of its type,
+so that the next time it becomes the oldest one, its accessed bit
+reflects whether it has been used again. Executable file pages are
+promoted on their first access, since reclaiming them causes typical
+I/O thrashing.
Eviction
--------
The eviction consumes old generations. Given an ``lruvec``, it
increments ``min_seq`` when ``lrugen->folios[]`` indexed by
-``min_seq%MAX_NR_GENS`` becomes empty. To select a type and a tier to
-evict from, it first compares ``min_seq[]`` to select the older type.
-If both types are equally old, it selects the one whose first tier has
-a lower refault percentage. The first tier contains single-use
-unmapped clean pages, which are the best bet. The eviction sorts a
-page according to its gen counter if the aging has found this page
-accessed through page tables and updated its gen counter. It also
-moves a page to the next generation, i.e., ``min_seq+1``, if this page
-was accessed multiple times through file descriptors and the feedback
-loop has detected outlying refaults from the tier this page is in. To
-this end, the feedback loop uses the first tier as the baseline, for
-the reason stated earlier.
+``min_seq%MAX_NR_GENS`` becomes empty. To select a type to evict from,
+it first compares ``min_seq[]`` to select the older type. If both
+types are equally old, it compares the refault percentages of all the
+tiers of each type proportional to the swappiness level. To select a
+tier to protect, it compares the refault percentage of each tier
+against that of the tiers below it. The lowest tier contains pages
+accessed at most once, which are the best bet. The eviction sorts a
+page according to its gen counter if the aging or an access has
+updated its gen counter. It also moves a page to the next generation,
+i.e., ``min_seq+1``, if the feedback loop has detected outlying
+refaults from the tier this page is in.
Working set protection
----------------------
--
2.56.0
^ permalink raw reply related [flat|nested] 59+ messages in thread
* Re: [PATCH v4 17/17] Documentation/mm: multi-gen LRU: update for frequency guided promotion
2026-10-15 18:16 ` Kairui Song
(?)
@ 2026-10-15 18:27 ` sashiko-bot
-1 siblings, 0 replies; 59+ messages in thread
From: sashiko-bot @ 2026-10-15 18:27 UTC (permalink / raw)
To: Kairui Song; +Cc: damon
> From: Kairui Song <kasong@tencent.com>
>
> The design doc still describes the old tier and promotion model:
>
> - Tiers are no longer order_base_2() of the file descriptor accesses.
> A folio's tier is derived from its referenced count, which counts
> accesses from both page tables and file descriptors.
>
> - A young PTE no longer always promotes a folio to the newest
> generation. The first access only moves a folio one generation
> forward if it is at the tail of its LRU, i.e., in one of the two
> oldest generations of its type, a second access promotes it, and
> executable file folios are promoted on their first access.
>
> - The eviction no longer uses the first tier as the only baseline.
> The type is selected by comparing all tiers of each type, and a tier
> is protected by comparing it against the tiers below it.
>
> Update the aging and eviction sections accordingly.
>
> Assisted-by: LLM
> Signed-off-by: Kairui Song <kasong@tencent.com>
Sashiko has reviewed this patch and found no issues. It looks great!
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261016-mglru-fg-v4-0-2106750c0bfa@tencent.com?part=17
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [PATCH v4 00/17] mm/mglru: frequency guided promotion (MGLRU-FG) and flag cleanup
2026-10-15 18:15 ` Kairui Song
` (17 preceding siblings ...)
(?)
@ 2026-10-16 8:42 ` Barry Song (Xiaomi)
-1 siblings, 0 replies; 59+ messages in thread
From: Barry Song (Xiaomi) @ 2026-10-16 8:42 UTC (permalink / raw)
To: devnull+kasong.tencent.com
Cc: akpm, axelrasmussen, baohua, baolin.wang, baoquan.he, cgroups,
chenridong, chrisl, damon, david, dev.jain, ehab.ababneh, gourry,
hannes, hughd, kasong, kunwu.chan, lance.yang, liam, lianux.mm,
linux-kernel, linux-mm, liuxiang.277, ljs, mhocko, muchun.song,
nico.pache, nphamcs, qi.zheng, rientjes, roman.gushchin,
ryan.roberts, ryncsn, shakeel.butt, shikemeng, sj, surenb, tz2294,
usama.arif, vbabka, vernon2gm, wangzicheng, weixugc, willy,
ying.huang, youngjun.park, yuanchu, yuzhao, zhaozhengzhuo, ziy,
stable
On Fri, Oct 16, 2026 at 2:15 AM Kairui Song via B4 Relay <devnull+kasong.tencent.com@kernel.org> wrote:
[...]
>
> LevelDB Scan/Get
> ================
> I also retested the LevelDB benchmark from the cache_ext paper [5].
> Interestingly, mainline MGLRU already beats CLRU on this one after a
> recent change in lru_gen_folio_seq that bumps new folios with refs == 1
> to the second-oldest generation as discussed. That change accidentally
> gave random reads a higher hotness level while making sequential reads
> much colder as described before [14]. However it's unreliable and
> incorrect because it depends on LRU drain timing, and hurts some workloads
> as reported, or where the sequential part is actually hotter (any workload
> involving hotter large file and many small cold files will be affected)
> which is reproduced below.
>
> This series improves it while covering ordinary workloads too and should
> fix regressions:
>
> LevelDB Scan / Get, Throughput Total:
> CLRU: 4708.3 ops/s
> MGLRU: 5026.9 ops/s (faster than CLRU, but hurts other workloads)
> MGLRU After: 5179.2 ops/s (even better in all cases, and no regression)
>
> The hot-sequential and cold-random workload can be easily reproduced with
> SQLite and grep. SQLite continuously scans and looks up a small hot
> portion of a DB file, while grep iterates over a set of small files much
> larger than RAM [6]:
>
> SQLite scan & lookup time: Grep iterate time:
> CLRU: 14.51ms 13281.37ms
> MGLRU mainline: 567.05ms 13694.47ms
> MGLRU After this series: 10.58ms 12930.43ms
>
Thanks, Kairui. I'll do a thorough review of the whole patchset over
the next two weeks.
BTW, do you think we need a quick fix patch like the one below
for the above problem before this patchset, so that it can be
backported to stable? I assume your patchset is too large to
backport?
From f9683205d2ab6e3b185022d7d5af3496dfe5f35f Mon Sep 17 00:00:00 2001
From: "Barry Song (Xiaomi)" <baohua@kernel.org>
Date: Fri, 16 Oct 2026 11:48:06 +0800
Subject: [PATCH] mm/mglru: fix using folio_mark_accessed to replace
folio_set_active
Commit 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to
replace folio_set_active") tried to avoid over-protecting
fault-around read-ahead file folios by placing them in the
second-oldest generation instead of the youngest generations.
However, it abuses `PG_referenced` on read-ahead file folios
for this purpose, potentially coupling their handling with
normal syscall-based file accesses that also use
`PG_referenced`.
This could regress some workloads, as reported by Kairui Song[1]
and Ehab Ababneh[2]. Many thanks to both for reporting these
issues!
The long-term fix is Kairui's MGLRU-FG, but in the meantime,
we also need a quick fix that can be backported to stable.
This patch reverts all of the original commit's behavior that
abuses `PG_referenced`, while preserving the semantics of
avoiding over-protection of fault-around read-ahead file
folios by adjusting only `lru_gen_folio_seq()`.
Specifically, fault-around read-ahead file folios with
`PG_active` set but `PG_workingset` clear are placed in the
second-oldest generation.
Activated file folios and refaulted workingset file folios
are still placed in the youngest generations.
Build kernel cases in a memcg shows it not only perserve
the benefit of original patch, but also even improve
performance further.
w/o fix:
real 1m49.094s
user 25m31.484s
sys 3m40.959s
pswpin: 662112
pswpout: 2504415
pgpgin: 4424336
pgpgout: 10355273
refault_file: 359626
refault_anon: 670724
w/ fix:
real 1m48.913s
user 25m31.639s
sys 3m30.875s
pswpin: 689815
pswpout: 2689894
pgpgin: 4422659
pgpgout: 11043464
refault_file: 326566
refault_anon: 697138
Revert original patch:
real 1m50.044s
user 25m37.137s
sys 4m4.493s
pswpin: 1349087
pswpout: 4317933
pgpgin: 7083597
pgpgout: 17602735
refault_file: 337676
refault_anon: 1368542
[1] https://lore.kernel.org/linux-mm/20261003-mglru-fg-v3-0-cbd4546a5bd9@tencent.com/
[2] https://lore.kernel.org/linux-mm/20260901180704.168106-1-ehab.ababneh@intel.com/
Fixes: 6cbdd9726fb5 ("mm/mglru: use folio_mark_accessed to replace folio_set_active")
Cc: <stable@vger.kernel.org>
Signed-off-by: Barry Song (Xiaomi) <baohua@kernel.org>
---
include/linux/mm_inline.h | 8 +++++---
mm/folio.c | 16 +++-------------
mm/vmscan.c | 9 ++-------
mm/workingset.c | 10 ++++------
4 files changed, 14 insertions(+), 29 deletions(-)
diff --git a/include/linux/mm_inline.h b/include/linux/mm_inline.h
index ab69b9930893..7aaff4e06e80 100644
--- a/include/linux/mm_inline.h
+++ b/include/linux/mm_inline.h
@@ -311,7 +311,8 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
* | Accessed through page tables and | Accessed through file descriptors |
* | promoted by folio_update_gen() | and protected by folio_inc_gen() |
* +-----------------------------------+-----------------------------------+
- * | PG_active (set while isolated) | |
+ * | PG_active (set while isolated) | PG_active (files| |
+ * | | fault-readahead)| |
* +-----------------+-----------------+-----------------+-----------------+
* | PG_workingset | PG_referenced | PG_workingset | LRU_REFS_FLAGS |
* +-----------------------------------+-----------------------------------+
@@ -319,7 +320,8 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
* |<---------------------------- MAX_NR_GENS ---------------------------->|
*/
if (folio_test_active(folio))
- gen = MIN_NR_GENS - folio_test_workingset(folio);
+ gen = MIN_NR_GENS - folio_test_workingset(folio) +
+ (type ? !folio_test_workingset(folio) : 0);
else if (reclaiming)
gen = MAX_NR_GENS;
else if ((!folio_is_file_lru(folio) && !folio_test_swapcache(folio)) ||
@@ -327,7 +329,7 @@ static inline unsigned long lru_gen_folio_seq(const struct lruvec *lruvec,
(folio_test_dirty(folio) || folio_test_writeback(folio))))
gen = MIN_NR_GENS;
else
- gen = MAX_NR_GENS - (folio_test_workingset(folio) || folio_test_referenced(folio));
+ gen = MAX_NR_GENS - folio_test_workingset(folio);
return max(READ_ONCE(lrugen->max_seq) - gen + 1, READ_ONCE(lrugen->min_seq[type]));
}
diff --git a/mm/folio.c b/mm/folio.c
index 35e242b48870..2af724cecb31 100644
--- a/mm/folio.c
+++ b/mm/folio.c
@@ -474,20 +474,10 @@ void folio_add_lru(struct folio *folio)
folio_test_unevictable(folio), folio);
VM_BUG_ON_FOLIO(folio_test_lru(folio), folio);
- /*
- * For refaulted workingset folios, set PG_active so they
- * can be added to active generations.
- * For prefaulted file folios, folio_mark_accessed() sets
- * PG_referenced so lru_gen_folio_seq() places them into
- * the second oldest generation.
- */
+ /* see the comment in lru_gen_folio_seq() */
if (lru_gen_enabled() && !folio_test_unevictable(folio) &&
- lru_gen_in_fault() && !(current->flags & PF_MEMALLOC)) {
- if (folio_test_workingset(folio))
- folio_set_active(folio);
- else if (!folio_test_referenced(folio))
- folio_mark_accessed(folio);
- }
+ lru_gen_in_fault() && !(current->flags & PF_MEMALLOC))
+ folio_set_active(folio);
folio_batch_add_and_move(folio, lru_add);
}
diff --git a/mm/vmscan.c b/mm/vmscan.c
index daa7e37c2236..e82ba737c028 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -952,13 +952,8 @@ static bool lru_gen_set_refs(struct folio *folio, const vma_flags_t *vma_flags)
return false;
}
- /* Promote on second access */
- if (folio_lru_refs(folio) > 1) {
- folio_set_workingset(folio);
- folio_set_lru_refs(folio, 0);
- } else {
- folio_mark_accessed(folio);
- }
+ folio_set_workingset(folio);
+ folio_set_lru_refs(folio, 0);
return true;
}
#else
diff --git a/mm/workingset.c b/mm/workingset.c
index 1504f91cdca5..10d4ce202cca 100644
--- a/mm/workingset.c
+++ b/mm/workingset.c
@@ -319,13 +319,11 @@ static void lru_gen_refault(struct folio *folio, void *shadow)
atomic_long_add(delta, &lrugen->refaulted[hist][type][tier]);
+ /* see folio_add_lru() where folio_set_active() will be called */
+ if (lru_gen_in_fault())
+ mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
+
if (workingset) {
- /*
- * see folio_add_lru(), where folio_set_active() is
- * called for workingset folios
- */
- if (lru_gen_in_fault())
- mod_lruvec_state(lruvec, WORKINGSET_ACTIVATE_BASE + type, delta);
folio_set_workingset(folio);
mod_lruvec_state(lruvec, WORKINGSET_RESTORE_BASE + type, delta);
} else
--
2.34.1
^ permalink raw reply related [flat|nested] 59+ messages in thread