* [PATCH mm-unstable v7] mm/vmscan: throttle MGLRU eviction when isolated folios pile up
@ 2026-10-15 8:01 Hui Zhu
2026-10-16 18:07 ` Kairui Song
0 siblings, 1 reply; 5+ messages in thread
From: Hui Zhu @ 2026-10-15 8:01 UTC (permalink / raw)
To: Andrew Morton, Kairui Song, Qi Zheng, Shakeel Butt, Barry Song,
Axel Rasmussen, Yuanchu Xie, Wei Xu, Johannes Weiner,
David Hildenbrand, Michal Hocko, Lorenzo Stoakes, Baolin Wang,
linux-mm, linux-kernel
Cc: Hui Zhu
From: Hui Zhu <zhuhui@kylinos.cn>
The legacy path throttles direct reclaim via too_many_isolated() when
isolated folios pile up, but MGLRU's evict_folios() isolates folios
without this check, which can lead to unnecessary swapping, thrashing
and OOM.
Add lru_gen_too_many_isolated() and throttle_evictable_types(), called
from evict_folios() before the lruvec lock is taken since throttling
sleeps.
The check is per lruvec rather than node-wide: reclaimers contend per
lruvec, and a node-wide check would let isolation in one memcg throttle
reclaim in another. The threshold is the evictable size of the type
divided by MIN_NR_GENS, approximating one generation.
Since isolate_folios() picks the type from refault feedback and may fall
back to the other one, throttle_evictable_types() collects the types
that are not over-isolated and have evictable folios, and sleeps only
when all of them are over-isolated. The mask is passed to
isolate_folios() to restrict its choice and fallback, so an
over-isolated type never blocks the other, and isolation never lands on
a throttled or empty type.
An lruvec with no evictable folios at all is not over-isolation:
reclaim stops immediately instead of waiting. Without this, allowed == 0
would conflate "all types are empty" with "all types are over-isolated"
and send the reclaimer to sleep, and since the kswapd exemption lives
inside the per-type check, which empty types bypass, even kswapd would
stall on an empty lruvec.
Like the legacy path, kswapd and !writeback_throttling_sane() reclaim
are exempt, the threshold is raised for GFP_NOIO/GFP_NOFS callers, and
a dying task fakes progress to exit reclaim quickly.
This builds on the recently merged NR_ISOLATED accounting fix for the
MGLRU reclaim path, which makes in-flight isolation visible in
/proc/vmstat; the throttle itself uses a separate per-lruvec counter.
The nr_isolated numbers in the test results below therefore come from
that fix rather than from this patch.
Tested on a 10 CPU / 8GB QEMU guest, cgroup v2, MGLRU enabled (0x0007).
"unpatched" and "patched" are the same tree built without/with this
patch; every configuration is run with identical parameters on both
kernels. Test scripts:
https://gist.github.com/teawater/3ef51251f2e91a5a600e3d26bb477e34
1) Synthetic (mglru_iso_repro.sh, mglru_iso_repro_v2.sh): threads
write-touching a 48MB anonymous working set inside a 16MB memcg,
with swap on a dm-delay device (50ms) to slow swap-out and lengthen
the isolation window.
64 threads, 60s - neither kernel OOMs at this concurrency; the gain
shows in reclaim quality:
unpatched patched
OOM kills 0 0
total touches 246,499,132,369 301,456,426,692 (+22%)
scan efficiency 0.0261 0.0194
mm_vmscan_throttled 0 35,645 (all
VMSCAN_THROTTLE_ISOLATED)
nr_isolated peak 0 230
The patched kernel completes ~22% more work in the same 60s: the
throttle keeps concurrent reclaimers from trampling the same
generation, so less CPU is burned in reclaim.
192 threads, 60s - over-isolation becomes fatal without the
throttle:
unpatched patched
OOM kills 1 (task killed) 0
memcg oom events 51 0
total touches 0 (killed) 682,300,039,725
mm_vmscan_throttled 0 512,292
nr_isolated peak 0 335
The unpatched kernel cannot keep reclaim ahead of allocation and the
task is OOM-killed; the patched kernel survives the full run and
keeps reclaim making progress.
2) Real workload (mglru_build_repro.sh): a CI-style build pod -
repeated from-scratch "make -j20 vmlinux" rounds inside a
memory-limited cgroup, with the object tree on ext4 over dm-delay
(10ms) to model slow storage. 20 concurrent compilers charge into
one memcg and enter direct reclaim at the same time, while dirty
object files stretch the isolate->shrink->putback window. Each
kernel runs 1800s per pressure level and the metrics are cumulative
counters over that window, which smooths run-to-run noise. The
headline metric is the number of objects actually compiled (a full
round is ~3,800 objects).
memory.max = 1024MB (extreme pressure):
unpatched patched
objects compiled 4,744 7,211 (+52%)
pgscan_direct 2,901,083,521 2,228,776,807 (-23%)
scan efficiency 0.195 0.242
workingset_refault 565,169,241 537,686,479
mm_vmscan_throttled 0 511,441 (all
VMSCAN_THROTTLE_ISOLATED)
nr_isolated peak 0 1,152
memory.max = 1536MB (heavy pressure):
unpatched patched
objects compiled 18,969 25,124 (+32%)
pgscan_direct 1,223,905,508 816,812,074 (-33%)
scan efficiency 0.320 0.338
workingset_refault 390,048,986 273,442,445 (-30%)
memcg charge fails 8,344,775 5,868,167 (-30%)
round wall time 199-391s 182-275s
mm_vmscan_throttled 0 503,117
nr_isolated peak 0 1,053
At both levels -j20 is oversubscribed (20 cc1 processes need ~3GB
of anon in total), so rounds end with a cc1 being memcg-OOM-killed
on either kernel. Normalized per completed round the kill rate is
the same (13.2 vs 13.1 at 1024MB, 5.8 vs 6.1 at 1536MB); the
patched kernel simply reaches that point sooner because less time
is burned thrashing, so it cycles through more rounds. This patch
does not claim to avoid those OOMs - the point is that it performs
23-33% fewer futile LRU scans and gets 32-52% more build work done
in the same wall time.
memory.max = 2048MB (light pressure - no-harm control):
unpatched patched
rounds completed 6 6
avg wall time/round 271s 270s
scan efficiency 0.984 0.978
mm_vmscan_throttled 0 0
nr_isolated peak 0 144
When reclaim keeps up the throttle never fires and both kernels
perform identically, so the check costs nothing on a system that is
not over-isolating.
3) Service no-impact (mglru_redis_repro.sh and
mglru_redis_shards_repro.sh): a redis-server serving a 1KB-value
dataset larger than its 96MB cgroup, with swap on dm-delay (50ms) -
first as a single instance (100k keys, 200 clients), then as 8
shards with AOF persistence on a second dm-delay device sharing one
cgroup (8 concurrent chargers). The bottleneck here is swap-in I/O
serialized on redis's single command thread, not isolation
contention: charge-induced reclaim is a trickle (~400 scans/s vs
10^5-10^6 scans/s in the build runs), isolation never piles up, and
the throttle correctly stays dormant on both kernels:
unpatched patched
single instance 41.6 ops/s 41.2 ops/s
(avg 2999ms) (avg 2999ms)
8 shards aggregate 333-419 ops/s 310-358 ops/s
(mean 376, (mean 335,
4 runs) (*) 4 runs)
mm_vmscan_throttled 0 0 (all runs)
nr_isolated peak 0 0
(*) the aggregate is dominated by a preload-end residency lottery
(which shards' datasets happen to stay hot) and swings by up to 26%
between runs of the SAME kernel; with 4 runs per kernel the
difference between the means is within that noise (Welch t=1.95,
p~0.11). The throttle never fired in any patched run (the tracepoint
covers preload as well), so both kernels execute the same reclaim
path here, and the kernel-side counters agree across all 8 runs:
steal/scan 0.59-0.63, swap-ins == refaults, zero OOM. These runs
document the no-impact side: the check is dormant unless isolation
actually piles up.
Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
---
Changelog:
v7:
Patch 1 was merged into mm-unstable, so this is resent as a single
patch.
According to the comments of Sashiko, Return instead of waiting when the
lruvec has no evictable folios at all: allowed == 0 previously conflated
"all empty" with "all over-isolated" and stalled even kswapd for 20ms.
Move the v6 cover-letter test data into the commit message.
v6:
According to the comments of Barry, add real-world workload tests:
a CI-style kernel build (repeated "make -j20 vmlinux") inside a
memory-limited cgroup with its object tree on dm-delay-backed storage,
measured at three pressure levels (1024/1536/2048MB), plus redis-based
cache service runs (single instance and 8 shards sharing one cgroup)
as no-impact data points. New and updated test scripts are in the
gist. Rebased onto the latest mm-stable; patch 2 is adapted to the
current isolate_folios() type-selection context (is_single_type_
reclaim()), with no semantic change. Re-run the tests and update the
test logs.
v5:
According to the comments of Kairun and feedback from the test scripts,
make the throttle check per lruvec (new nr_isolated counter) and skip
empty types in the allowed mask.
v4:
According to the commens of Baolin and Barry, rework patch 2:
drop the throttle_is_throttled() helper extracted
from shrink_inactive_list() and leave the legacy path untouched.
The new MGLRU-only throttle_evictable_types() only sleeps when all
evictable types are over-isolated - v3 throttled as soon as any of
them was, which unnecessarily blocked the reclaim of the other type
and passes the mask of the remaining types to isolate_folios(),
which restricts both its initial choice and its fallback, so that
isolation never lands on a throttled type. The v3 gate did not
constrain the type actually isolated, so the fallback could still
pick the over-isolated one.
Re-run the tests and update the test log.
v3:
According to the commens of Baolin, remove the redundant nr_isolated
check before restoring the NR_ISOLATED_* counters in evict_folios().
rename the extracted helper to throttle_is_throttled() to avoid
confusion with the existing wake_throttle_isolated() naming space.
Use for_each_evictable_type() in the MGLRU throttle to check each
evictable type's isolation instead of only the type returned by
get_type_to_scan(), since isolate_folios() may fall back to the
other type.
Re-run the tests and update the test log.
v2:
According to the commens of Kairun, Rebased on mm-unstable.
Split into two patches; patch 2 is new and adds the
too_many_isolated() throttling to the MGLRU eviction path, which v1
did not cover.
Add test infomations.
include/linux/mmzone.h | 2 +
mm/vmscan.c | 184 +++++++++++++++++++++++++++++++++++++----
2 files changed, 172 insertions(+), 14 deletions(-)
diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
index 3b96d6c7123b..7759a53326ba 100644
--- a/include/linux/mmzone.h
+++ b/include/linux/mmzone.h
@@ -564,6 +564,8 @@ struct lru_gen_folio {
/* can be modified without holding the LRU lock */
atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
+ /* the number of folios isolated from this lruvec and being reclaimed */
+ atomic_long_t nr_isolated[ANON_AND_FILE];
/* whether the multi-gen LRU is enabled */
bool enabled;
/* the memcg generation this lru_gen_folio belongs to */
diff --git a/mm/vmscan.c b/mm/vmscan.c
index daa7e37c2236..36887117d76a 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -4332,25 +4332,34 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control
sc->priority = clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY);
}
-static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
+static unsigned long lruvec_type_evictable_size(struct lruvec *lruvec, int type)
{
- int gen, type, zone;
+ int gen, zone;
unsigned long seq, total = 0;
struct lru_gen_folio *lrugen = &lruvec->lrugen;
DEFINE_MAX_SEQ(lruvec);
DEFINE_MIN_SEQ(lruvec);
- for_each_evictable_type(type, swappiness) {
- for (seq = min_seq[type]; seq <= max_seq; seq++) {
- gen = lru_gen_from_seq(seq);
- for (zone = 0; zone < MAX_NR_ZONES; zone++)
- total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
- }
+ for (seq = min_seq[type]; seq <= max_seq; seq++) {
+ gen = lru_gen_from_seq(seq);
+ for (zone = 0; zone < MAX_NR_ZONES; zone++)
+ total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
}
return total;
}
+static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
+{
+ unsigned long total = 0;
+ int type;
+
+ for_each_evictable_type(type, swappiness)
+ total += lruvec_type_evictable_size(lruvec, type);
+
+ return total;
+}
+
static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)
{
unsigned long total;
@@ -5012,16 +5021,142 @@ static inline bool is_single_type_reclaim(int swappiness)
swappiness == SWAPPINESS_ANON_ONLY;
}
+/*
+ * The MGLRU counterpart to too_many_isolated().
+ *
+ * too_many_isolated() compares node-wide isolated counts against
+ * node-wide inactive sizes. It predates per-memcg LRU lists, which
+ * moved isolation to per-lruvec granularity. MGLRU reclaimers contend
+ * per lruvec, so this check is per lruvec as well, and isolation in
+ * one memcg does not throttle reclaim in another.
+ *
+ * The legacy threshold, the size of the inactive list, is not usable
+ * here either: in MGLRU the inactive counters are only a compatibility
+ * shim - they jump around when swap runs out or a memcg hits its swap
+ * limit, and proactive aging can inflate them. Compare against the
+ * total number of evictable pages of the type divided by MIN_NR_GENS,
+ * approximating the size of one generation - the same unit the
+ * original MGLRU aging heuristics used.
+ */
+static bool lru_gen_too_many_isolated(struct lruvec *lruvec, int type,
+ struct scan_control *sc)
+{
+ unsigned long isolated, evictable;
+
+ if (current_is_kswapd())
+ return false;
+
+ if (!writeback_throttling_sane(sc))
+ return false;
+
+ isolated = atomic_long_read(&lruvec->lrugen.nr_isolated[type]);
+ evictable = lruvec_type_evictable_size(lruvec, type);
+
+ /*
+ * GFP_NOIO/GFP_NOFS callers are allowed to isolate more pages, so
+ * they won't be blocked by normal direct-reclaimers, forming a
+ * circular deadlock.
+ */
+ if (gfp_has_io_fs(sc->gfp_mask))
+ evictable >>= 3;
+
+ return isolated > evictable / MIN_NR_GENS;
+}
+
+/*
+ * Unlike the legacy path, where the LRU list to isolate from is known
+ * before isolation, isolate_folios() picks the type from the refault
+ * feedback and may fall back to the other one. Therefore, instead of
+ * throttling on a single type, collect the evictable types that do not
+ * have too many isolated folios, and only sleep when all of them do.
+ *
+ * Returns the mask of the types isolate_folios() may isolate from, or
+ * 0 if reclaim should stop. Also sets @fatal to tell the caller that
+ * the task received a fatal signal while waiting.
+ */
+static unsigned int throttle_evictable_types(struct lruvec *lruvec,
+ int swappiness,
+ struct scan_control *sc,
+ bool *fatal)
+{
+ unsigned int allowed;
+ bool stalled = false;
+ bool has_evictable;
+ struct pglist_data *pgdat = lruvec_pgdat(lruvec);
+ int i;
+
+ *fatal = false;
+
+ for (;;) {
+ allowed = 0;
+ has_evictable = false;
+ for_each_evictable_type(i, swappiness) {
+ /*
+ * A type with no evictable folios has nothing to
+ * isolate. Don't allow it, otherwise reclaim can be
+ * redirected to it and spin making no progress, e.g.
+ * when the only type with folios is over-isolated.
+ */
+ if (!lruvec_type_evictable_size(lruvec, i))
+ continue;
+
+ has_evictable = true;
+
+ if (!lru_gen_too_many_isolated(lruvec, i, sc))
+ allowed |= BIT(i);
+ }
+
+ /*
+ * Nothing evictable at all is not over-isolation. Stop
+ * reclaim instead of waiting, so that nobody - including
+ * kswapd, which is exempt from the check below - stalls on
+ * an empty lruvec.
+ */
+ if (!has_evictable)
+ return 0;
+
+ if (allowed) {
+ /* Wake up reclaimers waiting on the isolation to go down. */
+ wake_throttle_isolated(pgdat);
+ return allowed;
+ }
+
+ /*
+ * All evictable types are over-isolated. Like the legacy
+ * path, wait once for concurrent reclaimers to put their
+ * isolated folios back; give up if that makes no progress.
+ */
+ if (stalled)
+ return 0;
+
+ stalled = true;
+ reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED);
+
+ /* We are about to die and free our memory. Return now. */
+ if (fatal_signal_pending(current)) {
+ *fatal = true;
+ return 0;
+ }
+ }
+}
+
static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
struct scan_control *sc, int swappiness,
- struct list_head *list, int *isolated,
- int *isolate_type, int *isolate_scanned)
+ unsigned int allowed, struct list_head *list,
+ int *isolated, int *isolate_type, int *isolate_scanned)
{
bool type_fallback_allowed = !is_single_type_reclaim(swappiness);
int type = get_type_to_scan(lruvec, swappiness);
int total_scanned = 0, scanned, tier;
bool tried = false;
+ /*
+ * The preferred type may have been excluded by
+ * throttle_evictable_types(); start from the other one.
+ */
+ if (!(allowed & BIT(type)))
+ type = !type;
+
retry:
tier = get_tier_idx(lruvec, type);
scanned = scan_folios(nr_to_scan, lruvec, sc,
@@ -5036,9 +5171,11 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
/*
* We are running out of the current reclaim type. Fall back to
- * the other type if allowed.
+ * the other type if allowed. Only fall back to a type that is
+ * not throttled.
*/
- if (!scanned && !tried && type_fallback_allowed) {
+ if (!scanned && !tried && type_fallback_allowed &&
+ (allowed & BIT(!type))) {
type = !type;
tried = true;
type_fallback_allowed = false;
@@ -5072,18 +5209,36 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
bool skip_retry = false;
struct mem_cgroup *memcg = lruvec_memcg(lruvec);
struct pglist_data *pgdat = lruvec_pgdat(lruvec);
+ unsigned int allowed;
+ bool fatal;
+
+ allowed = throttle_evictable_types(lruvec, swappiness, sc, &fatal);
+ if (!allowed) {
+ /*
+ * We are about to die and free our memory. Like the legacy
+ * path, pretend some pages were reclaimed so reclaim
+ * unwinds quickly instead of looping back into the
+ * throttle.
+ */
+ if (fatal)
+ sc->nr_reclaimed += SWAP_CLUSTER_MAX;
+
+ return 0;
+ }
lruvec_lock_irq(lruvec);
/* In case folio deletion left empty old gens, flush them */
try_to_inc_min_seq(lruvec, swappiness);
- scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness,
+ scanned = isolate_folios(nr_to_scan, lruvec, sc, swappiness, allowed,
&list, &isolated, &type, &type_scanned);
nr_isolated = isolated;
- if (nr_isolated)
+ if (nr_isolated) {
__mod_node_page_state(pgdat, NR_ISOLATED_ANON + type,
nr_isolated);
+ atomic_long_add(nr_isolated, &lruvec->lrugen.nr_isolated[type]);
+ }
/* Scanning may have emptied the oldest gen, flush it */
if (scanned)
@@ -5152,6 +5307,7 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
goto retry;
}
+ atomic_long_sub(nr_isolated, &lruvec->lrugen.nr_isolated[type]);
mod_node_page_state(pgdat, NR_ISOLATED_ANON + type, -nr_isolated);
if (nr_isolated > total_reclaimed)
--
2.43.0
^ permalink raw reply related [flat|nested] 5+ messages in thread* Re: [PATCH mm-unstable v7] mm/vmscan: throttle MGLRU eviction when isolated folios pile up
2026-10-15 8:01 [PATCH mm-unstable v7] mm/vmscan: throttle MGLRU eviction when isolated folios pile up Hui Zhu
@ 2026-10-16 18:07 ` Kairui Song
2026-10-20 9:28 ` Hui Zhu
0 siblings, 1 reply; 5+ messages in thread
From: Kairui Song @ 2026-10-16 18:07 UTC (permalink / raw)
To: Hui Zhu
Cc: Andrew Morton, Qi Zheng, Shakeel Butt, Barry Song, Axel Rasmussen,
Yuanchu Xie, Wei Xu, Johannes Weiner, David Hildenbrand,
Michal Hocko, Lorenzo Stoakes, Baolin Wang, linux-mm,
linux-kernel, Hui Zhu
On Thu, Oct 15, 2026 at 4:02 PM Hui Zhu <hui.zhu@linux.dev> wrote:
>
> From: Hui Zhu <zhuhui@kylinos.cn>
>
> The legacy path throttles direct reclaim via too_many_isolated() when
> isolated folios pile up, but MGLRU's evict_folios() isolates folios
> without this check, which can lead to unnecessary swapping, thrashing
> and OOM.
>
> Add lru_gen_too_many_isolated() and throttle_evictable_types(), called
> from evict_folios() before the lruvec lock is taken since throttling
> sleeps.
>
> The check is per lruvec rather than node-wide: reclaimers contend per
> lruvec, and a node-wide check would let isolation in one memcg throttle
> reclaim in another. The threshold is the evictable size of the type
> divided by MIN_NR_GENS, approximating one generation.
>
> Since isolate_folios() picks the type from refault feedback and may fall
> back to the other one, throttle_evictable_types() collects the types
> that are not over-isolated and have evictable folios, and sleeps only
> when all of them are over-isolated. The mask is passed to
> isolate_folios() to restrict its choice and fallback, so an
> over-isolated type never blocks the other, and isolation never lands on
> a throttled or empty type.
>
> An lruvec with no evictable folios at all is not over-isolation:
> reclaim stops immediately instead of waiting. Without this, allowed == 0
> would conflate "all types are empty" with "all types are over-isolated"
> and send the reclaimer to sleep, and since the kswapd exemption lives
> inside the per-type check, which empty types bypass, even kswapd would
> stall on an empty lruvec.
>
> Like the legacy path, kswapd and !writeback_throttling_sane() reclaim
> are exempt, the threshold is raised for GFP_NOIO/GFP_NOFS callers, and
> a dying task fakes progress to exit reclaim quickly.
>
> This builds on the recently merged NR_ISOLATED accounting fix for the
> MGLRU reclaim path, which makes in-flight isolation visible in
> /proc/vmstat; the throttle itself uses a separate per-lruvec counter.
> The nr_isolated numbers in the test results below therefore come from
> that fix rather than from this patch.
>
> Tested on a 10 CPU / 8GB QEMU guest, cgroup v2, MGLRU enabled (0x0007).
> "unpatched" and "patched" are the same tree built without/with this
> patch; every configuration is run with identical parameters on both
> kernels. Test scripts:
> https://gist.github.com/teawater/3ef51251f2e91a5a600e3d26bb477e34
>
> 1) Synthetic (mglru_iso_repro.sh, mglru_iso_repro_v2.sh): threads
> write-touching a 48MB anonymous working set inside a 16MB memcg,
> with swap on a dm-delay device (50ms) to slow swap-out and lengthen
> the isolation window.
>
> 64 threads, 60s - neither kernel OOMs at this concurrency; the gain
> shows in reclaim quality:
> unpatched patched
> OOM kills 0 0
> total touches 246,499,132,369 301,456,426,692 (+22%)
> scan efficiency 0.0261 0.0194
> mm_vmscan_throttled 0 35,645 (all
> VMSCAN_THROTTLE_ISOLATED)
> nr_isolated peak 0 230
>
> The patched kernel completes ~22% more work in the same 60s: the
> throttle keeps concurrent reclaimers from trampling the same
> generation, so less CPU is burned in reclaim.
>
> 192 threads, 60s - over-isolation becomes fatal without the
> throttle:
> unpatched patched
> OOM kills 1 (task killed) 0
> memcg oom events 51 0
> total touches 0 (killed) 682,300,039,725
> mm_vmscan_throttled 0 512,292
> nr_isolated peak 0 335
>
> The unpatched kernel cannot keep reclaim ahead of allocation and the
> task is OOM-killed; the patched kernel survives the full run and
> keeps reclaim making progress.
>
> 2) Real workload (mglru_build_repro.sh): a CI-style build pod -
> repeated from-scratch "make -j20 vmlinux" rounds inside a
> memory-limited cgroup, with the object tree on ext4 over dm-delay
> (10ms) to model slow storage. 20 concurrent compilers charge into
> one memcg and enter direct reclaim at the same time, while dirty
> object files stretch the isolate->shrink->putback window. Each
> kernel runs 1800s per pressure level and the metrics are cumulative
> counters over that window, which smooths run-to-run noise. The
> headline metric is the number of objects actually compiled (a full
> round is ~3,800 objects).
>
> memory.max = 1024MB (extreme pressure):
> unpatched patched
> objects compiled 4,744 7,211 (+52%)
> pgscan_direct 2,901,083,521 2,228,776,807 (-23%)
> scan efficiency 0.195 0.242
> workingset_refault 565,169,241 537,686,479
> mm_vmscan_throttled 0 511,441 (all
> VMSCAN_THROTTLE_ISOLATED)
> nr_isolated peak 0 1,152
>
> memory.max = 1536MB (heavy pressure):
> unpatched patched
> objects compiled 18,969 25,124 (+32%)
> pgscan_direct 1,223,905,508 816,812,074 (-33%)
> scan efficiency 0.320 0.338
> workingset_refault 390,048,986 273,442,445 (-30%)
> memcg charge fails 8,344,775 5,868,167 (-30%)
> round wall time 199-391s 182-275s
> mm_vmscan_throttled 0 503,117
> nr_isolated peak 0 1,053
Hmm is this because of the type fallback? e.g. too many files isoalted
-> scan anon instead / too many anon isolate -> scan file instead.
>
> At both levels -j20 is oversubscribed (20 cc1 processes need ~3GB
> of anon in total), so rounds end with a cc1 being memcg-OOM-killed
> on either kernel. Normalized per completed round the kill rate is
> the same (13.2 vs 13.1 at 1024MB, 5.8 vs 6.1 at 1536MB); the
> patched kernel simply reaches that point sooner because less time
> is burned thrashing, so it cycles through more rounds. This patch
> does not claim to avoid those OOMs - the point is that it performs
> 23-33% fewer futile LRU scans and gets 32-52% more build work done
> in the same wall time.
>
> memory.max = 2048MB (light pressure - no-harm control):
> unpatched patched
> rounds completed 6 6
> avg wall time/round 271s 270s
> scan efficiency 0.984 0.978
> mm_vmscan_throttled 0 0
> nr_isolated peak 0 144
>
> When reclaim keeps up the throttle never fires and both kernels
> perform identically, so the check costs nothing on a system that is
> not over-isolating.
>
> 3) Service no-impact (mglru_redis_repro.sh and
> mglru_redis_shards_repro.sh): a redis-server serving a 1KB-value
> dataset larger than its 96MB cgroup, with swap on dm-delay (50ms) -
> first as a single instance (100k keys, 200 clients), then as 8
> shards with AOF persistence on a second dm-delay device sharing one
> cgroup (8 concurrent chargers). The bottleneck here is swap-in I/O
> serialized on redis's single command thread, not isolation
> contention: charge-induced reclaim is a trickle (~400 scans/s vs
> 10^5-10^6 scans/s in the build runs), isolation never piles up, and
> the throttle correctly stays dormant on both kernels:
>
> unpatched patched
> single instance 41.6 ops/s 41.2 ops/s
> (avg 2999ms) (avg 2999ms)
> 8 shards aggregate 333-419 ops/s 310-358 ops/s
> (mean 376, (mean 335,
> 4 runs) (*) 4 runs)
Hmm, I know this could be noise but it looks like a regression?
> mm_vmscan_throttled 0 0 (all runs)
> nr_isolated peak 0 0
>
> (*) the aggregate is dominated by a preload-end residency lottery
> (which shards' datasets happen to stay hot) and swings by up to 26%
> between runs of the SAME kernel; with 4 runs per kernel the
> difference between the means is within that noise (Welch t=1.95,
> p~0.11). The throttle never fired in any patched run (the tracepoint
> covers preload as well), so both kernels execute the same reclaim
> path here, and the kernel-side counters agree across all 8 runs:
> steal/scan 0.59-0.63, swap-ins == refaults, zero OOM. These runs
> document the no-impact side: the check is dormant unless isolation
> actually piles up.
>
> Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
Hi Hui
Looks like a good idea in general.
> ---
> Changelog:
> v7:
> Patch 1 was merged into mm-unstable, so this is resent as a single
> patch.
> According to the comments of Sashiko, Return instead of waiting when the
> lruvec has no evictable folios at all: allowed == 0 previously conflated
> "all empty" with "all over-isolated" and stalled even kswapd for 20ms.
> Move the v6 cover-letter test data into the commit message.
> v6:
> According to the comments of Barry, add real-world workload tests:
> a CI-style kernel build (repeated "make -j20 vmlinux") inside a
> memory-limited cgroup with its object tree on dm-delay-backed storage,
> measured at three pressure levels (1024/1536/2048MB), plus redis-based
> cache service runs (single instance and 8 shards sharing one cgroup)
> as no-impact data points. New and updated test scripts are in the
> gist. Rebased onto the latest mm-stable; patch 2 is adapted to the
> current isolate_folios() type-selection context (is_single_type_
> reclaim()), with no semantic change. Re-run the tests and update the
> test logs.
> v5:
> According to the comments of Kairun and feedback from the test scripts,
> make the throttle check per lruvec (new nr_isolated counter) and skip
> empty types in the allowed mask.
> v4:
> According to the commens of Baolin and Barry, rework patch 2:
> drop the throttle_is_throttled() helper extracted
> from shrink_inactive_list() and leave the legacy path untouched.
> The new MGLRU-only throttle_evictable_types() only sleeps when all
> evictable types are over-isolated - v3 throttled as soon as any of
> them was, which unnecessarily blocked the reclaim of the other type
> and passes the mask of the remaining types to isolate_folios(),
> which restricts both its initial choice and its fallback, so that
> isolation never lands on a throttled type. The v3 gate did not
> constrain the type actually isolated, so the fallback could still
> pick the over-isolated one.
> Re-run the tests and update the test log.
> v3:
> According to the commens of Baolin, remove the redundant nr_isolated
> check before restoring the NR_ISOLATED_* counters in evict_folios().
> rename the extracted helper to throttle_is_throttled() to avoid
> confusion with the existing wake_throttle_isolated() naming space.
> Use for_each_evictable_type() in the MGLRU throttle to check each
> evictable type's isolation instead of only the type returned by
> get_type_to_scan(), since isolate_folios() may fall back to the
> other type.
> Re-run the tests and update the test log.
> v2:
> According to the commens of Kairun, Rebased on mm-unstable.
> Split into two patches; patch 2 is new and adds the
> too_many_isolated() throttling to the MGLRU eviction path, which v1
> did not cover.
> Add test infomations.
>
> include/linux/mmzone.h | 2 +
> mm/vmscan.c | 184 +++++++++++++++++++++++++++++++++++++----
> 2 files changed, 172 insertions(+), 14 deletions(-)
>
> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> index 3b96d6c7123b..7759a53326ba 100644
> --- a/include/linux/mmzone.h
> +++ b/include/linux/mmzone.h
> @@ -564,6 +564,8 @@ struct lru_gen_folio {
> /* can be modified without holding the LRU lock */
> atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
> atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
> + /* the number of folios isolated from this lruvec and being reclaimed */
> + atomic_long_t nr_isolated[ANON_AND_FILE];
> /* whether the multi-gen LRU is enabled */
> bool enabled;
> /* the memcg generation this lru_gen_folio belongs to */
> diff --git a/mm/vmscan.c b/mm/vmscan.c
> index daa7e37c2236..36887117d76a 100644
> --- a/mm/vmscan.c
> +++ b/mm/vmscan.c
> @@ -4332,25 +4332,34 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control
> sc->priority = clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY);
> }
>
> -static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
> +static unsigned long lruvec_type_evictable_size(struct lruvec *lruvec, int type)
> {
> - int gen, type, zone;
> + int gen, zone;
> unsigned long seq, total = 0;
> struct lru_gen_folio *lrugen = &lruvec->lrugen;
> DEFINE_MAX_SEQ(lruvec);
> DEFINE_MIN_SEQ(lruvec);
>
> - for_each_evictable_type(type, swappiness) {
> - for (seq = min_seq[type]; seq <= max_seq; seq++) {
> - gen = lru_gen_from_seq(seq);
> - for (zone = 0; zone < MAX_NR_ZONES; zone++)
> - total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
> - }
> + for (seq = min_seq[type]; seq <= max_seq; seq++) {
> + gen = lru_gen_from_seq(seq);
> + for (zone = 0; zone < MAX_NR_ZONES; zone++)
> + total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
> }
>
> return total;
> }
>
> +static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
> +{
> + unsigned long total = 0;
> + int type;
> +
> + for_each_evictable_type(type, swappiness)
> + total += lruvec_type_evictable_size(lruvec, type);
> +
> + return total;
> +}
> +
The helper split looks clean; it could be a stand alone commit I think.
> static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)
> {
> unsigned long total;
> @@ -5012,16 +5021,142 @@ static inline bool is_single_type_reclaim(int swappiness)
> swappiness == SWAPPINESS_ANON_ONLY;
> }
>
> +/*
> + * The MGLRU counterpart to too_many_isolated().
> + *
> + * too_many_isolated() compares node-wide isolated counts against
> + * node-wide inactive sizes. It predates per-memcg LRU lists, which
> + * moved isolation to per-lruvec granularity. MGLRU reclaimers contend
> + * per lruvec, so this check is per lruvec as well, and isolation in
> + * one memcg does not throttle reclaim in another.
> + *
> + * The legacy threshold, the size of the inactive list, is not usable
> + * here either: in MGLRU the inactive counters are only a compatibility
> + * shim - they jump around when swap runs out or a memcg hits its swap
> + * limit, and proactive aging can inflate them. Compare against the
> + * total number of evictable pages of the type divided by MIN_NR_GENS,
> + * approximating the size of one generation - the same unit the
> + * original MGLRU aging heuristics used.
> + */
> +static bool lru_gen_too_many_isolated(struct lruvec *lruvec, int type,
> + struct scan_control *sc)
> +{
> + unsigned long isolated, evictable;
> +
> + if (current_is_kswapd())
> + return false;
> +
> + if (!writeback_throttling_sane(sc))
> + return false;
> +
> + isolated = atomic_long_read(&lruvec->lrugen.nr_isolated[type]);
> + evictable = lruvec_type_evictable_size(lruvec, type);
> +
> + /*
> + * GFP_NOIO/GFP_NOFS callers are allowed to isolate more pages, so
> + * they won't be blocked by normal direct-reclaimers, forming a
> + * circular deadlock.
> + */
> + if (gfp_has_io_fs(sc->gfp_mask))
> + evictable >>= 3;
> +
> + return isolated > evictable / MIN_NR_GENS;
> +}
The magic number of >>=3 looks a bit odd, others are fine to me.
> +
> +/*
> + * Unlike the legacy path, where the LRU list to isolate from is known
> + * before isolation, isolate_folios() picks the type from the refault
> + * feedback and may fall back to the other one. Therefore, instead of
> + * throttling on a single type, collect the evictable types that do not
> + * have too many isolated folios, and only sleep when all of them do.
> + *
> + * Returns the mask of the types isolate_folios() may isolate from, or
> + * 0 if reclaim should stop. Also sets @fatal to tell the caller that
> + * the task received a fatal signal while waiting.
> + */
> +static unsigned int throttle_evictable_types(struct lruvec *lruvec,
> + int swappiness,
> + struct scan_control *sc,
> + bool *fatal)
> +{
> + unsigned int allowed;
> + bool stalled = false;
> + bool has_evictable;
> + struct pglist_data *pgdat = lruvec_pgdat(lruvec);
> + int i;
> +
> + *fatal = false;
> +
> + for (;;) {
> + allowed = 0;
> + has_evictable = false;
> + for_each_evictable_type(i, swappiness) {
> + /*
> + * A type with no evictable folios has nothing to
> + * isolate. Don't allow it, otherwise reclaim can be
> + * redirected to it and spin making no progress, e.g.
> + * when the only type with folios is over-isolated.
> + */
> + if (!lruvec_type_evictable_size(lruvec, i))
> + continue;
> +
> + has_evictable = true;
> +
> + if (!lru_gen_too_many_isolated(lruvec, i, sc))
> + allowed |= BIT(i);
> + }
> +
> + /*
> + * Nothing evictable at all is not over-isolation. Stop
> + * reclaim instead of waiting, so that nobody - including
> + * kswapd, which is exempt from the check below - stalls on
> + * an empty lruvec.
> + */
> + if (!has_evictable)
> + return 0;
> +
> + if (allowed) {
> + /* Wake up reclaimers waiting on the isolation to go down. */
> + wake_throttle_isolated(pgdat);
> + return allowed;
This seems counterintuitive: we reach here because a reclaimer is
starting, and we wake up even more reclaimer?
> + }
> +
> + /*
> + * All evictable types are over-isolated. Like the legacy
> + * path, wait once for concurrent reclaimers to put their
> + * isolated folios back; give up if that makes no progress.
> + */
> + if (stalled)
> + return 0;
> +
> + stalled = true;
> + reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED);
> +
> + /* We are about to die and free our memory. Return now. */
> + if (fatal_signal_pending(current)) {
> + *fatal = true;
> + return 0;
> + }
> + }
> +}
> +
Hmm... this part looks a bit too complex, see below.
> static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
> struct scan_control *sc, int swappiness,
> - struct list_head *list, int *isolated,
> - int *isolate_type, int *isolate_scanned)
> + unsigned int allowed, struct list_head *list,
> + int *isolated, int *isolate_type, int *isolate_scanned)
> {
> bool type_fallback_allowed = !is_single_type_reclaim(swappiness);
> int type = get_type_to_scan(lruvec, swappiness);
> int total_scanned = 0, scanned, tier;
> bool tried = false;
>
> + /*
> + * The preferred type may have been excluded by
> + * throttle_evictable_types(); start from the other one.
> + */
> + if (!(allowed & BIT(type)))
> + type = !type;
> +
> retry:
> tier = get_tier_idx(lruvec, type);
> scanned = scan_folios(nr_to_scan, lruvec, sc,
> @@ -5036,9 +5171,11 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
>
> /*
> * We are running out of the current reclaim type. Fall back to
> - * the other type if allowed.
> + * the other type if allowed. Only fall back to a type that is
> + * not throttled.
> */
> - if (!scanned && !tried && type_fallback_allowed) {
> + if (!scanned && !tried && type_fallback_allowed &&
> + (allowed & BIT(!type))) {
> type = !type;
> tried = true;
> type_fallback_allowed = false;
> @@ -5072,18 +5209,36 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
> bool skip_retry = false;
> struct mem_cgroup *memcg = lruvec_memcg(lruvec);
> struct pglist_data *pgdat = lruvec_pgdat(lruvec);
> + unsigned int allowed;
> + bool fatal;
> +
> + allowed = throttle_evictable_types(lruvec, swappiness, sc, &fatal);
> + if (!allowed) {
> + /*
> + * We are about to die and free our memory. Like the legacy
> + * path, pretend some pages were reclaimed so reclaim
> + * unwinds quickly instead of looping back into the
> + * throttle.
> + */
> + if (fatal)
> + sc->nr_reclaimed += SWAP_CLUSTER_MAX;
And reporting a fake nr_reclaimed looks a bad thing to do in the first place.
It seems a lot of trouble is caused by the design of the reclaim loop,
so will it work better if we calculate the number & type to reclaim
first like CLRU?
There is a demonstration from Barry:
https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?h=kairui_scanbalance_0.3
And instead of throttle at the reclaim loop, balance the scan budge
based on (size - isolation) number, and throttle a while LRU simply
when the total isolated number is larger than total size /
MIN_NR_GENS.
Just an idea for reference, seems could help get rid of a lot of complex logic.
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH mm-unstable v7] mm/vmscan: throttle MGLRU eviction when isolated folios pile up
2026-10-16 18:07 ` Kairui Song
@ 2026-10-20 9:28 ` Hui Zhu
2026-10-22 22:20 ` Barry Song
2026-10-23 15:51 ` Kairui Song
0 siblings, 2 replies; 5+ messages in thread
From: Hui Zhu @ 2026-10-20 9:28 UTC (permalink / raw)
To: Kairui Song
Cc: Andrew Morton, Qi Zheng, Shakeel Butt, Barry Song, Axel Rasmussen,
Yuanchu Xie, Wei Xu, Johannes Weiner, David Hildenbrand,
Michal Hocko, Lorenzo Stoakes, Baolin Wang, linux-mm,
linux-kernel, Hui Zhu
2026/10/17 02:07, Kairui Song:
> On Thu, Oct 15, 2026 at 4:02 PM Hui Zhu <hui.zhu@linux.dev> wrote:
>> From: Hui Zhu <zhuhui@kylinos.cn>
>>
>> The legacy path throttles direct reclaim via too_many_isolated() when
>> isolated folios pile up, but MGLRU's evict_folios() isolates folios
>> without this check, which can lead to unnecessary swapping, thrashing
>> and OOM.
>>
>> Add lru_gen_too_many_isolated() and throttle_evictable_types(), called
>> from evict_folios() before the lruvec lock is taken since throttling
>> sleeps.
>>
>> The check is per lruvec rather than node-wide: reclaimers contend per
>> lruvec, and a node-wide check would let isolation in one memcg throttle
>> reclaim in another. The threshold is the evictable size of the type
>> divided by MIN_NR_GENS, approximating one generation.
>>
>> Since isolate_folios() picks the type from refault feedback and may fall
>> back to the other one, throttle_evictable_types() collects the types
>> that are not over-isolated and have evictable folios, and sleeps only
>> when all of them are over-isolated. The mask is passed to
>> isolate_folios() to restrict its choice and fallback, so an
>> over-isolated type never blocks the other, and isolation never lands on
>> a throttled or empty type.
>>
>> An lruvec with no evictable folios at all is not over-isolation:
>> reclaim stops immediately instead of waiting. Without this, allowed == 0
>> would conflate "all types are empty" with "all types are over-isolated"
>> and send the reclaimer to sleep, and since the kswapd exemption lives
>> inside the per-type check, which empty types bypass, even kswapd would
>> stall on an empty lruvec.
>>
>> Like the legacy path, kswapd and !writeback_throttling_sane() reclaim
>> are exempt, the threshold is raised for GFP_NOIO/GFP_NOFS callers, and
>> a dying task fakes progress to exit reclaim quickly.
>>
>> This builds on the recently merged NR_ISOLATED accounting fix for the
>> MGLRU reclaim path, which makes in-flight isolation visible in
>> /proc/vmstat; the throttle itself uses a separate per-lruvec counter.
>> The nr_isolated numbers in the test results below therefore come from
>> that fix rather than from this patch.
>>
>> Tested on a 10 CPU / 8GB QEMU guest, cgroup v2, MGLRU enabled (0x0007).
>> "unpatched" and "patched" are the same tree built without/with this
>> patch; every configuration is run with identical parameters on both
>> kernels. Test scripts:
>> https://gist.github.com/teawater/3ef51251f2e91a5a600e3d26bb477e34
>>
>> 1) Synthetic (mglru_iso_repro.sh, mglru_iso_repro_v2.sh): threads
>> write-touching a 48MB anonymous working set inside a 16MB memcg,
>> with swap on a dm-delay device (50ms) to slow swap-out and lengthen
>> the isolation window.
>>
>> 64 threads, 60s - neither kernel OOMs at this concurrency; the gain
>> shows in reclaim quality:
>> unpatched patched
>> OOM kills 0 0
>> total touches 246,499,132,369 301,456,426,692 (+22%)
>> scan efficiency 0.0261 0.0194
>> mm_vmscan_throttled 0 35,645 (all
>> VMSCAN_THROTTLE_ISOLATED)
>> nr_isolated peak 0 230
>>
>> The patched kernel completes ~22% more work in the same 60s: the
>> throttle keeps concurrent reclaimers from trampling the same
>> generation, so less CPU is burned in reclaim.
>>
>> 192 threads, 60s - over-isolation becomes fatal without the
>> throttle:
>> unpatched patched
>> OOM kills 1 (task killed) 0
>> memcg oom events 51 0
>> total touches 0 (killed) 682,300,039,725
>> mm_vmscan_throttled 0 512,292
>> nr_isolated peak 0 335
>>
>> The unpatched kernel cannot keep reclaim ahead of allocation and the
>> task is OOM-killed; the patched kernel survives the full run and
>> keeps reclaim making progress.
>>
>> 2) Real workload (mglru_build_repro.sh): a CI-style build pod -
>> repeated from-scratch "make -j20 vmlinux" rounds inside a
>> memory-limited cgroup, with the object tree on ext4 over dm-delay
>> (10ms) to model slow storage. 20 concurrent compilers charge into
>> one memcg and enter direct reclaim at the same time, while dirty
>> object files stretch the isolate->shrink->putback window. Each
>> kernel runs 1800s per pressure level and the metrics are cumulative
>> counters over that window, which smooths run-to-run noise. The
>> headline metric is the number of objects actually compiled (a full
>> round is ~3,800 objects).
>>
>> memory.max = 1024MB (extreme pressure):
>> unpatched patched
>> objects compiled 4,744 7,211 (+52%)
>> pgscan_direct 2,901,083,521 2,228,776,807 (-23%)
>> scan efficiency 0.195 0.242
>> workingset_refault 565,169,241 537,686,479
>> mm_vmscan_throttled 0 511,441 (all
>> VMSCAN_THROTTLE_ISOLATED)
>> nr_isolated peak 0 1,152
>>
>> memory.max = 1536MB (heavy pressure):
>> unpatched patched
>> objects compiled 18,969 25,124 (+32%)
>> pgscan_direct 1,223,905,508 816,812,074 (-33%)
>> scan efficiency 0.320 0.338
>> workingset_refault 390,048,986 273,442,445 (-30%)
>> memcg charge fails 8,344,775 5,868,167 (-30%)
>> round wall time 199-391s 182-275s
>> mm_vmscan_throttled 0 503,117
>> nr_isolated peak 0 1,053
> Hmm is this because of the type fallback? e.g. too many files isoalted
> -> scan anon instead / too many anon isolate -> scan file instead.
>> At both levels -j20 is oversubscribed (20 cc1 processes need ~3GB
>> of anon in total), so rounds end with a cc1 being memcg-OOM-killed
>> on either kernel. Normalized per completed round the kill rate is
>> the same (13.2 vs 13.1 at 1024MB, 5.8 vs 6.1 at 1536MB); the
>> patched kernel simply reaches that point sooner because less time
>> is burned thrashing, so it cycles through more rounds. This patch
>> does not claim to avoid those OOMs - the point is that it performs
>> 23-33% fewer futile LRU scans and gets 32-52% more build work done
>> in the same wall time.
>>
>> memory.max = 2048MB (light pressure - no-harm control):
>> unpatched patched
>> rounds completed 6 6
>> avg wall time/round 271s 270s
>> scan efficiency 0.984 0.978
>> mm_vmscan_throttled 0 0
>> nr_isolated peak 0 144
>>
>> When reclaim keeps up the throttle never fires and both kernels
>> perform identically, so the check costs nothing on a system that is
>> not over-isolating.
>>
>> 3) Service no-impact (mglru_redis_repro.sh and
>> mglru_redis_shards_repro.sh): a redis-server serving a 1KB-value
>> dataset larger than its 96MB cgroup, with swap on dm-delay (50ms) -
>> first as a single instance (100k keys, 200 clients), then as 8
>> shards with AOF persistence on a second dm-delay device sharing one
>> cgroup (8 concurrent chargers). The bottleneck here is swap-in I/O
>> serialized on redis's single command thread, not isolation
>> contention: charge-induced reclaim is a trickle (~400 scans/s vs
>> 10^5-10^6 scans/s in the build runs), isolation never piles up, and
>> the throttle correctly stays dormant on both kernels:
>>
>> unpatched patched
>> single instance 41.6 ops/s 41.2 ops/s
>> (avg 2999ms) (avg 2999ms)
>> 8 shards aggregate 333-419 ops/s 310-358 ops/s
>> (mean 376, (mean 335,
>> 4 runs) (*) 4 runs)
> Hmm, I know this could be noise but it looks like a regression?
>
>> mm_vmscan_throttled 0 0 (all runs)
>> nr_isolated peak 0 0
>>
>> (*) the aggregate is dominated by a preload-end residency lottery
>> (which shards' datasets happen to stay hot) and swings by up to 26%
>> between runs of the SAME kernel; with 4 runs per kernel the
>> difference between the means is within that noise (Welch t=1.95,
>> p~0.11). The throttle never fired in any patched run (the tracepoint
>> covers preload as well), so both kernels execute the same reclaim
>> path here, and the kernel-side counters agree across all 8 runs:
>> steal/scan 0.59-0.63, swap-ins == refaults, zero OOM. These runs
>> document the no-impact side: the check is dormant unless isolation
>> actually piles up.
>>
>> Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
> Hi Hui
>
> Looks like a good idea in general.
>> ---
>> Changelog:
>> v7:
>> Patch 1 was merged into mm-unstable, so this is resent as a single
>> patch.
>> According to the comments of Sashiko, Return instead of waiting when the
>> lruvec has no evictable folios at all: allowed == 0 previously conflated
>> "all empty" with "all over-isolated" and stalled even kswapd for 20ms.
>> Move the v6 cover-letter test data into the commit message.
>> v6:
>> According to the comments of Barry, add real-world workload tests:
>> a CI-style kernel build (repeated "make -j20 vmlinux") inside a
>> memory-limited cgroup with its object tree on dm-delay-backed storage,
>> measured at three pressure levels (1024/1536/2048MB), plus redis-based
>> cache service runs (single instance and 8 shards sharing one cgroup)
>> as no-impact data points. New and updated test scripts are in the
>> gist. Rebased onto the latest mm-stable; patch 2 is adapted to the
>> current isolate_folios() type-selection context (is_single_type_
>> reclaim()), with no semantic change. Re-run the tests and update the
>> test logs.
>> v5:
>> According to the comments of Kairun and feedback from the test scripts,
>> make the throttle check per lruvec (new nr_isolated counter) and skip
>> empty types in the allowed mask.
>> v4:
>> According to the commens of Baolin and Barry, rework patch 2:
>> drop the throttle_is_throttled() helper extracted
>> from shrink_inactive_list() and leave the legacy path untouched.
>> The new MGLRU-only throttle_evictable_types() only sleeps when all
>> evictable types are over-isolated - v3 throttled as soon as any of
>> them was, which unnecessarily blocked the reclaim of the other type
>> and passes the mask of the remaining types to isolate_folios(),
>> which restricts both its initial choice and its fallback, so that
>> isolation never lands on a throttled type. The v3 gate did not
>> constrain the type actually isolated, so the fallback could still
>> pick the over-isolated one.
>> Re-run the tests and update the test log.
>> v3:
>> According to the commens of Baolin, remove the redundant nr_isolated
>> check before restoring the NR_ISOLATED_* counters in evict_folios().
>> rename the extracted helper to throttle_is_throttled() to avoid
>> confusion with the existing wake_throttle_isolated() naming space.
>> Use for_each_evictable_type() in the MGLRU throttle to check each
>> evictable type's isolation instead of only the type returned by
>> get_type_to_scan(), since isolate_folios() may fall back to the
>> other type.
>> Re-run the tests and update the test log.
>> v2:
>> According to the commens of Kairun, Rebased on mm-unstable.
>> Split into two patches; patch 2 is new and adds the
>> too_many_isolated() throttling to the MGLRU eviction path, which v1
>> did not cover.
>> Add test infomations.
>>
>> include/linux/mmzone.h | 2 +
>> mm/vmscan.c | 184 +++++++++++++++++++++++++++++++++++++----
>> 2 files changed, 172 insertions(+), 14 deletions(-)
>>
>> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
>> index 3b96d6c7123b..7759a53326ba 100644
>> --- a/include/linux/mmzone.h
>> +++ b/include/linux/mmzone.h
>> @@ -564,6 +564,8 @@ struct lru_gen_folio {
>> /* can be modified without holding the LRU lock */
>> atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
>> atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
>> + /* the number of folios isolated from this lruvec and being reclaimed */
>> + atomic_long_t nr_isolated[ANON_AND_FILE];
>> /* whether the multi-gen LRU is enabled */
>> bool enabled;
>> /* the memcg generation this lru_gen_folio belongs to */
>> diff --git a/mm/vmscan.c b/mm/vmscan.c
>> index daa7e37c2236..36887117d76a 100644
>> --- a/mm/vmscan.c
>> +++ b/mm/vmscan.c
>> @@ -4332,25 +4332,34 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control
>> sc->priority = clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY);
>> }
>>
>> -static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
>> +static unsigned long lruvec_type_evictable_size(struct lruvec *lruvec, int type)
>> {
>> - int gen, type, zone;
>> + int gen, zone;
>> unsigned long seq, total = 0;
>> struct lru_gen_folio *lrugen = &lruvec->lrugen;
>> DEFINE_MAX_SEQ(lruvec);
>> DEFINE_MIN_SEQ(lruvec);
>>
>> - for_each_evictable_type(type, swappiness) {
>> - for (seq = min_seq[type]; seq <= max_seq; seq++) {
>> - gen = lru_gen_from_seq(seq);
>> - for (zone = 0; zone < MAX_NR_ZONES; zone++)
>> - total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
>> - }
>> + for (seq = min_seq[type]; seq <= max_seq; seq++) {
>> + gen = lru_gen_from_seq(seq);
>> + for (zone = 0; zone < MAX_NR_ZONES; zone++)
>> + total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
>> }
>>
>> return total;
>> }
>>
>> +static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
>> +{
>> + unsigned long total = 0;
>> + int type;
>> +
>> + for_each_evictable_type(type, swappiness)
>> + total += lruvec_type_evictable_size(lruvec, type);
>> +
>> + return total;
>> +}
>> +
> The helper split looks clean; it could be a stand alone commit I think.
>
>> static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)
>> {
>> unsigned long total;
>> @@ -5012,16 +5021,142 @@ static inline bool is_single_type_reclaim(int swappiness)
>> swappiness == SWAPPINESS_ANON_ONLY;
>> }
>>
>> +/*
>> + * The MGLRU counterpart to too_many_isolated().
>> + *
>> + * too_many_isolated() compares node-wide isolated counts against
>> + * node-wide inactive sizes. It predates per-memcg LRU lists, which
>> + * moved isolation to per-lruvec granularity. MGLRU reclaimers contend
>> + * per lruvec, so this check is per lruvec as well, and isolation in
>> + * one memcg does not throttle reclaim in another.
>> + *
>> + * The legacy threshold, the size of the inactive list, is not usable
>> + * here either: in MGLRU the inactive counters are only a compatibility
>> + * shim - they jump around when swap runs out or a memcg hits its swap
>> + * limit, and proactive aging can inflate them. Compare against the
>> + * total number of evictable pages of the type divided by MIN_NR_GENS,
>> + * approximating the size of one generation - the same unit the
>> + * original MGLRU aging heuristics used.
>> + */
>> +static bool lru_gen_too_many_isolated(struct lruvec *lruvec, int type,
>> + struct scan_control *sc)
>> +{
>> + unsigned long isolated, evictable;
>> +
>> + if (current_is_kswapd())
>> + return false;
>> +
>> + if (!writeback_throttling_sane(sc))
>> + return false;
>> +
>> + isolated = atomic_long_read(&lruvec->lrugen.nr_isolated[type]);
>> + evictable = lruvec_type_evictable_size(lruvec, type);
>> +
>> + /*
>> + * GFP_NOIO/GFP_NOFS callers are allowed to isolate more pages, so
>> + * they won't be blocked by normal direct-reclaimers, forming a
>> + * circular deadlock.
>> + */
>> + if (gfp_has_io_fs(sc->gfp_mask))
>> + evictable >>= 3;
>> +
>> + return isolated > evictable / MIN_NR_GENS;
>> +}
> The magic number of >>=3 looks a bit odd, others are fine to me.
>
>> +
>> +/*
>> + * Unlike the legacy path, where the LRU list to isolate from is known
>> + * before isolation, isolate_folios() picks the type from the refault
>> + * feedback and may fall back to the other one. Therefore, instead of
>> + * throttling on a single type, collect the evictable types that do not
>> + * have too many isolated folios, and only sleep when all of them do.
>> + *
>> + * Returns the mask of the types isolate_folios() may isolate from, or
>> + * 0 if reclaim should stop. Also sets @fatal to tell the caller that
>> + * the task received a fatal signal while waiting.
>> + */
>> +static unsigned int throttle_evictable_types(struct lruvec *lruvec,
>> + int swappiness,
>> + struct scan_control *sc,
>> + bool *fatal)
>> +{
>> + unsigned int allowed;
>> + bool stalled = false;
>> + bool has_evictable;
>> + struct pglist_data *pgdat = lruvec_pgdat(lruvec);
>> + int i;
>> +
>> + *fatal = false;
>> +
>> + for (;;) {
>> + allowed = 0;
>> + has_evictable = false;
>> + for_each_evictable_type(i, swappiness) {
>> + /*
>> + * A type with no evictable folios has nothing to
>> + * isolate. Don't allow it, otherwise reclaim can be
>> + * redirected to it and spin making no progress, e.g.
>> + * when the only type with folios is over-isolated.
>> + */
>> + if (!lruvec_type_evictable_size(lruvec, i))
>> + continue;
>> +
>> + has_evictable = true;
>> +
>> + if (!lru_gen_too_many_isolated(lruvec, i, sc))
>> + allowed |= BIT(i);
>> + }
>> +
>> + /*
>> + * Nothing evictable at all is not over-isolation. Stop
>> + * reclaim instead of waiting, so that nobody - including
>> + * kswapd, which is exempt from the check below - stalls on
>> + * an empty lruvec.
>> + */
>> + if (!has_evictable)
>> + return 0;
>> +
>> + if (allowed) {
>> + /* Wake up reclaimers waiting on the isolation to go down. */
>> + wake_throttle_isolated(pgdat);
>> + return allowed;
> This seems counterintuitive: we reach here because a reclaimer is
> starting, and we wake up even more reclaimer?
>
>> + }
>> +
>> + /*
>> + * All evictable types are over-isolated. Like the legacy
>> + * path, wait once for concurrent reclaimers to put their
>> + * isolated folios back; give up if that makes no progress.
>> + */
>> + if (stalled)
>> + return 0;
>> +
>> + stalled = true;
>> + reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED);
>> +
>> + /* We are about to die and free our memory. Return now. */
>> + if (fatal_signal_pending(current)) {
>> + *fatal = true;
>> + return 0;
>> + }
>> + }
>> +}
>> +
> Hmm... this part looks a bit too complex, see below.
>
>> static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
>> struct scan_control *sc, int swappiness,
>> - struct list_head *list, int *isolated,
>> - int *isolate_type, int *isolate_scanned)
>> + unsigned int allowed, struct list_head *list,
>> + int *isolated, int *isolate_type, int *isolate_scanned)
>> {
>> bool type_fallback_allowed = !is_single_type_reclaim(swappiness);
>> int type = get_type_to_scan(lruvec, swappiness);
>> int total_scanned = 0, scanned, tier;
>> bool tried = false;
>>
>> + /*
>> + * The preferred type may have been excluded by
>> + * throttle_evictable_types(); start from the other one.
>> + */
>> + if (!(allowed & BIT(type)))
>> + type = !type;
>> +
>> retry:
>> tier = get_tier_idx(lruvec, type);
>> scanned = scan_folios(nr_to_scan, lruvec, sc,
>> @@ -5036,9 +5171,11 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
>>
>> /*
>> * We are running out of the current reclaim type. Fall back to
>> - * the other type if allowed.
>> + * the other type if allowed. Only fall back to a type that is
>> + * not throttled.
>> */
>> - if (!scanned && !tried && type_fallback_allowed) {
>> + if (!scanned && !tried && type_fallback_allowed &&
>> + (allowed & BIT(!type))) {
>> type = !type;
>> tried = true;
>> type_fallback_allowed = false;
>> @@ -5072,18 +5209,36 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
>> bool skip_retry = false;
>> struct mem_cgroup *memcg = lruvec_memcg(lruvec);
>> struct pglist_data *pgdat = lruvec_pgdat(lruvec);
>> + unsigned int allowed;
>> + bool fatal;
>> +
>> + allowed = throttle_evictable_types(lruvec, swappiness, sc, &fatal);
>> + if (!allowed) {
>> + /*
>> + * We are about to die and free our memory. Like the legacy
>> + * path, pretend some pages were reclaimed so reclaim
>> + * unwinds quickly instead of looping back into the
>> + * throttle.
>> + */
>> + if (fatal)
>> + sc->nr_reclaimed += SWAP_CLUSTER_MAX;
> And reporting a fake nr_reclaimed looks a bad thing to do in the first place.
>
> It seems a lot of trouble is caused by the design of the reclaim loop,
> so will it work better if we calculate the number & type to reclaim
> first like CLRU?
>
> There is a demonstration from Barry:
> https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?h=kairui_scanbalance_0.3
>
> And instead of throttle at the reclaim loop, balance the scan budge
> based on (size - isolation) number, and throttle a while LRU simply
> when the total isolated number is larger than total size /
> MIN_NR_GENS.
>
> Just an idea for reference, seems could help get rid of a lot of complex logic.
Hi Kairui,
Thanks for the pointer. I had a rough look at kairui_scanbalance_0.3 and
I am quite interested in it.
I think that branch still needs an over-isolation guard. The scan budget
answers "which type and how much", but it is computed once per
try_to_shrink_lruvec() and does not see what other reclaimers are
holding. Each reclaimer's in-flight isolation is bounded by the batch
size rather than by the budget, so N reclaimers entering together all
see the same state and all proceed.
On the other hand, that branch would also make the guard much simpler
and clearer. Since evict_folios() already takes the type from its
caller, the allowed mask, the type rewrite and the fallback restriction
in isolate_folios() all go away, and the throttle reduces to a single
check on the totals in the shrink loop, which is basically the second
half of what you suggested.
So I would like to ask what the development plan for this branch is,
and whether you think it is worth moving the MGLRU too_many_isolated()
work over to it.
Best,
Hui
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [PATCH mm-unstable v7] mm/vmscan: throttle MGLRU eviction when isolated folios pile up
2026-10-20 9:28 ` Hui Zhu
@ 2026-10-22 22:20 ` Barry Song
2026-10-23 15:51 ` Kairui Song
1 sibling, 0 replies; 5+ messages in thread
From: Barry Song @ 2026-10-22 22:20 UTC (permalink / raw)
To: Hui Zhu
Cc: Kairui Song, Andrew Morton, Qi Zheng, Shakeel Butt,
Axel Rasmussen, Yuanchu Xie, Wei Xu, Johannes Weiner,
David Hildenbrand, Michal Hocko, Lorenzo Stoakes, Baolin Wang,
linux-mm, linux-kernel, Hui Zhu
On Tue, Oct 20, 2026 at 5:29 PM Hui Zhu <hui.zhu@linux.dev> wrote:
[...]
> >
> > It seems a lot of trouble is caused by the design of the reclaim loop,
> > so will it work better if we calculate the number & type to reclaim
> > first like CLRU?
> >
> > There is a demonstration from Barry:
> > https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?h=kairui_scanbalance_0.3
> >
> > And instead of throttle at the reclaim loop, balance the scan budge
> > based on (size - isolation) number, and throttle a while LRU simply
> > when the total isolated number is larger than total size /
> > MIN_NR_GENS.
> >
> > Just an idea for reference, seems could help get rid of a lot of complex logic.
>
> Hi Kairui,
>
> Thanks for the pointer. I had a rough look at kairui_scanbalance_0.3 and
> I am quite interested in it.
>
> I think that branch still needs an over-isolation guard. The scan budget
> answers "which type and how much", but it is computed once per
> try_to_shrink_lruvec() and does not see what other reclaimers are
> holding. Each reclaimer's in-flight isolation is bounded by the batch
> size rather than by the budget, so N reclaimers entering together all
> see the same state and all proceed.
>
> On the other hand, that branch would also make the guard much simpler
> and clearer. Since evict_folios() already takes the type from its
> caller, the allowed mask, the type rewrite and the fallback restriction
> in isolate_folios() all go away, and the throttle reduces to a single
> check on the totals in the shrink loop, which is basically the second
> half of what you suggested.
>
> So I would like to ask what the development plan for this branch is,
> and whether you think it is worth moving the MGLRU too_many_isolated()
> work over to it.
FYI, here is the RFC:
https://lore.kernel.org/linux-mm/20261022220203.91911-1-baohua@kernel.org/
Best Regards
Barry
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH mm-unstable v7] mm/vmscan: throttle MGLRU eviction when isolated folios pile up
2026-10-20 9:28 ` Hui Zhu
2026-10-22 22:20 ` Barry Song
@ 2026-10-23 15:51 ` Kairui Song
1 sibling, 0 replies; 5+ messages in thread
From: Kairui Song @ 2026-10-23 15:51 UTC (permalink / raw)
To: Hui Zhu
Cc: Andrew Morton, Qi Zheng, Shakeel Butt, Barry Song, Axel Rasmussen,
Yuanchu Xie, Wei Xu, Johannes Weiner, David Hildenbrand,
Michal Hocko, Lorenzo Stoakes, Baolin Wang, linux-mm,
linux-kernel, Hui Zhu
On Tue, Oct 20, 2026 at 5:29 PM Hui Zhu <hui.zhu@linux.dev> wrote:
> 2026/10/17 02:07, Kairui Song:
> > On Thu, Oct 15, 2026 at 4:02 PM Hui Zhu <hui.zhu@linux.dev> wrote:
...
> > Hmm, I know this could be noise but it looks like a regression?
> >
> >> mm_vmscan_throttled 0 0 (all runs)
> >> nr_isolated peak 0 0
> >>
> >> (*) the aggregate is dominated by a preload-end residency lottery
> >> (which shards' datasets happen to stay hot) and swings by up to 26%
> >> between runs of the SAME kernel; with 4 runs per kernel the
> >> difference between the means is within that noise (Welch t=1.95,
> >> p~0.11). The throttle never fired in any patched run (the tracepoint
> >> covers preload as well), so both kernels execute the same reclaim
> >> path here, and the kernel-side counters agree across all 8 runs:
> >> steal/scan 0.59-0.63, swap-ins == refaults, zero OOM. These runs
> >> document the no-impact side: the check is dormant unless isolation
> >> actually piles up.
> >>
> >> Signed-off-by: Hui Zhu <zhuhui@kylinos.cn>
> > Hi Hui
> >
> > Looks like a good idea in general.
> >> ---
> >> Changelog:
> >> v7:
> >> Patch 1 was merged into mm-unstable, so this is resent as a single
> >> patch.
> >> According to the comments of Sashiko, Return instead of waiting when the
> >> lruvec has no evictable folios at all: allowed == 0 previously conflated
> >> "all empty" with "all over-isolated" and stalled even kswapd for 20ms.
> >> Move the v6 cover-letter test data into the commit message.
> >> v6:
> >> According to the comments of Barry, add real-world workload tests:
> >> a CI-style kernel build (repeated "make -j20 vmlinux") inside a
> >> memory-limited cgroup with its object tree on dm-delay-backed storage,
> >> measured at three pressure levels (1024/1536/2048MB), plus redis-based
> >> cache service runs (single instance and 8 shards sharing one cgroup)
> >> as no-impact data points. New and updated test scripts are in the
> >> gist. Rebased onto the latest mm-stable; patch 2 is adapted to the
> >> current isolate_folios() type-selection context (is_single_type_
> >> reclaim()), with no semantic change. Re-run the tests and update the
> >> test logs.
> >> v5:
> >> According to the comments of Kairun and feedback from the test scripts,
> >> make the throttle check per lruvec (new nr_isolated counter) and skip
> >> empty types in the allowed mask.
> >> v4:
> >> According to the commens of Baolin and Barry, rework patch 2:
> >> drop the throttle_is_throttled() helper extracted
> >> from shrink_inactive_list() and leave the legacy path untouched.
> >> The new MGLRU-only throttle_evictable_types() only sleeps when all
> >> evictable types are over-isolated - v3 throttled as soon as any of
> >> them was, which unnecessarily blocked the reclaim of the other type
> >> and passes the mask of the remaining types to isolate_folios(),
> >> which restricts both its initial choice and its fallback, so that
> >> isolation never lands on a throttled type. The v3 gate did not
> >> constrain the type actually isolated, so the fallback could still
> >> pick the over-isolated one.
> >> Re-run the tests and update the test log.
> >> v3:
> >> According to the commens of Baolin, remove the redundant nr_isolated
> >> check before restoring the NR_ISOLATED_* counters in evict_folios().
> >> rename the extracted helper to throttle_is_throttled() to avoid
> >> confusion with the existing wake_throttle_isolated() naming space.
> >> Use for_each_evictable_type() in the MGLRU throttle to check each
> >> evictable type's isolation instead of only the type returned by
> >> get_type_to_scan(), since isolate_folios() may fall back to the
> >> other type.
> >> Re-run the tests and update the test log.
> >> v2:
> >> According to the commens of Kairun, Rebased on mm-unstable.
> >> Split into two patches; patch 2 is new and adds the
> >> too_many_isolated() throttling to the MGLRU eviction path, which v1
> >> did not cover.
> >> Add test infomations.
> >>
> >> include/linux/mmzone.h | 2 +
> >> mm/vmscan.c | 184 +++++++++++++++++++++++++++++++++++++----
> >> 2 files changed, 172 insertions(+), 14 deletions(-)
> >>
> >> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> >> index 3b96d6c7123b..7759a53326ba 100644
> >> --- a/include/linux/mmzone.h
> >> +++ b/include/linux/mmzone.h
> >> @@ -564,6 +564,8 @@ struct lru_gen_folio {
> >> /* can be modified without holding the LRU lock */
> >> atomic_long_t evicted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
> >> atomic_long_t refaulted[NR_HIST_GENS][ANON_AND_FILE][MAX_NR_TIERS];
> >> + /* the number of folios isolated from this lruvec and being reclaimed */
> >> + atomic_long_t nr_isolated[ANON_AND_FILE];
> >> /* whether the multi-gen LRU is enabled */
> >> bool enabled;
> >> /* the memcg generation this lru_gen_folio belongs to */
> >> diff --git a/mm/vmscan.c b/mm/vmscan.c
> >> index daa7e37c2236..36887117d76a 100644
> >> --- a/mm/vmscan.c
> >> +++ b/mm/vmscan.c
> >> @@ -4332,25 +4332,34 @@ static void set_initial_priority(struct pglist_data *pgdat, struct scan_control
> >> sc->priority = clamp(priority, DEF_PRIORITY / 2, DEF_PRIORITY);
> >> }
> >>
> >> -static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
> >> +static unsigned long lruvec_type_evictable_size(struct lruvec *lruvec, int type)
> >> {
> >> - int gen, type, zone;
> >> + int gen, zone;
> >> unsigned long seq, total = 0;
> >> struct lru_gen_folio *lrugen = &lruvec->lrugen;
> >> DEFINE_MAX_SEQ(lruvec);
> >> DEFINE_MIN_SEQ(lruvec);
> >>
> >> - for_each_evictable_type(type, swappiness) {
> >> - for (seq = min_seq[type]; seq <= max_seq; seq++) {
> >> - gen = lru_gen_from_seq(seq);
> >> - for (zone = 0; zone < MAX_NR_ZONES; zone++)
> >> - total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
> >> - }
> >> + for (seq = min_seq[type]; seq <= max_seq; seq++) {
> >> + gen = lru_gen_from_seq(seq);
> >> + for (zone = 0; zone < MAX_NR_ZONES; zone++)
> >> + total += max(READ_ONCE(lrugen->nr_pages[gen][type][zone]), 0L);
> >> }
> >>
> >> return total;
> >> }
> >>
> >> +static unsigned long lruvec_evictable_size(struct lruvec *lruvec, int swappiness)
> >> +{
> >> + unsigned long total = 0;
> >> + int type;
> >> +
> >> + for_each_evictable_type(type, swappiness)
> >> + total += lruvec_type_evictable_size(lruvec, type);
> >> +
> >> + return total;
> >> +}
> >> +
> > The helper split looks clean; it could be a stand alone commit I think.
> >
> >> static bool lruvec_is_sizable(struct lruvec *lruvec, struct scan_control *sc)
> >> {
> >> unsigned long total;
> >> @@ -5012,16 +5021,142 @@ static inline bool is_single_type_reclaim(int swappiness)
> >> swappiness == SWAPPINESS_ANON_ONLY;
> >> }
> >>
> >> +/*
> >> + * The MGLRU counterpart to too_many_isolated().
> >> + *
> >> + * too_many_isolated() compares node-wide isolated counts against
> >> + * node-wide inactive sizes. It predates per-memcg LRU lists, which
> >> + * moved isolation to per-lruvec granularity. MGLRU reclaimers contend
> >> + * per lruvec, so this check is per lruvec as well, and isolation in
> >> + * one memcg does not throttle reclaim in another.
> >> + *
> >> + * The legacy threshold, the size of the inactive list, is not usable
> >> + * here either: in MGLRU the inactive counters are only a compatibility
> >> + * shim - they jump around when swap runs out or a memcg hits its swap
> >> + * limit, and proactive aging can inflate them. Compare against the
> >> + * total number of evictable pages of the type divided by MIN_NR_GENS,
> >> + * approximating the size of one generation - the same unit the
> >> + * original MGLRU aging heuristics used.
> >> + */
> >> +static bool lru_gen_too_many_isolated(struct lruvec *lruvec, int type,
> >> + struct scan_control *sc)
> >> +{
> >> + unsigned long isolated, evictable;
> >> +
> >> + if (current_is_kswapd())
> >> + return false;
> >> +
> >> + if (!writeback_throttling_sane(sc))
> >> + return false;
> >> +
> >> + isolated = atomic_long_read(&lruvec->lrugen.nr_isolated[type]);
> >> + evictable = lruvec_type_evictable_size(lruvec, type);
> >> +
> >> + /*
> >> + * GFP_NOIO/GFP_NOFS callers are allowed to isolate more pages, so
> >> + * they won't be blocked by normal direct-reclaimers, forming a
> >> + * circular deadlock.
> >> + */
> >> + if (gfp_has_io_fs(sc->gfp_mask))
> >> + evictable >>= 3;
> >> +
> >> + return isolated > evictable / MIN_NR_GENS;
> >> +}
> > The magic number of >>=3 looks a bit odd, others are fine to me.
> >
> >> +
> >> +/*
> >> + * Unlike the legacy path, where the LRU list to isolate from is known
> >> + * before isolation, isolate_folios() picks the type from the refault
> >> + * feedback and may fall back to the other one. Therefore, instead of
> >> + * throttling on a single type, collect the evictable types that do not
> >> + * have too many isolated folios, and only sleep when all of them do.
> >> + *
> >> + * Returns the mask of the types isolate_folios() may isolate from, or
> >> + * 0 if reclaim should stop. Also sets @fatal to tell the caller that
> >> + * the task received a fatal signal while waiting.
> >> + */
> >> +static unsigned int throttle_evictable_types(struct lruvec *lruvec,
> >> + int swappiness,
> >> + struct scan_control *sc,
> >> + bool *fatal)
> >> +{
> >> + unsigned int allowed;
> >> + bool stalled = false;
> >> + bool has_evictable;
> >> + struct pglist_data *pgdat = lruvec_pgdat(lruvec);
> >> + int i;
> >> +
> >> + *fatal = false;
> >> +
> >> + for (;;) {
> >> + allowed = 0;
> >> + has_evictable = false;
> >> + for_each_evictable_type(i, swappiness) {
> >> + /*
> >> + * A type with no evictable folios has nothing to
> >> + * isolate. Don't allow it, otherwise reclaim can be
> >> + * redirected to it and spin making no progress, e.g.
> >> + * when the only type with folios is over-isolated.
> >> + */
> >> + if (!lruvec_type_evictable_size(lruvec, i))
> >> + continue;
> >> +
> >> + has_evictable = true;
> >> +
> >> + if (!lru_gen_too_many_isolated(lruvec, i, sc))
> >> + allowed |= BIT(i);
> >> + }
> >> +
> >> + /*
> >> + * Nothing evictable at all is not over-isolation. Stop
> >> + * reclaim instead of waiting, so that nobody - including
> >> + * kswapd, which is exempt from the check below - stalls on
> >> + * an empty lruvec.
> >> + */
> >> + if (!has_evictable)
> >> + return 0;
> >> +
> >> + if (allowed) {
> >> + /* Wake up reclaimers waiting on the isolation to go down. */
> >> + wake_throttle_isolated(pgdat);
> >> + return allowed;
> > This seems counterintuitive: we reach here because a reclaimer is
> > starting, and we wake up even more reclaimer?
> >
> >> + }
> >> +
> >> + /*
> >> + * All evictable types are over-isolated. Like the legacy
> >> + * path, wait once for concurrent reclaimers to put their
> >> + * isolated folios back; give up if that makes no progress.
> >> + */
> >> + if (stalled)
> >> + return 0;
> >> +
> >> + stalled = true;
> >> + reclaim_throttle(pgdat, VMSCAN_THROTTLE_ISOLATED);
> >> +
> >> + /* We are about to die and free our memory. Return now. */
> >> + if (fatal_signal_pending(current)) {
> >> + *fatal = true;
> >> + return 0;
> >> + }
> >> + }
> >> +}
> >> +
> > Hmm... this part looks a bit too complex, see below.
> >
> >> static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
> >> struct scan_control *sc, int swappiness,
> >> - struct list_head *list, int *isolated,
> >> - int *isolate_type, int *isolate_scanned)
> >> + unsigned int allowed, struct list_head *list,
> >> + int *isolated, int *isolate_type, int *isolate_scanned)
> >> {
> >> bool type_fallback_allowed = !is_single_type_reclaim(swappiness);
> >> int type = get_type_to_scan(lruvec, swappiness);
> >> int total_scanned = 0, scanned, tier;
> >> bool tried = false;
> >>
> >> + /*
> >> + * The preferred type may have been excluded by
> >> + * throttle_evictable_types(); start from the other one.
> >> + */
> >> + if (!(allowed & BIT(type)))
> >> + type = !type;
> >> +
> >> retry:
> >> tier = get_tier_idx(lruvec, type);
> >> scanned = scan_folios(nr_to_scan, lruvec, sc,
> >> @@ -5036,9 +5171,11 @@ static int isolate_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
> >>
> >> /*
> >> * We are running out of the current reclaim type. Fall back to
> >> - * the other type if allowed.
> >> + * the other type if allowed. Only fall back to a type that is
> >> + * not throttled.
> >> */
> >> - if (!scanned && !tried && type_fallback_allowed) {
> >> + if (!scanned && !tried && type_fallback_allowed &&
> >> + (allowed & BIT(!type))) {
> >> type = !type;
> >> tried = true;
> >> type_fallback_allowed = false;
> >> @@ -5072,18 +5209,36 @@ static int evict_folios(unsigned long nr_to_scan, struct lruvec *lruvec,
> >> bool skip_retry = false;
> >> struct mem_cgroup *memcg = lruvec_memcg(lruvec);
> >> struct pglist_data *pgdat = lruvec_pgdat(lruvec);
> >> + unsigned int allowed;
> >> + bool fatal;
> >> +
> >> + allowed = throttle_evictable_types(lruvec, swappiness, sc, &fatal);
> >> + if (!allowed) {
> >> + /*
> >> + * We are about to die and free our memory. Like the legacy
> >> + * path, pretend some pages were reclaimed so reclaim
> >> + * unwinds quickly instead of looping back into the
> >> + * throttle.
> >> + */
> >> + if (fatal)
> >> + sc->nr_reclaimed += SWAP_CLUSTER_MAX;
> > And reporting a fake nr_reclaimed looks a bad thing to do in the first place.
> >
> > It seems a lot of trouble is caused by the design of the reclaim loop,
> > so will it work better if we calculate the number & type to reclaim
> > first like CLRU?
> >
> > There is a demonstration from Barry:
> > https://git.kernel.org/pub/scm/linux/kernel/git/baohua/linux.git/log/?h=kairui_scanbalance_0.3
> >
> > And instead of throttle at the reclaim loop, balance the scan budge
> > based on (size - isolation) number, and throttle a while LRU simply
> > when the total isolated number is larger than total size /
> > MIN_NR_GENS.
> >
> > Just an idea for reference, seems could help get rid of a lot of complex logic.
>
> Hi Kairui,
>
> Thanks for the pointer. I had a rough look at kairui_scanbalance_0.3 and
> I am quite interested in it.
>
> I think that branch still needs an over-isolation guard. The scan budget
> answers "which type and how much", but it is computed once per
> try_to_shrink_lruvec() and does not see what other reclaimers are
> holding. Each reclaimer's in-flight isolation is bounded by the batch
> size rather than by the budget, so N reclaimers entering together all
> see the same state and all proceed.
Yeah, exactly.
> On the other hand, that branch would also make the guard much simpler
> and clearer. Since evict_folios() already takes the type from its
> caller, the allowed mask, the type rewrite and the fallback restriction
> in isolate_folios() all go away, and the throttle reduces to a single
> check on the totals in the shrink loop, which is basically the second
> half of what you suggested.
>
> So I would like to ask what the development plan for this branch is,
> and whether you think it is worth moving the MGLRU too_many_isolated()
> work over to it.
Thanks for looking into that!
I think the isolation protection is definitely needed; really
appreciate your work!
However, many things are currently happening in parallel for MGLRU.
Will the code be cleaner if worked on based on that branch? And
MGLRU's budge will be more similiar with CLRU, so I hope maybe it can
even be unified with CLRU for this isolation protection, just with a
different gating (size / MIN_NR_GEN vs size of inactive), will be much
lower LOC.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2026-10-23 15:51 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-15 8:01 [PATCH mm-unstable v7] mm/vmscan: throttle MGLRU eviction when isolated folios pile up Hui Zhu
2026-10-16 18:07 ` Kairui Song
2026-10-20 9:28 ` Hui Zhu
2026-10-22 22:20 ` Barry Song
2026-10-23 15:51 ` Kairui Song
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox