[PATCH v6 0/2] mm/vmscan: fix NR_ISOLATED accounting and throttle MGLRU eviction

[Hui Zhu <hui.zhu@linux.dev>]

From: Hui Zhu <zhuhui@kylinos.cn>

Barry asked for a real-world workload that suffers from the missing
throttling. This version adds one: a CI-style kernel build inside a
memory-limited cgroup with slow storage (mglru_build_repro.sh),
measured at three pressure levels, plus redis-based cache service runs
as no-impact data points.

The series is also rebased onto the latest mm-stable; patch 2 is
adapted to the current isolate_folios() type-selection context
(is_single_type_reclaim()), with no semantic change.

The legacy reclaim path has two mechanisms around isolated folios that
MGLRU lacks:

1. NR_ISOLATED_ANON/FILE counters are updated when folios are isolated
   from the inactive lists. Compaction's too_many_isolated() relies on
   them to decide when to back off. The MGLRU reclaim path never
   updates them, so compaction cannot see MGLRU's in-flight isolation.

2. shrink_inactive_list() throttles direct reclaim via
   too_many_isolated() when isolated folios pile up. MGLRU's
   evict_folios() isolates folios without any such check, so many
   concurrent reclaimers can over-isolate the same (oldest) generation,
   leading to unnecessary swapping, thrashing and premature memcg OOM.

Patch 1 fixes the counter accounting in the MGLRU isolation path.
Patch 2 adds a per-lruvec throttle, mirroring the legacy behavior but
adapted to MGLRU's per-lruvec contention and its dynamic type
selection/fallback.

Testing
=======
Five test scripts are provided: the two synthetic reproducers from v5
and three real-world workload harnesses new in v6. All are available
at:
https://gist.github.com/teawater/3ef51251f2e91a5a600e3d26bb477e34

Common environment: 10 CPU / 8GB QEMU guest, cgroup v2, MGLRU enabled
(0x0007). "unpatched" and "patched" are the same tree built
without/with this series; every configuration is run with identical
parameters on both kernels. Each cell below is one run; the build
metrics are cumulative counters over a 1800s window, which smooths
run-to-run noise.

Real workload 1: CI kernel build (mglru_build_repro.sh)
-------------------------------------------------------
Models a CI build pod (k8s / OBS / gitlab-runner): repeated
from-scratch "make -j20 vmlinux" rounds inside a memory-limited
cgroup, with the object tree on ext4 over dm-delay (10ms) to model
slow/congested storage. 20 concurrent compilers charge into one memcg
and enter direct reclaim at the same time, while dirty object files
stretch the isolate->shrink->putback window - the workload shape
too_many_isolated() was invented for. Each kernel runs 1800s per
pressure level; the headline metric is the number of objects actually
compiled in that window (a full round is ~3,800 objects).

memory.max = 1024MB (extreme pressure):
                        unpatched       patched
  objects compiled      4,744           7,211 (+52%)
  pgscan_direct         2,901,083,521   2,228,776,807 (-23%)
  scan efficiency       0.195           0.242
  workingset_refault    565,169,241     537,686,479
  mm_vmscan_throttled   0               511,441 (all
                                        VMSCAN_THROTTLE_ISOLATED)
  nr_isolated peak      0 (invisible)   1,152

memory.max = 1536MB (heavy pressure):
                        unpatched       patched
  objects compiled      18,969          25,124 (+32%)
  pgscan_direct         1,223,905,508   816,812,074 (-33%)
  scan efficiency       0.320           0.338
  workingset_refault    390,048,986     273,442,445 (-30%)
  memcg charge fails    8,344,775       5,868,167 (-30%)
  round wall time       199-391s        182-275s
  mm_vmscan_throttled   0               503,117
  nr_isolated peak      0 (invisible)   1,053

At both levels -j20 is oversubscribed (20 cc1 processes need ~3GB of
anon in total), so rounds end with a cc1 being memcg-OOM-killed on
either kernel. Normalized per completed round the kill rate is the
same (13.2 vs 13.1 at 1024MB, 5.8 vs 6.1 at 1536MB); the patched
kernel simply reaches that point sooner because less time is burned
thrashing, so it cycles through more rounds. The patches do not claim
to avoid these OOMs - the point is that the patched kernel performs
23-33% fewer futile LRU scans and gets 32-52% more build work done in
the same wall time.

memory.max = 2048MB (light pressure - no-harm control):
                        unpatched       patched
  rounds completed      6               6
  avg wall time/round   271s            270s
  scan efficiency       0.984           0.978
  mm_vmscan_throttled   0               0
  nr_isolated peak      0 (invisible)   144

When reclaim keeps up, the throttle never fires and both kernels
perform identically. Patch 1 still shows: the unpatched kernel reports
nr_isolated=0 even while isolation is in flight, which is exactly the
blind spot it fixes.

Real workload 2: redis cache service (mglru_redis_repro.sh,
mglru_redis_shards_repro.sh)
-------------------------------------------------------------
A redis-server serving a 1KB-value dataset larger than its 96MB
cgroup, with swap on dm-delay (50ms): first as a single instance
(100k keys, 200 clients), then as 8 shards with AOF persistence on a
second dm-delay device sharing one cgroup (8 concurrent chargers).
The bottleneck here is swap-in I/O serialized on redis's single
command thread, not isolation contention: charge-induced reclaim is a
trickle (~400 scans/s vs 10^5-10^6 scans/s in the build runs),
isolation never piles up, and the throttle correctly stays dormant on
both kernels:

                        unpatched       patched
  single instance       41.6 ops/s      41.2 ops/s
                        (avg 2999ms)    (avg 2999ms)
  8 shards aggregate    333-419 ops/s   310-358 ops/s
                        (mean 376,      (mean 335,
                        4 runs) (*)     4 runs)
  mm_vmscan_throttled   0               0 (all runs)
  nr_isolated peak      0               0

(*) the aggregate is dominated by a preload-end residency lottery
(which shards' datasets happen to stay hot) and swings by up to 26%
between runs of the SAME kernel; with 4 runs per kernel the
between-kernel difference of the means is within that noise (Welch
t=1.95, p~0.11). The throttle never fired in any patched run (the
tracepoint covers preload as well), so both kernels execute the same
reclaim path here, and the kernel-side counters agree across all 8
runs: steal/scan 0.59-0.63, swap-ins == refaults, zero OOM. These
runs document the no-impact side: the check is dormant unless
isolation actually piles up.

Synthetic reproducers (unchanged from v5)
-----------------------------------------
mglru_iso_repro.sh (64 threads, 48MB working set, 60s):
Drives concurrent direct reclaim inside the memcg and measures scan
efficiency, throttle events, in-flight isolation and throughput.
Neither kernel OOMs at this concurrency; the value of the patch shows
in reclaim quality:
                        unpatched       patched
  OOM kills             0               0
  mm_vmscan_throttled   0               35645 (all
                                        VMSCAN_THROTTLE_ISOLATED)
  nr_isolated peak      0 (invisible)   230
  total touches         246,499,132,369 301,456,426,692
  scan efficiency       0.0261          0.0194

The patched kernel completes ~22% more work in the same 60s: the
throttle keeps concurrent reclaimers from trampling the same
generation, so less CPU is burned in reclaim. Note nr_isolated is
always 0 on the unpatched kernel - the over-isolation is invisible
there, which is exactly what patch 1 fixes.

mglru_iso_repro_v2.sh (192 threads, 48MB working set, 60s):
Raises concurrency to the point where over-isolation becomes fatal:
                        unpatched       patched
  OOM kills             1 (task killed) 0
  memcg oom events      51              0
  mm_vmscan_throttled   0               512292 (all
                                        VMSCAN_THROTTLE_ISOLATED)
  total touches         0 (killed)      682,300,000,972

With 192 threads the unpatched kernel cannot keep reclaim ahead of
allocation and the task is OOM-killed; the patched kernel survives the
full run and keeps reclaim making progress.

Changelog:
v6:
According to the comments of Barry, add real-world workload tests:
a CI-style kernel build (repeated "make -j20 vmlinux") inside a
memory-limited cgroup with its object tree on dm-delay-backed storage,
measured at three pressure levels (1024/1536/2048MB), plus redis-based
cache service runs (single instance and 8 shards sharing one cgroup)
as no-impact data points. New and updated test scripts are in the
gist. Rebased onto the latest mm-stable; patch 2 is adapted to the
current isolate_folios() type-selection context (is_single_type_
reclaim()), with no semantic change. Re-run the tests and update the
test logs.
v5:
According to the comments of Kairun and feedback from the test scripts,
make the throttle check per lruvec (new nr_isolated counter) and skip
empty types in the allowed mask.
v4:
According to the commens of Baolin and Barry, rework patch 2:
drop the throttle_is_throttled() helper extracted
from shrink_inactive_list() and leave the legacy path untouched.
The new MGLRU-only throttle_evictable_types() only sleeps when all
evictable types are over-isolated - v3 throttled as soon as any of
them was, which unnecessarily blocked the reclaim of the other type
and passes the mask of the remaining types to isolate_folios(),
which restricts both its initial choice and its fallback, so that
isolation never lands on a throttled type.  The v3 gate did not
constrain the type actually isolated, so the fallback could still
pick the over-isolated one.
Re-run the tests and update the test log.
v3:
According to the commens of Baolin, remove the redundant nr_isolated
check before restoring the NR_ISOLATED_* counters in evict_folios().
rename the extracted helper to throttle_is_throttled() to avoid
confusion with the existing wake_throttle_isolated() naming space.
Use for_each_evictable_type() in the MGLRU throttle to check each
evictable type's isolation instead of only the type returned by
get_type_to_scan(), since isolate_folios() may fall back to the
other type.
Re-run the tests and update the test log.
v2:
According to the commens of Kairun, Rebased on mm-unstable.
Split into two patches; patch 2 is new and adds the
too_many_isolated() throttling to the MGLRU eviction path, which v1
did not cover.
Add test infomations.

Hui Zhu (2):
  mm/vmscan: fix missing NR_ISOLATED counter update in MGLRU reclaim
    path
  mm/vmscan: throttle MGLRU eviction when isolated folios pile up

 include/linux/mmzone.h |   2 +
 mm/vmscan.c            | 174 ++++++++++++++++++++++++++++++++++++++---
 2 files changed, 163 insertions(+), 13 deletions(-)

-- 
2.43.0