Linux-mm Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v4 0/3] mm: zswap: free cold writeback folios promptly
@ 2026-08-25 13:52 Alexandre Ghiti
  2026-08-25 13:52 ` [PATCH v4 1/3] mm: swap: move LRU insertion out of the swap cache allocator Alexandre Ghiti
                   ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Alexandre Ghiti @ 2026-08-25 13:52 UTC (permalink / raw)
  To: Johannes Weiner, Yosry Ahmed, Nhat Pham, Andrew Morton, Chris Li,
	Kairui Song
  Cc: Kairui Song, Chengming Zhou, Matthew Wilcox (Oracle), Jan Kara,
	Kemeng Shi, Baoquan He, Barry Song, Youngjun Park, Alexander Viro,
	Christian Brauner, David Hildenbrand, Lorenzo Stoakes,
	Michal Hocko, Axel Rasmussen, Qi Zheng, Shakeel Butt, Wei Xu,
	Yuanchu Xie, Kunwu Chan, linux-mm, linux-kernel, linux-fsdevel,
	Alexandre Ghiti

When zswap writes an entry back, it allocates an order-0 swap cache folio,
decompresses into it, and issues the write. The folio is cold by
construction, yet today it is left on the LRU for page reclaim to find and
free later. That wastes a reclaim scan and keeps cold memory resident
longer than necessary.

Rather than implement this in zswap, extend the existing dropbehind
mechanism to swap cache folios and have zswap opt into it (Yosry). A
PG_dropbehind folio is already dropped from its cache once writeback
completes instead of being left for reclaim; for a swap cache folio that
"drop" is removing it from the swap cache.

  Patch 1 - move LRU insertion out of the swap cache allocator into its
            callers, so zswap writeback can allocate off the LRU.

  Patch 2 - drop dropbehind swap cache folios on writeback completion.

  Patch 3 - zswap allocates its writeback folio off the LRU and marks it
            dropbehind, opting into the mechanism above.

Note: patch 1 also appears as patch 1 of the zswap writeback refault
series [1]. It is the same patch. Both series need it and both are meant
to apply on their own, so it is posted in each; whichever lands first, the
other should drop it.

This version is based on Linus' tree rather than mm-unstable, because it
builds on Tal Zussman's BIO_COMPLETE_IN_TASK work merged in the 7.3 block
pull, which has not reached the mm tree yet. Thanks to Matthew and Barry for
pointing out this series!

v1: https://lore.kernel.org/linux-mm/20260718093723.153324-1-alex@ghiti.fr/
v2: https://lore.kernel.org/linux-mm/20260727143618.1582318-1-alex@ghiti.fr/
v3: https://lore.kernel.org/linux-mm/20260818163221.589352-1-alex@ghiti.fr/

[1] https://lore.kernel.org/linux-mm/20260821093606.2231216-1-alex@ghiti.fr/

Changes in v4:
- Rebase on BIO_COMPLETE_IN_TASK: set it on dropbehind swap writeback like
  the file dropbehind paths do, and drop the folio directly from
  folio_end_writeback(). This removes the per-CPU llist, the workqueue and
  the reuse of folio->lru as the list node.
- zswap now drops its folio reference before starting writeback, so the
  swap cache holds the only one and remove_mapping() sees the refcount it
  expects. This fixes the drop on synchronous-IO devices and the race
  Sashiko reported, where the drop could run before zswap released its
  reference and fall back to the LRU. Verified on zram (the only
  SWP_SYNCHRONOUS_IO backend I have): over ~6.7M writebacks per run, 99.999%
  of the folios are dropped, and the refcount fallback fires 37-50 times.
- Use remove_mapping_reclaim() rather than adding a boolean argument to
  remove_mapping(), which keeps the calling code readable (David). This also
  leaves the existing remove_mapping() callers untouched.
- Patch 1: correct the changelog. The folio has to stay off the LRU because
  folio_add_lru() leaves a reference in the per-CPU LRU batch, not because
  of the free-time page-flag checks. Measured on zram, adding the folio to
  the LRU instead drops the freed rate from 99.999% to 2.7%.

Changes in v3:
- Drop the synchronous-IO special case in zswap writeback (Yosry, Nhat).
- Use mem_cgroup_tryget()/mem_cgroup_put(): struct mem_cgroup is only
  defined under CONFIG_MEMCG, so css_tryget()/css_put() failed to build
  with CONFIG_MEMCG=n.

Changes in v2:
- Make swap dropbehind a generic core-mm mechanism that zswap opts into,
  rather than a zswap-specific implementation (Yosry).
- Allocate off the LRU by moving folio_add_lru() out of the swap cache
  allocator into its callers; rename it to __swap_cache_alloc_folio()
  (Kairui).
- Skip the folio in the free path if it is still under writeback (Nhat).

Results
-------
Paired baseline vs series on async swap (NVMe). Each
workload runs confined to a memory cgroup (memory.max) small enough to force
zswap shrinker writeback.

Kernel build (defconfig, make -j4; memory.max = 600M):

  metric            baseline       series      delta
  pgrotated           441028         2521     -99.4%
  pgsteal_direct     3524343      2869004     -18.6%
  pgscan_direct      8393791      7765008      -7.5%
  zswpwb              705129       699019      -0.9%
  build time (s)        1155         1114      -3.6%

Of the 699019 folios written back, 698990 (99.996%) were freed promptly on
writeback completion; only 28 fell back to reclaim.

MySQL/OLTP (sysbench, 10 tables x 1M rows, 512M buffer pool, 8 threads, 300s;
memory.max = 256M):

  metric            baseline       series      delta
  transactions/s      153.87       163.30      +6.1%
  p95 latency (ms)    157.42       145.82      -7.4%
  avg latency (ms)     52.09        49.05      -5.9%
  pgrotated           743738        22460     -97.0%
  pgsteal_direct     6886490      5445278     -20.9%
  pgscan_direct     13462510     10820730     -19.6%

Future work
-----------
Barry suggested extending this to MADV_PAGEOUT and general reclaim. I
prototyped dropbehind for all reclaimed swap folios and it regressed
sysbench OLTP throughput by ~15% on NVMe swap: dropping the swap cache
immediately turns cheap in-cache refaults into disk reads and collapses
swap readahead clustering. Neither blk-wbt, mq-deadline nor a PG_workingset
gate recovered it. MADV_PAGEOUT alone may still be worth it, since there
userspace has explicitly declared the range cold, but I have not measured
that case in isolation yet.

Alexandre Ghiti (3):
  mm: swap: move LRU insertion out of the swap cache allocator
  mm: swap: drop dropbehind swap cache folios on writeback completion
  mm: zswap: drop cold writeback folios via swap dropbehind

 include/linux/swap.h |  5 ++++
 mm/filemap.c         | 19 ++++++++++++++
 mm/page_io.c         |  7 +++++
 mm/swap.h            |  6 ++---
 mm/swap_state.c      | 61 ++++++++++++++++++++++++++++++++++++++------
 mm/vmscan.c          | 48 ++++++++++++++++++++++++++--------
 mm/zswap.c           | 22 +++++++++++++---
 7 files changed, 143 insertions(+), 25 deletions(-)


base-commit: 55ab7e14222e5f0b0fd9f7711ca391d2924b35e3
-- 
2.53.0-Meta



^ permalink raw reply	[flat|nested] 12+ messages in thread
* [PATCH v4 0/3] mm: fix workingset refaults in the zswap writeback path
@ 2026-09-11  9:20 Alexandre Ghiti
  2026-09-11  9:20 ` [PATCH v4 1/3] mm: swap: move LRU insertion out of the swap cache allocator Alexandre Ghiti
  0 siblings, 1 reply; 12+ messages in thread
From: Alexandre Ghiti @ 2026-09-11  9:20 UTC (permalink / raw)
  To: Johannes Weiner, Yosry Ahmed, Nhat Pham, Chengming Zhou,
	Andrew Morton, David Hildenbrand, Lorenzo Stoakes,
	Liam R. Howlett, Vlastimil Babka, Mike Rapoport,
	Suren Baghdasaryan, Michal Hocko, Hugh Dickins, Baolin Wang,
	Chris Li, Kairui Song, Kemeng Shi, Baoquan He, Barry Song,
	Youngjun Park, Qi Zheng, Shakeel Butt, Axel Rasmussen,
	Yuanchu Xie, Wei Xu, Joonsoo Kim
  Cc: linux-mm, linux-kernel, Alexandre Ghiti

Note: patch 1 also appears as patch 1 of the zswap dropbehind series [1].
It is the same change. Both series need it and both are meant to apply on
their own, so it is posted in each; whichever lands first, the other
should drop it.

When an anonymous folio is reclaimed, workingset_eviction() stores a
"shadow" (the eviction cookie) in the swap slot so that a later swap-in
can be recognised as a refault and, if the refault distance is short
enough, the page can be re-activated. This is how anon workingset/refault
detection has worked since commit aae466b0052e ("mm/swap: implement
workingset detection for anonymous LRU").

zswap writeback breaks this in two independent ways:

  - Over-count at writeback: the shrinker allocates a buffer folio in the
    swap cache, and the allocation path counts that folio as a refault.

  - Lost eviction cookie at reclaim: adding the buffer to the swap cache
    overwrites the slot's shadow, so the original cookie is lost; when the
    buffer folio is finally reclaimed a fresh, inaccurate cookie is minted
    in its place.

This series preserves the shadow within zswap itself, without any new
swap-table or swap-slot state. At writeback, instead of erasing the freed
zswap entry, the captured shadow is parked in the zswap tree in its place,
so it outlives the writeback buffer folio.

Finally, the refault evaluation is moved out of the swap-cache allocator
into the swap-in callers, so allocating the writeback buffer is no longer
miscounted as a refault.

Results
-------

Measured with a sysbench OLTP (MariaDB) workload in a memory cgroup sized
so the dataset and InnoDB buffer pool both overcommit it, with the zswap
shrinker on so entries are continuously written back to an NVMe swap
device (classic LRU; MGLRU off). Anon workingset counters over the
measured window, baseline vs this series, mean +/- stddev over 10 runs:

  workingset_refault_anon    383,242 +/- 59,345  ->  199,355 +/- 27,042   -48%
  workingset_activate_anon    54,890 +/- 11,742  ->   22,607 +/-  3,118   -59%
  workingset_restore_anon     16,212 +/-  4,590  ->    7,599 +/-  1,190   -53%

Writeback volume is comparable (zswpwb 183k +/- 16k -> 178k +/- 15k), so
the reduction is not from doing less work. Normalised per transaction the
reduction holds (-53%/-49%/-39%) while the swap work per transaction is
unchanged. A kernel build under the same pressure moves all three
counters in the same direction.

The run-to-run variance of these counters drops as well.

Throughput is unaffected: over the same 10 runs, transactions/s is
18.50 +/- 1.08 -> 18.65 +/- 0.94, i.e. +0.8% with a 95% confidence
interval of +/- 7.7%.

Changes in v4:
- Usama pointed out that patch 1's changelog referred to "the next patch"
  and to the "upcoming" dropbehind series, which does not work for a patch
  posted in two series. Both references are gone.
- Kunwu Chan pointed out that mm/swapfile.c still refers to the function
  by its old name in a comment; patch 1 renames it there too.
- Usama pointed out that reading the slot's shadow in the swap-in path
  before allocating races: the allocation can sleep, so another swap-in
  can install a folio, have it reclaimed and leave a newer shadow behind,
  and this caller would then win the insertion but refault against the
  stale snapshot. __swap_cache_alloc_folio() now hands the shadow back
  from __swap_cache_add_check(), which captures it under ci->lock at the
  point the displacing insertion succeeds. zswap writeback took the same
  racy snapshot and now uses the same output.

Changes in v3:
- David pointed out that the boolean added to workingset_refault() in v2
  makes the calling code hard to read. The boolean is now internal to
  mm/workingset.c and the callers use workingset_refault() or
  workingset_refault_lru_managed(), the same way remove_mapping() and
  remove_mapping_reclaim() do. The three existing callers are left
  untouched.
- zswap_writeback_entry() now restores the shadow into the slot on the
  error paths taken after the buffer was allocated: the allocation had
  already overwritten it and nothing was parked yet, so it was lost.
- Dropped the "if (!shadow) shadow = ZSWAP_WRITEBACK_NO_SHADOW" fallback,
  which can never be taken: the swap table already marks a swapped out
  slot with xa_mk_value(0), so swap_cache_get_shadow() never returns NULL
  for one.
- Rebased on mm-new.

Changes in v2:
- Sashiko pointed out that v1 evaluated the refault after folio_add_lru(),
  which picks the MGLRU generation before PG_workingset is set. Patch 1
  now moves the LRU insertion out of the swap cache allocator so the
  refault is evaluated before it, as it was originally.
- Sashiko also pointed out that a failed writeback redirties the buffer
  and leaves it in the swap cache with its shadow still parked, so a
  later zswap_store() on it would hand the parked value to
  zswap_entry_free(). zswap_store() now bails out for such a folio:
  writeback already decided that data belongs on disk, so it is written
  there instead of being compressed again, which also keeps the parked
  shadow intact until the folio leaves the swap cache.
- The buffer folio a swap-in consumes is already on the LRU, so the
  refault there cannot activate it by setting PG_active: that leaves the
  flag disagreeing with the list the folio is on, which shows up as an
  mm/memcontrol.c lru_size underflow when it is freed. Such a folio is now
  activated with folio_activate() instead. One still sitting in a per-CPU
  batch cannot be moved safely and just misses the activation; a counter
  on that path measured 0.0007% of the activations over a 10 run test.
  This is only needed because zswap writeback still puts its buffer on the
  LRU: once it stops doing so, workingset_refault_lru_managed() has no
  caller left and can go away.

[1] https://lore.kernel.org/linux-mm/20260825135209.3135169-2-alex@ghiti.fr/

v1: https://lore.kernel.org/all/20260817144622.137133-1-alex@ghiti.fr/
v2: https://lore.kernel.org/linux-mm/20260821093606.2231216-1-alex@ghiti.fr/
v3: https://lore.kernel.org/linux-mm/20260825172604.3243589-1-alex@ghiti.fr/

Alexandre Ghiti (3):
  mm: swap: move LRU insertion out of the swap cache allocator
  mm: swap: refault on swap-in, not in the swap cache allocator
  mm: zswap: preserve the workingset shadow across writeback

 include/linux/zswap.h |  12 +++++
 mm/internal.h         |   1 +
 mm/memory.c           |   5 ++
 mm/shmem.c            |   5 ++
 mm/swap.h             |   7 +--
 mm/swap_state.c       |  52 +++++++++++++++-----
 mm/swapfile.c         |   2 +-
 mm/vmscan.c           |   4 +-
 mm/workingset.c       |  60 ++++++++++++++++++-----
 mm/zswap.c            | 108 ++++++++++++++++++++++++++++++++++++++++--
 10 files changed, 224 insertions(+), 32 deletions(-)


base-commit: 1a46b1e97bde62afa7d925bb0dcd9f9748a1d7c3
-- 
2.53.0-Meta



^ permalink raw reply	[flat|nested] 12+ messages in thread
end of thread, other threads:[~2026-09-11  9:21 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-08-25 13:52 [PATCH v4 0/3] mm: zswap: free cold writeback folios promptly Alexandre Ghiti
2026-08-25 13:52 ` [PATCH v4 1/3] mm: swap: move LRU insertion out of the swap cache allocator Alexandre Ghiti
2026-09-08  9:37   ` Kunwu Chan
2026-09-08 18:26   ` Nhat Pham
2026-08-25 13:52 ` [PATCH v4 2/3] mm: swap: drop dropbehind swap cache folios on writeback completion Alexandre Ghiti
2026-09-08  9:58   ` Kunwu Chan
2026-09-08 18:39   ` Nhat Pham
2026-08-25 13:52 ` [PATCH v4 3/3] mm: zswap: drop cold writeback folios via swap dropbehind Alexandre Ghiti
2026-08-25 15:51   ` Yosry Ahmed
2026-08-25 16:57     ` Alexandre Ghiti
2026-09-08  8:24   ` Kunwu Chan
  -- strict thread matches above, loose matches on Subject: below --
2026-09-11  9:20 [PATCH v4 0/3] mm: fix workingset refaults in the zswap writeback path Alexandre Ghiti
2026-09-11  9:20 ` [PATCH v4 1/3] mm: swap: move LRU insertion out of the swap cache allocator Alexandre Ghiti

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox