[PATCH mm-new v1 0/2] mm: vmscan: skip anon folios zswap would refuse when writeback is off

[PATCH mm-new v1 0/2] mm: vmscan: skip anon folios zswap would refuse when writeback is off

[Jiayuan Chen]

memory.zswap.writeback=0 is what we set for latency sensitive services:
their cold anon pages may go to zswap, but never to disk, a swapin
from disk is a latency spike they cannot afford. Such a memcg can only
swap into zswap. When zswap stops taking its pages, because the pool
hit max_pool_percent or the memcg hit memory.zswap.max, reclaim still
goes through the motions on its anon LRU: every folio it picks gets a
swap slot, is unmapped, is refused by zswap_store() and is activated
again. Nothing is freed, the folio now sits in the swap cache holding
a slot, and the owner takes a minor fault on the next access.

memory.reclaim below is just the easiest way to trigger it. The same
loop runs whenever the memcg is reclaimed: when it hits memory.max, or
when kswapd walks it under node pressure. Neither is something the
service can opt out of.

Patch 1 is a refactor so the memcg zswap check can be reused from
reclaim. Patch 2 adds the check to shrink_folio_list(). It cannot go
into can_reclaim_anon_pages(): zero-filled folios never reach zswap
and must stay reclaimable.

Numbers from a 16-CPU, 32G QEMU guest on mm-new, zswap with
max_pool_percent=2 (~640M), a memcg with memory.zswap.writeback=0
holding 3G of 2:1 compressible anon. The pool fills up after ~300k
pages. Then 5 x "echo 512M > memory.reclaim", and the task walks its
3G once:

                     minor faults on the walk
                     MGLRU        classic LRU
  before             786,454      786,452
  after              301,168      301,158

The ~300k after the patch are the pages that really are in zswap
(stored_pages was 301,133 / 301,165), swapped back in once each. The
other ~485k before the patch are pages zswap refused: unmapped, left
in the swap cache, mapped again on the next touch.

Jiayuan Chen (2):
  mm: memcontrol: split mem_cgroup_may_zswap() out of
    obj_cgroup_may_zswap()
  mm: vmscan: skip anon folios zswap would refuse when writeback is off

 include/linux/memcontrol.h |  5 +++++
 include/linux/zswap.h      |  5 +++++
 mm/memcontrol.c            | 26 ++++++++++++++++++--------
 mm/page_io.c               | 15 ++++++++++++++-
 mm/swap.h                  | 12 ++++++++++++
 mm/vmscan.c                | 17 +++++++++++++++++
 mm/zswap.c                 | 11 +++++++++++
 7 files changed, 82 insertions(+), 9 deletions(-)

-- 
2.43.0

[PATCH mm-new v1 1/2] mm: memcontrol: split mem_cgroup_may_zswap() out of obj_cgroup_may_zswap()

[Jiayuan Chen]

zswap_store() asks obj_cgroup_may_zswap() whether the memcg is under
its zswap limit. The next patch wants to ask the same question from
reclaim, where it has a memcg but no objcg.

Move the walk into mem_cgroup_may_zswap() taking a memcg and make
obj_cgroup_may_zswap() a wrapper around it. Either way the answer is
the one zswap_store() gets.

No functional change.

Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
---
 include/linux/memcontrol.h |  5 +++++
 mm/memcontrol.c            | 26 ++++++++++++++++++--------
 2 files changed, 23 insertions(+), 8 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 74110a324f9e..dab15db2c61a 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1943,11 +1943,16 @@ static inline void mem_cgroup_calculate_protection_path(struct mem_cgroup *root,
 #endif
 
 #if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP)
+bool mem_cgroup_may_zswap(struct mem_cgroup *memcg);
 bool obj_cgroup_may_zswap(struct obj_cgroup *objcg);
 void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size);
 void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size);
 bool mem_cgroup_zswap_writeback_enabled(const struct mem_cgroup *memcg);
 #else
+static inline bool mem_cgroup_may_zswap(struct mem_cgroup *memcg)
+{
+	return true;
+}
 static inline bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
 {
 	return true;
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index aad0498a7bd6..f32105a08a91 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -6261,8 +6261,8 @@ static struct cftype swap_files[] = {
 
 #ifdef CONFIG_ZSWAP
 /**
- * obj_cgroup_may_zswap - check if this cgroup can zswap
- * @objcg: the object cgroup
+ * mem_cgroup_may_zswap - check if this cgroup can zswap
+ * @memcg: the memory cgroup
  *
  * Check if the hierarchical zswap limit has been reached.
  *
@@ -6272,17 +6272,14 @@ static struct cftype swap_files[] = {
  * spending cycles on compression when there is already no room left
  * or zswap is disabled altogether somewhere in the hierarchy.
  */
-bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
+bool mem_cgroup_may_zswap(struct mem_cgroup *memcg)
 {
-	struct mem_cgroup *memcg, *original_memcg;
 	bool ret = true;
 
 	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
 		return true;
 
-	original_memcg = get_mem_cgroup_from_objcg(objcg);
-	for (memcg = original_memcg; !mem_cgroup_is_root(memcg);
-	     memcg = parent_mem_cgroup(memcg)) {
+	for (; !mem_cgroup_is_root(memcg); memcg = parent_mem_cgroup(memcg)) {
 		unsigned long max = READ_ONCE(memcg->zswap_max);
 		unsigned long pages;
 
@@ -6301,7 +6298,20 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
 		ret = false;
 		break;
 	}
-	mem_cgroup_put(original_memcg);
+	return ret;
+}
+
+bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
+{
+	struct mem_cgroup *memcg;
+	bool ret;
+
+	if (!cgroup_subsys_on_dfl(memory_cgrp_subsys))
+		return true;
+
+	memcg = get_mem_cgroup_from_objcg(objcg);
+	ret = mem_cgroup_may_zswap(memcg);
+	mem_cgroup_put(memcg);
 	return ret;
 }
 
-- 
2.43.0

[PATCH mm-new v1 2/2] mm: vmscan: skip anon folios zswap would refuse when writeback is off

[Jiayuan Chen]

A memcg with memory.zswap.writeback=0 can only swap into zswap. Once
zswap stops taking its pages, because the pool is at its limit or the
memcg is at memory.zswap.max, swap_writeout() has nowhere to put a
dirty folio and returns AOP_WRITEPAGE_ACTIVATE.

shrink_folio_list() only learns that at the very end. Every folio it
picks gets a swap slot, is unmapped, fails in swap_writeout() and goes
back to the active list, to be aged and tried again. Each round costs
a slot, an rmap walk, a TLB flush and a minor fault for the owner, and
frees nothing. The folio also stays in the swap cache holding its slot.

The obvious place for a zswap check is can_reclaim_anon_pages(), but
it cannot go there. Zero-filled folios never reach zswap:
swap_writeout() marks them in the swap table and returns before
zswap_store() is called. They need no pool space, they are common, and
a memcg-wide "no anon" decision would keep them in memory for as long
as zswap is full. With zswap full they must still be reclaimable.

So decide per folio in shrink_folio_list() instead. When
swap_writeout() is bound to return AOP_WRITEPAGE_ACTIVATE for a folio,
activate it right there, before it costs a swap slot and an unmap.
Zero-filled folios go on as before (that check reads the whole folio,
so it comes after the zswap one and only runs when zswap is blocked).
A swap cache folio with a clean flag may still be dirty in the page
table; it goes on too, unmap settles it and the next round sees the
dirty flag. An unmapped swap cache folio goes on as well: nobody holds
its slot any more, so swap_writeout() drops it in folio_free_swap()
and the folio is freed without a store. Large folios go on too. They
may still be split, before the swap allocation or when it fails, and
the base folios then come back through this check, zero-filled ones go
on, the rest is stopped. Keeping the split path as it is keeps the
change small.
zswap_store_blocked() asks zswap_check_limits() itself, the same call
zswap_store() makes first, so the pool-full state keeps being updated
while vmscan turns folios away and pool_limit_hit counts once per
batch instead of once per folio. A refused store would have kicked
the zswap shrinker to make room, so zswap_store_blocked() does that
too.

The check is made once per batch, at the first folio it could stop,
and the whole batch goes by it. A batch of file folios never asks
zswap anything. zswap can change state under us while the batch is
being worked through, that is fine: at worst the tail of one batch
takes the old path and is refused in swap_writeout() as before, or is
activated once more. Most folios are still stopped early, and the next
batch sees the new state.

Only anon folios are handled. shmem splits large folios and zeroes
fallocated ones on its own way to swap, that is not for vmscan to
second-guess.

Signed-off-by: Jiayuan Chen <jiayuan.chen@linux.dev>
---
 include/linux/zswap.h |  5 +++++
 mm/page_io.c          | 15 ++++++++++++++-
 mm/swap.h             | 12 ++++++++++++
 mm/vmscan.c           | 17 +++++++++++++++++
 mm/zswap.c            | 11 +++++++++++
 5 files changed, 59 insertions(+), 1 deletion(-)

diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index df6cafbe95dc..b64fcdeda575 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -33,6 +33,7 @@ void zswap_swapoff(int type);
 void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg);
 void zswap_lruvec_state_init(struct lruvec *lruvec);
 void zswap_folio_swapin(struct folio *folio);
+bool zswap_store_blocked(struct mem_cgroup *memcg);
 bool zswap_is_enabled(void);
 bool zswap_never_enabled(void);
 #else
@@ -62,6 +63,10 @@ static inline void zswap_swapoff(int type) {}
 static inline void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg) {}
 static inline void zswap_lruvec_state_init(struct lruvec *lruvec) {}
 static inline void zswap_folio_swapin(struct folio *folio) {}
+static inline bool zswap_store_blocked(struct mem_cgroup *memcg)
+{
+	return false;
+}
 
 static inline bool zswap_is_enabled(void)
 {
diff --git a/mm/page_io.c b/mm/page_io.c
index c6824fcd483e..b4a9fe401a3c 100644
--- a/mm/page_io.c
+++ b/mm/page_io.c
@@ -125,7 +125,7 @@ int generic_swapfile_activate(struct swap_info_struct *sis,
 	goto out;
 }
 
-static bool is_folio_zero_filled(struct folio *folio)
+bool is_folio_zero_filled(struct folio *folio)
 {
 	unsigned int pos, last_pos;
 	unsigned long *data;
@@ -197,6 +197,19 @@ static void swap_zeromap_folio_clear(struct folio *folio)
 	swap_cluster_unlock(ci);
 }
 
+/*
+ * True when swap_writeout() would only activate the dirty folios of @memcg:
+ * zswap refuses them and they may not go to the device. Zero-filled folios
+ * are the exception, see swap_writeout().
+ */
+bool swap_writeout_blocked(struct mem_cgroup *memcg)
+{
+	/* The device takes the folio, also when zswap is off */
+	if (mem_cgroup_zswap_writeback_enabled(memcg))
+		return false;
+	return zswap_store_blocked(memcg);
+}
+
 /*
  * We may have stale swap cache pages in memory: notice
  * them here and get rid of the unnecessary final write.
diff --git a/mm/swap.h b/mm/swap.h
index d5bf21f517dc..51d00e37870c 100644
--- a/mm/swap.h
+++ b/mm/swap.h
@@ -259,6 +259,8 @@ void swap_read_submit(struct swap_io_ctx *ctx);
 void swap_write_submit(struct swap_io_ctx *ctx);
 int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio);
 void __swap_writeout(struct swap_io_ctx *ctx, struct folio *folio);
+bool swap_writeout_blocked(struct mem_cgroup *memcg);
+bool is_folio_zero_filled(struct folio *folio);
 
 /* linux/mm/swap_state.c */
 extern struct address_space swap_space __read_mostly;
@@ -428,6 +430,16 @@ static inline int swap_writeout(struct swap_io_ctx *ctx, struct folio *folio)
 	return 0;
 }
 
+static inline bool swap_writeout_blocked(struct mem_cgroup *memcg)
+{
+	return false;
+}
+
+static inline bool is_folio_zero_filled(struct folio *folio)
+{
+	return false;
+}
+
 static inline int swap_retry_table_alloc(swp_entry_t entry, gfp_t gfp)
 {
 	return -EINVAL;
diff --git a/mm/vmscan.c b/mm/vmscan.c
index daa7e37c2236..d732cd378b55 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1169,6 +1169,7 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 	unsigned int nr_reclaimed = 0, nr_demoted = 0;
 	unsigned int pgactivate = 0;
 	bool do_demote_pass;
+	int swap_blocked = -1;
 	struct swap_io_ctx ctx = {};
 
 	folio_batch_init(&free_folios);
@@ -1355,6 +1356,22 @@ static unsigned int shrink_folio_list(struct list_head *folio_list,
 			continue;
 		}
 
+		/*
+		 * swap_writeout() would only activate this folio. Do that now,
+		 * before it costs a swap slot and an unmap. Zero-filled folios
+		 * are marked in the swap table without a store, an unmapped one
+		 * may only need its stale slot dropped and a large one may still
+		 * be split, let them go on.
+		 */
+		if (folio_test_anon(folio) && folio_test_swapbacked(folio) &&
+		    folio_mapped(folio) && !folio_test_large(folio) &&
+		    (!folio_test_swapcache(folio) || folio_test_dirty(folio))) {
+			if (swap_blocked < 0)
+				swap_blocked = swap_writeout_blocked(memcg);
+			if (swap_blocked && !is_folio_zero_filled(folio))
+				goto activate_locked;
+		}
+
 		/*
 		 * Anonymous process memory has backing store?
 		 * Try to allocate it some swap space here.
diff --git a/mm/zswap.c b/mm/zswap.c
index 572075bb1c1c..468bdec6931a 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -566,6 +566,17 @@ static bool zswap_check_limits(void)
 	return zswap_pool_reached_full;
 }
 
+/* True when zswap_store() would refuse every folio of @memcg right now */
+bool zswap_store_blocked(struct mem_cgroup *memcg)
+{
+	if (zswap_check_limits()) {
+		/* A refused store would kick the shrinker, do the same */
+		queue_work(shrink_wq, &zswap_shrink_work);
+		return true;
+	}
+	return !mem_cgroup_may_zswap(memcg);
+}
+
 /*********************************
 * param callbacks
 **********************************/
-- 
2.43.0