[PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[Hao Jia]

From: Hao Jia <jiahao1@lixiang.com>

Zswap writeback when the global pool limit is hit fails when memory
cgroup is disabled. The pool remains full until it is organically
drained by swapins or memory freeing, leading to zswap store failures
and pages bypassing getting written directly to the backing swap device,
causing LRU inversion (hotter pages with higher fault latency).

This happens because mem_cgroup_iter() always returns NULL when
memory cgroups are disabled. As a result, the global shrinker
shrink_worker() repeatedly takes empty walks. After MAX_RECLAIM_RETRIES
failed attempts, the worker gives up without writing back any pages.

Therefore, when memory cgroup is disabled, fall through with the !memcg
branch and shrink the root memcg directly.

With memcg disabled, shrink_memcg() only returns -ENOENT when the root
LRU is empty, which means the total pages are already below thr. In the
absence of heavy concurrent zswap stores, the loop then safely bails out
via the zswap_total_pages() <= thr check; otherwise, it will resume
shrinking the memcg after processing the reschedule check. For any other
return value from shrink_memcg(), the loop is guaranteed to terminate,
either after MAX_RECLAIM_RETRIES failures or once the threshold is met.

Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware")
Cc: stable@vger.kernel.org
Suggested-by: Nhat Pham <nphamcs@gmail.com>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Acked-by: Yosry Ahmed <yosry@kernel.org>
Reported-by: Yosry Ahmed <yosry@kernel.org>
Closes: https://lore.kernel.org/all/CAO9r8zPVzMKFbCixxD-qgtRrkFxWVrHiZZeLc=eyTPKPVQgX4g@mail.gmail.com
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
---
 mm/zswap.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)

diff --git a/mm/zswap.c b/mm/zswap.c
index b5a17ea20237..48fc7b575e24 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1356,11 +1356,12 @@ static void shrink_worker(struct work_struct *w)
 		} while (memcg && !mem_cgroup_tryget_online(memcg));
 		spin_unlock(&zswap_shrink_lock);
 
-		if (!memcg) {
-			/*
-			 * Continue shrinking without incrementing failures if
-			 * we found candidate memcgs in the last tree walk.
-			 */
+		/*
+		 * A NULL memcg ends a full hierarchy pass (except when memcg is
+		 * disabled, where it is always NULL: fall through to the root LRU).
+		 * Count a failure only if the last pass found no candidates.
+		 */
+		if (!memcg && !mem_cgroup_disabled()) {
 			if (!attempts && ++failures == MAX_RECLAIM_RETRIES)
 				break;
 
@@ -1379,7 +1380,7 @@ static void shrink_worker(struct work_struct *w)
 		 * and failures.
 		 */
 		if (ret == -ENOENT)
-			continue;
+			goto resched;
 		++attempts;
 
 		if (ret && ++failures == MAX_RECLAIM_RETRIES)
-- 
2.34.1

Re: [PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[Andrew Morton]

On Wed, 29 Jul 2026 16:42:05 +0800 Hao Jia <jiahao.kernel@gmail.com> wrote:

> Zswap writeback when the global pool limit is hit fails when memory
> cgroup is disabled. The pool remains full until it is organically
> drained by swapins or memory freeing, leading to zswap store failures
> and pages bypassing getting written directly to the backing swap device,
> causing LRU inversion (hotter pages with higher fault latency).
> 
> This happens because mem_cgroup_iter() always returns NULL when
> memory cgroups are disabled. As a result, the global shrinker
> shrink_worker() repeatedly takes empty walks. After MAX_RECLAIM_RETRIES
> failed attempts, the worker gives up without writing back any pages.
> 
> Therefore, when memory cgroup is disabled, fall through with the !memcg
> branch and shrink the root memcg directly.
> 
> With memcg disabled, shrink_memcg() only returns -ENOENT when the root
> LRU is empty, which means the total pages are already below thr. In the
> absence of heavy concurrent zswap stores, the loop then safely bails out
> via the zswap_total_pages() <= thr check; otherwise, it will resume
> shrinking the memcg after processing the reschedule check. For any other
> return value from shrink_memcg(), the loop is guaranteed to terminate,
> either after MAX_RECLAIM_RETRIES failures or once the threshold is met.
> 
> Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware")
> Cc: stable@vger.kernel.org

How does this affect users?  What behavior do they observe when it
occurs?

> Closes: https://lore.kernel.org/all/CAO9r8zPVzMKFbCixxD-qgtRrkFxWVrHiZZeLc=eyTPKPVQgX4g@mail.gmail.com

hm, that isn't really a bug report and doesn't answer the above
question.


AI review asked a couple of questions:
	https://sashiko.dev/#/patchset/20260729084206.77793-1-jiahao.kernel@gmail.com

Thanks.

Re: [PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[Yosry Ahmed]

On Wed, Jul 29, 2026 at 3:58 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Wed, 29 Jul 2026 16:42:05 +0800 Hao Jia <jiahao.kernel@gmail.com> wrote:
>
> > Zswap writeback when the global pool limit is hit fails when memory
> > cgroup is disabled. The pool remains full until it is organically
> > drained by swapins or memory freeing, leading to zswap store failures
> > and pages bypassing getting written directly to the backing swap device,
> > causing LRU inversion (hotter pages with higher fault latency).
> >
> > This happens because mem_cgroup_iter() always returns NULL when
> > memory cgroups are disabled. As a result, the global shrinker
> > shrink_worker() repeatedly takes empty walks. After MAX_RECLAIM_RETRIES
> > failed attempts, the worker gives up without writing back any pages.
> >
> > Therefore, when memory cgroup is disabled, fall through with the !memcg
> > branch and shrink the root memcg directly.
> >
> > With memcg disabled, shrink_memcg() only returns -ENOENT when the root
> > LRU is empty, which means the total pages are already below thr. In the
> > absence of heavy concurrent zswap stores, the loop then safely bails out
> > via the zswap_total_pages() <= thr check; otherwise, it will resume
> > shrinking the memcg after processing the reschedule check. For any other
> > return value from shrink_memcg(), the loop is guaranteed to terminate,
> > either after MAX_RECLAIM_RETRIES failures or once the threshold is met.
> >
> > Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware")
> > Cc: stable@vger.kernel.org
>
> How does this affect users?  What behavior do they observe when it
> occurs?

I think the first paragraph sums it up pretty well, especially the
last sentence "hotter pages with higher fault latency".

>
> > Closes: https://lore.kernel.org/all/CAO9r8zPVzMKFbCixxD-qgtRrkFxWVrHiZZeLc=eyTPKPVQgX4g@mail.gmail.com
>
> hm, that isn't really a bug report and doesn't answer the above
> question.

Yeah, it isn't. Probably we should drop "Closes". I assume Hao added
it because checkpatch annoyingly complains if you add "Reported-by"
without "Closes", so Hao just linked to the thread where I pointed out
the bug.

>
> AI review asked a couple of questions:
>         https://sashiko.dev/#/patchset/20260729084206.77793-1-jiahao.kernel@gmail.com

The review on patch #1 is something theoretical, we discussed it at
length in previous versions.

For patch #2:

> Does this batching logic break NUMA fairness?
>
> Because for_each_node_state() always starts from the lowest node
> ID and breaks when the scan budget is exhausted, subsequent
> calls to shrink_memcg() will restart at the lowest node ID again.
>
> If the lowest node (typically Node 0) consistently has enough
> items to exhaust the scan budget, wouldn't we exclusively evict
> pages from it while ignoring older pages on other nodes? Could
> this cause LRU inversion across nodes, keeping older pages in
> memory on Node 1 while hot pages on Node 0 are evicted?

Yes, unfairness is possible.

For global shrinking, it's probably not an issue. We reclaim until we
hit the acceptance threshold and it's very unlikely this will happen
before iterating all nodes (given that the batch size is 32 pages).
However, with the shrink_memcg() path, we only reclaim one batch, so
there's a chance we'll always reclaim it from node 0.

Maybe we should just drop the early bailout and accept potentially
doing more writeback than needed. Hao, WDYT?

If you respin, please also drop the batch size argument to
shrink_memcg() as it's now always SWAP_CLUSTER_MAX.

Re: [PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[Andrew Morton]

On Wed, 29 Jul 2026 17:30:56 -0700 Yosry Ahmed <yosry@kernel.org> wrote:

> On Wed, Jul 29, 2026 at 3:58 PM Andrew Morton <akpm@linux-foundation.org> wrote:
> >
> > On Wed, 29 Jul 2026 16:42:05 +0800 Hao Jia <jiahao.kernel@gmail.com> wrote:
> >
> > > Zswap writeback when the global pool limit is hit fails when memory
> > > cgroup is disabled. The pool remains full until it is organically
> > > drained by swapins or memory freeing, leading to zswap store failures
> > > and pages bypassing getting written directly to the backing swap device,
> > > causing LRU inversion (hotter pages with higher fault latency).
> > >
> > > This happens because mem_cgroup_iter() always returns NULL when
> > > memory cgroups are disabled. As a result, the global shrinker
> > > shrink_worker() repeatedly takes empty walks. After MAX_RECLAIM_RETRIES
> > > failed attempts, the worker gives up without writing back any pages.
> > >
> > > Therefore, when memory cgroup is disabled, fall through with the !memcg
> > > branch and shrink the root memcg directly.
> > >
> > > With memcg disabled, shrink_memcg() only returns -ENOENT when the root
> > > LRU is empty, which means the total pages are already below thr. In the
> > > absence of heavy concurrent zswap stores, the loop then safely bails out
> > > via the zswap_total_pages() <= thr check; otherwise, it will resume
> > > shrinking the memcg after processing the reschedule check. For any other
> > > return value from shrink_memcg(), the loop is guaranteed to terminate,
> > > either after MAX_RECLAIM_RETRIES failures or once the threshold is met.
> > >
> > > Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware")
> > > Cc: stable@vger.kernel.org
> >
> > How does this affect users?  What behavior do they observe when it
> > occurs?
> 
> I think the first paragraph sums it up pretty well, especially the
> last sentence "hotter pages with higher fault latency".

How do users observe that?

See, what I'm looking for here is an explanation for why we're
proposing a backport.  How are our users harmed by the current code and
how does this change benefit them?

> >
> > > Closes: https://lore.kernel.org/all/CAO9r8zPVzMKFbCixxD-qgtRrkFxWVrHiZZeLc=eyTPKPVQgX4g@mail.gmail.com
> >
> > hm, that isn't really a bug report and doesn't answer the above
> > question.
> 
> Yeah, it isn't. Probably we should drop "Closes". I assume Hao added
> it because checkpatch annoyingly complains if you add "Reported-by"
> without "Closes", so Hao just linked to the thread where I pointed out
> the bug.

OK.

Re: [PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[Yosry Ahmed]

On Wed, Jul 29, 2026 at 6:17 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>
> On Wed, 29 Jul 2026 17:30:56 -0700 Yosry Ahmed <yosry@kernel.org> wrote:
>
> > On Wed, Jul 29, 2026 at 3:58 PM Andrew Morton <akpm@linux-foundation.org> wrote:
> > >
> > > On Wed, 29 Jul 2026 16:42:05 +0800 Hao Jia <jiahao.kernel@gmail.com> wrote:
> > >
> > > > Zswap writeback when the global pool limit is hit fails when memory
> > > > cgroup is disabled. The pool remains full until it is organically
> > > > drained by swapins or memory freeing, leading to zswap store failures
> > > > and pages bypassing getting written directly to the backing swap device,
> > > > causing LRU inversion (hotter pages with higher fault latency).
> > > >
> > > > This happens because mem_cgroup_iter() always returns NULL when
> > > > memory cgroups are disabled. As a result, the global shrinker
> > > > shrink_worker() repeatedly takes empty walks. After MAX_RECLAIM_RETRIES
> > > > failed attempts, the worker gives up without writing back any pages.
> > > >
> > > > Therefore, when memory cgroup is disabled, fall through with the !memcg
> > > > branch and shrink the root memcg directly.
> > > >
> > > > With memcg disabled, shrink_memcg() only returns -ENOENT when the root
> > > > LRU is empty, which means the total pages are already below thr. In the
> > > > absence of heavy concurrent zswap stores, the loop then safely bails out
> > > > via the zswap_total_pages() <= thr check; otherwise, it will resume
> > > > shrinking the memcg after processing the reschedule check. For any other
> > > > return value from shrink_memcg(), the loop is guaranteed to terminate,
> > > > either after MAX_RECLAIM_RETRIES failures or once the threshold is met.
> > > >
> > > > Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware")
> > > > Cc: stable@vger.kernel.org
> > >
> > > How does this affect users?  What behavior do they observe when it
> > > occurs?
> >
> > I think the first paragraph sums it up pretty well, especially the
> > last sentence "hotter pages with higher fault latency".
>
> How do users observe that?
>
> See, what I'm looking for here is an explanation for why we're
> proposing a backport.  How are our users harmed by the current code and
> how does this change benefit them?

It's a potential performance regression when using zswap with memcg
disabled. Sorry I just realized you were looking for a high level
description not detailed symptoms :)

Re: [PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[Andrew Morton]

On Thu, 30 Jul 2026 09:52:38 -0700 Yosry Ahmed <yosry@kernel.org> wrote:

> > > >
> > > > How does this affect users?  What behavior do they observe when it
> > > > occurs?
> > >
> > > I think the first paragraph sums it up pretty well, especially the
> > > last sentence "hotter pages with higher fault latency".
> >
> > How do users observe that?
> >
> > See, what I'm looking for here is an explanation for why we're
> > proposing a backport.  How are our users harmed by the current code and
> > how does this change benefit them?
> 
> It's a potential performance regression when using zswap with memcg
> disabled. Sorry I just realized you were looking for a high level
> description not detailed symptoms :)

Are we able to guess how significant this is?  It should have
significant impact to justify a backport.

Sorry to push - I'm forever trying to train people to describe the
userspace-visible effects so we can make these backporting decisions!

And to justify those decisions to -stable maintainers

And so that someone who is experiencing a slowdown in some private
kernel branch can look at the changelog and decide whether this might
fix it for them.

Re: [PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[Yosry Ahmed]

On Thu, Jul 30, 2026 at 10:59 AM Andrew Morton
<akpm@linux-foundation.org> wrote:
>
> On Thu, 30 Jul 2026 09:52:38 -0700 Yosry Ahmed <yosry@kernel.org> wrote:
>
> > > > >
> > > > > How does this affect users?  What behavior do they observe when it
> > > > > occurs?
> > > >
> > > > I think the first paragraph sums it up pretty well, especially the
> > > > last sentence "hotter pages with higher fault latency".
> > >
> > > How do users observe that?
> > >
> > > See, what I'm looking for here is an explanation for why we're
> > > proposing a backport.  How are our users harmed by the current code and
> > > how does this change benefit them?
> >
> > It's a potential performance regression when using zswap with memcg
> > disabled. Sorry I just realized you were looking for a high level
> > description not detailed symptoms :)
>
> Are we able to guess how significant this is?  It should have
> significant impact to justify a backport.

It depends on the workload. I think a lot of workloads may not notice
tbh, I don't think it's common to use zswap without cgroups.

> Sorry to push - I'm forever trying to train people to describe the
> userspace-visible effects so we can make these backporting decisions!
>
> And to justify those decisions to -stable maintainers
>
> And so that someone who is experiencing a slowdown in some private
> kernel branch can look at the changelog and decide whether this might
> fix it for them.

I don't think it's super important to backport, but it is also
something that we broke at some point and the fix is straightforward
and simple, so imo the risk/reward function is leaning toward
backporting it just in case. That being said, I don't feel strongly.

Re: [PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[Hao Jia]

On 2026/7/30 08:30, Yosry Ahmed wrote:
> On Wed, Jul 29, 2026 at 3:58 PM Andrew Morton <akpm@linux-foundation.org> wrote:
>>
>> On Wed, 29 Jul 2026 16:42:05 +0800 Hao Jia <jiahao.kernel@gmail.com> wrote:
>>
>>> Zswap writeback when the global pool limit is hit fails when memory
>>> cgroup is disabled. The pool remains full until it is organically
>>> drained by swapins or memory freeing, leading to zswap store failures
>>> and pages bypassing getting written directly to the backing swap device,
>>> causing LRU inversion (hotter pages with higher fault latency).
>>>
>>> This happens because mem_cgroup_iter() always returns NULL when
>>> memory cgroups are disabled. As a result, the global shrinker
>>> shrink_worker() repeatedly takes empty walks. After MAX_RECLAIM_RETRIES
>>> failed attempts, the worker gives up without writing back any pages.
>>>
>>> Therefore, when memory cgroup is disabled, fall through with the !memcg
>>> branch and shrink the root memcg directly.
>>>
>>> With memcg disabled, shrink_memcg() only returns -ENOENT when the root
>>> LRU is empty, which means the total pages are already below thr. In the
>>> absence of heavy concurrent zswap stores, the loop then safely bails out
>>> via the zswap_total_pages() <= thr check; otherwise, it will resume
>>> shrinking the memcg after processing the reschedule check. For any other
>>> return value from shrink_memcg(), the loop is guaranteed to terminate,
>>> either after MAX_RECLAIM_RETRIES failures or once the threshold is met.
>>>
>>> Fixes: a65b0e7607cc ("zswap: make shrinking memcg-aware")
>>> Cc: stable@vger.kernel.org
>>
>> How does this affect users?  What behavior do they observe when it
>> occurs?
> 
> I think the first paragraph sums it up pretty well, especially the
> last sentence "hotter pages with higher fault latency".
> 
>>
>>> Closes: https://lore.kernel.org/all/CAO9r8zPVzMKFbCixxD-qgtRrkFxWVrHiZZeLc=eyTPKPVQgX4g@mail.gmail.com
>>
>> hm, that isn't really a bug report and doesn't answer the above
>> question.
> 
> Yeah, it isn't. Probably we should drop "Closes". I assume Hao added
> it because checkpatch annoyingly complains if you add "Reported-by"
> without "Closes", so Hao just linked to the thread where I pointed out
> the bug.

Yeah, checkpatch will complain if it's missing.

> 
>>
>> AI review asked a couple of questions:
>>          https://sashiko.dev/#/patchset/20260729084206.77793-1-jiahao.kernel@gmail.com
> 
> The review on patch #1 is something theoretical, we discussed it at
> length in previous versions.
> 
> For patch #2:
> 
>> Does this batching logic break NUMA fairness?
>>
>> Because for_each_node_state() always starts from the lowest node
>> ID and breaks when the scan budget is exhausted, subsequent
>> calls to shrink_memcg() will restart at the lowest node ID again.
>>
>> If the lowest node (typically Node 0) consistently has enough
>> items to exhaust the scan budget, wouldn't we exclusively evict
>> pages from it while ignoring older pages on other nodes? Could
>> this cause LRU inversion across nodes, keeping older pages in
>> memory on Node 1 while hot pages on Node 0 are evicted?
> 
> Yes, unfairness is possible.
> 
> For global shrinking, it's probably not an issue. We reclaim until we
> hit the acceptance threshold and it's very unlikely this will happen
> before iterating all nodes (given that the batch size is 32 pages).
> However, with the shrink_memcg() path, we only reclaim one batch, so
> there's a chance we'll always reclaim it from node 0.
> 
> Maybe we should just drop the early bailout and accept potentially
> doing more writeback than needed. Hao, WDYT?
> 
If we scan and attempt to write back SWAP_CLUSTER_MAX zswap entries per 
node, it might lead to excessive writeback on machines with many NUMA 
nodes. Furthermore, I'm concerned about introducing higher latency in 
synchronous shrink paths like zswap_store()—especially on systems with a 
large number of NUMA nodes, where it could end up writing back hundreds 
of pages in a single call.

Maybe we could do something like this instead? That way, in the 
worst-case scenario, it falls back to the baseline behavior without 
introducing any extra latency risks.

  static int shrink_memcg(struct mem_cgroup *memcg)
  {
-       int nid, shrunk = 0, scanned = 0;
+       unsigned long node_batch, scanned = 0;
+       int nid, shrunk = 0;

         if (!mem_cgroup_zswap_writeback_enabled(memcg))
                 return -ENOENT;
@@ -1289,14 +1313,26 @@ static int shrink_memcg(struct mem_cgroup *memcg)
         if (memcg && !mem_cgroup_online(memcg))
                 return -ENOENT;

+       node_batch = max(1UL, SWAP_CLUSTER_MAX / 
num_node_state(N_NORMAL_MEMORY));
         for_each_node_state(nid, N_NORMAL_MEMORY) {
-               unsigned long nr_to_walk = 1;
+               unsigned long nr_to_walk, budget;
+
+               /*
+                * Cap the scan at the per-node LRU length so each entry is
+                * scanned at most once per call.
+                */
+               budget = min(node_batch,
+                            list_lru_count_one(&zswap_list_lru, nid, 
memcg));
+               if (!budget)
+                       continue;

+               nr_to_walk = budget;
                 shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg,
                                             &shrink_memcg_cb, NULL, 
&nr_to_walk);
-               scanned += 1 - nr_to_walk;
+               scanned += budget - nr_to_walk;
         }

+       /* Nothing was scanned: every LRU under @memcg was empty. */
         if (!scanned)
                 return -ENOENT;

Thanks,
Hao
> If you respin, please also drop the batch size argument to
> shrink_memcg() as it's now always SWAP_CLUSTER_MAX.

Re: [PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[Yosry Ahmed]

> > Yeah, it isn't. Probably we should drop "Closes". I assume Hao added
> > it because checkpatch annoyingly complains if you add "Reported-by"
> > without "Closes", so Hao just linked to the thread where I pointed out
> > the bug.
>
> Yeah, checkpatch will complain if it's missing.

We often ignore checkpatch if it's unreasonable.

> >> AI review asked a couple of questions:
> >>          https://sashiko.dev/#/patchset/20260729084206.77793-1-jiahao.kernel@gmail.com
> >
> > The review on patch #1 is something theoretical, we discussed it at
> > length in previous versions.
> >
> > For patch #2:
> >
> >> Does this batching logic break NUMA fairness?
> >>
> >> Because for_each_node_state() always starts from the lowest node
> >> ID and breaks when the scan budget is exhausted, subsequent
> >> calls to shrink_memcg() will restart at the lowest node ID again.
> >>
> >> If the lowest node (typically Node 0) consistently has enough
> >> items to exhaust the scan budget, wouldn't we exclusively evict
> >> pages from it while ignoring older pages on other nodes? Could
> >> this cause LRU inversion across nodes, keeping older pages in
> >> memory on Node 1 while hot pages on Node 0 are evicted?
> >
> > Yes, unfairness is possible.
> >
> > For global shrinking, it's probably not an issue. We reclaim until we
> > hit the acceptance threshold and it's very unlikely this will happen
> > before iterating all nodes (given that the batch size is 32 pages).
> > However, with the shrink_memcg() path, we only reclaim one batch, so
> > there's a chance we'll always reclaim it from node 0.
> >
> > Maybe we should just drop the early bailout and accept potentially
> > doing more writeback than needed. Hao, WDYT?
> >
> If we scan and attempt to write back SWAP_CLUSTER_MAX zswap entries per
> node, it might lead to excessive writeback on machines with many NUMA
> nodes. Furthermore, I'm concerned about introducing higher latency in
> synchronous shrink paths like zswap_store()—especially on systems with a
> large number of NUMA nodes, where it could end up writing back hundreds
> of pages in a single call.
>
> Maybe we could do something like this instead? That way, in the
> worst-case scenario, it falls back to the baseline behavior without
> introducing any extra latency risks.

This basically errs on the side of honouring the batch size
(under-reclaim) instead of doing more writeback, right?

I think I prefer erring on the side of doing more writeback, as it
would ultimately result in less LRU inversion, and we are already
doing writeback proactively during reclaim through the shrinker.

The other option we can explore is keeping the existing logic with the
bailout, but start iterating the nodes at the folio's node in the
zswap_store() path. Basically pass an nid to shrink_memcg() and always
start there. This will avoid always reclaiming from node 0, and the
node of the folio being stored is more likely to be under pressure and
need the writeback. I am not sure how complex this would be and
whether it's worth doing.

Johannes, Nhats, any thoughts on this?

>
>   static int shrink_memcg(struct mem_cgroup *memcg)
>   {
> -       int nid, shrunk = 0, scanned = 0;
> +       unsigned long node_batch, scanned = 0;
> +       int nid, shrunk = 0;
>
>          if (!mem_cgroup_zswap_writeback_enabled(memcg))
>                  return -ENOENT;
> @@ -1289,14 +1313,26 @@ static int shrink_memcg(struct mem_cgroup *memcg)
>          if (memcg && !mem_cgroup_online(memcg))
>                  return -ENOENT;
>
> +       node_batch = max(1UL, SWAP_CLUSTER_MAX /
> num_node_state(N_NORMAL_MEMORY));
>          for_each_node_state(nid, N_NORMAL_MEMORY) {
> -               unsigned long nr_to_walk = 1;
> +               unsigned long nr_to_walk, budget;
> +
> +               /*
> +                * Cap the scan at the per-node LRU length so each entry is
> +                * scanned at most once per call.
> +                */
> +               budget = min(node_batch,
> +                            list_lru_count_one(&zswap_list_lru, nid,
> memcg));
> +               if (!budget)
> +                       continue;
>
> +               nr_to_walk = budget;
>                  shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg,
>                                              &shrink_memcg_cb, NULL,
> &nr_to_walk);
> -               scanned += 1 - nr_to_walk;
> +               scanned += budget - nr_to_walk;
>          }
>
> +       /* Nothing was scanned: every LRU under @memcg was empty. */
>          if (!scanned)
>                  return -ENOENT;
>
> Thanks,
> Hao
> > If you respin, please also drop the batch size argument to
> > shrink_memcg() as it's now always SWAP_CLUSTER_MAX.

Re: [PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[Johannes Weiner]

On Thu, Jul 30, 2026 at 10:48:38AM -0700, Yosry Ahmed wrote:
> > > Yeah, it isn't. Probably we should drop "Closes". I assume Hao added
> > > it because checkpatch annoyingly complains if you add "Reported-by"
> > > without "Closes", so Hao just linked to the thread where I pointed out
> > > the bug.
> >
> > Yeah, checkpatch will complain if it's missing.
> 
> We often ignore checkpatch if it's unreasonable.
> 
> > >> AI review asked a couple of questions:
> > >>          https://sashiko.dev/#/patchset/20260729084206.77793-1-jiahao.kernel@gmail.com
> > >
> > > The review on patch #1 is something theoretical, we discussed it at
> > > length in previous versions.
> > >
> > > For patch #2:
> > >
> > >> Does this batching logic break NUMA fairness?
> > >>
> > >> Because for_each_node_state() always starts from the lowest node
> > >> ID and breaks when the scan budget is exhausted, subsequent
> > >> calls to shrink_memcg() will restart at the lowest node ID again.
> > >>
> > >> If the lowest node (typically Node 0) consistently has enough
> > >> items to exhaust the scan budget, wouldn't we exclusively evict
> > >> pages from it while ignoring older pages on other nodes? Could
> > >> this cause LRU inversion across nodes, keeping older pages in
> > >> memory on Node 1 while hot pages on Node 0 are evicted?
> > >
> > > Yes, unfairness is possible.
> > >
> > > For global shrinking, it's probably not an issue. We reclaim until we
> > > hit the acceptance threshold and it's very unlikely this will happen
> > > before iterating all nodes (given that the batch size is 32 pages).
> > > However, with the shrink_memcg() path, we only reclaim one batch, so
> > > there's a chance we'll always reclaim it from node 0.
> > >
> > > Maybe we should just drop the early bailout and accept potentially
> > > doing more writeback than needed. Hao, WDYT?
> > >
> > If we scan and attempt to write back SWAP_CLUSTER_MAX zswap entries per
> > node, it might lead to excessive writeback on machines with many NUMA
> > nodes. Furthermore, I'm concerned about introducing higher latency in
> > synchronous shrink paths like zswap_store()—especially on systems with a
> > large number of NUMA nodes, where it could end up writing back hundreds
> > of pages in a single call.
> >
> > Maybe we could do something like this instead? That way, in the
> > worst-case scenario, it falls back to the baseline behavior without
> > introducing any extra latency risks.
> 
> This basically errs on the side of honouring the batch size
> (under-reclaim) instead of doing more writeback, right?
> 
> I think I prefer erring on the side of doing more writeback, as it
> would ultimately result in less LRU inversion, and we are already
> doing writeback proactively during reclaim through the shrinker.

Yes, LRU inversions are worse than reclaiming a few extra pages from
the cold tail of the LRU. It's not cumulative after all, it just means
we take a bigger bite and can take more stores before the next shrink.

> The other option we can explore is keeping the existing logic with the
> bailout, but start iterating the nodes at the folio's node in the
> zswap_store() path. Basically pass an nid to shrink_memcg() and always
> start there. This will avoid always reclaiming from node 0, and the
> node of the folio being stored is more likely to be under pressure and
> need the writeback. I am not sure how complex this would be and
> whether it's worth doing.
>
> Johannes, Nhats, any thoughts on this?

What happens if simply we do a hard SWAP_CLUSTER_MAX on each node?

With cgroups, we put the zswap entry on the list_lru head of the node
and memcg where the incoming page was.

So if you're going after one specific cgroup, it's not like you
*actually* reclaim 32 entries on each node of the system. You just
write back from nodes where that cgroup has entries.

The global shrinker also does one cgroup at a time (zswap_next_shrink)
before checking the limit again, so that's fairly fine-grained too.

Without cgroups, you could have zswap entries on all nodes. But if you
have so many nodes that this would constitute hundreds of pages,
presumably that's still a drop in the bucket compared to overall
memory capacity. SWAP_CLUSTER_MAX isn't that big a number.

So I don't think we need to get fancy. Just do SWAP_CLUSTER_MAX on
each node. Delete that nr_to_walk carry-over between them.

[PATCH v3 2/2] mm/zswap: Support batch writeback in shrink_memcg()

[Hao Jia]

From: Hao Jia <jiahao1@lixiang.com>

Currently, shrink_memcg() writes back at most one entry per-node during
its traversal. This makes shrink_worker() inefficient, as it must
repeatedly re-enter shrink_memcg() to make any substantial progress.
Under high memory pressure, this can cause the writeback speed to be
too slow to keep up with refaults, leading to zswap store failures and
forcing pages to skip zswap and go directly to disk, which results in
an LRU inversion.

To address this, extend shrink_memcg() and rewrite its LRU iteration logic,
enabling batch writeback for both the shrink_worker() and zswap_store() paths.
To prevent shrink unfairness across NUMA nodes caused by a shared global scan
quota, limit scanning to up to SWAP_CLUSTER_MAX pages per node and write back
any reclaimable entries found.

Test Setup:
- Total memory: 32 GB, 1 NUMA node.
- zswap settings: accept_threshold_percent=50, shrinker_enabled=N.

Test Case 1:
Set max_pool_percent=1, allocate 512MB of anonymous pages, and fill them
with random data (to avoid compression). Then, use cgroup memory.reclaim
to force a large amount of anonymous pages into zswap. At an interval of
2ms, allocate a 4K anonymous page where the first 4 bytes are random numbers
and the rest are zeros, and then trigger reclamation of this 4K page through
cgroup memory.reclaim. When the pool threshold is reached, shrink_memcg()
will be triggered.
The test data after running for 120s is as follows:
                                Baseline      Patched
shrink_worker wakeups              5,363          169
shrink_memcg calls            11,373,201      350,703
written_back pages                40,212       40,241
zswap_store calls                161,190      163,753
   store succeeded (ret=1)       102,743      117,183
   store rejected (ret=0)         58,447       46,570
   store reject rate                ~36%        ~28%
pool_limit_hit delta              55,826       33,760
pswpout                           98,659       86,811
pswpin                                 2            0

Test Case 2:
We evaluated the following two sub-configurations using stress-ng inside
a cgroup capped at memory.max=1G for 120 seconds:
  Test Case 2a (max_pool_percent=1): Continuously triggers the global
  zswap pool limit, thereby waking up shrink_worker() to perform asynchronous
  shrinking.
  Test Case 2b (zswap.max=320M, max_pool_percent=50): Continuously triggers
  the cgroup's zswap.max limit, thereby invoking synchronous shrinking.
Command executed for both setups:
   bash -c 'echo $$ > /sys/fs/cgroup/zswaptest/cgroup.procs ; \
   exec stress-ng --vm 4 --vm-bytes 4G --vm-keep --vm-method rand-set -t \
120s -q'

Test Case 2a (max_pool_percent=1):
                                Baseline       Patched
shrink_worker wakeups              5,640         1,308
shrink_memcg calls             8,481,500     3,140,972
written_back pages                   260       468,216
zswap_store calls              2,742,756     2,011,269
   store succeeded (ret=1)       934,640       947,988
   store rejected (ret=0)      1,808,116     1,063,281
   store reject rate                ~66%          ~52%
pool_limit_hit delta           1,181,310       196,882
pswpout                        1,808,376     1,531,497
pswpin                         4,288,497     3,635,365
Test Case 2b (zswap.max=320M, max_pool_percent=50):
                                Baseline       Patched
shrink_worker wakeups                 0              0
shrink_memcg calls              687,608         54,002
written_back pages              639,176        846,663
zswap_store calls             1,224,222      1,228,548
   store succeeded (ret=1)      992,816      1,208,123
   store rejected (ret=0)       231,431         20,425
   store reject rate               ~19%            ~2%
pool_limit_hit delta                  0              0
pswpout                         870,745        867,360
pswpin                        1,707,823      1,216,814

Under identical workloads and runtimes, batched zswap shrinking
exhibits a significant reduction in both shrink_worker() wakeups
and shrink_memcg() calls. Furthermore, the sharp drop in both pswpin
and zswap_store() rejections demonstrates that batching zswap shrink
operations effectively mitigates zswap_store() failures caused by
hitting the pool limit. This significantly prevents pages from bypassing
zswap and falling back directly to disk, thereby reducing LRU inversion.

Suggested-by: Yosry Ahmed <yosry@kernel.org>
Acked-by: Yosry Ahmed <yosry@kernel.org>
Acked-by: Nhat Pham <nphamcs@gmail.com>
Signed-off-by: Hao Jia <jiahao1@lixiang.com>
---
 mm/zswap.c | 30 ++++++++++++++++++++++++++++--
 1 file changed, 28 insertions(+), 2 deletions(-)

diff --git a/mm/zswap.c b/mm/zswap.c
index 48fc7b575e24..d406c14925d8 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1275,6 +1275,21 @@ static struct shrinker *zswap_alloc_shrinker(void)
 	return shrinker;
 }
 
+/*
+ * Scan up to SWAP_CLUSTER_MAX pages on each per-node zswap LRU of @memcg
+ * and write back the reclaimable ones.
+ *
+ * Since the second-chance algorithm rotates referenced entries to the
+ * LRU tail, the per-node scan is capped at the current LRU length so
+ * each entry is scanned at most once per call. It is up to the caller
+ * to handle retries, deciding whether to scan another memcg to complete
+ * the full iteration, or to rescan the current memcg to drain its zswap
+ * entries.
+ *
+ * Return: 0 if at least one entry was written back, -EAGAIN if entries
+ * were scanned but none could be written back, or -ENOENT if @memcg has
+ * writeback disabled, is a zombie cgroup, or has empty zswap LRUs.
+ */
 static int shrink_memcg(struct mem_cgroup *memcg)
 {
 	int nid, shrunk = 0, scanned = 0;
@@ -1290,13 +1305,24 @@ static int shrink_memcg(struct mem_cgroup *memcg)
 		return -ENOENT;
 
 	for_each_node_state(nid, N_NORMAL_MEMORY) {
-		unsigned long nr_to_walk = 1;
+		unsigned long nr_to_walk, node_budget;
+
+		/*
+		 * Cap the scan at the per-node LRU length so each entry is
+		 * scanned at most once per call.
+		 */
+		node_budget = min(SWAP_CLUSTER_MAX,
+			     list_lru_count_one(&zswap_list_lru, nid, memcg));
+		if (!node_budget)
+			continue;
 
+		nr_to_walk = node_budget;
 		shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg,
 					    &shrink_memcg_cb, NULL, &nr_to_walk);
-		scanned += 1 - nr_to_walk;
+		scanned += node_budget - nr_to_walk;
 	}
 
+	/* Nothing was scanned: every LRU under @memcg was empty. */
 	if (!scanned)
 		return -ENOENT;
 
-- 
2.34.1

Re: [PATCH v3 2/2] mm/zswap: Support batch writeback in shrink_memcg()

[Johannes Weiner]

On Fri, Jul 31, 2026 at 03:19:00PM +0800, Hao Jia wrote:
> From: Hao Jia <jiahao1@lixiang.com>
> 
> Currently, shrink_memcg() writes back at most one entry per-node during
> its traversal. This makes shrink_worker() inefficient, as it must
> repeatedly re-enter shrink_memcg() to make any substantial progress.
> Under high memory pressure, this can cause the writeback speed to be
> too slow to keep up with refaults, leading to zswap store failures and
> forcing pages to skip zswap and go directly to disk, which results in
> an LRU inversion.
> 
> To address this, extend shrink_memcg() and rewrite its LRU iteration logic,
> enabling batch writeback for both the shrink_worker() and zswap_store() paths.
> To prevent shrink unfairness across NUMA nodes caused by a shared global scan
> quota, limit scanning to up to SWAP_CLUSTER_MAX pages per node and write back
> any reclaimable entries found.
> 
> Test Setup:
> - Total memory: 32 GB, 1 NUMA node.
> - zswap settings: accept_threshold_percent=50, shrinker_enabled=N.
> 
> Test Case 1:
> Set max_pool_percent=1, allocate 512MB of anonymous pages, and fill them
> with random data (to avoid compression). Then, use cgroup memory.reclaim
> to force a large amount of anonymous pages into zswap. At an interval of
> 2ms, allocate a 4K anonymous page where the first 4 bytes are random numbers
> and the rest are zeros, and then trigger reclamation of this 4K page through
> cgroup memory.reclaim. When the pool threshold is reached, shrink_memcg()
> will be triggered.
> The test data after running for 120s is as follows:
>                                 Baseline      Patched
> shrink_worker wakeups              5,363          169
> shrink_memcg calls            11,373,201      350,703
> written_back pages                40,212       40,241
> zswap_store calls                161,190      163,753
>    store succeeded (ret=1)       102,743      117,183
>    store rejected (ret=0)         58,447       46,570
>    store reject rate                ~36%        ~28%
> pool_limit_hit delta              55,826       33,760
> pswpout                           98,659       86,811
> pswpin                                 2            0
> 
> Test Case 2:
> We evaluated the following two sub-configurations using stress-ng inside
> a cgroup capped at memory.max=1G for 120 seconds:
>   Test Case 2a (max_pool_percent=1): Continuously triggers the global
>   zswap pool limit, thereby waking up shrink_worker() to perform asynchronous
>   shrinking.
>   Test Case 2b (zswap.max=320M, max_pool_percent=50): Continuously triggers
>   the cgroup's zswap.max limit, thereby invoking synchronous shrinking.
> Command executed for both setups:
>    bash -c 'echo $$ > /sys/fs/cgroup/zswaptest/cgroup.procs ; \
>    exec stress-ng --vm 4 --vm-bytes 4G --vm-keep --vm-method rand-set -t \
> 120s -q'
> 
> Test Case 2a (max_pool_percent=1):
>                                 Baseline       Patched
> shrink_worker wakeups              5,640         1,308
> shrink_memcg calls             8,481,500     3,140,972
> written_back pages                   260       468,216
> zswap_store calls              2,742,756     2,011,269
>    store succeeded (ret=1)       934,640       947,988
>    store rejected (ret=0)      1,808,116     1,063,281
>    store reject rate                ~66%          ~52%
> pool_limit_hit delta           1,181,310       196,882
> pswpout                        1,808,376     1,531,497
> pswpin                         4,288,497     3,635,365
> Test Case 2b (zswap.max=320M, max_pool_percent=50):
>                                 Baseline       Patched
> shrink_worker wakeups                 0              0
> shrink_memcg calls              687,608         54,002
> written_back pages              639,176        846,663
> zswap_store calls             1,224,222      1,228,548
>    store succeeded (ret=1)      992,816      1,208,123
>    store rejected (ret=0)       231,431         20,425
>    store reject rate               ~19%            ~2%
> pool_limit_hit delta                  0              0
> pswpout                         870,745        867,360
> pswpin                        1,707,823      1,216,814
> 
> Under identical workloads and runtimes, batched zswap shrinking
> exhibits a significant reduction in both shrink_worker() wakeups
> and shrink_memcg() calls. Furthermore, the sharp drop in both pswpin
> and zswap_store() rejections demonstrates that batching zswap shrink
> operations effectively mitigates zswap_store() failures caused by
> hitting the pool limit. This significantly prevents pages from bypassing
> zswap and falling back directly to disk, thereby reducing LRU inversion.
> 
> Suggested-by: Yosry Ahmed <yosry@kernel.org>
> Acked-by: Yosry Ahmed <yosry@kernel.org>
> Acked-by: Nhat Pham <nphamcs@gmail.com>
> Signed-off-by: Hao Jia <jiahao1@lixiang.com>
> ---
>  mm/zswap.c | 30 ++++++++++++++++++++++++++++--
>  1 file changed, 28 insertions(+), 2 deletions(-)
> 
> diff --git a/mm/zswap.c b/mm/zswap.c
> index 48fc7b575e24..d406c14925d8 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
> @@ -1275,6 +1275,21 @@ static struct shrinker *zswap_alloc_shrinker(void)
>  	return shrinker;
>  }
>  
> +/*
> + * Scan up to SWAP_CLUSTER_MAX pages on each per-node zswap LRU of @memcg
> + * and write back the reclaimable ones.
> + *
> + * Since the second-chance algorithm rotates referenced entries to the
> + * LRU tail, the per-node scan is capped at the current LRU length so
> + * each entry is scanned at most once per call. It is up to the caller
> + * to handle retries, deciding whether to scan another memcg to complete
> + * the full iteration, or to rescan the current memcg to drain its zswap
> + * entries.
> + *
> + * Return: 0 if at least one entry was written back, -EAGAIN if entries
> + * were scanned but none could be written back, or -ENOENT if @memcg has
> + * writeback disabled, is a zombie cgroup, or has empty zswap LRUs.
> + */
>  static int shrink_memcg(struct mem_cgroup *memcg)
>  {
>  	int nid, shrunk = 0, scanned = 0;
> @@ -1290,13 +1305,24 @@ static int shrink_memcg(struct mem_cgroup *memcg)
>  		return -ENOENT;
>  
>  	for_each_node_state(nid, N_NORMAL_MEMORY) {
> -		unsigned long nr_to_walk = 1;
> +		unsigned long nr_to_walk, node_budget;
> +
> +		/*
> +		 * Cap the scan at the per-node LRU length so each entry is
> +		 * scanned at most once per call.
> +		 */
> +		node_budget = min(SWAP_CLUSTER_MAX,
> +			     list_lru_count_one(&zswap_list_lru, nid, memcg));

AFAICS you can just do unsigned long nr_to_walk = SWAP_CLUSTER_MAX.

__list_lru_walk_one() does a list_for_each_safe() that will exit the
same way whether you hit !nr_to_walk or run out of items.

> +		if (!node_budget)
> +			continue;
>  
> +		nr_to_walk = node_budget;
>  		shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg,
>  					    &shrink_memcg_cb, NULL, &nr_to_walk);
> -		scanned += 1 - nr_to_walk;
> +		scanned += node_budget - nr_to_walk;

scanned += SWAP_CLUSTER_MAX - nr_to_walk;

Re: [PATCH v3 2/2] mm/zswap: Support batch writeback in shrink_memcg()

[Hao Jia]

On 2026/7/31 23:17, Johannes Weiner wrote:
> On Fri, Jul 31, 2026 at 03:19:00PM +0800, Hao Jia wrote:
>> From: Hao Jia <jiahao1@lixiang.com>
>>
>> Currently, shrink_memcg() writes back at most one entry per-node during
>> its traversal. This makes shrink_worker() inefficient, as it must
>> repeatedly re-enter shrink_memcg() to make any substantial progress.
>> Under high memory pressure, this can cause the writeback speed to be
>> too slow to keep up with refaults, leading to zswap store failures and
>> forcing pages to skip zswap and go directly to disk, which results in
>> an LRU inversion.
>>
>> To address this, extend shrink_memcg() and rewrite its LRU iteration logic,
>> enabling batch writeback for both the shrink_worker() and zswap_store() paths.
>> To prevent shrink unfairness across NUMA nodes caused by a shared global scan
>> quota, limit scanning to up to SWAP_CLUSTER_MAX pages per node and write back
>> any reclaimable entries found.
>>
>> Test Setup:
>> - Total memory: 32 GB, 1 NUMA node.
>> - zswap settings: accept_threshold_percent=50, shrinker_enabled=N.
>>
>> Test Case 1:
>> Set max_pool_percent=1, allocate 512MB of anonymous pages, and fill them
>> with random data (to avoid compression). Then, use cgroup memory.reclaim
>> to force a large amount of anonymous pages into zswap. At an interval of
>> 2ms, allocate a 4K anonymous page where the first 4 bytes are random numbers
>> and the rest are zeros, and then trigger reclamation of this 4K page through
>> cgroup memory.reclaim. When the pool threshold is reached, shrink_memcg()
>> will be triggered.
>> The test data after running for 120s is as follows:
>>                                  Baseline      Patched
>> shrink_worker wakeups              5,363          169
>> shrink_memcg calls            11,373,201      350,703
>> written_back pages                40,212       40,241
>> zswap_store calls                161,190      163,753
>>     store succeeded (ret=1)       102,743      117,183
>>     store rejected (ret=0)         58,447       46,570
>>     store reject rate                ~36%        ~28%
>> pool_limit_hit delta              55,826       33,760
>> pswpout                           98,659       86,811
>> pswpin                                 2            0
>>
>> Test Case 2:
>> We evaluated the following two sub-configurations using stress-ng inside
>> a cgroup capped at memory.max=1G for 120 seconds:
>>    Test Case 2a (max_pool_percent=1): Continuously triggers the global
>>    zswap pool limit, thereby waking up shrink_worker() to perform asynchronous
>>    shrinking.
>>    Test Case 2b (zswap.max=320M, max_pool_percent=50): Continuously triggers
>>    the cgroup's zswap.max limit, thereby invoking synchronous shrinking.
>> Command executed for both setups:
>>     bash -c 'echo $$ > /sys/fs/cgroup/zswaptest/cgroup.procs ; \
>>     exec stress-ng --vm 4 --vm-bytes 4G --vm-keep --vm-method rand-set -t \
>> 120s -q'
>>
>> Test Case 2a (max_pool_percent=1):
>>                                  Baseline       Patched
>> shrink_worker wakeups              5,640         1,308
>> shrink_memcg calls             8,481,500     3,140,972
>> written_back pages                   260       468,216
>> zswap_store calls              2,742,756     2,011,269
>>     store succeeded (ret=1)       934,640       947,988
>>     store rejected (ret=0)      1,808,116     1,063,281
>>     store reject rate                ~66%          ~52%
>> pool_limit_hit delta           1,181,310       196,882
>> pswpout                        1,808,376     1,531,497
>> pswpin                         4,288,497     3,635,365
>> Test Case 2b (zswap.max=320M, max_pool_percent=50):
>>                                  Baseline       Patched
>> shrink_worker wakeups                 0              0
>> shrink_memcg calls              687,608         54,002
>> written_back pages              639,176        846,663
>> zswap_store calls             1,224,222      1,228,548
>>     store succeeded (ret=1)      992,816      1,208,123
>>     store rejected (ret=0)       231,431         20,425
>>     store reject rate               ~19%            ~2%
>> pool_limit_hit delta                  0              0
>> pswpout                         870,745        867,360
>> pswpin                        1,707,823      1,216,814
>>
>> Under identical workloads and runtimes, batched zswap shrinking
>> exhibits a significant reduction in both shrink_worker() wakeups
>> and shrink_memcg() calls. Furthermore, the sharp drop in both pswpin
>> and zswap_store() rejections demonstrates that batching zswap shrink
>> operations effectively mitigates zswap_store() failures caused by
>> hitting the pool limit. This significantly prevents pages from bypassing
>> zswap and falling back directly to disk, thereby reducing LRU inversion.
>>
>> Suggested-by: Yosry Ahmed <yosry@kernel.org>
>> Acked-by: Yosry Ahmed <yosry@kernel.org>
>> Acked-by: Nhat Pham <nphamcs@gmail.com>
>> Signed-off-by: Hao Jia <jiahao1@lixiang.com>
>> ---
>>   mm/zswap.c | 30 ++++++++++++++++++++++++++++--
>>   1 file changed, 28 insertions(+), 2 deletions(-)
>>
>> diff --git a/mm/zswap.c b/mm/zswap.c
>> index 48fc7b575e24..d406c14925d8 100644
>> --- a/mm/zswap.c
>> +++ b/mm/zswap.c
>> @@ -1275,6 +1275,21 @@ static struct shrinker *zswap_alloc_shrinker(void)
>>   	return shrinker;
>>   }
>>   
>> +/*
>> + * Scan up to SWAP_CLUSTER_MAX pages on each per-node zswap LRU of @memcg
>> + * and write back the reclaimable ones.
>> + *
>> + * Since the second-chance algorithm rotates referenced entries to the
>> + * LRU tail, the per-node scan is capped at the current LRU length so
>> + * each entry is scanned at most once per call. It is up to the caller
>> + * to handle retries, deciding whether to scan another memcg to complete
>> + * the full iteration, or to rescan the current memcg to drain its zswap
>> + * entries.
>> + *
>> + * Return: 0 if at least one entry was written back, -EAGAIN if entries
>> + * were scanned but none could be written back, or -ENOENT if @memcg has
>> + * writeback disabled, is a zombie cgroup, or has empty zswap LRUs.
>> + */
>>   static int shrink_memcg(struct mem_cgroup *memcg)
>>   {
>>   	int nid, shrunk = 0, scanned = 0;
>> @@ -1290,13 +1305,24 @@ static int shrink_memcg(struct mem_cgroup *memcg)
>>   		return -ENOENT;
>>   
>>   	for_each_node_state(nid, N_NORMAL_MEMORY) {
>> -		unsigned long nr_to_walk = 1;
>> +		unsigned long nr_to_walk, node_budget;
>> +
>> +		/*
>> +		 * Cap the scan at the per-node LRU length so each entry is
>> +		 * scanned at most once per call.
>> +		 */
>> +		node_budget = min(SWAP_CLUSTER_MAX,
>> +			     list_lru_count_one(&zswap_list_lru, nid, memcg));
> 
> AFAICS you can just do unsigned long nr_to_walk = SWAP_CLUSTER_MAX.
> 
> __list_lru_walk_one() does a list_for_each_safe() that will exit the
> same way whether you hit !nr_to_walk or run out of items.

Wouldn't it be better to ensure that each entry is scanned at most once 
per call, particularly when list_lru_count_one(nid) < SWAP_CLUSTER_MAX?

On one hand, this avoids scanning the same entry multiple times within a 
single pass. Since the second-chance algorithm rotates referenced 
entries to the tail of the LRU, entries on nodes with a large number of 
zswap entries require at least two shrink_memcg() calls to be written 
back, whereas entries on nodes with fewer entries might get written back 
in a single shrink_memcg() call instead. On the other hand, we also 
avoid spinning repeatedly on entries that fail writeback.

Thanks,
Hao

> 
>> +		if (!node_budget)
>> +			continue;
>>   
>> +		nr_to_walk = node_budget;
>>   		shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg,
>>   					    &shrink_memcg_cb, NULL, &nr_to_walk);
>> -		scanned += 1 - nr_to_walk;
>> +		scanned += node_budget - nr_to_walk;
> 
> scanned += SWAP_CLUSTER_MAX - nr_to_walk;

Re: [PATCH v3 2/2] mm/zswap: Support batch writeback in shrink_memcg()

[Hao Jia]

On 2026/8/1 08:31, Hao Jia wrote:
> 
> 
> On 2026/7/31 23:17, Johannes Weiner wrote:
>> On Fri, Jul 31, 2026 at 03:19:00PM +0800, Hao Jia wrote:
>>> From: Hao Jia <jiahao1@lixiang.com>
>>>
>>> Currently, shrink_memcg() writes back at most one entry per-node during
>>> its traversal. This makes shrink_worker() inefficient, as it must
>>> repeatedly re-enter shrink_memcg() to make any substantial progress.
>>> Under high memory pressure, this can cause the writeback speed to be
>>> too slow to keep up with refaults, leading to zswap store failures and
>>> forcing pages to skip zswap and go directly to disk, which results in
>>> an LRU inversion.
>>>
>>> To address this, extend shrink_memcg() and rewrite its LRU iteration 
>>> logic,
>>> enabling batch writeback for both the shrink_worker() and 
>>> zswap_store() paths.
>>> To prevent shrink unfairness across NUMA nodes caused by a shared 
>>> global scan
>>> quota, limit scanning to up to SWAP_CLUSTER_MAX pages per node and 
>>> write back
>>> any reclaimable entries found.
>>>
>>> Test Setup:
>>> - Total memory: 32 GB, 1 NUMA node.
>>> - zswap settings: accept_threshold_percent=50, shrinker_enabled=N.
>>>
>>> Test Case 1:
>>> Set max_pool_percent=1, allocate 512MB of anonymous pages, and fill them
>>> with random data (to avoid compression). Then, use cgroup memory.reclaim
>>> to force a large amount of anonymous pages into zswap. At an interval of
>>> 2ms, allocate a 4K anonymous page where the first 4 bytes are random 
>>> numbers
>>> and the rest are zeros, and then trigger reclamation of this 4K page 
>>> through
>>> cgroup memory.reclaim. When the pool threshold is reached, 
>>> shrink_memcg()
>>> will be triggered.
>>> The test data after running for 120s is as follows:
>>>                                  Baseline      Patched
>>> shrink_worker wakeups              5,363          169
>>> shrink_memcg calls            11,373,201      350,703
>>> written_back pages                40,212       40,241
>>> zswap_store calls                161,190      163,753
>>>     store succeeded (ret=1)       102,743      117,183
>>>     store rejected (ret=0)         58,447       46,570
>>>     store reject rate                ~36%        ~28%
>>> pool_limit_hit delta              55,826       33,760
>>> pswpout                           98,659       86,811
>>> pswpin                                 2            0
>>>
>>> Test Case 2:
>>> We evaluated the following two sub-configurations using stress-ng inside
>>> a cgroup capped at memory.max=1G for 120 seconds:
>>>    Test Case 2a (max_pool_percent=1): Continuously triggers the global
>>>    zswap pool limit, thereby waking up shrink_worker() to perform 
>>> asynchronous
>>>    shrinking.
>>>    Test Case 2b (zswap.max=320M, max_pool_percent=50): Continuously 
>>> triggers
>>>    the cgroup's zswap.max limit, thereby invoking synchronous shrinking.
>>> Command executed for both setups:
>>>     bash -c 'echo $$ > /sys/fs/cgroup/zswaptest/cgroup.procs ; \
>>>     exec stress-ng --vm 4 --vm-bytes 4G --vm-keep --vm-method 
>>> rand-set -t \
>>> 120s -q'
>>>
>>> Test Case 2a (max_pool_percent=1):
>>>                                  Baseline       Patched
>>> shrink_worker wakeups              5,640         1,308
>>> shrink_memcg calls             8,481,500     3,140,972
>>> written_back pages                   260       468,216
>>> zswap_store calls              2,742,756     2,011,269
>>>     store succeeded (ret=1)       934,640       947,988
>>>     store rejected (ret=0)      1,808,116     1,063,281
>>>     store reject rate                ~66%          ~52%
>>> pool_limit_hit delta           1,181,310       196,882
>>> pswpout                        1,808,376     1,531,497
>>> pswpin                         4,288,497     3,635,365
>>> Test Case 2b (zswap.max=320M, max_pool_percent=50):
>>>                                  Baseline       Patched
>>> shrink_worker wakeups                 0              0
>>> shrink_memcg calls              687,608         54,002
>>> written_back pages              639,176        846,663
>>> zswap_store calls             1,224,222      1,228,548
>>>     store succeeded (ret=1)      992,816      1,208,123
>>>     store rejected (ret=0)       231,431         20,425
>>>     store reject rate               ~19%            ~2%
>>> pool_limit_hit delta                  0              0
>>> pswpout                         870,745        867,360
>>> pswpin                        1,707,823      1,216,814
>>>
>>> Under identical workloads and runtimes, batched zswap shrinking
>>> exhibits a significant reduction in both shrink_worker() wakeups
>>> and shrink_memcg() calls. Furthermore, the sharp drop in both pswpin
>>> and zswap_store() rejections demonstrates that batching zswap shrink
>>> operations effectively mitigates zswap_store() failures caused by
>>> hitting the pool limit. This significantly prevents pages from bypassing
>>> zswap and falling back directly to disk, thereby reducing LRU inversion.
>>>
>>> Suggested-by: Yosry Ahmed <yosry@kernel.org>
>>> Acked-by: Yosry Ahmed <yosry@kernel.org>
>>> Acked-by: Nhat Pham <nphamcs@gmail.com>
>>> Signed-off-by: Hao Jia <jiahao1@lixiang.com>
>>> ---
>>>   mm/zswap.c | 30 ++++++++++++++++++++++++++++--
>>>   1 file changed, 28 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/mm/zswap.c b/mm/zswap.c
>>> index 48fc7b575e24..d406c14925d8 100644
>>> --- a/mm/zswap.c
>>> +++ b/mm/zswap.c
>>> @@ -1275,6 +1275,21 @@ static struct shrinker 
>>> *zswap_alloc_shrinker(void)
>>>       return shrinker;
>>>   }
>>> +/*
>>> + * Scan up to SWAP_CLUSTER_MAX pages on each per-node zswap LRU of 
>>> @memcg
>>> + * and write back the reclaimable ones.
>>> + *
>>> + * Since the second-chance algorithm rotates referenced entries to the
>>> + * LRU tail, the per-node scan is capped at the current LRU length so
>>> + * each entry is scanned at most once per call. It is up to the caller
>>> + * to handle retries, deciding whether to scan another memcg to 
>>> complete
>>> + * the full iteration, or to rescan the current memcg to drain its 
>>> zswap
>>> + * entries.
>>> + *
>>> + * Return: 0 if at least one entry was written back, -EAGAIN if entries
>>> + * were scanned but none could be written back, or -ENOENT if @memcg 
>>> has
>>> + * writeback disabled, is a zombie cgroup, or has empty zswap LRUs.
>>> + */
>>>   static int shrink_memcg(struct mem_cgroup *memcg)
>>>   {
>>>       int nid, shrunk = 0, scanned = 0;
>>> @@ -1290,13 +1305,24 @@ static int shrink_memcg(struct mem_cgroup 
>>> *memcg)
>>>           return -ENOENT;
>>>       for_each_node_state(nid, N_NORMAL_MEMORY) {
>>> -        unsigned long nr_to_walk = 1;
>>> +        unsigned long nr_to_walk, node_budget;
>>> +
>>> +        /*
>>> +         * Cap the scan at the per-node LRU length so each entry is
>>> +         * scanned at most once per call.
>>> +         */
>>> +        node_budget = min(SWAP_CLUSTER_MAX,
>>> +                 list_lru_count_one(&zswap_list_lru, nid, memcg));
>>
>> AFAICS you can just do unsigned long nr_to_walk = SWAP_CLUSTER_MAX.
>>
>> __list_lru_walk_one() does a list_for_each_safe() that will exit the
>> same way whether you hit !nr_to_walk or run out of items.
> 
> Wouldn't it be better to ensure that each entry is scanned at most once 
> per call, particularly when list_lru_count_one(nid) < SWAP_CLUSTER_MAX?
> 
> On one hand, this avoids scanning the same entry multiple times within a 
> single pass. Since the second-chance algorithm rotates referenced 
> entries to the tail of the LRU, entries on nodes with a large number of 
> zswap entries require at least two shrink_memcg() calls to be written 
> back, whereas entries on nodes with fewer entries might get written back 
> in a single shrink_memcg() call instead. On the other hand, we also 
> avoid spinning repeatedly on entries that fail writeback.
> 
> Thanks,
> Hao
> 

Johannes, Yosry, Nhats, any thoughts on this?

>>
>>> +        if (!node_budget)
>>> +            continue;
>>> +        nr_to_walk = node_budget;
>>>           shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg,
>>>                           &shrink_memcg_cb, NULL, &nr_to_walk);
>>> -        scanned += 1 - nr_to_walk;
>>> +        scanned += node_budget - nr_to_walk;
>>
>> scanned += SWAP_CLUSTER_MAX - nr_to_walk;

Re: [PATCH v3 2/2] mm/zswap: Support batch writeback in shrink_memcg()

[Nhat Pham]

On Tue, Aug 4, 2026 at 11:21 PM Hao Jia <jiahao.kernel@gmail.com> wrote:
>
>
>
> On 2026/8/1 08:31, Hao Jia wrote:
> >
> >
> > On one hand, this avoids scanning the same entry multiple times within a
> > single pass. Since the second-chance algorithm rotates referenced
> > entries to the tail of the LRU, entries on nodes with a large number of
> > zswap entries require at least two shrink_memcg() calls to be written
> > back, whereas entries on nodes with fewer entries might get written back
> > in a single shrink_memcg() call instead. On the other hand, we also

Same thing happens if you have more entries, no? Say you have 64
entries and 96 entries in node 1 and node 2, respectively. Then you
need to go to round 3 to reclaim an entry in node 1, and round 4 for
node 2. What's special about 1/2 rounds vs 3/4 rounds?

I think it's probably fine to just pass SWAP_CLUSTER_MAX as nr_to_walk
- we're applying equal amount of reclaim pressure to 2 nodes. Without
prior information, seems like a reasonable strategy to me. I think
what you did is already a pretty big improvements to status quo - we
can always add more fine tuning on top of it later if needs be.

> > avoid spinning repeatedly on entries that fail writeback.

SWAP_CLUSTER_MAX is 32, so I assume it's not going to spin for a
terribly long amount of time?

Re: [PATCH v3 1/2] mm/zswap: Fix global shrinker when memory cgroup is disabled

[Hao Jia]

On 2026/7/31 03:01, Johannes Weiner wrote:
> On Thu, Jul 30, 2026 at 10:48:38AM -0700, Yosry Ahmed wrote:
>>>> Yeah, it isn't. Probably we should drop "Closes". I assume Hao added
>>>> it because checkpatch annoyingly complains if you add "Reported-by"
>>>> without "Closes", so Hao just linked to the thread where I pointed out
>>>> the bug.
>>>
>>> Yeah, checkpatch will complain if it's missing.
>>
>> We often ignore checkpatch if it's unreasonable.
>>
>>>>> AI review asked a couple of questions:
>>>>>           https://sashiko.dev/#/patchset/20260729084206.77793-1-jiahao.kernel@gmail.com
>>>>
>>>> The review on patch #1 is something theoretical, we discussed it at
>>>> length in previous versions.
>>>>
>>>> For patch #2:
>>>>
>>>>> Does this batching logic break NUMA fairness?
>>>>>
>>>>> Because for_each_node_state() always starts from the lowest node
>>>>> ID and breaks when the scan budget is exhausted, subsequent
>>>>> calls to shrink_memcg() will restart at the lowest node ID again.
>>>>>
>>>>> If the lowest node (typically Node 0) consistently has enough
>>>>> items to exhaust the scan budget, wouldn't we exclusively evict
>>>>> pages from it while ignoring older pages on other nodes? Could
>>>>> this cause LRU inversion across nodes, keeping older pages in
>>>>> memory on Node 1 while hot pages on Node 0 are evicted?
>>>>
>>>> Yes, unfairness is possible.
>>>>
>>>> For global shrinking, it's probably not an issue. We reclaim until we
>>>> hit the acceptance threshold and it's very unlikely this will happen
>>>> before iterating all nodes (given that the batch size is 32 pages).
>>>> However, with the shrink_memcg() path, we only reclaim one batch, so
>>>> there's a chance we'll always reclaim it from node 0.
>>>>
>>>> Maybe we should just drop the early bailout and accept potentially
>>>> doing more writeback than needed. Hao, WDYT?
>>>>
>>> If we scan and attempt to write back SWAP_CLUSTER_MAX zswap entries per
>>> node, it might lead to excessive writeback on machines with many NUMA
>>> nodes. Furthermore, I'm concerned about introducing higher latency in
>>> synchronous shrink paths like zswap_store()—especially on systems with a
>>> large number of NUMA nodes, where it could end up writing back hundreds
>>> of pages in a single call.
>>>
>>> Maybe we could do something like this instead? That way, in the
>>> worst-case scenario, it falls back to the baseline behavior without
>>> introducing any extra latency risks.
>>
>> This basically errs on the side of honouring the batch size
>> (under-reclaim) instead of doing more writeback, right?
>>
>> I think I prefer erring on the side of doing more writeback, as it
>> would ultimately result in less LRU inversion, and we are already
>> doing writeback proactively during reclaim through the shrinker.
> 
> Yes, LRU inversions are worse than reclaiming a few extra pages from
> the cold tail of the LRU. It's not cumulative after all, it just means
> we take a bigger bite and can take more stores before the next shrink.
> 
>> The other option we can explore is keeping the existing logic with the
>> bailout, but start iterating the nodes at the folio's node in the
>> zswap_store() path. Basically pass an nid to shrink_memcg() and always
>> start there. This will avoid always reclaiming from node 0, and the
>> node of the folio being stored is more likely to be under pressure and
>> need the writeback. I am not sure how complex this would be and
>> whether it's worth doing.
>>
>> Johannes, Nhats, any thoughts on this?
> 
> What happens if simply we do a hard SWAP_CLUSTER_MAX on each node?
> 
> With cgroups, we put the zswap entry on the list_lru head of the node
> and memcg where the incoming page was.
> 
> So if you're going after one specific cgroup, it's not like you
> *actually* reclaim 32 entries on each node of the system. You just
> write back from nodes where that cgroup has entries.
> 
> The global shrinker also does one cgroup at a time (zswap_next_shrink)
> before checking the limit again, so that's fairly fine-grained too.
> 
> Without cgroups, you could have zswap entries on all nodes. But if you
> have so many nodes that this would constitute hundreds of pages,
> presumably that's still a drop in the bucket compared to overall
> memory capacity. SWAP_CLUSTER_MAX isn't that big a number.
> 
> So I don't think we need to get fancy. Just do SWAP_CLUSTER_MAX on
> each node. Delete that nr_to_walk carry-over between them.

Thanks for the input! I've posted the new implementation—please take a 
look when you have time.

https://lore.kernel.org/all/20260731071900.38942-1-jiahao.kernel@gmail.com

Thanks,
Hao