[PATCH v4 0/2] mm/swap: skip empty clusters in the swapoff scan

[PATCH v4 0/2] mm/swap: skip empty clusters in the swapoff scan

[Youngjun Park]

find_next_to_unuse() walks a swap device one offset at a time.  Slot
state now lives in a per cluster swap table, so patch 2 dismisses an
empty cluster with one counter read instead of SWAPFILE_CLUSTER table
reads.

Patch 1 is an unrelated one line comment fix noticed on the way.

A debug test confirmed the skip path runs, and swapoff completed
under load with no DEBUG_VM or lockdep splats.

Changes in v4:
- 2/2: put the measured swapoff times in the changelog (Andrew)
- 2/2: pick up Kairui's Acked-by and Baoquan's Reviewed-by
- Rebased on mm-new
- Link to v3: https://lore.kernel.org/r/20260806193228.458685-1-youngjun.park@lge.com

Changes in v3:
- 2/2: clamp the scan end with min_t() so it stops at si->max, rather
  than running into the masked tail of the last cluster, which drops the
  need to explain why walking that tail was safe (Barry)
- 2/2: compute ci_off only where it is used
- 1/2, 2/2: pick up Barry's Reviewed-by
- Rebased on mm-new
- Link to v2: https://lore.kernel.org/r/20260805141146.127776-1-youngjun.park@lge.com

Youngjun Park (2):
  mm/swap: fix stale comment on swap_info_struct::cluster_info
  mm/swap: scan by cluster in find_next_to_unuse()

 include/linux/swap.h |  2 +-
 mm/swapfile.c        | 43 ++++++++++++++++++++++++++++++-------------
 2 files changed, 31 insertions(+), 14 deletions(-)


base-commit: 0bffe67ab8a72b3725cc31c4b525709bd3a3e223
-- 
2.48.1

[PATCH v4 1/2] mm/swap: fix stale comment on swap_info_struct::cluster_info

[Youngjun Park]

setup_swap_clusters_info() allocates cluster_info for every swap area,
not only for SSDs.

Signed-off-by: Youngjun Park <youngjun.park@lge.com>
Acked-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Barry Song <baohua@kernel.org>
---
 include/linux/swap.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/include/linux/swap.h b/include/linux/swap.h
index 8032a0f3c319..78974da6810e 100644
--- a/include/linux/swap.h
+++ b/include/linux/swap.h
@@ -240,7 +240,7 @@ struct swap_info_struct {
 	struct plist_node list;		/* entry in swap_active_head */
 	signed char	type;		/* strange name for an index */
 	unsigned int	max;		/* size of this swap device */
-	struct swap_cluster_info *cluster_info; /* cluster info. Only for SSD */
+	struct swap_cluster_info *cluster_info; /* array, one entry per cluster */
 	struct list_head free_clusters; /* free clusters list */
 	struct list_head full_clusters; /* full clusters list */
 	struct list_head nonfull_clusters[SWAP_NR_ORDERS];
-- 
2.48.1

[PATCH v4 2/2] mm/swap: scan by cluster in find_next_to_unuse()

[Youngjun Park]

find_next_to_unuse() walks every offset from 0 to si->max, and swapoff
restarts that walk on each retry, so the cost scales with the size of
the device rather than with the few slots the shmem and mmlist passes
could not free.  It has caused stalls before.

The flat walk predates the swap table.  Slot state now lives in a per
cluster table, and wait_for_allocation() stops all allocation before
try_to_unuse() runs, so a cluster that holds no slot in use stays that
way.  Skip such a cluster instead of reading all of its entries.

Fill a 1 TiB swap up to some amount, then swapoff.
What is left sits at the top of what was filled, so every slot below it
is free.  Medians over 11 pairs at 32 and 128 GiB, 3 pairs at 256 and
512.

	filled		swapoff
			old	new
	 32 GiB		 92.4ms	 66.3ms
	128 GiB		157.6ms	 94.4ms
	256 GiB		209.7ms	 63.8ms
	512 GiB		391.8ms	 73.4ms

old grows with how much was filled, new does not.

In the ordinary case swap still holds real data and swapoff spends its
time reading it back. it is tested 4 GiB on an 8 GiB device, where the scan
is 1.4% of try_to_unuse(), and there is no difference either way.

Commit dc644a073769 ("mm: add three more cond_resched() in swapoff")
answered those stalls with a cond_resched() every 256 offsets.  A walk
bounded by one cluster no longer needs that counter.  The loop now runs
at most SWAPFILE_CLUSTER times before it returns or reschedules, the
same bound swap_reclaim_full_clusters() already scans between
cond_resched() calls.

The scan end is clamped to si->max, so the walk stops there rather than
running into the masked tail of the last cluster.

ci->count is read without ci->lock, so READ_ONCE() marks the read for
KCSAN.  Allocation is already stopped, so the count can only drop, and a
slot stops being counted only after its folio has left the swap cache.
An empty cluster therefore holds nothing for try_to_unuse() to act on.

Signed-off-by: Youngjun Park <youngjun.park@lge.com>
Reviewed-by: Barry Song <baohua@kernel.org>
Acked-by: Kairui Song <kasong@tencent.com>
Reviewed-by: Baoquan He <baoquan.he@linux.dev>
---
 mm/swapfile.c | 43 ++++++++++++++++++++++++++++++-------------
 1 file changed, 30 insertions(+), 13 deletions(-)

diff --git a/mm/swapfile.c b/mm/swapfile.c
index 0a3a3b2218c7..05d3408396f9 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -370,8 +370,6 @@ static void discard_swap_cluster(struct swap_info_struct *si,
 	}
 }
 
-#define LATENCY_LIMIT		256
-
 static inline bool cluster_is_empty(struct swap_cluster_info *info)
 {
 	return info->count == 0;
@@ -2787,7 +2785,9 @@ static int unuse_mm(struct mm_struct *mm, unsigned int type)
 static unsigned int find_next_to_unuse(struct swap_info_struct *si,
 					unsigned int prev)
 {
-	unsigned int i;
+	struct swap_cluster_info *ci;
+	unsigned long i, end;
+	unsigned int ci_off;
 	unsigned long swp_tb;
 
 	/*
@@ -2796,19 +2796,36 @@ static unsigned int find_next_to_unuse(struct swap_info_struct *si,
 	 * hits are okay, and sys_swapoff() has already prevented new
 	 * allocations from this area (while holding swap_lock).
 	 */
-	for (i = prev + 1; i < si->max; i++) {
-		swp_tb = swap_table_get(__swap_offset_to_cluster(si, i),
-					i % SWAPFILE_CLUSTER);
-		if (!swp_tb_is_null(swp_tb) && !swp_tb_is_bad(swp_tb))
-			break;
-		if ((i % LATENCY_LIMIT) == 0)
+	i = prev + 1;
+	while (i < si->max) {
+		ci = __swap_offset_to_cluster(si, i);
+		end = min_t(unsigned long,
+			    ALIGN_DOWN(i, SWAPFILE_CLUSTER) + SWAPFILE_CLUSTER,
+			    si->max);
+
+		/*
+		 * An empty cluster has no slot in use, so skip it whole.
+		 * A slot is uncounted only after its folio left the swap
+		 * cache, so there is nothing here for try_to_unuse() to act on.
+		 * Count only drops here, so a READ_ONCE() without ci->lock is
+		 * enough, unlike in every other cluster_is_empty() caller.
+		 */
+		if (!READ_ONCE(ci->count)) {
+			i = end;
 			cond_resched();
-	}
+			continue;
+		}
 
-	if (i == si->max)
-		i = 0;
+		ci_off = i % SWAPFILE_CLUSTER;
+		for (; i < end; ci_off++, i++) {
+			swp_tb = swap_table_get(ci, ci_off);
+			if (!swp_tb_is_null(swp_tb) && !swp_tb_is_bad(swp_tb))
+				return i;
+		}
+		cond_resched();
+	}
 
-	return i;
+	return 0;
 }
 
 static int try_to_unuse(unsigned int type)
-- 
2.48.1

Re: [PATCH v4 0/2] mm/swap: skip empty clusters in the swapoff scan

[Andrew Morton]

On Thu, 10 Sep 2026 01:15:50 +0900 Youngjun Park <youngjun.park@lge.com> wrote:

> find_next_to_unuse() walks a swap device one offset at a time.  Slot
> state now lives in a per cluster swap table, so patch 2 dismisses an
> empty cluster with one counter read instead of SWAPFILE_CLUSTER table
> reads.
> 
> Patch 1 is an unrelated one line comment fix noticed on the way.
> 
> A debug test confirmed the skip path runs, and swapoff completed
> under load with no DEBUG_VM or lockdep splats.

Please let's try to make the [0/N] words provide a summary of the
patchset's effects.  The above didn't even tell people that this is a
swapoff optimization!

I asked an LLM for this summary and we came up with

:  Speed up swapoff and reduce scanning stalls on large, mostly empty
:  swap devices by skipping empty clusters.  Reduce swapoff time on a 1
:  TiB device from 158 ms to 94 ms after filling 128 GiB, and from 392
:  ms to 73 ms after filling 512 GiB; expect little benefit when
:  substantial swap data remains.

Does that sound OK?

Re: [PATCH v4 2/2] mm/swap: scan by cluster in find_next_to_unuse()

[Nhat Pham]

On Wed, Sep 9, 2026 at 9:16 AM Youngjun Park <youngjun.park@lge.com> wrote:
>
> find_next_to_unuse() walks every offset from 0 to si->max, and swapoff
> restarts that walk on each retry, so the cost scales with the size of
> the device rather than with the few slots the shmem and mmlist passes
> could not free.  It has caused stalls before.
>
> The flat walk predates the swap table.  Slot state now lives in a per
> cluster table, and wait_for_allocation() stops all allocation before
> try_to_unuse() runs, so a cluster that holds no slot in use stays that
> way.  Skip such a cluster instead of reading all of its entries.
>
> Fill a 1 TiB swap up to some amount, then swapoff.
> What is left sits at the top of what was filled, so every slot below it
> is free.  Medians over 11 pairs at 32 and 128 GiB, 3 pairs at 256 and
> 512.
>
>         filled          swapoff
>                         old     new
>          32 GiB          92.4ms  66.3ms
>         128 GiB         157.6ms  94.4ms
>         256 GiB         209.7ms  63.8ms
>         512 GiB         391.8ms  73.4ms
>
> old grows with how much was filled, new does not.
>
> In the ordinary case swap still holds real data and swapoff spends its
> time reading it back. it is tested 4 GiB on an 8 GiB device, where the scan
> is 1.4% of try_to_unuse(), and there is no difference either way.
>
> Commit dc644a073769 ("mm: add three more cond_resched() in swapoff")
> answered those stalls with a cond_resched() every 256 offsets.  A walk
> bounded by one cluster no longer needs that counter.  The loop now runs
> at most SWAPFILE_CLUSTER times before it returns or reschedules, the
> same bound swap_reclaim_full_clusters() already scans between
> cond_resched() calls.
>
> The scan end is clamped to si->max, so the walk stops there rather than
> running into the masked tail of the last cluster.
>
> ci->count is read without ci->lock, so READ_ONCE() marks the read for
> KCSAN.  Allocation is already stopped, so the count can only drop, and a
> slot stops being counted only after its folio has left the swap cache.
> An empty cluster therefore holds nothing for try_to_unuse() to act on.
>
> Signed-off-by: Youngjun Park <youngjun.park@lge.com>
> Reviewed-by: Barry Song <baohua@kernel.org>
> Acked-by: Kairui Song <kasong@tencent.com>
> Reviewed-by: Baoquan He <baoquan.he@linux.dev>

Reviewed-by: Nhat Pham <nphamcs@gmail.com>

Re: [PATCH v4 0/2] mm/swap: skip empty clusters in the swapoff scan

[Youngjun Park]

On Wed, Sep 09, 2026 at 10:49:39AM -0700, Andrew Morton wrote:
> On Thu, 10 Sep 2026 01:15:50 +0900 Youngjun Park <youngjun.park@lge.com> wrote:
> 
> > find_next_to_unuse() walks a swap device one offset at a time.  Slot
> > state now lives in a per cluster swap table, so patch 2 dismisses an
> > empty cluster with one counter read instead of SWAPFILE_CLUSTER table
> > reads.
> > 
> > Patch 1 is an unrelated one line comment fix noticed on the way.
> > 
> > A debug test confirmed the skip path runs, and swapoff completed
> > under load with no DEBUG_VM or lockdep splats.
> 
> Please let's try to make the [0/N] words provide a summary of the
> patchset's effects.  The above didn't even tell people that this is a
> swapoff optimization!

Hello Andrew.

I will keep in mind, Thank you for pointing it out.

> I asked an LLM for this summary and we came up with
> 
> :  Speed up swapoff and reduce scanning stalls on large, mostly empty
> :  swap devices by skipping empty clusters.  Reduce swapoff time on a 1
> :  TiB device from 158 ms to 94 ms after filling 128 GiB, and from 392
> :  ms to 73 ms after filling 512 GiB; expect little benefit when
> :  substantial swap data remains.
> 
> Does that sound OK?

Yes, sounds good. summary seems good enough!

Best regards,
Youngjun