[PATCH v3 0/3] mm: zswap: optimize zswap invalidate and store

[PATCH v3 0/3] mm: zswap: optimize zswap invalidate and store

[Kefeng Wang]

This series range-ifies zswap_invalidate() to skip xarray lookups
when zswap is unused, and reuses it in zswap_store() to eliminate
redundant per-slot lookups and open-coded logic.

v3:
- make zswap_invalidate() to take nr_entries and add
  zswap_never_enabled() into it
- since all patches refreshed, so no ACK/RB added

v2:
- split patches and refactor zswap_invalidate(), suggested by
  Johannes Weiner
- add patch4 to skip zswap_invalidate() in swap_range_free()

Kefeng Wang (3):
  mm: zswap: convert zswap_invalidate() to take a range
  mm: zswap: skip xarray walk in zswap_invalidate() when zswap is unused
  mm: zswap: reuse zswap_invalidate() in zswap_store()

 include/linux/zswap.h |  8 ++++++--
 mm/swapfile.c         |  4 +---
 mm/zswap.c            | 45 ++++++++++++++++++++++---------------------
 3 files changed, 30 insertions(+), 27 deletions(-)

-- 
2.55.0

[PATCH v3 1/3] mm: zswap: convert zswap_invalidate() to take a range

[Kefeng Wang]

zswap_invalidate() takes a swp_entry_t only to unpack it right back
into type and offset, and both callers already have those values in
hand. Pass type, offset, and nr_entries directly so swap_range_free()
can invalidate an entire range in one call instead of looping in the
caller.

Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
---
 include/linux/zswap.h |  8 ++++++--
 mm/swapfile.c         |  4 +---
 mm/zswap.c            | 29 +++++++++++++++++++----------
 3 files changed, 26 insertions(+), 15 deletions(-)

diff --git a/include/linux/zswap.h b/include/linux/zswap.h
index 30c193a1207e..df6cafbe95dc 100644
--- a/include/linux/zswap.h
+++ b/include/linux/zswap.h
@@ -27,7 +27,7 @@ struct zswap_lruvec_state {
 unsigned long zswap_total_pages(void);
 bool zswap_store(struct folio *folio);
 int zswap_load(struct folio *folio);
-void zswap_invalidate(swp_entry_t swp);
+void zswap_invalidate(int type, pgoff_t offset, unsigned long nr_entries);
 int zswap_swapon(int type, unsigned long nr_pages);
 void zswap_swapoff(int type);
 void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg);
@@ -49,7 +49,11 @@ static inline int zswap_load(struct folio *folio)
 	return -ENOENT;
 }
 
-static inline void zswap_invalidate(swp_entry_t swp) {}
+static inline void zswap_invalidate(int type, pgoff_t offset,
+		unsigned long nr_entries)
+{
+}
+
 static inline int zswap_swapon(int type, unsigned long nr_pages)
 {
 	return 0;
diff --git a/mm/swapfile.c b/mm/swapfile.c
index 05d3408396f9..891379c95a01 100644
--- a/mm/swapfile.c
+++ b/mm/swapfile.c
@@ -1316,10 +1316,8 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
 {
 	unsigned long end = offset + nr_entries - 1;
 	void (*swap_slot_free_notify)(struct block_device *, unsigned long);
-	unsigned int i;
 
-	for (i = 0; i < nr_entries; i++)
-		zswap_invalidate(swp_entry(si->type, offset + i));
+	zswap_invalidate(si->type, offset, nr_entries);
 
 	if (si->flags & SWP_BLKDEV)
 		swap_slot_free_notify =
diff --git a/mm/zswap.c b/mm/zswap.c
index a12b6452a225..79a66093d7ec 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -228,10 +228,15 @@ static bool zswap_has_pool;
 /* One swap address space for each 64M swap space */
 #define ZSWAP_ADDRESS_SPACE_SHIFT 14
 #define ZSWAP_ADDRESS_SPACE_PAGES (1 << ZSWAP_ADDRESS_SPACE_SHIFT)
+
+static inline struct xarray *zswap_tree(int type, pgoff_t offset)
+{
+	return &zswap_trees[type][offset >> ZSWAP_ADDRESS_SPACE_SHIFT];
+}
+
 static inline struct xarray *swap_zswap_tree(swp_entry_t swp)
 {
-	return &zswap_trees[swp_type(swp)][swp_offset(swp)
-		>> ZSWAP_ADDRESS_SPACE_SHIFT];
+	return zswap_tree(swp_type(swp), swp_offset(swp));
 }
 
 #define zswap_pool_debug(msg, p)			\
@@ -1659,18 +1664,22 @@ int zswap_load(struct folio *folio)
 	return 0;
 }
 
-void zswap_invalidate(swp_entry_t swp)
+void zswap_invalidate(int type, pgoff_t offset, unsigned long nr_entries)
 {
-	pgoff_t offset = swp_offset(swp);
-	struct xarray *tree = swap_zswap_tree(swp);
 	struct zswap_entry *entry;
+	struct xarray *tree;
+	unsigned long i;
 
-	if (xa_empty(tree))
-		return;
+	for (i = 0; i < nr_entries; i++) {
+		tree = zswap_tree(type, offset + i);
 
-	entry = xa_erase(tree, offset);
-	if (entry)
-		zswap_entry_free(entry);
+		if (xa_empty(tree))
+			continue;
+
+		entry = xa_erase(tree, offset + i);
+		if (entry)
+			zswap_entry_free(entry);
+	}
 }
 
 int zswap_swapon(int type, unsigned long nr_pages)
-- 
2.55.0

[PATCH v3 2/3] mm: zswap: skip xarray walk in zswap_invalidate() when zswap is unused

[Kefeng Wang]

zswap_invalidate() still walks the per-area xarray even when zswap has
never been enabled. Add a zswap_never_enabled() to skip it.

Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
---
 mm/zswap.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/mm/zswap.c b/mm/zswap.c
index 79a66093d7ec..132e97748ed2 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1670,6 +1670,9 @@ void zswap_invalidate(int type, pgoff_t offset, unsigned long nr_entries)
 	struct xarray *tree;
 	unsigned long i;
 
+	if (zswap_never_enabled())
+		return;
+
 	for (i = 0; i < nr_entries; ++i) {
 		tree = zswap_tree(type, offset + i);
 
-- 
2.55.0

[PATCH v3 3/3] mm: zswap: reuse zswap_invalidate() in zswap_store()

[Kefeng Wang]

Reuse zswap_invalidate() in zswap_store() check_old path. Its
zswap_never_enabled() and xa_empty() guards avoid redundant xarray
lookups and deduplicate the per-slot free logic.

Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>
---
 mm/zswap.c | 15 ++-------------
 1 file changed, 2 insertions(+), 13 deletions(-)

diff --git a/mm/zswap.c b/mm/zswap.c
index 132e97748ed2..e6edbaf97f0d 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -1546,19 +1546,8 @@ bool zswap_store(struct folio *folio)
 	 * offsets corresponding to each page of the folio. Otherwise,
 	 * writeback could overwrite the new data in the swapfile.
 	 */
-	if (!ret) {
-		unsigned type = swp_type(swp);
-		pgoff_t offset = swp_offset(swp);
-		struct zswap_entry *entry;
-		struct xarray *tree;
-
-		for (index = 0; index < nr_pages; ++index) {
-			tree = swap_zswap_tree(swp_entry(type, offset + index));
-			entry = xa_erase(tree, offset + index);
-			if (entry)
-				zswap_entry_free(entry);
-		}
-	}
+	if (!ret)
+		zswap_invalidate(swp_type(swp), swp_offset(swp), nr_pages);
 
 	return ret;
 }
-- 
2.55.0

Re: [PATCH v3 1/3] mm: zswap: convert zswap_invalidate() to take a range

[Yosry Ahmed]

On Thu, Sep 10, 2026 at 5:36 AM Kefeng Wang <wangkefeng.wang@huawei.com> wrote:
>
> zswap_invalidate() takes a swp_entry_t only to unpack it right back
> into type and offset, and both callers already have those values in
> hand. Pass type, offset, and nr_entries directly so swap_range_free()
> can invalidate an entire range in one call instead of looping in the
> caller.
>
> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>

Acked-by: Yosry Ahmed <yosry@kernel.org>

> ---
>  include/linux/zswap.h |  8 ++++++--
>  mm/swapfile.c         |  4 +---
>  mm/zswap.c            | 29 +++++++++++++++++++----------
>  3 files changed, 26 insertions(+), 15 deletions(-)
>
> diff --git a/include/linux/zswap.h b/include/linux/zswap.h
> index 30c193a1207e..df6cafbe95dc 100644
> --- a/include/linux/zswap.h
> +++ b/include/linux/zswap.h
> @@ -27,7 +27,7 @@ struct zswap_lruvec_state {
>  unsigned long zswap_total_pages(void);
>  bool zswap_store(struct folio *folio);
>  int zswap_load(struct folio *folio);
> -void zswap_invalidate(swp_entry_t swp);
> +void zswap_invalidate(int type, pgoff_t offset, unsigned long nr_entries);
>  int zswap_swapon(int type, unsigned long nr_pages);
>  void zswap_swapoff(int type);
>  void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg);
> @@ -49,7 +49,11 @@ static inline int zswap_load(struct folio *folio)
>         return -ENOENT;
>  }
>
> -static inline void zswap_invalidate(swp_entry_t swp) {}
> +static inline void zswap_invalidate(int type, pgoff_t offset,
> +               unsigned long nr_entries)
> +{
> +}
> +
>  static inline int zswap_swapon(int type, unsigned long nr_pages)
>  {
>         return 0;
> diff --git a/mm/swapfile.c b/mm/swapfile.c
> index 05d3408396f9..891379c95a01 100644
> --- a/mm/swapfile.c
> +++ b/mm/swapfile.c
> @@ -1316,10 +1316,8 @@ static void swap_range_free(struct swap_info_struct *si, unsigned long offset,
>  {
>         unsigned long end = offset + nr_entries - 1;
>         void (*swap_slot_free_notify)(struct block_device *, unsigned long);
> -       unsigned int i;
>
> -       for (i = 0; i < nr_entries; i++)
> -               zswap_invalidate(swp_entry(si->type, offset + i));
> +       zswap_invalidate(si->type, offset, nr_entries);
>
>         if (si->flags & SWP_BLKDEV)
>                 swap_slot_free_notify =
> diff --git a/mm/zswap.c b/mm/zswap.c
> index a12b6452a225..79a66093d7ec 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
> @@ -228,10 +228,15 @@ static bool zswap_has_pool;
>  /* One swap address space for each 64M swap space */
>  #define ZSWAP_ADDRESS_SPACE_SHIFT 14
>  #define ZSWAP_ADDRESS_SPACE_PAGES (1 << ZSWAP_ADDRESS_SPACE_SHIFT)
> +
> +static inline struct xarray *zswap_tree(int type, pgoff_t offset)
> +{
> +       return &zswap_trees[type][offset >> ZSWAP_ADDRESS_SPACE_SHIFT];
> +}
> +
>  static inline struct xarray *swap_zswap_tree(swp_entry_t swp)
>  {
> -       return &zswap_trees[swp_type(swp)][swp_offset(swp)
> -               >> ZSWAP_ADDRESS_SPACE_SHIFT];
> +       return zswap_tree(swp_type(swp), swp_offset(swp));
>  }
>
>  #define zswap_pool_debug(msg, p)                       \
> @@ -1659,18 +1664,22 @@ int zswap_load(struct folio *folio)
>         return 0;
>  }
>
> -void zswap_invalidate(swp_entry_t swp)
> +void zswap_invalidate(int type, pgoff_t offset, unsigned long nr_entries)
>  {
> -       pgoff_t offset = swp_offset(swp);
> -       struct xarray *tree = swap_zswap_tree(swp);
>         struct zswap_entry *entry;
> +       struct xarray *tree;
> +       unsigned long i;
>
> -       if (xa_empty(tree))
> -               return;
> +       for (i = 0; i < nr_entries; i++) {
> +               tree = zswap_tree(type, offset + i);
>
> -       entry = xa_erase(tree, offset);
> -       if (entry)
> -               zswap_entry_free(entry);
> +               if (xa_empty(tree))
> +                       continue;
> +
> +               entry = xa_erase(tree, offset + i);
> +               if (entry)
> +                       zswap_entry_free(entry);
> +       }
>  }
>
>  int zswap_swapon(int type, unsigned long nr_pages)
> --
> 2.55.0
>
>

Re: [PATCH v3 2/3] mm: zswap: skip xarray walk in zswap_invalidate() when zswap is unused

[Yosry Ahmed]

On Thu, Sep 10, 2026 at 5:36 AM Kefeng Wang <wangkefeng.wang@huawei.com> wrote:
>
> zswap_invalidate() still walks the per-area xarray even when zswap has
> never been enabled. Add a zswap_never_enabled() to skip it.
>
> Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>

Acked-by: Yosry Ahmed <yosry@kernel.org>

Re: [PATCH v3 3/3] mm: zswap: reuse zswap_invalidate() in zswap_store()

[Yosry Ahmed]

On Thu, Sep 10, 2026 at 5:36 AM Kefeng Wang <wangkefeng.wang@huawei.com> wrote:
>
> Reuse zswap_invalidate() in zswap_store() check_old path. Its
> zswap_never_enabled() and xa_empty() guards avoid redundant xarray
> lookups and deduplicate the per-slot free logic.
>
> Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>

Acked-by: Yosry Ahmed <yosry@kernel.org>

Re: [PATCH v3 1/3] mm: zswap: convert zswap_invalidate() to take a range

[Johannes Weiner]

On Thu, Sep 10, 2026 at 08:35:42PM +0800, Kefeng Wang wrote:
> zswap_invalidate() takes a swp_entry_t only to unpack it right back
> into type and offset, and both callers already have those values in
> hand. Pass type, offset, and nr_entries directly so swap_range_free()
> can invalidate an entire range in one call instead of looping in the
> caller.
> 
> Suggested-by: Johannes Weiner <hannes@cmpxchg.org>
> Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>

It keeps getting better :)

Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>

Re: [PATCH v3 2/3] mm: zswap: skip xarray walk in zswap_invalidate() when zswap is unused

[Johannes Weiner]

On Thu, Sep 10, 2026 at 08:35:43PM +0800, Kefeng Wang wrote:
> zswap_invalidate() still walks the per-area xarray even when zswap has
> never been enabled. Add a zswap_never_enabled() to skip it.
> 
> Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>

Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>

Re: [PATCH v3 3/3] mm: zswap: reuse zswap_invalidate() in zswap_store()

[Johannes Weiner]

On Thu, Sep 10, 2026 at 08:35:44PM +0800, Kefeng Wang wrote:
> Reuse zswap_invalidate() in zswap_store() check_old path. Its
> zswap_never_enabled() and xa_empty() guards avoid redundant xarray
> lookups and deduplicate the per-slot free logic.
> 
> Signed-off-by: Kefeng Wang <wangkefeng.wang@huawei.com>

Reviewed-by: Johannes Weiner <hannes@cmpxchg.org>