Linux-mm Archive on lore.kernel.org
 help / color / mirror / Atom feed
* [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages
@ 2026-11-15  2:44 Nanzhe Zhao
  2026-11-15  2:44 ` [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages Nanzhe Zhao
                   ` (3 more replies)
  0 siblings, 4 replies; 8+ messages in thread
From: Nanzhe Zhao @ 2026-11-15  2:44 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Nanzhe Zhao, linux-mm, linux-block, linux-kernel, Jens Axboe,
	Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner,
	Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li,
	David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts,
	Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee,
	Nick Terrell, Barry Song, Tangquan Zheng, Bo Zhang

When large folios are compressed at a larger granularity, we observe
a notable reduction in CPU usage and a significant improvement in
compression ratios.

mTHP's ability to be swapped out without splitting and swapped back in
as a whole allows compression and decompression at larger granularities.

This patchset enhances zsmalloc and zram by adding support for dividing
large folios into multi-page blocks, typically configured with a
2-order granularity. Without this patchset, a large folio is always
divided into nr_pages 4KiB blocks.

The granularity can be set using the ZSMALLOC_MULTI_PAGES_ORDER
setting, where the default of 2 allows all anonymous THP to benefit.

Examples include:
* A 16KiB large folio will be compressed and stored as a single 16KiB
  block.
* A 64KiB large folio will be compressed and stored as four 16KiB
  blocks.

We captured anonymous pages from four Android applications on a Pixel 6.
To evaluate throughput, we allocate a buffer via mmap(MAP_PRIVATE |
MAP_ANONYMOUS), populate it with the captured pages, and invoke
madvise(MADV_PAGEOUT) to swap the buffer out to zram, measuring the
pageout duration. We then sequentially touch every 4 KiB page to fault
them back in from zram, measuring the swapin duration.

For example, testing the 143 MiB WeChat corpus (times in seconds;
w/o: without the patch, w/: with the patch):

Algorithm  pageout w/o (s)  pageout w/            swapin w/o (s)  swapin w/
lz4        0.247            0.210 (-14.8%)        0.137           0.113 (-17.6%)
lzo-rle    0.240            0.207 (-13.7%)        0.149           0.123 (-17.5%)
zstd       0.567            0.452 (-20.4%)        0.258           0.192 (-25.6%)

v3 left two issues to discuss:

The first issue is internal fragmentation. A compressed 16 KiB block
is rounded up to the next size-class boundary. If it lands in a class
with a large gap, the gap becomes per-object padding.

On all four applications, unused pages in partially filled zspages
increase by ~1.8 to 2.4 MiB, and the per-object class gap is wider.
However, because the number of allocated objects drops by ~74% (from
four 4 KiB objects to one 16 KiB object), total rounding waste
actually falls by 13% to 31%. Combined with 5% to 15% better
compression ratios from multi-page compression, total zsmalloc memory
occupancy (mem_used_total) is reduced across all applications.

For example, below is the memory breakdown under lz4 for our four apps
(lzo-rle and zstd show the same trend, saving 3.8% to 9.3% total memory):

App          objects (w/o -> w/)        rounding waste (MiB)    unused pages (MiB)      mem_used_total (MiB)
WPS          27,530 ->  7,285 (-73.5%)  0.80 -> 0.56 (-29.8%)   1.07 -> 3.35 (+214.0%)   36.4 ->  34.9 (-4.0%)
WeChat       32,460 ->  8,450 (-74.0%)  0.89 -> 0.62 (-30.6%)   1.35 -> 3.32 (+145.8%)   42.8 ->  40.0 (-6.7%)
Douyin lite  62,668 -> 16,218 (-74.1%)  2.07 -> 1.61 (-22.3%)   1.37 -> 3.78 (+176.0%)  100.8 ->  93.6 (-7.1%)
JK chess     71,950 -> 18,584 (-74.2%)  1.75 -> 1.52 (-13.0%)   1.43 -> 3.28 (+129.7%)  117.0 -> 112.7 (-3.6%)

Here, objects is obj_allocated, rounding waste is total class padding
across all objects, unused pages is memory left in partially filled
zspages, and mem_used_total is the total memory allocated by zsmalloc
(including fragmentation and metadata).

We also built a 256 MiB lz4 synthetic extreme case: 16 KiB blocks with
the first K bytes random and the rest zero, with K tuned so the object
lands one byte above the previous class. Below is the code to generate
this pattern:

#define BLOCK 16384

static void make_block(unsigned char *buf, size_t k)
{
	size_t i;

	for (i = 0; i < k; i++)
		buf[i] = rand();
	memset(buf + k, 0, BLOCK - k);
}

static void write_corpus(int fd, size_t k, size_t nblocks)
{
	unsigned char *buf = malloc(BLOCK);
	size_t i;

	make_block(buf, k);
	for (i = 0; i < nblocks; i++)
		write(fd, buf, BLOCK);
}

With the default CONFIG_ZSMALLOC_CHAIN_SIZE=8, the largest non-huge
class gap is 576 bytes. Setting K to 10,781 bytes lands the object in
class 11,456 with a 575 B/object padding, increasing total memory by
5.0% over the baseline without the patch (179.3 MiB vs 170.7 MiB).

With CONFIG_ZSMALLOC_CHAIN_SIZE=10, the classes are denser and the
largest gap drops to 448 bytes. At K=10,781 bytes, the memory overhead
falls to 3.9% (177.3 MiB vs 170.7 MiB).

A potential future optimization is during size-class merging: we can
avoid merging classes if the resulting gap exceeds a threshold (e.g.
ZS_MAX_MERGE_GAP = 256 bytes), for example:

--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ can_merge() @@
+#define ZS_MAX_MERGE_GAP	256
+
+	if (prev->size - size > ZS_MAX_MERGE_GAP)
+		return false;
+
 	if (prev->zpdescs_per_zspage == zpdescs_per_zspage &&
 		prev->objs_per_zspage == objs_per_zspage)
 		return true;

The second issue is partial read. If the order-2 allocation for the
swapped-in folio fails and falls back to order-0, we must decompress
the entire 16 KiB block just for a 4 KiB swapin.

v3's 4/4 ("mm: fall back to four small folios if mTHP allocation
fails") handled this by building four small folios. Usama asked whether
defrag=always could replace this patch. But we only need direct reclaim
for the do_swap_page order-2 allocation; otherwise all anonymous mTHP
allocations would enter direct reclaim. So the new 4/4 adds
__GFP_RECLAIM to the order-2 swapin allocation in do_swap_page().

We measured the cost of entering direct reclaim in the swapin path on
an i7-14700HX with no_turbo=1: a 1 GiB memcg, zram (zstd), 16 KiB mTHP,
and an arm64 defconfig vmlinux build with -j20 in the background. The
bench was pinned to one CPU and the build to the rest. We compared
forcing direct reclaim with the existing behavior, 10 totals per case,
each total 30 rounds of 100 MiB anonymous pageout/swapin.

#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <time.h>

static double now(void)
{
	struct timespec ts;

	clock_gettime(CLOCK_MONOTONIC, &ts);
	return ts.tv_sec + ts.tv_nsec / 1e9;
}

int main(void)
{
	const size_t size = 100UL << 20;
	unsigned long *p, sum = 0;
	size_t i, off;
	double t0, t1;

	p = mmap(NULL, size, PROT_READ | PROT_WRITE,
		 MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
	if (p == MAP_FAILED) {
		perror("mmap");
		return 1;
	}

	for (i = 0; i < size / sizeof(*p); i++)
		p[i] = i & 0xfff;

	t0 = now();
	if (madvise(p, size, MADV_PAGEOUT)) {
		perror("madvise(MADV_PAGEOUT)");
		return 1;
	}
	t1 = now();
	printf("pageout %.6f\n", t1 - t0);

	t0 = now();
	for (off = 0; off < size; off += 4096)
		sum += *(volatile unsigned long *)((char *)p + off);
	t1 = now();
	printf("swapin %.6f sum %lu\n", t1 - t0, sum);

	return 0;
}

                         w/o DR          DR
partial reads            235,832         218
swapin total (s)         24.85 +/- 0.56    22.80 +/- 0.73
pswpout (pages)          11.4M           11.3M

Here, partial reads is the zram_bio_read_multi_pages_partial counter.
The direct-reclaim case did not show a measurable cost. Almost all
partial decompressions were eliminated, making swapin 8.3% faster.

-v4:
 * Rebased onto latest akpm mm-new.
 * Patch 1: Fixed multi-page page accounting (NR_ZSPAGES, zs_compact,
   and debugfs) to use physical pages via class_pages_per_zspage().
 * Patch 2: Adapted to upstream's new zcomp read/write stream separation;
   added a preallocated decomp_buf in zcomp_dstrm for partial reads to
   avoid memory allocation in the swapin path.
 * Patch 3: Updated zstd backend to pass ZCOMP_MULTI_PAGES_SIZE to
   zstd_setup_params().
 * Patch 4: Dropped v3's 4/4 fallback and added a new patch to allow
   direct reclaim for synchronous multi-page swapin.

[1] v3: https://lore.kernel.org/all/20241121222521.83458-1-21cnbao@gmail.com/

Barry Song (1):
  zram: backend_zstd: use multi-page size for zstd parameter setup

Nanzhe Zhao (1):
  mm: swap: allow direct reclaim for synchronous multi-page swapin

Tangquan Zheng (2):
  mm: zsmalloc: support objects compressed based on multiple pages
  zram: support compression at the granularity of multi-pages

 drivers/block/zram/Kconfig        |   9 +
 drivers/block/zram/backend_zstd.c |   4 +-
 drivers/block/zram/zcomp.c        |  31 +-
 drivers/block/zram/zcomp.h        |  19 +-
 drivers/block/zram/zram_drv.c     | 499 +++++++++++++++++++++++++++---
 drivers/block/zram/zram_drv.h     |  43 +++
 include/linux/zsmalloc.h          |  10 +-
 mm/Kconfig                        |  29 +-
 mm/swap_state.c                   |   7 +
 mm/zsmalloc.c                     | 242 ++++++++++-----
 10 files changed, 757 insertions(+), 136 deletions(-)

-- 
2.43.0



^ permalink raw reply	[flat|nested] 8+ messages in thread

* [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages
  2026-11-15  2:44 [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages Nanzhe Zhao
@ 2026-11-15  2:44 ` Nanzhe Zhao
  2026-11-16  2:46   ` sashiko-bot
  2026-11-15  2:44 ` [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages Nanzhe Zhao
                   ` (2 subsequent siblings)
  3 siblings, 1 reply; 8+ messages in thread
From: Nanzhe Zhao @ 2026-11-15  2:44 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Tangquan Zheng, linux-mm, linux-block, linux-kernel, Jens Axboe,
	Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner,
	Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li,
	David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts,
	Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee,
	Nick Terrell, Barry Song, Bo Zhang, Nanzhe Zhao

From: Tangquan Zheng <zhengtangquan@oppo.com>

This patch introduces support for zsmalloc to store compressed objects
based on multi-pages. Previously, a large folio with nr_pages subpages
would undergo compression one by one, each at the granularity of
PAGE_SIZE. However, by compressing them at a larger granularity, we
can conserve both memory and CPU resources.

We define the granularity with a configuration option called
ZSMALLOC_MULTI_PAGES_ORDER, set to a default value of 2, which matches
the minimum order of anonymous mTHP. As a result, a large folio with
8 subpages will now be split into 2 parts instead of 8.

The introduction of the multi-pages feature necessitates the creation
of new size classes to accommodate it.

Signed-off-by: Tangquan Zheng <zhengtangquan@oppo.com>
Co-developed-by: Barry Song <baohua@kernel.org>
Signed-off-by: Barry Song <baohua@kernel.org>
Co-developed-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com>
Signed-off-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com>
---
 drivers/block/zram/zram_drv.c |   3 +-
 include/linux/zsmalloc.h      |  10 +-
 mm/Kconfig                    |  29 +++-
 mm/zsmalloc.c                 | 245 +++++++++++++++++++++++-----------
 4 files changed, 204 insertions(+), 83 deletions(-)

diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 4f479d905142..24eefcf143de 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -2024,7 +2024,8 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize)
 	}
 
 	if (!huge_class_size)
-		huge_class_size = zs_huge_class_size(zram->mem_pool);
+		huge_class_size = zs_huge_class_size(zram->mem_pool,
+						ZSMALLOC_TYPE_BASEPAGE);
 
 	lockdep_register_key(&zram->table_lock_key);
 	lockdep_init_map(&zram->table_lock_map, "zram->table[index].lock", &zram->table_lock_key, 0);
diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h
index 8971eef333ba..738260358a9a 100644
--- a/include/linux/zsmalloc.h
+++ b/include/linux/zsmalloc.h
@@ -17,6 +17,14 @@
 #include <linux/compiler.h>
 #include <linux/types.h>
 
+enum zsmalloc_type {
+	ZSMALLOC_TYPE_BASEPAGE,
+#ifdef CONFIG_ZSMALLOC_MULTI_PAGES
+	ZSMALLOC_TYPE_MULTI_PAGES,
+#endif
+	ZSMALLOC_TYPE_MAX,
+};
+
 struct zs_pool_stats {
 	/* How many pages were migrated (freed) */
 	atomic_long_t pages_compacted;
@@ -55,7 +63,7 @@ void zs_free(struct zs_pool *pool, struct zs_handle *handle);
 void zs_rebind_handle(struct zs_pool *pool, struct zs_handle *dst,
 		      struct zs_handle *src);
 
-size_t zs_huge_class_size(struct zs_pool *pool);
+size_t zs_huge_class_size(struct zs_pool *pool, enum zsmalloc_type type);
 
 unsigned long zs_get_total_pages(struct zs_pool *pool);
 unsigned long zs_compact(struct zs_pool *pool);
diff --git a/mm/Kconfig b/mm/Kconfig
index afff1b25b543..b68720f6fb6b 100644
--- a/mm/Kconfig
+++ b/mm/Kconfig
@@ -140,14 +140,15 @@ config ZSMALLOC_STAT
 	  If unsure, say N.
 
 config ZSMALLOC_CHAIN_SIZE
-	int "Maximum number of physical pages per-zspage"
+	int "Maximum number of chain elements per-zspage"
 	default 8
 	range 4 16
 	help
-	  This option sets the upper limit on the number of physical pages
-	  that a zmalloc page (zspage) can consist of. The optimal zspage
-	  chain size is calculated for each size class during the
-	  initialization of the pool.
+	  This option sets the upper limit on the number of chain elements
+	  that a zsmalloc page (zspage) can consist of. Each element is a
+	  single page, or a multi-page allocation when ZSMALLOC_MULTI_PAGES
+	  is enabled. The optimal zspage chain size is calculated for each
+	  size class during the initialization of the pool.
 
 	  Changing this option can alter the characteristics of size classes,
 	  such as the number of pages per zspage and the number of objects
@@ -157,6 +158,24 @@ config ZSMALLOC_CHAIN_SIZE
 
 	  For more information, see zsmalloc documentation.
 
+config ZSMALLOC_MULTI_PAGES
+	bool "support zsmalloc multiple pages"
+	depends on ZSMALLOC && !HIGHMEM
+	help
+	  This option configures zsmalloc to support allocations larger than
+	  PAGE_SIZE, enabling compression across multiple pages. The size of
+	  these multiple pages is determined by the configured
+	  ZSMALLOC_MULTI_PAGES_ORDER.
+
+config ZSMALLOC_MULTI_PAGES_ORDER
+	int "zsmalloc multiple pages order"
+	default 2
+	range 1 9
+	depends on ZSMALLOC_MULTI_PAGES
+	help
+	  This option is used to configure zsmalloc to support the compression
+	  of multiple pages.
+
 endmenu
 
 menu "Slab allocator options"
diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 4a6ec43f418b..2d83fcc8610d 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -48,6 +48,12 @@
 
 #define ZSPAGE_MAGIC	0x58
 
+#ifdef CONFIG_ZSMALLOC_MULTI_PAGES
+#define ZSMALLOC_MULTI_PAGES_ORDER	(_AC(CONFIG_ZSMALLOC_MULTI_PAGES_ORDER, UL))
+#define ZSMALLOC_MULTI_PAGES_NR		(1 << ZSMALLOC_MULTI_PAGES_ORDER)
+#define ZSMALLOC_MULTI_PAGES_SIZE	(PAGE_SIZE * ZSMALLOC_MULTI_PAGES_NR)
+#endif
+
 /*
  * This must be power of 2 and greater than or equal to sizeof(link_free).
  * These two conditions ensure that any 'struct link_free' itself doesn't
@@ -114,7 +120,7 @@
 
 #define HUGE_BITS	1
 #define FULLNESS_BITS	4
-#define CLASS_BITS	8
+#define CLASS_BITS	9
 #define MAGIC_VAL_BITS	8
 
 #define ZS_MAX_PAGES_PER_ZSPAGE	(_AC(CONFIG_ZSMALLOC_CHAIN_SIZE, UL))
@@ -172,7 +178,11 @@ static_assert(ZS_OBJ_IDX_BITS >= ZS_PAGES_PER_ZSPAGE_BITS,
 	      "zsmalloc: ZS_MIN_ALLOC_SIZE would exceed ZS_MAX_ALLOC_SIZE");
 
 /* each chunk includes extra space to keep handle */
+#ifdef CONFIG_ZSMALLOC_MULTI_PAGES
+#define ZS_MAX_ALLOC_SIZE	ZSMALLOC_MULTI_PAGES_SIZE
+#else
 #define ZS_MAX_ALLOC_SIZE	PAGE_SIZE
+#endif
 
 /*
  * On systems with 4K page size, this gives 255 size classes! There is a
@@ -187,14 +197,45 @@ static_assert(ZS_OBJ_IDX_BITS >= ZS_PAGES_PER_ZSPAGE_BITS,
  *  ZS_MIN_ALLOC_SIZE and ZS_SIZE_CLASS_DELTA must be multiple of ZS_ALIGN
  *  (reason above)
  */
-#define ZS_SIZE_CLASS_DELTA	(PAGE_SIZE >> CLASS_BITS)
+#define ZS_SIZE_CLASS_DELTA	(PAGE_SIZE >> (CLASS_BITS - 1))
 #define _ZS_MIN_ALLOC_RAW \
 	DIV_ROUND_UP(ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT, ZS_OBJ_IDX_MASK)
 #define ZS_MIN_ALLOC_SIZE \
 	MIN(ZS_MAX_ALLOC_SIZE, \
 	    MAX(32UL, ALIGN(_ZS_MIN_ALLOC_RAW, ZS_ALIGN)))
-#define ZS_SIZE_CLASSES	(DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - ZS_MIN_ALLOC_SIZE, \
-				      ZS_SIZE_CLASS_DELTA) + 1)
+
+#define ZS_PAGE_SIZE_CLASSES	(DIV_ROUND_UP(PAGE_SIZE - ZS_MIN_ALLOC_SIZE, \
+					    ZS_SIZE_CLASS_DELTA) + 1)
+
+#ifdef CONFIG_ZSMALLOC_MULTI_PAGES
+#define ZS_MULTI_PAGES_SIZE_CLASS_DELTA	(ZSMALLOC_MULTI_PAGES_SIZE >> (CLASS_BITS - 1))
+#define ZS_MULTI_PAGES_SIZE_CLASSES	(DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - PAGE_SIZE, \
+					      ZS_MULTI_PAGES_SIZE_CLASS_DELTA) + 1)
+#define ZS_SIZE_CLASSES		(ZS_PAGE_SIZE_CLASSES + ZS_MULTI_PAGES_SIZE_CLASSES)
+#else
+#define ZS_SIZE_CLASSES		(DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - ZS_MIN_ALLOC_SIZE, \
+					      ZS_SIZE_CLASS_DELTA) + 1)
+#endif
+
+#ifdef CONFIG_ZSMALLOC_MULTI_PAGES
+static inline unsigned int class_size_to_zs_order(unsigned long size)
+{
+	if (size > PAGE_SIZE)
+		return ZSMALLOC_MULTI_PAGES_ORDER;
+
+	return 0;
+}
+#else
+static inline unsigned int class_size_to_zs_order(unsigned long size)
+{
+	return 0;
+}
+#endif
+
+static inline unsigned long class_size_to_zs_size(unsigned long size)
+{
+	return PAGE_SIZE * (1 << class_size_to_zs_order(size));
+}
 
 /*
  * Pages are distinguished by the ratio of used memory (that is the ratio
@@ -230,7 +271,7 @@ struct zs_size_stat {
 static struct dentry *zs_stat_root;
 #endif
 
-static size_t huge_class_size;
+static size_t huge_class_size[ZSMALLOC_TYPE_MAX];
 
 struct size_class {
 	spinlock_t lock;
@@ -241,13 +282,20 @@ struct size_class {
 	 */
 	int size;
 	int objs_per_zspage;
-	/* Number of PAGE_SIZE sized pages to combine to form a 'zspage' */
-	int pages_per_zspage;
+	/* Number of zpdescs to combine to form a 'zspage' */
+	int zpdescs_per_zspage;
 
 	unsigned int index;
 	struct zs_size_stat stats;
 };
 
+/* Number of physical pages in a zspage */
+static inline unsigned long class_pages_per_zspage(const struct size_class *class)
+{
+	return (unsigned long)class->zpdescs_per_zspage <<
+	       class_size_to_zs_order(class->size);
+}
+
 /*
  * Placed within free objects to form a singly linked list.
  * For every zspage, zspage->freeobj gives head of this list.
@@ -295,27 +343,28 @@ struct zs_pool {
 
 static inline void zpdesc_inc_zone_page_state(struct zpdesc *zpdesc)
 {
-	inc_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES);
+	__zone_stat_add_folio(zpdesc_folio(zpdesc), NR_ZSPAGES);
 }
 
 static inline void zpdesc_dec_zone_page_state(struct zpdesc *zpdesc)
 {
-	dec_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES);
+	__zone_stat_sub_folio(zpdesc_folio(zpdesc), NR_ZSPAGES);
 }
 
-static inline struct zpdesc *alloc_zpdesc(gfp_t gfp, const int nid)
+static inline struct zpdesc *alloc_zpdesc(gfp_t gfp, const int nid,
+					  unsigned int order)
 {
-	struct page *page = alloc_pages_node(nid, gfp, 0);
+	struct page *page = alloc_pages_node(nid, gfp | __GFP_COMP, order);
 
 	return page_zpdesc(page);
 }
 
-static inline void free_zpdesc(struct zpdesc *zpdesc)
+static inline void free_zpdesc(struct zpdesc *zpdesc, unsigned int order)
 {
 	struct page *page = zpdesc_page(zpdesc);
 
 	/* PageZsmalloc is sticky until the page is freed to the buddy. */
-	__free_page(page);
+	__free_pages(page, order);
 }
 
 #define ZS_PAGE_UNLOCKED	0
@@ -420,7 +469,7 @@ static void zspage_write_unlock(struct zspage *zspage)
 	spin_unlock(&zsl->lock);
 }
 
-/* huge object: pages_per_zspage == 1 && maxobj_per_zspage == 1 */
+/* huge object: zpdescs_per_zspage == 1 && maxobj_per_zspage == 1 */
 static void SetZsHugePage(struct zspage *zspage)
 {
 	zspage->huge = 1;
@@ -535,6 +584,15 @@ static int get_size_class_index(int size)
 {
 	int idx = 0;
 
+#ifdef CONFIG_ZSMALLOC_MULTI_PAGES
+	if (size > PAGE_SIZE + ZS_HANDLE_SIZE) {
+		idx = ZS_PAGE_SIZE_CLASSES;
+		idx += DIV_ROUND_UP(size - PAGE_SIZE,
+				    ZS_MULTI_PAGES_SIZE_CLASS_DELTA);
+		return min_t(int, ZS_SIZE_CLASSES - 1, idx);
+	}
+#endif
+
 	if (likely(size > ZS_MIN_ALLOC_SIZE))
 		idx = DIV_ROUND_UP(size - ZS_MIN_ALLOC_SIZE,
 				ZS_SIZE_CLASS_DELTA);
@@ -594,11 +652,11 @@ static int zs_stats_size_show(struct seq_file *s, void *v)
 	unsigned long total_freeable = 0;
 	unsigned long inuse_totals[NR_FULLNESS_GROUPS] = {0, };
 
-	seq_printf(s, " %5s %5s %9s %9s %9s %9s %9s %9s %9s %9s %9s %9s %9s %13s %10s %10s %16s %8s\n",
-			"class", "size", "10%", "20%", "30%", "40%",
+	seq_printf(s, " %5s %5s %5s %9s %9s %9s %9s %9s %9s %9s %9s %9s %9s %9s %13s %10s %10s %16s %16s %8s\n",
+			"class", "size", "order", "10%", "20%", "30%", "40%",
 			"50%", "60%", "70%", "80%", "90%", "99%", "100%",
 			"obj_allocated", "obj_used", "pages_used",
-			"pages_per_zspage", "freeable");
+			"pages_per_zspage", "objs_per_zspage", "freeable");
 
 	for (i = 0; i < ZS_SIZE_CLASSES; i++) {
 
@@ -609,7 +667,8 @@ static int zs_stats_size_show(struct seq_file *s, void *v)
 
 		spin_lock(&class->lock);
 
-		seq_printf(s, " %5u %5u ", i, class->size);
+		seq_printf(s, " %5u %5u %5u", i, class->size,
+			   class_size_to_zs_order(class->size));
 		for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) {
 			inuse_totals[fg] += class_stat_read(class, fg);
 			seq_printf(s, "%9lu ", class_stat_read(class, fg));
@@ -622,11 +681,12 @@ static int zs_stats_size_show(struct seq_file *s, void *v)
 
 		objs_per_zspage = class->objs_per_zspage;
 		pages_used = obj_allocated / objs_per_zspage *
-				class->pages_per_zspage;
+				class_pages_per_zspage(class);
 
-		seq_printf(s, "%13lu %10lu %10lu %16d %8lu\n",
+		seq_printf(s, "%13lu %10lu %10lu %16lu %16d %8lu\n",
 			   obj_allocated, obj_used, pages_used,
-			   class->pages_per_zspage, freeable);
+			   class_pages_per_zspage(class), objs_per_zspage,
+			   freeable);
 
 		total_objs += obj_allocated;
 		total_used_objs += obj_used;
@@ -634,13 +694,13 @@ static int zs_stats_size_show(struct seq_file *s, void *v)
 		total_freeable += freeable;
 	}
 
-	seq_printf(s, "\n %5s %5s ", "Total", "");
+	seq_printf(s, "\n %5s %5s %5s ", "Total", "", "");
 
 	for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++)
 		seq_printf(s, "%9lu ", inuse_totals[fg]);
 
-	seq_printf(s, "%13lu %10lu %10lu %16s %8lu\n",
-		   total_objs, total_used_objs, total_pages, "",
+	seq_printf(s, "%13lu %10lu %10lu %16s %16s %8lu\n",
+		   total_objs, total_used_objs, total_pages, "", "",
 		   total_freeable);
 
 	return 0;
@@ -915,7 +975,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class,
 	assert_spin_locked(&class->lock);
 	__free_zspage_lockless(zspage);
 	class_stat_sub(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage);
-	atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated);
+	atomic_long_sub(class_pages_per_zspage(class), &pool->pages_allocated);
 }
 
 static void free_zspage(struct zs_pool *pool, struct size_class *class,
@@ -943,6 +1003,7 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 {
 	unsigned int freeobj = 1;
 	unsigned long off = 0;
+	unsigned long page_size = class_size_to_zs_size(class->size);
 	struct zpdesc *zpdesc = get_first_zpdesc(zspage);
 
 	while (zpdesc) {
@@ -955,7 +1016,7 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 		vaddr = kmap_local_zpdesc(zpdesc);
 		link = (struct link_free *)vaddr + off / sizeof(*link);
 
-		while ((off += class->size) < PAGE_SIZE) {
+		while ((off += class->size) < page_size) {
 			link->next = freeobj++ << OBJ_TAG_BITS;
 			link += class->size / sizeof(*link);
 		}
@@ -977,7 +1038,7 @@ static void init_zspage(struct size_class *class, struct zspage *zspage)
 		}
 		kunmap_local(vaddr);
 		zpdesc = next_zpdesc;
-		off %= PAGE_SIZE;
+		off %= page_size;
 	}
 
 	set_freeobj(zspage, 0);
@@ -989,7 +1050,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage,
 	int i;
 	struct zpdesc *zpdesc;
 	struct zpdesc *prev_zpdesc = NULL;
-	int nr_zpdescs = class->pages_per_zspage;
+	int nr_zpdescs = class->zpdescs_per_zspage;
 
 	/*
 	 * Allocate individual pages and link them together as:
@@ -1006,7 +1067,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage,
 		if (i == 0) {
 			zspage->first_zpdesc = zpdesc;
 			if (unlikely(class->objs_per_zspage == 1 &&
-					class->pages_per_zspage == 1))
+					class->zpdescs_per_zspage == 1))
 				SetZsHugePage(zspage);
 		} else {
 			prev_zpdesc->next = zpdesc;
@@ -1023,6 +1084,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool,
 				   gfp_t gfp, const int nid)
 {
 	int i;
+	unsigned int order = class_size_to_zs_order(class->size);
 	struct zpdesc *zpdescs[ZS_MAX_PAGES_PER_ZSPAGE];
 	struct zspage *zspage = cache_alloc_zspage(gfp);
 
@@ -1032,19 +1094,24 @@ static struct zspage *alloc_zspage(struct zs_pool *pool,
 	if (!IS_ENABLED(CONFIG_COMPACTION))
 		gfp &= ~__GFP_MOVABLE;
 
+	/*
+	 * Multi-page zspages are not movable, keep them out of ZONE_MOVABLE.
+	 */
+	if (order)
+		gfp &= ~__GFP_MOVABLE;
+
 	zspage->magic = ZSPAGE_MAGIC;
 	zspage->pool = pool;
 	zspage->class = class->index;
 	zspage_lock_init(zspage);
 
-	for (i = 0; i < class->pages_per_zspage; i++) {
-		struct zpdesc *zpdesc;
+	for (i = 0; i < class->zpdescs_per_zspage; i++) {
+		struct zpdesc *zpdesc = alloc_zpdesc(gfp, nid, order);
 
-		zpdesc = alloc_zpdesc(gfp, nid);
 		if (!zpdesc) {
 			while (--i >= 0) {
 				zpdesc_dec_zone_page_state(zpdescs[i]);
-				free_zpdesc(zpdescs[i]);
+				free_zpdesc(zpdescs[i], order);
 			}
 			cache_free_zspage(zspage);
 			return NULL;
@@ -1076,10 +1143,10 @@ static struct zspage *find_get_zspage(struct size_class *class)
 	return zspage;
 }
 
-static bool can_merge(struct size_class *prev, int pages_per_zspage,
+static bool can_merge(struct size_class *prev, int zpdescs_per_zspage,
 					int objs_per_zspage)
 {
-	if (prev->pages_per_zspage == pages_per_zspage &&
+	if (prev->zpdescs_per_zspage == zpdescs_per_zspage &&
 		prev->objs_per_zspage == objs_per_zspage)
 		return true;
 
@@ -1128,7 +1195,7 @@ void zs_obj_read_sg_begin(struct zs_pool *pool, const struct zs_handle *handle,
 {
 	struct zspage *zspage;
 	struct zpdesc *zpdesc;
-	unsigned long obj, off;
+	unsigned long obj, off, page_size;
 	unsigned int obj_idx;
 	struct size_class *class;
 
@@ -1143,12 +1210,13 @@ void zs_obj_read_sg_begin(struct zs_pool *pool, const struct zs_handle *handle,
 	read_unlock(&pool->lock);
 
 	class = zspage_class(pool, zspage);
-	off = offset_in_page(class->size * obj_idx);
+	page_size = class_size_to_zs_size(class->size);
+	off = (class->size * obj_idx) & (page_size - 1);
 
 	if (!ZsHugePage(zspage))
 		off += ZS_HANDLE_SIZE;
 
-	if (off + mem_len <= PAGE_SIZE) {
+	if (off + mem_len <= page_size) {
 		/* this object is contained entirely within a page */
 		sg_init_table(sg, 1);
 		sg_set_page(sg, zpdesc_page(zpdesc), mem_len, off);
@@ -1156,7 +1224,7 @@ void zs_obj_read_sg_begin(struct zs_pool *pool, const struct zs_handle *handle,
 		size_t sizes[2];
 
 		/* this object spans two pages */
-		sizes[0] = PAGE_SIZE - off;
+		sizes[0] = page_size - off;
 		sizes[1] = mem_len - sizes[0];
 
 		sg_init_table(sg, 2);
@@ -1189,16 +1257,17 @@ static void obj_write(struct size_class *class, struct zspage *zspage,
 		      unsigned long obj, const void *src, size_t size)
 {
 	struct zpdesc *zpdesc;
+	unsigned long page_size = class_size_to_zs_size(class->size);
 	unsigned long off;
 	unsigned int obj_idx;
 
 	obj_to_location(obj, &zpdesc, &obj_idx);
-	off = offset_in_page(class->size * obj_idx);
+	off = (class->size * obj_idx) & (page_size - 1);
 
 	if (!ZsHugePage(zspage))
 		off += ZS_HANDLE_SIZE;
 
-	if (off + size <= PAGE_SIZE) {
+	if (off + size <= page_size) {
 		/* this object is contained entirely within a page */
 		void *dst = kmap_local_zpdesc(zpdesc);
 
@@ -1208,7 +1277,7 @@ static void obj_write(struct size_class *class, struct zspage *zspage,
 		/* this object spans two pages */
 		size_t sizes[2];
 
-		sizes[0] = PAGE_SIZE - off;
+		sizes[0] = page_size - off;
 		sizes[1] = size - sizes[0];
 
 		memcpy_to_page(zpdesc_page(zpdesc), off, src, sizes[0]);
@@ -1230,9 +1299,9 @@ static void obj_write(struct size_class *class, struct zspage *zspage,
  *
  * Return: the size (in bytes) of the first huge zsmalloc &size_class.
  */
-size_t zs_huge_class_size(struct zs_pool *pool)
+size_t zs_huge_class_size(struct zs_pool *pool, enum zsmalloc_type type)
 {
-	return huge_class_size;
+	return huge_class_size[type];
 }
 EXPORT_SYMBOL_GPL(zs_huge_class_size);
 
@@ -1245,15 +1314,17 @@ static unsigned long obj_malloc(struct zs_pool *pool,
 	struct size_class *class;
 
 	struct zpdesc *m_zpdesc;
-	unsigned long m_offset;
+	unsigned long m_offset, page_size, page_shift;
 	void *vaddr;
 
 	class = pool->size_class[zspage->class];
 	obj = get_freeobj(zspage);
 
+	page_size = class_size_to_zs_size(class->size);
+	page_shift = PAGE_SHIFT + class_size_to_zs_order(class->size);
 	offset = obj * class->size;
-	nr_zpdesc = offset >> PAGE_SHIFT;
-	m_offset = offset_in_page(offset);
+	nr_zpdesc = offset >> page_shift;
+	m_offset = offset & (page_size - 1);
 	m_zpdesc = get_first_zpdesc(zspage);
 
 	for (i = 0; i < nr_zpdesc; i++)
@@ -1348,12 +1419,13 @@ int zs_store(struct zs_pool *pool, struct zs_handle *handle, const void *src,
 	spin_lock(&class->lock);
 	newfg = get_fullness_group(class, zspage);
 	insert_zspage(class, zspage, newfg);
-	atomic_long_add(class->pages_per_zspage, &pool->pages_allocated);
+	atomic_long_add(class_pages_per_zspage(class), &pool->pages_allocated);
 	class_stat_add(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage);
 	class_stat_add(class, ZS_OBJS_INUSE, 1);
 
 	/* We completely set up zspage so mark them as movable */
-	SetZsPageMovable(pool, zspage);
+	if (class_size_to_zs_order(class->size) == 0)
+		SetZsPageMovable(pool, zspage);
 	spin_unlock(&class->lock);
 
 	return 0;
@@ -1365,13 +1437,15 @@ static void obj_free(int class_size, unsigned long obj)
 	struct link_free *link;
 	struct zspage *zspage;
 	struct zpdesc *f_zpdesc;
-	unsigned long f_offset;
+	unsigned long f_offset, page_size, page_mask;
 	unsigned int f_objidx;
 	void *vaddr;
 
 
 	obj_to_location(obj, &f_zpdesc, &f_objidx);
-	f_offset = offset_in_page(class_size * f_objidx);
+	page_size = class_size_to_zs_size(class_size);
+	page_mask = ~(page_size - 1);
+	f_offset = (class_size * f_objidx) & ~page_mask;
 	zspage = get_zspage(f_zpdesc);
 
 	vaddr = kmap_local_zpdesc(f_zpdesc);
@@ -1482,7 +1556,7 @@ void zs_free(struct zs_pool *pool, struct zs_handle *handle)
 
 	if (zspage_to_free) {
 		__free_zspage_lockless(zspage_to_free);
-		atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated);
+		atomic_long_sub(class_pages_per_zspage(class), &pool->pages_allocated);
 	}
 }
 EXPORT_SYMBOL_GPL(zs_free);
@@ -1540,6 +1614,8 @@ static void zs_object_copy(struct size_class *class, unsigned long dst,
 	struct zpdesc *s_zpdesc, *d_zpdesc;
 	unsigned int s_objidx, d_objidx;
 	unsigned long s_off, d_off;
+	unsigned long page_size = class_size_to_zs_size(class->size);
+	unsigned long page_mask = ~(page_size - 1);
 	void *s_addr, *d_addr;
 	int s_size, d_size, size;
 	int written = 0;
@@ -1549,14 +1625,14 @@ static void zs_object_copy(struct size_class *class, unsigned long dst,
 	obj_to_location(src, &s_zpdesc, &s_objidx);
 	obj_to_location(dst, &d_zpdesc, &d_objidx);
 
-	s_off = offset_in_page(class->size * s_objidx);
-	d_off = offset_in_page(class->size * d_objidx);
+	s_off = (class->size * s_objidx) & ~page_mask;
+	d_off = (class->size * d_objidx) & ~page_mask;
 
-	if (s_off + class->size > PAGE_SIZE)
-		s_size = PAGE_SIZE - s_off;
+	if (s_off + class->size > page_size)
+		s_size = page_size - s_off;
 
-	if (d_off + class->size > PAGE_SIZE)
-		d_size = PAGE_SIZE - d_off;
+	if (d_off + class->size > page_size)
+		d_size = page_size - d_off;
 
 	s_addr = kmap_local_zpdesc(s_zpdesc);
 	d_addr = kmap_local_zpdesc(d_zpdesc);
@@ -1581,7 +1657,7 @@ static void zs_object_copy(struct size_class *class, unsigned long dst,
 		 * kunmap_local(d_addr). For more details see
 		 * Documentation/mm/highmem.rst.
 		 */
-		if (s_off >= PAGE_SIZE) {
+		if (s_off >= page_size) {
 			kunmap_local(d_addr);
 			kunmap_local(s_addr);
 			s_zpdesc = get_next_zpdesc(s_zpdesc);
@@ -1591,7 +1667,7 @@ static void zs_object_copy(struct size_class *class, unsigned long dst,
 			s_off = 0;
 		}
 
-		if (d_off >= PAGE_SIZE) {
+		if (d_off >= page_size) {
 			kunmap_local(d_addr);
 			d_zpdesc = get_next_zpdesc(d_zpdesc);
 			d_addr = kmap_local_zpdesc(d_zpdesc);
@@ -1615,13 +1691,13 @@ static struct zs_handle *find_alloced_obj(struct size_class *class,
 	int index = *obj_idx;
 	struct zs_handle *handle = NULL;
 	void *addr = kmap_local_zpdesc(zpdesc);
+	unsigned long page_size = class_size_to_zs_size(class->size);
 
 	offset = get_first_obj_offset(zpdesc);
 	offset += class->size * index;
 
-	while (offset < PAGE_SIZE) {
-		handle = obj_allocated(zpdesc, addr + offset);
-		if (handle)
+	while (offset < page_size) {
+		if (obj_allocated(zpdesc, addr + offset, &handle))
 			break;
 
 		offset += class->size;
@@ -2002,7 +2078,7 @@ static unsigned long zs_can_compact(struct size_class *class)
 	obj_wasted = obj_allocated - obj_used;
 	obj_wasted /= class->objs_per_zspage;
 
-	return obj_wasted * class->pages_per_zspage;
+	return obj_wasted * class_pages_per_zspage(class);
 }
 
 static unsigned long __zs_compact(struct zs_pool *pool,
@@ -2041,7 +2117,7 @@ static unsigned long __zs_compact(struct zs_pool *pool,
 		fg = putback_zspage(class, src_zspage);
 		if (fg == ZS_INUSE_RATIO_0) {
 			free_zspage(pool, class, src_zspage);
-			pages_freed += class->pages_per_zspage;
+			pages_freed += class_pages_per_zspage(class);
 		}
 		src_zspage = NULL;
 
@@ -2164,6 +2240,7 @@ static int calculate_zspage_chain_size(int class_size)
 {
 	int i, min_waste = INT_MAX;
 	int chain_size = 1;
+	unsigned long page_size = class_size_to_zs_size(class_size);
 
 	if (is_power_of_2(class_size))
 		return chain_size;
@@ -2171,7 +2248,7 @@ static int calculate_zspage_chain_size(int class_size)
 	for (i = 1; i <= ZS_MAX_PAGES_PER_ZSPAGE; i++) {
 		int waste;
 
-		waste = (i * PAGE_SIZE) % class_size;
+		waste = (i * page_size) % class_size;
 		if (waste < min_waste) {
 			min_waste = waste;
 			chain_size = i;
@@ -2215,16 +2292,32 @@ struct zs_pool *zs_create_pool(const char *name)
 	 */
 	for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) {
 		int size;
-		int pages_per_zspage;
+		int zpdescs_per_zspage;
 		int objs_per_zspage;
 		struct size_class *class;
 		int fullness;
+		int idx = ZSMALLOC_TYPE_BASEPAGE;
+#ifdef CONFIG_ZSMALLOC_MULTI_PAGES
+		int order = 0;
+#endif
 
-		size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA;
+		if (i < ZS_PAGE_SIZE_CLASSES)
+			size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA;
+#ifdef CONFIG_ZSMALLOC_MULTI_PAGES
+		if (i >= ZS_PAGE_SIZE_CLASSES)
+			size = PAGE_SIZE + (i - ZS_PAGE_SIZE_CLASSES) *
+			       ZS_MULTI_PAGES_SIZE_CLASS_DELTA;
+#endif
 		if (size > ZS_MAX_ALLOC_SIZE)
 			size = ZS_MAX_ALLOC_SIZE;
-		pages_per_zspage = calculate_zspage_chain_size(size);
-		objs_per_zspage = pages_per_zspage * PAGE_SIZE / size;
+#ifdef CONFIG_ZSMALLOC_MULTI_PAGES
+		order = class_size_to_zs_order(size);
+		if (order == ZSMALLOC_MULTI_PAGES_ORDER)
+			idx = ZSMALLOC_TYPE_MULTI_PAGES;
+#endif
+		zpdescs_per_zspage = calculate_zspage_chain_size(size);
+		objs_per_zspage = zpdescs_per_zspage *
+				  class_size_to_zs_size(size) / size;
 
 		/*
 		 * We iterate from biggest down to smallest classes,
@@ -2232,9 +2325,9 @@ struct zs_pool *zs_create_pool(const char *name)
 		 * class. Any object bigger than or equal to that will
 		 * endup in the huge class.
 		 */
-		if (pages_per_zspage != 1 && objs_per_zspage != 1 &&
-				!huge_class_size) {
-			huge_class_size = size;
+		if (zpdescs_per_zspage != 1 && objs_per_zspage != 1 &&
+				!huge_class_size[idx]) {
+			huge_class_size[idx] = size;
 			/*
 			 * The object uses ZS_HANDLE_SIZE bytes to store the
 			 * handle. We need to subtract it, because zs_store()
@@ -2244,7 +2337,7 @@ struct zs_pool *zs_create_pool(const char *name)
 			 * class because it grows by ZS_HANDLE_SIZE extra bytes
 			 * right before class lookup.
 			 */
-			huge_class_size -= (ZS_HANDLE_SIZE - 1);
+			huge_class_size[idx] -= (ZS_HANDLE_SIZE - 1);
 		}
 
 		/*
@@ -2257,7 +2350,7 @@ struct zs_pool *zs_create_pool(const char *name)
 		 * previous size_class if possible.
 		 */
 		if (prev_class) {
-			if (can_merge(prev_class, pages_per_zspage, objs_per_zspage)) {
+			if (can_merge(prev_class, zpdescs_per_zspage, objs_per_zspage)) {
 				pool->size_class[i] = prev_class;
 				continue;
 			}
@@ -2269,7 +2362,7 @@ struct zs_pool *zs_create_pool(const char *name)
 
 		class->size = size;
 		class->index = i;
-		class->pages_per_zspage = pages_per_zspage;
+		class->zpdescs_per_zspage = zpdescs_per_zspage;
 		class->objs_per_zspage = objs_per_zspage;
 		spin_lock_init(&class->lock);
 		pool->size_class[i] = class;
-- 
2.43.0



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages
  2026-11-15  2:44 [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages Nanzhe Zhao
  2026-11-15  2:44 ` [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages Nanzhe Zhao
@ 2026-11-15  2:44 ` Nanzhe Zhao
  2026-11-16  2:46   ` sashiko-bot
  2026-11-15  2:44 ` [RFC PATCH v4 3/4] zram: backend_zstd: use multi-page size for zstd parameter setup Nanzhe Zhao
  2026-11-15  2:44 ` [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin Nanzhe Zhao
  3 siblings, 1 reply; 8+ messages in thread
From: Nanzhe Zhao @ 2026-11-15  2:44 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Tangquan Zheng, linux-mm, linux-block, linux-kernel, Jens Axboe,
	Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner,
	Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li,
	David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts,
	Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee,
	Nick Terrell, Barry Song, Bo Zhang, Nanzhe Zhao

From: Tangquan Zheng <zhengtangquan@oppo.com>

Currently, when a large folio with nr_pages is submitted to zram, it is
divided into nr_pages parts for compression and storage individually.
By transitioning to a higher granularity, we can notably enhance
compression rates while simultaneously reducing CPU consumption.

This patch introduces the capability for large folios to be divided
based on the granularity specified by ZSMALLOC_MULTI_PAGES_ORDER, which
defaults to 2. For instance, for folios sized at 128KiB, compression
will occur in eight 16KiB multi-pages.

This modification will notably reduce CPU consumption and enhance
compression ratios. The following data illustrates the time and
compressed data for typical anonymous pages gathered from Android
phones.

Signed-off-by: Tangquan Zheng <zhengtangquan@oppo.com>
Co-developed-by: Barry Song <baohua@kernel.org>
Signed-off-by: Barry Song <baohua@kernel.org>
Co-developed-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com>
Signed-off-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com>
---
 drivers/block/zram/Kconfig    |   9 +
 drivers/block/zram/zcomp.c    |  31 ++-
 drivers/block/zram/zcomp.h    |  19 +-
 drivers/block/zram/zram_drv.c | 500 ++++++++++++++++++++++++++++++----
 drivers/block/zram/zram_drv.h |  43 +++
 mm/zsmalloc.c                 |   3 +-
 6 files changed, 549 insertions(+), 56 deletions(-)

diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig
index 402b7b175863..716e92c5fdfe 100644
--- a/drivers/block/zram/Kconfig
+++ b/drivers/block/zram/Kconfig
@@ -145,3 +145,12 @@ config ZRAM_MULTI_COMP
 	  re-compress pages using a potentially slower but more effective
 	  compression algorithm. Note, that IDLE page recompression
 	  requires ZRAM_TRACK_ENTRY_ACTIME.
+
+config ZRAM_MULTI_PAGES
+	bool "Enable multiple pages compression and decompression"
+	depends on ZRAM && ZSMALLOC_MULTI_PAGES
+	help
+	  Initially, zram divided large folios into blocks of nr_pages, each sized
+	  equal to PAGE_SIZE, for compression. This option fine-tunes zram to
+	  improve compression granularity by dividing large folios into larger
+	  parts defined by the configuration option ZSMALLOC_MULTI_PAGES_ORDER.
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index ad8c7dc73922..8fa2a44adaa3 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -71,7 +71,7 @@ static int zcomp_cstrm_init(struct zcomp *comp, struct zcomp_cstrm *zstrm)
 	 * allocate 2 pages. 1 for compressed data, plus 1 extra for the
 	 * case when compressed size is larger than the original one
 	 */
-	zstrm->buffer = vzalloc(2 * PAGE_SIZE);
+	zstrm->buffer = vzalloc(2 * ZCOMP_MULTI_PAGES_SIZE);
 	if (!zstrm->buffer) {
 		zcomp_cstrm_free(comp, zstrm);
 		return -ENOMEM;
@@ -83,6 +83,10 @@ static void zcomp_dstrm_free(struct zcomp *comp, struct zcomp_dstrm *zstrm)
 {
 	if (comp->ops->destroy_dctx)
 		comp->ops->destroy_dctx(&zstrm->ctx);
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	vfree(zstrm->decomp_buf);
+	zstrm->decomp_buf = NULL;
+#endif
 	vfree(zstrm->local_copy);
 	zstrm->local_copy = NULL;
 	zstrm->ctx.context = NULL;
@@ -101,11 +105,23 @@ static int zcomp_dstrm_init(struct zcomp *comp, struct zcomp_dstrm *zstrm)
 		}
 	}
 
-	zstrm->local_copy = vzalloc(PAGE_SIZE);
+	zstrm->local_copy = vzalloc(ZCOMP_MULTI_PAGES_SIZE);
 	if (!zstrm->local_copy) {
 		zcomp_dstrm_free(comp, zstrm);
 		return -ENOMEM;
 	}
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	/*
+	 * Preallocate a decompression buffer for partial multi-page reads to
+	 * avoid memory allocation in the swapin path.
+	 */
+	zstrm->decomp_buf = vzalloc(ZCOMP_MULTI_PAGES_SIZE);
+	if (!zstrm->decomp_buf) {
+		zcomp_dstrm_free(comp, zstrm);
+		return -ENOMEM;
+	}
+	mutex_init(&zstrm->buf_lock);
+#endif
 	return 0;
 }
 
@@ -255,13 +271,13 @@ void zcomp_dstrm_put(struct zcomp *comp, struct zcomp_dstrm *zstrm,
 }
 
 int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm,
-		   const void *src, unsigned int *dst_len)
+		   const void *src, unsigned int src_len, unsigned int *dst_len)
 {
 	struct zcomp_req req = {
 		.src = src,
 		.dst = zstrm->buffer,
-		.src_len = PAGE_SIZE,
-		.dst_len = 2 * PAGE_SIZE,
+		.src_len = src_len,
+		.dst_len = 2 * src_len,
 	};
 	int ret;
 
@@ -282,12 +298,13 @@ static int zcomp_decompress_buf(struct zcomp *comp, struct zcomp_dstrm *zstrm,
 }
 
 int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm,
-		     struct scatterlist *sg, unsigned int src_len, void *dst)
+		     struct scatterlist *sg, unsigned int src_len, void *dst,
+		     unsigned int dst_len)
 {
 	struct zcomp_req req = {
 		.dst = dst,
 		.src_len = src_len,
-		.dst_len = PAGE_SIZE,
+		.dst_len = dst_len,
 	};
 	void *src;
 	int ret;
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index 39efefc43a13..fee105f876b4 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -47,6 +47,15 @@ struct zcomp_ctx {
 	void *context;
 };
 
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+#define ZCOMP_MULTI_PAGES_ORDER	(_AC(CONFIG_ZSMALLOC_MULTI_PAGES_ORDER, UL))
+#define ZCOMP_MULTI_PAGES_NR	(1 << ZCOMP_MULTI_PAGES_ORDER)
+#define ZCOMP_MULTI_PAGES_SIZE	(PAGE_SIZE * ZCOMP_MULTI_PAGES_NR)
+#define MULTI_PAGE_SHIFT (ZCOMP_MULTI_PAGES_ORDER + PAGE_SHIFT)
+#else
+#define ZCOMP_MULTI_PAGES_SIZE	PAGE_SIZE
+#endif
+
 struct zcomp_cstrm {
 	struct mutex lock;
 	/* compression buffer */
@@ -65,6 +74,11 @@ struct zcomp_dstrm {
 	struct mutex excl_lock;
 	/* local copy of handle memory */
 	void *local_copy;
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	/* Temporary buffer and lock for partial multi-page decompression */
+	void *decomp_buf;
+	struct mutex buf_lock;
+#endif
 	struct zcomp_ctx ctx;
 };
 
@@ -124,8 +138,9 @@ void zcomp_dstrm_put(struct zcomp *comp, struct zcomp_dstrm *zstrm,
 		     enum zcomp_dstrm_mode mode);
 
 int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm,
-		   const void *src, unsigned int *dst_len);
+		   const void *src, unsigned int src_len, unsigned int *dst_len);
 int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm,
-		     struct scatterlist *sg, unsigned int src_len, void *dst);
+		     struct scatterlist *sg, unsigned int src_len, void *dst,
+		     unsigned int dst_len);
 
 #endif /* _ZCOMP_H_ */
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 24eefcf143de..0d33220f4b22 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -53,11 +53,12 @@ static unsigned int num_devices = 1;
  * Pages that compress to sizes equals or greater than this are stored
  * uncompressed in memory.
  */
-static size_t huge_class_size;
+static size_t huge_class_size[ZSMALLOC_TYPE_MAX];
 
 static const struct block_device_operations zram_devops;
 
 static void slot_free(struct zram *zram, unsigned long index);
+static void slot_free_npages(struct zram *zram, unsigned long index, int nr_pages);
 
 /*
  * entry locking rules:
@@ -321,11 +322,12 @@ static inline void zram_fill_page(void *ptr, unsigned long len,
 	memset_l(ptr, value, len / sizeof(unsigned long));
 }
 
-static bool page_same_filled(void *ptr, unsigned long *element)
+static bool page_same_filled(void *ptr, unsigned long *element,
+			     unsigned int page_size)
 {
 	unsigned long *page;
 	unsigned long val;
-	unsigned int pos, last_pos = PAGE_SIZE / sizeof(*page) - 1;
+	unsigned int pos, last_pos = page_size / sizeof(*page) - 1;
 
 	page = (unsigned long *)ptr;
 	val = page[0];
@@ -484,7 +486,7 @@ static ssize_t idle_store(struct device *dev, struct device_attribute *attr,
 static int read_from_zspool_raw(struct zram *zram, struct page *page,
 				unsigned long index);
 static int read_from_zspool(struct zram *zram, struct page *page,
-			    unsigned long index);
+			    unsigned long index, unsigned long page_size);
 
 struct zram_wb_ctl {
 	/* idle list is accessed only by the writeback task, no concurency */
@@ -1092,7 +1094,7 @@ static int zram_writeback_slots(struct zram *zram,
 		if (zram->compressed_wb)
 			err = read_from_zspool_raw(zram, req->page, index);
 		else
-			err = read_from_zspool(zram, req->page, index);
+			err = read_from_zspool(zram, req->page, index, PAGE_SIZE);
 		if (err)
 			goto next;
 		slot_unlock(zram, index);
@@ -1379,7 +1381,7 @@ static int decompress_bdev_page(struct zram *zram, struct page *page,
 
 	zstrm = zcomp_dstrm_get(zram->comps[prio], ZCOMP_DSTRM_EXCLUSIVE);
 	ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size,
-			       zstrm->local_copy);
+			       zstrm->local_copy, PAGE_SIZE);
 	if (!ret) {
 		void *dst = kmap_local_page(page);
 
@@ -1989,6 +1991,31 @@ static ssize_t debug_stat_show(struct device *dev,
 	return ret;
 }
 
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+static ssize_t multi_pages_debug_stat_show(struct device *dev,
+		struct device_attribute *attr, char *buf)
+{
+	struct zram *zram = dev_to_zram(dev);
+	ssize_t ret;
+
+	guard(rwsem_read)(&zram->dev_lock);
+	ret = sysfs_emit(buf,
+			"zram_bio write/read multi_pages count:%8llu %8llu\n"
+			"zram_bio failed write/read multi_pages count%8llu %8llu\n"
+			"zram_bio partial write/read multi_pages count%8llu %8llu\n"
+			"multi_pages_miss_free %8llu\n",
+			(u64)atomic64_read(&zram->stats.zram_bio_write_multi_pages_count),
+			(u64)atomic64_read(&zram->stats.zram_bio_read_multi_pages_count),
+			(u64)atomic64_read(&zram->stats.multi_pages_failed_writes),
+			(u64)atomic64_read(&zram->stats.multi_pages_failed_reads),
+			(u64)atomic64_read(&zram->stats.zram_bio_write_multi_pages_partial_count),
+			(u64)atomic64_read(&zram->stats.zram_bio_read_multi_pages_partial_count),
+			(u64)atomic64_read(&zram->stats.multi_pages_miss_free));
+
+	return ret;
+}
+#endif
+
 static void zram_meta_free(struct zram *zram, u64 disksize)
 {
 	unsigned long num_pages = disksize >> PAGE_SHIFT;
@@ -2010,6 +2037,7 @@ static void zram_meta_free(struct zram *zram, u64 disksize)
 static bool zram_meta_alloc(struct zram *zram, u64 disksize)
 {
 	unsigned long num_pages;
+	int i;
 
 	num_pages = disksize >> PAGE_SHIFT;
 	zram->table = vzalloc(array_size(num_pages, sizeof(*zram->table)));
@@ -2023,9 +2051,10 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize)
 		return false;
 	}
 
-	if (!huge_class_size)
-		huge_class_size = zs_huge_class_size(zram->mem_pool,
-						ZSMALLOC_TYPE_BASEPAGE);
+	for (i = 0; i < ZSMALLOC_TYPE_MAX; i++) {
+		if (!huge_class_size[i])
+			huge_class_size[i] = zs_huge_class_size(zram->mem_pool, i);
+	}
 
 	lockdep_register_key(&zram->table_lock_key);
 	lockdep_init_map(&zram->table_lock_map, "zram->table[index].lock", &zram->table_lock_key, 0);
@@ -2033,10 +2062,18 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize)
 	return true;
 }
 
-static void slot_free(struct zram *zram, unsigned long index)
+static void slot_free_npages(struct zram *zram, unsigned long index,
+			     int nr_pages)
 {
 	struct zs_handle *handle;
 
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	if (test_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES)) {
+		clear_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES);
+		nr_pages = max(nr_pages, ZCOMP_MULTI_PAGES_NR);
+	}
+#endif
+
 #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME
 	zram->table[index].attr.ac_time = 0;
 #endif
@@ -2068,7 +2105,7 @@ static void slot_free(struct zram *zram, unsigned long index)
 	 */
 	if (test_slot_flag(zram, index, ZRAM_SAME)) {
 		clear_slot_flag(zram, index, ZRAM_SAME);
-		atomic64_dec(&zram->stats.same_pages);
+		atomic64_sub(nr_pages, &zram->stats.same_pages);
 		goto out;
 	}
 
@@ -2081,44 +2118,49 @@ static void slot_free(struct zram *zram, unsigned long index)
 	atomic64_sub(get_slot_size(zram, index),
 		     &zram->stats.compr_data_size);
 out:
-	atomic64_dec(&zram->stats.pages_stored);
+	atomic64_sub(nr_pages, &zram->stats.pages_stored);
 	zram->table[index].element = 0;
 	set_slot_size(zram, index, 0);
 }
 
+static void slot_free(struct zram *zram, unsigned long index)
+{
+	slot_free_npages(zram, index, 1);
+}
+
 static int read_same_filled_page(struct zram *zram, struct page *page,
-				 unsigned long index)
+				 unsigned long index, unsigned long page_size)
 {
 	void *mem;
 
 	mem = kmap_local_page(page);
-	zram_fill_page(mem, PAGE_SIZE, zram->table[index].element);
+	zram_fill_page(mem, page_size, zram->table[index].element);
 	kunmap_local(mem);
 	return 0;
 }
 
 static int read_incompressible_page(struct zram *zram, struct page *page,
-				    unsigned long index)
+				    unsigned long index,
+				    unsigned long page_size)
 {
 	struct scatterlist sg[2];
 	struct zs_handle *handle;
-	void *src, *dst;
+	void *dst;
 
 	handle = &zram->table[index].handle;
-	zs_obj_read_sg_begin(zram->mem_pool, handle, sg, PAGE_SIZE);
+	zs_obj_read_sg_begin(zram->mem_pool, handle, sg, page_size);
 	/* an incompressible object never spans two pages */
-	src = kmap_local_page(sg_page(sg));
 	dst = kmap_local_page(page);
-	copy_page(dst, src);
+	sg_copy_to_buffer(sg, sg_nents(sg), dst, page_size);
 	kunmap_local(dst);
-	kunmap_local(src);
 	zs_obj_read_sg_end(zram->mem_pool, handle);
 
 	return 0;
 }
 
 static int read_compressed_page(struct zram *zram, struct page *page,
-				unsigned long index)
+				unsigned long index,
+				unsigned long page_size)
 {
 	enum zcomp_dstrm_mode mode = ZCOMP_DSTRM_EXCLUSIVE;
 	struct zcomp_dstrm *zstrm;
@@ -2138,7 +2180,8 @@ static int read_compressed_page(struct zram *zram, struct page *page,
 
 	zstrm = zcomp_dstrm_get(zram->comps[prio], mode);
 	dst = kmap_local_page(page);
-	ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size, dst);
+	ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size, dst,
+			       page_size);
 	kunmap_local(dst);
 	zcomp_dstrm_put(zram->comps[prio], zstrm, mode);
 	zs_obj_read_sg_end(zram->mem_pool, handle);
@@ -2179,20 +2222,21 @@ static int read_from_zspool_raw(struct zram *zram, struct page *page,
  * Corresponding ZRAM slot should be locked.
  */
 static int read_from_zspool(struct zram *zram, struct page *page,
-			    unsigned long index)
+			    unsigned long index, unsigned long page_size)
 {
 	if (test_slot_flag(zram, index, ZRAM_SAME) ||
 	    !zs_handle_is_allocated(&zram->table[index].handle))
-		return read_same_filled_page(zram, page, index);
+		return read_same_filled_page(zram, page, index, page_size);
 
 	if (!test_slot_flag(zram, index, ZRAM_HUGE))
-		return read_compressed_page(zram, page, index);
+		return read_compressed_page(zram, page, index, page_size);
 	else
-		return read_incompressible_page(zram, page, index);
+		return read_incompressible_page(zram, page, index, page_size);
 }
 
-static int zram_read_page(struct zram *zram, struct page *page,
-			  unsigned long index, struct bio *parent)
+static int zram_read_page_size(struct zram *zram, struct page *page,
+			       unsigned long index, struct bio *parent,
+			       unsigned long page_size)
 {
 	int ret;
 
@@ -2200,7 +2244,7 @@ static int zram_read_page(struct zram *zram, struct page *page,
 	mark_slot_accessed(zram, index);
 	if (!test_slot_flag(zram, index, ZRAM_WB)) {
 		/* Slot should be locked through out the function call */
-		ret = read_from_zspool(zram, page, index);
+		ret = read_from_zspool(zram, page, index, page_size);
 		slot_unlock(zram, index);
 	} else {
 		unsigned long blk_idx = zram->table[index].blk_idx;
@@ -2216,6 +2260,12 @@ static int zram_read_page(struct zram *zram, struct page *page,
 	return ret;
 }
 
+static int zram_read_page(struct zram *zram, struct page *page,
+			  unsigned long index, struct bio *parent)
+{
+	return zram_read_page_size(zram, page, index, parent, PAGE_SIZE);
+}
+
 /*
  * Use a temporary buffer to decompress the page, as the decompressor
  * always expects a full page for the output.
@@ -2244,23 +2294,32 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec,
 }
 
 static int write_same_filled_page(struct zram *zram, unsigned long fill,
-				  unsigned long index)
+				  unsigned long index, int nr_pages)
 {
 	slot_lock(zram, index);
 	slot_free(zram, index);
 	set_slot_flag(zram, index, ZRAM_SAME);
 	zram->table[index].element = fill;
 	mark_slot_accessed(zram, index);
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	if (nr_pages > 1) {
+		int i;
+
+		for (i = 0; i < nr_pages; i++)
+			set_slot_flag(zram, index + i, ZRAM_COMP_MULTI_PAGES);
+	}
+#endif
 	slot_unlock(zram, index);
 
-	atomic64_inc(&zram->stats.same_pages);
-	atomic64_inc(&zram->stats.pages_stored);
+	atomic64_add(nr_pages, &zram->stats.same_pages);
+	atomic64_add(nr_pages, &zram->stats.pages_stored);
 
 	return 0;
 }
 
 static int write_incompressible_page(struct zram *zram, struct page *page,
-				     unsigned long index)
+				     unsigned long index, unsigned int page_size,
+				     int nr_pages)
 {
 	void *src;
 	int ret;
@@ -2269,7 +2328,7 @@ static int write_incompressible_page(struct zram *zram, struct page *page,
 	slot_free(zram, index);
 	src = kmap_local_page(page);
 	ret = zs_store(zram->mem_pool, &zram->table[index].handle, src,
-		       PAGE_SIZE, GFP_NOIO | __GFP_NOWARN |
+		       page_size, GFP_NOIO | __GFP_NOWARN |
 		       __GFP_HIGHMEM | __GFP_MOVABLE, page_to_nid(page));
 	kunmap_local(src);
 	if (ret) {
@@ -2280,14 +2339,22 @@ static int write_incompressible_page(struct zram *zram, struct page *page,
 	update_used_max(zram);
 
 	set_slot_flag(zram, index, ZRAM_HUGE);
-	set_slot_size(zram, index, PAGE_SIZE);
+	set_slot_size(zram, index, page_size);
 	mark_slot_accessed(zram, index);
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	if (nr_pages > 1) {
+		int i;
+
+		for (i = 0; i < nr_pages; i++)
+			set_slot_flag(zram, index + i, ZRAM_COMP_MULTI_PAGES);
+	}
+#endif
 	slot_unlock(zram, index);
 
-	atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size);
+	atomic64_add(page_size, &zram->stats.compr_data_size);
 	atomic64_inc(&zram->stats.huge_pages);
 	atomic64_inc(&zram->stats.huge_pages_since);
-	atomic64_inc(&zram->stats.pages_stored);
+	atomic64_add(nr_pages, &zram->stats.pages_stored);
 
 	return 0;
 }
@@ -2301,12 +2368,27 @@ static int zram_write_page(struct zram *zram, struct page *page,
 	struct zcomp_cstrm *zstrm;
 	unsigned long element;
 	bool same_filled;
+	unsigned long page_size = PAGE_SIZE;
+	int huge_class_idx = ZSMALLOC_TYPE_BASEPAGE;
+	int nr_pages = 1;
+
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	/*
+	 * Multi-page compression packs ZCOMP_MULTI_PAGES_NR pages into one
+	 * compressed object.
+	 */
+	if (folio_size(page_folio(page)) >= ZCOMP_MULTI_PAGES_SIZE) {
+		page_size = ZCOMP_MULTI_PAGES_SIZE;
+		huge_class_idx = ZSMALLOC_TYPE_MULTI_PAGES;
+		nr_pages = ZCOMP_MULTI_PAGES_NR;
+	}
+#endif
 
 	mem = kmap_local_page(page);
-	same_filled = page_same_filled(mem, &element);
+	same_filled = page_same_filled(mem, &element, page_size);
 	kunmap_local(mem);
 	if (same_filled)
-		return write_same_filled_page(zram, element, index);
+		return write_same_filled_page(zram, element, index, nr_pages);
 
 	if (!zram_can_store_page(zram))
 		return -ENOMEM;
@@ -2314,7 +2396,7 @@ static int zram_write_page(struct zram *zram, struct page *page,
 	zstrm = zcomp_cstrm_get(zram->comps[ZRAM_PRIMARY_COMP]);
 	mem = kmap_local_page(page);
 	ret = zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm,
-			     mem, &comp_len);
+			     mem, page_size, &comp_len);
 	kunmap_local(mem);
 
 	if (unlikely(ret)) {
@@ -2323,9 +2405,10 @@ static int zram_write_page(struct zram *zram, struct page *page,
 		return ret;
 	}
 
-	if (comp_len >= huge_class_size) {
+	if (comp_len >= huge_class_size[huge_class_idx]) {
 		zcomp_cstrm_put(zstrm);
-		return write_incompressible_page(zram, page, index);
+		return write_incompressible_page(zram, page, index, page_size,
+						 nr_pages);
 	}
 
 	slot_lock(zram, index);
@@ -2344,10 +2427,18 @@ static int zram_write_page(struct zram *zram, struct page *page,
 
 	set_slot_size(zram, index, comp_len);
 	mark_slot_accessed(zram, index);
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	if (nr_pages > 1) {
+		int i;
+
+		for (i = 0; i < nr_pages; i++)
+			set_slot_flag(zram, index + i, ZRAM_COMP_MULTI_PAGES);
+	}
+#endif
 	slot_unlock(zram, index);
 
 	/* Update stats */
-	atomic64_inc(&zram->stats.pages_stored);
+	atomic64_add(nr_pages, &zram->stats.pages_stored);
 	atomic64_add(comp_len, &zram->stats.compr_data_size);
 
 	return ret;
@@ -2467,7 +2558,7 @@ static int recompress_slot(struct zram *zram, unsigned long index,
 	if (comp_len_old < threshold)
 		return 0;
 
-	ret = read_from_zspool(zram, page, index);
+	ret = read_from_zspool(zram, page, index, PAGE_SIZE);
 	if (ret)
 		return ret;
 
@@ -2480,7 +2571,8 @@ static int recompress_slot(struct zram *zram, unsigned long index,
 
 	zstrm = zcomp_cstrm_get(zram->comps[prio]);
 	src = kmap_local_page(page);
-	ret = zcomp_compress(zram->comps[prio], zstrm, src, &comp_len_new);
+	ret = zcomp_compress(zram->comps[prio], zstrm, src, PAGE_SIZE,
+			      &comp_len_new);
 	kunmap_local(src);
 
 	/*
@@ -2615,7 +2707,7 @@ static ssize_t recompress_store(struct device *dev,
 		}
 	}
 
-	if (threshold >= huge_class_size)
+	if (threshold >= huge_class_size[ZSMALLOC_TYPE_BASEPAGE])
 		return -EINVAL;
 
 	guard(rwsem_write)(&zram->dev_lock);
@@ -2727,8 +2819,283 @@ static void zram_bio_discard(struct zram *zram, struct bio *bio)
 	bio_endio(bio);
 }
 
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+
+static inline unsigned long zram_multi_pages_head_index(unsigned long index)
+{
+	return index & ~((unsigned long)ZCOMP_MULTI_PAGES_NR - 1);
+}
+
+static inline int __test_multi_pages_comp(struct zram *zram,
+					 unsigned long index)
+{
+	unsigned long head = zram_multi_pages_head_index(index);
+	int i, count = 0;
+
+	for (i = 0; i < ZCOMP_MULTI_PAGES_NR; i++) {
+		if (test_slot_flag(zram, head + i, ZRAM_COMP_MULTI_PAGES))
+			count++;
+	}
+
+	return count;
+}
+
+static inline bool want_multi_pages_comp(struct zram *zram, struct bio *bio)
+{
+	unsigned long index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT;
+
+	if (bio->bi_io_vec->bv_len >= ZCOMP_MULTI_PAGES_SIZE)
+		return true;
+
+	slot_lock(zram, index);
+	if (__test_multi_pages_comp(zram, index)) {
+		slot_unlock(zram, index);
+		return true;
+	}
+	slot_unlock(zram, index);
+
+	return false;
+}
+
+static inline bool test_multi_pages_comp(struct zram *zram, struct bio *bio)
+{
+	unsigned long index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT;
+
+	return !!__test_multi_pages_comp(zram, index);
+}
+
+static inline bool is_multi_pages_partial_io(struct bio_vec *bvec)
+{
+	return bvec->bv_len != ZCOMP_MULTI_PAGES_SIZE;
+}
+
+static int zram_read_partial_from_zspool(struct zram *zram, struct page *page,
+					unsigned long index, int offset)
+{
+	struct zs_handle *handle = &zram->table[index].handle;
+	struct zcomp_dstrm *zstrm;
+	struct scatterlist sg[2];
+	enum zcomp_dstrm_mode mode;
+	unsigned int size;
+	void *src, *dst;
+	u32 prio;
+	int ret;
+	unsigned long page_size = ZCOMP_MULTI_PAGES_SIZE;
+
+	if (test_slot_flag(zram, index, ZRAM_SAME) ||
+	    !zs_handle_is_allocated(handle)) {
+		void *mem;
+
+		mem = kmap_local_page(page);
+		zram_fill_page(mem, PAGE_SIZE, zram->table[index].element);
+		kunmap_local(mem);
+		return 0;
+	}
+
+	size = get_slot_size(zram, index);
+	zs_obj_read_sg_begin(zram->mem_pool, handle, sg, size);
+
+	if (size == page_size) {
+		src = kmap_local_page(sg_page(sg));
+		dst = kmap_local_page(page);
+		memcpy(dst, src + offset, PAGE_SIZE);
+		kunmap_local(dst);
+		kunmap_local(src);
+		ret = 0;
+	} else {
+		prio = get_slot_comp_priority(zram, index);
+		mode = ZCOMP_DSTRM_EXCLUSIVE;
+		if (zcomp_shared_decomp(zram->comps[prio]))
+			mode = ZCOMP_DSTRM_SHARED;
+
+		zstrm = zcomp_dstrm_get(zram->comps[prio], mode);
+		mutex_lock(&zstrm->buf_lock);
+		ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size,
+				       zstrm->decomp_buf, page_size);
+		if (!ret) {
+			dst = kmap_local_page(page);
+			memcpy(dst, zstrm->decomp_buf + offset, PAGE_SIZE);
+			kunmap_local(dst);
+		}
+		mutex_unlock(&zstrm->buf_lock);
+		zcomp_dstrm_put(zram->comps[prio], zstrm, mode);
+	}
+	zs_obj_read_sg_end(zram->mem_pool, handle);
+
+	return ret;
+}
+
+static int zram_bvec_read_multi_pages_partial(struct zram *zram,
+					      struct bio_vec *bvec,
+					      unsigned long index, int offset,
+					      struct bio *bio)
+{
+	int ret;
+
+	slot_lock(zram, index);
+	if (!test_slot_flag(zram, index, ZRAM_WB)) {
+		ret = zram_read_partial_from_zspool(zram, bvec->bv_page,
+						    index, offset);
+		slot_unlock(zram, index);
+	} else {
+		unsigned long blk_idx = zram->table[index].blk_idx;
+
+		slot_unlock(zram, index);
+		ret = read_from_bdev(zram, bvec->bv_page, index, blk_idx, bio);
+	}
+
+	if (WARN_ON(ret < 0))
+		pr_err("Decompression failed! err=%d, page=%lu offset=%d\n",
+		       ret, index, offset);
+	if (!ret)
+		atomic64_inc(&zram->stats.zram_bio_read_multi_pages_partial_count);
+
+	return ret;
+}
+
+static int zram_bvec_read_multi_pages(struct zram *zram, struct bio_vec *bvec,
+				      unsigned long index, int offset,
+				      struct bio *bio)
+{
+	if (is_multi_pages_partial_io(bvec))
+		return zram_bvec_read_multi_pages_partial(zram, bvec, index,
+							  offset, bio);
+
+	return zram_read_page_size(zram, bvec->bv_page, index, bio,
+				   ZCOMP_MULTI_PAGES_SIZE);
+}
+
+static int zram_bvec_write_multi_pages_partial(struct zram *zram,
+					       struct bio_vec *bvec,
+					       unsigned long index, int offset,
+					       struct bio *bio)
+{
+	struct page *page;
+	void *src, *dst;
+	int ret;
+
+	page = alloc_pages(GFP_NOIO | __GFP_COMP, ZCOMP_MULTI_PAGES_ORDER);
+	if (!page)
+		return -ENOMEM;
+
+	ret = zram_read_page_size(zram, page, index, bio,
+				  ZCOMP_MULTI_PAGES_SIZE);
+	if (!ret) {
+		src = kmap_local_page(bvec->bv_page);
+		dst = kmap_local_page(page);
+		memcpy(dst + offset, src + bvec->bv_offset, bvec->bv_len);
+		kunmap_local(dst);
+		kunmap_local(src);
+
+		atomic64_inc(&zram->stats.zram_bio_write_multi_pages_partial_count);
+		ret = zram_write_page(zram, page, index);
+	}
+
+	__free_pages(page, ZCOMP_MULTI_PAGES_ORDER);
+	return ret;
+}
+
+static int zram_bvec_write_multi_pages(struct zram *zram, struct bio_vec *bvec,
+				       unsigned long index, int offset,
+				       struct bio *bio)
+{
+	if (is_multi_pages_partial_io(bvec))
+		return zram_bvec_write_multi_pages_partial(zram, bvec, index,
+							   offset, bio);
+
+	return zram_write_page(zram, bvec->bv_page, index);
+}
+
+static void zram_bio_read_multi_pages(struct zram *zram, struct bio *bio)
+{
+	unsigned long start_time = bio_start_io_acct(bio);
+	struct bvec_iter iter = bio->bi_iter;
+
+	do {
+		unsigned long index = zram_multi_pages_head_index(
+			iter.bi_sector >> SECTORS_PER_PAGE_SHIFT);
+		u32 offset = (iter.bi_sector & (SECTORS_PER_MULTI_PAGE - 1)) <<
+			     SECTOR_SHIFT;
+		struct bio_vec bv = multi_pages_bio_iter_iovec(bio, iter);
+
+		atomic64_inc(&zram->stats.zram_bio_read_multi_pages_count);
+		bv.bv_len = min_t(u32, bv.bv_len,
+				  ZCOMP_MULTI_PAGES_SIZE - offset);
+
+		if (zram_bvec_read_multi_pages(zram, &bv, index, offset, bio) < 0) {
+			atomic64_inc(&zram->stats.multi_pages_failed_reads);
+			bio->bi_status = BLK_STS_IOERR;
+			break;
+		}
+		flush_dcache_page(bv.bv_page);
+
+		slot_lock(zram, index);
+		mark_slot_accessed(zram, index);
+		slot_unlock(zram, index);
+
+		bio_advance_iter_single(bio, &iter, bv.bv_len);
+	} while (iter.bi_size);
+
+	bio_end_io_acct(bio, start_time);
+	bio_endio(bio);
+}
+
+static void zram_bio_write_multi_pages(struct zram *zram, struct bio *bio)
+{
+	unsigned long start_time = bio_start_io_acct(bio);
+	struct bvec_iter iter = bio->bi_iter;
+
+	do {
+		unsigned long index = zram_multi_pages_head_index(
+			iter.bi_sector >> SECTORS_PER_PAGE_SHIFT);
+		u32 offset = (iter.bi_sector & (SECTORS_PER_MULTI_PAGE - 1)) <<
+			     SECTOR_SHIFT;
+		struct bio_vec bv = multi_pages_bio_iter_iovec(bio, iter);
+
+		bv.bv_len = min_t(u32, bv.bv_len,
+				  ZCOMP_MULTI_PAGES_SIZE - offset);
+
+		atomic64_inc(&zram->stats.zram_bio_write_multi_pages_count);
+		if (zram_bvec_write_multi_pages(zram, &bv, index, offset, bio) < 0) {
+			atomic64_inc(&zram->stats.multi_pages_failed_writes);
+			bio->bi_status = BLK_STS_IOERR;
+			break;
+		}
+
+		slot_lock(zram, index);
+		mark_slot_accessed(zram, index);
+		slot_unlock(zram, index);
+
+		bio_advance_iter_single(bio, &iter, bv.bv_len);
+	} while (iter.bi_size);
+
+	bio_end_io_acct(bio, start_time);
+	bio_endio(bio);
+}
+#else /* CONFIG_ZRAM_MULTI_PAGES */
+static inline bool test_multi_pages_comp(struct zram *zram, struct bio *bio)
+{
+	return false;
+}
+
+static inline bool want_multi_pages_comp(struct zram *zram, struct bio *bio)
+{
+	return false;
+}
+
+static inline void zram_bio_read_multi_pages(struct zram *zram,
+					     struct bio *bio) { }
+static inline void zram_bio_write_multi_pages(struct zram *zram,
+					      struct bio *bio) { }
+#endif
+
 static void zram_bio_read(struct zram *zram, struct bio *bio)
 {
+	if (test_multi_pages_comp(zram, bio)) {
+		zram_bio_read_multi_pages(zram, bio);
+		return;
+	}
+
 	unsigned long start_time = bio_start_io_acct(bio);
 	struct bvec_iter iter = bio->bi_iter;
 
@@ -2756,6 +3123,11 @@ static void zram_bio_read(struct zram *zram, struct bio *bio)
 
 static void zram_bio_write(struct zram *zram, struct bio *bio)
 {
+	if (want_multi_pages_comp(zram, bio)) {
+		zram_bio_write_multi_pages(zram, bio);
+		return;
+	}
+
 	unsigned long start_time = bio_start_io_acct(bio);
 	struct bvec_iter iter = bio->bi_iter;
 
@@ -2808,6 +3180,9 @@ static void zram_slot_free_notify(struct block_device *bdev,
 				unsigned long index)
 {
 	struct zram *zram;
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	int nr_pages_to_free = 1;
+#endif
 
 	zram = bdev->bd_disk->private_data;
 
@@ -2817,7 +3192,34 @@ static void zram_slot_free_notify(struct block_device *bdev,
 		return;
 	}
 
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	{
+		int comp_count = __test_multi_pages_comp(zram, index);
+
+		if (comp_count > 1) {
+			clear_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES);
+			slot_unlock(zram, index);
+			return;
+		}
+
+		if (comp_count == 1) {
+			clear_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES);
+			slot_unlock(zram, index);
+			index = zram_multi_pages_head_index(index);
+			nr_pages_to_free = ZCOMP_MULTI_PAGES_NR;
+			if (!slot_trylock(zram, index)) {
+				atomic64_inc(&zram->stats.multi_pages_miss_free);
+				return;
+			}
+		}
+	}
+#endif
+
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	slot_free_npages(zram, index, nr_pages_to_free);
+#else
 	slot_free(zram, index);
+#endif
 	slot_unlock(zram, index);
 }
 
@@ -2991,6 +3393,9 @@ static const struct block_device_operations zram_devops = {
 static DEVICE_ATTR_RO(io_stat);
 static DEVICE_ATTR_RO(mm_stat);
 static DEVICE_ATTR_RO(debug_stat);
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+static DEVICE_ATTR_RO(multi_pages_debug_stat);
+#endif
 static DEVICE_ATTR_WO(compact);
 static DEVICE_ATTR_RW(disksize);
 static DEVICE_ATTR_RO(initstate);
@@ -3035,6 +3440,9 @@ static struct attribute *zram_disk_attrs[] = {
 	&dev_attr_io_stat.attr,
 	&dev_attr_mm_stat.attr,
 	&dev_attr_debug_stat.attr,
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	&dev_attr_multi_pages_debug_stat.attr,
+#endif
 #ifdef CONFIG_ZRAM_MULTI_COMP
 	&dev_attr_recomp_algorithm.attr,
 	&dev_attr_recompress.attr,
diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h
index d64c99b74214..1e3ebfec972e 100644
--- a/drivers/block/zram/zram_drv.h
+++ b/drivers/block/zram/zram_drv.h
@@ -28,6 +28,11 @@
 #define ZRAM_SECTOR_PER_LOGICAL_BLOCK	\
 	(1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT))
 
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+#define SECTORS_PER_MULTI_PAGE_SHIFT	(MULTI_PAGE_SHIFT - SECTOR_SHIFT)
+#define SECTORS_PER_MULTI_PAGE		(1 << SECTORS_PER_MULTI_PAGE_SHIFT)
+#endif
+
 /*
  * ZRAM is mainly used for memory efficiency so we want to keep memory
  * footprint small and thus squeeze size and zram pageflags into a flags
@@ -37,7 +42,12 @@
  *
  * We use BUILD_BUG_ON() to make sure that zram pageflags don't overflow.
  */
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+#define ZRAM_FLAG_SHIFT (PAGE_SHIFT + CONFIG_ZSMALLOC_MULTI_PAGES_ORDER + 1)
+#else
 #define ZRAM_FLAG_SHIFT (PAGE_SHIFT + 1)
+#endif
+
 
 /* Only 2 bits are allowed for comp priority index */
 #define ZRAM_COMP_PRIORITY_MASK	0x3
@@ -55,6 +65,10 @@ enum zram_pageflags {
 	ZRAM_COMP_PRIORITY_BIT1, /* First bit of comp priority index */
 	ZRAM_COMP_PRIORITY_BIT2, /* Second bit of comp priority index */
 
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	ZRAM_COMP_MULTI_PAGES,	/* Compressed by multi-pages */
+#endif
+
 	__NR_ZRAM_PAGEFLAGS,
 };
 
@@ -110,6 +124,15 @@ struct zram_stats {
 	atomic64_t bd_reads;		/* no. of reads from backing device */
 	atomic64_t bd_writes;		/* no. of writes from backing device */
 #endif
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+	atomic64_t zram_bio_write_multi_pages_count;
+	atomic64_t zram_bio_read_multi_pages_count;
+	atomic64_t multi_pages_failed_writes;
+	atomic64_t multi_pages_failed_reads;
+	atomic64_t zram_bio_write_multi_pages_partial_count;
+	atomic64_t zram_bio_read_multi_pages_partial_count;
+	atomic64_t multi_pages_miss_free;
+#endif
 };
 
 #ifdef CONFIG_ZRAM_MULTI_COMP
@@ -167,4 +190,24 @@ struct zram {
 	struct dentry *debugfs_dir;
 #endif
 };
+
+#ifdef CONFIG_ZRAM_MULTI_PAGES
+#define multi_pages_bvec_iter_offset(bvec, iter)				\
+	(mp_bvec_iter_offset((bvec), (iter)) % ZCOMP_MULTI_PAGES_SIZE)
+
+#define multi_pages_bvec_iter_len(bvec, iter)				\
+	min_t(unsigned int, mp_bvec_iter_len((bvec), (iter)),		\
+	      ZCOMP_MULTI_PAGES_SIZE -					\
+	      multi_pages_bvec_iter_offset((bvec), (iter)))
+
+#define multi_pages_bvec_iter_bvec(bvec, iter)				\
+((struct bio_vec) {							\
+	.bv_page	= mp_bvec_iter_page((bvec), (iter)),		\
+	.bv_len		= multi_pages_bvec_iter_len((bvec), (iter)),	\
+	.bv_offset	= multi_pages_bvec_iter_offset((bvec), (iter)),	\
+})
+
+#define multi_pages_bio_iter_iovec(bio, iter)				\
+	multi_pages_bvec_iter_bvec((bio)->bi_io_vec, (iter))
+#endif
 #endif
diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
index 2d83fcc8610d..81d908d6d5a8 100644
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ -1697,7 +1697,8 @@ static struct zs_handle *find_alloced_obj(struct size_class *class,
 	offset += class->size * index;
 
 	while (offset < page_size) {
-		if (obj_allocated(zpdesc, addr + offset, &handle))
+		handle = obj_allocated(zpdesc, addr + offset);
+		if (handle)
 			break;
 
 		offset += class->size;
-- 
2.43.0



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [RFC PATCH v4 3/4] zram: backend_zstd: use multi-page size for zstd parameter setup
  2026-11-15  2:44 [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages Nanzhe Zhao
  2026-11-15  2:44 ` [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages Nanzhe Zhao
  2026-11-15  2:44 ` [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages Nanzhe Zhao
@ 2026-11-15  2:44 ` Nanzhe Zhao
  2026-11-15  2:44 ` [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin Nanzhe Zhao
  3 siblings, 0 replies; 8+ messages in thread
From: Nanzhe Zhao @ 2026-11-15  2:44 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Barry Song, linux-mm, linux-block, linux-kernel, Jens Axboe,
	Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner,
	Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li,
	David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts,
	Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee,
	Nick Terrell, Tangquan Zheng, Bo Zhang, Nanzhe Zhao

From: Barry Song <baohua@kernel.org>

zstd_setup_params() passes PAGE_SIZE to zstd_get_params() and
zstd_get_cparams() as the source size hint. When multi-page compression
is enabled, compressing 16 KiB blocks with 4 KiB-tuned parameters limits
zstd's match window to a single page and prevents it from finding
cross-page redundancy.

Pass ZCOMP_MULTI_PAGES_SIZE instead of PAGE_SIZE so that zstd configures
its parameters for the multi-page block size.

Signed-off-by: Barry Song <baohua@kernel.org>
Co-developed-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com>
Signed-off-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com>
---
 drivers/block/zram/backend_zstd.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
index d55615f34300..9438691791ef 100644
--- a/drivers/block/zram/backend_zstd.c
+++ b/drivers/block/zram/backend_zstd.c
@@ -73,12 +73,12 @@ static int zstd_setup_params(struct zcomp_params *params)
 		goto error;
 	}
 
-	zp->cprm = zstd_get_params(params->level, PAGE_SIZE);
+	zp->cprm = zstd_get_params(params->level, ZCOMP_MULTI_PAGES_SIZE);
 
 	zp->custom_mem.customAlloc = zstd_custom_alloc;
 	zp->custom_mem.customFree = zstd_custom_free;
 
-	prm = zstd_get_cparams(params->level, PAGE_SIZE,
+	prm = zstd_get_cparams(params->level, ZCOMP_MULTI_PAGES_SIZE,
 			       params->dict_sz);
 
 	if (!params->dict || !params->dict_sz)
-- 
2.43.0



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin
  2026-11-15  2:44 [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages Nanzhe Zhao
                   ` (2 preceding siblings ...)
  2026-11-15  2:44 ` [RFC PATCH v4 3/4] zram: backend_zstd: use multi-page size for zstd parameter setup Nanzhe Zhao
@ 2026-11-15  2:44 ` Nanzhe Zhao
  2026-11-16  2:46   ` sashiko-bot
  3 siblings, 1 reply; 8+ messages in thread
From: Nanzhe Zhao @ 2026-11-15  2:44 UTC (permalink / raw)
  To: Andrew Morton
  Cc: Nanzhe Zhao, linux-mm, linux-block, linux-kernel, Jens Axboe,
	Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner,
	Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li,
	David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts,
	Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee,
	Nick Terrell, Barry Song, Tangquan Zheng, Bo Zhang

Synchronous swap devices such as zram complete swap I/O in the faulting
context. With multi-page compression enabled, a 16 KiB block is
stored and decompressed as a single object.

The swapin allocation inherits the VMA's THP gfp mask, which drops
__GFP_DIRECT_RECLAIM for VMAs without MADV_HUGEPAGE. Under memory
pressure, the order-2 allocation fails and falls back to order-0,
requiring the entire 16 KiB block to be decompressed just to swap in
an order-0 page.

Allow direct reclaim for order-2 swapin allocation to improve the
allocation success rate and avoid partial reads.

Signed-off-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com>
---
 mm/swap_state.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/mm/swap_state.c b/mm/swap_state.c
index da65d75ceead..912b0e21a781 100644
--- a/mm/swap_state.c
+++ b/mm/swap_state.c
@@ -462,6 +462,13 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
 	if (mpol || !vmf) {
 		folio = folio_alloc_mpol(gfp, order, mpol, ilx, numa_node_id());
 	} else {
+		/*
+		 * zsmalloc uses order-2 pages to store compressed data.
+		 * Allow direct reclaim to improve allocation success and
+		 * avoid partial reads.
+		 */
+		if (order == 2)
+			gfp |= __GFP_RECLAIM;
 		address = round_down(vmf->address, PAGE_SIZE << order);
 		folio = vma_alloc_folio(gfp, order, vmf->vma, address);
 	}
-- 
2.43.0



^ permalink raw reply related	[flat|nested] 8+ messages in thread

* Re: [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages
  2026-11-15  2:44 ` [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages Nanzhe Zhao
@ 2026-11-16  2:46   ` sashiko-bot
  0 siblings, 0 replies; 8+ messages in thread
From: sashiko-bot @ 2026-11-16  2:46 UTC (permalink / raw)
  To: Nanzhe Zhao, Andrew Morton
  Cc: Tangquan Zheng, linux-mm, Jens Axboe, Sergey Senozhatsky,
	Minchan Kim, Kairui Song, Johannes Weiner, Nhat Pham, Usama Arif,
	Yosry Ahmed, Chengming Zhou, Chris Li, David Hildenbrand,
	Ying Huang, Suren Baghdasaryan, Ryan Roberts, Kanchana P Sridhar,
	Wajdi K Feghali, Bala Seshasayee, Nick Terrell, Barry Song,
	Bo Zhang

Thank you for your contribution! Sashiko AI review found 8 potential
issues to consider:
- [High] mm: zsmalloc: build failure in find_alloced_obj() calling
  obj_allocated()
- [High] mm: zsmalloc: VM_BUG_ON in obj_write() calling single-page
  memcpy_to_page()
- [High] mm: zsmalloc: memory corruption in zs_rebind_handle() using
  offset_in_page()
- [High] mm: zsmalloc: unclamped basepage size class in zs_create_pool()
- [Medium] mm: zsmalloc: 24-bit offset overflow in
  set_first_obj_offset() on 64K pages
- [Medium] mm: zsmalloc: preemptible __zone_stat_add_folio() in zpdesc
  state helpers
- [Low] mm: zsmalloc: missing @type kernel-doc in zs_huge_class_size()
- [Low] mm: zsmalloc: missing space after order column in
  zs_stats_size_show()
--

commit de88105665ba4d7f0292945bc4a5ae1786f09efe
Author: Tangquan Zheng <zhengtangquan@oppo.com>

mm: zsmalloc: support objects compressed based on multiple pages

This commit adds multi-page object support to zsmalloc under
CONFIG_ZSMALLOC_MULTI_PAGES, introducing compound zpdesc allocations and size
classes above PAGE_SIZE up to ZSMALLOC_MULTI_PAGES_ORDER.

> diff --git a/mm/Kconfig b/mm/Kconfig
> index afff1b25b5438..b68720f6fb6bd 100644
> --- a/mm/Kconfig
> +++ b/mm/Kconfig
> @@ -157,6 +158,24 @@ config ZSMALLOC_CHAIN_SIZE
>  
>  	  For more information, see zsmalloc documentation.
>  
> +config ZSMALLOC_MULTI_PAGES
> +	bool "support zsmalloc multiple pages"
> +	depends on ZSMALLOC && !HIGHMEM
> +	help
> +	  This option configures zsmalloc to support allocations larger than
> +	  PAGE_SIZE, enabling compression across multiple pages. The size of
> +	  these multiple pages is determined by the configured
> +	  ZSMALLOC_MULTI_PAGES_ORDER.
> +
> +config ZSMALLOC_MULTI_PAGES_ORDER
> +	int "zsmalloc multiple pages order"
> +	default 2
> +	range 1 9
> +	depends on ZSMALLOC_MULTI_PAGES
> +	help
> +	  This option is used to configure zsmalloc to support the compression
> +	  of multiple pages.
> +
>  endmenu

[Severity: Medium]
Can ZSMALLOC_MULTI_PAGES_ORDER=9 overflow the 24-bit first_obj_offset field
on 64 KiB page systems?

With 64 KiB pages, order 9 gives a 32 MiB zpdesc, while
set_first_obj_offset() masks offsets with FIRST_OBJ_PAGE_TYPE_MASK (16 MiB)
and only checks BUILD_BUG_ON(PAGE_SIZE > SZ_16M):

mm/zsmalloc.c:set_first_obj_offset() {
	/* With 24 bits available, we can support offsets into 16 MiB pages. */
	BUILD_BUG_ON(PAGE_SIZE > SZ_16M);
	VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc)));
	VM_WARN_ON_ONCE(offset & ~FIRST_OBJ_PAGE_TYPE_MASK);
	zpdesc->first_obj_offset &= ~FIRST_OBJ_PAGE_TYPE_MASK;
	zpdesc->first_obj_offset |= offset & FIRST_OBJ_PAGE_TYPE_MASK;
}

When init_zspage() sets an initial object offset at or above 16 MiB, the
offset wraps modulo 16 MiB, causing find_alloced_obj() during compaction to
scan from the wrong offset inside an object payload.

> diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
> index 4a6ec43f418b2..2d83fcc8610da 100644
> --- a/mm/zsmalloc.c
> +++ b/mm/zsmalloc.c

[ ... ]

> @@ -295,27 +343,28 @@ struct zs_pool {
>  
>  static inline void zpdesc_inc_zone_page_state(struct zpdesc *zpdesc)
>  {
> -	inc_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES);
> +	__zone_stat_add_folio(zpdesc_folio(zpdesc), NR_ZSPAGES);
>  }
>  
>  static inline void zpdesc_dec_zone_page_state(struct zpdesc *zpdesc)
>  {
> -	dec_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES);
> +	__zone_stat_sub_folio(zpdesc_folio(zpdesc), NR_ZSPAGES);
>  }

[Severity: Medium]
Should these helpers use zone_stat_add_folio() and zone_stat_sub_folio()
instead of the double-underscore variants?

__zone_stat_add_folio() and __zone_stat_sub_folio() require preemption or
interrupts to be disabled, whereas alloc_zspage(), zs_free() ->
__free_zspage_lockless(), and zs_page_migrate() call these helpers in
preemptible context with no locks held, triggering preemption-disabled
assertions and racy per-CPU vmstat updates.

[ ... ]

> @@ -609,7 +667,8 @@ static int zs_stats_size_show(struct seq_file *s, void *v)
>  
>  		spin_lock(&class->lock);
>  
> -		seq_printf(s, " %5u %5u ", i, class->size);
> +		seq_printf(s, " %5u %5u %5u", i, class->size,
> +			   class_size_to_zs_order(class->size));

[Severity: Low]
Does the format string need a trailing space after the third "%5u" to match
the header and Total lines?  Without it, all subsequent columns in each
class row are shifted left by one character.

[ ... ]

> @@ -1189,16 +1257,17 @@ static void obj_write(struct size_class *class, struct zspage *zspage,
>  		      unsigned long obj, const void *src, size_t size)
>  {
>  	struct zpdesc *zpdesc;
> +	unsigned long page_size = class_size_to_zs_size(class->size);
>  	unsigned long off;
>  	unsigned int obj_idx;
>  
>  	obj_to_location(obj, &zpdesc, &obj_idx);
> -	off = offset_in_page(class->size * obj_idx);
> +	off = (class->size * obj_idx) & (page_size - 1);
>  
>  	if (!ZsHugePage(zspage))
>  		off += ZS_HANDLE_SIZE;
>  
> -	if (off + size <= PAGE_SIZE) {
> +	if (off + size <= page_size) {
>  		/* this object is contained entirely within a page */
>  		void *dst = kmap_local_zpdesc(zpdesc);
>  
> @@ -1208,7 +1277,7 @@ static void obj_write(struct size_class *class, struct zspage *zspage,
>  		/* this object spans two pages */
>  		size_t sizes[2];
>  
> -		sizes[0] = PAGE_SIZE - off;
> +		sizes[0] = page_size - off;
>  		sizes[1] = size - sizes[0];
>  
>  		memcpy_to_page(zpdesc_page(zpdesc), off, src, sizes[0]);

[Severity: High]
Does calling memcpy_to_page() here trigger VM_BUG_ON(offset + len > PAGE_SIZE)
when writing a multi-page object that spans two zpdescs?

For a multi-page class, page_size is larger than PAGE_SIZE, so off + sizes[0]
equals page_size and exceeds PAGE_SIZE in the first memcpy_to_page() call
(and sizes[1] can also exceed PAGE_SIZE in the second call).

> @@ -1230,9 +1299,9 @@ static void obj_write(struct size_class *class, struct zspage *zspage,
>   *
>   * Return: the size (in bytes) of the first huge zsmalloc &size_class.
>   */
> -size_t zs_huge_class_size(struct zs_pool *pool)
> +size_t zs_huge_class_size(struct zs_pool *pool, enum zsmalloc_type type)
>  {
> -	return huge_class_size;
> +	return huge_class_size[type];
>  }
>  EXPORT_SYMBOL_GPL(zs_huge_class_size);

[Severity: Low]
The kernel-doc comment above zs_huge_class_size() is missing an @type entry
for the new parameter.

[ ... ]

> @@ -1365,13 +1437,15 @@ static void obj_free(int class_size, unsigned long obj)
>  	struct link_free *link;
>  	struct zspage *zspage;
>  	struct zpdesc *f_zpdesc;
> -	unsigned long f_offset;
> +	unsigned long f_offset, page_size, page_mask;
>  	unsigned int f_objidx;
>  	void *vaddr;
>  
>  
>  	obj_to_location(obj, &f_zpdesc, &f_objidx);
> -	f_offset = offset_in_page(class_size * f_objidx);
> +	page_size = class_size_to_zs_size(class_size);
> +	page_mask = ~(page_size - 1);
> +	f_offset = (class_size * f_objidx) & ~page_mask;
>  	zspage = get_zspage(f_zpdesc);

[Severity: High]
Does zs_rebind_handle() also need to mask the object offset with
page_size - 1 instead of offset_in_page()?

mm/zsmalloc.c:zs_rebind_handle() {
    ...
	if (likely(!ZsHugePage(zspage))) {
		obj_to_location(obj, &zpdesc, &obj_idx);
		off = offset_in_page(class->size * obj_idx);
		vaddr = kmap_local_zpdesc(zpdesc);
		link = (struct link_free *)(vaddr + off);
		link->handle = (unsigned long)dst | OBJ_ALLOCATED_TAG;
		kunmap_local(vaddr);
    ...
}

When a non-huge multi-page object sits at an offset >= PAGE_SIZE within its
zpdesc, zs_rebind_handle() overwrites 8 bytes in the first subpage and leaves
the object's own header pointing to the old src handle, so later compaction
updates src instead of dst.

[ ... ]

> @@ -1615,13 +1691,13 @@ static struct zs_handle *find_alloced_obj(struct size_class *class,
>  	int index = *obj_idx;
>  	struct zs_handle *handle = NULL;
>  	void *addr = kmap_local_zpdesc(zpdesc);
> +	unsigned long page_size = class_size_to_zs_size(class->size);
>  
>  	offset = get_first_obj_offset(zpdesc);
>  	offset += class->size * index;
>  
> -	while (offset < PAGE_SIZE) {
> -		handle = obj_allocated(zpdesc, addr + offset);
> -		if (handle)
> +	while (offset < page_size) {
> +		if (obj_allocated(zpdesc, addr + offset, &handle))
>  			break;

[Severity: High]
Does this break compilation at this commit?

obj_allocated() only takes two arguments (zpdesc and obj) and returns the
handle pointer, so passing &handle fails to build until commit "zram: support
compression at the granularity of multi-pages" reverts this line back to the
two-argument call.

[ ... ]

> @@ -2215,16 +2292,32 @@ struct zs_pool *zs_create_pool(const char *name)
>  	 */
>  	for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) {
>  		int size;
> -		int pages_per_zspage;
> +		int zpdescs_per_zspage;
>  		int objs_per_zspage;
>  		struct size_class *class;
>  		int fullness;
> +		int idx = ZSMALLOC_TYPE_BASEPAGE;
> +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES
> +		int order = 0;
> +#endif
>  
> -		size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA;
> +		if (i < ZS_PAGE_SIZE_CLASSES)
> +			size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA;
> +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES
> +		if (i >= ZS_PAGE_SIZE_CLASSES)
> +			size = PAGE_SIZE + (i - ZS_PAGE_SIZE_CLASSES) *
> +			       ZS_MULTI_PAGES_SIZE_CLASS_DELTA;
> +#endif
>  		if (size > ZS_MAX_ALLOC_SIZE)
>  			size = ZS_MAX_ALLOC_SIZE;

[Severity: High]
Can the last base-page size class exceed PAGE_SIZE when
CONFIG_ZSMALLOC_MULTI_PAGES is enabled?

Previously, ZS_MAX_ALLOC_SIZE was PAGE_SIZE, which clamped class
ZS_PAGE_SIZE_CLASSES - 1 to PAGE_SIZE when PAGE_SIZE - ZS_MIN_ALLOC_SIZE was
not a multiple of ZS_SIZE_CLASS_DELTA (such as on 16 KiB and 64 KiB page
kernels).  With ZS_MAX_ALLOC_SIZE raised to ZSMALLOC_MULTI_PAGES_SIZE, class
ZS_PAGE_SIZE_CLASSES - 1 gets size == PAGE_SIZE + 32 on 16 KiB and 64 KiB
page systems, turning it into a non-huge multi-page class while class
ZS_PAGE_SIZE_CLASSES starts at PAGE_SIZE and is unreachable from
get_size_class_index().  Storing an incompressible PAGE_SIZE page then lands
in a non-huge class with an 8-byte handle prefix, corrupting incompressible
page reads in zram and triggering length warnings in zswap.

Should sizes for i < ZS_PAGE_SIZE_CLASSES be clamped to PAGE_SIZE, and should
the multi-page classes start at i - ZS_PAGE_SIZE_CLASSES + 1?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261115024455.1465726-1-zhaonanzhe@xiaomi.com?part=1


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages
  2026-11-15  2:44 ` [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages Nanzhe Zhao
@ 2026-11-16  2:46   ` sashiko-bot
  0 siblings, 0 replies; 8+ messages in thread
From: sashiko-bot @ 2026-11-16  2:46 UTC (permalink / raw)
  To: Nanzhe Zhao, Andrew Morton
  Cc: Tangquan Zheng, linux-mm, Jens Axboe, Sergey Senozhatsky,
	Minchan Kim, Kairui Song, Johannes Weiner, Nhat Pham, Usama Arif,
	Yosry Ahmed, Chengming Zhou, Chris Li, David Hildenbrand,
	Ying Huang, Suren Baghdasaryan, Ryan Roberts, Kanchana P Sridhar,
	Wajdi K Feghali, Bala Seshasayee, Nick Terrell, Barry Song,
	Bo Zhang

Thank you for your contribution! Sashiko AI review found 14 potential
issues to consider:
- [Critical] zram: partial read in zram_read_partial_from_zspool()
  ignores bv_len/offset
- [Critical] zram: OOB access and data loss in zram_write_page() using
  folio_size()
- [Critical] zram: bio-wide routing in want_multi_pages_comp() and
  test_multi_pages_comp()
- [Critical] zram: multi_pages_bvec_iter_bvec() and partial read corrupt
  multi-page I/O
- [High] zram: unlocked flag updates and leaked handles in
  zram_write_page()
- [High] zram: sibling race and premature flag clear in
  zram_slot_free_notify()
- [High] zram: unlocked RMW in zram_bvec_write_multi_pages_partial()
  loses writes
- [High] zram: use-after-free in zram_bvec_write_multi_pages_partial()
  on ZRAM_WB
- [High] zram: out-of-bounds access in place_pp_slot() and
  post-processing overflow
- [High] mm: zsmalloc: VM_BUG_ON in obj_write() and bad offset in
  zs_rebind_handle()
- [Medium] zram: out-of-bounds zram->table access when disksize is not
  multi-page aligned
- [Medium] zram: ZRAM_FLAG_SHIFT overflows 32-bit attr.flags on 16K/64K
  page configs
- [Medium] zram: inverted lock order between excl_lock and buf_lock in
  zcomp
- [Low] zram: misplaced zsmalloc build fix, missing commit data, and
  stat format
--

commit d614cb6134b7b12651ec475514fd15b492598ab0
Author: Tangquan Zheng <zhengtangquan@oppo.com>

zram: support compression at the granularity of multi-pages

This commit adds CONFIG_ZRAM_MULTI_PAGES to compress and store large
folios in ZCOMP_MULTI_PAGES_SIZE chunks instead of individual PAGE_SIZE
pages.

> diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
> index ad8c7dc739223..8fa2a44adaa30 100644
> --- a/drivers/block/zram/zcomp.c
> +++ b/drivers/block/zram/zcomp.c

[ ... ]

> @@ -101,11 +105,23 @@ static int zcomp_dstrm_init(struct zcomp *comp, struct zcomp_dstrm *zstrm)
>  		}
>  	}
>  
> -	zstrm->local_copy = vzalloc(PAGE_SIZE);
> +	zstrm->local_copy = vzalloc(ZCOMP_MULTI_PAGES_SIZE);
>  	if (!zstrm->local_copy) {
>  		zcomp_dstrm_free(comp, zstrm);
>  		return -ENOMEM;
>  	}
> +#ifdef CONFIG_ZRAM_MULTI_PAGES
> +	/*
> +	 * Preallocate a decompression buffer for partial multi-page reads to
> +	 * avoid memory allocation in the swapin path.
> +	 */
> +	zstrm->decomp_buf = vzalloc(ZCOMP_MULTI_PAGES_SIZE);
> +	if (!zstrm->decomp_buf) {
> +		zcomp_dstrm_free(comp, zstrm);
> +		return -ENOMEM;
> +	}
> +	mutex_init(&zstrm->buf_lock);
> +#endif
>  	return 0;
>  }

[Severity: Medium]
Can zstrm->excl_lock and zstrm->buf_lock be acquired in opposite orders?

In zram_read_partial_from_zspool(), an exclusive decompression backend
(ZCOMP_DSTRM_EXCLUSIVE) takes zstrm->excl_lock inside zcomp_dstrm_get()
and then locks zstrm->buf_lock.  A shared decompression backend
(ZCOMP_DSTRM_SHARED) locks zstrm->buf_lock first and then calls
zcomp_decompress(), which locks zstrm->excl_lock when a two-page object
falls back to zstrm->local_copy:

CPU0                              CPU1
----                              ----
lock(&zstrm->excl_lock)
                                  lock(&zstrm->buf_lock)
lock(&zstrm->buf_lock)
                                  lock(&zstrm->excl_lock)

 *** DEADLOCK ***

CPU0: zram_read_partial_from_zspool() -> zcomp_dstrm_get()
CPU1: zram_read_partial_from_zspool() -> zcomp_decompress()

Because all zstrm->excl_lock and zstrm->buf_lock instances share static
lockdep classes across devices and CONFIG_ZRAM_MULTI_COMP priorities,
this triggers a lockdep circular locking splat.

Also, zcomp_dstrm_init() runs from the CPUHP_ZCOMP_PREPARE hotplug
callback every time a CPU comes online, so mutex_init(&zstrm->buf_lock)
re-initializes an already initialized mutex instead of initializing it
once per possible CPU in zcomp_init().

> diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
> index 8b7d6926da266..e258289679015 100644
> --- a/drivers/block/zram/zram_drv.c
> +++ b/drivers/block/zram/zram_drv.c

[ ... ]

> @@ -484,7 +486,7 @@ static ssize_t idle_store(struct device *dev, struct device_attribute *attr,
>  static int read_from_zspool_raw(struct zram *zram, struct page *page,
>  				unsigned long index);
>  static int read_from_zspool(struct zram *zram, struct page *page,
> -			    unsigned long index);
> +			    unsigned long index, unsigned long page_size);
>  
>  struct zram_wb_ctl {
>  	/* idle list is accessed only by the writeback task, no concurency */
> @@ -1092,7 +1094,7 @@ static int zram_writeback_slots(struct zram *zram,
>  		if (zram->compressed_wb)
>  			err = read_from_zspool_raw(zram, req->page, index);
>  		else
> -			err = read_from_zspool(zram, req->page, index);
> +			err = read_from_zspool(zram, req->page, index, PAGE_SIZE);
>  		if (err)
>  			goto next;
>  		slot_unlock(zram, index);

[Severity: High]
Can post-processing a multi-page slot overflow ctl->pp_buckets[] and
req->page?

Neither scan_slots_for_writeback() nor scan_slots_for_recompress() skips
ZRAM_COMP_MULTI_PAGES slots, where get_slot_size() can be up to
ZCOMP_MULTI_PAGES_SIZE.  In place_pp_slot(), NUM_PP_BUCKETS is sized for
PAGE_SIZE:

drivers/block/zram/zram_drv.c:place_pp_slot() {
	...
	bid = get_slot_size(zram, pps->index) / PP_BUCKET_SIZE_RANGE;
	list_add(&pps->entry, &ctl->pp_buckets[bid]);
}

When a multi-page slot's compressed size exceeds PAGE_SIZE, bid exceeds
NUM_PP_BUCKETS and corrupts memory past struct zram_pp_ctl.  In
zram_writeback_slots(), req->page is a single PAGE_SIZE page, so
read_from_zspool_raw() also overflows req->page and wraps PAGE_SIZE - size
in memzero_page(), while uncompressed writeback and recompress_slot() pass
PAGE_SIZE for a multi-page object.

[ ... ]

> @@ -2298,12 +2365,27 @@ static int zram_write_page(struct zram *zram, struct page *page,
>  	struct zcomp_cstrm *zstrm;
>  	unsigned long element;
>  	bool same_filled;
> +	unsigned long page_size = PAGE_SIZE;
> +	int huge_class_idx = ZSMALLOC_TYPE_BASEPAGE;
> +	int nr_pages = 1;
> +
> +#ifdef CONFIG_ZRAM_MULTI_PAGES
> +	/*
> +	 * Multi-page compression packs ZCOMP_MULTI_PAGES_NR pages into one
> +	 * compressed object.
> +	 */
> +	if (folio_size(page_folio(page)) >= ZCOMP_MULTI_PAGES_SIZE) {
> +		page_size = ZCOMP_MULTI_PAGES_SIZE;
> +		huge_class_idx = ZSMALLOC_TYPE_MULTI_PAGES;
> +		nr_pages = ZCOMP_MULTI_PAGES_NR;
> +	}
> +#endif
>  
>  	mem = kmap_local_page(page);
> -	same_filled = page_same_filled(mem, &element);
> +	same_filled = page_same_filled(mem, &element, page_size);
>  	kunmap_local(mem);
>  	if (same_filled)
> -		return write_same_filled_page(zram, element, index);
> +		return write_same_filled_page(zram, element, index, nr_pages);

[Severity: Critical]
Does sizing the write from folio_size(page_folio(page)) instead of the I/O
length cause out-of-bounds accesses and dropped pages?

When a single-page bio (bv_len == PAGE_SIZE) writes a subpage belonging to
a large folio, zram_bio_write() takes the single-page loop and calls
zram_write_page(), where folio_size(page_folio(page)) >=
ZCOMP_MULTI_PAGES_SIZE is still true.  If page is near the end of the
folio, reading ZCOMP_MULTI_PAGES_SIZE bytes starting at page reads past
the end of the folio, and setting ZRAM_COMP_MULTI_PAGES across index ..
index + nr_pages - 1 on an unaligned index corrupts adjacent slots or
writes past the end of zram->table.

Conversely, if bio_add_page() merges ZCOMP_MULTI_PAGES_NR physically
contiguous order-0 pages into one bvec of length ZCOMP_MULTI_PAGES_SIZE,
zram_bio_write_multi_pages() advances the iterator by
ZCOMP_MULTI_PAGES_SIZE while zram_write_page() sees folio_size() ==
PAGE_SIZE and stores only the first page.

> @@ -2320,9 +2402,10 @@ static int zram_write_page(struct zram *zram, struct page *page,
>  		return ret;
>  	}
>  
> -	if (comp_len >= huge_class_size) {
> +	if (comp_len >= huge_class_size[huge_class_idx]) {
>  		zcomp_cstrm_put(zstrm);
> -		return write_incompressible_page(zram, page, index);
> +		return write_incompressible_page(zram, page, index, page_size,
> +						 nr_pages);
>  	}
>  
>  	slot_lock(zram, index);
> @@ -2341,10 +2424,18 @@ static int zram_write_page(struct zram *zram, struct page *page,
>  
>  	set_slot_size(zram, index, comp_len);
>  	mark_slot_accessed(zram, index);
> +#ifdef CONFIG_ZRAM_MULTI_PAGES
> +	if (nr_pages > 1) {
> +		int i;
> +
> +		for (i = 0; i < nr_pages; i++)
> +			set_slot_flag(zram, index + i, ZRAM_COMP_MULTI_PAGES);
> +	}
> +#endif
>  	slot_unlock(zram, index);

[Severity: High]
Does setting ZRAM_COMP_MULTI_PAGES on index + 1 .. index + nr_pages - 1
without locking or freeing those tail slots corrupt the slot bit-lock and
leak existing objects?

zram_write_page(), write_same_filled_page(), and
write_incompressible_page() only lock and call slot_free() on the head
slot index:

1. Because table[index + i].attr.flags aliases table[index + i].__lock,
   the unlocked non-atomic |= in set_slot_flag(zram, index + i, ...) races
   with concurrent slot_lock() / slot_unlock() on tail slots and can
   corrupt ZRAM_ENTRY_LOCK_BIT, hanging waiters in wait_on_bit_lock().
2. Any single-page zsmalloc handles already stored in the tail slots are
   never freed before ZRAM_COMP_MULTI_PAGES is set on them, leaking those
   objects.
3. During a partial write after some sub-slots were already freed by
   zram_slot_free_notify(), re-setting ZRAM_COMP_MULTI_PAGES on all
   nr_pages sub-slots marks already-freed swap slots that will never
   receive another free notification, so the multi-page object can never
   be freed when the remaining active sub-slots are released.
4. If the head sub-slot was freed first via zram_slot_free_notify() (which
   clears ZRAM_COMP_MULTI_PAGES on index), a later slot_free(zram, index)
   sees ZRAM_COMP_MULTI_PAGES clear on index and subtracts 1 instead of
   ZCOMP_MULTI_PAGES_NR from pages_stored and same_pages, while
   slot_free_npages() on discard or zs_store() failure leaves
   ZRAM_COMP_MULTI_PAGES set on the tail slots.
5. On a zsmalloc multi-page size class, zs_store() -> obj_write() calls
   memcpy_to_page() with off + sizes[0] == ZSMALLOC_MULTI_PAGES_SIZE when
   an object spans two zpdescs, which triggers
   VM_BUG_ON(offset + len > PAGE_SIZE) in memcpy_to_page(); similarly,
   zs_rebind_handle() uses offset_in_page() instead of masking with
   page_size - 1, corrupting object headers at offsets >= PAGE_SIZE.

[ ... ]

> +#ifdef CONFIG_ZRAM_MULTI_PAGES
> +
> +static inline unsigned long zram_multi_pages_head_index(unsigned long index)
> +{
> +	return index & ~((unsigned long)ZCOMP_MULTI_PAGES_NR - 1);
> +}
> +
> +static inline int __test_multi_pages_comp(struct zram *zram,
> +					 unsigned long index)
> +{
> +	unsigned long head = zram_multi_pages_head_index(index);
> +	int i, count = 0;
> +
> +	for (i = 0; i < ZCOMP_MULTI_PAGES_NR; i++) {
> +		if (test_slot_flag(zram, head + i, ZRAM_COMP_MULTI_PAGES))
> +			count++;
> +	}
> +
> +	return count;
> +}

[Severity: Medium]
Can __test_multi_pages_comp() access past the end of zram->table when
disksize is not a multiple of ZCOMP_MULTI_PAGES_SIZE?

disksize_store() only aligns disksize with PAGE_ALIGN(disksize).  When
num_pages is not a multiple of ZCOMP_MULTI_PAGES_NR, an access to the
final slots has head + i >= num_pages, so __test_multi_pages_comp() and
the set_slot_flag() loops in the multi-page write path read and write past
the end of zram->table.

> +static inline bool want_multi_pages_comp(struct zram *zram, struct bio *bio)
> +{
> +	unsigned long index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT;
> +
> +	if (bio->bi_io_vec->bv_len >= ZCOMP_MULTI_PAGES_SIZE)
> +		return true;
> +
> +	slot_lock(zram, index);
> +	if (__test_multi_pages_comp(zram, index)) {
> +		slot_unlock(zram, index);
> +		return true;
> +	}
> +	slot_unlock(zram, index);
> +
> +	return false;
> +}
> +
> +static inline bool test_multi_pages_comp(struct zram *zram, struct bio *bio)
> +{
> +	unsigned long index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT;
> +
> +	return !!__test_multi_pages_comp(zram, index);
> +}

[Severity: Critical]
Can routing the entire bio through the multi-page or single-page path
based only on the first sector crash or corrupt data?

want_multi_pages_comp() and test_multi_pages_comp() inspect only the bio's
initial sector and bio->bi_io_vec->bv_len before iterating the bio.  If an
unaligned write bio with bv_len >= ZCOMP_MULTI_PAGES_SIZE or a multi-bvec
bio enters the multi-page path and lands on a head slot that holds a
single-page (PAGE_SIZE) object, zram_bvec_write_multi_pages_partial() or
zram_bvec_read_multi_pages() calls zram_read_page_size(...,
ZCOMP_MULTI_PAGES_SIZE) on that single-page slot:

- If the single-page head slot has ZRAM_HUGE set (a 1-page ZsHugePage),
  read_incompressible_page() calls zs_obj_read_sg_begin() with
  ZCOMP_MULTI_PAGES_SIZE, which takes the two-page branch, gets NULL from
  get_next_zpdesc(zpdesc), and dereferences NULL in sg_copy_to_buffer().
- If the single-page head slot holds a compressed PAGE_SIZE object during
  zram_bvec_write_multi_pages_partial(), zcomp_decompress() only fills the
  first PAGE_SIZE bytes of the unzeroed order-ZCOMP_MULTI_PAGES_ORDER page,
  and zram_write_page() compresses and stores the remaining uninitialized
  kernel memory.

Conversely, if a read bio starts on a single-page slot and crosses into a
multi-page slot group, zram_bio_read() stays in the single-page loop,
failing decompression on the head slot and reading the unallocated tail
slots as zero-filled pages.

> +static inline bool is_multi_pages_partial_io(struct bio_vec *bvec)
> +{
> +	return bvec->bv_len != ZCOMP_MULTI_PAGES_SIZE;
> +}
> +
> +static int zram_read_partial_from_zspool(struct zram *zram, struct page *page,
> +					unsigned long index, int offset)
> +{
> +	struct zs_handle *handle = &zram->table[index].handle;
> +	struct zcomp_dstrm *zstrm;
> +	struct scatterlist sg[2];
> +	enum zcomp_dstrm_mode mode;
> +	unsigned int size;
> +	void *src, *dst;
> +	u32 prio;
> +	int ret;
> +	unsigned long page_size = ZCOMP_MULTI_PAGES_SIZE;
> +
> +	if (test_slot_flag(zram, index, ZRAM_SAME) ||
> +	    !zs_handle_is_allocated(handle)) {
> +		void *mem;
> +
> +		mem = kmap_local_page(page);
> +		zram_fill_page(mem, PAGE_SIZE, zram->table[index].element);
> +		kunmap_local(mem);
> +		return 0;
> +	}
> +
> +	size = get_slot_size(zram, index);
> +	zs_obj_read_sg_begin(zram->mem_pool, handle, sg, size);
> +
> +	if (size == page_size) {
> +		src = kmap_local_page(sg_page(sg));
> +		dst = kmap_local_page(page);
> +		memcpy(dst, src + offset, PAGE_SIZE);
> +		kunmap_local(dst);
> +		kunmap_local(src);
> +		ret = 0;
> +	} else {
> +		prio = get_slot_comp_priority(zram, index);
> +		mode = ZCOMP_DSTRM_EXCLUSIVE;
> +		if (zcomp_shared_decomp(zram->comps[prio]))
> +			mode = ZCOMP_DSTRM_SHARED;
> +
> +		zstrm = zcomp_dstrm_get(zram->comps[prio], mode);
> +		mutex_lock(&zstrm->buf_lock);
> +		ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size,
> +				       zstrm->decomp_buf, page_size);
> +		if (!ret) {
> +			dst = kmap_local_page(page);
> +			memcpy(dst, zstrm->decomp_buf + offset, PAGE_SIZE);
> +			kunmap_local(dst);
> +		}
> +		mutex_unlock(&zstrm->buf_lock);
> +		zcomp_dstrm_put(zram->comps[prio], zstrm, mode);
> +	}
> +	zs_obj_read_sg_end(zram->mem_pool, handle);
> +
> +	return ret;
> +}

[Severity: Critical]
Does zram_read_partial_from_zspool() ignore bvec->bv_len and
bvec->bv_offset?

zram_bvec_read_multi_pages_partial() passes only bvec->bv_page and the
slot offset to zram_read_partial_from_zspool(), and all three branches
above always write PAGE_SIZE bytes to offset 0 of page.  When a partial
multi-page read has bv.bv_len > PAGE_SIZE (for example, 2 or 3 pages) or a
non-zero bvec->bv_offset, zram_bio_read_multi_pages() still advances the
bio iterator by bv.bv_len, writing to the wrong offset of bv_page or
leaving the remaining pages of the read uninitialized.

[ ... ]

> +static int zram_bvec_write_multi_pages_partial(struct zram *zram,
> +					       struct bio_vec *bvec,
> +					       unsigned long index, int offset,
> +					       struct bio *bio)
> +{
> +	struct page *page;
> +	void *src, *dst;
> +	int ret;
> +
> +	page = alloc_pages(GFP_NOIO | __GFP_COMP, ZCOMP_MULTI_PAGES_ORDER);
> +	if (!page)
> +		return -ENOMEM;
> +
> +	ret = zram_read_page_size(zram, page, index, bio,
> +				  ZCOMP_MULTI_PAGES_SIZE);
> +	if (!ret) {
> +		src = kmap_local_page(bvec->bv_page);
> +		dst = kmap_local_page(page);
> +		memcpy(dst + offset, src + bvec->bv_offset, bvec->bv_len);
> +		kunmap_local(dst);
> +		kunmap_local(src);
> +
> +		atomic64_inc(&zram->stats.zram_bio_write_multi_pages_partial_count);
> +		ret = zram_write_page(zram, page, index);
> +	}
> +
> +	__free_pages(page, ZCOMP_MULTI_PAGES_ORDER);
> +	return ret;
> +}

[Severity: High]
Can two concurrent partial writes to different sub-slots in the same
multi-page group overwrite each other?

zram_read_page_size() unlocks slot_lock(zram, index) after reading the
multi-page into the temporary page, and zram_write_page() only re-acquires
slot_lock(zram, index) after compressing the updated buffer.  If two CPUs
concurrently perform partial writes to different offsets of the same head
index, both read the old multi-page before either writes back, so the
second writer overwrites the first writer's sub-page with stale data.

[Severity: High]
Does passing bio as parent to zram_read_page_size() cause a use-after-free
of page when ZRAM_WB is set?

When test_slot_flag(zram, index, ZRAM_WB) is true, zram_read_page_size()
calls read_from_bdev(zram, page, index, blk_idx, parent) with a non-NULL
parent bio:

drivers/block/zram/zram_drv.c:read_from_bdev() {
	atomic64_inc(&zram->stats.bd_reads);
	if (!parent) {
		if (WARN_ON_ONCE(!IS_ENABLED(ZRAM_PARTIAL_IO)))
			return -EIO;
		return read_from_bdev_sync(zram, page, index, blk_idx);
	}
	return read_from_bdev_async(zram, page, index, blk_idx, parent);
}

read_from_bdev_async() submits an asynchronous child bio referencing page
(inheriting REQ_OP_WRITE from the write bio) and returns 0 immediately,
after which zram_bvec_write_multi_pages_partial() frees page via
__free_pages() while the backing-device I/O and completion work are still
using it.

[ ... ]

> @@ -2814,7 +3189,34 @@ static void zram_slot_free_notify(struct block_device *bdev,
>  		return;
>  	}
>  
> +#ifdef CONFIG_ZRAM_MULTI_PAGES
> +	{
> +		int comp_count = __test_multi_pages_comp(zram, index);
> +
> +		if (comp_count > 1) {
> +			clear_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES);
> +			slot_unlock(zram, index);
> +			return;
> +		}
> +
> +		if (comp_count == 1) {
> +			clear_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES);
> +			slot_unlock(zram, index);
> +			index = zram_multi_pages_head_index(index);
> +			nr_pages_to_free = ZCOMP_MULTI_PAGES_NR;
> +			if (!slot_trylock(zram, index)) {
> +				atomic64_inc(&zram->stats.multi_pages_miss_free);
> +				return;
> +			}
> +		}
> +	}
> +#endif

[Severity: High]
Can concurrent zram_slot_free_notify() calls on sibling sub-slots leak the
multi-page object?

__test_multi_pages_comp() counts ZRAM_COMP_MULTI_PAGES across the group
while only holding slot_trylock(zram, index) on the current sub-slot:

CPU0                                  CPU1
zram_slot_free_notify(head + 1)       zram_slot_free_notify(head + 2)
  slot_trylock(zram, head + 1)          slot_trylock(zram, head + 2)
  __test_multi_pages_comp() -> 2        __test_multi_pages_comp() -> 2
  clear_slot_flag(head + 1, ...)        clear_slot_flag(head + 2, ...)
  slot_unlock(zram, head + 1)           slot_unlock(zram, head + 2)
  return;                               return;  <-- head is never freed

In addition, when comp_count == 1, clearing ZRAM_COMP_MULTI_PAGES and
unlocking index before slot_trylock(zram, head) means that if
slot_trylock(zram, head) fails, the last ZRAM_COMP_MULTI_PAGES flag has
already been cleared, and another CPU can also reallocate and write to
head in the unlocked window before slot_trylock(zram, head) runs.
__test_multi_pages_comp() also does not check whether index itself has
ZRAM_COMP_MULTI_PAGES set before treating it as part of a multi-page
group.

> diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h
> index d64c99b74214b..1e3ebfec972ec 100644
> --- a/drivers/block/zram/zram_drv.h
> +++ b/drivers/block/zram/zram_drv.h

[ ... ]

> @@ -37,7 +42,12 @@
>   *
>   * We use BUILD_BUG_ON() to make sure that zram pageflags don't overflow.
>   */
> +#ifdef CONFIG_ZRAM_MULTI_PAGES
> +#define ZRAM_FLAG_SHIFT (PAGE_SHIFT + CONFIG_ZSMALLOC_MULTI_PAGES_ORDER + 1)
> +#else
>  #define ZRAM_FLAG_SHIFT (PAGE_SHIFT + 1)
> +#endif

[Severity: Medium]
Does ZRAM_FLAG_SHIFT overflow the 32-bit attr.flags field when
PAGE_SHIFT + CONFIG_ZSMALLOC_MULTI_PAGES_ORDER > 21?

CONFIG_ZSMALLOC_MULTI_PAGES_ORDER allows values up to 9, and enum
zram_pageflags defines 10 flag bits above ZRAM_FLAG_SHIFT.  On 16K or 64K
page configurations with larger orders (for example, PAGE_SHIFT == 16 with
order >= 6), __NR_ZRAM_PAGEFLAGS exceeds 32 and triggers
BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > sizeof(zram_te.attr.flags) * 8) in
zram_init().

[ ... ]

> @@ -167,4 +190,24 @@ struct zram {
>  	struct dentry *debugfs_dir;
>  #endif
>  };
> +
> +#ifdef CONFIG_ZRAM_MULTI_PAGES
> +#define multi_pages_bvec_iter_offset(bvec, iter)				\
> +	(mp_bvec_iter_offset((bvec), (iter)) % ZCOMP_MULTI_PAGES_SIZE)
> +
> +#define multi_pages_bvec_iter_len(bvec, iter)				\
> +	min_t(unsigned int, mp_bvec_iter_len((bvec), (iter)),		\
> +	      ZCOMP_MULTI_PAGES_SIZE -					\
> +	      multi_pages_bvec_iter_offset((bvec), (iter)))
> +
> +#define multi_pages_bvec_iter_bvec(bvec, iter)				\
> +((struct bio_vec) {							\
> +	.bv_page	= mp_bvec_iter_page((bvec), (iter)),		\
> +	.bv_len		= multi_pages_bvec_iter_len((bvec), (iter)),	\
> +	.bv_offset	= multi_pages_bvec_iter_offset((bvec), (iter)),	\
> +})
> +
> +#define multi_pages_bio_iter_iovec(bio, iter)				\
> +	multi_pages_bvec_iter_bvec((bio)->bi_io_vec, (iter))
> +#endif
>  #endif

[Severity: Critical]
Does using mp_bvec_iter_page() in multi_pages_bvec_iter_bvec() keep
.bv_page stuck at the first page of a large folio across iterations?

Unlike bvec_iter_page(), mp_bvec_iter_page(bvec, iter) always returns
bvec[iter.bi_idx].bv_page without adding mp_bvec_iter_page_idx(bvec,
iter), while multi_pages_bvec_iter_offset() wraps .bv_offset modulo
ZCOMP_MULTI_PAGES_SIZE.  When a folio larger than ZCOMP_MULTI_PAGES_SIZE
(such as a 64KiB or 128KiB folio in a single multi-page bvec) is read or
written, every iteration of zram_bio_write_multi_pages() and
zram_bio_read_multi_pages() accesses subpage 0 of the folio, repeatedly
compressing or overwriting the first ZCOMP_MULTI_PAGES_SIZE bytes and
leaving the rest of the folio unread or unwritten.

> diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c
> index 2d83fcc8610da..81d908d6d5a8d 100644
> --- a/mm/zsmalloc.c
> +++ b/mm/zsmalloc.c
> @@ -1697,7 +1697,8 @@ static struct zs_handle *find_alloced_obj(struct size_class *class,
>  	offset += class->size * index;
>  
>  	while (offset < page_size) {
> -		if (obj_allocated(zpdesc, addr + offset, &handle))
> +		handle = obj_allocated(zpdesc, addr + offset);
> +		if (handle)
>  			break;
>  
>  		offset += class->size;

[Severity: Low]
Should this find_alloced_obj() change be folded into the preceding commit
"mm: zsmalloc: support objects compressed based on multiple pages" so the
series stays bisectable?

Also, the commit message ends with "The following data illustrates the
time and compressed data for typical anonymous pages gathered from Android
phones." without including the data table, and multi_pages_debug_stat_show()
is missing a colon or space before "%8llu" on the failed and partial count
lines.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261115024455.1465726-1-zhaonanzhe@xiaomi.com?part=2


^ permalink raw reply	[flat|nested] 8+ messages in thread

* Re: [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin
  2026-11-15  2:44 ` [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin Nanzhe Zhao
@ 2026-11-16  2:46   ` sashiko-bot
  0 siblings, 0 replies; 8+ messages in thread
From: sashiko-bot @ 2026-11-16  2:46 UTC (permalink / raw)
  To: Nanzhe Zhao, Andrew Morton
  Cc: linux-mm, Jens Axboe, Sergey Senozhatsky, Minchan Kim,
	Kairui Song, Johannes Weiner, Nhat Pham, Usama Arif, Yosry Ahmed,
	Chengming Zhou, Chris Li, David Hildenbrand, Ying Huang,
	Suren Baghdasaryan, Ryan Roberts, Kanchana P Sridhar,
	Wajdi K Feghali, Bala Seshasayee, Nick Terrell, Barry Song,
	Tangquan Zheng, Bo Zhang

Thank you for your contribution! Sashiko AI review found 2 potential
issues to consider:
- [High] mm: __GFP_RECLAIM without __GFP_NORETRY in __swap_cache_alloc()
  triggers OOM
- [Medium] mm: hardcoded order-2 __GFP_RECLAIM in __swap_cache_alloc()
  bypasses policies
--

commit cb747b328cc1a8c3f1c43755cbc281a7af714c90
Author: Nanzhe Zhao <zhaonanzhe@xiaomi.com>

mm: swap: allow direct reclaim for synchronous multi-page swapin

This commit enables __GFP_RECLAIM for order-2 folio allocations in
__swap_cache_alloc() when allocating via vma_alloc_folio() to improve
multi-page swapin allocation success.

> diff --git a/mm/swap_state.c b/mm/swap_state.c
> index da65d75ceead1..912b0e21a781e 100644
> --- a/mm/swap_state.c
> +++ b/mm/swap_state.c
> @@ -462,6 +462,13 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci,
>  	if (mpol || !vmf) {
>  		folio = folio_alloc_mpol(gfp, order, mpol, ilx, numa_node_id());
>  	} else {
> +		/*
> +		 * zsmalloc uses order-2 pages to store compressed data.
> +		 * Allow direct reclaim to improve allocation success and
> +		 * avoid partial reads.
> +		 */
> +		if (order == 2)
> +			gfp |= __GFP_RECLAIM;
>  		address = round_down(vmf->address, PAGE_SIZE << order);
>  		folio = vma_alloc_folio(gfp, order, vmf->vma, address);
>  	}

[Severity: High]
Can setting __GFP_RECLAIM without __GFP_NORETRY or __GFP_RETRY_MAYFAIL
trigger premature OOM kills on order-2 swapin faults instead of falling
back to order 0?

Under the default THP defrag policy (madvise), as well as defer,
defer+madvise, and never, vma_thp_gfp_mask() returns GFP_TRANSHUGE_LIGHT
without __GFP_NORETRY.  Because order 2 is within PAGE_ALLOC_COSTLY_ORDER
(3), adding __GFP_DIRECT_RECLAIM without __GFP_NORETRY causes
__alloc_pages_slowpath() to enter full reclaim/compaction retry loops and
invoke __alloc_pages_may_oom() -> out_of_memory() when an order-2 page
cannot be formed.  The same gfp mask is also passed to
mem_cgroup_swapin_charge_folio() -> try_charge_memcg(), so a cgroup with 1
to 3 pages of headroom triggers mem_cgroup_oom() instead of returning
-ENOMEM and letting __swap_cache_alloc_folio() fall back to order 0.

[Severity: Medium]
Should __swap_cache_alloc() unconditionally override the GFP mask for
order 2 here?

__swap_cache_alloc() is shared by all swap devices even when zram or
CONFIG_ZRAM_MULTI_PAGES is not used, and when multi-page compression is
enabled its order is configured via CONFIG_ZSMALLOC_MULTI_PAGES_ORDER
(1..9) rather than hardcoded to 2.  Or'ing __GFP_RECLAIM after
thp_shmem_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp) also overrides
defrag=never/defer and caller GFP restrictions, and placing it only in the
else branch leaves out the mpol path while causing shmem_fault() swapins
to behave differently between CONFIG_NUMA=y (where
shmem_get_pgoff_policy() returns a non-NULL policy) and CONFIG_NUMA=n
(where it returns NULL).

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261115024455.1465726-1-zhaonanzhe@xiaomi.com?part=4


^ permalink raw reply	[flat|nested] 8+ messages in thread

end of thread, other threads:[~2026-11-16  2:47 UTC | newest]

Thread overview: 8+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-15  2:44 [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages Nanzhe Zhao
2026-11-15  2:44 ` [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages Nanzhe Zhao
2026-11-16  2:46   ` sashiko-bot
2026-11-15  2:44 ` [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages Nanzhe Zhao
2026-11-16  2:46   ` sashiko-bot
2026-11-15  2:44 ` [RFC PATCH v4 3/4] zram: backend_zstd: use multi-page size for zstd parameter setup Nanzhe Zhao
2026-11-15  2:44 ` [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin Nanzhe Zhao
2026-11-16  2:46   ` sashiko-bot

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox