* [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages
@ 2026-11-15 2:44 Nanzhe Zhao
2026-11-15 2:44 ` [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages Nanzhe Zhao
` (4 more replies)
0 siblings, 5 replies; 9+ messages in thread
From: Nanzhe Zhao @ 2026-11-15 2:44 UTC (permalink / raw)
To: Andrew Morton
Cc: Nanzhe Zhao, linux-mm, linux-block, linux-kernel, Jens Axboe,
Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner,
Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li,
David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts,
Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee,
Nick Terrell, Barry Song, Tangquan Zheng, Bo Zhang
When large folios are compressed at a larger granularity, we observe
a notable reduction in CPU usage and a significant improvement in
compression ratios.
mTHP's ability to be swapped out without splitting and swapped back in
as a whole allows compression and decompression at larger granularities.
This patchset enhances zsmalloc and zram by adding support for dividing
large folios into multi-page blocks, typically configured with a
2-order granularity. Without this patchset, a large folio is always
divided into nr_pages 4KiB blocks.
The granularity can be set using the ZSMALLOC_MULTI_PAGES_ORDER
setting, where the default of 2 allows all anonymous THP to benefit.
Examples include:
* A 16KiB large folio will be compressed and stored as a single 16KiB
block.
* A 64KiB large folio will be compressed and stored as four 16KiB
blocks.
We captured anonymous pages from four Android applications on a Pixel 6.
To evaluate throughput, we allocate a buffer via mmap(MAP_PRIVATE |
MAP_ANONYMOUS), populate it with the captured pages, and invoke
madvise(MADV_PAGEOUT) to swap the buffer out to zram, measuring the
pageout duration. We then sequentially touch every 4 KiB page to fault
them back in from zram, measuring the swapin duration.
For example, testing the 143 MiB WeChat corpus (times in seconds;
w/o: without the patch, w/: with the patch):
Algorithm pageout w/o (s) pageout w/ swapin w/o (s) swapin w/
lz4 0.247 0.210 (-14.8%) 0.137 0.113 (-17.6%)
lzo-rle 0.240 0.207 (-13.7%) 0.149 0.123 (-17.5%)
zstd 0.567 0.452 (-20.4%) 0.258 0.192 (-25.6%)
v3 left two issues to discuss:
The first issue is internal fragmentation. A compressed 16 KiB block
is rounded up to the next size-class boundary. If it lands in a class
with a large gap, the gap becomes per-object padding.
On all four applications, unused pages in partially filled zspages
increase by ~1.8 to 2.4 MiB, and the per-object class gap is wider.
However, because the number of allocated objects drops by ~74% (from
four 4 KiB objects to one 16 KiB object), total rounding waste
actually falls by 13% to 31%. Combined with 5% to 15% better
compression ratios from multi-page compression, total zsmalloc memory
occupancy (mem_used_total) is reduced across all applications.
For example, below is the memory breakdown under lz4 for our four apps
(lzo-rle and zstd show the same trend, saving 3.8% to 9.3% total memory):
App objects (w/o -> w/) rounding waste (MiB) unused pages (MiB) mem_used_total (MiB)
WPS 27,530 -> 7,285 (-73.5%) 0.80 -> 0.56 (-29.8%) 1.07 -> 3.35 (+214.0%) 36.4 -> 34.9 (-4.0%)
WeChat 32,460 -> 8,450 (-74.0%) 0.89 -> 0.62 (-30.6%) 1.35 -> 3.32 (+145.8%) 42.8 -> 40.0 (-6.7%)
Douyin lite 62,668 -> 16,218 (-74.1%) 2.07 -> 1.61 (-22.3%) 1.37 -> 3.78 (+176.0%) 100.8 -> 93.6 (-7.1%)
JK chess 71,950 -> 18,584 (-74.2%) 1.75 -> 1.52 (-13.0%) 1.43 -> 3.28 (+129.7%) 117.0 -> 112.7 (-3.6%)
Here, objects is obj_allocated, rounding waste is total class padding
across all objects, unused pages is memory left in partially filled
zspages, and mem_used_total is the total memory allocated by zsmalloc
(including fragmentation and metadata).
We also built a 256 MiB lz4 synthetic extreme case: 16 KiB blocks with
the first K bytes random and the rest zero, with K tuned so the object
lands one byte above the previous class. Below is the code to generate
this pattern:
#define BLOCK 16384
static void make_block(unsigned char *buf, size_t k)
{
size_t i;
for (i = 0; i < k; i++)
buf[i] = rand();
memset(buf + k, 0, BLOCK - k);
}
static void write_corpus(int fd, size_t k, size_t nblocks)
{
unsigned char *buf = malloc(BLOCK);
size_t i;
make_block(buf, k);
for (i = 0; i < nblocks; i++)
write(fd, buf, BLOCK);
}
With the default CONFIG_ZSMALLOC_CHAIN_SIZE=8, the largest non-huge
class gap is 576 bytes. Setting K to 10,781 bytes lands the object in
class 11,456 with a 575 B/object padding, increasing total memory by
5.0% over the baseline without the patch (179.3 MiB vs 170.7 MiB).
With CONFIG_ZSMALLOC_CHAIN_SIZE=10, the classes are denser and the
largest gap drops to 448 bytes. At K=10,781 bytes, the memory overhead
falls to 3.9% (177.3 MiB vs 170.7 MiB).
A potential future optimization is during size-class merging: we can
avoid merging classes if the resulting gap exceeds a threshold (e.g.
ZS_MAX_MERGE_GAP = 256 bytes), for example:
--- a/mm/zsmalloc.c
+++ b/mm/zsmalloc.c
@@ can_merge() @@
+#define ZS_MAX_MERGE_GAP 256
+
+ if (prev->size - size > ZS_MAX_MERGE_GAP)
+ return false;
+
if (prev->zpdescs_per_zspage == zpdescs_per_zspage &&
prev->objs_per_zspage == objs_per_zspage)
return true;
The second issue is partial read. If the order-2 allocation for the
swapped-in folio fails and falls back to order-0, we must decompress
the entire 16 KiB block just for a 4 KiB swapin.
v3's 4/4 ("mm: fall back to four small folios if mTHP allocation
fails") handled this by building four small folios. Usama asked whether
defrag=always could replace this patch. But we only need direct reclaim
for the do_swap_page order-2 allocation; otherwise all anonymous mTHP
allocations would enter direct reclaim. So the new 4/4 adds
__GFP_RECLAIM to the order-2 swapin allocation in do_swap_page().
We measured the cost of entering direct reclaim in the swapin path on
an i7-14700HX with no_turbo=1: a 1 GiB memcg, zram (zstd), 16 KiB mTHP,
and an arm64 defconfig vmlinux build with -j20 in the background. The
bench was pinned to one CPU and the build to the rest. We compared
forcing direct reclaim with the existing behavior, 10 totals per case,
each total 30 rounds of 100 MiB anonymous pageout/swapin.
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <sys/mman.h>
#include <time.h>
static double now(void)
{
struct timespec ts;
clock_gettime(CLOCK_MONOTONIC, &ts);
return ts.tv_sec + ts.tv_nsec / 1e9;
}
int main(void)
{
const size_t size = 100UL << 20;
unsigned long *p, sum = 0;
size_t i, off;
double t0, t1;
p = mmap(NULL, size, PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS, -1, 0);
if (p == MAP_FAILED) {
perror("mmap");
return 1;
}
for (i = 0; i < size / sizeof(*p); i++)
p[i] = i & 0xfff;
t0 = now();
if (madvise(p, size, MADV_PAGEOUT)) {
perror("madvise(MADV_PAGEOUT)");
return 1;
}
t1 = now();
printf("pageout %.6f\n", t1 - t0);
t0 = now();
for (off = 0; off < size; off += 4096)
sum += *(volatile unsigned long *)((char *)p + off);
t1 = now();
printf("swapin %.6f sum %lu\n", t1 - t0, sum);
return 0;
}
w/o DR DR
partial reads 235,832 218
swapin total (s) 24.85 +/- 0.56 22.80 +/- 0.73
pswpout (pages) 11.4M 11.3M
Here, partial reads is the zram_bio_read_multi_pages_partial counter.
The direct-reclaim case did not show a measurable cost. Almost all
partial decompressions were eliminated, making swapin 8.3% faster.
-v4:
* Rebased onto latest akpm mm-new.
* Patch 1: Fixed multi-page page accounting (NR_ZSPAGES, zs_compact,
and debugfs) to use physical pages via class_pages_per_zspage().
* Patch 2: Adapted to upstream's new zcomp read/write stream separation;
added a preallocated decomp_buf in zcomp_dstrm for partial reads to
avoid memory allocation in the swapin path.
* Patch 3: Updated zstd backend to pass ZCOMP_MULTI_PAGES_SIZE to
zstd_setup_params().
* Patch 4: Dropped v3's 4/4 fallback and added a new patch to allow
direct reclaim for synchronous multi-page swapin.
[1] v3: https://lore.kernel.org/all/20241121222521.83458-1-21cnbao@gmail.com/
Barry Song (1):
zram: backend_zstd: use multi-page size for zstd parameter setup
Nanzhe Zhao (1):
mm: swap: allow direct reclaim for synchronous multi-page swapin
Tangquan Zheng (2):
mm: zsmalloc: support objects compressed based on multiple pages
zram: support compression at the granularity of multi-pages
drivers/block/zram/Kconfig | 9 +
drivers/block/zram/backend_zstd.c | 4 +-
drivers/block/zram/zcomp.c | 31 +-
drivers/block/zram/zcomp.h | 19 +-
drivers/block/zram/zram_drv.c | 499 +++++++++++++++++++++++++++---
drivers/block/zram/zram_drv.h | 43 +++
include/linux/zsmalloc.h | 10 +-
mm/Kconfig | 29 +-
mm/swap_state.c | 7 +
mm/zsmalloc.c | 242 ++++++++++-----
10 files changed, 757 insertions(+), 136 deletions(-)
--
2.43.0
^ permalink raw reply [flat|nested] 9+ messages in thread* [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages 2026-11-15 2:44 [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages Nanzhe Zhao @ 2026-11-15 2:44 ` Nanzhe Zhao 2026-11-16 2:46 ` sashiko-bot 2026-11-15 2:44 ` [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages Nanzhe Zhao ` (3 subsequent siblings) 4 siblings, 1 reply; 9+ messages in thread From: Nanzhe Zhao @ 2026-11-15 2:44 UTC (permalink / raw) To: Andrew Morton Cc: Tangquan Zheng, linux-mm, linux-block, linux-kernel, Jens Axboe, Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner, Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li, David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts, Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee, Nick Terrell, Barry Song, Bo Zhang, Nanzhe Zhao From: Tangquan Zheng <zhengtangquan@oppo.com> This patch introduces support for zsmalloc to store compressed objects based on multi-pages. Previously, a large folio with nr_pages subpages would undergo compression one by one, each at the granularity of PAGE_SIZE. However, by compressing them at a larger granularity, we can conserve both memory and CPU resources. We define the granularity with a configuration option called ZSMALLOC_MULTI_PAGES_ORDER, set to a default value of 2, which matches the minimum order of anonymous mTHP. As a result, a large folio with 8 subpages will now be split into 2 parts instead of 8. The introduction of the multi-pages feature necessitates the creation of new size classes to accommodate it. Signed-off-by: Tangquan Zheng <zhengtangquan@oppo.com> Co-developed-by: Barry Song <baohua@kernel.org> Signed-off-by: Barry Song <baohua@kernel.org> Co-developed-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com> Signed-off-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com> --- drivers/block/zram/zram_drv.c | 3 +- include/linux/zsmalloc.h | 10 +- mm/Kconfig | 29 +++- mm/zsmalloc.c | 245 +++++++++++++++++++++++----------- 4 files changed, 204 insertions(+), 83 deletions(-) diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 4f479d905142..24eefcf143de 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -2024,7 +2024,8 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) } if (!huge_class_size) - huge_class_size = zs_huge_class_size(zram->mem_pool); + huge_class_size = zs_huge_class_size(zram->mem_pool, + ZSMALLOC_TYPE_BASEPAGE); lockdep_register_key(&zram->table_lock_key); lockdep_init_map(&zram->table_lock_map, "zram->table[index].lock", &zram->table_lock_key, 0); diff --git a/include/linux/zsmalloc.h b/include/linux/zsmalloc.h index 8971eef333ba..738260358a9a 100644 --- a/include/linux/zsmalloc.h +++ b/include/linux/zsmalloc.h @@ -17,6 +17,14 @@ #include <linux/compiler.h> #include <linux/types.h> +enum zsmalloc_type { + ZSMALLOC_TYPE_BASEPAGE, +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES + ZSMALLOC_TYPE_MULTI_PAGES, +#endif + ZSMALLOC_TYPE_MAX, +}; + struct zs_pool_stats { /* How many pages were migrated (freed) */ atomic_long_t pages_compacted; @@ -55,7 +63,7 @@ void zs_free(struct zs_pool *pool, struct zs_handle *handle); void zs_rebind_handle(struct zs_pool *pool, struct zs_handle *dst, struct zs_handle *src); -size_t zs_huge_class_size(struct zs_pool *pool); +size_t zs_huge_class_size(struct zs_pool *pool, enum zsmalloc_type type); unsigned long zs_get_total_pages(struct zs_pool *pool); unsigned long zs_compact(struct zs_pool *pool); diff --git a/mm/Kconfig b/mm/Kconfig index afff1b25b543..b68720f6fb6b 100644 --- a/mm/Kconfig +++ b/mm/Kconfig @@ -140,14 +140,15 @@ config ZSMALLOC_STAT If unsure, say N. config ZSMALLOC_CHAIN_SIZE - int "Maximum number of physical pages per-zspage" + int "Maximum number of chain elements per-zspage" default 8 range 4 16 help - This option sets the upper limit on the number of physical pages - that a zmalloc page (zspage) can consist of. The optimal zspage - chain size is calculated for each size class during the - initialization of the pool. + This option sets the upper limit on the number of chain elements + that a zsmalloc page (zspage) can consist of. Each element is a + single page, or a multi-page allocation when ZSMALLOC_MULTI_PAGES + is enabled. The optimal zspage chain size is calculated for each + size class during the initialization of the pool. Changing this option can alter the characteristics of size classes, such as the number of pages per zspage and the number of objects @@ -157,6 +158,24 @@ config ZSMALLOC_CHAIN_SIZE For more information, see zsmalloc documentation. +config ZSMALLOC_MULTI_PAGES + bool "support zsmalloc multiple pages" + depends on ZSMALLOC && !HIGHMEM + help + This option configures zsmalloc to support allocations larger than + PAGE_SIZE, enabling compression across multiple pages. The size of + these multiple pages is determined by the configured + ZSMALLOC_MULTI_PAGES_ORDER. + +config ZSMALLOC_MULTI_PAGES_ORDER + int "zsmalloc multiple pages order" + default 2 + range 1 9 + depends on ZSMALLOC_MULTI_PAGES + help + This option is used to configure zsmalloc to support the compression + of multiple pages. + endmenu menu "Slab allocator options" diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 4a6ec43f418b..2d83fcc8610d 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -48,6 +48,12 @@ #define ZSPAGE_MAGIC 0x58 +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES +#define ZSMALLOC_MULTI_PAGES_ORDER (_AC(CONFIG_ZSMALLOC_MULTI_PAGES_ORDER, UL)) +#define ZSMALLOC_MULTI_PAGES_NR (1 << ZSMALLOC_MULTI_PAGES_ORDER) +#define ZSMALLOC_MULTI_PAGES_SIZE (PAGE_SIZE * ZSMALLOC_MULTI_PAGES_NR) +#endif + /* * This must be power of 2 and greater than or equal to sizeof(link_free). * These two conditions ensure that any 'struct link_free' itself doesn't @@ -114,7 +120,7 @@ #define HUGE_BITS 1 #define FULLNESS_BITS 4 -#define CLASS_BITS 8 +#define CLASS_BITS 9 #define MAGIC_VAL_BITS 8 #define ZS_MAX_PAGES_PER_ZSPAGE (_AC(CONFIG_ZSMALLOC_CHAIN_SIZE, UL)) @@ -172,7 +178,11 @@ static_assert(ZS_OBJ_IDX_BITS >= ZS_PAGES_PER_ZSPAGE_BITS, "zsmalloc: ZS_MIN_ALLOC_SIZE would exceed ZS_MAX_ALLOC_SIZE"); /* each chunk includes extra space to keep handle */ +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES +#define ZS_MAX_ALLOC_SIZE ZSMALLOC_MULTI_PAGES_SIZE +#else #define ZS_MAX_ALLOC_SIZE PAGE_SIZE +#endif /* * On systems with 4K page size, this gives 255 size classes! There is a @@ -187,14 +197,45 @@ static_assert(ZS_OBJ_IDX_BITS >= ZS_PAGES_PER_ZSPAGE_BITS, * ZS_MIN_ALLOC_SIZE and ZS_SIZE_CLASS_DELTA must be multiple of ZS_ALIGN * (reason above) */ -#define ZS_SIZE_CLASS_DELTA (PAGE_SIZE >> CLASS_BITS) +#define ZS_SIZE_CLASS_DELTA (PAGE_SIZE >> (CLASS_BITS - 1)) #define _ZS_MIN_ALLOC_RAW \ DIV_ROUND_UP(ZS_MAX_PAGES_PER_ZSPAGE << PAGE_SHIFT, ZS_OBJ_IDX_MASK) #define ZS_MIN_ALLOC_SIZE \ MIN(ZS_MAX_ALLOC_SIZE, \ MAX(32UL, ALIGN(_ZS_MIN_ALLOC_RAW, ZS_ALIGN))) -#define ZS_SIZE_CLASSES (DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - ZS_MIN_ALLOC_SIZE, \ - ZS_SIZE_CLASS_DELTA) + 1) + +#define ZS_PAGE_SIZE_CLASSES (DIV_ROUND_UP(PAGE_SIZE - ZS_MIN_ALLOC_SIZE, \ + ZS_SIZE_CLASS_DELTA) + 1) + +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES +#define ZS_MULTI_PAGES_SIZE_CLASS_DELTA (ZSMALLOC_MULTI_PAGES_SIZE >> (CLASS_BITS - 1)) +#define ZS_MULTI_PAGES_SIZE_CLASSES (DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - PAGE_SIZE, \ + ZS_MULTI_PAGES_SIZE_CLASS_DELTA) + 1) +#define ZS_SIZE_CLASSES (ZS_PAGE_SIZE_CLASSES + ZS_MULTI_PAGES_SIZE_CLASSES) +#else +#define ZS_SIZE_CLASSES (DIV_ROUND_UP(ZS_MAX_ALLOC_SIZE - ZS_MIN_ALLOC_SIZE, \ + ZS_SIZE_CLASS_DELTA) + 1) +#endif + +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES +static inline unsigned int class_size_to_zs_order(unsigned long size) +{ + if (size > PAGE_SIZE) + return ZSMALLOC_MULTI_PAGES_ORDER; + + return 0; +} +#else +static inline unsigned int class_size_to_zs_order(unsigned long size) +{ + return 0; +} +#endif + +static inline unsigned long class_size_to_zs_size(unsigned long size) +{ + return PAGE_SIZE * (1 << class_size_to_zs_order(size)); +} /* * Pages are distinguished by the ratio of used memory (that is the ratio @@ -230,7 +271,7 @@ struct zs_size_stat { static struct dentry *zs_stat_root; #endif -static size_t huge_class_size; +static size_t huge_class_size[ZSMALLOC_TYPE_MAX]; struct size_class { spinlock_t lock; @@ -241,13 +282,20 @@ struct size_class { */ int size; int objs_per_zspage; - /* Number of PAGE_SIZE sized pages to combine to form a 'zspage' */ - int pages_per_zspage; + /* Number of zpdescs to combine to form a 'zspage' */ + int zpdescs_per_zspage; unsigned int index; struct zs_size_stat stats; }; +/* Number of physical pages in a zspage */ +static inline unsigned long class_pages_per_zspage(const struct size_class *class) +{ + return (unsigned long)class->zpdescs_per_zspage << + class_size_to_zs_order(class->size); +} + /* * Placed within free objects to form a singly linked list. * For every zspage, zspage->freeobj gives head of this list. @@ -295,27 +343,28 @@ struct zs_pool { static inline void zpdesc_inc_zone_page_state(struct zpdesc *zpdesc) { - inc_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES); + __zone_stat_add_folio(zpdesc_folio(zpdesc), NR_ZSPAGES); } static inline void zpdesc_dec_zone_page_state(struct zpdesc *zpdesc) { - dec_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES); + __zone_stat_sub_folio(zpdesc_folio(zpdesc), NR_ZSPAGES); } -static inline struct zpdesc *alloc_zpdesc(gfp_t gfp, const int nid) +static inline struct zpdesc *alloc_zpdesc(gfp_t gfp, const int nid, + unsigned int order) { - struct page *page = alloc_pages_node(nid, gfp, 0); + struct page *page = alloc_pages_node(nid, gfp | __GFP_COMP, order); return page_zpdesc(page); } -static inline void free_zpdesc(struct zpdesc *zpdesc) +static inline void free_zpdesc(struct zpdesc *zpdesc, unsigned int order) { struct page *page = zpdesc_page(zpdesc); /* PageZsmalloc is sticky until the page is freed to the buddy. */ - __free_page(page); + __free_pages(page, order); } #define ZS_PAGE_UNLOCKED 0 @@ -420,7 +469,7 @@ static void zspage_write_unlock(struct zspage *zspage) spin_unlock(&zsl->lock); } -/* huge object: pages_per_zspage == 1 && maxobj_per_zspage == 1 */ +/* huge object: zpdescs_per_zspage == 1 && maxobj_per_zspage == 1 */ static void SetZsHugePage(struct zspage *zspage) { zspage->huge = 1; @@ -535,6 +584,15 @@ static int get_size_class_index(int size) { int idx = 0; +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES + if (size > PAGE_SIZE + ZS_HANDLE_SIZE) { + idx = ZS_PAGE_SIZE_CLASSES; + idx += DIV_ROUND_UP(size - PAGE_SIZE, + ZS_MULTI_PAGES_SIZE_CLASS_DELTA); + return min_t(int, ZS_SIZE_CLASSES - 1, idx); + } +#endif + if (likely(size > ZS_MIN_ALLOC_SIZE)) idx = DIV_ROUND_UP(size - ZS_MIN_ALLOC_SIZE, ZS_SIZE_CLASS_DELTA); @@ -594,11 +652,11 @@ static int zs_stats_size_show(struct seq_file *s, void *v) unsigned long total_freeable = 0; unsigned long inuse_totals[NR_FULLNESS_GROUPS] = {0, }; - seq_printf(s, " %5s %5s %9s %9s %9s %9s %9s %9s %9s %9s %9s %9s %9s %13s %10s %10s %16s %8s\n", - "class", "size", "10%", "20%", "30%", "40%", + seq_printf(s, " %5s %5s %5s %9s %9s %9s %9s %9s %9s %9s %9s %9s %9s %9s %13s %10s %10s %16s %16s %8s\n", + "class", "size", "order", "10%", "20%", "30%", "40%", "50%", "60%", "70%", "80%", "90%", "99%", "100%", "obj_allocated", "obj_used", "pages_used", - "pages_per_zspage", "freeable"); + "pages_per_zspage", "objs_per_zspage", "freeable"); for (i = 0; i < ZS_SIZE_CLASSES; i++) { @@ -609,7 +667,8 @@ static int zs_stats_size_show(struct seq_file *s, void *v) spin_lock(&class->lock); - seq_printf(s, " %5u %5u ", i, class->size); + seq_printf(s, " %5u %5u %5u", i, class->size, + class_size_to_zs_order(class->size)); for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) { inuse_totals[fg] += class_stat_read(class, fg); seq_printf(s, "%9lu ", class_stat_read(class, fg)); @@ -622,11 +681,12 @@ static int zs_stats_size_show(struct seq_file *s, void *v) objs_per_zspage = class->objs_per_zspage; pages_used = obj_allocated / objs_per_zspage * - class->pages_per_zspage; + class_pages_per_zspage(class); - seq_printf(s, "%13lu %10lu %10lu %16d %8lu\n", + seq_printf(s, "%13lu %10lu %10lu %16lu %16d %8lu\n", obj_allocated, obj_used, pages_used, - class->pages_per_zspage, freeable); + class_pages_per_zspage(class), objs_per_zspage, + freeable); total_objs += obj_allocated; total_used_objs += obj_used; @@ -634,13 +694,13 @@ static int zs_stats_size_show(struct seq_file *s, void *v) total_freeable += freeable; } - seq_printf(s, "\n %5s %5s ", "Total", ""); + seq_printf(s, "\n %5s %5s %5s ", "Total", "", ""); for (fg = ZS_INUSE_RATIO_10; fg < NR_FULLNESS_GROUPS; fg++) seq_printf(s, "%9lu ", inuse_totals[fg]); - seq_printf(s, "%13lu %10lu %10lu %16s %8lu\n", - total_objs, total_used_objs, total_pages, "", + seq_printf(s, "%13lu %10lu %10lu %16s %16s %8lu\n", + total_objs, total_used_objs, total_pages, "", "", total_freeable); return 0; @@ -915,7 +975,7 @@ static void __free_zspage(struct zs_pool *pool, struct size_class *class, assert_spin_locked(&class->lock); __free_zspage_lockless(zspage); class_stat_sub(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage); - atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated); + atomic_long_sub(class_pages_per_zspage(class), &pool->pages_allocated); } static void free_zspage(struct zs_pool *pool, struct size_class *class, @@ -943,6 +1003,7 @@ static void init_zspage(struct size_class *class, struct zspage *zspage) { unsigned int freeobj = 1; unsigned long off = 0; + unsigned long page_size = class_size_to_zs_size(class->size); struct zpdesc *zpdesc = get_first_zpdesc(zspage); while (zpdesc) { @@ -955,7 +1016,7 @@ static void init_zspage(struct size_class *class, struct zspage *zspage) vaddr = kmap_local_zpdesc(zpdesc); link = (struct link_free *)vaddr + off / sizeof(*link); - while ((off += class->size) < PAGE_SIZE) { + while ((off += class->size) < page_size) { link->next = freeobj++ << OBJ_TAG_BITS; link += class->size / sizeof(*link); } @@ -977,7 +1038,7 @@ static void init_zspage(struct size_class *class, struct zspage *zspage) } kunmap_local(vaddr); zpdesc = next_zpdesc; - off %= PAGE_SIZE; + off %= page_size; } set_freeobj(zspage, 0); @@ -989,7 +1050,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage, int i; struct zpdesc *zpdesc; struct zpdesc *prev_zpdesc = NULL; - int nr_zpdescs = class->pages_per_zspage; + int nr_zpdescs = class->zpdescs_per_zspage; /* * Allocate individual pages and link them together as: @@ -1006,7 +1067,7 @@ static void create_page_chain(struct size_class *class, struct zspage *zspage, if (i == 0) { zspage->first_zpdesc = zpdesc; if (unlikely(class->objs_per_zspage == 1 && - class->pages_per_zspage == 1)) + class->zpdescs_per_zspage == 1)) SetZsHugePage(zspage); } else { prev_zpdesc->next = zpdesc; @@ -1023,6 +1084,7 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, gfp_t gfp, const int nid) { int i; + unsigned int order = class_size_to_zs_order(class->size); struct zpdesc *zpdescs[ZS_MAX_PAGES_PER_ZSPAGE]; struct zspage *zspage = cache_alloc_zspage(gfp); @@ -1032,19 +1094,24 @@ static struct zspage *alloc_zspage(struct zs_pool *pool, if (!IS_ENABLED(CONFIG_COMPACTION)) gfp &= ~__GFP_MOVABLE; + /* + * Multi-page zspages are not movable, keep them out of ZONE_MOVABLE. + */ + if (order) + gfp &= ~__GFP_MOVABLE; + zspage->magic = ZSPAGE_MAGIC; zspage->pool = pool; zspage->class = class->index; zspage_lock_init(zspage); - for (i = 0; i < class->pages_per_zspage; i++) { - struct zpdesc *zpdesc; + for (i = 0; i < class->zpdescs_per_zspage; i++) { + struct zpdesc *zpdesc = alloc_zpdesc(gfp, nid, order); - zpdesc = alloc_zpdesc(gfp, nid); if (!zpdesc) { while (--i >= 0) { zpdesc_dec_zone_page_state(zpdescs[i]); - free_zpdesc(zpdescs[i]); + free_zpdesc(zpdescs[i], order); } cache_free_zspage(zspage); return NULL; @@ -1076,10 +1143,10 @@ static struct zspage *find_get_zspage(struct size_class *class) return zspage; } -static bool can_merge(struct size_class *prev, int pages_per_zspage, +static bool can_merge(struct size_class *prev, int zpdescs_per_zspage, int objs_per_zspage) { - if (prev->pages_per_zspage == pages_per_zspage && + if (prev->zpdescs_per_zspage == zpdescs_per_zspage && prev->objs_per_zspage == objs_per_zspage) return true; @@ -1128,7 +1195,7 @@ void zs_obj_read_sg_begin(struct zs_pool *pool, const struct zs_handle *handle, { struct zspage *zspage; struct zpdesc *zpdesc; - unsigned long obj, off; + unsigned long obj, off, page_size; unsigned int obj_idx; struct size_class *class; @@ -1143,12 +1210,13 @@ void zs_obj_read_sg_begin(struct zs_pool *pool, const struct zs_handle *handle, read_unlock(&pool->lock); class = zspage_class(pool, zspage); - off = offset_in_page(class->size * obj_idx); + page_size = class_size_to_zs_size(class->size); + off = (class->size * obj_idx) & (page_size - 1); if (!ZsHugePage(zspage)) off += ZS_HANDLE_SIZE; - if (off + mem_len <= PAGE_SIZE) { + if (off + mem_len <= page_size) { /* this object is contained entirely within a page */ sg_init_table(sg, 1); sg_set_page(sg, zpdesc_page(zpdesc), mem_len, off); @@ -1156,7 +1224,7 @@ void zs_obj_read_sg_begin(struct zs_pool *pool, const struct zs_handle *handle, size_t sizes[2]; /* this object spans two pages */ - sizes[0] = PAGE_SIZE - off; + sizes[0] = page_size - off; sizes[1] = mem_len - sizes[0]; sg_init_table(sg, 2); @@ -1189,16 +1257,17 @@ static void obj_write(struct size_class *class, struct zspage *zspage, unsigned long obj, const void *src, size_t size) { struct zpdesc *zpdesc; + unsigned long page_size = class_size_to_zs_size(class->size); unsigned long off; unsigned int obj_idx; obj_to_location(obj, &zpdesc, &obj_idx); - off = offset_in_page(class->size * obj_idx); + off = (class->size * obj_idx) & (page_size - 1); if (!ZsHugePage(zspage)) off += ZS_HANDLE_SIZE; - if (off + size <= PAGE_SIZE) { + if (off + size <= page_size) { /* this object is contained entirely within a page */ void *dst = kmap_local_zpdesc(zpdesc); @@ -1208,7 +1277,7 @@ static void obj_write(struct size_class *class, struct zspage *zspage, /* this object spans two pages */ size_t sizes[2]; - sizes[0] = PAGE_SIZE - off; + sizes[0] = page_size - off; sizes[1] = size - sizes[0]; memcpy_to_page(zpdesc_page(zpdesc), off, src, sizes[0]); @@ -1230,9 +1299,9 @@ static void obj_write(struct size_class *class, struct zspage *zspage, * * Return: the size (in bytes) of the first huge zsmalloc &size_class. */ -size_t zs_huge_class_size(struct zs_pool *pool) +size_t zs_huge_class_size(struct zs_pool *pool, enum zsmalloc_type type) { - return huge_class_size; + return huge_class_size[type]; } EXPORT_SYMBOL_GPL(zs_huge_class_size); @@ -1245,15 +1314,17 @@ static unsigned long obj_malloc(struct zs_pool *pool, struct size_class *class; struct zpdesc *m_zpdesc; - unsigned long m_offset; + unsigned long m_offset, page_size, page_shift; void *vaddr; class = pool->size_class[zspage->class]; obj = get_freeobj(zspage); + page_size = class_size_to_zs_size(class->size); + page_shift = PAGE_SHIFT + class_size_to_zs_order(class->size); offset = obj * class->size; - nr_zpdesc = offset >> PAGE_SHIFT; - m_offset = offset_in_page(offset); + nr_zpdesc = offset >> page_shift; + m_offset = offset & (page_size - 1); m_zpdesc = get_first_zpdesc(zspage); for (i = 0; i < nr_zpdesc; i++) @@ -1348,12 +1419,13 @@ int zs_store(struct zs_pool *pool, struct zs_handle *handle, const void *src, spin_lock(&class->lock); newfg = get_fullness_group(class, zspage); insert_zspage(class, zspage, newfg); - atomic_long_add(class->pages_per_zspage, &pool->pages_allocated); + atomic_long_add(class_pages_per_zspage(class), &pool->pages_allocated); class_stat_add(class, ZS_OBJS_ALLOCATED, class->objs_per_zspage); class_stat_add(class, ZS_OBJS_INUSE, 1); /* We completely set up zspage so mark them as movable */ - SetZsPageMovable(pool, zspage); + if (class_size_to_zs_order(class->size) == 0) + SetZsPageMovable(pool, zspage); spin_unlock(&class->lock); return 0; @@ -1365,13 +1437,15 @@ static void obj_free(int class_size, unsigned long obj) struct link_free *link; struct zspage *zspage; struct zpdesc *f_zpdesc; - unsigned long f_offset; + unsigned long f_offset, page_size, page_mask; unsigned int f_objidx; void *vaddr; obj_to_location(obj, &f_zpdesc, &f_objidx); - f_offset = offset_in_page(class_size * f_objidx); + page_size = class_size_to_zs_size(class_size); + page_mask = ~(page_size - 1); + f_offset = (class_size * f_objidx) & ~page_mask; zspage = get_zspage(f_zpdesc); vaddr = kmap_local_zpdesc(f_zpdesc); @@ -1482,7 +1556,7 @@ void zs_free(struct zs_pool *pool, struct zs_handle *handle) if (zspage_to_free) { __free_zspage_lockless(zspage_to_free); - atomic_long_sub(class->pages_per_zspage, &pool->pages_allocated); + atomic_long_sub(class_pages_per_zspage(class), &pool->pages_allocated); } } EXPORT_SYMBOL_GPL(zs_free); @@ -1540,6 +1614,8 @@ static void zs_object_copy(struct size_class *class, unsigned long dst, struct zpdesc *s_zpdesc, *d_zpdesc; unsigned int s_objidx, d_objidx; unsigned long s_off, d_off; + unsigned long page_size = class_size_to_zs_size(class->size); + unsigned long page_mask = ~(page_size - 1); void *s_addr, *d_addr; int s_size, d_size, size; int written = 0; @@ -1549,14 +1625,14 @@ static void zs_object_copy(struct size_class *class, unsigned long dst, obj_to_location(src, &s_zpdesc, &s_objidx); obj_to_location(dst, &d_zpdesc, &d_objidx); - s_off = offset_in_page(class->size * s_objidx); - d_off = offset_in_page(class->size * d_objidx); + s_off = (class->size * s_objidx) & ~page_mask; + d_off = (class->size * d_objidx) & ~page_mask; - if (s_off + class->size > PAGE_SIZE) - s_size = PAGE_SIZE - s_off; + if (s_off + class->size > page_size) + s_size = page_size - s_off; - if (d_off + class->size > PAGE_SIZE) - d_size = PAGE_SIZE - d_off; + if (d_off + class->size > page_size) + d_size = page_size - d_off; s_addr = kmap_local_zpdesc(s_zpdesc); d_addr = kmap_local_zpdesc(d_zpdesc); @@ -1581,7 +1657,7 @@ static void zs_object_copy(struct size_class *class, unsigned long dst, * kunmap_local(d_addr). For more details see * Documentation/mm/highmem.rst. */ - if (s_off >= PAGE_SIZE) { + if (s_off >= page_size) { kunmap_local(d_addr); kunmap_local(s_addr); s_zpdesc = get_next_zpdesc(s_zpdesc); @@ -1591,7 +1667,7 @@ static void zs_object_copy(struct size_class *class, unsigned long dst, s_off = 0; } - if (d_off >= PAGE_SIZE) { + if (d_off >= page_size) { kunmap_local(d_addr); d_zpdesc = get_next_zpdesc(d_zpdesc); d_addr = kmap_local_zpdesc(d_zpdesc); @@ -1615,13 +1691,13 @@ static struct zs_handle *find_alloced_obj(struct size_class *class, int index = *obj_idx; struct zs_handle *handle = NULL; void *addr = kmap_local_zpdesc(zpdesc); + unsigned long page_size = class_size_to_zs_size(class->size); offset = get_first_obj_offset(zpdesc); offset += class->size * index; - while (offset < PAGE_SIZE) { - handle = obj_allocated(zpdesc, addr + offset); - if (handle) + while (offset < page_size) { + if (obj_allocated(zpdesc, addr + offset, &handle)) break; offset += class->size; @@ -2002,7 +2078,7 @@ static unsigned long zs_can_compact(struct size_class *class) obj_wasted = obj_allocated - obj_used; obj_wasted /= class->objs_per_zspage; - return obj_wasted * class->pages_per_zspage; + return obj_wasted * class_pages_per_zspage(class); } static unsigned long __zs_compact(struct zs_pool *pool, @@ -2041,7 +2117,7 @@ static unsigned long __zs_compact(struct zs_pool *pool, fg = putback_zspage(class, src_zspage); if (fg == ZS_INUSE_RATIO_0) { free_zspage(pool, class, src_zspage); - pages_freed += class->pages_per_zspage; + pages_freed += class_pages_per_zspage(class); } src_zspage = NULL; @@ -2164,6 +2240,7 @@ static int calculate_zspage_chain_size(int class_size) { int i, min_waste = INT_MAX; int chain_size = 1; + unsigned long page_size = class_size_to_zs_size(class_size); if (is_power_of_2(class_size)) return chain_size; @@ -2171,7 +2248,7 @@ static int calculate_zspage_chain_size(int class_size) for (i = 1; i <= ZS_MAX_PAGES_PER_ZSPAGE; i++) { int waste; - waste = (i * PAGE_SIZE) % class_size; + waste = (i * page_size) % class_size; if (waste < min_waste) { min_waste = waste; chain_size = i; @@ -2215,16 +2292,32 @@ struct zs_pool *zs_create_pool(const char *name) */ for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) { int size; - int pages_per_zspage; + int zpdescs_per_zspage; int objs_per_zspage; struct size_class *class; int fullness; + int idx = ZSMALLOC_TYPE_BASEPAGE; +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES + int order = 0; +#endif - size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA; + if (i < ZS_PAGE_SIZE_CLASSES) + size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA; +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES + if (i >= ZS_PAGE_SIZE_CLASSES) + size = PAGE_SIZE + (i - ZS_PAGE_SIZE_CLASSES) * + ZS_MULTI_PAGES_SIZE_CLASS_DELTA; +#endif if (size > ZS_MAX_ALLOC_SIZE) size = ZS_MAX_ALLOC_SIZE; - pages_per_zspage = calculate_zspage_chain_size(size); - objs_per_zspage = pages_per_zspage * PAGE_SIZE / size; +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES + order = class_size_to_zs_order(size); + if (order == ZSMALLOC_MULTI_PAGES_ORDER) + idx = ZSMALLOC_TYPE_MULTI_PAGES; +#endif + zpdescs_per_zspage = calculate_zspage_chain_size(size); + objs_per_zspage = zpdescs_per_zspage * + class_size_to_zs_size(size) / size; /* * We iterate from biggest down to smallest classes, @@ -2232,9 +2325,9 @@ struct zs_pool *zs_create_pool(const char *name) * class. Any object bigger than or equal to that will * endup in the huge class. */ - if (pages_per_zspage != 1 && objs_per_zspage != 1 && - !huge_class_size) { - huge_class_size = size; + if (zpdescs_per_zspage != 1 && objs_per_zspage != 1 && + !huge_class_size[idx]) { + huge_class_size[idx] = size; /* * The object uses ZS_HANDLE_SIZE bytes to store the * handle. We need to subtract it, because zs_store() @@ -2244,7 +2337,7 @@ struct zs_pool *zs_create_pool(const char *name) * class because it grows by ZS_HANDLE_SIZE extra bytes * right before class lookup. */ - huge_class_size -= (ZS_HANDLE_SIZE - 1); + huge_class_size[idx] -= (ZS_HANDLE_SIZE - 1); } /* @@ -2257,7 +2350,7 @@ struct zs_pool *zs_create_pool(const char *name) * previous size_class if possible. */ if (prev_class) { - if (can_merge(prev_class, pages_per_zspage, objs_per_zspage)) { + if (can_merge(prev_class, zpdescs_per_zspage, objs_per_zspage)) { pool->size_class[i] = prev_class; continue; } @@ -2269,7 +2362,7 @@ struct zs_pool *zs_create_pool(const char *name) class->size = size; class->index = i; - class->pages_per_zspage = pages_per_zspage; + class->zpdescs_per_zspage = zpdescs_per_zspage; class->objs_per_zspage = objs_per_zspage; spin_lock_init(&class->lock); pool->size_class[i] = class; -- 2.43.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages 2026-11-15 2:44 ` [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages Nanzhe Zhao @ 2026-11-16 2:46 ` sashiko-bot 0 siblings, 0 replies; 9+ messages in thread From: sashiko-bot @ 2026-11-16 2:46 UTC (permalink / raw) To: Nanzhe Zhao, Andrew Morton Cc: Tangquan Zheng, linux-mm, Jens Axboe, Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner, Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li, David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts, Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee, Nick Terrell, Barry Song, Bo Zhang Thank you for your contribution! Sashiko AI review found 8 potential issues to consider: - [High] mm: zsmalloc: build failure in find_alloced_obj() calling obj_allocated() - [High] mm: zsmalloc: VM_BUG_ON in obj_write() calling single-page memcpy_to_page() - [High] mm: zsmalloc: memory corruption in zs_rebind_handle() using offset_in_page() - [High] mm: zsmalloc: unclamped basepage size class in zs_create_pool() - [Medium] mm: zsmalloc: 24-bit offset overflow in set_first_obj_offset() on 64K pages - [Medium] mm: zsmalloc: preemptible __zone_stat_add_folio() in zpdesc state helpers - [Low] mm: zsmalloc: missing @type kernel-doc in zs_huge_class_size() - [Low] mm: zsmalloc: missing space after order column in zs_stats_size_show() -- commit de88105665ba4d7f0292945bc4a5ae1786f09efe Author: Tangquan Zheng <zhengtangquan@oppo.com> mm: zsmalloc: support objects compressed based on multiple pages This commit adds multi-page object support to zsmalloc under CONFIG_ZSMALLOC_MULTI_PAGES, introducing compound zpdesc allocations and size classes above PAGE_SIZE up to ZSMALLOC_MULTI_PAGES_ORDER. > diff --git a/mm/Kconfig b/mm/Kconfig > index afff1b25b5438..b68720f6fb6bd 100644 > --- a/mm/Kconfig > +++ b/mm/Kconfig > @@ -157,6 +158,24 @@ config ZSMALLOC_CHAIN_SIZE > > For more information, see zsmalloc documentation. > > +config ZSMALLOC_MULTI_PAGES > + bool "support zsmalloc multiple pages" > + depends on ZSMALLOC && !HIGHMEM > + help > + This option configures zsmalloc to support allocations larger than > + PAGE_SIZE, enabling compression across multiple pages. The size of > + these multiple pages is determined by the configured > + ZSMALLOC_MULTI_PAGES_ORDER. > + > +config ZSMALLOC_MULTI_PAGES_ORDER > + int "zsmalloc multiple pages order" > + default 2 > + range 1 9 > + depends on ZSMALLOC_MULTI_PAGES > + help > + This option is used to configure zsmalloc to support the compression > + of multiple pages. > + > endmenu [Severity: Medium] Can ZSMALLOC_MULTI_PAGES_ORDER=9 overflow the 24-bit first_obj_offset field on 64 KiB page systems? With 64 KiB pages, order 9 gives a 32 MiB zpdesc, while set_first_obj_offset() masks offsets with FIRST_OBJ_PAGE_TYPE_MASK (16 MiB) and only checks BUILD_BUG_ON(PAGE_SIZE > SZ_16M): mm/zsmalloc.c:set_first_obj_offset() { /* With 24 bits available, we can support offsets into 16 MiB pages. */ BUILD_BUG_ON(PAGE_SIZE > SZ_16M); VM_WARN_ON_ONCE(!PageZsmalloc(zpdesc_page(zpdesc))); VM_WARN_ON_ONCE(offset & ~FIRST_OBJ_PAGE_TYPE_MASK); zpdesc->first_obj_offset &= ~FIRST_OBJ_PAGE_TYPE_MASK; zpdesc->first_obj_offset |= offset & FIRST_OBJ_PAGE_TYPE_MASK; } When init_zspage() sets an initial object offset at or above 16 MiB, the offset wraps modulo 16 MiB, causing find_alloced_obj() during compaction to scan from the wrong offset inside an object payload. > diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c > index 4a6ec43f418b2..2d83fcc8610da 100644 > --- a/mm/zsmalloc.c > +++ b/mm/zsmalloc.c [ ... ] > @@ -295,27 +343,28 @@ struct zs_pool { > > static inline void zpdesc_inc_zone_page_state(struct zpdesc *zpdesc) > { > - inc_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES); > + __zone_stat_add_folio(zpdesc_folio(zpdesc), NR_ZSPAGES); > } > > static inline void zpdesc_dec_zone_page_state(struct zpdesc *zpdesc) > { > - dec_zone_page_state(zpdesc_page(zpdesc), NR_ZSPAGES); > + __zone_stat_sub_folio(zpdesc_folio(zpdesc), NR_ZSPAGES); > } [Severity: Medium] Should these helpers use zone_stat_add_folio() and zone_stat_sub_folio() instead of the double-underscore variants? __zone_stat_add_folio() and __zone_stat_sub_folio() require preemption or interrupts to be disabled, whereas alloc_zspage(), zs_free() -> __free_zspage_lockless(), and zs_page_migrate() call these helpers in preemptible context with no locks held, triggering preemption-disabled assertions and racy per-CPU vmstat updates. [ ... ] > @@ -609,7 +667,8 @@ static int zs_stats_size_show(struct seq_file *s, void *v) > > spin_lock(&class->lock); > > - seq_printf(s, " %5u %5u ", i, class->size); > + seq_printf(s, " %5u %5u %5u", i, class->size, > + class_size_to_zs_order(class->size)); [Severity: Low] Does the format string need a trailing space after the third "%5u" to match the header and Total lines? Without it, all subsequent columns in each class row are shifted left by one character. [ ... ] > @@ -1189,16 +1257,17 @@ static void obj_write(struct size_class *class, struct zspage *zspage, > unsigned long obj, const void *src, size_t size) > { > struct zpdesc *zpdesc; > + unsigned long page_size = class_size_to_zs_size(class->size); > unsigned long off; > unsigned int obj_idx; > > obj_to_location(obj, &zpdesc, &obj_idx); > - off = offset_in_page(class->size * obj_idx); > + off = (class->size * obj_idx) & (page_size - 1); > > if (!ZsHugePage(zspage)) > off += ZS_HANDLE_SIZE; > > - if (off + size <= PAGE_SIZE) { > + if (off + size <= page_size) { > /* this object is contained entirely within a page */ > void *dst = kmap_local_zpdesc(zpdesc); > > @@ -1208,7 +1277,7 @@ static void obj_write(struct size_class *class, struct zspage *zspage, > /* this object spans two pages */ > size_t sizes[2]; > > - sizes[0] = PAGE_SIZE - off; > + sizes[0] = page_size - off; > sizes[1] = size - sizes[0]; > > memcpy_to_page(zpdesc_page(zpdesc), off, src, sizes[0]); [Severity: High] Does calling memcpy_to_page() here trigger VM_BUG_ON(offset + len > PAGE_SIZE) when writing a multi-page object that spans two zpdescs? For a multi-page class, page_size is larger than PAGE_SIZE, so off + sizes[0] equals page_size and exceeds PAGE_SIZE in the first memcpy_to_page() call (and sizes[1] can also exceed PAGE_SIZE in the second call). > @@ -1230,9 +1299,9 @@ static void obj_write(struct size_class *class, struct zspage *zspage, > * > * Return: the size (in bytes) of the first huge zsmalloc &size_class. > */ > -size_t zs_huge_class_size(struct zs_pool *pool) > +size_t zs_huge_class_size(struct zs_pool *pool, enum zsmalloc_type type) > { > - return huge_class_size; > + return huge_class_size[type]; > } > EXPORT_SYMBOL_GPL(zs_huge_class_size); [Severity: Low] The kernel-doc comment above zs_huge_class_size() is missing an @type entry for the new parameter. [ ... ] > @@ -1365,13 +1437,15 @@ static void obj_free(int class_size, unsigned long obj) > struct link_free *link; > struct zspage *zspage; > struct zpdesc *f_zpdesc; > - unsigned long f_offset; > + unsigned long f_offset, page_size, page_mask; > unsigned int f_objidx; > void *vaddr; > > > obj_to_location(obj, &f_zpdesc, &f_objidx); > - f_offset = offset_in_page(class_size * f_objidx); > + page_size = class_size_to_zs_size(class_size); > + page_mask = ~(page_size - 1); > + f_offset = (class_size * f_objidx) & ~page_mask; > zspage = get_zspage(f_zpdesc); [Severity: High] Does zs_rebind_handle() also need to mask the object offset with page_size - 1 instead of offset_in_page()? mm/zsmalloc.c:zs_rebind_handle() { ... if (likely(!ZsHugePage(zspage))) { obj_to_location(obj, &zpdesc, &obj_idx); off = offset_in_page(class->size * obj_idx); vaddr = kmap_local_zpdesc(zpdesc); link = (struct link_free *)(vaddr + off); link->handle = (unsigned long)dst | OBJ_ALLOCATED_TAG; kunmap_local(vaddr); ... } When a non-huge multi-page object sits at an offset >= PAGE_SIZE within its zpdesc, zs_rebind_handle() overwrites 8 bytes in the first subpage and leaves the object's own header pointing to the old src handle, so later compaction updates src instead of dst. [ ... ] > @@ -1615,13 +1691,13 @@ static struct zs_handle *find_alloced_obj(struct size_class *class, > int index = *obj_idx; > struct zs_handle *handle = NULL; > void *addr = kmap_local_zpdesc(zpdesc); > + unsigned long page_size = class_size_to_zs_size(class->size); > > offset = get_first_obj_offset(zpdesc); > offset += class->size * index; > > - while (offset < PAGE_SIZE) { > - handle = obj_allocated(zpdesc, addr + offset); > - if (handle) > + while (offset < page_size) { > + if (obj_allocated(zpdesc, addr + offset, &handle)) > break; [Severity: High] Does this break compilation at this commit? obj_allocated() only takes two arguments (zpdesc and obj) and returns the handle pointer, so passing &handle fails to build until commit "zram: support compression at the granularity of multi-pages" reverts this line back to the two-argument call. [ ... ] > @@ -2215,16 +2292,32 @@ struct zs_pool *zs_create_pool(const char *name) > */ > for (i = ZS_SIZE_CLASSES - 1; i >= 0; i--) { > int size; > - int pages_per_zspage; > + int zpdescs_per_zspage; > int objs_per_zspage; > struct size_class *class; > int fullness; > + int idx = ZSMALLOC_TYPE_BASEPAGE; > +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES > + int order = 0; > +#endif > > - size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA; > + if (i < ZS_PAGE_SIZE_CLASSES) > + size = ZS_MIN_ALLOC_SIZE + i * ZS_SIZE_CLASS_DELTA; > +#ifdef CONFIG_ZSMALLOC_MULTI_PAGES > + if (i >= ZS_PAGE_SIZE_CLASSES) > + size = PAGE_SIZE + (i - ZS_PAGE_SIZE_CLASSES) * > + ZS_MULTI_PAGES_SIZE_CLASS_DELTA; > +#endif > if (size > ZS_MAX_ALLOC_SIZE) > size = ZS_MAX_ALLOC_SIZE; [Severity: High] Can the last base-page size class exceed PAGE_SIZE when CONFIG_ZSMALLOC_MULTI_PAGES is enabled? Previously, ZS_MAX_ALLOC_SIZE was PAGE_SIZE, which clamped class ZS_PAGE_SIZE_CLASSES - 1 to PAGE_SIZE when PAGE_SIZE - ZS_MIN_ALLOC_SIZE was not a multiple of ZS_SIZE_CLASS_DELTA (such as on 16 KiB and 64 KiB page kernels). With ZS_MAX_ALLOC_SIZE raised to ZSMALLOC_MULTI_PAGES_SIZE, class ZS_PAGE_SIZE_CLASSES - 1 gets size == PAGE_SIZE + 32 on 16 KiB and 64 KiB page systems, turning it into a non-huge multi-page class while class ZS_PAGE_SIZE_CLASSES starts at PAGE_SIZE and is unreachable from get_size_class_index(). Storing an incompressible PAGE_SIZE page then lands in a non-huge class with an 8-byte handle prefix, corrupting incompressible page reads in zram and triggering length warnings in zswap. Should sizes for i < ZS_PAGE_SIZE_CLASSES be clamped to PAGE_SIZE, and should the multi-page classes start at i - ZS_PAGE_SIZE_CLASSES + 1? -- Sashiko AI review · https://sashiko.dev/#/patchset/20261115024455.1465726-1-zhaonanzhe@xiaomi.com?part=1 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages 2026-11-15 2:44 [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages Nanzhe Zhao 2026-11-15 2:44 ` [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages Nanzhe Zhao @ 2026-11-15 2:44 ` Nanzhe Zhao 2026-11-16 2:46 ` sashiko-bot 2026-11-15 2:44 ` [RFC PATCH v4 3/4] zram: backend_zstd: use multi-page size for zstd parameter setup Nanzhe Zhao ` (2 subsequent siblings) 4 siblings, 1 reply; 9+ messages in thread From: Nanzhe Zhao @ 2026-11-15 2:44 UTC (permalink / raw) To: Andrew Morton Cc: Tangquan Zheng, linux-mm, linux-block, linux-kernel, Jens Axboe, Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner, Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li, David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts, Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee, Nick Terrell, Barry Song, Bo Zhang, Nanzhe Zhao From: Tangquan Zheng <zhengtangquan@oppo.com> Currently, when a large folio with nr_pages is submitted to zram, it is divided into nr_pages parts for compression and storage individually. By transitioning to a higher granularity, we can notably enhance compression rates while simultaneously reducing CPU consumption. This patch introduces the capability for large folios to be divided based on the granularity specified by ZSMALLOC_MULTI_PAGES_ORDER, which defaults to 2. For instance, for folios sized at 128KiB, compression will occur in eight 16KiB multi-pages. This modification will notably reduce CPU consumption and enhance compression ratios. The following data illustrates the time and compressed data for typical anonymous pages gathered from Android phones. Signed-off-by: Tangquan Zheng <zhengtangquan@oppo.com> Co-developed-by: Barry Song <baohua@kernel.org> Signed-off-by: Barry Song <baohua@kernel.org> Co-developed-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com> Signed-off-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com> --- drivers/block/zram/Kconfig | 9 + drivers/block/zram/zcomp.c | 31 ++- drivers/block/zram/zcomp.h | 19 +- drivers/block/zram/zram_drv.c | 500 ++++++++++++++++++++++++++++++---- drivers/block/zram/zram_drv.h | 43 +++ mm/zsmalloc.c | 3 +- 6 files changed, 549 insertions(+), 56 deletions(-) diff --git a/drivers/block/zram/Kconfig b/drivers/block/zram/Kconfig index 402b7b175863..716e92c5fdfe 100644 --- a/drivers/block/zram/Kconfig +++ b/drivers/block/zram/Kconfig @@ -145,3 +145,12 @@ config ZRAM_MULTI_COMP re-compress pages using a potentially slower but more effective compression algorithm. Note, that IDLE page recompression requires ZRAM_TRACK_ENTRY_ACTIME. + +config ZRAM_MULTI_PAGES + bool "Enable multiple pages compression and decompression" + depends on ZRAM && ZSMALLOC_MULTI_PAGES + help + Initially, zram divided large folios into blocks of nr_pages, each sized + equal to PAGE_SIZE, for compression. This option fine-tunes zram to + improve compression granularity by dividing large folios into larger + parts defined by the configuration option ZSMALLOC_MULTI_PAGES_ORDER. diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c index ad8c7dc73922..8fa2a44adaa3 100644 --- a/drivers/block/zram/zcomp.c +++ b/drivers/block/zram/zcomp.c @@ -71,7 +71,7 @@ static int zcomp_cstrm_init(struct zcomp *comp, struct zcomp_cstrm *zstrm) * allocate 2 pages. 1 for compressed data, plus 1 extra for the * case when compressed size is larger than the original one */ - zstrm->buffer = vzalloc(2 * PAGE_SIZE); + zstrm->buffer = vzalloc(2 * ZCOMP_MULTI_PAGES_SIZE); if (!zstrm->buffer) { zcomp_cstrm_free(comp, zstrm); return -ENOMEM; @@ -83,6 +83,10 @@ static void zcomp_dstrm_free(struct zcomp *comp, struct zcomp_dstrm *zstrm) { if (comp->ops->destroy_dctx) comp->ops->destroy_dctx(&zstrm->ctx); +#ifdef CONFIG_ZRAM_MULTI_PAGES + vfree(zstrm->decomp_buf); + zstrm->decomp_buf = NULL; +#endif vfree(zstrm->local_copy); zstrm->local_copy = NULL; zstrm->ctx.context = NULL; @@ -101,11 +105,23 @@ static int zcomp_dstrm_init(struct zcomp *comp, struct zcomp_dstrm *zstrm) } } - zstrm->local_copy = vzalloc(PAGE_SIZE); + zstrm->local_copy = vzalloc(ZCOMP_MULTI_PAGES_SIZE); if (!zstrm->local_copy) { zcomp_dstrm_free(comp, zstrm); return -ENOMEM; } +#ifdef CONFIG_ZRAM_MULTI_PAGES + /* + * Preallocate a decompression buffer for partial multi-page reads to + * avoid memory allocation in the swapin path. + */ + zstrm->decomp_buf = vzalloc(ZCOMP_MULTI_PAGES_SIZE); + if (!zstrm->decomp_buf) { + zcomp_dstrm_free(comp, zstrm); + return -ENOMEM; + } + mutex_init(&zstrm->buf_lock); +#endif return 0; } @@ -255,13 +271,13 @@ void zcomp_dstrm_put(struct zcomp *comp, struct zcomp_dstrm *zstrm, } int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm, - const void *src, unsigned int *dst_len) + const void *src, unsigned int src_len, unsigned int *dst_len) { struct zcomp_req req = { .src = src, .dst = zstrm->buffer, - .src_len = PAGE_SIZE, - .dst_len = 2 * PAGE_SIZE, + .src_len = src_len, + .dst_len = 2 * src_len, }; int ret; @@ -282,12 +298,13 @@ static int zcomp_decompress_buf(struct zcomp *comp, struct zcomp_dstrm *zstrm, } int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm, - struct scatterlist *sg, unsigned int src_len, void *dst) + struct scatterlist *sg, unsigned int src_len, void *dst, + unsigned int dst_len) { struct zcomp_req req = { .dst = dst, .src_len = src_len, - .dst_len = PAGE_SIZE, + .dst_len = dst_len, }; void *src; int ret; diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h index 39efefc43a13..fee105f876b4 100644 --- a/drivers/block/zram/zcomp.h +++ b/drivers/block/zram/zcomp.h @@ -47,6 +47,15 @@ struct zcomp_ctx { void *context; }; +#ifdef CONFIG_ZRAM_MULTI_PAGES +#define ZCOMP_MULTI_PAGES_ORDER (_AC(CONFIG_ZSMALLOC_MULTI_PAGES_ORDER, UL)) +#define ZCOMP_MULTI_PAGES_NR (1 << ZCOMP_MULTI_PAGES_ORDER) +#define ZCOMP_MULTI_PAGES_SIZE (PAGE_SIZE * ZCOMP_MULTI_PAGES_NR) +#define MULTI_PAGE_SHIFT (ZCOMP_MULTI_PAGES_ORDER + PAGE_SHIFT) +#else +#define ZCOMP_MULTI_PAGES_SIZE PAGE_SIZE +#endif + struct zcomp_cstrm { struct mutex lock; /* compression buffer */ @@ -65,6 +74,11 @@ struct zcomp_dstrm { struct mutex excl_lock; /* local copy of handle memory */ void *local_copy; +#ifdef CONFIG_ZRAM_MULTI_PAGES + /* Temporary buffer and lock for partial multi-page decompression */ + void *decomp_buf; + struct mutex buf_lock; +#endif struct zcomp_ctx ctx; }; @@ -124,8 +138,9 @@ void zcomp_dstrm_put(struct zcomp *comp, struct zcomp_dstrm *zstrm, enum zcomp_dstrm_mode mode); int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm, - const void *src, unsigned int *dst_len); + const void *src, unsigned int src_len, unsigned int *dst_len); int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm, - struct scatterlist *sg, unsigned int src_len, void *dst); + struct scatterlist *sg, unsigned int src_len, void *dst, + unsigned int dst_len); #endif /* _ZCOMP_H_ */ diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 24eefcf143de..0d33220f4b22 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -53,11 +53,12 @@ static unsigned int num_devices = 1; * Pages that compress to sizes equals or greater than this are stored * uncompressed in memory. */ -static size_t huge_class_size; +static size_t huge_class_size[ZSMALLOC_TYPE_MAX]; static const struct block_device_operations zram_devops; static void slot_free(struct zram *zram, unsigned long index); +static void slot_free_npages(struct zram *zram, unsigned long index, int nr_pages); /* * entry locking rules: @@ -321,11 +322,12 @@ static inline void zram_fill_page(void *ptr, unsigned long len, memset_l(ptr, value, len / sizeof(unsigned long)); } -static bool page_same_filled(void *ptr, unsigned long *element) +static bool page_same_filled(void *ptr, unsigned long *element, + unsigned int page_size) { unsigned long *page; unsigned long val; - unsigned int pos, last_pos = PAGE_SIZE / sizeof(*page) - 1; + unsigned int pos, last_pos = page_size / sizeof(*page) - 1; page = (unsigned long *)ptr; val = page[0]; @@ -484,7 +486,7 @@ static ssize_t idle_store(struct device *dev, struct device_attribute *attr, static int read_from_zspool_raw(struct zram *zram, struct page *page, unsigned long index); static int read_from_zspool(struct zram *zram, struct page *page, - unsigned long index); + unsigned long index, unsigned long page_size); struct zram_wb_ctl { /* idle list is accessed only by the writeback task, no concurency */ @@ -1092,7 +1094,7 @@ static int zram_writeback_slots(struct zram *zram, if (zram->compressed_wb) err = read_from_zspool_raw(zram, req->page, index); else - err = read_from_zspool(zram, req->page, index); + err = read_from_zspool(zram, req->page, index, PAGE_SIZE); if (err) goto next; slot_unlock(zram, index); @@ -1379,7 +1381,7 @@ static int decompress_bdev_page(struct zram *zram, struct page *page, zstrm = zcomp_dstrm_get(zram->comps[prio], ZCOMP_DSTRM_EXCLUSIVE); ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size, - zstrm->local_copy); + zstrm->local_copy, PAGE_SIZE); if (!ret) { void *dst = kmap_local_page(page); @@ -1989,6 +1991,31 @@ static ssize_t debug_stat_show(struct device *dev, return ret; } +#ifdef CONFIG_ZRAM_MULTI_PAGES +static ssize_t multi_pages_debug_stat_show(struct device *dev, + struct device_attribute *attr, char *buf) +{ + struct zram *zram = dev_to_zram(dev); + ssize_t ret; + + guard(rwsem_read)(&zram->dev_lock); + ret = sysfs_emit(buf, + "zram_bio write/read multi_pages count:%8llu %8llu\n" + "zram_bio failed write/read multi_pages count%8llu %8llu\n" + "zram_bio partial write/read multi_pages count%8llu %8llu\n" + "multi_pages_miss_free %8llu\n", + (u64)atomic64_read(&zram->stats.zram_bio_write_multi_pages_count), + (u64)atomic64_read(&zram->stats.zram_bio_read_multi_pages_count), + (u64)atomic64_read(&zram->stats.multi_pages_failed_writes), + (u64)atomic64_read(&zram->stats.multi_pages_failed_reads), + (u64)atomic64_read(&zram->stats.zram_bio_write_multi_pages_partial_count), + (u64)atomic64_read(&zram->stats.zram_bio_read_multi_pages_partial_count), + (u64)atomic64_read(&zram->stats.multi_pages_miss_free)); + + return ret; +} +#endif + static void zram_meta_free(struct zram *zram, u64 disksize) { unsigned long num_pages = disksize >> PAGE_SHIFT; @@ -2010,6 +2037,7 @@ static void zram_meta_free(struct zram *zram, u64 disksize) static bool zram_meta_alloc(struct zram *zram, u64 disksize) { unsigned long num_pages; + int i; num_pages = disksize >> PAGE_SHIFT; zram->table = vzalloc(array_size(num_pages, sizeof(*zram->table))); @@ -2023,9 +2051,10 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) return false; } - if (!huge_class_size) - huge_class_size = zs_huge_class_size(zram->mem_pool, - ZSMALLOC_TYPE_BASEPAGE); + for (i = 0; i < ZSMALLOC_TYPE_MAX; i++) { + if (!huge_class_size[i]) + huge_class_size[i] = zs_huge_class_size(zram->mem_pool, i); + } lockdep_register_key(&zram->table_lock_key); lockdep_init_map(&zram->table_lock_map, "zram->table[index].lock", &zram->table_lock_key, 0); @@ -2033,10 +2062,18 @@ static bool zram_meta_alloc(struct zram *zram, u64 disksize) return true; } -static void slot_free(struct zram *zram, unsigned long index) +static void slot_free_npages(struct zram *zram, unsigned long index, + int nr_pages) { struct zs_handle *handle; +#ifdef CONFIG_ZRAM_MULTI_PAGES + if (test_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES)) { + clear_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES); + nr_pages = max(nr_pages, ZCOMP_MULTI_PAGES_NR); + } +#endif + #ifdef CONFIG_ZRAM_TRACK_ENTRY_ACTIME zram->table[index].attr.ac_time = 0; #endif @@ -2068,7 +2105,7 @@ static void slot_free(struct zram *zram, unsigned long index) */ if (test_slot_flag(zram, index, ZRAM_SAME)) { clear_slot_flag(zram, index, ZRAM_SAME); - atomic64_dec(&zram->stats.same_pages); + atomic64_sub(nr_pages, &zram->stats.same_pages); goto out; } @@ -2081,44 +2118,49 @@ static void slot_free(struct zram *zram, unsigned long index) atomic64_sub(get_slot_size(zram, index), &zram->stats.compr_data_size); out: - atomic64_dec(&zram->stats.pages_stored); + atomic64_sub(nr_pages, &zram->stats.pages_stored); zram->table[index].element = 0; set_slot_size(zram, index, 0); } +static void slot_free(struct zram *zram, unsigned long index) +{ + slot_free_npages(zram, index, 1); +} + static int read_same_filled_page(struct zram *zram, struct page *page, - unsigned long index) + unsigned long index, unsigned long page_size) { void *mem; mem = kmap_local_page(page); - zram_fill_page(mem, PAGE_SIZE, zram->table[index].element); + zram_fill_page(mem, page_size, zram->table[index].element); kunmap_local(mem); return 0; } static int read_incompressible_page(struct zram *zram, struct page *page, - unsigned long index) + unsigned long index, + unsigned long page_size) { struct scatterlist sg[2]; struct zs_handle *handle; - void *src, *dst; + void *dst; handle = &zram->table[index].handle; - zs_obj_read_sg_begin(zram->mem_pool, handle, sg, PAGE_SIZE); + zs_obj_read_sg_begin(zram->mem_pool, handle, sg, page_size); /* an incompressible object never spans two pages */ - src = kmap_local_page(sg_page(sg)); dst = kmap_local_page(page); - copy_page(dst, src); + sg_copy_to_buffer(sg, sg_nents(sg), dst, page_size); kunmap_local(dst); - kunmap_local(src); zs_obj_read_sg_end(zram->mem_pool, handle); return 0; } static int read_compressed_page(struct zram *zram, struct page *page, - unsigned long index) + unsigned long index, + unsigned long page_size) { enum zcomp_dstrm_mode mode = ZCOMP_DSTRM_EXCLUSIVE; struct zcomp_dstrm *zstrm; @@ -2138,7 +2180,8 @@ static int read_compressed_page(struct zram *zram, struct page *page, zstrm = zcomp_dstrm_get(zram->comps[prio], mode); dst = kmap_local_page(page); - ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size, dst); + ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size, dst, + page_size); kunmap_local(dst); zcomp_dstrm_put(zram->comps[prio], zstrm, mode); zs_obj_read_sg_end(zram->mem_pool, handle); @@ -2179,20 +2222,21 @@ static int read_from_zspool_raw(struct zram *zram, struct page *page, * Corresponding ZRAM slot should be locked. */ static int read_from_zspool(struct zram *zram, struct page *page, - unsigned long index) + unsigned long index, unsigned long page_size) { if (test_slot_flag(zram, index, ZRAM_SAME) || !zs_handle_is_allocated(&zram->table[index].handle)) - return read_same_filled_page(zram, page, index); + return read_same_filled_page(zram, page, index, page_size); if (!test_slot_flag(zram, index, ZRAM_HUGE)) - return read_compressed_page(zram, page, index); + return read_compressed_page(zram, page, index, page_size); else - return read_incompressible_page(zram, page, index); + return read_incompressible_page(zram, page, index, page_size); } -static int zram_read_page(struct zram *zram, struct page *page, - unsigned long index, struct bio *parent) +static int zram_read_page_size(struct zram *zram, struct page *page, + unsigned long index, struct bio *parent, + unsigned long page_size) { int ret; @@ -2200,7 +2244,7 @@ static int zram_read_page(struct zram *zram, struct page *page, mark_slot_accessed(zram, index); if (!test_slot_flag(zram, index, ZRAM_WB)) { /* Slot should be locked through out the function call */ - ret = read_from_zspool(zram, page, index); + ret = read_from_zspool(zram, page, index, page_size); slot_unlock(zram, index); } else { unsigned long blk_idx = zram->table[index].blk_idx; @@ -2216,6 +2260,12 @@ static int zram_read_page(struct zram *zram, struct page *page, return ret; } +static int zram_read_page(struct zram *zram, struct page *page, + unsigned long index, struct bio *parent) +{ + return zram_read_page_size(zram, page, index, parent, PAGE_SIZE); +} + /* * Use a temporary buffer to decompress the page, as the decompressor * always expects a full page for the output. @@ -2244,23 +2294,32 @@ static int zram_bvec_read(struct zram *zram, struct bio_vec *bvec, } static int write_same_filled_page(struct zram *zram, unsigned long fill, - unsigned long index) + unsigned long index, int nr_pages) { slot_lock(zram, index); slot_free(zram, index); set_slot_flag(zram, index, ZRAM_SAME); zram->table[index].element = fill; mark_slot_accessed(zram, index); +#ifdef CONFIG_ZRAM_MULTI_PAGES + if (nr_pages > 1) { + int i; + + for (i = 0; i < nr_pages; i++) + set_slot_flag(zram, index + i, ZRAM_COMP_MULTI_PAGES); + } +#endif slot_unlock(zram, index); - atomic64_inc(&zram->stats.same_pages); - atomic64_inc(&zram->stats.pages_stored); + atomic64_add(nr_pages, &zram->stats.same_pages); + atomic64_add(nr_pages, &zram->stats.pages_stored); return 0; } static int write_incompressible_page(struct zram *zram, struct page *page, - unsigned long index) + unsigned long index, unsigned int page_size, + int nr_pages) { void *src; int ret; @@ -2269,7 +2328,7 @@ static int write_incompressible_page(struct zram *zram, struct page *page, slot_free(zram, index); src = kmap_local_page(page); ret = zs_store(zram->mem_pool, &zram->table[index].handle, src, - PAGE_SIZE, GFP_NOIO | __GFP_NOWARN | + page_size, GFP_NOIO | __GFP_NOWARN | __GFP_HIGHMEM | __GFP_MOVABLE, page_to_nid(page)); kunmap_local(src); if (ret) { @@ -2280,14 +2339,22 @@ static int write_incompressible_page(struct zram *zram, struct page *page, update_used_max(zram); set_slot_flag(zram, index, ZRAM_HUGE); - set_slot_size(zram, index, PAGE_SIZE); + set_slot_size(zram, index, page_size); mark_slot_accessed(zram, index); +#ifdef CONFIG_ZRAM_MULTI_PAGES + if (nr_pages > 1) { + int i; + + for (i = 0; i < nr_pages; i++) + set_slot_flag(zram, index + i, ZRAM_COMP_MULTI_PAGES); + } +#endif slot_unlock(zram, index); - atomic64_add(PAGE_SIZE, &zram->stats.compr_data_size); + atomic64_add(page_size, &zram->stats.compr_data_size); atomic64_inc(&zram->stats.huge_pages); atomic64_inc(&zram->stats.huge_pages_since); - atomic64_inc(&zram->stats.pages_stored); + atomic64_add(nr_pages, &zram->stats.pages_stored); return 0; } @@ -2301,12 +2368,27 @@ static int zram_write_page(struct zram *zram, struct page *page, struct zcomp_cstrm *zstrm; unsigned long element; bool same_filled; + unsigned long page_size = PAGE_SIZE; + int huge_class_idx = ZSMALLOC_TYPE_BASEPAGE; + int nr_pages = 1; + +#ifdef CONFIG_ZRAM_MULTI_PAGES + /* + * Multi-page compression packs ZCOMP_MULTI_PAGES_NR pages into one + * compressed object. + */ + if (folio_size(page_folio(page)) >= ZCOMP_MULTI_PAGES_SIZE) { + page_size = ZCOMP_MULTI_PAGES_SIZE; + huge_class_idx = ZSMALLOC_TYPE_MULTI_PAGES; + nr_pages = ZCOMP_MULTI_PAGES_NR; + } +#endif mem = kmap_local_page(page); - same_filled = page_same_filled(mem, &element); + same_filled = page_same_filled(mem, &element, page_size); kunmap_local(mem); if (same_filled) - return write_same_filled_page(zram, element, index); + return write_same_filled_page(zram, element, index, nr_pages); if (!zram_can_store_page(zram)) return -ENOMEM; @@ -2314,7 +2396,7 @@ static int zram_write_page(struct zram *zram, struct page *page, zstrm = zcomp_cstrm_get(zram->comps[ZRAM_PRIMARY_COMP]); mem = kmap_local_page(page); ret = zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm, - mem, &comp_len); + mem, page_size, &comp_len); kunmap_local(mem); if (unlikely(ret)) { @@ -2323,9 +2405,10 @@ static int zram_write_page(struct zram *zram, struct page *page, return ret; } - if (comp_len >= huge_class_size) { + if (comp_len >= huge_class_size[huge_class_idx]) { zcomp_cstrm_put(zstrm); - return write_incompressible_page(zram, page, index); + return write_incompressible_page(zram, page, index, page_size, + nr_pages); } slot_lock(zram, index); @@ -2344,10 +2427,18 @@ static int zram_write_page(struct zram *zram, struct page *page, set_slot_size(zram, index, comp_len); mark_slot_accessed(zram, index); +#ifdef CONFIG_ZRAM_MULTI_PAGES + if (nr_pages > 1) { + int i; + + for (i = 0; i < nr_pages; i++) + set_slot_flag(zram, index + i, ZRAM_COMP_MULTI_PAGES); + } +#endif slot_unlock(zram, index); /* Update stats */ - atomic64_inc(&zram->stats.pages_stored); + atomic64_add(nr_pages, &zram->stats.pages_stored); atomic64_add(comp_len, &zram->stats.compr_data_size); return ret; @@ -2467,7 +2558,7 @@ static int recompress_slot(struct zram *zram, unsigned long index, if (comp_len_old < threshold) return 0; - ret = read_from_zspool(zram, page, index); + ret = read_from_zspool(zram, page, index, PAGE_SIZE); if (ret) return ret; @@ -2480,7 +2571,8 @@ static int recompress_slot(struct zram *zram, unsigned long index, zstrm = zcomp_cstrm_get(zram->comps[prio]); src = kmap_local_page(page); - ret = zcomp_compress(zram->comps[prio], zstrm, src, &comp_len_new); + ret = zcomp_compress(zram->comps[prio], zstrm, src, PAGE_SIZE, + &comp_len_new); kunmap_local(src); /* @@ -2615,7 +2707,7 @@ static ssize_t recompress_store(struct device *dev, } } - if (threshold >= huge_class_size) + if (threshold >= huge_class_size[ZSMALLOC_TYPE_BASEPAGE]) return -EINVAL; guard(rwsem_write)(&zram->dev_lock); @@ -2727,8 +2819,283 @@ static void zram_bio_discard(struct zram *zram, struct bio *bio) bio_endio(bio); } +#ifdef CONFIG_ZRAM_MULTI_PAGES + +static inline unsigned long zram_multi_pages_head_index(unsigned long index) +{ + return index & ~((unsigned long)ZCOMP_MULTI_PAGES_NR - 1); +} + +static inline int __test_multi_pages_comp(struct zram *zram, + unsigned long index) +{ + unsigned long head = zram_multi_pages_head_index(index); + int i, count = 0; + + for (i = 0; i < ZCOMP_MULTI_PAGES_NR; i++) { + if (test_slot_flag(zram, head + i, ZRAM_COMP_MULTI_PAGES)) + count++; + } + + return count; +} + +static inline bool want_multi_pages_comp(struct zram *zram, struct bio *bio) +{ + unsigned long index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; + + if (bio->bi_io_vec->bv_len >= ZCOMP_MULTI_PAGES_SIZE) + return true; + + slot_lock(zram, index); + if (__test_multi_pages_comp(zram, index)) { + slot_unlock(zram, index); + return true; + } + slot_unlock(zram, index); + + return false; +} + +static inline bool test_multi_pages_comp(struct zram *zram, struct bio *bio) +{ + unsigned long index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; + + return !!__test_multi_pages_comp(zram, index); +} + +static inline bool is_multi_pages_partial_io(struct bio_vec *bvec) +{ + return bvec->bv_len != ZCOMP_MULTI_PAGES_SIZE; +} + +static int zram_read_partial_from_zspool(struct zram *zram, struct page *page, + unsigned long index, int offset) +{ + struct zs_handle *handle = &zram->table[index].handle; + struct zcomp_dstrm *zstrm; + struct scatterlist sg[2]; + enum zcomp_dstrm_mode mode; + unsigned int size; + void *src, *dst; + u32 prio; + int ret; + unsigned long page_size = ZCOMP_MULTI_PAGES_SIZE; + + if (test_slot_flag(zram, index, ZRAM_SAME) || + !zs_handle_is_allocated(handle)) { + void *mem; + + mem = kmap_local_page(page); + zram_fill_page(mem, PAGE_SIZE, zram->table[index].element); + kunmap_local(mem); + return 0; + } + + size = get_slot_size(zram, index); + zs_obj_read_sg_begin(zram->mem_pool, handle, sg, size); + + if (size == page_size) { + src = kmap_local_page(sg_page(sg)); + dst = kmap_local_page(page); + memcpy(dst, src + offset, PAGE_SIZE); + kunmap_local(dst); + kunmap_local(src); + ret = 0; + } else { + prio = get_slot_comp_priority(zram, index); + mode = ZCOMP_DSTRM_EXCLUSIVE; + if (zcomp_shared_decomp(zram->comps[prio])) + mode = ZCOMP_DSTRM_SHARED; + + zstrm = zcomp_dstrm_get(zram->comps[prio], mode); + mutex_lock(&zstrm->buf_lock); + ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size, + zstrm->decomp_buf, page_size); + if (!ret) { + dst = kmap_local_page(page); + memcpy(dst, zstrm->decomp_buf + offset, PAGE_SIZE); + kunmap_local(dst); + } + mutex_unlock(&zstrm->buf_lock); + zcomp_dstrm_put(zram->comps[prio], zstrm, mode); + } + zs_obj_read_sg_end(zram->mem_pool, handle); + + return ret; +} + +static int zram_bvec_read_multi_pages_partial(struct zram *zram, + struct bio_vec *bvec, + unsigned long index, int offset, + struct bio *bio) +{ + int ret; + + slot_lock(zram, index); + if (!test_slot_flag(zram, index, ZRAM_WB)) { + ret = zram_read_partial_from_zspool(zram, bvec->bv_page, + index, offset); + slot_unlock(zram, index); + } else { + unsigned long blk_idx = zram->table[index].blk_idx; + + slot_unlock(zram, index); + ret = read_from_bdev(zram, bvec->bv_page, index, blk_idx, bio); + } + + if (WARN_ON(ret < 0)) + pr_err("Decompression failed! err=%d, page=%lu offset=%d\n", + ret, index, offset); + if (!ret) + atomic64_inc(&zram->stats.zram_bio_read_multi_pages_partial_count); + + return ret; +} + +static int zram_bvec_read_multi_pages(struct zram *zram, struct bio_vec *bvec, + unsigned long index, int offset, + struct bio *bio) +{ + if (is_multi_pages_partial_io(bvec)) + return zram_bvec_read_multi_pages_partial(zram, bvec, index, + offset, bio); + + return zram_read_page_size(zram, bvec->bv_page, index, bio, + ZCOMP_MULTI_PAGES_SIZE); +} + +static int zram_bvec_write_multi_pages_partial(struct zram *zram, + struct bio_vec *bvec, + unsigned long index, int offset, + struct bio *bio) +{ + struct page *page; + void *src, *dst; + int ret; + + page = alloc_pages(GFP_NOIO | __GFP_COMP, ZCOMP_MULTI_PAGES_ORDER); + if (!page) + return -ENOMEM; + + ret = zram_read_page_size(zram, page, index, bio, + ZCOMP_MULTI_PAGES_SIZE); + if (!ret) { + src = kmap_local_page(bvec->bv_page); + dst = kmap_local_page(page); + memcpy(dst + offset, src + bvec->bv_offset, bvec->bv_len); + kunmap_local(dst); + kunmap_local(src); + + atomic64_inc(&zram->stats.zram_bio_write_multi_pages_partial_count); + ret = zram_write_page(zram, page, index); + } + + __free_pages(page, ZCOMP_MULTI_PAGES_ORDER); + return ret; +} + +static int zram_bvec_write_multi_pages(struct zram *zram, struct bio_vec *bvec, + unsigned long index, int offset, + struct bio *bio) +{ + if (is_multi_pages_partial_io(bvec)) + return zram_bvec_write_multi_pages_partial(zram, bvec, index, + offset, bio); + + return zram_write_page(zram, bvec->bv_page, index); +} + +static void zram_bio_read_multi_pages(struct zram *zram, struct bio *bio) +{ + unsigned long start_time = bio_start_io_acct(bio); + struct bvec_iter iter = bio->bi_iter; + + do { + unsigned long index = zram_multi_pages_head_index( + iter.bi_sector >> SECTORS_PER_PAGE_SHIFT); + u32 offset = (iter.bi_sector & (SECTORS_PER_MULTI_PAGE - 1)) << + SECTOR_SHIFT; + struct bio_vec bv = multi_pages_bio_iter_iovec(bio, iter); + + atomic64_inc(&zram->stats.zram_bio_read_multi_pages_count); + bv.bv_len = min_t(u32, bv.bv_len, + ZCOMP_MULTI_PAGES_SIZE - offset); + + if (zram_bvec_read_multi_pages(zram, &bv, index, offset, bio) < 0) { + atomic64_inc(&zram->stats.multi_pages_failed_reads); + bio->bi_status = BLK_STS_IOERR; + break; + } + flush_dcache_page(bv.bv_page); + + slot_lock(zram, index); + mark_slot_accessed(zram, index); + slot_unlock(zram, index); + + bio_advance_iter_single(bio, &iter, bv.bv_len); + } while (iter.bi_size); + + bio_end_io_acct(bio, start_time); + bio_endio(bio); +} + +static void zram_bio_write_multi_pages(struct zram *zram, struct bio *bio) +{ + unsigned long start_time = bio_start_io_acct(bio); + struct bvec_iter iter = bio->bi_iter; + + do { + unsigned long index = zram_multi_pages_head_index( + iter.bi_sector >> SECTORS_PER_PAGE_SHIFT); + u32 offset = (iter.bi_sector & (SECTORS_PER_MULTI_PAGE - 1)) << + SECTOR_SHIFT; + struct bio_vec bv = multi_pages_bio_iter_iovec(bio, iter); + + bv.bv_len = min_t(u32, bv.bv_len, + ZCOMP_MULTI_PAGES_SIZE - offset); + + atomic64_inc(&zram->stats.zram_bio_write_multi_pages_count); + if (zram_bvec_write_multi_pages(zram, &bv, index, offset, bio) < 0) { + atomic64_inc(&zram->stats.multi_pages_failed_writes); + bio->bi_status = BLK_STS_IOERR; + break; + } + + slot_lock(zram, index); + mark_slot_accessed(zram, index); + slot_unlock(zram, index); + + bio_advance_iter_single(bio, &iter, bv.bv_len); + } while (iter.bi_size); + + bio_end_io_acct(bio, start_time); + bio_endio(bio); +} +#else /* CONFIG_ZRAM_MULTI_PAGES */ +static inline bool test_multi_pages_comp(struct zram *zram, struct bio *bio) +{ + return false; +} + +static inline bool want_multi_pages_comp(struct zram *zram, struct bio *bio) +{ + return false; +} + +static inline void zram_bio_read_multi_pages(struct zram *zram, + struct bio *bio) { } +static inline void zram_bio_write_multi_pages(struct zram *zram, + struct bio *bio) { } +#endif + static void zram_bio_read(struct zram *zram, struct bio *bio) { + if (test_multi_pages_comp(zram, bio)) { + zram_bio_read_multi_pages(zram, bio); + return; + } + unsigned long start_time = bio_start_io_acct(bio); struct bvec_iter iter = bio->bi_iter; @@ -2756,6 +3123,11 @@ static void zram_bio_read(struct zram *zram, struct bio *bio) static void zram_bio_write(struct zram *zram, struct bio *bio) { + if (want_multi_pages_comp(zram, bio)) { + zram_bio_write_multi_pages(zram, bio); + return; + } + unsigned long start_time = bio_start_io_acct(bio); struct bvec_iter iter = bio->bi_iter; @@ -2808,6 +3180,9 @@ static void zram_slot_free_notify(struct block_device *bdev, unsigned long index) { struct zram *zram; +#ifdef CONFIG_ZRAM_MULTI_PAGES + int nr_pages_to_free = 1; +#endif zram = bdev->bd_disk->private_data; @@ -2817,7 +3192,34 @@ static void zram_slot_free_notify(struct block_device *bdev, return; } +#ifdef CONFIG_ZRAM_MULTI_PAGES + { + int comp_count = __test_multi_pages_comp(zram, index); + + if (comp_count > 1) { + clear_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES); + slot_unlock(zram, index); + return; + } + + if (comp_count == 1) { + clear_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES); + slot_unlock(zram, index); + index = zram_multi_pages_head_index(index); + nr_pages_to_free = ZCOMP_MULTI_PAGES_NR; + if (!slot_trylock(zram, index)) { + atomic64_inc(&zram->stats.multi_pages_miss_free); + return; + } + } + } +#endif + +#ifdef CONFIG_ZRAM_MULTI_PAGES + slot_free_npages(zram, index, nr_pages_to_free); +#else slot_free(zram, index); +#endif slot_unlock(zram, index); } @@ -2991,6 +3393,9 @@ static const struct block_device_operations zram_devops = { static DEVICE_ATTR_RO(io_stat); static DEVICE_ATTR_RO(mm_stat); static DEVICE_ATTR_RO(debug_stat); +#ifdef CONFIG_ZRAM_MULTI_PAGES +static DEVICE_ATTR_RO(multi_pages_debug_stat); +#endif static DEVICE_ATTR_WO(compact); static DEVICE_ATTR_RW(disksize); static DEVICE_ATTR_RO(initstate); @@ -3035,6 +3440,9 @@ static struct attribute *zram_disk_attrs[] = { &dev_attr_io_stat.attr, &dev_attr_mm_stat.attr, &dev_attr_debug_stat.attr, +#ifdef CONFIG_ZRAM_MULTI_PAGES + &dev_attr_multi_pages_debug_stat.attr, +#endif #ifdef CONFIG_ZRAM_MULTI_COMP &dev_attr_recomp_algorithm.attr, &dev_attr_recompress.attr, diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h index d64c99b74214..1e3ebfec972e 100644 --- a/drivers/block/zram/zram_drv.h +++ b/drivers/block/zram/zram_drv.h @@ -28,6 +28,11 @@ #define ZRAM_SECTOR_PER_LOGICAL_BLOCK \ (1 << (ZRAM_LOGICAL_BLOCK_SHIFT - SECTOR_SHIFT)) +#ifdef CONFIG_ZRAM_MULTI_PAGES +#define SECTORS_PER_MULTI_PAGE_SHIFT (MULTI_PAGE_SHIFT - SECTOR_SHIFT) +#define SECTORS_PER_MULTI_PAGE (1 << SECTORS_PER_MULTI_PAGE_SHIFT) +#endif + /* * ZRAM is mainly used for memory efficiency so we want to keep memory * footprint small and thus squeeze size and zram pageflags into a flags @@ -37,7 +42,12 @@ * * We use BUILD_BUG_ON() to make sure that zram pageflags don't overflow. */ +#ifdef CONFIG_ZRAM_MULTI_PAGES +#define ZRAM_FLAG_SHIFT (PAGE_SHIFT + CONFIG_ZSMALLOC_MULTI_PAGES_ORDER + 1) +#else #define ZRAM_FLAG_SHIFT (PAGE_SHIFT + 1) +#endif + /* Only 2 bits are allowed for comp priority index */ #define ZRAM_COMP_PRIORITY_MASK 0x3 @@ -55,6 +65,10 @@ enum zram_pageflags { ZRAM_COMP_PRIORITY_BIT1, /* First bit of comp priority index */ ZRAM_COMP_PRIORITY_BIT2, /* Second bit of comp priority index */ +#ifdef CONFIG_ZRAM_MULTI_PAGES + ZRAM_COMP_MULTI_PAGES, /* Compressed by multi-pages */ +#endif + __NR_ZRAM_PAGEFLAGS, }; @@ -110,6 +124,15 @@ struct zram_stats { atomic64_t bd_reads; /* no. of reads from backing device */ atomic64_t bd_writes; /* no. of writes from backing device */ #endif +#ifdef CONFIG_ZRAM_MULTI_PAGES + atomic64_t zram_bio_write_multi_pages_count; + atomic64_t zram_bio_read_multi_pages_count; + atomic64_t multi_pages_failed_writes; + atomic64_t multi_pages_failed_reads; + atomic64_t zram_bio_write_multi_pages_partial_count; + atomic64_t zram_bio_read_multi_pages_partial_count; + atomic64_t multi_pages_miss_free; +#endif }; #ifdef CONFIG_ZRAM_MULTI_COMP @@ -167,4 +190,24 @@ struct zram { struct dentry *debugfs_dir; #endif }; + +#ifdef CONFIG_ZRAM_MULTI_PAGES +#define multi_pages_bvec_iter_offset(bvec, iter) \ + (mp_bvec_iter_offset((bvec), (iter)) % ZCOMP_MULTI_PAGES_SIZE) + +#define multi_pages_bvec_iter_len(bvec, iter) \ + min_t(unsigned int, mp_bvec_iter_len((bvec), (iter)), \ + ZCOMP_MULTI_PAGES_SIZE - \ + multi_pages_bvec_iter_offset((bvec), (iter))) + +#define multi_pages_bvec_iter_bvec(bvec, iter) \ +((struct bio_vec) { \ + .bv_page = mp_bvec_iter_page((bvec), (iter)), \ + .bv_len = multi_pages_bvec_iter_len((bvec), (iter)), \ + .bv_offset = multi_pages_bvec_iter_offset((bvec), (iter)), \ +}) + +#define multi_pages_bio_iter_iovec(bio, iter) \ + multi_pages_bvec_iter_bvec((bio)->bi_io_vec, (iter)) +#endif #endif diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c index 2d83fcc8610d..81d908d6d5a8 100644 --- a/mm/zsmalloc.c +++ b/mm/zsmalloc.c @@ -1697,7 +1697,8 @@ static struct zs_handle *find_alloced_obj(struct size_class *class, offset += class->size * index; while (offset < page_size) { - if (obj_allocated(zpdesc, addr + offset, &handle)) + handle = obj_allocated(zpdesc, addr + offset); + if (handle) break; offset += class->size; -- 2.43.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages 2026-11-15 2:44 ` [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages Nanzhe Zhao @ 2026-11-16 2:46 ` sashiko-bot 0 siblings, 0 replies; 9+ messages in thread From: sashiko-bot @ 2026-11-16 2:46 UTC (permalink / raw) To: Nanzhe Zhao, Andrew Morton Cc: Tangquan Zheng, linux-mm, Jens Axboe, Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner, Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li, David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts, Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee, Nick Terrell, Barry Song, Bo Zhang Thank you for your contribution! Sashiko AI review found 14 potential issues to consider: - [Critical] zram: partial read in zram_read_partial_from_zspool() ignores bv_len/offset - [Critical] zram: OOB access and data loss in zram_write_page() using folio_size() - [Critical] zram: bio-wide routing in want_multi_pages_comp() and test_multi_pages_comp() - [Critical] zram: multi_pages_bvec_iter_bvec() and partial read corrupt multi-page I/O - [High] zram: unlocked flag updates and leaked handles in zram_write_page() - [High] zram: sibling race and premature flag clear in zram_slot_free_notify() - [High] zram: unlocked RMW in zram_bvec_write_multi_pages_partial() loses writes - [High] zram: use-after-free in zram_bvec_write_multi_pages_partial() on ZRAM_WB - [High] zram: out-of-bounds access in place_pp_slot() and post-processing overflow - [High] mm: zsmalloc: VM_BUG_ON in obj_write() and bad offset in zs_rebind_handle() - [Medium] zram: out-of-bounds zram->table access when disksize is not multi-page aligned - [Medium] zram: ZRAM_FLAG_SHIFT overflows 32-bit attr.flags on 16K/64K page configs - [Medium] zram: inverted lock order between excl_lock and buf_lock in zcomp - [Low] zram: misplaced zsmalloc build fix, missing commit data, and stat format -- commit d614cb6134b7b12651ec475514fd15b492598ab0 Author: Tangquan Zheng <zhengtangquan@oppo.com> zram: support compression at the granularity of multi-pages This commit adds CONFIG_ZRAM_MULTI_PAGES to compress and store large folios in ZCOMP_MULTI_PAGES_SIZE chunks instead of individual PAGE_SIZE pages. > diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c > index ad8c7dc739223..8fa2a44adaa30 100644 > --- a/drivers/block/zram/zcomp.c > +++ b/drivers/block/zram/zcomp.c [ ... ] > @@ -101,11 +105,23 @@ static int zcomp_dstrm_init(struct zcomp *comp, struct zcomp_dstrm *zstrm) > } > } > > - zstrm->local_copy = vzalloc(PAGE_SIZE); > + zstrm->local_copy = vzalloc(ZCOMP_MULTI_PAGES_SIZE); > if (!zstrm->local_copy) { > zcomp_dstrm_free(comp, zstrm); > return -ENOMEM; > } > +#ifdef CONFIG_ZRAM_MULTI_PAGES > + /* > + * Preallocate a decompression buffer for partial multi-page reads to > + * avoid memory allocation in the swapin path. > + */ > + zstrm->decomp_buf = vzalloc(ZCOMP_MULTI_PAGES_SIZE); > + if (!zstrm->decomp_buf) { > + zcomp_dstrm_free(comp, zstrm); > + return -ENOMEM; > + } > + mutex_init(&zstrm->buf_lock); > +#endif > return 0; > } [Severity: Medium] Can zstrm->excl_lock and zstrm->buf_lock be acquired in opposite orders? In zram_read_partial_from_zspool(), an exclusive decompression backend (ZCOMP_DSTRM_EXCLUSIVE) takes zstrm->excl_lock inside zcomp_dstrm_get() and then locks zstrm->buf_lock. A shared decompression backend (ZCOMP_DSTRM_SHARED) locks zstrm->buf_lock first and then calls zcomp_decompress(), which locks zstrm->excl_lock when a two-page object falls back to zstrm->local_copy: CPU0 CPU1 ---- ---- lock(&zstrm->excl_lock) lock(&zstrm->buf_lock) lock(&zstrm->buf_lock) lock(&zstrm->excl_lock) *** DEADLOCK *** CPU0: zram_read_partial_from_zspool() -> zcomp_dstrm_get() CPU1: zram_read_partial_from_zspool() -> zcomp_decompress() Because all zstrm->excl_lock and zstrm->buf_lock instances share static lockdep classes across devices and CONFIG_ZRAM_MULTI_COMP priorities, this triggers a lockdep circular locking splat. Also, zcomp_dstrm_init() runs from the CPUHP_ZCOMP_PREPARE hotplug callback every time a CPU comes online, so mutex_init(&zstrm->buf_lock) re-initializes an already initialized mutex instead of initializing it once per possible CPU in zcomp_init(). > diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c > index 8b7d6926da266..e258289679015 100644 > --- a/drivers/block/zram/zram_drv.c > +++ b/drivers/block/zram/zram_drv.c [ ... ] > @@ -484,7 +486,7 @@ static ssize_t idle_store(struct device *dev, struct device_attribute *attr, > static int read_from_zspool_raw(struct zram *zram, struct page *page, > unsigned long index); > static int read_from_zspool(struct zram *zram, struct page *page, > - unsigned long index); > + unsigned long index, unsigned long page_size); > > struct zram_wb_ctl { > /* idle list is accessed only by the writeback task, no concurency */ > @@ -1092,7 +1094,7 @@ static int zram_writeback_slots(struct zram *zram, > if (zram->compressed_wb) > err = read_from_zspool_raw(zram, req->page, index); > else > - err = read_from_zspool(zram, req->page, index); > + err = read_from_zspool(zram, req->page, index, PAGE_SIZE); > if (err) > goto next; > slot_unlock(zram, index); [Severity: High] Can post-processing a multi-page slot overflow ctl->pp_buckets[] and req->page? Neither scan_slots_for_writeback() nor scan_slots_for_recompress() skips ZRAM_COMP_MULTI_PAGES slots, where get_slot_size() can be up to ZCOMP_MULTI_PAGES_SIZE. In place_pp_slot(), NUM_PP_BUCKETS is sized for PAGE_SIZE: drivers/block/zram/zram_drv.c:place_pp_slot() { ... bid = get_slot_size(zram, pps->index) / PP_BUCKET_SIZE_RANGE; list_add(&pps->entry, &ctl->pp_buckets[bid]); } When a multi-page slot's compressed size exceeds PAGE_SIZE, bid exceeds NUM_PP_BUCKETS and corrupts memory past struct zram_pp_ctl. In zram_writeback_slots(), req->page is a single PAGE_SIZE page, so read_from_zspool_raw() also overflows req->page and wraps PAGE_SIZE - size in memzero_page(), while uncompressed writeback and recompress_slot() pass PAGE_SIZE for a multi-page object. [ ... ] > @@ -2298,12 +2365,27 @@ static int zram_write_page(struct zram *zram, struct page *page, > struct zcomp_cstrm *zstrm; > unsigned long element; > bool same_filled; > + unsigned long page_size = PAGE_SIZE; > + int huge_class_idx = ZSMALLOC_TYPE_BASEPAGE; > + int nr_pages = 1; > + > +#ifdef CONFIG_ZRAM_MULTI_PAGES > + /* > + * Multi-page compression packs ZCOMP_MULTI_PAGES_NR pages into one > + * compressed object. > + */ > + if (folio_size(page_folio(page)) >= ZCOMP_MULTI_PAGES_SIZE) { > + page_size = ZCOMP_MULTI_PAGES_SIZE; > + huge_class_idx = ZSMALLOC_TYPE_MULTI_PAGES; > + nr_pages = ZCOMP_MULTI_PAGES_NR; > + } > +#endif > > mem = kmap_local_page(page); > - same_filled = page_same_filled(mem, &element); > + same_filled = page_same_filled(mem, &element, page_size); > kunmap_local(mem); > if (same_filled) > - return write_same_filled_page(zram, element, index); > + return write_same_filled_page(zram, element, index, nr_pages); [Severity: Critical] Does sizing the write from folio_size(page_folio(page)) instead of the I/O length cause out-of-bounds accesses and dropped pages? When a single-page bio (bv_len == PAGE_SIZE) writes a subpage belonging to a large folio, zram_bio_write() takes the single-page loop and calls zram_write_page(), where folio_size(page_folio(page)) >= ZCOMP_MULTI_PAGES_SIZE is still true. If page is near the end of the folio, reading ZCOMP_MULTI_PAGES_SIZE bytes starting at page reads past the end of the folio, and setting ZRAM_COMP_MULTI_PAGES across index .. index + nr_pages - 1 on an unaligned index corrupts adjacent slots or writes past the end of zram->table. Conversely, if bio_add_page() merges ZCOMP_MULTI_PAGES_NR physically contiguous order-0 pages into one bvec of length ZCOMP_MULTI_PAGES_SIZE, zram_bio_write_multi_pages() advances the iterator by ZCOMP_MULTI_PAGES_SIZE while zram_write_page() sees folio_size() == PAGE_SIZE and stores only the first page. > @@ -2320,9 +2402,10 @@ static int zram_write_page(struct zram *zram, struct page *page, > return ret; > } > > - if (comp_len >= huge_class_size) { > + if (comp_len >= huge_class_size[huge_class_idx]) { > zcomp_cstrm_put(zstrm); > - return write_incompressible_page(zram, page, index); > + return write_incompressible_page(zram, page, index, page_size, > + nr_pages); > } > > slot_lock(zram, index); > @@ -2341,10 +2424,18 @@ static int zram_write_page(struct zram *zram, struct page *page, > > set_slot_size(zram, index, comp_len); > mark_slot_accessed(zram, index); > +#ifdef CONFIG_ZRAM_MULTI_PAGES > + if (nr_pages > 1) { > + int i; > + > + for (i = 0; i < nr_pages; i++) > + set_slot_flag(zram, index + i, ZRAM_COMP_MULTI_PAGES); > + } > +#endif > slot_unlock(zram, index); [Severity: High] Does setting ZRAM_COMP_MULTI_PAGES on index + 1 .. index + nr_pages - 1 without locking or freeing those tail slots corrupt the slot bit-lock and leak existing objects? zram_write_page(), write_same_filled_page(), and write_incompressible_page() only lock and call slot_free() on the head slot index: 1. Because table[index + i].attr.flags aliases table[index + i].__lock, the unlocked non-atomic |= in set_slot_flag(zram, index + i, ...) races with concurrent slot_lock() / slot_unlock() on tail slots and can corrupt ZRAM_ENTRY_LOCK_BIT, hanging waiters in wait_on_bit_lock(). 2. Any single-page zsmalloc handles already stored in the tail slots are never freed before ZRAM_COMP_MULTI_PAGES is set on them, leaking those objects. 3. During a partial write after some sub-slots were already freed by zram_slot_free_notify(), re-setting ZRAM_COMP_MULTI_PAGES on all nr_pages sub-slots marks already-freed swap slots that will never receive another free notification, so the multi-page object can never be freed when the remaining active sub-slots are released. 4. If the head sub-slot was freed first via zram_slot_free_notify() (which clears ZRAM_COMP_MULTI_PAGES on index), a later slot_free(zram, index) sees ZRAM_COMP_MULTI_PAGES clear on index and subtracts 1 instead of ZCOMP_MULTI_PAGES_NR from pages_stored and same_pages, while slot_free_npages() on discard or zs_store() failure leaves ZRAM_COMP_MULTI_PAGES set on the tail slots. 5. On a zsmalloc multi-page size class, zs_store() -> obj_write() calls memcpy_to_page() with off + sizes[0] == ZSMALLOC_MULTI_PAGES_SIZE when an object spans two zpdescs, which triggers VM_BUG_ON(offset + len > PAGE_SIZE) in memcpy_to_page(); similarly, zs_rebind_handle() uses offset_in_page() instead of masking with page_size - 1, corrupting object headers at offsets >= PAGE_SIZE. [ ... ] > +#ifdef CONFIG_ZRAM_MULTI_PAGES > + > +static inline unsigned long zram_multi_pages_head_index(unsigned long index) > +{ > + return index & ~((unsigned long)ZCOMP_MULTI_PAGES_NR - 1); > +} > + > +static inline int __test_multi_pages_comp(struct zram *zram, > + unsigned long index) > +{ > + unsigned long head = zram_multi_pages_head_index(index); > + int i, count = 0; > + > + for (i = 0; i < ZCOMP_MULTI_PAGES_NR; i++) { > + if (test_slot_flag(zram, head + i, ZRAM_COMP_MULTI_PAGES)) > + count++; > + } > + > + return count; > +} [Severity: Medium] Can __test_multi_pages_comp() access past the end of zram->table when disksize is not a multiple of ZCOMP_MULTI_PAGES_SIZE? disksize_store() only aligns disksize with PAGE_ALIGN(disksize). When num_pages is not a multiple of ZCOMP_MULTI_PAGES_NR, an access to the final slots has head + i >= num_pages, so __test_multi_pages_comp() and the set_slot_flag() loops in the multi-page write path read and write past the end of zram->table. > +static inline bool want_multi_pages_comp(struct zram *zram, struct bio *bio) > +{ > + unsigned long index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; > + > + if (bio->bi_io_vec->bv_len >= ZCOMP_MULTI_PAGES_SIZE) > + return true; > + > + slot_lock(zram, index); > + if (__test_multi_pages_comp(zram, index)) { > + slot_unlock(zram, index); > + return true; > + } > + slot_unlock(zram, index); > + > + return false; > +} > + > +static inline bool test_multi_pages_comp(struct zram *zram, struct bio *bio) > +{ > + unsigned long index = bio->bi_iter.bi_sector >> SECTORS_PER_PAGE_SHIFT; > + > + return !!__test_multi_pages_comp(zram, index); > +} [Severity: Critical] Can routing the entire bio through the multi-page or single-page path based only on the first sector crash or corrupt data? want_multi_pages_comp() and test_multi_pages_comp() inspect only the bio's initial sector and bio->bi_io_vec->bv_len before iterating the bio. If an unaligned write bio with bv_len >= ZCOMP_MULTI_PAGES_SIZE or a multi-bvec bio enters the multi-page path and lands on a head slot that holds a single-page (PAGE_SIZE) object, zram_bvec_write_multi_pages_partial() or zram_bvec_read_multi_pages() calls zram_read_page_size(..., ZCOMP_MULTI_PAGES_SIZE) on that single-page slot: - If the single-page head slot has ZRAM_HUGE set (a 1-page ZsHugePage), read_incompressible_page() calls zs_obj_read_sg_begin() with ZCOMP_MULTI_PAGES_SIZE, which takes the two-page branch, gets NULL from get_next_zpdesc(zpdesc), and dereferences NULL in sg_copy_to_buffer(). - If the single-page head slot holds a compressed PAGE_SIZE object during zram_bvec_write_multi_pages_partial(), zcomp_decompress() only fills the first PAGE_SIZE bytes of the unzeroed order-ZCOMP_MULTI_PAGES_ORDER page, and zram_write_page() compresses and stores the remaining uninitialized kernel memory. Conversely, if a read bio starts on a single-page slot and crosses into a multi-page slot group, zram_bio_read() stays in the single-page loop, failing decompression on the head slot and reading the unallocated tail slots as zero-filled pages. > +static inline bool is_multi_pages_partial_io(struct bio_vec *bvec) > +{ > + return bvec->bv_len != ZCOMP_MULTI_PAGES_SIZE; > +} > + > +static int zram_read_partial_from_zspool(struct zram *zram, struct page *page, > + unsigned long index, int offset) > +{ > + struct zs_handle *handle = &zram->table[index].handle; > + struct zcomp_dstrm *zstrm; > + struct scatterlist sg[2]; > + enum zcomp_dstrm_mode mode; > + unsigned int size; > + void *src, *dst; > + u32 prio; > + int ret; > + unsigned long page_size = ZCOMP_MULTI_PAGES_SIZE; > + > + if (test_slot_flag(zram, index, ZRAM_SAME) || > + !zs_handle_is_allocated(handle)) { > + void *mem; > + > + mem = kmap_local_page(page); > + zram_fill_page(mem, PAGE_SIZE, zram->table[index].element); > + kunmap_local(mem); > + return 0; > + } > + > + size = get_slot_size(zram, index); > + zs_obj_read_sg_begin(zram->mem_pool, handle, sg, size); > + > + if (size == page_size) { > + src = kmap_local_page(sg_page(sg)); > + dst = kmap_local_page(page); > + memcpy(dst, src + offset, PAGE_SIZE); > + kunmap_local(dst); > + kunmap_local(src); > + ret = 0; > + } else { > + prio = get_slot_comp_priority(zram, index); > + mode = ZCOMP_DSTRM_EXCLUSIVE; > + if (zcomp_shared_decomp(zram->comps[prio])) > + mode = ZCOMP_DSTRM_SHARED; > + > + zstrm = zcomp_dstrm_get(zram->comps[prio], mode); > + mutex_lock(&zstrm->buf_lock); > + ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size, > + zstrm->decomp_buf, page_size); > + if (!ret) { > + dst = kmap_local_page(page); > + memcpy(dst, zstrm->decomp_buf + offset, PAGE_SIZE); > + kunmap_local(dst); > + } > + mutex_unlock(&zstrm->buf_lock); > + zcomp_dstrm_put(zram->comps[prio], zstrm, mode); > + } > + zs_obj_read_sg_end(zram->mem_pool, handle); > + > + return ret; > +} [Severity: Critical] Does zram_read_partial_from_zspool() ignore bvec->bv_len and bvec->bv_offset? zram_bvec_read_multi_pages_partial() passes only bvec->bv_page and the slot offset to zram_read_partial_from_zspool(), and all three branches above always write PAGE_SIZE bytes to offset 0 of page. When a partial multi-page read has bv.bv_len > PAGE_SIZE (for example, 2 or 3 pages) or a non-zero bvec->bv_offset, zram_bio_read_multi_pages() still advances the bio iterator by bv.bv_len, writing to the wrong offset of bv_page or leaving the remaining pages of the read uninitialized. [ ... ] > +static int zram_bvec_write_multi_pages_partial(struct zram *zram, > + struct bio_vec *bvec, > + unsigned long index, int offset, > + struct bio *bio) > +{ > + struct page *page; > + void *src, *dst; > + int ret; > + > + page = alloc_pages(GFP_NOIO | __GFP_COMP, ZCOMP_MULTI_PAGES_ORDER); > + if (!page) > + return -ENOMEM; > + > + ret = zram_read_page_size(zram, page, index, bio, > + ZCOMP_MULTI_PAGES_SIZE); > + if (!ret) { > + src = kmap_local_page(bvec->bv_page); > + dst = kmap_local_page(page); > + memcpy(dst + offset, src + bvec->bv_offset, bvec->bv_len); > + kunmap_local(dst); > + kunmap_local(src); > + > + atomic64_inc(&zram->stats.zram_bio_write_multi_pages_partial_count); > + ret = zram_write_page(zram, page, index); > + } > + > + __free_pages(page, ZCOMP_MULTI_PAGES_ORDER); > + return ret; > +} [Severity: High] Can two concurrent partial writes to different sub-slots in the same multi-page group overwrite each other? zram_read_page_size() unlocks slot_lock(zram, index) after reading the multi-page into the temporary page, and zram_write_page() only re-acquires slot_lock(zram, index) after compressing the updated buffer. If two CPUs concurrently perform partial writes to different offsets of the same head index, both read the old multi-page before either writes back, so the second writer overwrites the first writer's sub-page with stale data. [Severity: High] Does passing bio as parent to zram_read_page_size() cause a use-after-free of page when ZRAM_WB is set? When test_slot_flag(zram, index, ZRAM_WB) is true, zram_read_page_size() calls read_from_bdev(zram, page, index, blk_idx, parent) with a non-NULL parent bio: drivers/block/zram/zram_drv.c:read_from_bdev() { atomic64_inc(&zram->stats.bd_reads); if (!parent) { if (WARN_ON_ONCE(!IS_ENABLED(ZRAM_PARTIAL_IO))) return -EIO; return read_from_bdev_sync(zram, page, index, blk_idx); } return read_from_bdev_async(zram, page, index, blk_idx, parent); } read_from_bdev_async() submits an asynchronous child bio referencing page (inheriting REQ_OP_WRITE from the write bio) and returns 0 immediately, after which zram_bvec_write_multi_pages_partial() frees page via __free_pages() while the backing-device I/O and completion work are still using it. [ ... ] > @@ -2814,7 +3189,34 @@ static void zram_slot_free_notify(struct block_device *bdev, > return; > } > > +#ifdef CONFIG_ZRAM_MULTI_PAGES > + { > + int comp_count = __test_multi_pages_comp(zram, index); > + > + if (comp_count > 1) { > + clear_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES); > + slot_unlock(zram, index); > + return; > + } > + > + if (comp_count == 1) { > + clear_slot_flag(zram, index, ZRAM_COMP_MULTI_PAGES); > + slot_unlock(zram, index); > + index = zram_multi_pages_head_index(index); > + nr_pages_to_free = ZCOMP_MULTI_PAGES_NR; > + if (!slot_trylock(zram, index)) { > + atomic64_inc(&zram->stats.multi_pages_miss_free); > + return; > + } > + } > + } > +#endif [Severity: High] Can concurrent zram_slot_free_notify() calls on sibling sub-slots leak the multi-page object? __test_multi_pages_comp() counts ZRAM_COMP_MULTI_PAGES across the group while only holding slot_trylock(zram, index) on the current sub-slot: CPU0 CPU1 zram_slot_free_notify(head + 1) zram_slot_free_notify(head + 2) slot_trylock(zram, head + 1) slot_trylock(zram, head + 2) __test_multi_pages_comp() -> 2 __test_multi_pages_comp() -> 2 clear_slot_flag(head + 1, ...) clear_slot_flag(head + 2, ...) slot_unlock(zram, head + 1) slot_unlock(zram, head + 2) return; return; <-- head is never freed In addition, when comp_count == 1, clearing ZRAM_COMP_MULTI_PAGES and unlocking index before slot_trylock(zram, head) means that if slot_trylock(zram, head) fails, the last ZRAM_COMP_MULTI_PAGES flag has already been cleared, and another CPU can also reallocate and write to head in the unlocked window before slot_trylock(zram, head) runs. __test_multi_pages_comp() also does not check whether index itself has ZRAM_COMP_MULTI_PAGES set before treating it as part of a multi-page group. > diff --git a/drivers/block/zram/zram_drv.h b/drivers/block/zram/zram_drv.h > index d64c99b74214b..1e3ebfec972ec 100644 > --- a/drivers/block/zram/zram_drv.h > +++ b/drivers/block/zram/zram_drv.h [ ... ] > @@ -37,7 +42,12 @@ > * > * We use BUILD_BUG_ON() to make sure that zram pageflags don't overflow. > */ > +#ifdef CONFIG_ZRAM_MULTI_PAGES > +#define ZRAM_FLAG_SHIFT (PAGE_SHIFT + CONFIG_ZSMALLOC_MULTI_PAGES_ORDER + 1) > +#else > #define ZRAM_FLAG_SHIFT (PAGE_SHIFT + 1) > +#endif [Severity: Medium] Does ZRAM_FLAG_SHIFT overflow the 32-bit attr.flags field when PAGE_SHIFT + CONFIG_ZSMALLOC_MULTI_PAGES_ORDER > 21? CONFIG_ZSMALLOC_MULTI_PAGES_ORDER allows values up to 9, and enum zram_pageflags defines 10 flag bits above ZRAM_FLAG_SHIFT. On 16K or 64K page configurations with larger orders (for example, PAGE_SHIFT == 16 with order >= 6), __NR_ZRAM_PAGEFLAGS exceeds 32 and triggers BUILD_BUG_ON(__NR_ZRAM_PAGEFLAGS > sizeof(zram_te.attr.flags) * 8) in zram_init(). [ ... ] > @@ -167,4 +190,24 @@ struct zram { > struct dentry *debugfs_dir; > #endif > }; > + > +#ifdef CONFIG_ZRAM_MULTI_PAGES > +#define multi_pages_bvec_iter_offset(bvec, iter) \ > + (mp_bvec_iter_offset((bvec), (iter)) % ZCOMP_MULTI_PAGES_SIZE) > + > +#define multi_pages_bvec_iter_len(bvec, iter) \ > + min_t(unsigned int, mp_bvec_iter_len((bvec), (iter)), \ > + ZCOMP_MULTI_PAGES_SIZE - \ > + multi_pages_bvec_iter_offset((bvec), (iter))) > + > +#define multi_pages_bvec_iter_bvec(bvec, iter) \ > +((struct bio_vec) { \ > + .bv_page = mp_bvec_iter_page((bvec), (iter)), \ > + .bv_len = multi_pages_bvec_iter_len((bvec), (iter)), \ > + .bv_offset = multi_pages_bvec_iter_offset((bvec), (iter)), \ > +}) > + > +#define multi_pages_bio_iter_iovec(bio, iter) \ > + multi_pages_bvec_iter_bvec((bio)->bi_io_vec, (iter)) > +#endif > #endif [Severity: Critical] Does using mp_bvec_iter_page() in multi_pages_bvec_iter_bvec() keep .bv_page stuck at the first page of a large folio across iterations? Unlike bvec_iter_page(), mp_bvec_iter_page(bvec, iter) always returns bvec[iter.bi_idx].bv_page without adding mp_bvec_iter_page_idx(bvec, iter), while multi_pages_bvec_iter_offset() wraps .bv_offset modulo ZCOMP_MULTI_PAGES_SIZE. When a folio larger than ZCOMP_MULTI_PAGES_SIZE (such as a 64KiB or 128KiB folio in a single multi-page bvec) is read or written, every iteration of zram_bio_write_multi_pages() and zram_bio_read_multi_pages() accesses subpage 0 of the folio, repeatedly compressing or overwriting the first ZCOMP_MULTI_PAGES_SIZE bytes and leaving the rest of the folio unread or unwritten. > diff --git a/mm/zsmalloc.c b/mm/zsmalloc.c > index 2d83fcc8610da..81d908d6d5a8d 100644 > --- a/mm/zsmalloc.c > +++ b/mm/zsmalloc.c > @@ -1697,7 +1697,8 @@ static struct zs_handle *find_alloced_obj(struct size_class *class, > offset += class->size * index; > > while (offset < page_size) { > - if (obj_allocated(zpdesc, addr + offset, &handle)) > + handle = obj_allocated(zpdesc, addr + offset); > + if (handle) > break; > > offset += class->size; [Severity: Low] Should this find_alloced_obj() change be folded into the preceding commit "mm: zsmalloc: support objects compressed based on multiple pages" so the series stays bisectable? Also, the commit message ends with "The following data illustrates the time and compressed data for typical anonymous pages gathered from Android phones." without including the data table, and multi_pages_debug_stat_show() is missing a colon or space before "%8llu" on the failed and partial count lines. -- Sashiko AI review · https://sashiko.dev/#/patchset/20261115024455.1465726-1-zhaonanzhe@xiaomi.com?part=2 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [RFC PATCH v4 3/4] zram: backend_zstd: use multi-page size for zstd parameter setup 2026-11-15 2:44 [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages Nanzhe Zhao 2026-11-15 2:44 ` [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages Nanzhe Zhao 2026-11-15 2:44 ` [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages Nanzhe Zhao @ 2026-11-15 2:44 ` Nanzhe Zhao 2026-11-15 2:44 ` [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin Nanzhe Zhao 2026-11-19 6:59 ` [syzbot ci] Re: mTHP-friendly compression in zsmalloc and zram based on multi-pages syzbot ci 4 siblings, 0 replies; 9+ messages in thread From: Nanzhe Zhao @ 2026-11-15 2:44 UTC (permalink / raw) To: Andrew Morton Cc: Barry Song, linux-mm, linux-block, linux-kernel, Jens Axboe, Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner, Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li, David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts, Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee, Nick Terrell, Tangquan Zheng, Bo Zhang, Nanzhe Zhao From: Barry Song <baohua@kernel.org> zstd_setup_params() passes PAGE_SIZE to zstd_get_params() and zstd_get_cparams() as the source size hint. When multi-page compression is enabled, compressing 16 KiB blocks with 4 KiB-tuned parameters limits zstd's match window to a single page and prevents it from finding cross-page redundancy. Pass ZCOMP_MULTI_PAGES_SIZE instead of PAGE_SIZE so that zstd configures its parameters for the multi-page block size. Signed-off-by: Barry Song <baohua@kernel.org> Co-developed-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com> Signed-off-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com> --- drivers/block/zram/backend_zstd.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c index d55615f34300..9438691791ef 100644 --- a/drivers/block/zram/backend_zstd.c +++ b/drivers/block/zram/backend_zstd.c @@ -73,12 +73,12 @@ static int zstd_setup_params(struct zcomp_params *params) goto error; } - zp->cprm = zstd_get_params(params->level, PAGE_SIZE); + zp->cprm = zstd_get_params(params->level, ZCOMP_MULTI_PAGES_SIZE); zp->custom_mem.customAlloc = zstd_custom_alloc; zp->custom_mem.customFree = zstd_custom_free; - prm = zstd_get_cparams(params->level, PAGE_SIZE, + prm = zstd_get_cparams(params->level, ZCOMP_MULTI_PAGES_SIZE, params->dict_sz); if (!params->dict || !params->dict_sz) -- 2.43.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin 2026-11-15 2:44 [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages Nanzhe Zhao ` (2 preceding siblings ...) 2026-11-15 2:44 ` [RFC PATCH v4 3/4] zram: backend_zstd: use multi-page size for zstd parameter setup Nanzhe Zhao @ 2026-11-15 2:44 ` Nanzhe Zhao 2026-11-16 2:46 ` sashiko-bot 2026-11-19 6:59 ` [syzbot ci] Re: mTHP-friendly compression in zsmalloc and zram based on multi-pages syzbot ci 4 siblings, 1 reply; 9+ messages in thread From: Nanzhe Zhao @ 2026-11-15 2:44 UTC (permalink / raw) To: Andrew Morton Cc: Nanzhe Zhao, linux-mm, linux-block, linux-kernel, Jens Axboe, Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner, Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li, David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts, Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee, Nick Terrell, Barry Song, Tangquan Zheng, Bo Zhang Synchronous swap devices such as zram complete swap I/O in the faulting context. With multi-page compression enabled, a 16 KiB block is stored and decompressed as a single object. The swapin allocation inherits the VMA's THP gfp mask, which drops __GFP_DIRECT_RECLAIM for VMAs without MADV_HUGEPAGE. Under memory pressure, the order-2 allocation fails and falls back to order-0, requiring the entire 16 KiB block to be decompressed just to swap in an order-0 page. Allow direct reclaim for order-2 swapin allocation to improve the allocation success rate and avoid partial reads. Signed-off-by: Nanzhe Zhao <zhaonanzhe@xiaomi.com> --- mm/swap_state.c | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/mm/swap_state.c b/mm/swap_state.c index da65d75ceead..912b0e21a781 100644 --- a/mm/swap_state.c +++ b/mm/swap_state.c @@ -462,6 +462,13 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci, if (mpol || !vmf) { folio = folio_alloc_mpol(gfp, order, mpol, ilx, numa_node_id()); } else { + /* + * zsmalloc uses order-2 pages to store compressed data. + * Allow direct reclaim to improve allocation success and + * avoid partial reads. + */ + if (order == 2) + gfp |= __GFP_RECLAIM; address = round_down(vmf->address, PAGE_SIZE << order); folio = vma_alloc_folio(gfp, order, vmf->vma, address); } -- 2.43.0 ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin 2026-11-15 2:44 ` [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin Nanzhe Zhao @ 2026-11-16 2:46 ` sashiko-bot 0 siblings, 0 replies; 9+ messages in thread From: sashiko-bot @ 2026-11-16 2:46 UTC (permalink / raw) To: Nanzhe Zhao, Andrew Morton Cc: linux-mm, Jens Axboe, Sergey Senozhatsky, Minchan Kim, Kairui Song, Johannes Weiner, Nhat Pham, Usama Arif, Yosry Ahmed, Chengming Zhou, Chris Li, David Hildenbrand, Ying Huang, Suren Baghdasaryan, Ryan Roberts, Kanchana P Sridhar, Wajdi K Feghali, Bala Seshasayee, Nick Terrell, Barry Song, Tangquan Zheng, Bo Zhang Thank you for your contribution! Sashiko AI review found 2 potential issues to consider: - [High] mm: __GFP_RECLAIM without __GFP_NORETRY in __swap_cache_alloc() triggers OOM - [Medium] mm: hardcoded order-2 __GFP_RECLAIM in __swap_cache_alloc() bypasses policies -- commit cb747b328cc1a8c3f1c43755cbc281a7af714c90 Author: Nanzhe Zhao <zhaonanzhe@xiaomi.com> mm: swap: allow direct reclaim for synchronous multi-page swapin This commit enables __GFP_RECLAIM for order-2 folio allocations in __swap_cache_alloc() when allocating via vma_alloc_folio() to improve multi-page swapin allocation success. > diff --git a/mm/swap_state.c b/mm/swap_state.c > index da65d75ceead1..912b0e21a781e 100644 > --- a/mm/swap_state.c > +++ b/mm/swap_state.c > @@ -462,6 +462,13 @@ static struct folio *__swap_cache_alloc(struct swap_cluster_info *ci, > if (mpol || !vmf) { > folio = folio_alloc_mpol(gfp, order, mpol, ilx, numa_node_id()); > } else { > + /* > + * zsmalloc uses order-2 pages to store compressed data. > + * Allow direct reclaim to improve allocation success and > + * avoid partial reads. > + */ > + if (order == 2) > + gfp |= __GFP_RECLAIM; > address = round_down(vmf->address, PAGE_SIZE << order); > folio = vma_alloc_folio(gfp, order, vmf->vma, address); > } [Severity: High] Can setting __GFP_RECLAIM without __GFP_NORETRY or __GFP_RETRY_MAYFAIL trigger premature OOM kills on order-2 swapin faults instead of falling back to order 0? Under the default THP defrag policy (madvise), as well as defer, defer+madvise, and never, vma_thp_gfp_mask() returns GFP_TRANSHUGE_LIGHT without __GFP_NORETRY. Because order 2 is within PAGE_ALLOC_COSTLY_ORDER (3), adding __GFP_DIRECT_RECLAIM without __GFP_NORETRY causes __alloc_pages_slowpath() to enter full reclaim/compaction retry loops and invoke __alloc_pages_may_oom() -> out_of_memory() when an order-2 page cannot be formed. The same gfp mask is also passed to mem_cgroup_swapin_charge_folio() -> try_charge_memcg(), so a cgroup with 1 to 3 pages of headroom triggers mem_cgroup_oom() instead of returning -ENOMEM and letting __swap_cache_alloc_folio() fall back to order 0. [Severity: Medium] Should __swap_cache_alloc() unconditionally override the GFP mask for order 2 here? __swap_cache_alloc() is shared by all swap devices even when zram or CONFIG_ZRAM_MULTI_PAGES is not used, and when multi-page compression is enabled its order is configured via CONFIG_ZSMALLOC_MULTI_PAGES_ORDER (1..9) rather than hardcoded to 2. Or'ing __GFP_RECLAIM after thp_shmem_limit_gfp_mask(vma_thp_gfp_mask(vma), gfp) also overrides defrag=never/defer and caller GFP restrictions, and placing it only in the else branch leaves out the mpol path while causing shmem_fault() swapins to behave differently between CONFIG_NUMA=y (where shmem_get_pgoff_policy() returns a non-NULL policy) and CONFIG_NUMA=n (where it returns NULL). -- Sashiko AI review · https://sashiko.dev/#/patchset/20261115024455.1465726-1-zhaonanzhe@xiaomi.com?part=4 ^ permalink raw reply [flat|nested] 9+ messages in thread
* [syzbot ci] Re: mTHP-friendly compression in zsmalloc and zram based on multi-pages 2026-11-15 2:44 [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages Nanzhe Zhao ` (3 preceding siblings ...) 2026-11-15 2:44 ` [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin Nanzhe Zhao @ 2026-11-19 6:59 ` syzbot ci 4 siblings, 0 replies; 9+ messages in thread From: syzbot ci @ 2026-11-19 6:59 UTC (permalink / raw) To: akpm, axboe, bala.seshasayee, baohua, chengming.zhou, chrisl, david, hannes, kanchana.p.sridhar, kasong, linux-block, linux-kernel, linux-mm, minchan, nphamcs, ryan.roberts, senozhatsky, surenb, terrelln, usama.arif, wajdi.k.feghali, ying.huang, yosry, zhangbo56, zhaonanzhe, zhengtangquan Cc: syzbot, syzkaller-bugs syzbot ci has tested the following series [v4] mTHP-friendly compression in zsmalloc and zram based on multi-pages https://lore.kernel.org/all/20261115024455.1465726-1-zhaonanzhe@xiaomi.com * [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages * [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages * [RFC PATCH v4 3/4] zram: backend_zstd: use multi-page size for zstd parameter setup * [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin and found the following issues: * KASAN: vmalloc-out-of-bounds Read in zram_submit_bio * WARNING in __mod_zone_page_state Full report is available here: https://ci.syzbot.org/series/fbfb33eb-ef1c-41ea-aa01-54cbecf7c03c *** KASAN: vmalloc-out-of-bounds Read in zram_submit_bio tree: mm-new URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/akpm/mm.git base: 40a41254dc1cf8babbd9178a1c87535695255bf1 arch: amd64 compiler: Debian clang version 22.1.8 (1~deb13u4), Debian LLD 22.1.8 config: https://ci.syzbot.org/builds/0826548c-74e7-4858-8bd7-d0550838a241/config syz repro: https://ci.syzbot.org/findings/50cc4432-dd6a-4eee-9faf-f854d755d423/syz_repro zram0: detected capacity change from 0 to 8 ================================================================== BUG: KASAN: vmalloc-out-of-bounds in test_slot_flag drivers/block/zram/zram_drv.c:118 [inline] BUG: KASAN: vmalloc-out-of-bounds in __test_multi_pages_comp drivers/block/zram/zram_drv.c:2836 [inline] BUG: KASAN: vmalloc-out-of-bounds in test_multi_pages_comp drivers/block/zram/zram_drv.c:2864 [inline] BUG: KASAN: vmalloc-out-of-bounds in zram_bio_read drivers/block/zram/zram_drv.c:3094 [inline] BUG: KASAN: vmalloc-out-of-bounds in zram_submit_bio+0x2593/0x2790 drivers/block/zram/zram_drv.c:3164 Read of size 4 at addr ffffc9000343c018 by task syz.1.18/5883 CPU: 1 UID: 0 PID: 5883 Comm: syz.1.18 Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 Call Trace: <TASK> dump_stack_lvl+0xe8/0x150 lib/dump_stack.c:120 print_address_description+0x55/0x1e0 mm/kasan/report.c:378 print_report+0x58/0x70 mm/kasan/report.c:482 kasan_report+0x117/0x150 mm/kasan/report.c:595 test_slot_flag drivers/block/zram/zram_drv.c:118 [inline] __test_multi_pages_comp drivers/block/zram/zram_drv.c:2836 [inline] test_multi_pages_comp drivers/block/zram/zram_drv.c:2864 [inline] zram_bio_read drivers/block/zram/zram_drv.c:3094 [inline] zram_submit_bio+0x2593/0x2790 drivers/block/zram/zram_drv.c:3164 __submit_bio+0x27f/0x340 block/blk-core.c:682 __submit_bio_noacct block/blk-core.c:725 [inline] submit_bio_noacct_nocheck+0x4f2/0xb10 block/blk-core.c:793 bh_submit fs/buffer.c:1150 [inline] __bh_read fs/buffer.c:2944 [inline] bh_read_nowait include/linux/buffer_head.h:504 [inline] __block_write_begin_int+0x147c/0x1930 fs/buffer.c:2060 iomap_write_begin+0x1075/0x1530 fs/iomap/buffered-io.c:1105 iomap_write_iter fs/iomap/buffered-io.c:1233 [inline] iomap_file_buffered_write+0x494/0xbb0 fs/iomap/buffered-io.c:1318 blkdev_buffered_write+0xee/0x160 block/fops.c:739 blkdev_write_iter+0x4f5/0x820 block/fops.c:829 new_sync_write fs/read_write.c:595 [inline] vfs_write+0x612/0xba0 fs/read_write.c:687 ksys_pwrite64 fs/read_write.c:794 [inline] __do_sys_pwrite64 fs/read_write.c:802 [inline] __se_sys_pwrite64 fs/read_write.c:799 [inline] __x64_sys_pwrite64+0x210/0x2c0 fs/read_write.c:799 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline] do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7f06c93a8fd9 Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007f06ca353028 EFLAGS: 00000246 ORIG_RAX: 0000000000000012 RAX: ffffffffffffffda RBX: 00007f06c9625fa0 RCX: 00007f06c93a8fd9 RDX: 0000000000000001 RSI: 00002000000007c0 RDI: 0000000000000004 RBP: 00007f06c9437fed R08: 0000000000000000 R09: 0000000000000000 R10: 00000000000000c4 R11: 0000000000000246 R12: 0000000000000000 R13: 00007f06c9626038 R14: 00007f06c9625fa0 R15: 00007ffe748226b8 </TASK> The buggy address belongs to a 1-page vmalloc region starting at 0xffffc9000343c000 allocated at zram_meta_alloc drivers/block/zram/zram_drv.c:2043 [inline] The buggy address belongs to a 1-page vmalloc region starting at 0xffffc9000343c000 allocated at disksize_store+0x11d/0x430 drivers/block/zram/zram_drv.c:3304 The buggy address belongs to the physical page: page: refcount:1 mapcount:0 mapping:0000000000000000 index:0xffff888170b2c000 pfn:0x170b2c flags: 0x57ff00000000000(node=1|zone=2|lastcpupid=0x7ff) raw: 057ff00000000000 0000000000000000 dead000000000122 0000000000000000 raw: ffff888170b2c000 0000000000000000 00000001ffffffff 0000000000000000 page dumped because: kasan: bad access detected page_owner tracks the page as allocated page last allocated via order 0, migratetype Unmovable, gfp_mask 0x2dc2(GFP_KERNEL|__GFP_HIGHMEM|__GFP_ZERO|__GFP_NOWARN), pid 5883, tgid 5882 (syz.1.18), ts 56638265335 set_page_owner include/linux/page_owner.h:33 [inline] post_alloc_hook+0x1f9/0x250 mm/page_alloc.c:1858 prep_new_page mm/page_alloc.c:1866 [inline] get_page_from_freelist+0x2224/0x2290 mm/page_alloc.c:3939 __alloc_frozen_pages_noprof+0x1fe/0x410 mm/page_alloc.c:5463 __alloc_pages_noprof+0x10/0x100 mm/page_alloc.c:5498 alloc_pages_bulk_noprof+0x67a/0x8a0 mm/page_alloc.c:5322 vm_area_alloc_pages mm/vmalloc.c:3948 [inline] __vmalloc_area_node mm/vmalloc.c:4123 [inline] __vmalloc_node_range_noprof+0xad1/0x16f0 mm/vmalloc.c:4323 __vmalloc_node_noprof mm/vmalloc.c:4384 [inline] vzalloc_noprof+0xb2/0xe0 mm/vmalloc.c:4462 zram_meta_alloc drivers/block/zram/zram_drv.c:2043 [inline] disksize_store+0x11d/0x430 drivers/block/zram/zram_drv.c:3304 kernfs_fop_write_iter+0x3a4/0x540 fs/kernfs/file.c:345 new_sync_write fs/read_write.c:595 [inline] vfs_write+0x612/0xba0 fs/read_write.c:687 ksys_write+0x150/0x270 fs/read_write.c:739 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline] do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84 entry_SYSCALL_64_after_hwframe+0x77/0x7f page last free pid 5883 tgid 5882 ts 56638249897 stack trace: reset_page_owner include/linux/page_owner.h:26 [inline] __free_pages_prepare mm/page_alloc.c:1402 [inline] __free_frozen_pages+0xc97/0xd90 mm/page_alloc.c:2956 __kasan_populate_vmalloc_do mm/kasan/shadow.c:393 [inline] __kasan_populate_vmalloc+0x1a8/0x1c0 mm/kasan/shadow.c:424 kasan_populate_vmalloc include/linux/kasan.h:587 [inline] alloc_vmap_area+0xd10/0x1410 mm/vmalloc.c:2190 __get_vm_area_node+0x24a/0x330 mm/vmalloc.c:3334 __vmalloc_node_range_noprof+0x36e/0x16f0 mm/vmalloc.c:4283 __vmalloc_node_noprof mm/vmalloc.c:4384 [inline] vzalloc_noprof+0xb2/0xe0 mm/vmalloc.c:4462 zram_meta_alloc drivers/block/zram/zram_drv.c:2043 [inline] disksize_store+0x11d/0x430 drivers/block/zram/zram_drv.c:3304 kernfs_fop_write_iter+0x3a4/0x540 fs/kernfs/file.c:345 new_sync_write fs/read_write.c:595 [inline] vfs_write+0x612/0xba0 fs/read_write.c:687 ksys_write+0x150/0x270 fs/read_write.c:739 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline] do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84 entry_SYSCALL_64_after_hwframe+0x77/0x7f Memory state around the buggy address: ffffc9000343bf00: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 ffffc9000343bf80: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 >ffffc9000343c000: 00 00 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 ^ ffffc9000343c080: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 ffffc9000343c100: f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 f8 ================================================================== *** WARNING in __mod_zone_page_state tree: mm-new URL: https://kernel.googlesource.com/pub/scm/linux/kernel/git/akpm/mm.git base: 40a41254dc1cf8babbd9178a1c87535695255bf1 arch: amd64 compiler: Debian clang version 22.1.8 (1~deb13u4), Debian LLD 22.1.8 config: https://ci.syzbot.org/builds/0826548c-74e7-4858-8bd7-d0550838a241/config syz repro: https://ci.syzbot.org/findings/02951484-1645-42b3-8301-7ebe9e49ee59/syz_repro ------------[ cut here ]------------ IS_ENABLED(CONFIG_PREEMPT_COUNT) && __lockdep_enabled && (preempt_count() == 0 && this_cpu_read(hardirqs_enabled)) WARNING: mm/vmstat.c:361 at __mod_zone_page_state+0xfc/0x140 mm/vmstat.c:361, CPU#0: syz.0.21/5857 Modules linked in: CPU: 0 UID: 0 PID: 5857 Comm: syz.0.21 Not tainted syzkaller #0 PREEMPT(full) Hardware name: QEMU Standard PC (Q35 + ICH9, 2009), BIOS 1.16.2-debian-1.16.2-1 04/01/2014 RIP: 0010:__mod_zone_page_state+0xfc/0x140 mm/vmstat.c:361 Code: d3 e8 09 65 43 88 1c 34 5b 41 5c 41 5d 41 5e 41 5f 5d c3 cc cc cc cc cc 48 89 df 4c 89 fe 89 ea e8 59 00 00 00 31 db eb ce 90 <0f> 0b 90 eb 8b 48 c7 c7 00 44 26 90 4c 89 f6 e8 70 4d 10 03 e9 34 RSP: 0018:ffffc90003a2e2e8 EFLAGS: 00010202 RAX: 0000000000000001 RBX: 0000000000000001 RCX: ffff888103b33c00 RDX: 0000000000000001 RSI: 0000000000000009 RDI: ffff88815fffbb00 RBP: 0000000000000009 R08: ffffea000080de07 R09: 1ffffd4000101bc0 R10: dffffc0000000000 R11: fffff94000101bc1 R12: ffffffff94e3d1c0 R13: dffffc0000000000 R14: 0000000000000009 R15: ffff88815fffbb00 FS: 00007fca0f3e26c0(0000) GS:ffff88818be05000(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 CR2: 00007fca0e471a20 CR3: 00000001bab1c000 CR4: 00000000000006f0 Call Trace: <TASK> __zone_stat_add_folio include/linux/vmstat.h:437 [inline] zpdesc_inc_zone_page_state mm/zsmalloc.c:346 [inline] alloc_zspage+0x73c/0x1510 mm/zsmalloc.c:1121 zs_store+0x906/0xd50 mm/zsmalloc.c:1407 zswap_compress mm/zswap.c:953 [inline] zswap_store_page mm/zswap.c:1542 [inline] zswap_store+0x12aa/0x2230 mm/zswap.c:1657 swap_writeout+0x9bc/0x1700 mm/page_io.c:248 pageout mm/vmscan.c:619 [inline] shrink_folio_list+0x3135/0x4e90 mm/vmscan.c:1392 reclaim_folio_list+0x141/0x4a0 mm/vmscan.c:2176 reclaim_pages+0x452/0x530 mm/vmscan.c:2213 madvise_cold_or_pageout_pte_range+0x1b7e/0x2390 mm/madvise.c:574 walk_pmd_range mm/pagewalk.c:147 [inline] walk_pud_range mm/pagewalk.c:237 [inline] walk_p4d_range mm/pagewalk.c:278 [inline] walk_pgd_range+0x102d/0x1dc0 mm/pagewalk.c:319 __walk_page_range+0x16d/0x720 mm/pagewalk.c:427 walk_page_range_vma_unsafe+0x309/0x410 mm/pagewalk.c:735 madvise_pageout_page_range mm/madvise.c:633 [inline] madvise_pageout mm/madvise.c:658 [inline] madvise_vma_behavior+0x2db7/0x47f0 mm/madvise.c:1393 madvise_walk_vmas+0x576/0xb10 mm/madvise.c:1751 madvise_do_behavior+0x385/0x540 mm/madvise.c:1947 do_madvise+0x33a/0x3c0 mm/madvise.c:2045 __do_sys_madvise mm/madvise.c:2054 [inline] __se_sys_madvise mm/madvise.c:2052 [inline] __x64_sys_madvise+0xa6/0xc0 mm/madvise.c:2052 do_syscall_x64 arch/x86/entry/syscall_64.c:61 [inline] do_syscall_64+0x166/0x520 arch/x86/entry/syscall_64.c:84 entry_SYSCALL_64_after_hwframe+0x77/0x7f RIP: 0033:0x7fca0e5a8fd9 Code: ff c3 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 44 00 00 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 e8 ff ff ff f7 d8 64 89 01 48 RSP: 002b:00007fca0f3e2028 EFLAGS: 00000246 ORIG_RAX: 000000000000001c RAX: ffffffffffffffda RBX: 00007fca0e825fa0 RCX: 00007fca0e5a8fd9 RDX: 0000000000000015 RSI: 0000000000600003 RDI: 0000200000000000 RBP: 00007fca0e637fed R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000 R13: 00007fca0e826038 R14: 00007fca0e825fa0 R15: 00007fff2ba215c8 </TASK> *** If these findings have caused you to resend the series or submit a separate fix, please add the following tag to your commit message: Tested-by: syzbot@syzkaller.appspotmail.com --- This report is generated by a bot. It may contain errors. syzbot ci engineers can be reached at syzkaller@googlegroups.com. ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-11-19 6:59 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-15 2:44 [RFC PATCH v4 0/4] mTHP-friendly compression in zsmalloc and zram based on multi-pages Nanzhe Zhao 2026-11-15 2:44 ` [RFC PATCH v4 1/4] mm: zsmalloc: support objects compressed based on multiple pages Nanzhe Zhao 2026-11-16 2:46 ` sashiko-bot 2026-11-15 2:44 ` [RFC PATCH v4 2/4] zram: support compression at the granularity of multi-pages Nanzhe Zhao 2026-11-16 2:46 ` sashiko-bot 2026-11-15 2:44 ` [RFC PATCH v4 3/4] zram: backend_zstd: use multi-page size for zstd parameter setup Nanzhe Zhao 2026-11-15 2:44 ` [RFC PATCH v4 4/4] mm: swap: allow direct reclaim for synchronous multi-page swapin Nanzhe Zhao 2026-11-16 2:46 ` sashiko-bot 2026-11-19 6:59 ` [syzbot ci] Re: mTHP-friendly compression in zsmalloc and zram based on multi-pages syzbot ci
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox