* [PATCH v2 0/8] Batched zswap_store() framework with compression batching
@ 2026-10-28 3:04 Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
` (7 more replies)
0 siblings, 8 replies; 16+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:04 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
v2: Batched zswap_store() framework with compression batching
=============================================================
This series is a spin-off of "v14: zswap compression batching with optimized
iaa_crypto driver" [1], and focuses on the generic crypto_acomp and zswap
compression batching enablers. Resuming work on this series following a gap
since v14 (January 2026). Please note that my email address and affiliation have
changed from kanchana.p.sridhar@intel.com to kanchanapsridhar2026@gmail.com. I
have retained the original Intel sign-offs on prior patches for historical
attribution while signing off v2 of this series with my current address.
The main focus of v2 was to establish a reliable and accurate performance
measurement methodology, that addressed the problems with variation in v1. This
is accomplished and used to validate v2, by reducing usemem to 2 processes and
enabling PMD THP. As the data shows, we get exactly the same "2048kB-zswpout"
counts in the baseline and with this series.
To summarize the main contribution of this patch series: Batching is introduced
in zswap_store() of large folios: we carve out a large folio into batches of
ZSWAP_MAX_BATCH_SIZE (8 pages) for software compressors, and store each
batch. Moving the fundamental operations that need to be happen to store a page
to now work at the granularity of a batch offers opportunities to optimize them
from these perspectives:
1) Utilize existing kernel API that minimize obtaining locks for bulk
operations, e.g. kmem_cache_alloc_bulk()/kmem_cache_free_bulk().
2) We keep the working set for each of the core store functionalities
(alloc/initialize zswap entries, compress pages, store pages in zsmalloc,
pool/objcg reference increments, objcg charging) to be focused on a
batch, thereby minimizing cache evictions and benefiting from cache
locality.
3) Along the lines of (1), the main zswap_compress() batching routine obtains
the comp_ctx mutex lock once per batch, and releases it after it has
processed all pages in that batch.
4) Any zswap store functionality that works on a batch can potentially be
optimized via bulk-processing/parallelized as a next step, to further improve
performance.
Performance data: usemem with 56% over-commit:
==============================================
Validation was done with mm-unstable as of 10-25-2026 (Baseline) and with this
patch series. To minimize variation in the data, I made these changes to my
performance measurement methodology:
1) Reduced the number of usemem processes to 2. Each process tried to allocate
2400M, in a 3G cgroup memory.high budget, effectively accomplishing a 56.25%
over-commit.
2) Increased the runtime of the workload to ~9 minutes by enabling PMD THP,
i.e., 2048kB mTHP folios.
3) The use of PMD THP forces zswap_store()/zswap_compress() latency to be in the
critical path, and thus enables very reliable measurements with bpftrace.
4) I used lhist() instead of hist() in bpftrace to gather latencies at a finer
level of granularity and accuracy.
Testing was done on a Chromebook with a QEMU "vng" VM running in Crostini.
Chromebook Diagnostics reports:
Intel(R) Core(TM) Ultra 5 115U (10 CPUs, 4.200GHz)
RAM: 16GB
vng setup:
Memory: 7G
Swap: 3G
Pinned CPUs: 2 (E-Cores: 2,3)
zswap compressor:
ZSTD
mTHP enabled:
2048kB (PMD THP)
I used the help of an LLM to write the bpftrace data post-processing scripts
that generate latency percentiles.
vm-scalability/usemem setup:
cgroup memory.high: 3G
2 usemem processes, each allocates 2400M anonymous memory, reads the
Silesia dataset [2] from a pre-fork-allocated buffer into the 2400M,
writes to each unsigned long in the 2400M, sleeps for 10 sec, then
frees the memory and exits.
usemem -w --read-silesia -O -b 1 -s 10 -n 2 2400M
usemem metrics are reported from a single run after starting the QEMU VM.
Further, I have tabulated the bpftrace p50/p95/p99 latencies with these labels:
"zswap_store_higher_order":
zswap_store() of PMD folios.
"zswap_store_order_0":
zswap_store() of order-0 folios.
"zswap_batch_compress_per_page":
zswap_compress() per-page latency of store batches generated from PMD
folios.
"zswap_single_compress_per_page":
zswap_compress() per-page latency of order-0 folios.
----------------------------------------------------------------------------
usemem Baseline zswap zswap Change with
PMD THP mm-unstable store compress compress
10-25-2026 batching batching batching
(patch 6) (patch 8) %
----------------------------------------------------------------------------
usemem process 1 6,353 7,240 7,428
Throughput (KB/s)
usemem process 2 6,352 7,236 7,422
Throughput (KB/s)
Average Throughput (KB/s) 6,353 7,238 7,425 17%
----------------------------------------------------------------------------
usemem process 1 484,580 308,903 257,230
Time to free memory (usecs)
usemem process 2 518,928 438,585 446,958
Time to free memory (usecs)
Average Time to 501,754 373,744 352,094 -30%
Free Memory (usecs)
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
usemem Baseline zswap zswap Change
PMD THP mm-unstable store compress with
BPFTRACE 10-25-2026 batching batching compress
(patch 6) (patch 8) batching
(ns)
-------------------------------------------------------------------------------
zswap_batch_compress_per_page (ns):
p50 40,086 1,569
p95 54,689 -64,505
p99 83,248 -88,488
-------------------------------------------------------------------------------
zswap_single_compress_per_page (ns):
p50 38,517 38,352 36,650 -1,867
p95 119,194 121,174 133,678 14,484
p99 171,736 158,601 169,010 -2,726
-------------------------------------------------------------------------------
zswap_store_higher_order (ns):
p50 21,012,222 21,322,885 20,176,667 -835,556
p95 25,643,500 26,204,375 24,485,000 -1,158,500
p99 28,567,400 38,577,750 26,158,000 -2,409,400
-------------------------------------------------------------------------------
zswap_store_order_0 (ns):
p50 37,601 36,994 37,568 -33
p95 123,731 137,034 128,366 4,635
p99 159,965 169,282 175,618 15,653
-------------------------------------------------------------------------------
vmstat summary:
-------------------------------------------------------------------------------
zswpin 1,104,226 993,412 1,091,146
zswpout 1,688,181 1,582,664 1,669,671
2048kB-alloc 2,500 2,505 2,500
2048kB-zswpout 1,535 1,535 1,535
2048kB-swpout-fallback 927 929 875
-------------------------------------------------------------------------------
Detailed vmstat & zswap stats after workload finishes:
======================================================
------------------------------------------------------
usemem Baseline zswap
PMD THP mm-unstable compress
10-25-2026 batching
(patch 8)
------------------------------------------------------
vmstat:
------------------------------------------------------
pswpin 0 0
pswpout 0 0
thp_swpout 0 0
thp_swpout_fallback 927 875
swpin_zero 0 0
swpout_zero 0 0
zswpin 1,104,226 1,091,146
zswpout 1,688,181 1,669,671
zswpwb 0 0
nrswpin 0 0
nrswpout 0 0
pgmajfault 173,554 173,774
nr_free_pages 1,753,879 1,755,195
nr_free_pages_blocks 1,651,712 1,667,072
2048kB-alloc 2,500 2,500
2048kB-zswpout 1,535 1,535
2048kB-swpout 0 0
2048kB-swpout-fallback 927 875
2048kB-swpin 0 0
memory.current 352,256 499,712
memory.swap.current 0 0
------------------------------------------------------
zswap stats:
------------------------------------------------------
pool_limit_hit 0 0
written_back_pages 0 0
reject_alloc_fail 0 0
reject_kmemcache_fail 0 0
reject_reclaim_fail 0 0
reject_compress_fail 0 0
reject_compress_poor 0 0
decompress_fail 0 0
stored_incompressible_pages 0 0
stored_pages 0 0
pool_total_size 0 0
------------------------------------------------------
Conclusions based on the data:
==============================
zswap compress batching improves on both, zswap_store() latency and
zswap_compress() latency wrt the baseline, pretty consistently; except for p50
batch compress latency which can be optimized further in subsequent
patches. The batching framework established in this series allows us to make
locking and bulk-processing improvements in each stage of zswap_store(). One of
the ideas that has been showing very promising results is to replace zsmalloc
with SLUB compressed caches and will be submitted separately. This is expected
to resolve the p50 batch compress latency issue since it will be directly
optimizing zswap_update_entries().
The speedup with zswap store batching of large folios has a propelling effect
on other parts of mm, especially, by enabling large folios to be swapped out
without splitting, as the data shows. This enables other mm components that are
optimized for large folios, such as munmap(), folio refcounts/mapcounts
updates, PTE invalidation upon freeing memory, zswap_invalidate(); which
benefit from xarray offsets/range continuity. As a result, subsequent large
folio swapouts are more likely to find contiguous swap slots, there are fewer
large folio splits during reclaim and the cycle continues. As the data
demonstrates, this directly contributes to more memory over-commit
opportunity.
[1]: https://patchwork.kernel.org/project/linux-mm/cover/20260125033537.334628-1-kanchana.p.sridhar@intel.com/
[2]: https://wanos.co/assets/silesia.tar
Changes since v1:
=================
1) Rebased to mm-unstable as of 10-25-2026, commit a2abad0d7f45.
2) Addressed Herbert's comments on checking for the req->unit_size being equal
to the req->slen to identify singleton acomp_reqs and to quickly
short-circuit with setting the dst SG list length instead of going through
the request chaining flow. Further, the most common code path is the very
first check, with the short-circuit, in crypto_acomp_compress().
3) Addressed Yosry's comments wrt:
- struct zswap_entry "length" field build-time assertion to have sufficient
bit-fields for the architecture's PAGE_SIZE.
- Use kmem_cache_alloc() instead of kmem_cache_alloc_node() in
zswap_entry_cache_alloc().
4) Established a reliable and accurate performance validation methodology, based
on Yosry's comments.
5) Refactored the sub-components of zswap_store_pages() where each routine
processes a batch. The new zswap_account_stores() gets refs, updates
statistics and charges cgroup for the batch, rather than for a page at a
time. zswap_store_pages() is more readable and maintainable. This batching
framework can be immediately useful for implementing more ideas such as SLUB
bulk allocation of compressed data, reclaiming a batch of folios from
shrink_folio_list(), etc.
Changes since [1]:
==================
1) Better cacheline placement of struct zswap_pool and struct zswap_entry
members for temporal memory access efficiency in zswap store/load hot paths,
while minimizing holes.
2) Refactored the core compression batching functionality to a separate
__zswap_compress() function, which is called by zswap_compress(), as per
Nhat's and Yosry's comments. Hopefully this helps explain the outer "store
batching" zswap_compress() vis-a-vis the inner "compress batching"
__zswap_compress() distinction.
3) The acomp_ctx is locked once and is passed to the routine that compresses a
batch, per the algorithm's batch-size. Similarly, the folio's node id and
writeback_enabled status are computed once and passed to the routines that
implement the different store functionalities, to save latency by avoid
localized recomputes of information that doesn't change.
4) Moved the folio node id being stored in zswap_entry to a separate patch, per
Yosry's comment.
5) Addressed the kunmap_local() issue flagged by Jixin He in the
IS_ERR_VALUE(handle) error path
(https://patchwork.kernel.org/comment/27095925) - thanks!
6) Moved comments inline, per Yosry's suggestions.
Thanks,
Kanchana
Kanchana P. Sridhar (8):
crypto: acomp - Define a unit_size in struct acomp_req to enable
batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE.
crypto: acomp - Add bit to indicate segmentation support.
crypto: acomp - Add trivial segmentation wrapper.
crypto: acomp - Add API to get a compression algorithm's batch-size.
mm: zswap: Store the folio's node id in the zswap_entry.
mm: zswap: Store large folios in batches.
mm: zswap: Batched zswap_compress() for storing large folios in
batches.
mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
crypto/acompress.c | 53 ++-
include/crypto/acompress.h | 64 ++-
include/crypto/algapi.h | 5 +
include/crypto/internal/acompress.h | 10 +-
include/linux/crypto.h | 5 +-
mm/zswap.c | 635 ++++++++++++++++++++--------
6 files changed, 592 insertions(+), 180 deletions(-)
--
2.39.5
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
@ 2026-10-28 3:04 ` Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 2/8] crypto: acomp - Add bit to indicate segmentation support Kanchana P. Sridhar
` (6 subsequent siblings)
7 siblings, 0 replies; 16+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:04 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
We add a new @unit_size data member to struct acomp_req along with a
helper function acomp_request_set_unit_size() for kernel modules to set
the unit size to use while breaking down the request's src/dst
scatterlists.
An acomp_alg can implement batching by using the @req->unit_size to
break down the SG lists passed in via @req->dst and/or @req->src, to
submit individual @req->slen/@req->unit_size compress jobs or
@req->dlen/@req->unit_size decompress jobs, for batch compression and
batch decompression respectively.
In case of batch compression, the folio's pages for the batch can be
retrieved from the @req->src scatterlist by using a struct sg_page_iter
after determining the number of pages as @req->slen/@req->unit_size.
Please note that acomp_request_set_callback() sets the @req->unit_size
to 0.
In zswap_acomp_req_init(), after the call to
zswap_set_acomp_req_callback(), we set the unit size for zswap to
PAGE_SIZE.
acomp_request_set_unit_size(areq->req, PAGE_SIZE);
Suggested-by: Herbert Xu <herbert@gondor.apana.org.au>
Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
include/crypto/acompress.h | 49 ++++++++++++++++++++++++++++++++++++++
mm/zswap.c | 6 +++++
2 files changed, 55 insertions(+)
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index 5d5358dfab73..6462fc46f4ec 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -79,6 +79,7 @@ struct acomp_req_chain {
* @dvirt: Destination virtual address
* @slen: Size of the input buffer
* @dlen: Size of the output buffer and number of bytes produced
+ * @unit_size: Unit size for the request for use in batching
* @chain: Private API code data, do not use
* @__ctx: Start of private context data
*/
@@ -94,6 +95,7 @@ struct acomp_req {
};
unsigned int slen;
unsigned int dlen;
+ unsigned int unit_size;
struct acomp_req_chain chain;
@@ -329,9 +331,56 @@ static inline void acomp_request_set_callback(struct acomp_req *req,
{
flgs &= ~CRYPTO_ACOMP_REQ_PRIVATE;
flgs |= req->base.flags & CRYPTO_ACOMP_REQ_PRIVATE;
+ req->unit_size = 0;
crypto_request_set_callback(&req->base, flgs, cmpl, data);
}
+/**
+ * acomp_request_set_unit_size() -- Sets the unit size for the request.
+ *
+ * This is a helper function that enables batching for zswap, IPComp, etc.
+ * It allows multiple independent compression (or decompression) operations to
+ * be submitted in a single request's SG lists, where each SG list ("segment")
+ * is processed independently. The unit size helps derive segments from a
+ * single request. crypto_acomp does not expect the segments to be related in
+ * any way.
+ *
+ * Example usage model:
+ *
+ * A module such as zswap that's configured to use a batching compressor, can
+ * accomplish batch compression of "nr_pages" with crypto_acomp by creating an
+ * output SG table for the batch, initialized to contain "nr_pages" SG
+ * lists. Each scatterlist is mapped to the nth destination buffer for the
+ * batch. Depending on whether the @req is used for batch compress/decompress,
+ * zswap must set the @req's source/destination length to be
+ * "nr_pages * @req->unit_size" respectively.
+ *
+ * An acomp_alg can implement batch compression by using the @req->unit_size
+ * to break down the SG lists passed in via @req->dst to submit individual
+ * "@req->slen/@req->unit_size" compress jobs to be processed as a batch.
+ *
+ * Similarly, zswap can implement batch decompression by passing an
+ * SG table with "nr_pages" SG lists via @req->src to process
+ * "@req->dlen/@req->unit_size" decompress jobs as a batch.
+ *
+ * This API must be called after acomp_request_set_callback(),
+ * which sets @req->unit_size to 0. This makes it easy for users of
+ * crypto_acomp to rely on a default of not opting in to batching.
+ * Users such as zswap opt in to batching by defining @req->unit_size
+ * to a non-zero value for use by acomp_algs supporting batching.
+ *
+ * @du would be PAGE_SIZE for zswap, it could be the MTU for IPsec.
+ *
+ * @req: asynchronous compress/decompress request
+ * @du: data unit size of the input/output buffer scatterlist.
+ */
+static inline void acomp_request_set_unit_size(struct acomp_req *req,
+ unsigned int du)
+{
+ req->unit_size = du;
+}
+
+
/**
* acomp_request_set_params() -- Sets request parameters
*
diff --git a/mm/zswap.c b/mm/zswap.c
index ad01c691a263..e1d57a7d9595 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -826,6 +826,12 @@ static int zswap_acomp_req_init(struct zswap_acomp_req *areq,
crypto_init_wait(&areq->wait);
zswap_set_acomp_req_callback(areq->req, &areq->wait);
+ /*
+ * This should be called after acomp_request_set_callback(),
+ * which initializes req->unit_size to 0.
+ */
+ acomp_request_set_unit_size(areq->req, PAGE_SIZE);
+
mutex_init(&areq->mutex);
return 0;
}
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 2/8] crypto: acomp - Add bit to indicate segmentation support.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
@ 2026-10-28 3:04 ` Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 3/8] crypto: acomp - Add trivial segmentation wrapper Kanchana P. Sridhar
` (5 subsequent siblings)
7 siblings, 0 replies; 16+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:04 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
This patch adds segmentation support for compression.
Add a bit to the crypto_alg flags to indicate support for segmentation.
Also add a helper for acomp to test whether a given tfm supports
segmentation.
Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
include/crypto/algapi.h | 5 +++++
include/crypto/internal/acompress.h | 5 +++++
include/linux/crypto.h | 3 +++
3 files changed, 13 insertions(+)
diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
index 05deea9dac5e..7d406cfe5751 100644
--- a/include/crypto/algapi.h
+++ b/include/crypto/algapi.h
@@ -280,6 +280,11 @@ static inline bool crypto_tfm_req_virt(struct crypto_tfm *tfm)
return tfm->__crt_alg->cra_flags & CRYPTO_ALG_REQ_VIRT;
}
+static inline bool crypto_tfm_req_seg(struct crypto_tfm *tfm)
+{
+ return tfm->__crt_alg->cra_flags & CRYPTO_ALG_REQ_SEG;
+}
+
static inline u32 crypto_request_flags(struct crypto_async_request *req)
{
return req->flags & ~CRYPTO_TFM_REQ_ON_STACK;
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 9cd37df32dc4..93ae59623f13 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -189,6 +189,11 @@ static inline bool crypto_acomp_req_virt(struct crypto_acomp *tfm)
return crypto_tfm_req_virt(&tfm->base);
}
+static inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm)
+{
+ return crypto_tfm_req_seg(&tfm->base);
+}
+
void crypto_acomp_free_streams(struct crypto_acomp_streams *s);
int crypto_acomp_alloc_streams(struct crypto_acomp_streams *s);
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index b7c97f1c47c9..78c699ac3f82 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -139,6 +139,9 @@
/* Set if the algorithm cannot have a fallback (e.g., phmac). */
#define CRYPTO_ALG_NO_FALLBACK 0x00080000
+/* Set if the algorithm supports segmentation. */
+#define CRYPTO_ALG_REQ_SEG 0x00100000
+
/* The high bits 0xff000000 are reserved for type-specific flags. */
/*
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 3/8] crypto: acomp - Add trivial segmentation wrapper.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 2/8] crypto: acomp - Add bit to indicate segmentation support Kanchana P. Sridhar
@ 2026-10-28 3:04 ` Kanchana P. Sridhar
2026-10-29 3:05 ` sashiko-bot
2026-10-28 3:04 ` [PATCH v2 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
` (4 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:04 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
This patch provides a wrapper for existing algorithms so that they can
accept a single segment while returning the compressed length or error
through the dst SG list length.
This trivial segmentation wrapper only supports compression with a
segment count of exactly one.
The reason is that the first user zswap will only allocate the
extra memory if the underlying algorithm supports segmentation,
and otherwise only one segment will be given at a time.
Having this wrapper means that the same calling convention can
be used for all algorithms, regardless of segmentation support.
To gain latency in the zswap_store() hot path impacted by this patch:
1) The most common code path is invoked first in
crypto_acomp_compress().
2) The helper functions used in (1) are made __always_inline.
Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
crypto/acompress.c | 39 ++++++++++++++++++++++++-----
include/crypto/acompress.h | 3 ++-
include/crypto/internal/acompress.h | 4 +--
include/linux/crypto.h | 2 +-
4 files changed, 38 insertions(+), 10 deletions(-)
diff --git a/crypto/acompress.c b/crypto/acompress.c
index c8b4b8589769..c55b0e8eff3d 100644
--- a/crypto/acompress.c
+++ b/crypto/acompress.c
@@ -166,14 +166,20 @@ static void acomp_save_req(struct acomp_req *req, crypto_completion_t cplt)
state->compl = req->base.complete;
state->data = req->base.data;
+ state->unit_size = req->unit_size;
+ state->flags = req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
+ CRYPTO_ACOMP_REQ_DST_VIRT);
+
req->base.complete = cplt;
req->base.data = req;
+ req->unit_size = 0;
}
static void acomp_restore_req(struct acomp_req *req)
{
req->base.complete = req->chain.compl;
req->base.data = req->chain.data;
+ req->unit_size = req->chain.unit_size;
}
static void acomp_reqchain_virt(struct acomp_req *req)
@@ -192,9 +198,6 @@ static void acomp_virt_to_sg(struct acomp_req *req)
{
struct acomp_req_chain *state = &req->chain;
- state->flags = req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
- CRYPTO_ACOMP_REQ_DST_VIRT);
-
if (acomp_request_src_isvirt(req)) {
unsigned int slen = req->slen;
const u8 *svirt = req->svirt;
@@ -242,6 +245,10 @@ static int acomp_reqchain_finish(struct acomp_req *req, int err)
{
acomp_reqchain_virt(req);
acomp_restore_req(req);
+
+ if (req->unit_size)
+ req->dst->length = unlikely(err) ? err : req->dlen;
+
return err;
}
@@ -266,10 +273,13 @@ static int acomp_do_req_chain(struct acomp_req *req, bool comp)
{
int err;
+ if (unlikely(req->unit_size && req->slen > req->unit_size))
+ return -ENOSYS;
+
acomp_save_req(req, acomp_reqchain_done);
err = acomp_do_one_req(req, comp);
- if (err == -EBUSY || err == -EINPROGRESS)
+ if (unlikely(err == -EBUSY || err == -EINPROGRESS))
return err;
return acomp_reqchain_finish(req, err);
@@ -279,10 +289,27 @@ int crypto_acomp_compress(struct acomp_req *req)
{
struct crypto_acomp *tfm = crypto_acomp_reqtfm(req);
+ if (likely((req->unit_size == req->slen) && acomp_request_issg(req) &&
+ !crypto_acomp_req_seg(tfm) && !acomp_req_on_stack(req))) {
+ int err = tfm->compress(req);
+
+ req->dst->length = unlikely(err) ? err : req->dlen;
+ return err;
+ }
+
if (acomp_req_on_stack(req) && acomp_is_async(tfm))
return -EAGAIN;
+
+ if (req->unit_size && acomp_request_issg(req)) {
+ if (!crypto_acomp_req_seg(tfm))
+ return acomp_do_req_chain(req, true);
+
+ return tfm->compress(req);
+ }
+
if (crypto_acomp_req_virt(tfm) || acomp_request_issg(req))
- return crypto_acomp_reqtfm(req)->compress(req);
+ return tfm->compress(req);
+
return acomp_do_req_chain(req, true);
}
EXPORT_SYMBOL_GPL(crypto_acomp_compress);
@@ -294,7 +321,7 @@ int crypto_acomp_decompress(struct acomp_req *req)
if (acomp_req_on_stack(req) && acomp_is_async(tfm))
return -EAGAIN;
if (crypto_acomp_req_virt(tfm) || acomp_request_issg(req))
- return crypto_acomp_reqtfm(req)->decompress(req);
+ return tfm->decompress(req);
return acomp_do_req_chain(req, false);
}
EXPORT_SYMBOL_GPL(crypto_acomp_decompress);
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index 6462fc46f4ec..e64cf422510b 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -67,6 +67,7 @@ struct acomp_req_chain {
struct folio *dfolio;
};
u32 flags;
+ u32 unit_size;
};
/**
@@ -294,7 +295,7 @@ static inline void *acomp_request_extra(struct acomp_req *req)
return (void *)((char *)req + len);
}
-static inline bool acomp_req_on_stack(struct acomp_req *req)
+static __always_inline bool acomp_req_on_stack(struct acomp_req *req)
{
return crypto_req_on_stack(&req->base);
}
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 93ae59623f13..0c0d3fc8a9fc 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -146,7 +146,7 @@ void crypto_unregister_acomp(struct acomp_alg *alg);
int crypto_register_acomps(struct acomp_alg *algs, int count);
void crypto_unregister_acomps(struct acomp_alg *algs, int count);
-static inline bool acomp_request_issg(struct acomp_req *req)
+static __always_inline bool acomp_request_issg(struct acomp_req *req)
{
return !(req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
CRYPTO_ACOMP_REQ_DST_VIRT));
@@ -189,7 +189,7 @@ static inline bool crypto_acomp_req_virt(struct crypto_acomp *tfm)
return crypto_tfm_req_virt(&tfm->base);
}
-static inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm)
+static __always_inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm)
{
return crypto_tfm_req_seg(&tfm->base);
}
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 78c699ac3f82..0628b62e4946 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -492,7 +492,7 @@ static inline bool crypto_tfm_is_async(struct crypto_tfm *tfm)
return tfm->__crt_alg->cra_flags & CRYPTO_ALG_ASYNC;
}
-static inline bool crypto_req_on_stack(struct crypto_async_request *req)
+static __always_inline bool crypto_req_on_stack(struct crypto_async_request *req)
{
return req->flags & CRYPTO_TFM_REQ_ON_STACK;
}
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
` (2 preceding siblings ...)
2026-10-28 3:04 ` [PATCH v2 3/8] crypto: acomp - Add trivial segmentation wrapper Kanchana P. Sridhar
@ 2026-10-28 3:04 ` Kanchana P. Sridhar
2026-10-29 3:06 ` sashiko-bot
2026-10-28 3:05 ` [PATCH v2 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
` (3 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:04 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
This commit adds a @batch_size data member to struct acomp_alg.
An acomp_alg compression algorithm that supports batching of
compressions and decompressions must provide a @batch_size greater
than 1, representing the maximum batch-size that the compressor
supports, so that kernel users of crypto_acomp, such as zswap, can
allocate resources for submitting multiple compress/decompress jobs that
can be batched, and invoke batching of [de]compressions.
The new crypto_acomp_batch_size() API queries the crypto_acomp's
acomp_alg for the batch-size. If the acomp_alg has registered a
@batch_size greater than 1, this is returned. If not, a default of "1"
is returned.
zswap can invoke crypto_acomp_batch_size() to query the maximum number
of requests that can be batch [de]compressed. Based on this, zswap
can use the minimum of any zswap-specific upper limits for batch-size
and the compressor's max @batch_size, to allocate batching resources.
This enables zswap to compress/decompress multiple pages as a batch
to improve swapout/swapin performance and memory savings.
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
crypto/acompress.c | 14 ++++++++++++++
include/crypto/acompress.h | 12 ++++++++++++
include/crypto/internal/acompress.h | 3 +++
3 files changed, 29 insertions(+)
diff --git a/crypto/acompress.c b/crypto/acompress.c
index c55b0e8eff3d..8e285cb7477e 100644
--- a/crypto/acompress.c
+++ b/crypto/acompress.c
@@ -326,6 +326,20 @@ int crypto_acomp_decompress(struct acomp_req *req)
}
EXPORT_SYMBOL_GPL(crypto_acomp_decompress);
+unsigned int crypto_acomp_batch_size(struct crypto_acomp *tfm)
+{
+ if (acomp_is_async(tfm) &&
+ (crypto_comp_alg_common(tfm)->base.cra_flags & CRYPTO_ALG_TYPE_ACOMPRESS)) {
+ struct acomp_alg *alg = crypto_acomp_alg(tfm);
+
+ if (alg && alg->batch_size > 1)
+ return alg->batch_size;
+ }
+
+ return 1;
+}
+EXPORT_SYMBOL_GPL(crypto_acomp_batch_size);
+
void comp_prepare_alg(struct comp_alg_common *alg)
{
struct crypto_alg *base = &alg->base;
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index e64cf422510b..a71532ccb8dc 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -593,6 +593,18 @@ int crypto_acomp_compress(struct acomp_req *req);
*/
int crypto_acomp_decompress(struct acomp_req *req);
+/**
+ * crypto_acomp_batch_size() -- Get the algorithm's batch size
+ *
+ * Function returns the algorithm's batch size for batching operations
+ *
+ * @tfm: ACOMPRESS tfm handle allocated with crypto_alloc_acomp()
+ *
+ * Return: @tfm's acomp_alg's @batch_size, if it has defined a
+ * @batch_size greater than 1; else return 1.
+ */
+unsigned int crypto_acomp_batch_size(struct crypto_acomp *tfm);
+
static inline struct acomp_req *acomp_request_on_stack_init(
char *buf, struct crypto_acomp *tfm)
{
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 0c0d3fc8a9fc..2d4d302d1b3f 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -28,6 +28,8 @@
*
* @compress: Function performs a compress operation
* @decompress: Function performs a de-compress operation
+ * @batch_size: Maximum batch-size for batching compress/decompress
+ * operations.
* @init: Initialize the cryptographic transformation object.
* This function is used to initialize the cryptographic
* transformation object. This function is called only once at
@@ -47,6 +49,7 @@
struct acomp_alg {
int (*compress)(struct acomp_req *req);
int (*decompress)(struct acomp_req *req);
+ unsigned int batch_size;
int (*init)(struct crypto_acomp *tfm);
void (*exit)(struct crypto_acomp *tfm);
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 5/8] mm: zswap: Store the folio's node id in the zswap_entry.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
` (3 preceding siblings ...)
2026-10-28 3:04 ` [PATCH v2 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
@ 2026-10-28 3:05 ` Kanchana P. Sridhar
2026-10-29 3:06 ` sashiko-bot
2026-10-28 3:05 ` [PATCH v2 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
` (2 subsequent siblings)
7 siblings, 1 reply; 16+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:05 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
When storing a folio in zswap, save the folio's node id in the zswap
entry instead of relying on the zswap_entry being allocated on the same
node. This enables optimizations such as batch allocation of zswap
entries for large folios using kmem_cache_alloc_bulk() that does not
support specifying a node id.
Accordingly, replace the SLUB allocation in zswap_entry_cache_alloc() to
be node-agnostic.
The size of the zswap_entry remains unchanged as 'referenced' is lumped
in with the 'length' (as it doesn't need a full unsigned int anyway).
A build time assertion checks for the zswap_entry's 'length' bitfield
being at least as large as the PAGE_SHIFT, which can be 18 on some
architectures.
Suggested-by: Nhat Pham <nphamcs@gmail.com>
Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 33 ++++++++++++++++++---------------
1 file changed, 18 insertions(+), 15 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index e1d57a7d9595..19b7d60bc7d2 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -180,6 +180,7 @@ struct zswap_pool {
*/
#define ZSWAP_FIRST_POOL_ID 1
#define ZSWAP_MAX_POOL_ID U8_MAX
+#define ZSWAP_MAX_COMPR_LENGTH_SHIFT 18U
static DEFINE_XARRAY_FLAGS(zswap_pools, XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_BH);
static struct zswap_pool __rcu *zswap_current_pool;
@@ -206,15 +207,19 @@ static struct shrinker *zswap_shrinker;
* logic if referenced is unset. See comments in the shrinker
* section for context.
* pool_idx - id of the zswap_pool that the entry's data is in.
+ * nid - NUMA node id of the page for which this is the zswap entry.
* handle - zsmalloc allocation handle that stores the compressed page data
* objcg - the obj_cgroup that the compressed memory is charged to
* lru - handle to the pool's lru used to evict pages.
*/
struct zswap_entry {
swp_entry_t swpentry;
- unsigned int length;
- bool referenced;
- u8 pool_idx;
+ struct {
+ unsigned int length:ZSWAP_MAX_COMPR_LENGTH_SHIFT;
+ bool referenced:1U;
+ u8 pool_idx;
+ };
+ int nid;
unsigned long handle;
struct obj_cgroup *objcg;
struct list_head lru;
@@ -679,14 +684,8 @@ static int zswap_enabled_param_set(const char *val,
* lru functions
**********************************/
-static inline int entry_to_nid(struct zswap_entry *entry)
-{
- return page_to_nid(virt_to_page(entry));
-}
-
static void zswap_lru_add(struct zswap_entry *entry)
{
- int nid = entry_to_nid(entry);
struct mem_cgroup *memcg;
/*
@@ -703,19 +702,18 @@ static void zswap_lru_add(struct zswap_entry *entry)
rcu_read_lock();
memcg = obj_cgroup_memcg(entry->objcg);
/* will always succeed */
- list_lru_add(&zswap_list_lru, &entry->lru, nid, memcg);
+ list_lru_add(&zswap_list_lru, &entry->lru, entry->nid, memcg);
rcu_read_unlock();
}
static void zswap_lru_del(struct zswap_entry *entry)
{
- int nid = entry_to_nid(entry);
struct mem_cgroup *memcg;
rcu_read_lock();
memcg = obj_cgroup_memcg(entry->objcg);
/* will always succeed */
- list_lru_del(&zswap_list_lru, &entry->lru, nid, memcg);
+ list_lru_del(&zswap_list_lru, &entry->lru, entry->nid, memcg);
rcu_read_unlock();
}
@@ -763,10 +761,11 @@ void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg)
**********************************/
static struct kmem_cache *zswap_entry_cache;
-static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp, int nid)
+static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp)
{
struct zswap_entry *entry;
- entry = kmem_cache_alloc_node(zswap_entry_cache, gfp, nid);
+
+ entry = kmem_cache_alloc(zswap_entry_cache, gfp);
if (!entry)
return NULL;
return entry;
@@ -1507,9 +1506,10 @@ static bool zswap_store_page(struct folio *folio, long index,
{
swp_entry_t page_swpentry = folio_swap_entry(folio, index);
struct zswap_entry *entry, *old;
+ int nid = folio_nid(folio);
/* allocate entry */
- entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio));
+ entry = zswap_entry_cache_alloc(GFP_KERNEL);
if (!entry) {
zswap_reject_kmemcache_fail++;
return false;
@@ -1524,6 +1524,7 @@ static bool zswap_store_page(struct folio *folio, long index,
* to an unrelated live pool.
*/
entry->pool_idx = pool->idx;
+ entry->nid = nid;
old = xa_store(swap_zswap_tree(page_swpentry),
swp_offset(page_swpentry),
@@ -1897,6 +1898,8 @@ static int zswap_setup(void)
struct zswap_pool *pool;
int ret;
+ BUILD_BUG_ON(ZSWAP_MAX_COMPR_LENGTH_SHIFT < PAGE_SHIFT);
+
zswap_entry_cache = KMEM_CACHE(zswap_entry, 0);
if (!zswap_entry_cache) {
pr_err("entry cache creation failed\n");
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 6/8] mm: zswap: Store large folios in batches.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
` (4 preceding siblings ...)
2026-10-28 3:05 ` [PATCH v2 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
@ 2026-10-28 3:05 ` Kanchana P. Sridhar
2026-10-29 3:06 ` sashiko-bot
2026-10-28 3:05 ` [PATCH v2 7/8] mm: zswap: Batched zswap_compress() for storing " Kanchana P. Sridhar
2026-10-28 3:05 ` [PATCH v2 8/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations Kanchana P. Sridhar
7 siblings, 1 reply; 16+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:05 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
Use batching when storing large folios in zswap. If the underlying
compressor supports batching (e.g. hardware parallel compression),
allocate multiple compression buffers, otherwise allocate one. The
number of buffers is bounded by a new constant, ZSWAP_MAX_BATCH_SIZE, to
limit the memory overhead. For existing software compressors, the only
extra overhead is the extra 'buffers' pointer, so 8 bytes per-CPU on
x86_64. Only the first buffer is currently used.
Regardless of compression batching, always process large folios in
batches. For hardware compressors, the batch size is the compressor
batch size, otherwise ZSWAP_MAX_BATCH_SIZE is used.
zswap_store_page() is replaced with zswap_store_pages(), which processes
a batch of pages and allows for batching optimizations. For now, only
optimize allocating entries by using batch allocations from the slab
cache.
Avoid repeatedly calling mem_cgroup_zswap_writeback_enabled() for every
page and only call it once for the folio, since the entire folio is
charged to a single memcg. Similarly, obtain the nid once to use during
each batch store of a folio. To facilitate this, a
struct zswap_store_context is introduced to keep function parameter
lists concise.
Also, place and align the members of struct zswap_pool such that members
used together in compress/decompress hot paths are able to share the same
cacheline. Specifically, these members are now on the same cacheline:
struct zswap_pool {
struct zs_pool *zs_pool;
struct crypto_acomp_ctx __percpu *acomp_ctx;
struct percpu_ref ref;
u8 idx;
u8 compr_batch_size;
// ...
};
The size of struct zswap_pool is unchanged.
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 323 +++++++++++++++++++++++++++++++++++++----------------
1 file changed, 227 insertions(+), 96 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index 19b7d60bc7d2..c8890e8a9b83 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -84,6 +84,9 @@ static bool zswap_pool_reached_full;
#define ZSWAP_PARAM_UNSET ""
+/* Limit the batch size to limit per-CPU memory usage for dst buffers. */
+#define ZSWAP_MAX_BATCH_SIZE 8U
+
static int zswap_setup(void);
/* Enable/disable zswap */
@@ -143,10 +146,10 @@ struct zswap_acomp_req {
struct mutex mutex;
};
-/* The compression mutex also protects the output buffer. */
+/* The compression mutex also protects the output buffer(s). */
struct zswap_comp_ctx {
struct zswap_acomp_req areq;
- u8 *buffer;
+ u8 **buffers;
};
/* Separate requests, so that decompression does not wait for compression. */
@@ -161,14 +164,18 @@ struct crypto_acomp_ctx {
* The only case where lru_lock is not acquired while holding tree.lock is
* when a zswap_entry is taken off the lru for writeback, in that case it
* needs to be verified that it's still valid in the tree.
+ *
+ * @compr_batch_size: The max batch size of the compression algorithm,
+ * bounded by ZSWAP_MAX_BATCH_SIZE.
*/
struct zswap_pool {
struct zs_pool *zs_pool;
struct crypto_acomp_ctx __percpu *acomp_ctx;
struct percpu_ref ref;
- struct rcu_work release_rwork;
- struct hlist_node node;
u8 idx;
+ u8 compr_batch_size;
+ struct hlist_node node;
+ struct rcu_work release_rwork;
char tfm_name[CRYPTO_MAX_ALG_NAME];
};
@@ -225,6 +232,15 @@ struct zswap_entry {
struct list_head lru;
};
+struct zswap_store_context {
+ struct folio *folio;
+ long folio_start;
+ bool wb_enabled;
+ int nid;
+ u8 nr_pages;
+ struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE];
+};
+
/*
* No RCU section is needed around the returned pointer: a stored entry pins
* its pool via percpu_ref (taken in zswap_store_page()), so the id cannot be
@@ -281,8 +297,10 @@ static inline struct xarray *swap_zswap_tree(swp_entry_t swp)
**********************************/
static void __zswap_pool_empty(struct percpu_ref *ref);
-static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
+static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx, u8 nr_buffers)
{
+ u8 i;
+
if (!acomp_ctx)
return;
@@ -301,8 +319,12 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
acomp_ctx->acomp = NULL;
- kfree(acomp_ctx->comp.buffer);
- acomp_ctx->comp.buffer = NULL;
+ if (acomp_ctx->comp.buffers) {
+ for (i = 0; i < nr_buffers; ++i)
+ kfree(acomp_ctx->comp.buffers[i]);
+ kfree(acomp_ctx->comp.buffers);
+ }
+ acomp_ctx->comp.buffers = NULL;
}
static struct zswap_pool *zswap_pool_create(char *compressor)
@@ -386,7 +408,9 @@ static struct zswap_pool *zswap_pool_create(char *compressor)
cpuhp_add_fail:
for_each_possible_cpu(cpu)
- acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu));
+ acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu),
+ pool->compr_batch_size);
+
error:
if (pool->acomp_ctx)
free_percpu(pool->acomp_ctx);
@@ -424,7 +448,8 @@ static void zswap_pool_destroy(struct zswap_pool *pool)
cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node);
for_each_possible_cpu(cpu)
- acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu));
+ acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu),
+ pool->compr_batch_size);
free_percpu(pool->acomp_ctx);
@@ -776,6 +801,39 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)
kmem_cache_free(zswap_entry_cache, entry);
}
+static inline void zswap_entries_cache_free_batch(
+ struct zswap_entry **entries,
+ u8 nr_entries)
+{
+ /*
+ * It is okay to use this to free entries allocated separately
+ * by zswap_entry_cache_alloc().
+ */
+ kmem_cache_free_bulk(zswap_entry_cache, nr_entries, (void **)entries);
+}
+
+static bool zswap_entries_cache_alloc_batch(
+ struct zswap_store_context *zc,
+ gfp_t gfp)
+{
+ if (unlikely(!kmem_cache_alloc_bulk(zswap_entry_cache, gfp,
+ zc->nr_pages, (void **)zc->entries))) {
+ u8 i;
+
+ for (i = 0; i < zc->nr_pages; ++i) {
+ zc->entries[i] = zswap_entry_cache_alloc(GFP_KERNEL);
+
+ if (unlikely(!zc->entries[i])) {
+ zswap_reject_kmemcache_fail++;
+ zswap_entries_cache_free_batch(zc->entries, i);
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
/*
* Carries out the common pattern of freeing an entry's zsmalloc allocation,
* freeing the entry itself, and decrementing the number of stored pages.
@@ -839,7 +897,9 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
{
struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node);
struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu);
+ int nid = cpu_to_node(cpu);
int ret = -ENOMEM;
+ u8 i;
/*
* To handle cases where the CPU goes through online-offline-online
@@ -850,15 +910,11 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
return 0;
}
- acomp_ctx->comp.buffer = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu));
- if (!acomp_ctx->comp.buffer)
- return ret;
-
/*
* In case of an error, crypto_alloc_acomp_node() returns an
* error pointer, never NULL.
*/
- acomp_ctx->acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu));
+ acomp_ctx->acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, nid);
if (IS_ERR(acomp_ctx->acomp)) {
pr_err("could not alloc crypto acomp %s : %pe\n",
pool->tfm_name, acomp_ctx->acomp);
@@ -866,10 +922,28 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
goto fail;
}
+ /*
+ * Allocate up to ZSWAP_MAX_BATCH_SIZE dst buffers if the
+ * compressor supports batching.
+ */
+ pool->compr_batch_size = min(ZSWAP_MAX_BATCH_SIZE,
+ crypto_acomp_batch_size(acomp_ctx->acomp));
+
ret = zswap_acomp_req_init(&acomp_ctx->comp.areq, acomp_ctx->acomp);
if (ret)
goto req_fail;
+ acomp_ctx->comp.buffers = kcalloc_node(pool->compr_batch_size, sizeof(u8 *),
+ GFP_KERNEL, nid);
+ if (!acomp_ctx->comp.buffers)
+ goto req_fail;
+
+ for (i = 0; i < pool->compr_batch_size; ++i) {
+ acomp_ctx->comp.buffers[i] = kmalloc_node(PAGE_SIZE, GFP_KERNEL, nid);
+ if (!acomp_ctx->comp.buffers[i])
+ goto req_fail;
+ }
+
/* Skip the per-CPU request when decompression can use the stack. */
if (acomp_is_async(acomp_ctx->acomp)) {
ret = zswap_acomp_req_init(&acomp_ctx->decomp, acomp_ctx->acomp);
@@ -882,12 +956,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
req_fail:
pr_err("could not alloc crypto acomp_request %s\n", pool->tfm_name);
fail:
- acomp_ctx_free(acomp_ctx);
+ acomp_ctx_free(acomp_ctx, pool->compr_batch_size);
return ret;
}
static bool zswap_compress(struct folio *folio, long index,
- struct zswap_entry *entry, struct zswap_pool *pool)
+ struct zswap_entry *entry, struct zswap_pool *pool,
+ bool wb_enabled)
{
struct zswap_comp_ctx *comp_ctx;
struct scatterlist input, output;
@@ -901,7 +976,7 @@ static bool zswap_compress(struct folio *folio, long index,
comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
mutex_lock(&comp_ctx->areq.mutex);
- dst = comp_ctx->buffer;
+ dst = comp_ctx->buffers[0];
sg_init_table(&input, 1);
sg_set_folio(&input, folio, PAGE_SIZE, index * PAGE_SIZE);
@@ -940,12 +1015,8 @@ static bool zswap_compress(struct folio *folio, long index,
* to the active LRU list in the case.
*/
if (comp_ret) {
- rcu_read_lock();
- if (!mem_cgroup_zswap_writeback_enabled(folio_memcg(folio))) {
- rcu_read_unlock();
+ if (!wb_enabled)
goto unlock;
- }
- rcu_read_unlock();
comp_ret = 0;
dlen = PAGE_SIZE;
dst = kmap_local_folio(folio, index * PAGE_SIZE);
@@ -1500,91 +1571,124 @@ static void shrink_worker(struct work_struct *w)
* main API
**********************************/
-static bool zswap_store_page(struct folio *folio, long index,
- struct obj_cgroup *objcg,
- struct zswap_pool *pool)
+/*
+ * Store multiple pages in @folio, starting from the page at index @start up to
+ * the page at index @end-1.
+ */
+static bool zswap_store_pages(struct zswap_store_context *zc,
+ long start,
+ long end,
+ struct zswap_pool *pool,
+ struct obj_cgroup *objcg)
{
- swp_entry_t page_swpentry = folio_swap_entry(folio, index);
- struct zswap_entry *entry, *old;
- int nid = folio_nid(folio);
+ u8 i, store_fail_idx = 0;
- /* allocate entry */
- entry = zswap_entry_cache_alloc(GFP_KERNEL);
- if (!entry) {
- zswap_reject_kmemcache_fail++;
- return false;
- }
+ zc->nr_pages = end - start;
+ zc->folio_start = start;
- if (!zswap_compress(folio, index, entry, pool))
- goto compress_failed;
+ VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
+
+ if (unlikely(!zswap_entries_cache_alloc_batch(zc, GFP_KERNEL)))
+ return false;
/*
- * Set pool_idx before the xa_store() below publishes the entry, or a
- * concurrent reader could resolve a stale pool_idx left by slab reuse
- * to an unrelated live pool.
+ * We co-locate entry initialization as much as possible here to
+ * minimize potential cache misses.
*/
- entry->pool_idx = pool->idx;
- entry->nid = nid;
-
- old = xa_store(swap_zswap_tree(page_swpentry),
- swp_offset(page_swpentry),
- entry, GFP_KERNEL);
- if (xa_is_err(old)) {
- int err = xa_err(old);
+ for (i = 0; i < zc->nr_pages; ++i) {
+ zc->entries[i]->handle = (unsigned long)ERR_PTR(-EINVAL);
+ /*
+ * Set pool_idx before the xa_store() below publishes the entry, or a
+ * concurrent reader could resolve a stale pool_idx left by slab reuse
+ * to an unrelated live pool.
+ */
+ zc->entries[i]->pool_idx = pool->idx;
+ zc->entries[i]->swpentry = folio_swap_entry(zc->folio,
+ zc->folio_start + i);
+ zc->entries[i]->objcg = objcg;
+ zc->entries[i]->referenced = true;
+ zc->entries[i]->nid = zc->nid;
+ INIT_LIST_HEAD(&zc->entries[i]->lru);
+ }
- WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err);
- zswap_reject_alloc_fail++;
- goto store_failed;
+ for (i = 0; i < zc->nr_pages; ++i) {
+ if (!zswap_compress(zc->folio, zc->folio_start + i,
+ zc->entries[i], pool, zc->wb_enabled))
+ goto store_pages_failed;
}
- /*
- * We may have had an existing entry that became stale when
- * the folio was redirtied and now the new version is being
- * swapped out. Get rid of the old.
- */
- if (old)
- zswap_entry_free(old);
+ for (i = 0; i < zc->nr_pages; ++i) {
+ struct zswap_entry *old, *entry = zc->entries[i];
+
+ old = xa_store(swap_zswap_tree(entry->swpentry),
+ swp_offset(entry->swpentry),
+ entry, GFP_KERNEL);
+ if (unlikely(xa_is_err(old))) {
+ int err = xa_err(old);
+
+ WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err);
+ zswap_reject_alloc_fail++;
+ /*
+ * Entries up to this point have been stored in the
+ * xarray. zswap_store() will erase them from the xarray
+ * and call zswap_entry_free(). Local cleanup in
+ * 'store_pages_failed' only needs to happen for
+ * entries from [@i to @nr_pages).
+ */
+ store_fail_idx = i;
+ goto store_pages_failed;
+ }
- /*
- * The entry is successfully compressed and stored in the tree, there is
- * no further possibility of failure. Grab refs to the pool and objcg,
- * charge zswap memory, and increment zswap_stored_pages.
- * The opposite actions will be performed by zswap_entry_free()
- * when the entry is removed from the tree.
- */
- zswap_pool_get(pool);
- if (objcg) {
- obj_cgroup_get(objcg);
- obj_cgroup_charge_zswap(objcg, entry->length);
- }
- atomic_long_inc(&zswap_stored_pages);
- if (entry->length == PAGE_SIZE)
- atomic_long_inc(&zswap_stored_incompressible_pages);
+ /*
+ * We may have had an existing entry that became stale when
+ * the folio was redirtied and now the new version is being
+ * swapped out. Get rid of the old.
+ */
+ if (unlikely(old))
+ zswap_entry_free(old);
- /*
- * We finish initializing the entry while it's already in xarray.
- * This is safe because:
- *
- * 1. Concurrent stores and invalidations are excluded by folio lock.
- *
- * 2. Writeback is excluded by the entry not being on the LRU yet.
- * The publishing order matters to prevent writeback from seeing
- * an incoherent entry.
- */
- entry->swpentry = page_swpentry;
- entry->objcg = objcg;
- entry->referenced = true;
- if (entry->length) {
- INIT_LIST_HEAD(&entry->lru);
- zswap_lru_add(entry);
+ /*
+ * The entry is successfully compressed and stored in the tree,
+ * and further failures will be cleaned up in zswap_store().
+ * Grab refs to the pool and objcg, charge zswap memory, and
+ * increment zswap_stored_pages. The opposite actions will be
+ * performed by zswap_entry_free() when the entry is removed
+ * from the tree.
+ */
+ zswap_pool_get(pool);
+ if (objcg) {
+ obj_cgroup_get(objcg);
+ obj_cgroup_charge_zswap(objcg, entry->length);
+ }
+ atomic_long_inc(&zswap_stored_pages);
+ if (unlikely(entry->length == PAGE_SIZE))
+ atomic_long_inc(&zswap_stored_incompressible_pages);
+
+ /*
+ * We finish by adding the entry to the LRU while it's already
+ * in xarray. This is safe because:
+ *
+ * 1. Concurrent stores and invalidations are excluded by folio lock.
+ *
+ * 2. Writeback is excluded by the entry not being on the LRU yet.
+ * The publishing order matters to prevent writeback from seeing
+ * an incoherent entry.
+ */
+ if (likely(entry->length))
+ zswap_lru_add(entry);
}
return true;
-store_failed:
- zs_free(pool->zs_pool, entry->handle);
-compress_failed:
- zswap_entry_cache_free(entry);
+store_pages_failed:
+ for (i = store_fail_idx; i < zc->nr_pages; ++i) {
+ if (!IS_ERR_VALUE(zc->entries[i]->handle))
+ zs_free(pool->zs_pool, zc->entries[i]->handle);
+ }
+
+ zswap_entries_cache_free_batch(&zc->entries[store_fail_idx],
+ zc->nr_pages - store_fail_idx);
+
return false;
}
@@ -1594,9 +1698,15 @@ bool zswap_store(struct folio *folio)
swp_entry_t swp = folio->swap;
struct obj_cgroup *objcg = NULL;
struct mem_cgroup *memcg = NULL;
- struct zswap_pool *pool;
+ int nid = folio_nid(folio);
bool ret = false;
- long index;
+ struct zswap_pool *pool;
+ u8 store_batch_size;
+ long start, end;
+ struct zswap_store_context zc = {
+ .folio = folio,
+ .nid = nid,
+ };
VM_WARN_ON_ONCE(!folio_test_locked(folio));
VM_WARN_ON_ONCE(!folio_test_swapcache(folio));
@@ -1630,8 +1740,29 @@ bool zswap_store(struct folio *folio)
mem_cgroup_put(memcg);
}
- for (index = 0; index < nr_pages; ++index) {
- if (!zswap_store_page(folio, index, objcg, pool))
+ rcu_read_lock();
+ zc.wb_enabled = mem_cgroup_zswap_writeback_enabled(folio_memcg(folio));
+ rcu_read_unlock();
+
+ /*
+ * For batching compressors, store the folio in batches of the
+ * compressor's batch_size.
+ *
+ * For non-batching compressors, store the folio in batches
+ * of ZSWAP_MAX_BATCH_SIZE, where each page in the batch is
+ * compressed sequentially. This gives better performance than
+ * invoking zswap_store_pages() per-page, due to cache locality
+ * of working set structures.
+ */
+ store_batch_size = (pool->compr_batch_size > 1) ?
+ pool->compr_batch_size : ZSWAP_MAX_BATCH_SIZE;
+
+ for (start = 0; start < nr_pages; start += store_batch_size) {
+ end = min(start + store_batch_size, nr_pages);
+
+ ret = zswap_store_pages(&zc, start, end, pool, objcg);
+
+ if (unlikely(!ret))
goto put_pool;
}
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 7/8] mm: zswap: Batched zswap_compress() for storing large folios in batches.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
` (5 preceding siblings ...)
2026-10-28 3:05 ` [PATCH v2 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
@ 2026-10-28 3:05 ` Kanchana P. Sridhar
2026-10-29 3:06 ` sashiko-bot
2026-10-28 3:05 ` [PATCH v2 8/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations Kanchana P. Sridhar
7 siblings, 1 reply; 16+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:05 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
A new batching implementation of zswap_compress() is provided for
compressors that do and do not support batching. This eliminates code
duplication and facilitates code maintainability with the introduction
of compress batching.
The vectorized implementation of calling the earlier zswap_compress()
sequentially, one page at a time in zswap_store_pages(), is replaced
with this new version of zswap_compress() that accepts multiple pages to
compress as a batch.
If the compressor does not support batching, each page in the batch is
compressed and stored sequentially.
If the batch is compressed without errors, the compressed buffers for
the batch are stored in zsmalloc. In case of compression errors, the
current behavior based on whether the folio is enabled for zswap
writeback, is preserved.
The batched zswap_compress() incorporates Herbert's suggestion for
SG lists to represent the batch's inputs/outputs to interface with the
crypto API [1].
The struct zswap_store_context is extended to facilitate passing
batching related constraints and data to zswap_compress(). The code that
allocates zsmalloc memory to store compression results is refactored
into a new routine, zswap_update_entries(). Subsequent patches will be
submitted to further optimize this phase by replacing zsmalloc with
the SLUB allocator to improve zswap_store()/zswap_load() latencies.
[1]: https://lore.kernel.org/all/aJ7Fk6RpNc815Ivd@gondor.apana.org.au/T/#m99aea2ce3d284e6c5a3253061d97b08c4752a798
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 271 +++++++++++++++++++++++++++++++++++++----------------
1 file changed, 189 insertions(+), 82 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index c8890e8a9b83..4476fb4aeb5f 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -150,6 +150,7 @@ struct zswap_acomp_req {
struct zswap_comp_ctx {
struct zswap_acomp_req areq;
u8 **buffers;
+ struct sg_table *sg_table;
};
/* Separate requests, so that decompression does not wait for compression. */
@@ -235,9 +236,13 @@ struct zswap_entry {
struct zswap_store_context {
struct folio *folio;
long folio_start;
+ unsigned int slen;
+ u8 nr_batch_pages;
bool wb_enabled;
int nid;
u8 nr_pages;
+ struct zs_pool *zs_pool;
+ struct zswap_comp_ctx *comp_ctx;
struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE];
};
@@ -325,6 +330,12 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx, u8 nr_buffers)
kfree(acomp_ctx->comp.buffers);
}
acomp_ctx->comp.buffers = NULL;
+
+ if (acomp_ctx->comp.sg_table) {
+ sg_free_table(acomp_ctx->comp.sg_table);
+ kfree(acomp_ctx->comp.sg_table);
+ }
+ acomp_ctx->comp.sg_table = NULL;
}
static struct zswap_pool *zswap_pool_create(char *compressor)
@@ -898,6 +909,7 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node);
struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu);
int nid = cpu_to_node(cpu);
+ struct scatterlist *sg;
int ret = -ENOMEM;
u8 i;
@@ -944,6 +956,21 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
goto req_fail;
}
+ acomp_ctx->comp.sg_table = kmalloc(sizeof(*acomp_ctx->comp.sg_table), GFP_KERNEL);
+ if (!acomp_ctx->comp.sg_table)
+ goto req_fail;
+
+ if (sg_alloc_table(acomp_ctx->comp.sg_table, pool->compr_batch_size,
+ GFP_KERNEL))
+ goto req_fail;
+
+ /*
+ * Statically map the per-CPU destination buffers to the per-CPU
+ * SG lists; crucial for performance.
+ */
+ for_each_sg(acomp_ctx->comp.sg_table->sgl, sg, pool->compr_batch_size, i)
+ sg_set_buf(sg, acomp_ctx->comp.buffers[i], PAGE_SIZE);
+
/* Skip the per-CPU request when decompression can use the stack. */
if (acomp_is_async(acomp_ctx->acomp)) {
ret = zswap_acomp_req_init(&acomp_ctx->decomp, acomp_ctx->acomp);
@@ -960,92 +987,161 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
return ret;
}
-static bool zswap_compress(struct folio *folio, long index,
- struct zswap_entry *entry, struct zswap_pool *pool,
- bool wb_enabled)
+static bool zswap_update_entries(struct zswap_store_context *zc,
+ u8 batch_start)
{
- struct zswap_comp_ctx *comp_ctx;
- struct scatterlist input, output;
- int comp_ret = 0, alloc_ret = 0;
- unsigned int dlen = PAGE_SIZE;
+ gfp_t gfp = GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE;
+ u8 i, batch_iter, comp_batch_iter;
unsigned long handle;
- gfp_t gfp;
+ unsigned int dlen;
u8 *dst;
- bool mapped = false;
-
- comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
- mutex_lock(&comp_ctx->areq.mutex);
-
- dst = comp_ctx->buffers[0];
- sg_init_table(&input, 1);
- sg_set_folio(&input, folio, PAGE_SIZE, index * PAGE_SIZE);
-
- sg_init_one(&output, dst, PAGE_SIZE);
- acomp_request_set_params(comp_ctx->areq.req, &input, &output,
- PAGE_SIZE, dlen);
/*
- * it maybe looks a little bit silly that we send an asynchronous request,
- * then wait for its completion synchronously. This makes the process look
- * synchronous in fact.
- * Theoretically, acomp supports users send multiple acomp requests in one
- * acomp instance, then get those requests done simultaneously. but in this
- * case, zswap actually does store and load page by page, there is no
- * existing method to send the second page before the first page is done
- * in one thread doing zswap.
- * but in different threads running on different cpu, we have different
- * acomp instance, and compression and decompression use separate
- * requests, so multiple threads can do (de)compression in parallel.
+ * For each output SG list in @acomp_ctx->req->sg_table->sgl,
+ * the @sg->length should be set to either the page's compressed
+ * length (success), or it's negative value compression error.
*/
- comp_ret = crypto_wait_req(crypto_acomp_compress(comp_ctx->areq.req),
- &comp_ctx->areq.wait);
- dlen = comp_ctx->areq.req->dlen;
- if (!comp_ret) {
- if (dlen == PAGE_SIZE)
- comp_ret = -ENOSPC;
- else if (WARN_ON_ONCE(!dlen || dlen > PAGE_SIZE))
- comp_ret = -EINVAL;
+ for (comp_batch_iter = 0; comp_batch_iter < zc->nr_batch_pages;
+ ++comp_batch_iter) {
+ batch_iter = batch_start + comp_batch_iter;
+ dlen = zc->entries[batch_iter]->length;
+ handle = zs_malloc(zc->zs_pool, dlen, gfp, zc->nid);
+
+ if (unlikely(IS_ERR_VALUE(handle))) {
+ if (PTR_ERR((void *)handle) == -ENOSPC)
+ zswap_reject_compress_poor++;
+ else
+ zswap_reject_alloc_fail++;
+
+ for (i = 0; i < comp_batch_iter; ++i)
+ zs_free(zc->zs_pool, zc->entries[batch_start + i]->handle);
+
+ return false;
+ }
+
+ zc->entries[batch_iter]->handle = handle;
+
+ if (likely(dlen < PAGE_SIZE)) {
+ dst = zc->comp_ctx->buffers[comp_batch_iter];
+ zs_obj_write(zc->zs_pool, handle, dst, dlen);
+ } else {
+ dst = kmap_local_folio(zc->folio,
+ (zc->folio_start + batch_iter) * PAGE_SIZE);
+ zs_obj_write(zc->zs_pool, handle, dst, dlen);
+ kunmap_local(dst);
+ }
}
+ return true;
+}
+
+/*
+ * zswap_compress() batching implementation for sequential and batching
+ * compressors.
+ *
+ * Description:
+ * ============
+ * Compress multiple @nr_pages in @folio starting from the @folio_start index
+ * in batches of @nr_batch_pages; the latter representing the lesser of the
+ * @nr_pages to be batch compressed and the compressor's batch size.
+ *
+ * @nr_pages can be in (1, ZSWAP_MAX_BATCH_SIZE] even if the compressor does not
+ * support batching.
+ *
+ * If @nr_batch_pages is 1, each page is processed sequentially.
+ *
+ * If @nr_batch_pages is > 1, the compressor may use batching to optimize
+ * compression.
+ */
+static bool zswap_compress(struct zswap_store_context *zc)
+{
+ u8 batch_start, comp_batch_iter, batch_iter;
+ struct scatterlist *sg;
+ bool ret;
+ int err;
+
/*
- * If a page cannot be compressed into a size smaller than PAGE_SIZE,
- * save the content as is without a compression, to keep the LRU order
- * of writebacks. If writeback is disabled, reject the page since it
- * only adds metadata overhead. swap_writeout() will put the page back
- * to the active LRU list in the case.
+ * Compress the @nr_pages in @folio starting at index @folio_start
+ * in batches of @nr_batch_pages.
*/
- if (comp_ret) {
- if (!wb_enabled)
- goto unlock;
- comp_ret = 0;
- dlen = PAGE_SIZE;
- dst = kmap_local_folio(folio, index * PAGE_SIZE);
- mapped = true;
- }
+ for (batch_start = 0; batch_start < zc->nr_pages;
+ batch_start += zc->nr_batch_pages) {
+ /*
+ * Send @nr_batch_pages to crypto_acomp for compression:
+ *
+ * These pages are in @folio's range of indices in the interval
+ * [@folio_start + @batch_start,
+ * @folio_start + @batch_start + @nr_batch_pages).
+ *
+ * @slen indicates the total source length bytes for
+ * @nr_batch_pages.
+ *
+ * The source folio pages in the batch are directly submitted to
+ * crypto_acomp via acomp_request_set_src_folio().
+ *
+ * The pool's compressor batch size is at least @nr_batch_pages,
+ * hence the acomp_ctx has at least @nr_batch_pages dst @buffers.
+ */
+ acomp_request_set_src_folio(zc->comp_ctx->areq.req, zc->folio,
+ (zc->folio_start + batch_start) * PAGE_SIZE,
+ zc->slen);
+
+ zc->comp_ctx->sg_table->sgl->length = zc->slen;
+
+ /*
+ * The per-CPU @zc->comp_ctx->sg_table scatterlists are statically
+ * mapped to the per-CPU dst @buffers at pool creation time.
+ */
+ acomp_request_set_dst_sg(zc->comp_ctx->areq.req,
+ zc->comp_ctx->sg_table->sgl, zc->slen);
+
+ err = crypto_wait_req(crypto_acomp_compress(zc->comp_ctx->areq.req),
+ &zc->comp_ctx->areq.wait);
- gfp = GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE;
- handle = zs_malloc(pool->zs_pool, dlen, gfp, folio_nid(folio));
- if (IS_ERR_VALUE(handle)) {
- alloc_ret = PTR_ERR((void *)handle);
- goto unlock;
+ /*
+ * If a page cannot be compressed into a size smaller than
+ * PAGE_SIZE, save the content as is without a compression, to
+ * keep the LRU order of writebacks. If writeback is disabled,
+ * reject the page since it only adds metadata overhead.
+ * swap_writeout() will put the page back to the active LRU list
+ * in the case.
+ *
+ * It is assumed that any compressor that sets the output length
+ * to 0 or a value >= PAGE_SIZE will also return a negative
+ * error status in @err; i.e, will not return a successful
+ * compression status in @err in this case.
+ */
+ if (unlikely(err && !zc->wb_enabled))
+ goto compress_error;
+
+ for_each_sg(zc->comp_ctx->sg_table->sgl, sg,
+ zc->nr_batch_pages, comp_batch_iter) {
+ batch_iter = batch_start + comp_batch_iter;
+ if (unlikely(sg->length <= 0 || sg->length >= PAGE_SIZE))
+ sg->length = PAGE_SIZE;
+
+ zc->entries[batch_iter]->length = sg->length;
+ }
+
+ ret = zswap_update_entries(zc, batch_start);
+ if (unlikely(!ret))
+ return false;
}
- zs_obj_write(pool->zs_pool, handle, dst, dlen);
- entry->handle = handle;
- entry->length = dlen;
+ return true;
-unlock:
- if (mapped)
- kunmap_local(dst);
- if (comp_ret == -ENOSPC || alloc_ret == -ENOSPC)
- zswap_reject_compress_poor++;
- else if (comp_ret)
- zswap_reject_compress_fail++;
- else if (alloc_ret)
- zswap_reject_alloc_fail++;
-
- mutex_unlock(&comp_ctx->areq.mutex);
- return comp_ret == 0 && alloc_ret == 0;
+compress_error:
+ for_each_sg(zc->comp_ctx->sg_table->sgl, sg,
+ zc->nr_batch_pages, comp_batch_iter) {
+ if ((int)sg->length < 0) {
+ if ((int)sg->length == -ENOSPC)
+ zswap_reject_compress_poor++;
+ else
+ zswap_reject_compress_fail++;
+ }
+ }
+
+ return false;
}
static bool __zswap_decompress(struct zswap_entry *entry,
@@ -1582,6 +1678,7 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
struct obj_cgroup *objcg)
{
u8 i, store_fail_idx = 0;
+ bool ret;
zc->nr_pages = end - start;
zc->folio_start = start;
@@ -1596,7 +1693,6 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
* minimize potential cache misses.
*/
for (i = 0; i < zc->nr_pages; ++i) {
- zc->entries[i]->handle = (unsigned long)ERR_PTR(-EINVAL);
/*
* Set pool_idx before the xa_store() below publishes the entry, or a
* concurrent reader could resolve a stale pool_idx left by slab reuse
@@ -1611,11 +1707,19 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
INIT_LIST_HEAD(&zc->entries[i]->lru);
}
- for (i = 0; i < zc->nr_pages; ++i) {
- if (!zswap_compress(zc->folio, zc->folio_start + i,
- zc->entries[i], pool, zc->wb_enabled))
- goto store_pages_failed;
- }
+ /*
+ * Locking the comp_ctx mutex once per store batch results in better
+ * performance as compared to locking per compress batch.
+ */
+ zc->comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
+ mutex_lock(&zc->comp_ctx->areq.mutex);
+
+ ret = zswap_compress(zc);
+
+ mutex_unlock(&zc->comp_ctx->areq.mutex);
+
+ if (unlikely(!ret))
+ goto compress_failed;
for (i = 0; i < zc->nr_pages; ++i) {
struct zswap_entry *old, *entry = zc->entries[i];
@@ -1681,11 +1785,10 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
return true;
store_pages_failed:
- for (i = store_fail_idx; i < zc->nr_pages; ++i) {
- if (!IS_ERR_VALUE(zc->entries[i]->handle))
- zs_free(pool->zs_pool, zc->entries[i]->handle);
- }
+ for (i = store_fail_idx; i < zc->nr_pages; ++i)
+ zs_free(pool->zs_pool, zc->entries[i]->handle);
+compress_failed:
zswap_entries_cache_free_batch(&zc->entries[store_fail_idx],
zc->nr_pages - store_fail_idx);
@@ -1757,6 +1860,10 @@ bool zswap_store(struct folio *folio)
store_batch_size = (pool->compr_batch_size > 1) ?
pool->compr_batch_size : ZSWAP_MAX_BATCH_SIZE;
+ zc.nr_batch_pages = pool->compr_batch_size;
+ zc.slen = zc.nr_batch_pages * PAGE_SIZE;
+ zc.zs_pool = pool->zs_pool;
+
for (start = 0; start < nr_pages; start += store_batch_size) {
end = min(start + store_batch_size, nr_pages);
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [PATCH v2 8/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
` (6 preceding siblings ...)
2026-10-28 3:05 ` [PATCH v2 7/8] mm: zswap: Batched zswap_compress() for storing " Kanchana P. Sridhar
@ 2026-10-28 3:05 ` Kanchana P. Sridhar
2026-10-29 3:06 ` sashiko-bot
7 siblings, 1 reply; 16+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-28 3:05 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
This various stages of zswap_store_pages() are broken down into routines
that work on a batch of data when a large folio is being processed:
zswap_init_entries()
zswap_compress()
zswap_tree_stores()
zswap_account_stores()
The struct zswap_store_context is used to pass constraints and data
among these batching components. The resulting code is highly simplified
and maintainable, where each of the batching sub-routines of
zswap_store() can be optimized modularly.
For instance, the zswap_account_stores() now does the zswap atomics
counter increments, getting pool references, getting objcg references
and objcg zswap charging at the batch level.
Subsequent patch series will be submitted, that implement ideas building
upon this batching framework:
1) Replacing zsmalloc with SLUB allocator, comprising of kmem_caches
for a finite # of compressed size bins. This enables bulk allocation
of compressed buffers in a batch, that belong to the same size
bin. It gets rid of significant zsmalloc latency overhead (locks,
compressed data spanning zspages, etc.), without compromising memory
efficiency.
2) vmscan reclaim batching of hybrid any-order folios, that will submit
a batch of folios to zswap_store()'s batching framework to optimize
storing of batches of pages from different folios, and also within
large folios.
The batching framework introduced in this series improves zswap_store()
latency by a lot.
Performance data: usemem with 56% over-commit:
==============================================
Validation was done with mm-unstable as of 10-25-2026 (Baseline) and with this
patch series. To minimize variation in the data, I made these changes to my
performance measurement methodology:
1) Reduced the number of usemem processes to 2. Each process tried to allocate
2400M, in a 3G cgroup memory.high budget, effectively accomplishing a 56.25%
over-commit.
2) Increased the runtime of the workload to ~9 minutes by enabling PMD THP,
i.e., 2048kB mTHP folios.
3) The use of PMD THP forces zswap_store()/zswap_compress() latency to be in the
critical path, and thus enables very reliable measurements with bpftrace.
4) I used lhist() instead of hist() in bpftrace to gather latencies at a finer
level of granularity and accuracy.
Testing was done on a Chromebook with a QEMU "vng" VM running in Crostini.
Chromebook Diagnostics reports:
Intel(R) Core(TM) Ultra 5 115U (10 CPUs, 4.200GHz)
RAM: 16GB
vng setup:
Memory: 7G
Swap: 3G
Pinned CPUs: 2 (E-Cores: 2,3)
zswap compressor:
ZSTD
mTHP enabled:
2048kB (PMD THP)
I used the help of an LLM to write the bpftrace data post-processing scripts
that generate latency percentiles.
vm-scalability/usemem setup:
cgroup memory.high: 3G
2 usemem processes, each allocates 2400M anonymous memory, reads the
Silesia dataset [1] from a pre-fork-allocated buffer into the 2400M,
writes to each unsigned long in the 2400M, sleeps for 10 sec, then
frees the memory and exits.
usemem -w --read-silesia -O -b 1 -s 10 -n 2 2400M
usemem metrics are reported from a single run after starting the QEMU VM.
Further, I have tabulated the bpftrace p50/p95/p99 latencies with these labels:
"zswap_store_higher_order":
zswap_store() of PMD folios.
"zswap_store_order_0":
zswap_store() of order-0 folios.
"zswap_batch_compress_per_page":
zswap_compress() per-page latency of store batches generated from PMD
folios.
"zswap_single_compress_per_page":
zswap_compress() per-page latency of order-0 folios.
----------------------------------------------------------------------------
usemem Baseline zswap zswap Change with
PMD THP mm-unstable store compress compress
10-25-2026 batching batching batching
(patch 6) (patch 8) %
----------------------------------------------------------------------------
usemem process 1 6,353 7,240 7,428
Throughput (KB/s)
usemem process 2 6,352 7,236 7,422
Throughput (KB/s)
Average Throughput (KB/s) 6,353 7,238 7,425 17%
----------------------------------------------------------------------------
usemem process 1 484,580 308,903 257,230
Time to free memory (usecs)
usemem process 2 518,928 438,585 446,958
Time to free memory (usecs)
Average Time to 501,754 373,744 352,094 -30%
Free Memory (usecs)
-------------------------------------------------------------------------------
-------------------------------------------------------------------------------
usemem Baseline zswap zswap Change
PMD THP mm-unstable store compress with
BPFTRACE 10-25-2026 batching batching compress
(patch 6) (patch 8) batching
(ns)
-------------------------------------------------------------------------------
zswap_batch_compress_per_page (ns):
p50 40,086 1,569
p95 54,689 -64,505
p99 83,248 -88,488
-------------------------------------------------------------------------------
zswap_single_compress_per_page (ns):
p50 38,517 38,352 36,650 -1,867
p95 119,194 121,174 133,678 14,484
p99 171,736 158,601 169,010 -2,726
-------------------------------------------------------------------------------
zswap_store_higher_order (ns):
p50 21,012,222 21,322,885 20,176,667 -835,556
p95 25,643,500 26,204,375 24,485,000 -1,158,500
p99 28,567,400 38,577,750 26,158,000 -2,409,400
-------------------------------------------------------------------------------
zswap_store_order_0 (ns):
p50 37,601 36,994 37,568 -33
p95 123,731 137,034 128,366 4,635
p99 159,965 169,282 175,618 15,653
-------------------------------------------------------------------------------
vmstat summary:
-------------------------------------------------------------------------------
zswpin 1,104,226 993,412 1,091,146
zswpout 1,688,181 1,582,664 1,669,671
2048kB-alloc 2,500 2,505 2,500
2048kB-zswpout 1,535 1,535 1,535
2048kB-swpout-fallback 927 929 875
-------------------------------------------------------------------------------
zswap compress batching improves on both, zswap_store() latency and
zswap_compress() latency wrt the baseline, pretty consistently; except for p50
batch compress latency which can be optimized further in subsequent
patches. The batching framework established in this series allows us to make
locking and bulk-processing improvements in each stage of zswap_store(). One of
the ideas that has been showing very promising results is to replace zsmalloc
with SLUB compressed caches and will be submitted separately. This is expected
to resolve the p50 batch compress latency issue since it will be directly
optimizing zswap_update_entries().
The speedup with zswap store batching of large folios has a propelling effect
on other parts of mm, especially, by enabling large folios to be swapped out
without splitting, as the data shows. This enables other mm components that are
optimized for large folios, such as munmap(), folio refcounts/mapcounts
updates, PTE invalidation upon freeing memory, zswap_invalidate(); which
benefit from xarray offsets/range continuity. As a result, subsequent large
folio swapouts are more likely to find contiguous swap slots, there are fewer
large folio splits during reclaim and the cycle continues. As the data
demonstrates, this directly contributes to more memory over-commit
opportunity.
Architectural considerations for the zswap batching framework:
==============================================================
The zswap batching framework is designed to be hardware-agnostic.
It can be leveraged by any hardware accelerator or software-based
compressor.
Potential future clients of the batching framework:
===================================================
This patch-series demonstrates the performance benefits of compression
batching when used in zswap_store() of large folios. Compression
batching can be used for other use cases such as batching compression in
zram, batch compression of different folios during reclaim, kcompressd,
file systems, etc. Decompression batching can be used to improve
efficiency of zswap writeback (Thanks Nhat for this idea), batching
decompressions in zram, etc.
[1]: https://wanos.co/assets/silesia.tar
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 178 +++++++++++++++++++++++++++++++++--------------------
1 file changed, 112 insertions(+), 66 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index 4476fb4aeb5f..2831284cfc90 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -504,12 +504,18 @@ static int __must_check zswap_pool_tryget(struct zswap_pool *pool)
}
/* The caller must already have a reference. */
-static void zswap_pool_get(struct zswap_pool *pool)
+static void __maybe_unused zswap_pool_get(struct zswap_pool *pool)
{
percpu_ref_get(&pool->ref);
}
-static void zswap_pool_put(struct zswap_pool *pool)
+/* The caller must already have a reference. */
+static inline void zswap_pool_get_many(struct zswap_pool *pool, u8 nr)
+{
+ percpu_ref_get_many(&pool->ref, nr);
+}
+
+static inline void zswap_pool_put(struct zswap_pool *pool)
{
percpu_ref_put(&pool->ref);
}
@@ -1668,25 +1674,13 @@ static void shrink_worker(struct work_struct *w)
**********************************/
/*
- * Store multiple pages in @folio, starting from the page at index @start up to
- * the page at index @end-1.
+ * Functions called by zswap_store(), that operate on batches of pages.
*/
-static bool zswap_store_pages(struct zswap_store_context *zc,
- long start,
- long end,
- struct zswap_pool *pool,
- struct obj_cgroup *objcg)
+static inline void zswap_init_entries(struct zswap_store_context *zc,
+ struct zswap_pool *pool,
+ struct obj_cgroup *objcg)
{
- u8 i, store_fail_idx = 0;
- bool ret;
-
- zc->nr_pages = end - start;
- zc->folio_start = start;
-
- VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
-
- if (unlikely(!zswap_entries_cache_alloc_batch(zc, GFP_KERNEL)))
- return false;
+ u8 i;
/*
* We co-locate entry initialization as much as possible here to
@@ -1694,9 +1688,9 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
*/
for (i = 0; i < zc->nr_pages; ++i) {
/*
- * Set pool_idx before the xa_store() below publishes the entry, or a
- * concurrent reader could resolve a stale pool_idx left by slab reuse
- * to an unrelated live pool.
+ * Set pool_idx before the xa_store() in zswap_tree_stores()
+ * publishes the entry, or a concurrent reader could resolve a
+ * stale pool_idx left by slab reuse to an unrelated live pool.
*/
zc->entries[i]->pool_idx = pool->idx;
zc->entries[i]->swpentry = folio_swap_entry(zc->folio,
@@ -1706,27 +1700,17 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
zc->entries[i]->nid = zc->nid;
INIT_LIST_HEAD(&zc->entries[i]->lru);
}
+}
- /*
- * Locking the comp_ctx mutex once per store batch results in better
- * performance as compared to locking per compress batch.
- */
- zc->comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
- mutex_lock(&zc->comp_ctx->areq.mutex);
-
- ret = zswap_compress(zc);
-
- mutex_unlock(&zc->comp_ctx->areq.mutex);
-
- if (unlikely(!ret))
- goto compress_failed;
+static bool zswap_tree_stores(struct zswap_store_context *zc)
+{
+ struct zswap_entry *old;
+ u8 i, store_fail_idx = 0;
for (i = 0; i < zc->nr_pages; ++i) {
- struct zswap_entry *old, *entry = zc->entries[i];
-
- old = xa_store(swap_zswap_tree(entry->swpentry),
- swp_offset(entry->swpentry),
- entry, GFP_KERNEL);
+ old = xa_store(swap_zswap_tree(zc->entries[i]->swpentry),
+ swp_offset(zc->entries[i]->swpentry),
+ zc->entries[i], GFP_KERNEL);
if (unlikely(xa_is_err(old))) {
int err = xa_err(old);
@@ -1734,13 +1718,11 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
zswap_reject_alloc_fail++;
/*
* Entries up to this point have been stored in the
- * xarray. zswap_store() will erase them from the xarray
- * and call zswap_entry_free(). Local cleanup in
- * 'store_pages_failed' only needs to happen for
- * entries from [@i to @nr_pages).
+ * xarray. Local cleanup in 'tree_store_failed' needs to
+ * erase entries up until now.
*/
store_fail_idx = i;
- goto store_pages_failed;
+ goto tree_store_failed;
}
/*
@@ -1750,24 +1732,30 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
*/
if (unlikely(old))
zswap_entry_free(old);
+ }
- /*
- * The entry is successfully compressed and stored in the tree,
- * and further failures will be cleaned up in zswap_store().
- * Grab refs to the pool and objcg, charge zswap memory, and
- * increment zswap_stored_pages. The opposite actions will be
- * performed by zswap_entry_free() when the entry is removed
- * from the tree.
- */
- zswap_pool_get(pool);
- if (objcg) {
- obj_cgroup_get(objcg);
- obj_cgroup_charge_zswap(objcg, entry->length);
- }
- atomic_long_inc(&zswap_stored_pages);
- if (unlikely(entry->length == PAGE_SIZE))
+ return true;
+
+tree_store_failed:
+ for (i = 0; i < store_fail_idx; ++i)
+ xa_erase(swap_zswap_tree(zc->entries[i]->swpentry),
+ swp_offset(zc->entries[i]->swpentry));
+ return false;
+}
+
+static inline void zswap_account_stores(struct zswap_store_context *zc,
+ struct zswap_pool *pool,
+ struct obj_cgroup *objcg)
+{
+ size_t total_batch_comp_length = 0;
+ u8 i;
+
+ for (i = 0; i < zc->nr_pages; ++i) {
+ if (unlikely(zc->entries[i]->length == PAGE_SIZE))
atomic_long_inc(&zswap_stored_incompressible_pages);
+ total_batch_comp_length += zc->entries[i]->length;
+
/*
* We finish by adding the entry to the LRU while it's already
* in xarray. This is safe because:
@@ -1778,20 +1766,78 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
* The publishing order matters to prevent writeback from seeing
* an incoherent entry.
*/
- if (likely(entry->length))
- zswap_lru_add(entry);
+ if (likely(zc->entries[i]->length))
+ zswap_lru_add(zc->entries[i]);
}
+ /*
+ * The entry is successfully compressed and stored in the tree,
+ * and further failures will be cleaned up in zswap_store().
+ * For the batch, grab refs to the pool and objcg, charge zswap memory,
+ * and increment zswap_stored_pages by @zc->nr_pages. The opposite
+ * actions will be performed by zswap_entry_free() when the entry is
+ * removed from the tree.
+ */
+ zswap_pool_get_many(pool, zc->nr_pages);
+ if (objcg) {
+ obj_cgroup_get_many(objcg, zc->nr_pages);
+ obj_cgroup_charge_zswap(objcg, total_batch_comp_length);
+ }
+ atomic_long_add(zc->nr_pages, &zswap_stored_pages);
+}
+
+/*
+ * Store multiple pages in @folio, starting from the page at index @start up to
+ * the page at index @end-1.
+ */
+static bool zswap_store_pages(struct zswap_store_context *zc,
+ long start,
+ long end,
+ struct zswap_pool *pool,
+ struct obj_cgroup *objcg)
+{
+ bool ret;
+ u8 i;
+
+ zc->nr_pages = end - start;
+ zc->folio_start = start;
+
+ VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
+
+ if (unlikely(!zswap_entries_cache_alloc_batch(zc, GFP_KERNEL)))
+ return false;
+
+ zswap_init_entries(zc, pool, objcg);
+
+ /*
+ * Locking the comp_ctx mutex once per store batch results in better
+ * performance as compared to locking per compress batch.
+ */
+ zc->comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
+ mutex_lock(&zc->comp_ctx->areq.mutex);
+
+ ret = zswap_compress(zc);
+
+ mutex_unlock(&zc->comp_ctx->areq.mutex);
+
+ if (unlikely(!ret))
+ goto compress_failed;
+
+ ret = zswap_tree_stores(zc);
+
+ if (unlikely(!ret))
+ goto store_pages_failed;
+
+ zswap_account_stores(zc, pool, objcg);
+
return true;
store_pages_failed:
- for (i = store_fail_idx; i < zc->nr_pages; ++i)
+ for (i = 0; i < zc->nr_pages; ++i)
zs_free(pool->zs_pool, zc->entries[i]->handle);
compress_failed:
- zswap_entries_cache_free_batch(&zc->entries[store_fail_idx],
- zc->nr_pages - store_fail_idx);
-
+ zswap_entries_cache_free_batch(zc->entries, zc->nr_pages);
return false;
}
--
2.39.5
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [PATCH v2 3/8] crypto: acomp - Add trivial segmentation wrapper.
2026-10-28 3:04 ` [PATCH v2 3/8] crypto: acomp - Add trivial segmentation wrapper Kanchana P. Sridhar
@ 2026-10-29 3:05 ` sashiko-bot
2026-10-29 5:09 ` Kanchana P. Sridhar
0 siblings, 1 reply; 16+ messages in thread
From: sashiko-bot @ 2026-10-29 3:05 UTC (permalink / raw)
To: Kanchana P. Sridhar, hannes, yosry, nphamcs, chengming.zhou, akpm,
linux-mm, herbert, davem, linux-crypto
Cc: rientjes, willy, senozhatsky
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [Critical] crypto: acomp - missing async check in crypto_acomp_compress() fast path
- [High] crypto: acomp: uninitialized state->src/dst restore in acomp_reqchain_virt()
--
commit 7144b07b2cc5b6704699fb53fac44c4270d31f7b
Author: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
crypto: acomp - Add trivial segmentation wrapper.
This commit adds a single-segment compression wrapper in crypto/acompress.c
that stores the compressed length or error code into req->dst->length when
req->unit_size is set on transforms that do not support segmentation. It also
adds an inline fast path in crypto_acomp_compress() for single-segment
scatterlist requests.
> diff --git a/crypto/acompress.c b/crypto/acompress.c
> index c8b4b8589769e..c55b0e8eff3d8 100644
> --- a/crypto/acompress.c
> +++ b/crypto/acompress.c
> @@ -166,14 +166,20 @@ static void acomp_save_req(struct acomp_req *req, crypto_completion_t cplt)
>
> state->compl = req->base.complete;
> state->data = req->base.data;
> + state->unit_size = req->unit_size;
> + state->flags = req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
> + CRYPTO_ACOMP_REQ_DST_VIRT);
[Severity: High]
Can moving the state->flags assignment from acomp_virt_to_sg() into
acomp_save_req() corrupt req->svirt and req->dvirt when a request is
configured with acomp_request_set_src_nondma() or
acomp_request_set_dst_nondma()?
When a caller sets up a non-dma virtual request via
acomp_request_set_src_nondma() or acomp_request_set_dst_nondma(), both the
non-dma and virtual request flags are set in req->base.flags. When
crypto_acomp_compress() or crypto_acomp_decompress() calls
acomp_do_req_chain(), the call chain is:
crypto_acomp_compress()
acomp_do_req_chain()
acomp_save_req()
state->flags = req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
CRYPTO_ACOMP_REQ_DST_VIRT);
acomp_do_one_req()
if (acomp_request_isnondma(req))
return acomp_do_nondma(req, comp);
Because acomp_do_one_req() returns early through acomp_do_nondma() without
calling acomp_virt_to_sg(), state->src and state->dst are never initialized
from req->svirt and req->dvirt.
When acomp_reqchain_finish() then calls acomp_reqchain_virt() in
crypto/acompress.c:
static void acomp_reqchain_virt(struct acomp_req *req)
{
struct acomp_req_chain *state = &req->chain;
unsigned int slen = req->slen;
unsigned int dlen = req->dlen;
if (state->flags & CRYPTO_ACOMP_REQ_SRC_VIRT)
acomp_request_set_src_dma(req, state->src, slen);
if (state->flags & CRYPTO_ACOMP_REQ_DST_VIRT)
acomp_request_set_dst_dma(req, state->dst, dlen);
}
won't acomp_reqchain_virt() see the virtual flags set in state->flags and
overwrite req->svirt and req->dvirt with the uninitialized state->src and
state->dst pointers (while also clearing the non-dma flags), causing a null
or wild pointer dereference if acomp_reqchain_finish() then writes to
req->dst->length when req->unit_size is non-zero?
[ ... ]
> @@ -279,10 +289,27 @@ int crypto_acomp_compress(struct acomp_req *req)
> {
> struct crypto_acomp *tfm = crypto_acomp_reqtfm(req);
>
> + if (likely((req->unit_size == req->slen) && acomp_request_issg(req) &&
> + !crypto_acomp_req_seg(tfm) && !acomp_req_on_stack(req))) {
> + int err = tfm->compress(req);
> +
> + req->dst->length = unlikely(err) ? err : req->dlen;
> + return err;
> + }
[Severity: Critical]
Should this fast path in crypto_acomp_compress() in crypto/acompress.c check
!acomp_is_async(tfm) instead of !acomp_req_on_stack(req)?
Callers using asynchronous hardware compression transforms (where
acomp_is_async(tfm) is true, such as qat, hisi_zip, or iaa in async mode)
allocate requests on the heap with acomp_request_alloc(), so
!acomp_req_on_stack(req) is true for asynchronous requests.
When zswap_compress() submits a single-segment scatterlist request with
req->unit_size == req->slen to an asynchronous transform that does not
support segmentation, this fast path will be taken instead of
acomp_do_req_chain(req, true):
crypto/acompress.c:crypto_acomp_compress() {
if (likely((req->unit_size == req->slen) && acomp_request_issg(req) &&
!crypto_acomp_req_seg(tfm) && !acomp_req_on_stack(req))) {
int err = tfm->compress(req);
req->dst->length = unlikely(err) ? err : req->dlen;
return err;
}
}
When tfm->compress(req) submits the descriptor to hardware and returns an
in-progress or busy status, won't this immediately overwrite req->dst->length
with the negative error value while hardware dma is still in flight on
req->dst, and skip registering acomp_reqchain_done() via acomp_save_req() so
that req->dst->length is never updated to req->dlen upon async completion?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261028030503.17923-1-kanchanapsridhar2026@gmail.com?part=3
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size.
2026-10-28 3:04 ` [PATCH v2 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
@ 2026-10-29 3:06 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-10-29 3:06 UTC (permalink / raw)
To: Kanchana P. Sridhar, hannes, yosry, nphamcs, chengming.zhou, akpm,
linux-mm, herbert, davem, linux-crypto
Cc: rientjes, willy, senozhatsky
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] crypto: acomp: broken type check and async gate in crypto_acomp_batch_size()
--
commit a38b95e87019ff2c11b51ebf159a082ec340456f
Author: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
crypto: acomp - Add API to get a compression algorithm's batch-size.
This commit adds a batch_size member to struct acomp_alg and introduces
crypto_acomp_batch_size() so callers such as zswap can query the maximum
number of requests an acomp algorithm can process in a batch.
> diff --git a/crypto/acompress.c b/crypto/acompress.c
> index c55b0e8eff3d8..8e285cb7477e9 100644
> --- a/crypto/acompress.c
> +++ b/crypto/acompress.c
> @@ -326,6 +326,20 @@ int crypto_acomp_decompress(struct acomp_req *req)
> }
> EXPORT_SYMBOL_GPL(crypto_acomp_decompress);
>
> +unsigned int crypto_acomp_batch_size(struct crypto_acomp *tfm)
> +{
> + if (acomp_is_async(tfm) &&
> + (crypto_comp_alg_common(tfm)->base.cra_flags & CRYPTO_ALG_TYPE_ACOMPRESS)) {
> + struct acomp_alg *alg = crypto_acomp_alg(tfm);
> +
> + if (alg && alg->batch_size > 1)
> + return alg->batch_size;
> + }
[Severity: Medium]
Does the cra_flags check in crypto_acomp_batch_size() distinguish
struct acomp_alg from struct scomp_alg as intended?
In include/linux/crypto.h, algorithm types are 4-bit values under
CRYPTO_ALG_TYPE_MASK rather than single-bit flags:
#define CRYPTO_ALG_TYPE_MASK 0x0000000f
...
#define CRYPTO_ALG_TYPE_ACOMPRESS 0x0000000a
#define CRYPTO_ALG_TYPE_SCOMPRESS 0x0000000b
Because (0x0b & 0x0a) == 0x0a, testing cra_flags & CRYPTO_ALG_TYPE_ACOMPRESS
without masking by CRYPTO_ALG_TYPE_MASK evaluates to true for both
CRYPTO_ALG_TYPE_ACOMPRESS and CRYPTO_ALG_TYPE_SCOMPRESS.
A struct crypto_acomp *tfm allocated via crypto_alloc_acomp() can wrap either
a struct acomp_alg or a struct scomp_alg. If an scomp_alg has CRYPTO_ALG_ASYNC
set in cra_flags (or if the acomp_is_async(tfm) check is removed), calling
crypto_acomp_alg(tfm) performs container_of() to struct acomp_alg on a
struct scomp_alg and reads alg->batch_size from unrelated memory inside
struct scomp_alg::streams.
Elsewhere in crypto/acompress.c, such as in crypto_acomp_init_tfm():
if (tfm->__crt_alg->cra_type != &crypto_acomp_type)
return crypto_init_scomp_ops_async(tfm);
the code checks cra_type == &crypto_acomp_type (or masks cra_flags with
CRYPTO_ALG_TYPE_MASK before comparing against CRYPTO_ALG_TYPE_ACOMPRESS) to
verify that the tfm is backed by a struct acomp_alg.
Also, should crypto_acomp_batch_size() gate on acomp_is_async(tfm)? A
synchronous struct acomp_alg registered via crypto_register_acomp() does not
set CRYPTO_ALG_ASYNC, which causes crypto_acomp_batch_size() to always return 1
even when batch_size > 1, whereas segmented request handling in
crypto_acomp_compress() and acomp_do_req_chain() checks
crypto_acomp_req_seg(tfm) (CRYPTO_ALG_REQ_SEG) instead.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261028030503.17923-1-kanchanapsridhar2026@gmail.com?part=4
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 5/8] mm: zswap: Store the folio's node id in the zswap_entry.
2026-10-28 3:05 ` [PATCH v2 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
@ 2026-10-29 3:06 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-10-29 3:06 UTC (permalink / raw)
To: Kanchana P. Sridhar, hannes, yosry, nphamcs, chengming.zhou, akpm,
linux-mm, herbert, davem, linux-crypto
Cc: rientjes, willy, senozhatsky
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] mm: zswap: off-by-one bitfield width for zswap_entry.length on 256KB pages
- [Medium] mm: zswap: shrink_memcg() skips non-N_NORMAL_MEMORY nodes in zswap_list_lru
--
commit c4aa889f59a75fcf871276fd68b4a4738afe132b
Author: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
mm: zswap: Store the folio's node id in the zswap_entry.
This commit stores the folio's NUMA node ID directly in struct zswap_entry
instead of deriving it from the slab page backing the entry, and converts
zswap_entry_cache_alloc() to use node-agnostic slab allocation. To keep the
structure size unchanged, it packs length and referenced into bitfields.
> diff --git a/mm/zswap.c b/mm/zswap.c
> index e1d57a7d95950..19b7d60bc7d2a 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
> @@ -180,6 +180,7 @@ struct zswap_pool {
> */
> #define ZSWAP_FIRST_POOL_ID 1
> #define ZSWAP_MAX_POOL_ID U8_MAX
> +#define ZSWAP_MAX_COMPR_LENGTH_SHIFT 18U
> static DEFINE_XARRAY_FLAGS(zswap_pools, XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_BH);
> static struct zswap_pool __rcu *zswap_current_pool;
>
> @@ -206,15 +207,19 @@ static struct shrinker *zswap_shrinker;
> * logic if referenced is unset. See comments in the shrinker
> * section for context.
> * pool_idx - id of the zswap_pool that the entry's data is in.
> + * nid - NUMA node id of the page for which this is the zswap entry.
> * handle - zsmalloc allocation handle that stores the compressed page data
> * objcg - the obj_cgroup that the compressed memory is charged to
> * lru - handle to the pool's lru used to evict pages.
> */
> struct zswap_entry {
> swp_entry_t swpentry;
> - unsigned int length;
> - bool referenced;
> - u8 pool_idx;
> + struct {
> + unsigned int length:ZSWAP_MAX_COMPR_LENGTH_SHIFT;
> + bool referenced:1U;
> + u8 pool_idx;
> + };
[Severity: High]
Can entry->length overflow to 0 when storing an incompressible page on a
kernel with 256KB pages (PAGE_SHIFT == 18)?
With ZSWAP_MAX_COMPR_LENGTH_SHIFT defined as 18U, the 18-bit unsigned
bitfield can only hold values up to (1U << 18) - 1, whereas PAGE_SIZE is
(1UL << 18) when PAGE_SHIFT == 18.
When an incompressible page is stored with writeback enabled in
zswap_store_page() -> zswap_compress():
mm/zswap.c:zswap_compress() {
...
if (comp_ret) {
...
comp_ret = 0;
dlen = PAGE_SIZE;
dst = kmap_local_folio(folio, index * PAGE_SIZE);
mapped = true;
}
...
entry->handle = handle;
entry->length = dlen;
}
Assigning PAGE_SIZE to entry->length wraps the 18-bit bitfield to 0. In
zswap_store_page(), entry->length == 0 then skips initializing entry->lru and
adding it to the LRU:
mm/zswap.c:zswap_store_page() {
...
if (entry->length == PAGE_SIZE)
atomic_long_inc(&zswap_stored_incompressible_pages);
...
if (entry->length) {
INIT_LIST_HEAD(&entry->lru);
zswap_lru_add(entry);
}
}
Later, zswap_entry_free() unconditionally calls zswap_lru_del(entry) on the
uninitialized entry->lru list head, and __zswap_decompress() fails on swapin
because entry->length == PAGE_SIZE is false.
Should ZSWAP_MAX_COMPR_LENGTH_SHIFT be at least 19U (and the check in
zswap_setup() be BUILD_BUG_ON(ZSWAP_MAX_COMPR_LENGTH_SHIFT <= PAGE_SHIFT)) so
that entry->length can represent PAGE_SIZE?
[ ... ]
> @@ -1507,9 +1506,10 @@ static bool zswap_store_page(struct folio *folio, long index,
> {
> swp_entry_t page_swpentry = folio_swap_entry(folio, index);
> struct zswap_entry *entry, *old;
> + int nid = folio_nid(folio);
>
> /* allocate entry */
> - entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio));
> + entry = zswap_entry_cache_alloc(GFP_KERNEL);
> if (!entry) {
> zswap_reject_kmemcache_fail++;
> return false;
> @@ -1524,6 +1524,7 @@ static bool zswap_store_page(struct folio *folio, long index,
> * to an unrelated live pool.
> */
> entry->pool_idx = pool->idx;
> + entry->nid = nid;
[Severity: Medium]
What happens during cgroup or pool reclaim in shrink_memcg() if the swapped
folio resides on a NUMA node without normal memory (for example, a
ZONE_MOVABLE-only or ZONE_HIGHMEM-only node)?
Previously, entry_to_nid() derived the node from virt_to_page(entry), which
was allocated from slab with GFP_KERNEL and therefore always resided on a
node in N_NORMAL_MEMORY.
With entry->nid now set to folio_nid(folio) in zswap_store_page(), anonymous
and shmem folios allocated with GFP_HIGHUSER_MOVABLE can have an entry->nid
that is in N_MEMORY, but not in N_NORMAL_MEMORY. zswap_lru_add() inserts
those entries into zswap_list_lru for entry->nid, but shrink_memcg() only
walks N_NORMAL_MEMORY nodes:
mm/zswap.c:shrink_memcg() {
...
for_each_node_state(nid, N_NORMAL_MEMORY) {
unsigned long nr_to_walk = SWAP_CLUSTER_MAX;
shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg,
&shrink_memcg_cb, NULL, &nr_to_walk);
scanned += SWAP_CLUSTER_MAX - nr_to_walk;
}
}
Should shrink_memcg() iterate over N_MEMORY instead of N_NORMAL_MEMORY so
that entries from movable-only or highmem-only nodes can be reclaimed by
shrink_worker() and mem_cgroup_try_charge_zswap()?
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261028030503.17923-1-kanchanapsridhar2026@gmail.com?part=5
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 6/8] mm: zswap: Store large folios in batches.
2026-10-28 3:05 ` [PATCH v2 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
@ 2026-10-29 3:06 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-10-29 3:06 UTC (permalink / raw)
To: Kanchana P. Sridhar, hannes, yosry, nphamcs, chengming.zhou, akpm,
linux-mm, herbert, davem, linux-crypto
Cc: rientjes, willy, senozhatsky
Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
- [High] mm: zswap: fix error return on buffer alloc failure in zswap_cpu_comp_prepare()
- [Medium] mm: zswap: unsynchronized write to pool->compr_batch_size on CPU hotplug
--
commit 13a7e2ffeb93e24e6a8f400496ff51f01ce0c408
Author: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
mm: zswap: Store large folios in batches.
This commit introduces batching when storing large folios in zswap, allocating
multiple per-CPU compression destination buffers when supported by the
compressor and batch-allocating zswap entries from the slab cache. It also
caches the memcg writeback setting and folio NUMA node ID in a store context
and reorganizes struct zswap_pool for cacheline locality.
> diff --git a/mm/zswap.c b/mm/zswap.c
> index 19b7d60bc7d2a..c8890e8a9b836 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
[ ... ]
> @@ -866,10 +922,28 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
> goto fail;
> }
>
> + /*
> + * Allocate up to ZSWAP_MAX_BATCH_SIZE dst buffers if the
> + * compressor supports batching.
> + */
> + pool->compr_batch_size = min(ZSWAP_MAX_BATCH_SIZE,
> + crypto_acomp_batch_size(acomp_ctx->acomp));
[Severity: Medium]
Can updating pool->compr_batch_size inside the per-CPU hotplug callback race
with concurrent readers or cause a buffer count mismatch across CPUs?
In zswap_pool_create(), cpuhp_state_add_instance() runs zswap_cpu_comp_prepare()
only on currently online CPUs. If an offline CPU is brought online later while
the pool is active, zswap_cpu_comp_prepare() writes to pool->compr_batch_size
without synchronization while zswap_store() concurrently reads it:
store_batch_size = (pool->compr_batch_size > 1) ?
pool->compr_batch_size : ZSWAP_MAX_BATCH_SIZE;
Additionally, because each CPU independently calls crypto_alloc_acomp_node()
when brought online, if a higher-priority acomp implementation for
pool->tfm_name with a different batch_size is registered before a CPU comes
online, pool->compr_batch_size will be overwritten with the new value. When
zswap_pool_destroy() later frees each CPU's buffers using the single
pool->compr_batch_size:
for_each_possible_cpu(cpu)
acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu),
pool->compr_batch_size);
acomp_ctx_free() would iterate past the end of acomp_ctx->comp.buffers (or
leak buffers) for CPUs that were initialized with the earlier batch size.
> +
> ret = zswap_acomp_req_init(&acomp_ctx->comp.areq, acomp_ctx->acomp);
> if (ret)
> goto req_fail;
>
> + acomp_ctx->comp.buffers = kcalloc_node(pool->compr_batch_size, sizeof(u8 *),
> + GFP_KERNEL, nid);
> + if (!acomp_ctx->comp.buffers)
> + goto req_fail;
> +
> + for (i = 0; i < pool->compr_batch_size; ++i) {
> + acomp_ctx->comp.buffers[i] = kmalloc_node(PAGE_SIZE, GFP_KERNEL, nid);
> + if (!acomp_ctx->comp.buffers[i])
> + goto req_fail;
> + }
> +
> /* Skip the per-CPU request when decompression can use the stack. */
> if (acomp_is_async(acomp_ctx->acomp)) {
> ret = zswap_acomp_req_init(&acomp_ctx->decomp, acomp_ctx->acomp);
> @@ -882,12 +956,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
> req_fail:
> pr_err("could not alloc crypto acomp_request %s\n", pool->tfm_name);
> fail:
> - acomp_ctx_free(acomp_ctx);
> + acomp_ctx_free(acomp_ctx, pool->compr_batch_size);
> return ret;
> }
[Severity: High]
Does zswap_cpu_comp_prepare() return 0 (success) if kcalloc_node() or
kmalloc_node() fails when allocating acomp_ctx->comp.buffers?
Earlier in zswap_cpu_comp_prepare(), ret is initialized to -ENOMEM, but then
zswap_acomp_req_init() is called:
ret = zswap_acomp_req_init(&acomp_ctx->comp.areq, acomp_ctx->acomp);
if (ret)
goto req_fail;
When zswap_acomp_req_init() succeeds, ret is 0. If the subsequent
kcalloc_node() for acomp_ctx->comp.buffers or kmalloc_node() for
acomp_ctx->comp.buffers[i] fails and jumps to req_fail, ret is still 0.
At req_fail, acomp_ctx_free() frees the acomp state and sets
acomp_ctx->comp.buffers, acomp_ctx->comp.areq.req, and acomp_ctx->acomp to
NULL, and zswap_cpu_comp_prepare() returns 0 to the CPU hotplug core.
Subsequent calls to zswap_compress() on that CPU will then dereference NULL at
dst = comp_ctx->buffers[0].
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261028030503.17923-1-kanchanapsridhar2026@gmail.com?part=6
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 7/8] mm: zswap: Batched zswap_compress() for storing large folios in batches.
2026-10-28 3:05 ` [PATCH v2 7/8] mm: zswap: Batched zswap_compress() for storing " Kanchana P. Sridhar
@ 2026-10-29 3:06 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-10-29 3:06 UTC (permalink / raw)
To: Kanchana P. Sridhar, hannes, yosry, nphamcs, chengming.zhou, akpm,
linux-mm, herbert, davem, linux-crypto
Cc: rientjes, willy, senozhatsky
Thank you for your contribution! Sashiko AI review found 6 potential issue(s) to consider:
- [Critical] mm: zswap: unclamped nr_batch_pages in zswap_compress() causes OOB and crash
- [High] mm: zswap: zsmalloc handle leak on partial batch failure in zswap_compress()
- [High] mm: zswap: invalid and unreset dst scatterlist lengths in zswap_compress()
- [High] mm: zswap: zswap_cpu_comp_prepare() returns 0 on buffer or sg_table alloc fail
- [High] mm: zswap: zswap_compress() reads -EINPROGRESS sg->length for async acomp
- [Medium] mm: zswap: zswap_compress() stores PAGE_SIZE outputs when writeback disabled
--
commit 61c906de6e16dcc53db7f835142f9c05558dca5f
Author: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
mm: zswap: Batched zswap_compress() for storing large folios in batches.
This commit introduces a batched zswap_compress() implementation using
per-CPU scatterlists and refactors zsmalloc allocation into
zswap_update_entries().
> diff --git a/mm/zswap.c b/mm/zswap.c
> index c8890e8a9b836..4476fb4aeb5fd 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
[ ... ]
> @@ -944,6 +956,21 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
> goto req_fail;
> }
>
> + acomp_ctx->comp.sg_table = kmalloc(sizeof(*acomp_ctx->comp.sg_table), GFP_KERNEL);
> + if (!acomp_ctx->comp.sg_table)
> + goto req_fail;
> +
> + if (sg_alloc_table(acomp_ctx->comp.sg_table, pool->compr_batch_size,
> + GFP_KERNEL))
> + goto req_fail;
> +
> + /*
> + * Statically map the per-CPU destination buffers to the per-CPU
> + * SG lists; crucial for performance.
> + */
> + for_each_sg(acomp_ctx->comp.sg_table->sgl, sg, pool->compr_batch_size, i)
> + sg_set_buf(sg, acomp_ctx->comp.buffers[i], PAGE_SIZE);
> +
> /* Skip the per-CPU request when decompression can use the stack. */
> if (acomp_is_async(acomp_ctx->acomp)) {
> ret = zswap_acomp_req_init(&acomp_ctx->decomp, acomp_ctx->acomp);
> @@ -960,92 +987,161 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
> return ret;
> }
[Severity: High]
Does zswap_cpu_comp_prepare() return 0 on failure if allocating
acomp_ctx->comp.buffers, acomp_ctx->comp.sg_table, or sg_alloc_table()
fails?
Earlier in zswap_cpu_comp_prepare(), ret is overwritten by
zswap_acomp_req_init():
ret = zswap_acomp_req_init(&acomp_ctx->comp.areq, acomp_ctx->acomp);
if (ret)
goto req_fail;
When that call succeeds, ret is 0. If any of the subsequent allocations
for buffers or sg_table fail and jump to req_fail, acomp_ctx_free() tears
down and zeroes acomp_ctx, and zswap_cpu_comp_prepare() returns ret (0) to
the CPU hotplug core, leading to a NULL pointer dereference when zswap
later uses the per-CPU context.
> -static bool zswap_compress(struct folio *folio, long index,
> - struct zswap_entry *entry, struct zswap_pool *pool,
> - bool wb_enabled)
> +static bool zswap_update_entries(struct zswap_store_context *zc,
> + u8 batch_start)
> {
> - struct zswap_comp_ctx *comp_ctx;
> - struct scatterlist input, output;
> - int comp_ret = 0, alloc_ret = 0;
> - unsigned int dlen = PAGE_SIZE;
> + gfp_t gfp = GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE;
> + u8 i, batch_iter, comp_batch_iter;
> unsigned long handle;
> - gfp_t gfp;
> + unsigned int dlen;
> u8 *dst;
> - bool mapped = false;
> -
> - comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
> - mutex_lock(&comp_ctx->areq.mutex);
> -
> - dst = comp_ctx->buffers[0];
> - sg_init_table(&input, 1);
> - sg_set_folio(&input, folio, PAGE_SIZE, index * PAGE_SIZE);
> -
> - sg_init_one(&output, dst, PAGE_SIZE);
> - acomp_request_set_params(comp_ctx->areq.req, &input, &output,
> - PAGE_SIZE, dlen);
>
> /*
> - * it maybe looks a little bit silly that we send an asynchronous request,
> - * then wait for its completion synchronously. This makes the process look
> - * synchronous in fact.
> - * Theoretically, acomp supports users send multiple acomp requests in one
> - * acomp instance, then get those requests done simultaneously. but in this
> - * case, zswap actually does store and load page by page, there is no
> - * existing method to send the second page before the first page is done
> - * in one thread doing zswap.
> - * but in different threads running on different cpu, we have different
> - * acomp instance, and compression and decompression use separate
> - * requests, so multiple threads can do (de)compression in parallel.
> + * For each output SG list in @acomp_ctx->req->sg_table->sgl,
> + * the @sg->length should be set to either the page's compressed
> + * length (success), or it's negative value compression error.
> */
> - comp_ret = crypto_wait_req(crypto_acomp_compress(comp_ctx->areq.req),
> - &comp_ctx->areq.wait);
> - dlen = comp_ctx->areq.req->dlen;
> - if (!comp_ret) {
> - if (dlen == PAGE_SIZE)
> - comp_ret = -ENOSPC;
> - else if (WARN_ON_ONCE(!dlen || dlen > PAGE_SIZE))
> - comp_ret = -EINVAL;
> + for (comp_batch_iter = 0; comp_batch_iter < zc->nr_batch_pages;
> + ++comp_batch_iter) {
> + batch_iter = batch_start + comp_batch_iter;
> + dlen = zc->entries[batch_iter]->length;
> + handle = zs_malloc(zc->zs_pool, dlen, gfp, zc->nid);
> +
> + if (unlikely(IS_ERR_VALUE(handle))) {
> + if (PTR_ERR((void *)handle) == -ENOSPC)
> + zswap_reject_compress_poor++;
> + else
> + zswap_reject_alloc_fail++;
> +
> + for (i = 0; i < comp_batch_iter; ++i)
> + zs_free(zc->zs_pool, zc->entries[batch_start + i]->handle);
> +
> + return false;
> + }
> +
> + zc->entries[batch_iter]->handle = handle;
> +
> + if (likely(dlen < PAGE_SIZE)) {
> + dst = zc->comp_ctx->buffers[comp_batch_iter];
> + zs_obj_write(zc->zs_pool, handle, dst, dlen);
> + } else {
> + dst = kmap_local_folio(zc->folio,
> + (zc->folio_start + batch_iter) * PAGE_SIZE);
> + zs_obj_write(zc->zs_pool, handle, dst, dlen);
> + kunmap_local(dst);
> + }
> }
>
> + return true;
> +}
> +
> +/*
> + * zswap_compress() batching implementation for sequential and batching
> + * compressors.
> + *
> + * Description:
> + * ============
> + * Compress multiple @nr_pages in @folio starting from the @folio_start index
> + * in batches of @nr_batch_pages; the latter representing the lesser of the
> + * @nr_pages to be batch compressed and the compressor's batch size.
> + *
> + * @nr_pages can be in (1, ZSWAP_MAX_BATCH_SIZE] even if the compressor does not
> + * support batching.
> + *
> + * If @nr_batch_pages is 1, each page is processed sequentially.
> + *
> + * If @nr_batch_pages is > 1, the compressor may use batching to optimize
> + * compression.
> + */
> +static bool zswap_compress(struct zswap_store_context *zc)
> +{
> + u8 batch_start, comp_batch_iter, batch_iter;
> + struct scatterlist *sg;
> + bool ret;
> + int err;
> +
> /*
> - * If a page cannot be compressed into a size smaller than PAGE_SIZE,
> - * save the content as is without a compression, to keep the LRU order
> - * of writebacks. If writeback is disabled, reject the page since it
> - * only adds metadata overhead. swap_writeout() will put the page back
> - * to the active LRU list in the case.
> + * Compress the @nr_pages in @folio starting at index @folio_start
> + * in batches of @nr_batch_pages.
> */
> - if (comp_ret) {
> - if (!wb_enabled)
> - goto unlock;
> - comp_ret = 0;
> - dlen = PAGE_SIZE;
> - dst = kmap_local_folio(folio, index * PAGE_SIZE);
> - mapped = true;
> - }
> + for (batch_start = 0; batch_start < zc->nr_pages;
> + batch_start += zc->nr_batch_pages) {
> + /*
> + * Send @nr_batch_pages to crypto_acomp for compression:
> + *
> + * These pages are in @folio's range of indices in the interval
> + * [@folio_start + @batch_start,
> + * @folio_start + @batch_start + @nr_batch_pages).
> + *
> + * @slen indicates the total source length bytes for
> + * @nr_batch_pages.
> + *
> + * The source folio pages in the batch are directly submitted to
> + * crypto_acomp via acomp_request_set_src_folio().
> + *
> + * The pool's compressor batch size is at least @nr_batch_pages,
> + * hence the acomp_ctx has at least @nr_batch_pages dst @buffers.
> + */
> + acomp_request_set_src_folio(zc->comp_ctx->areq.req, zc->folio,
> + (zc->folio_start + batch_start) * PAGE_SIZE,
> + zc->slen);
> +
> + zc->comp_ctx->sg_table->sgl->length = zc->slen;
> +
> + /*
> + * The per-CPU @zc->comp_ctx->sg_table scatterlists are statically
> + * mapped to the per-CPU dst @buffers at pool creation time.
> + */
> + acomp_request_set_dst_sg(zc->comp_ctx->areq.req,
> + zc->comp_ctx->sg_table->sgl, zc->slen);
[Severity: High]
How are the destination scatterlist entry lengths in
zc->comp_ctx->sg_table->sgl maintained across batches when
zc->nr_batch_pages > 1?
In zswap_cpu_comp_prepare(), each acomp_ctx->comp.buffers[i] is allocated
with size PAGE_SIZE and mapped to sgl[i] via sg_set_buf(..., PAGE_SIZE).
Here in zswap_compress(), setting zc->comp_ctx->sg_table->sgl->length =
zc->slen sets sgl[0].length to zc->nr_batch_pages * PAGE_SIZE, which
exceeds the single PAGE_SIZE allocation backing buffers[0] when
zc->nr_batch_pages > 1.
At the same time, sgl[1 .. zc->nr_batch_pages - 1].length are never reset
to PAGE_SIZE before calling crypto_acomp_compress(). After the first batch
completes (or after a batch fails and jumps to compress_error), those SG
entries retain their previous compressed lengths or negative error codes on
subsequent requests.
> +
> + err = crypto_wait_req(crypto_acomp_compress(zc->comp_ctx->areq.req),
> + &zc->comp_ctx->areq.wait);
>
> - gfp = GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE;
> - handle = zs_malloc(pool->zs_pool, dlen, gfp, folio_nid(folio));
> - if (IS_ERR_VALUE(handle)) {
> - alloc_ret = PTR_ERR((void *)handle);
> - goto unlock;
> + /*
> + * If a page cannot be compressed into a size smaller than
> + * PAGE_SIZE, save the content as is without a compression, to
> + * keep the LRU order of writebacks. If writeback is disabled,
> + * reject the page since it only adds metadata overhead.
> + * swap_writeout() will put the page back to the active LRU list
> + * in the case.
> + *
> + * It is assumed that any compressor that sets the output length
> + * to 0 or a value >= PAGE_SIZE will also return a negative
> + * error status in @err; i.e, will not return a successful
> + * compression status in @err in this case.
> + */
> + if (unlikely(err && !zc->wb_enabled))
> + goto compress_error;
> +
> + for_each_sg(zc->comp_ctx->sg_table->sgl, sg,
> + zc->nr_batch_pages, comp_batch_iter) {
> + batch_iter = batch_start + comp_batch_iter;
> + if (unlikely(sg->length <= 0 || sg->length >= PAGE_SIZE))
> + sg->length = PAGE_SIZE;
> +
> + zc->entries[batch_iter]->length = sg->length;
> + }
[Severity: High]
What happens in zswap_compress() when using an asynchronous non-segmented
acomp driver?
In crypto/acompress.c:crypto_acomp_compress(), when req->unit_size ==
req->slen and !crypto_acomp_req_seg(tfm), the fast path runs:
if (likely((req->unit_size == req->slen) && acomp_request_issg(req) &&
!crypto_acomp_req_seg(tfm) && !acomp_req_on_stack(req))) {
int err = tfm->compress(req);
req->dst->length = unlikely(err) ? err : req->dlen;
return err;
}
When tfm->compress(req) is asynchronous and returns -EINPROGRESS or -EBUSY,
req->dst->length (sg->length) is set to -EINPROGRESS before
crypto_wait_req() waits for completion, and nothing updates sg->length to
req->dlen when the request finishes.
Wouldn't zswap_compress() then see sg->length == (unsigned int)-EINPROGRESS
after crypto_wait_req() returns 0, clamp sg->length to PAGE_SIZE, and store
every page uncompressed instead of using req->dlen?
[Severity: Medium]
Also, is the assumption that compressors never return 0 when the output
length is PAGE_SIZE true for software compressors?
Software compressors such as lz4 and zstd return 0 with dlen == PAGE_SIZE
when a page compresses to exactly PAGE_SIZE bytes. Previously,
zswap_compress() checked:
if (!comp_ret) {
if (dlen == PAGE_SIZE)
comp_ret = -ENOSPC;
else if (WARN_ON_ONCE(!dlen || dlen > PAGE_SIZE))
comp_ret = -EINVAL;
}
Without that check, when err == 0 and sg->length == PAGE_SIZE with
!zc->wb_enabled, won't zswap_compress() and zswap_update_entries() allocate
a PAGE_SIZE object in zsmalloc and store the uncompressed page even though
writeback is disabled?
> +
> + ret = zswap_update_entries(zc, batch_start);
> + if (unlikely(!ret))
> + return false;
> }
[ ... ]
> @@ -1681,11 +1785,10 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
> return true;
>
> store_pages_failed:
> - for (i = store_fail_idx; i < zc->nr_pages; ++i) {
> - if (!IS_ERR_VALUE(zc->entries[i]->handle))
> - zs_free(pool->zs_pool, zc->entries[i]->handle);
> - }
> + for (i = store_fail_idx; i < zc->nr_pages; ++i)
> + zs_free(pool->zs_pool, zc->entries[i]->handle);
>
> +compress_failed:
> zswap_entries_cache_free_batch(&zc->entries[store_fail_idx],
> zc->nr_pages - store_fail_idx);
>
[Severity: High]
Does this leak zsmalloc handles in zc->entries[0 .. batch_start - 1] when a
later sub-batch (batch_start > 0) fails in zswap_compress()?
For example, with a non-batching compressor (zc->nr_batch_pages == 1) and a
large folio (zc->nr_pages == 8), zswap_compress() iterates batch_start from
0 to zc->nr_pages - 1, calling zswap_update_entries(zc, batch_start) on
each iteration.
If sub-batch 0 succeeds (allocating zc->entries[0]->handle via zs_malloc())
and a later sub-batch at batch_start > 0 fails either in
crypto_acomp_compress() (with !zc->wb_enabled) or in zs_malloc() inside
zswap_update_entries(), zswap_update_entries() only frees handles in the
current sub-batch [batch_start, batch_start + comp_batch_iter):
for (i = 0; i < comp_batch_iter; ++i)
zs_free(zc->zs_pool, zc->entries[batch_start + i]->handle);
zswap_store_pages() then jumps to compress_failed, which skips the
zs_free() loop at store_pages_failed and frees zc->entries[], leaking all
zsmalloc handles allocated in [0, batch_start).
> @@ -1757,6 +1860,10 @@ bool zswap_store(struct folio *folio)
> store_batch_size = (pool->compr_batch_size > 1) ?
> pool->compr_batch_size : ZSWAP_MAX_BATCH_SIZE;
>
> + zc.nr_batch_pages = pool->compr_batch_size;
> + zc.slen = zc.nr_batch_pages * PAGE_SIZE;
> + zc.zs_pool = pool->zs_pool;
> +
> for (start = 0; start < nr_pages; start += store_batch_size) {
> end = min(start + store_batch_size, nr_pages);
>
[Severity: Critical]
What happens in zswap_store() -> zswap_store_pages() -> zswap_compress()
when pool->compr_batch_size > 1 and the folio (or trailing batch end -
start) has fewer pages than pool->compr_batch_size, such as an order-0
folio where nr_pages == 1?
In zswap_store(), zc.nr_batch_pages is set to pool->compr_batch_size and
zc.slen is set to zc.nr_batch_pages * PAGE_SIZE, while zswap_store_pages()
sets zc->nr_pages = end - start and only allocates zc->nr_pages entries in
zc->entries[0 .. zc->nr_pages - 1].
Because zc->nr_batch_pages and zc->slen are never clamped to zc->nr_pages,
when zc->nr_pages < zc->nr_batch_pages:
1. acomp_request_set_src_folio() in zswap_compress() maps zc->slen bytes,
reading past the end of zc->folio.
2. Both the for_each_sg() loop in zswap_compress() and the loop in
zswap_update_entries() iterate comp_batch_iter up to zc->nr_batch_pages,
dereferencing zc->entries[batch_iter] for batch_iter >= zc->nr_pages. On
the first batch, zc->entries[zc->nr_pages] is NULL (causing a NULL
pointer dereference); on a trailing partial batch after a full batch, it
contains stale zswap_entry pointers from the previous batch (corrupting
live entries and leaking the newly allocated zsmalloc handles).
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261028030503.17923-1-kanchanapsridhar2026@gmail.com?part=7
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 8/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
2026-10-28 3:05 ` [PATCH v2 8/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations Kanchana P. Sridhar
@ 2026-10-29 3:06 ` sashiko-bot
0 siblings, 0 replies; 16+ messages in thread
From: sashiko-bot @ 2026-10-29 3:06 UTC (permalink / raw)
To: Kanchana P. Sridhar, hannes, yosry, nphamcs, chengming.zhou, akpm,
linux-mm, herbert, davem, linux-crypto
Cc: rientjes, willy, senozhatsky
Thank you for your contribution! Sashiko AI review found 3 potential issue(s) to consider:
- [High] mm: zswap: memcg counter corruption in zswap_account_stores()
- [High] mm: zswap: missing zs_free() of prior sub-batches when zswap_compress() fails
- [Low] mm: zswap: unused zswap_pool_get() retained with __maybe_unused
--
commit 5a1c181c752a45f404bc7cfeb7961fd6accb0903
Author: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
This commit refactors zswap_store_pages() into smaller helper functions
(zswap_init_entries(), zswap_tree_stores(), and zswap_account_stores()) to
process batches of pages when storing a large folio. It also batches pool and
obj_cgroup reference counting, memcg charging, and zswap_stored_pages updates
across the batch.
> diff --git a/mm/zswap.c b/mm/zswap.c
> index 4476fb4aeb5fd..2831284cfc903 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
> @@ -504,12 +504,18 @@ static int __must_check zswap_pool_tryget(struct zswap_pool *pool)
> }
>
> /* The caller must already have a reference. */
> -static void zswap_pool_get(struct zswap_pool *pool)
> +static void __maybe_unused zswap_pool_get(struct zswap_pool *pool)
> {
> percpu_ref_get(&pool->ref);
> }
[Severity: Low]
Since the only call site in zswap_store_pages() is replaced by
zswap_pool_get_many(), zswap_pool_get() no longer has any callers in the
file. Could zswap_pool_get() be removed outright instead of marking it
__maybe_unused?
>
> -static void zswap_pool_put(struct zswap_pool *pool)
> +/* The caller must already have a reference. */
> +static inline void zswap_pool_get_many(struct zswap_pool *pool, u8 nr)
> +{
> + percpu_ref_get_many(&pool->ref, nr);
> +}
[ ... ]
> @@ -1750,24 +1732,30 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
[ ... ]
> +static inline void zswap_account_stores(struct zswap_store_context *zc,
> + struct zswap_pool *pool,
> + struct obj_cgroup *objcg)
> +{
> + size_t total_batch_comp_length = 0;
> + u8 i;
> +
> + for (i = 0; i < zc->nr_pages; ++i) {
> + if (unlikely(zc->entries[i]->length == PAGE_SIZE))
> atomic_long_inc(&zswap_stored_incompressible_pages);
>
> + total_batch_comp_length += zc->entries[i]->length;
> +
> /*
> * We finish by adding the entry to the LRU while it's already
> * in xarray. This is safe because:
> @@ -1778,20 +1766,78 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
> * The publishing order matters to prevent writeback from seeing
> * an incoherent entry.
> */
> - if (likely(entry->length))
> - zswap_lru_add(entry);
> + if (likely(zc->entries[i]->length))
> + zswap_lru_add(zc->entries[i]);
> }
>
> + /*
> + * The entry is successfully compressed and stored in the tree,
> + * and further failures will be cleaned up in zswap_store().
> + * For the batch, grab refs to the pool and objcg, charge zswap memory,
> + * and increment zswap_stored_pages by @zc->nr_pages. The opposite
> + * actions will be performed by zswap_entry_free() when the entry is
> + * removed from the tree.
> + */
> + zswap_pool_get_many(pool, zc->nr_pages);
> + if (objcg) {
> + obj_cgroup_get_many(objcg, zc->nr_pages);
> + obj_cgroup_charge_zswap(objcg, total_batch_comp_length);
> + }
> + atomic_long_add(zc->nr_pages, &zswap_stored_pages);
> +}
[Severity: High]
Does calling obj_cgroup_charge_zswap(objcg, total_batch_comp_length) once per
batch corrupt the MEMCG_ZSWAPPED and MEMCG_ZSWAP_INCOMP counters when
zc->nr_pages > 1?
Looking at obj_cgroup_charge_zswap() in mm/memcontrol.c, it assumes @size
represents a single page:
mm/memcontrol.c:obj_cgroup_charge_zswap() {
...
rcu_read_lock();
memcg = obj_cgroup_memcg(objcg);
mod_memcg_state(memcg, MEMCG_ZSWAP_B, size);
mod_memcg_state(memcg, MEMCG_ZSWAPPED, 1);
if (size == PAGE_SIZE)
mod_memcg_state(memcg, MEMCG_ZSWAP_INCOMP, 1);
rcu_read_unlock();
}
When a batch of zc->nr_pages > 1 pages from a large folio is stored via
zswap_store() -> zswap_store_pages() -> zswap_account_stores(),
MEMCG_ZSWAPPED is only incremented by 1 instead of zc->nr_pages, and
MEMCG_ZSWAP_INCOMP compares the summed total_batch_comp_length against
PAGE_SIZE rather than counting individual incompressible pages.
Later, when each entry in the batch is freed via zswap_entry_free():
mm/zswap.c:zswap_entry_free() {
...
if (entry->objcg) {
obj_cgroup_uncharge_zswap(entry->objcg, entry->length);
obj_cgroup_put(entry->objcg);
}
...
}
obj_cgroup_uncharge_zswap() is called once per entry, decrementing
MEMCG_ZSWAPPED by zc->nr_pages total and decrementing MEMCG_ZSWAP_INCOMP for
each entry whose length == PAGE_SIZE.
This causes both MEMCG_ZSWAPPED and MEMCG_ZSWAP_INCOMP to be miscounted and
underflow, which also breaks zswap_shrinker_count() where MEMCG_ZSWAPPED is
used as nr_stored to scale reclaim targets.
> +
> +/*
> + * Store multiple pages in @folio, starting from the page at index @start up to
> + * the page at index @end-1.
> + */
> +static bool zswap_store_pages(struct zswap_store_context *zc,
> + long start,
> + long end,
> + struct zswap_pool *pool,
> + struct obj_cgroup *objcg)
> +{
> + bool ret;
> + u8 i;
> +
> + zc->nr_pages = end - start;
> + zc->folio_start = start;
> +
> + VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
> +
> + if (unlikely(!zswap_entries_cache_alloc_batch(zc, GFP_KERNEL)))
> + return false;
> +
> + zswap_init_entries(zc, pool, objcg);
> +
> + /*
> + * Locking the comp_ctx mutex once per store batch results in better
> + * performance as compared to locking per compress batch.
> + */
> + zc->comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
> + mutex_lock(&zc->comp_ctx->areq.mutex);
> +
> + ret = zswap_compress(zc);
> +
> + mutex_unlock(&zc->comp_ctx->areq.mutex);
> +
> + if (unlikely(!ret))
> + goto compress_failed;
> +
> + ret = zswap_tree_stores(zc);
> +
> + if (unlikely(!ret))
> + goto store_pages_failed;
> +
> + zswap_account_stores(zc, pool, objcg);
> +
> return true;
>
> store_pages_failed:
> - for (i = store_fail_idx; i < zc->nr_pages; ++i)
> + for (i = 0; i < zc->nr_pages; ++i)
> zs_free(pool->zs_pool, zc->entries[i]->handle);
>
> compress_failed:
> - zswap_entries_cache_free_batch(&zc->entries[store_fail_idx],
> - zc->nr_pages - store_fail_idx);
> -
> + zswap_entries_cache_free_batch(zc->entries, zc->nr_pages);
> return false;
> }
[Severity: High]
Can jumping to compress_failed when zswap_compress() returns false leak the
zsmalloc handles allocated in earlier compression sub-batches?
In zswap_compress(), pages in [0, zc->nr_pages) are compressed in sub-batches
of zc->nr_batch_pages (which is 1 for software compressors such as ZSTD or
LZ4 when zc->nr_pages is up to ZSWAP_MAX_BATCH_SIZE):
mm/zswap.c:zswap_compress() {
for (batch_start = 0; batch_start < zc->nr_pages;
batch_start += zc->nr_batch_pages) {
...
err = crypto_wait_req(crypto_acomp_compress(zc->comp_ctx->areq.req),
&zc->comp_ctx->areq.wait);
if (unlikely(err && !zc->wb_enabled))
goto compress_error;
...
ret = zswap_update_entries(zc, batch_start);
if (unlikely(!ret))
return false;
}
}
Each successful call to zswap_update_entries(zc, batch_start) allocates
zs_malloc() handles for zc->entries[batch_start .. batch_start +
zc->nr_batch_pages - 1]. If a subsequent iteration with batch_start > 0 fails
at goto compress_error (when !zc->wb_enabled and a page is incompressible) or
fails in zswap_update_entries() (which only frees handles in the current
sub-batch starting at batch_start):
mm/zswap.c:zswap_update_entries() {
...
if (unlikely(IS_ERR_VALUE(handle))) {
...
for (i = 0; i < comp_batch_iter; ++i)
zs_free(zc->zs_pool, zc->entries[batch_start + i]->handle);
return false;
}
...
}
zswap_compress() returns false while zc->entries[0 .. batch_start - 1]->handle
still hold allocated zsmalloc objects. In zswap_store_pages(), jumping to
compress_failed then frees zc->entries back to the slab cache without calling
zs_free() on those handles.
--
Sashiko AI review · https://sashiko.dev/#/patchset/20261028030503.17923-1-kanchanapsridhar2026@gmail.com?part=8
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 3/8] crypto: acomp - Add trivial segmentation wrapper.
2026-10-29 3:05 ` sashiko-bot
@ 2026-10-29 5:09 ` Kanchana P. Sridhar
0 siblings, 0 replies; 16+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-29 5:09 UTC (permalink / raw)
To: sashiko-reviews
Cc: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, rientjes, willy, senozhatsky,
Kanchana P. Sridhar
On Wed, Oct 28, 2026 at 8:06 PM <sashiko-bot@kernel.org> wrote:
>
> Thank you for your contribution! Sashiko AI review found 2 potential issue(s) to consider:
> - [Critical] crypto: acomp - missing async check in crypto_acomp_compress() fast path
> - [High] crypto: acomp: uninitialized state->src/dst restore in acomp_reqchain_virt()
> --
Thanks for the detailed code review!
>
> commit 7144b07b2cc5b6704699fb53fac44c4270d31f7b
> Author: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
>
> crypto: acomp - Add trivial segmentation wrapper.
>
> This commit adds a single-segment compression wrapper in crypto/acompress.c
> that stores the compressed length or error code into req->dst->length when
> req->unit_size is set on transforms that do not support segmentation. It also
> adds an inline fast path in crypto_acomp_compress() for single-segment
> scatterlist requests.
>
> > diff --git a/crypto/acompress.c b/crypto/acompress.c
> > index c8b4b8589769e..c55b0e8eff3d8 100644
> > --- a/crypto/acompress.c
> > +++ b/crypto/acompress.c
> > @@ -166,14 +166,20 @@ static void acomp_save_req(struct acomp_req *req, crypto_completion_t cplt)
> >
> > state->compl = req->base.complete;
> > state->data = req->base.data;
> > + state->unit_size = req->unit_size;
> > + state->flags = req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
> > + CRYPTO_ACOMP_REQ_DST_VIRT);
>
> [Severity: High]
> Can moving the state->flags assignment from acomp_virt_to_sg() into
> acomp_save_req() corrupt req->svirt and req->dvirt when a request is
> configured with acomp_request_set_src_nondma() or
> acomp_request_set_dst_nondma()?
>
> When a caller sets up a non-dma virtual request via
> acomp_request_set_src_nondma() or acomp_request_set_dst_nondma(), both the
> non-dma and virtual request flags are set in req->base.flags. When
> crypto_acomp_compress() or crypto_acomp_decompress() calls
> acomp_do_req_chain(), the call chain is:
>
> crypto_acomp_compress()
> acomp_do_req_chain()
> acomp_save_req()
> state->flags = req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
> CRYPTO_ACOMP_REQ_DST_VIRT);
> acomp_do_one_req()
> if (acomp_request_isnondma(req))
> return acomp_do_nondma(req, comp);
>
> Because acomp_do_one_req() returns early through acomp_do_nondma() without
> calling acomp_virt_to_sg(), state->src and state->dst are never initialized
> from req->svirt and req->dvirt.
>
> When acomp_reqchain_finish() then calls acomp_reqchain_virt() in
> crypto/acompress.c:
>
> static void acomp_reqchain_virt(struct acomp_req *req)
> {
> struct acomp_req_chain *state = &req->chain;
> unsigned int slen = req->slen;
> unsigned int dlen = req->dlen;
>
> if (state->flags & CRYPTO_ACOMP_REQ_SRC_VIRT)
> acomp_request_set_src_dma(req, state->src, slen);
> if (state->flags & CRYPTO_ACOMP_REQ_DST_VIRT)
> acomp_request_set_dst_dma(req, state->dst, dlen);
> }
>
> won't acomp_reqchain_virt() see the virtual flags set in state->flags and
> overwrite req->svirt and req->dvirt with the uninitialized state->src and
> state->dst pointers (while also clearing the non-dma flags), causing a null
> or wild pointer dereference if acomp_reqchain_finish() then writes to
> req->dst->length when req->unit_size is non-zero?
>
Herbert, can you please take a look at this and suggest what changes
need to be made here, since you are the original developer of this
portion of the patch. Thanks.
> [ ... ]
>
> > @@ -279,10 +289,27 @@ int crypto_acomp_compress(struct acomp_req *req)
> > {
> > struct crypto_acomp *tfm = crypto_acomp_reqtfm(req);
> >
> > + if (likely((req->unit_size == req->slen) && acomp_request_issg(req) &&
> > + !crypto_acomp_req_seg(tfm) && !acomp_req_on_stack(req))) {
> > + int err = tfm->compress(req);
> > +
> > + req->dst->length = unlikely(err) ? err : req->dlen;
> > + return err;
> > + }
>
> [Severity: Critical]
> Should this fast path in crypto_acomp_compress() in crypto/acompress.c check
> !acomp_is_async(tfm) instead of !acomp_req_on_stack(req)?
Thanks for the catch! Yes, I will add (!acomp_is_async(tfm) &&
!acomp_req_on_stack(req)) here.
Since zswap is the only current user who, with this patch series,
submits the req as SG lists, I was assuming that async algorithms
would set the segmentation bit, and avoid this short-circuit. But
until that happens, the above check will take care of this.
>
> Callers using asynchronous hardware compression transforms (where
> acomp_is_async(tfm) is true, such as qat, hisi_zip, or iaa in async mode)
> allocate requests on the heap with acomp_request_alloc(), so
> !acomp_req_on_stack(req) is true for asynchronous requests.
>
> When zswap_compress() submits a single-segment scatterlist request with
> req->unit_size == req->slen to an asynchronous transform that does not
> support segmentation, this fast path will be taken instead of
> acomp_do_req_chain(req, true):
>
> crypto/acompress.c:crypto_acomp_compress() {
> if (likely((req->unit_size == req->slen) && acomp_request_issg(req) &&
> !crypto_acomp_req_seg(tfm) && !acomp_req_on_stack(req))) {
> int err = tfm->compress(req);
>
> req->dst->length = unlikely(err) ? err : req->dlen;
> return err;
> }
> }
>
> When tfm->compress(req) submits the descriptor to hardware and returns an
> in-progress or busy status, won't this immediately overwrite req->dst->length
> with the negative error value while hardware dma is still in flight on
> req->dst, and skip registering acomp_reqchain_done() via acomp_save_req() so
> that req->dst->length is never updated to req->dlen upon async completion?
>
The augmented check for async algs should take care of this.
Thanks,
Kanchana
> --
> Sashiko AI review · https://sashiko.dev/#/patchset/20261028030503.17923-1-kanchanapsridhar2026@gmail.com?part=3
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2026-10-29 5:09 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-10-28 3:04 [PATCH v2 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 2/8] crypto: acomp - Add bit to indicate segmentation support Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 3/8] crypto: acomp - Add trivial segmentation wrapper Kanchana P. Sridhar
2026-10-29 3:05 ` sashiko-bot
2026-10-29 5:09 ` Kanchana P. Sridhar
2026-10-28 3:04 ` [PATCH v2 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
2026-10-29 3:06 ` sashiko-bot
2026-10-28 3:05 ` [PATCH v2 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
2026-10-29 3:06 ` sashiko-bot
2026-10-28 3:05 ` [PATCH v2 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
2026-10-29 3:06 ` sashiko-bot
2026-10-28 3:05 ` [PATCH v2 7/8] mm: zswap: Batched zswap_compress() for storing " Kanchana P. Sridhar
2026-10-29 3:06 ` sashiko-bot
2026-10-28 3:05 ` [PATCH v2 8/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations Kanchana P. Sridhar
2026-10-29 3:06 ` sashiko-bot
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox