* [PATCH v1 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE.
2026-10-08 18:28 [PATCH v1 0/8] Batched zswap_store() with compression batching Kanchana P. Sridhar
@ 2026-10-08 18:28 ` Kanchana P. Sridhar
2026-10-08 18:28 ` [PATCH v1 2/8] crypto: acomp - Add bit to indicate segmentation support Kanchana P. Sridhar
` (6 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-08 18:28 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
We add a new @unit_size data member to struct acomp_req along with a
helper function acomp_request_set_unit_size() for kernel modules to set
the unit size to use while breaking down the request's src/dst
scatterlists.
An acomp_alg can implement batching by using the @req->unit_size to
break down the SG lists passed in via @req->dst and/or @req->src, to
submit individual @req->slen/@req->unit_size compress jobs or
@req->dlen/@req->unit_size decompress jobs, for batch compression and
batch decompression respectively.
In case of batch compression, the folio's pages for the batch can be
retrieved from the @req->src scatterlist by using a struct sg_page_iter
after determining the number of pages as @req->slen/@req->unit_size.
Please note that acomp_request_set_callback() sets the @req->unit_size
to 0.
In zswap_cpu_comp_prepare(), after the call to
acomp_request_set_callback(), we set the unit size for zswap to
PAGE_SIZE.
acomp_request_set_unit_size(acomp_ctx->req, PAGE_SIZE);
Suggested-by: Herbert Xu <herbert@gondor.apana.org.au>
Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
include/crypto/acompress.h | 49 ++++++++++++++++++++++++++++++++++++++
mm/zswap.c | 6 +++++
2 files changed, 55 insertions(+)
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index 5d5358dfab73..6462fc46f4ec 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -79,6 +79,7 @@ struct acomp_req_chain {
* @dvirt: Destination virtual address
* @slen: Size of the input buffer
* @dlen: Size of the output buffer and number of bytes produced
+ * @unit_size: Unit size for the request for use in batching
* @chain: Private API code data, do not use
* @__ctx: Start of private context data
*/
@@ -94,6 +95,7 @@ struct acomp_req {
};
unsigned int slen;
unsigned int dlen;
+ unsigned int unit_size;
struct acomp_req_chain chain;
@@ -329,9 +331,56 @@ static inline void acomp_request_set_callback(struct acomp_req *req,
{
flgs &= ~CRYPTO_ACOMP_REQ_PRIVATE;
flgs |= req->base.flags & CRYPTO_ACOMP_REQ_PRIVATE;
+ req->unit_size = 0;
crypto_request_set_callback(&req->base, flgs, cmpl, data);
}
+/**
+ * acomp_request_set_unit_size() -- Sets the unit size for the request.
+ *
+ * This is a helper function that enables batching for zswap, IPComp, etc.
+ * It allows multiple independent compression (or decompression) operations to
+ * be submitted in a single request's SG lists, where each SG list ("segment")
+ * is processed independently. The unit size helps derive segments from a
+ * single request. crypto_acomp does not expect the segments to be related in
+ * any way.
+ *
+ * Example usage model:
+ *
+ * A module such as zswap that's configured to use a batching compressor, can
+ * accomplish batch compression of "nr_pages" with crypto_acomp by creating an
+ * output SG table for the batch, initialized to contain "nr_pages" SG
+ * lists. Each scatterlist is mapped to the nth destination buffer for the
+ * batch. Depending on whether the @req is used for batch compress/decompress,
+ * zswap must set the @req's source/destination length to be
+ * "nr_pages * @req->unit_size" respectively.
+ *
+ * An acomp_alg can implement batch compression by using the @req->unit_size
+ * to break down the SG lists passed in via @req->dst to submit individual
+ * "@req->slen/@req->unit_size" compress jobs to be processed as a batch.
+ *
+ * Similarly, zswap can implement batch decompression by passing an
+ * SG table with "nr_pages" SG lists via @req->src to process
+ * "@req->dlen/@req->unit_size" decompress jobs as a batch.
+ *
+ * This API must be called after acomp_request_set_callback(),
+ * which sets @req->unit_size to 0. This makes it easy for users of
+ * crypto_acomp to rely on a default of not opting in to batching.
+ * Users such as zswap opt in to batching by defining @req->unit_size
+ * to a non-zero value for use by acomp_algs supporting batching.
+ *
+ * @du would be PAGE_SIZE for zswap, it could be the MTU for IPsec.
+ *
+ * @req: asynchronous compress/decompress request
+ * @du: data unit size of the input/output buffer scatterlist.
+ */
+static inline void acomp_request_set_unit_size(struct acomp_req *req,
+ unsigned int du)
+{
+ req->unit_size = du;
+}
+
+
/**
* acomp_request_set_params() -- Sets request parameters
*
diff --git a/mm/zswap.c b/mm/zswap.c
index ae19e301fced..ca513a9f89e3 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -845,6 +845,12 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
acomp_request_set_callback(acomp_ctx->req, CRYPTO_TFM_REQ_MAY_BACKLOG,
crypto_req_done, &acomp_ctx->wait);
+ /*
+ * This should be called after acomp_request_set_callback(),
+ * which initializes req->unit_size to 0.
+ */
+ acomp_request_set_unit_size(acomp_ctx->req, PAGE_SIZE);
+
mutex_init(&acomp_ctx->mutex);
return 0;
--
2.39.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v1 2/8] crypto: acomp - Add bit to indicate segmentation support.
2026-10-08 18:28 [PATCH v1 0/8] Batched zswap_store() with compression batching Kanchana P. Sridhar
2026-10-08 18:28 ` [PATCH v1 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
@ 2026-10-08 18:28 ` Kanchana P. Sridhar
2026-10-08 18:28 ` [PATCH v1 3/8] crypto: acomp - Add trivial segmentation wrapper Kanchana P. Sridhar
` (5 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-08 18:28 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
This patch adds segmentation support for compression.
Add a bit to the crypto_alg flags to indicate support for segmentation.
Also add a helper for acomp to test whether a given tfm supports
segmentation.
Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
include/crypto/algapi.h | 5 +++++
include/crypto/internal/acompress.h | 5 +++++
include/linux/crypto.h | 3 +++
3 files changed, 13 insertions(+)
diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
index 05deea9dac5e..7d406cfe5751 100644
--- a/include/crypto/algapi.h
+++ b/include/crypto/algapi.h
@@ -280,6 +280,11 @@ static inline bool crypto_tfm_req_virt(struct crypto_tfm *tfm)
return tfm->__crt_alg->cra_flags & CRYPTO_ALG_REQ_VIRT;
}
+static inline bool crypto_tfm_req_seg(struct crypto_tfm *tfm)
+{
+ return tfm->__crt_alg->cra_flags & CRYPTO_ALG_REQ_SEG;
+}
+
static inline u32 crypto_request_flags(struct crypto_async_request *req)
{
return req->flags & ~CRYPTO_TFM_REQ_ON_STACK;
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 9cd37df32dc4..93ae59623f13 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -189,6 +189,11 @@ static inline bool crypto_acomp_req_virt(struct crypto_acomp *tfm)
return crypto_tfm_req_virt(&tfm->base);
}
+static inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm)
+{
+ return crypto_tfm_req_seg(&tfm->base);
+}
+
void crypto_acomp_free_streams(struct crypto_acomp_streams *s);
int crypto_acomp_alloc_streams(struct crypto_acomp_streams *s);
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index b7c97f1c47c9..78c699ac3f82 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -139,6 +139,9 @@
/* Set if the algorithm cannot have a fallback (e.g., phmac). */
#define CRYPTO_ALG_NO_FALLBACK 0x00080000
+/* Set if the algorithm supports segmentation. */
+#define CRYPTO_ALG_REQ_SEG 0x00100000
+
/* The high bits 0xff000000 are reserved for type-specific flags. */
/*
--
2.39.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v1 3/8] crypto: acomp - Add trivial segmentation wrapper.
2026-10-08 18:28 [PATCH v1 0/8] Batched zswap_store() with compression batching Kanchana P. Sridhar
2026-10-08 18:28 ` [PATCH v1 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
2026-10-08 18:28 ` [PATCH v1 2/8] crypto: acomp - Add bit to indicate segmentation support Kanchana P. Sridhar
@ 2026-10-08 18:28 ` Kanchana P. Sridhar
2026-10-08 18:28 ` [PATCH v1 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
` (4 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-08 18:28 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
This patch provides a wrapper for existing algorithms so that they can
accept a single segment while returning the compressed length or error
through the dst SG list length.
This trivial segmentation wrapper only supports compression with a
segment count of exactly one.
The reason is that the first user zswap will only allocate the
extra memory if the underlying algorithm supports segmentation,
and otherwise only one segment will be given at a time.
Having this wrapper means that the same calling convention can
be used for all algorithms, regardless of segmentation support.
Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
crypto/acompress.c | 33 ++++++++++++++++++++++++++-------
include/crypto/acompress.h | 1 +
2 files changed, 27 insertions(+), 7 deletions(-)
diff --git a/crypto/acompress.c b/crypto/acompress.c
index 4de1a2ad577f..8c9a76073159 100644
--- a/crypto/acompress.c
+++ b/crypto/acompress.c
@@ -166,14 +166,20 @@ static void acomp_save_req(struct acomp_req *req, crypto_completion_t cplt)
state->compl = req->base.complete;
state->data = req->base.data;
+ state->unit_size = req->unit_size;
+ state->flags = req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
+ CRYPTO_ACOMP_REQ_DST_VIRT);
+
req->base.complete = cplt;
req->base.data = req;
+ req->unit_size = 0;
}
static void acomp_restore_req(struct acomp_req *req)
{
req->base.complete = req->chain.compl;
req->base.data = req->chain.data;
+ req->unit_size = req->chain.unit_size;
}
static void acomp_reqchain_virt(struct acomp_req *req)
@@ -192,9 +198,6 @@ static void acomp_virt_to_sg(struct acomp_req *req)
{
struct acomp_req_chain *state = &req->chain;
- state->flags = req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
- CRYPTO_ACOMP_REQ_DST_VIRT);
-
if (acomp_request_src_isvirt(req)) {
unsigned int slen = req->slen;
const u8 *svirt = req->svirt;
@@ -242,6 +245,10 @@ static int acomp_reqchain_finish(struct acomp_req *req, int err)
{
acomp_reqchain_virt(req);
acomp_restore_req(req);
+
+ if (req->unit_size)
+ req->dst->length = unlikely(err) ? err : req->dlen;
+
return err;
}
@@ -266,10 +273,13 @@ static int acomp_do_req_chain(struct acomp_req *req, bool comp)
{
int err;
+ if (unlikely(req->unit_size && req->slen > req->unit_size))
+ return -ENOSYS;
+
acomp_save_req(req, acomp_reqchain_done);
err = acomp_do_one_req(req, comp);
- if (err == -EBUSY || err == -EINPROGRESS)
+ if (unlikely(err == -EBUSY || err == -EINPROGRESS))
return err;
return acomp_reqchain_finish(req, err);
@@ -279,10 +289,19 @@ int crypto_acomp_compress(struct acomp_req *req)
{
struct crypto_acomp *tfm = crypto_acomp_reqtfm(req);
- if (acomp_req_on_stack(req) && acomp_is_async(tfm))
+ if (unlikely(acomp_req_on_stack(req) && acomp_is_async(tfm)))
return -EAGAIN;
+
+ if (likely(req->unit_size && acomp_request_issg(req))) {
+ if (!crypto_acomp_req_seg(tfm))
+ return acomp_do_req_chain(req, true);
+
+ return tfm->compress(req);
+ }
+
if (crypto_acomp_req_virt(tfm) || acomp_request_issg(req))
- return crypto_acomp_reqtfm(req)->compress(req);
+ return tfm->compress(req);
+
return acomp_do_req_chain(req, true);
}
EXPORT_SYMBOL_GPL(crypto_acomp_compress);
@@ -294,7 +313,7 @@ int crypto_acomp_decompress(struct acomp_req *req)
if (acomp_req_on_stack(req) && acomp_is_async(tfm))
return -EAGAIN;
if (crypto_acomp_req_virt(tfm) || acomp_request_issg(req))
- return crypto_acomp_reqtfm(req)->decompress(req);
+ return tfm->decompress(req);
return acomp_do_req_chain(req, false);
}
EXPORT_SYMBOL_GPL(crypto_acomp_decompress);
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index 6462fc46f4ec..fc0665b77a8f 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -67,6 +67,7 @@ struct acomp_req_chain {
struct folio *dfolio;
};
u32 flags;
+ u32 unit_size;
};
/**
--
2.39.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v1 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size.
2026-10-08 18:28 [PATCH v1 0/8] Batched zswap_store() with compression batching Kanchana P. Sridhar
` (2 preceding siblings ...)
2026-10-08 18:28 ` [PATCH v1 3/8] crypto: acomp - Add trivial segmentation wrapper Kanchana P. Sridhar
@ 2026-10-08 18:28 ` Kanchana P. Sridhar
2026-10-08 18:28 ` [PATCH v1 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
` (3 subsequent siblings)
7 siblings, 0 replies; 11+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-08 18:28 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
This commit adds a @batch_size data member to struct acomp_alg.
An acomp_alg compression algorithm that supports batching of
compressions and decompressions must provide a @batch_size greater
than 1, representing the maximum batch-size that the compressor
supports, so that kernel users of crypto_acomp, such as zswap, can
allocate resources for submitting multiple compress/decompress jobs that
can be batched, and invoke batching of [de]compressions.
The new crypto_acomp_batch_size() API queries the crypto_acomp's
acomp_alg for the batch-size. If the acomp_alg has registered a
@batch_size greater than 1, this is returned. If not, a default of "1"
is returned.
zswap can invoke crypto_acomp_batch_size() to query the maximum number
of requests that can be batch [de]compressed. Based on this, zswap
can use the minimum of any zswap-specific upper limits for batch-size
and the compressor's max @batch_size, to allocate batching resources.
This enables zswap to compress/decompress multiple pages as a batch
to improve swapout/swapin performance and memory savings.
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
crypto/acompress.c | 14 ++++++++++++++
include/crypto/acompress.h | 12 ++++++++++++
include/crypto/internal/acompress.h | 3 +++
3 files changed, 29 insertions(+)
diff --git a/crypto/acompress.c b/crypto/acompress.c
index 8c9a76073159..192ea996e779 100644
--- a/crypto/acompress.c
+++ b/crypto/acompress.c
@@ -318,6 +318,20 @@ int crypto_acomp_decompress(struct acomp_req *req)
}
EXPORT_SYMBOL_GPL(crypto_acomp_decompress);
+unsigned int crypto_acomp_batch_size(struct crypto_acomp *tfm)
+{
+ if (acomp_is_async(tfm) &&
+ (crypto_comp_alg_common(tfm)->base.cra_flags & CRYPTO_ALG_TYPE_ACOMPRESS)) {
+ struct acomp_alg *alg = crypto_acomp_alg(tfm);
+
+ if (alg && alg->batch_size > 1)
+ return alg->batch_size;
+ }
+
+ return 1;
+}
+EXPORT_SYMBOL_GPL(crypto_acomp_batch_size);
+
void comp_prepare_alg(struct comp_alg_common *alg)
{
struct crypto_alg *base = &alg->base;
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index fc0665b77a8f..06578c4367f6 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -593,6 +593,18 @@ int crypto_acomp_compress(struct acomp_req *req);
*/
int crypto_acomp_decompress(struct acomp_req *req);
+/**
+ * crypto_acomp_batch_size() -- Get the algorithm's batch size
+ *
+ * Function returns the algorithm's batch size for batching operations
+ *
+ * @tfm: ACOMPRESS tfm handle allocated with crypto_alloc_acomp()
+ *
+ * Return: @tfm's acomp_alg's @batch_size, if it has defined a
+ * @batch_size greater than 1; else return 1.
+ */
+unsigned int crypto_acomp_batch_size(struct crypto_acomp *tfm);
+
static inline struct acomp_req *acomp_request_on_stack_init(
char *buf, struct crypto_acomp *tfm)
{
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 93ae59623f13..b790722c263a 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -28,6 +28,8 @@
*
* @compress: Function performs a compress operation
* @decompress: Function performs a de-compress operation
+ * @batch_size: Maximum batch-size for batching compress/decompress
+ * operations.
* @init: Initialize the cryptographic transformation object.
* This function is used to initialize the cryptographic
* transformation object. This function is called only once at
@@ -47,6 +49,7 @@
struct acomp_alg {
int (*compress)(struct acomp_req *req);
int (*decompress)(struct acomp_req *req);
+ unsigned int batch_size;
int (*init)(struct crypto_acomp *tfm);
void (*exit)(struct crypto_acomp *tfm);
--
2.39.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v1 5/8] mm: zswap: Store the folio's node id in the zswap_entry.
2026-10-08 18:28 [PATCH v1 0/8] Batched zswap_store() with compression batching Kanchana P. Sridhar
` (3 preceding siblings ...)
2026-10-08 18:28 ` [PATCH v1 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
@ 2026-10-08 18:28 ` Kanchana P. Sridhar
2026-10-14 0:18 ` Yosry Ahmed
2026-10-08 18:28 ` [PATCH v1 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
` (2 subsequent siblings)
7 siblings, 1 reply; 11+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-08 18:28 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
When storing a folio in zswap, save the folio's node id in the zswap
entry instead of relying on the zswap_entry being allocated on the same
node. This enables optimizations such as batch allocation of zswap
entries for large folios using kmem_cache_alloc_bulk() that does not
support specifying a node id.
The size of the zswap_entry remains unchanged as 'referenced' is lumped
in with the 'length' (as it doesn't need a full unsigned int anyway).
Suggested-by: Nhat Pham <nphamcs@gmail.com>
Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 23 +++++++++++------------
1 file changed, 11 insertions(+), 12 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index ca513a9f89e3..c2731d3acb2c 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -194,6 +194,7 @@ static struct shrinker *zswap_shrinker;
* writeback logic. The entry is only reclaimed by the writeback
* logic if referenced is unset. See comments in the shrinker
* section for context.
+ * nid - NUMA node id of the page for which this is the zswap entry.
* pool_idx - id of the zswap_pool that the entry's data is in.
* handle - zsmalloc allocation handle that stores the compressed page data
* objcg - the obj_cgroup that the compressed memory is charged to
@@ -201,8 +202,11 @@ static struct shrinker *zswap_shrinker;
*/
struct zswap_entry {
swp_entry_t swpentry;
- unsigned int length;
- bool referenced;
+ struct {
+ unsigned int length:31;
+ bool referenced:1;
+ };
+ int nid;
u8 pool_idx;
unsigned long handle;
struct obj_cgroup *objcg;
@@ -672,14 +676,8 @@ static int zswap_enabled_param_set(const char *val,
* lru functions
**********************************/
-static inline int entry_to_nid(struct zswap_entry *entry)
-{
- return page_to_nid(virt_to_page(entry));
-}
-
static void zswap_lru_add(struct zswap_entry *entry)
{
- int nid = entry_to_nid(entry);
struct mem_cgroup *memcg;
/*
@@ -696,19 +694,18 @@ static void zswap_lru_add(struct zswap_entry *entry)
rcu_read_lock();
memcg = obj_cgroup_memcg(entry->objcg);
/* will always succeed */
- list_lru_add(&zswap_list_lru, &entry->lru, nid, memcg);
+ list_lru_add(&zswap_list_lru, &entry->lru, entry->nid, memcg);
rcu_read_unlock();
}
static void zswap_lru_del(struct zswap_entry *entry)
{
- int nid = entry_to_nid(entry);
struct mem_cgroup *memcg;
rcu_read_lock();
memcg = obj_cgroup_memcg(entry->objcg);
/* will always succeed */
- list_lru_del(&zswap_list_lru, &entry->lru, nid, memcg);
+ list_lru_del(&zswap_list_lru, &entry->lru, entry->nid, memcg);
rcu_read_unlock();
}
@@ -1449,9 +1446,10 @@ static bool zswap_store_page(struct folio *folio, long index,
{
swp_entry_t page_swpentry = folio_swap_entry(folio, index);
struct zswap_entry *entry, *old;
+ int nid = folio_nid(folio);
/* allocate entry */
- entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio));
+ entry = zswap_entry_cache_alloc(GFP_KERNEL, nid);
if (!entry) {
zswap_reject_kmemcache_fail++;
return false;
@@ -1466,6 +1464,7 @@ static bool zswap_store_page(struct folio *folio, long index,
* to an unrelated live pool.
*/
entry->pool_idx = pool->idx;
+ entry->nid = nid;
old = xa_store(swap_zswap_tree(page_swpentry),
swp_offset(page_swpentry),
--
2.39.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* Re: [PATCH v1 5/8] mm: zswap: Store the folio's node id in the zswap_entry.
2026-10-08 18:28 ` [PATCH v1 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
@ 2026-10-14 0:18 ` Yosry Ahmed
2026-10-14 1:17 ` Kanchana P. Sridhar
0 siblings, 1 reply; 11+ messages in thread
From: Yosry Ahmed @ 2026-10-14 0:18 UTC (permalink / raw)
To: Kanchana P. Sridhar
Cc: hannes, nphamcs, chengming.zhou, akpm, linux-mm, herbert, davem,
linux-crypto, linux-kernel, rientjes, willy, senozhatsky
On Thu, Oct 08, 2026 at 11:28:11AM -0700, Kanchana P. Sridhar wrote:
> When storing a folio in zswap, save the folio's node id in the zswap
> entry instead of relying on the zswap_entry being allocated on the same
> node. This enables optimizations such as batch allocation of zswap
> entries for large folios using kmem_cache_alloc_bulk() that does not
> support specifying a node id.
>
> The size of the zswap_entry remains unchanged as 'referenced' is lumped
> in with the 'length' (as it doesn't need a full unsigned int anyway).
>
> Suggested-by: Nhat Pham <nphamcs@gmail.com>
> Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
> Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
> Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
> ---
> mm/zswap.c | 23 +++++++++++------------
> 1 file changed, 11 insertions(+), 12 deletions(-)
>
> diff --git a/mm/zswap.c b/mm/zswap.c
> index ca513a9f89e3..c2731d3acb2c 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
> @@ -194,6 +194,7 @@ static struct shrinker *zswap_shrinker;
> * writeback logic. The entry is only reclaimed by the writeback
> * logic if referenced is unset. See comments in the shrinker
> * section for context.
> + * nid - NUMA node id of the page for which this is the zswap entry.
> * pool_idx - id of the zswap_pool that the entry's data is in.
> * handle - zsmalloc allocation handle that stores the compressed page data
> * objcg - the obj_cgroup that the compressed memory is charged to
> @@ -201,8 +202,11 @@ static struct shrinker *zswap_shrinker;
> */
> struct zswap_entry {
> swp_entry_t swpentry;
> - unsigned int length;
> - bool referenced;
> + struct {
> + unsigned int length:31;
> + bool referenced:1;
> + };
> + int nid;
This will increase the size of the struct now with pool_idx, right?
Before you patch on x86_64:
struct zswap_entry {
swp_entry_t swpentry; /* 0 8 */
unsigned int length; /* 8 4 */
bool referenced; /* 12 1 */
u8 pool_idx; /* 13 1 */
/* XXX 2 bytes hole, try to pack */
unsigned long handle; /* 16 8 */
struct obj_cgroup * objcg; /* 24 8 */
struct list_head lru; /* 32 16 */
/* size: 48, cachelines: 1, members: 7 */
/* sum members: 46, holes: 1, sum holes: 2 */
/* last cacheline: 48 bytes */
};
After:
struct zswap_entry {
swp_entry_t swpentry; /* 0 8 */
struct {
unsigned int length:31; /* 8: 0 4 */
/* Bitfield combined with next fields */
bool referenced:1; /* 11: 7 1 */
}; /* 8 4 */
struct {
unsigned int length:31; /* 0: 0 4 */
/* Bitfield combined with next fields */
bool referenced:1; /* 3: 7 1 */
/* size: 4, cachelines: 1, members: 2 */
/* last cacheline: 4 bytes */
};
int nid; /* 12 4 */
u8 pool_idx; /* 16 1 */
/* XXX 7 bytes hole, try to pack */
unsigned long handle; /* 24 8 */
struct obj_cgroup * objcg; /* 32 8 */
struct list_head lru; /* 40 16 */
/* size: 56, cachelines: 1, members: 7 */
/* sum members: 49, holes: 1, sum holes: 7 */
/* last cacheline: 56 bytes */
};
I think we want to reorder the fields. We can probably also avoid the
bitfield if the maximum page size is 64K (is it?)
struct zswap_entry {
swp_entry_t swpentry; /* 0 8 */
u16 length; /* 8 2 */
u8 pool_idx; /* 10 1 */
bool referenced; /* 11 1 */
int nid; /* 12 4 */
unsigned long handle; /* 16 8 */
struct obj_cgroup * objcg; /* 24 8 */
struct list_head lru; /* 32 16 */
/* size: 48, cachelines: 1, members: 8 */
/* last cacheline: 48 bytes */
};
> u8 pool_idx;
> unsigned long handle;
> struct obj_cgroup *objcg;
> @@ -672,14 +676,8 @@ static int zswap_enabled_param_set(const char *val,
> * lru functions
> **********************************/
>
> -static inline int entry_to_nid(struct zswap_entry *entry)
> -{
> - return page_to_nid(virt_to_page(entry));
> -}
> -
> static void zswap_lru_add(struct zswap_entry *entry)
> {
> - int nid = entry_to_nid(entry);
> struct mem_cgroup *memcg;
>
> /*
> @@ -696,19 +694,18 @@ static void zswap_lru_add(struct zswap_entry *entry)
> rcu_read_lock();
> memcg = obj_cgroup_memcg(entry->objcg);
> /* will always succeed */
> - list_lru_add(&zswap_list_lru, &entry->lru, nid, memcg);
> + list_lru_add(&zswap_list_lru, &entry->lru, entry->nid, memcg);
> rcu_read_unlock();
> }
>
> static void zswap_lru_del(struct zswap_entry *entry)
> {
> - int nid = entry_to_nid(entry);
> struct mem_cgroup *memcg;
>
> rcu_read_lock();
> memcg = obj_cgroup_memcg(entry->objcg);
> /* will always succeed */
> - list_lru_del(&zswap_list_lru, &entry->lru, nid, memcg);
> + list_lru_del(&zswap_list_lru, &entry->lru, entry->nid, memcg);
> rcu_read_unlock();
> }
>
> @@ -1449,9 +1446,10 @@ static bool zswap_store_page(struct folio *folio, long index,
> {
> swp_entry_t page_swpentry = folio_swap_entry(folio, index);
> struct zswap_entry *entry, *old;
> + int nid = folio_nid(folio);
>
> /* allocate entry */
> - entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio));
> + entry = zswap_entry_cache_alloc(GFP_KERNEL, nid);
We don't need to allocate the entry on the same node anymore, should we
drop this and let the slab allocator handle it as it sees fit?
> if (!entry) {
> zswap_reject_kmemcache_fail++;
> return false;
> @@ -1466,6 +1464,7 @@ static bool zswap_store_page(struct folio *folio, long index,
> * to an unrelated live pool.
> */
> entry->pool_idx = pool->idx;
> + entry->nid = nid;
>
> old = xa_store(swap_zswap_tree(page_swpentry),
> swp_offset(page_swpentry),
> --
> 2.39.5
>
^ permalink raw reply [flat|nested] 11+ messages in thread* Re: [PATCH v1 5/8] mm: zswap: Store the folio's node id in the zswap_entry.
2026-10-14 0:18 ` Yosry Ahmed
@ 2026-10-14 1:17 ` Kanchana P. Sridhar
0 siblings, 0 replies; 11+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-14 1:17 UTC (permalink / raw)
To: Yosry Ahmed
Cc: hannes, nphamcs, chengming.zhou, akpm, linux-mm, herbert, davem,
linux-crypto, linux-kernel, rientjes, willy, senozhatsky,
Kanchana P. Sridhar
On Tue, Oct 13, 2026 at 5:18 PM Yosry Ahmed <yosry@kernel.org> wrote:
>
> On Thu, Oct 08, 2026 at 11:28:11AM -0700, Kanchana P. Sridhar wrote:
> > When storing a folio in zswap, save the folio's node id in the zswap
> > entry instead of relying on the zswap_entry being allocated on the same
> > node. This enables optimizations such as batch allocation of zswap
> > entries for large folios using kmem_cache_alloc_bulk() that does not
> > support specifying a node id.
> >
> > The size of the zswap_entry remains unchanged as 'referenced' is lumped
> > in with the 'length' (as it doesn't need a full unsigned int anyway).
> >
> > Suggested-by: Nhat Pham <nphamcs@gmail.com>
> > Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
> > Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
> > Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
> > ---
> > mm/zswap.c | 23 +++++++++++------------
> > 1 file changed, 11 insertions(+), 12 deletions(-)
> >
> > diff --git a/mm/zswap.c b/mm/zswap.c
> > index ca513a9f89e3..c2731d3acb2c 100644
> > --- a/mm/zswap.c
> > +++ b/mm/zswap.c
> > @@ -194,6 +194,7 @@ static struct shrinker *zswap_shrinker;
> > * writeback logic. The entry is only reclaimed by the writeback
> > * logic if referenced is unset. See comments in the shrinker
> > * section for context.
> > + * nid - NUMA node id of the page for which this is the zswap entry.
> > * pool_idx - id of the zswap_pool that the entry's data is in.
> > * handle - zsmalloc allocation handle that stores the compressed page data
> > * objcg - the obj_cgroup that the compressed memory is charged to
> > @@ -201,8 +202,11 @@ static struct shrinker *zswap_shrinker;
> > */
> > struct zswap_entry {
> > swp_entry_t swpentry;
> > - unsigned int length;
> > - bool referenced;
> > + struct {
> > + unsigned int length:31;
> > + bool referenced:1;
> > + };
> > + int nid;
>
> This will increase the size of the struct now with pool_idx, right?
I was gathering the struct size with the struct {unsigned int
length:31; bool referenced:1} as suggested in v14, for my before/after
comparisons:
before:
struct zswap_entry {
swp_entry_t swpentry; /* 0 8 */
struct {
unsigned int length:31; /* 8: 0 4 */
/* Bitfield combined with next fields */
bool referenced:1; /* 11: 7 1 */
}; /* 8 4 */
int nid; /* 12 4 */
u8 pool_idx; /* 16 1 */
/* XXX 7 bytes hole, try to pack */
long unsigned int handle; /* 24 8 */
struct obj_cgroup * objcg; /* 32 8 */
struct list_head lru; /* 40 16 */
/* size: 56, cachelines: 1, members: 7 */
/* sum members: 49, holes: 1, sum holes: 7 */
/* last cacheline: 56 bytes */
};
after:
struct zswap_entry {
swp_entry_t swpentry; /* 0 8 */
struct {
unsigned int length:31; /* 8: 0 4 */
/* Bitfield combined with next fields */
bool referenced:1; /* 11: 7 1 */
}; /* 8 4 */
int nid; /* 12 4 */
long unsigned int handle; /* 16 8 */
struct obj_cgroup * objcg; /* 24 8 */
struct list_head lru; /* 32 16 */
u8 pool_idx; /* 48 1 */
/* size: 56, cachelines: 1, members: 7 */
/* padding: 7 */
/* last cacheline: 56 bytes */
};
And decided to go with the placement in patch 7, since it also
separates out members by compress/decompress path accesses. pool_idx
is set once, and never changes. Further, moving pool_idx to the end
gets rid of holes.
>
> Before you patch on x86_64:
>
> struct zswap_entry {
> swp_entry_t swpentry; /* 0 8 */
> unsigned int length; /* 8 4 */
> bool referenced; /* 12 1 */
> u8 pool_idx; /* 13 1 */
>
> /* XXX 2 bytes hole, try to pack */
>
> unsigned long handle; /* 16 8 */
> struct obj_cgroup * objcg; /* 24 8 */
> struct list_head lru; /* 32 16 */
>
> /* size: 48, cachelines: 1, members: 7 */
> /* sum members: 46, holes: 1, sum holes: 2 */
> /* last cacheline: 48 bytes */
> };
>
> After:
>
> struct zswap_entry {
> swp_entry_t swpentry; /* 0 8 */
> struct {
> unsigned int length:31; /* 8: 0 4 */
>
> /* Bitfield combined with next fields */
>
> bool referenced:1; /* 11: 7 1 */
> }; /* 8 4 */
> struct {
> unsigned int length:31; /* 0: 0 4 */
>
> /* Bitfield combined with next fields */
>
> bool referenced:1; /* 3: 7 1 */
>
> /* size: 4, cachelines: 1, members: 2 */
> /* last cacheline: 4 bytes */
> };
>
> int nid; /* 12 4 */
> u8 pool_idx; /* 16 1 */
>
> /* XXX 7 bytes hole, try to pack */
>
> unsigned long handle; /* 24 8 */
> struct obj_cgroup * objcg; /* 32 8 */
> struct list_head lru; /* 40 16 */
>
> /* size: 56, cachelines: 1, members: 7 */
> /* sum members: 49, holes: 1, sum holes: 7 */
> /* last cacheline: 56 bytes */
> };
>
> I think we want to reorder the fields. We can probably also avoid the
> bitfield if the maximum page size is 64K (is it?)
It seems on ARM64, the max page size is 64K, however on PPC, it can be
256K. So it appears we need at least 18 bits on currently supported
architectures.
If you agree with the pool_idx placement to be moved to the end as
proposed in patch 7, I can try and optimize this further, by limiting
the "unsigned int length:18", if you think this makes sense.
>
> struct zswap_entry {
> swp_entry_t swpentry; /* 0 8 */
> u16 length; /* 8 2 */
> u8 pool_idx; /* 10 1 */
> bool referenced; /* 11 1 */
> int nid; /* 12 4 */
> unsigned long handle; /* 16 8 */
> struct obj_cgroup * objcg; /* 24 8 */
> struct list_head lru; /* 32 16 */
>
> /* size: 48, cachelines: 1, members: 8 */
> /* last cacheline: 48 bytes */
> };
>
>
> > u8 pool_idx;
> > unsigned long handle;
> > struct obj_cgroup *objcg;
> > @@ -672,14 +676,8 @@ static int zswap_enabled_param_set(const char *val,
> > * lru functions
> > **********************************/
> >
> > -static inline int entry_to_nid(struct zswap_entry *entry)
> > -{
> > - return page_to_nid(virt_to_page(entry));
> > -}
> > -
> > static void zswap_lru_add(struct zswap_entry *entry)
> > {
> > - int nid = entry_to_nid(entry);
> > struct mem_cgroup *memcg;
> >
> > /*
> > @@ -696,19 +694,18 @@ static void zswap_lru_add(struct zswap_entry *entry)
> > rcu_read_lock();
> > memcg = obj_cgroup_memcg(entry->objcg);
> > /* will always succeed */
> > - list_lru_add(&zswap_list_lru, &entry->lru, nid, memcg);
> > + list_lru_add(&zswap_list_lru, &entry->lru, entry->nid, memcg);
> > rcu_read_unlock();
> > }
> >
> > static void zswap_lru_del(struct zswap_entry *entry)
> > {
> > - int nid = entry_to_nid(entry);
> > struct mem_cgroup *memcg;
> >
> > rcu_read_lock();
> > memcg = obj_cgroup_memcg(entry->objcg);
> > /* will always succeed */
> > - list_lru_del(&zswap_list_lru, &entry->lru, nid, memcg);
> > + list_lru_del(&zswap_list_lru, &entry->lru, entry->nid, memcg);
> > rcu_read_unlock();
> > }
> >
> > @@ -1449,9 +1446,10 @@ static bool zswap_store_page(struct folio *folio, long index,
> > {
> > swp_entry_t page_swpentry = folio_swap_entry(folio, index);
> > struct zswap_entry *entry, *old;
> > + int nid = folio_nid(folio);
> >
> > /* allocate entry */
> > - entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio));
> > + entry = zswap_entry_cache_alloc(GFP_KERNEL, nid);
>
> We don't need to allocate the entry on the same node anymore, should we
> drop this and let the slab allocator handle it as it sees fit?
Was this a recent change? I see the folio_nid(folio) on my version of
mm-unstable as of 10-3-2026. Let me check, and yes, allowing the slab
allocated handle this is probably a good idea.
Along the same lines, I have been working on replacing zsmalloc with
direct SLAB allocations for the zswap compressed data storage and see
very promising results: memory savings and latency improvements. I
will submit an RFC shortly.
>
> > if (!entry) {
> > zswap_reject_kmemcache_fail++;
> > return false;
> > @@ -1466,6 +1464,7 @@ static bool zswap_store_page(struct folio *folio, long index,
> > * to an unrelated live pool.
> > */
> > entry->pool_idx = pool->idx;
> > + entry->nid = nid;
> >
> > old = xa_store(swap_zswap_tree(page_swpentry),
> > swp_offset(page_swpentry),
> > --
> > 2.39.5
> >
^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH v1 6/8] mm: zswap: Store large folios in batches.
2026-10-08 18:28 [PATCH v1 0/8] Batched zswap_store() with compression batching Kanchana P. Sridhar
` (4 preceding siblings ...)
2026-10-08 18:28 ` [PATCH v1 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
@ 2026-10-08 18:28 ` Kanchana P. Sridhar
2026-10-08 18:28 ` [PATCH v1 7/8] mm: zswap: Optimally place members of struct zswap_pool, struct zswap_entry Kanchana P. Sridhar
2026-10-08 18:28 ` [PATCH v1 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches Kanchana P. Sridhar
7 siblings, 0 replies; 11+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-08 18:28 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
Use batching when storing large folios in zswap. If the underlying
compressor supports batching (e.g. hardware parallel compression),
allocate multiple compression buffers, otherwise allocate one. The
number of buffers is bounded by a new constant, ZSWAP_MAX_BATCH_SIZE, to
limit the memory overhead. For existing software compressors, the only
extra overhead is the extra 'buffers' pointer, so 8 bytes per-CPU on
x86_64. Only the first buffer is currently used.
Regardless of compression batching, always process large folios in
batches. For hardware compressors, the batch size is the compressor
batch size, otherwise ZSWAP_MAX_BATCH_SIZE is used.
zswap_store_page() is replaced with zswap_store_pages(), which processes
a batch of pages and allows for batching optimizations. For now, only
optimize allocating entries by using batch allocations from the slab
cache.
Avoid repeatedly calling mem_cgroup_zswap_writeback_enabled() for every
page and only call it once for the folio, since the entire folio is
charged to a single memcg. Similarly, obtain the nid once to use during
each batch store of a folio.
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 304 +++++++++++++++++++++++++++++++++++++----------------
1 file changed, 212 insertions(+), 92 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index c2731d3acb2c..8ae67fd10522 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -84,6 +84,9 @@ static bool zswap_pool_reached_full;
#define ZSWAP_PARAM_UNSET ""
+/* Limit the batch size to limit per-CPU memory usage for dst buffers. */
+#define ZSWAP_MAX_BATCH_SIZE 8U
+
static int zswap_setup(void);
/* Enable/disable zswap */
@@ -141,7 +144,7 @@ struct crypto_acomp_ctx {
struct crypto_acomp *acomp;
struct acomp_req *req;
struct crypto_wait wait;
- u8 *buffer;
+ u8 **buffers;
struct mutex mutex;
};
@@ -150,10 +153,14 @@ struct crypto_acomp_ctx {
* The only case where lru_lock is not acquired while holding tree.lock is
* when a zswap_entry is taken off the lru for writeback, in that case it
* needs to be verified that it's still valid in the tree.
+ *
+ * @compr_batch_size: The max batch size of the compression algorithm,
+ * bounded by ZSWAP_MAX_BATCH_SIZE.
*/
struct zswap_pool {
struct zs_pool *zs_pool;
struct crypto_acomp_ctx __percpu *acomp_ctx;
+ u8 compr_batch_size;
struct percpu_ref ref;
struct rcu_work release_rwork;
struct hlist_node node;
@@ -269,8 +276,10 @@ static inline struct xarray *swap_zswap_tree(swp_entry_t swp)
**********************************/
static void __zswap_pool_empty(struct percpu_ref *ref);
-static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
+static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx, u8 nr_buffers)
{
+ u8 i;
+
if (!acomp_ctx)
return;
@@ -293,8 +302,12 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
acomp_ctx->acomp = NULL;
- kfree(acomp_ctx->buffer);
- acomp_ctx->buffer = NULL;
+ if (acomp_ctx->buffers) {
+ for (i = 0; i < nr_buffers; ++i)
+ kfree(acomp_ctx->buffers[i]);
+ kfree(acomp_ctx->buffers);
+ }
+ acomp_ctx->buffers = NULL;
}
static struct zswap_pool *zswap_pool_create(char *compressor)
@@ -378,7 +391,9 @@ static struct zswap_pool *zswap_pool_create(char *compressor)
cpuhp_add_fail:
for_each_possible_cpu(cpu)
- acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu));
+ acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu),
+ pool->compr_batch_size);
+
error:
if (pool->acomp_ctx)
free_percpu(pool->acomp_ctx);
@@ -416,7 +431,8 @@ static void zswap_pool_destroy(struct zswap_pool *pool)
cpuhp_state_remove_instance(CPUHP_MM_ZSWP_POOL_PREPARE, &pool->node);
for_each_possible_cpu(cpu)
- acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu));
+ acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu),
+ pool->compr_batch_size);
free_percpu(pool->acomp_ctx);
@@ -767,6 +783,41 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)
kmem_cache_free(zswap_entry_cache, entry);
}
+static __always_inline void zswap_entries_cache_free_batch(
+ struct zswap_entry **entries,
+ u8 nr_entries)
+{
+ /*
+ * It is okay to use this to free entries allocated separately
+ * by zswap_entry_cache_alloc().
+ */
+ kmem_cache_free_bulk(zswap_entry_cache, nr_entries, (void **)entries);
+}
+
+static bool zswap_entries_cache_alloc_batch(
+ struct zswap_entry **entries,
+ u8 nr_entries,
+ gfp_t gfp,
+ int nid)
+{
+ if (unlikely(!kmem_cache_alloc_bulk(zswap_entry_cache, gfp,
+ nr_entries, (void **)entries))) {
+ u8 i;
+
+ for (i = 0; i < nr_entries; ++i) {
+ entries[i] = zswap_entry_cache_alloc(GFP_KERNEL, nid);
+
+ if (unlikely(!entries[i])) {
+ zswap_reject_kmemcache_fail++;
+ zswap_entries_cache_free_batch(entries, i);
+ return false;
+ }
+ }
+ }
+
+ return true;
+}
+
/*
* Carries out the common pattern of freeing an entry's zsmalloc allocation,
* freeing the entry itself, and decrementing the number of stored pages.
@@ -797,7 +848,9 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
{
struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node);
struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu);
+ int nid = cpu_to_node(cpu);
int ret = -ENOMEM;
+ u8 i;
/*
* To handle cases where the CPU goes through online-offline-online
@@ -808,15 +861,11 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
return 0;
}
- acomp_ctx->buffer = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu));
- if (!acomp_ctx->buffer)
- return ret;
-
/*
* In case of an error, crypto_alloc_acomp_node() returns an
* error pointer, never NULL.
*/
- acomp_ctx->acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu));
+ acomp_ctx->acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, nid);
if (IS_ERR(acomp_ctx->acomp)) {
pr_err("could not alloc crypto acomp %s : %pe\n",
pool->tfm_name, acomp_ctx->acomp);
@@ -824,6 +873,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
goto fail;
}
+ /*
+ * Allocate up to ZSWAP_MAX_BATCH_SIZE dst buffers if the
+ * compressor supports batching.
+ */
+ pool->compr_batch_size = min(ZSWAP_MAX_BATCH_SIZE,
+ crypto_acomp_batch_size(acomp_ctx->acomp));
+
/* acomp_request_alloc() returns NULL in case of an error. */
acomp_ctx->req = acomp_request_alloc(acomp_ctx->acomp);
if (!acomp_ctx->req) {
@@ -832,6 +888,17 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
goto fail;
}
+ acomp_ctx->buffers = kcalloc_node(pool->compr_batch_size, sizeof(u8 *),
+ GFP_KERNEL, nid);
+ if (!acomp_ctx->buffers)
+ goto fail;
+
+ for (i = 0; i < pool->compr_batch_size; ++i) {
+ acomp_ctx->buffers[i] = kmalloc_node(PAGE_SIZE, GFP_KERNEL, nid);
+ if (!acomp_ctx->buffers[i])
+ goto fail;
+ }
+
crypto_init_wait(&acomp_ctx->wait);
/*
@@ -852,12 +919,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
return 0;
fail:
- acomp_ctx_free(acomp_ctx);
+ acomp_ctx_free(acomp_ctx, pool->compr_batch_size);
return ret;
}
static bool zswap_compress(struct folio *folio, long index,
- struct zswap_entry *entry, struct zswap_pool *pool)
+ struct zswap_entry *entry, struct zswap_pool *pool,
+ bool wb_enabled)
{
struct crypto_acomp_ctx *acomp_ctx;
struct scatterlist input, output;
@@ -871,7 +939,7 @@ static bool zswap_compress(struct folio *folio, long index,
acomp_ctx = raw_cpu_ptr(pool->acomp_ctx);
mutex_lock(&acomp_ctx->mutex);
- dst = acomp_ctx->buffer;
+ dst = acomp_ctx->buffers[0];
sg_init_table(&input, 1);
sg_set_folio(&input, folio, PAGE_SIZE, index * PAGE_SIZE);
@@ -901,13 +969,10 @@ static bool zswap_compress(struct folio *folio, long index,
* to the active LRU list in the case.
*/
if (comp_ret || !dlen || dlen >= PAGE_SIZE) {
- rcu_read_lock();
- if (!mem_cgroup_zswap_writeback_enabled(folio_memcg(folio))) {
- rcu_read_unlock();
+ if (!wb_enabled) {
comp_ret = comp_ret ? comp_ret : -EINVAL;
goto unlock;
}
- rcu_read_unlock();
comp_ret = 0;
dlen = PAGE_SIZE;
dst = kmap_local_folio(folio, index * PAGE_SIZE);
@@ -1440,91 +1505,122 @@ static void shrink_worker(struct work_struct *w)
* main API
**********************************/
-static bool zswap_store_page(struct folio *folio, long index,
- struct obj_cgroup *objcg,
- struct zswap_pool *pool)
+/*
+ * Store multiple pages in @folio, starting from the page at index @start up to
+ * the page at index @end-1.
+ */
+static bool zswap_store_pages(struct folio *folio,
+ long start,
+ long end,
+ struct zswap_pool *pool,
+ int nid,
+ bool wb_enabled,
+ struct obj_cgroup *objcg)
{
- swp_entry_t page_swpentry = folio_swap_entry(folio, index);
- struct zswap_entry *entry, *old;
- int nid = folio_nid(folio);
+ struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE];
+ u8 i, store_fail_idx = 0, nr_pages = end - start;
- /* allocate entry */
- entry = zswap_entry_cache_alloc(GFP_KERNEL, nid);
- if (!entry) {
- zswap_reject_kmemcache_fail++;
- return false;
- }
+ VM_WARN_ON_ONCE(nr_pages <= 0 || nr_pages > ZSWAP_MAX_BATCH_SIZE);
- if (!zswap_compress(folio, index, entry, pool))
- goto compress_failed;
+ if (unlikely(!zswap_entries_cache_alloc_batch(entries, nr_pages,
+ GFP_KERNEL, nid)))
+ return false;
/*
- * Set pool_idx before the xa_store() below publishes the entry, or a
- * concurrent reader could resolve a stale pool_idx left by slab reuse
- * to an unrelated live pool.
+ * We co-locate entry initialization as much as possible here to
+ * minimize potential cache misses.
*/
- entry->pool_idx = pool->idx;
- entry->nid = nid;
-
- old = xa_store(swap_zswap_tree(page_swpentry),
- swp_offset(page_swpentry),
- entry, GFP_KERNEL);
- if (xa_is_err(old)) {
- int err = xa_err(old);
+ for (i = 0; i < nr_pages; ++i) {
+ entries[i]->handle = (unsigned long)ERR_PTR(-EINVAL);
+ /*
+ * Set pool_idx before the xa_store() below publishes the entry, or a
+ * concurrent reader could resolve a stale pool_idx left by slab reuse
+ * to an unrelated live pool.
+ */
+ entries[i]->pool_idx = pool->idx;
+ entries[i]->swpentry = folio_swap_entry(folio, start + i);
+ entries[i]->objcg = objcg;
+ entries[i]->referenced = true;
+ entries[i]->nid = nid;
+ INIT_LIST_HEAD(&entries[i]->lru);
+ }
- WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err);
- zswap_reject_alloc_fail++;
- goto store_failed;
+ for (i = 0; i < nr_pages; ++i) {
+ if (!zswap_compress(folio, start + i, entries[i], pool, wb_enabled))
+ goto store_pages_failed;
}
- /*
- * We may have had an existing entry that became stale when
- * the folio was redirtied and now the new version is being
- * swapped out. Get rid of the old.
- */
- if (old)
- zswap_entry_free(old);
+ for (i = 0; i < nr_pages; ++i) {
+ struct zswap_entry *old, *entry = entries[i];
+
+ old = xa_store(swap_zswap_tree(entry->swpentry),
+ swp_offset(entry->swpentry),
+ entry, GFP_KERNEL);
+ if (unlikely(xa_is_err(old))) {
+ int err = xa_err(old);
+
+ WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err);
+ zswap_reject_alloc_fail++;
+ /*
+ * Entries up to this point have been stored in the
+ * xarray. zswap_store() will erase them from the xarray
+ * and call zswap_entry_free(). Local cleanup in
+ * 'store_pages_failed' only needs to happen for
+ * entries from [@i to @nr_pages).
+ */
+ store_fail_idx = i;
+ goto store_pages_failed;
+ }
- /*
- * The entry is successfully compressed and stored in the tree, there is
- * no further possibility of failure. Grab refs to the pool and objcg,
- * charge zswap memory, and increment zswap_stored_pages.
- * The opposite actions will be performed by zswap_entry_free()
- * when the entry is removed from the tree.
- */
- zswap_pool_get(pool);
- if (objcg) {
- obj_cgroup_get(objcg);
- obj_cgroup_charge_zswap(objcg, entry->length);
- }
- atomic_long_inc(&zswap_stored_pages);
- if (entry->length == PAGE_SIZE)
- atomic_long_inc(&zswap_stored_incompressible_pages);
+ /*
+ * We may have had an existing entry that became stale when
+ * the folio was redirtied and now the new version is being
+ * swapped out. Get rid of the old.
+ */
+ if (unlikely(old))
+ zswap_entry_free(old);
- /*
- * We finish initializing the entry while it's already in xarray.
- * This is safe because:
- *
- * 1. Concurrent stores and invalidations are excluded by folio lock.
- *
- * 2. Writeback is excluded by the entry not being on the LRU yet.
- * The publishing order matters to prevent writeback from seeing
- * an incoherent entry.
- */
- entry->swpentry = page_swpentry;
- entry->objcg = objcg;
- entry->referenced = true;
- if (entry->length) {
- INIT_LIST_HEAD(&entry->lru);
- zswap_lru_add(entry);
+ /*
+ * The entry is successfully compressed and stored in the tree,
+ * and further failures will be cleaned up in zswap_store().
+ * Grab refs to the pool and objcg, charge zswap memory, and
+ * increment zswap_stored_pages. The opposite actions will be
+ * performed by zswap_entry_free() when the entry is removed
+ * from the tree.
+ */
+ zswap_pool_get(pool);
+ if (objcg) {
+ obj_cgroup_get(objcg);
+ obj_cgroup_charge_zswap(objcg, entry->length);
+ }
+ atomic_long_inc(&zswap_stored_pages);
+ if (entry->length == PAGE_SIZE)
+ atomic_long_inc(&zswap_stored_incompressible_pages);
+
+ /*
+ * We finish by adding the entry to the LRU while it's already
+ * in xarray. This is safe because:
+ *
+ * 1. Concurrent stores and invalidations are excluded by folio lock.
+ *
+ * 2. Writeback is excluded by the entry not being on the LRU yet.
+ * The publishing order matters to prevent writeback from seeing
+ * an incoherent entry.
+ */
+ if (likely(entry->length))
+ zswap_lru_add(entry);
}
return true;
-store_failed:
- zs_free(pool->zs_pool, entry->handle);
-compress_failed:
- zswap_entry_cache_free(entry);
+store_pages_failed:
+ for (i = store_fail_idx; i < nr_pages; ++i) {
+ if (!IS_ERR_VALUE(entries[i]->handle))
+ zs_free(pool->zs_pool, entries[i]->handle);
+ }
+ zswap_entries_cache_free_batch(&entries[store_fail_idx],
+ nr_pages - store_fail_idx);
+
return false;
}
@@ -1534,9 +1630,11 @@ bool zswap_store(struct folio *folio)
swp_entry_t swp = folio->swap;
struct obj_cgroup *objcg = NULL;
struct mem_cgroup *memcg = NULL;
+ bool wb_enabled, ret = false;
+ int nid = folio_nid(folio);
struct zswap_pool *pool;
- bool ret = false;
- long index;
+ u8 store_batch_size;
+ long start, end;
VM_WARN_ON_ONCE(!folio_test_locked(folio));
VM_WARN_ON_ONCE(!folio_test_swapcache(folio));
@@ -1570,8 +1668,30 @@ bool zswap_store(struct folio *folio)
mem_cgroup_put(memcg);
}
- for (index = 0; index < nr_pages; ++index) {
- if (!zswap_store_page(folio, index, objcg, pool))
+ rcu_read_lock();
+ wb_enabled = mem_cgroup_zswap_writeback_enabled(folio_memcg(folio));
+ rcu_read_unlock();
+
+ /*
+ * For batching compressors, store the folio in batches of the
+ * compressor's batch_size.
+ *
+ * For non-batching compressors, store the folio in batches
+ * of ZSWAP_MAX_BATCH_SIZE, where each page in the batch is
+ * compressed sequentially. This gives better performance than
+ * invoking zswap_store_pages() per-page, due to cache locality
+ * of working set structures.
+ */
+ store_batch_size = (pool->compr_batch_size > 1) ?
+ pool->compr_batch_size : ZSWAP_MAX_BATCH_SIZE;
+
+ for (start = 0; start < nr_pages; start += store_batch_size) {
+ end = min(start + store_batch_size, nr_pages);
+
+ ret = zswap_store_pages(folio, start, end, pool,
+ nid, wb_enabled, objcg);
+
+ if (unlikely(!ret))
goto put_pool;
}
--
2.39.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v1 7/8] mm: zswap: Optimally place members of struct zswap_pool, struct zswap_entry.
2026-10-08 18:28 [PATCH v1 0/8] Batched zswap_store() with compression batching Kanchana P. Sridhar
` (5 preceding siblings ...)
2026-10-08 18:28 ` [PATCH v1 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
@ 2026-10-08 18:28 ` Kanchana P. Sridhar
2026-10-08 18:28 ` [PATCH v1 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches Kanchana P. Sridhar
7 siblings, 0 replies; 11+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-08 18:28 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
Place and align the members of struct zswap_pool and struct zswap_entry
such that members used together in compress/decompress hot paths are
able to share the same cacheline. Minimize holes within these structs
while doing so.
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index 8ae67fd10522..ade84ac4076e 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -160,11 +160,11 @@ struct crypto_acomp_ctx {
struct zswap_pool {
struct zs_pool *zs_pool;
struct crypto_acomp_ctx __percpu *acomp_ctx;
- u8 compr_batch_size;
struct percpu_ref ref;
- struct rcu_work release_rwork;
- struct hlist_node node;
u8 idx;
+ u8 compr_batch_size;
+ struct hlist_node node;
+ struct rcu_work release_rwork;
char tfm_name[CRYPTO_MAX_ALG_NAME];
};
@@ -202,10 +202,10 @@ static struct shrinker *zswap_shrinker;
* logic if referenced is unset. See comments in the shrinker
* section for context.
* nid - NUMA node id of the page for which this is the zswap entry.
- * pool_idx - id of the zswap_pool that the entry's data is in.
* handle - zsmalloc allocation handle that stores the compressed page data
* objcg - the obj_cgroup that the compressed memory is charged to
* lru - handle to the pool's lru used to evict pages.
+ * pool_idx - id of the zswap_pool that the entry's data is in.
*/
struct zswap_entry {
swp_entry_t swpentry;
@@ -214,10 +214,10 @@ struct zswap_entry {
bool referenced:1;
};
int nid;
- u8 pool_idx;
unsigned long handle;
struct obj_cgroup *objcg;
struct list_head lru;
+ u8 pool_idx;
};
/*
--
2.39.5
^ permalink raw reply related [flat|nested] 11+ messages in thread* [PATCH v1 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches.
2026-10-08 18:28 [PATCH v1 0/8] Batched zswap_store() with compression batching Kanchana P. Sridhar
` (6 preceding siblings ...)
2026-10-08 18:28 ` [PATCH v1 7/8] mm: zswap: Optimally place members of struct zswap_pool, struct zswap_entry Kanchana P. Sridhar
@ 2026-10-08 18:28 ` Kanchana P. Sridhar
7 siblings, 0 replies; 11+ messages in thread
From: Kanchana P. Sridhar @ 2026-10-08 18:28 UTC (permalink / raw)
To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
davem, linux-crypto, kanchanapsridhar2026, linux-kernel
Cc: rientjes, willy, senozhatsky
A new batching implementation of zswap_compress() is provided for
compressors that do and do not support batching. This eliminates code
duplication and facilitates code maintainability with the introduction
of compress batching.
The vectorized implementation of calling the earlier zswap_compress()
sequentially, one page at a time in zswap_store_pages(), is replaced
with this new version of zswap_compress() that accepts multiple pages to
compress as a batch.
If the compressor does not support batching, each page in the batch is
compressed and stored sequentially.
If the batch is compressed without errors, the compressed buffers for
the batch are stored in zsmalloc. In case of compression errors, the
current behavior based on whether the folio is enabled for zswap
writeback, is preserved.
The batched zswap_compress() incorporates Herbert's suggestion for
SG lists to represent the batch's inputs/outputs to interface with the
crypto API [1].
Performance data: usemem with 60% over-commit:
==============================================
This patch series was tested on a Chromebook with a QEMU "vng" VM
running in Crostini.
Chromebook Diagnostics reports:
Intel(R) Core(TM) Ultra 5 115U (10 CPUs, 4.200GHz)
RAM: 16GB
vng setup:
Memory: 7G
Swap: 3G
Pinned CPUs: 4 (E-Cores: 4,5,6,7)
zswap compressor:
ZSTD
mTHP enabled:
64kB
vm-scalability/usemem setup:
cgroup memory.high: 3G
4 usemem processes, each allocates 1200M anonymous memory, reads the
Silesia dataset [2] from a pre-fork-allocated buffer into the 1200M,
writes to each unsigned long in the 1200M, sleeps for 10 sec, then
exits.
usemem metrics averaged across 10 iterations*:
================================================
--------------------------------------------------------------------------
Baseline Patch Change
mm-unstable series with
10-3-2026 v1 v1
--------------------------------------------------------------------------
Average Throughput (KB/s) 11,794 17,810 +51.00%
Total Throughput (KB/s) 47,179 71,242 +51.00%
Average Time to Free Memory (usecs) 490,973 294,920 -39.93%
Total Time to Free Memory (usecs) 1,963,892 1,179,682 -39.93%
--------------------------------------------------------------------------
* vmstats are as reported at the end of the 10th iteration:
64kB-alloc 834,567 828,199 -0.76%
64kB-zswpout 700,138 743,697 6.22%
64kB-swpout-fallback 35,342 17,978 -49.13%
64kB-swpout 0 0
--------------------------------------------------------------------------
The speedup with zswap store batching of large folios has a propelling effect
on other parts of mm, especially, by enabling large folios to be swapped out
without splitting, as the data shows. This enables other mm components that are
optimized for large folios, such as munmap(), folio refcounts/mapcounts
updates, PTE invalidation upon freeing memory, zswap_invalidate(); which
benefit from xarray offsets/range continuity. As a result, subsequent large
folio swapouts are more likely to find contiguous swap slots, there are fewer
large folio splits during reclaim, more mTHP zswpouts and the cycle
continues. As the data demonstrates, this directly contributes to more memory
over-commit opportunity.
Architectural considerations for the zswap batching framework:
==============================================================
The zswap batching framework is designed to be hardware-agnostic.
It can be leveraged by any hardware accelerator or software-based
compressor.
Potential future clients of the batching framework:
===================================================
This patch-series demonstrates the performance benefits of compression
batching when used in zswap_store() of large folios. Compression
batching can be used for other use cases such as batching compression in
zram, batch compression of different folios during reclaim, kcompressd,
file systems, etc. Decompression batching can be used to improve
efficiency of zswap writeback (Thanks Nhat for this idea), batching
decompressions in zram, etc.
[1]: https://lore.kernel.org/all/aJ7Fk6RpNc815Ivd@gondor.apana.org.au/T/#m99aea2ce3d284e6c5a3253061d97b08c4752a798
[2]: https://wanos.co/assets/silesia.tar
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
mm/zswap.c | 250 +++++++++++++++++++++++++++++++++++++++--------------
1 file changed, 187 insertions(+), 63 deletions(-)
diff --git a/mm/zswap.c b/mm/zswap.c
index ade84ac4076e..68748349c3ae 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -145,6 +145,7 @@ struct crypto_acomp_ctx {
struct acomp_req *req;
struct crypto_wait wait;
u8 **buffers;
+ struct sg_table *sg_table;
struct mutex mutex;
};
@@ -308,6 +309,12 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx, u8 nr_buffers)
kfree(acomp_ctx->buffers);
}
acomp_ctx->buffers = NULL;
+
+ if (acomp_ctx->sg_table) {
+ sg_free_table(acomp_ctx->sg_table);
+ kfree(acomp_ctx->sg_table);
+ }
+ acomp_ctx->sg_table = NULL;
}
static struct zswap_pool *zswap_pool_create(char *compressor)
@@ -849,6 +856,7 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node);
struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu);
int nid = cpu_to_node(cpu);
+ struct scatterlist *sg;
int ret = -ENOMEM;
u8 i;
@@ -899,6 +907,21 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
goto fail;
}
+ acomp_ctx->sg_table = kmalloc(sizeof(*acomp_ctx->sg_table), GFP_KERNEL);
+ if (!acomp_ctx->sg_table)
+ goto fail;
+
+ if (sg_alloc_table(acomp_ctx->sg_table, pool->compr_batch_size,
+ GFP_KERNEL))
+ goto fail;
+
+ /*
+ * Statically map the per-CPU destination buffers to the per-CPU
+ * SG lists; crucial for performance.
+ */
+ for_each_sg(acomp_ctx->sg_table->sgl, sg, pool->compr_batch_size, i)
+ sg_set_buf(sg, acomp_ctx->buffers[i], PAGE_SIZE);
+
crypto_init_wait(&acomp_ctx->wait);
/*
@@ -923,85 +946,184 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
return ret;
}
-static bool zswap_compress(struct folio *folio, long index,
- struct zswap_entry *entry, struct zswap_pool *pool,
- bool wb_enabled)
+/*
+ * The inner compress batching and pool storage function.
+ */
+static bool __zswap_compress(struct folio *folio,
+ long folio_start,
+ u8 batch_start,
+ u8 nr_batch_pages,
+ struct crypto_acomp_ctx *acomp_ctx,
+ struct zswap_entry *entries[],
+ struct zswap_pool *pool,
+ int nid,
+ bool wb_enabled)
{
- struct crypto_acomp_ctx *acomp_ctx;
- struct scatterlist input, output;
- int comp_ret = 0, alloc_ret = 0;
- unsigned int dlen = PAGE_SIZE;
+ gfp_t gfp = GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE;
+ unsigned int slen = nr_batch_pages * PAGE_SIZE;
+ u8 batch_iter, comp_batch_iter;
+ struct scatterlist *sg;
unsigned long handle;
- gfp_t gfp;
+ int err, dlen;
u8 *dst;
- bool mapped = false;
- acomp_ctx = raw_cpu_ptr(pool->acomp_ctx);
- mutex_lock(&acomp_ctx->mutex);
+ /*
+ * @slen indicates the total source length bytes for
+ * @nr_batch_pages.
+ *
+ * The source folio pages in the batch are directly submitted to
+ * crypto_acomp via acomp_request_set_src_folio().
+ *
+ * The pool's compressor batch size is at least @nr_batch_pages,
+ * hence the acomp_ctx has at least @nr_batch_pages dst @buffers.
+ */
+ acomp_request_set_src_folio(acomp_ctx->req, folio,
+ (folio_start + batch_start) * PAGE_SIZE,
+ slen);
+
+ acomp_ctx->sg_table->sgl->length = slen;
- dst = acomp_ctx->buffers[0];
- sg_init_table(&input, 1);
- sg_set_folio(&input, folio, PAGE_SIZE, index * PAGE_SIZE);
+ /*
+ * The per-CPU @acomp_ctx->sg_table scatterlists are statically
+ * mapped to the per-CPU dst @buffers at pool creation time.
+ */
+ acomp_request_set_dst_sg(acomp_ctx->req,
+ acomp_ctx->sg_table->sgl,
+ slen);
- sg_init_one(&output, dst, PAGE_SIZE);
- acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen);
+ err = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req),
+ &acomp_ctx->wait);
/*
- * it maybe looks a little bit silly that we send an asynchronous request,
- * then wait for its completion synchronously. This makes the process look
- * synchronous in fact.
- * Theoretically, acomp supports users send multiple acomp requests in one
- * acomp instance, then get those requests done simultaneously. but in this
- * case, zswap actually does store and load page by page, there is no
- * existing method to send the second page before the first page is done
- * in one thread doing zswap.
- * but in different threads running on different cpu, we have different
- * acomp instance, so multiple threads can do (de)compression in parallel.
+ * If a page cannot be compressed into a size smaller than
+ * PAGE_SIZE, save the content as is without a compression, to
+ * keep the LRU order of writebacks. If writeback is disabled,
+ * reject the page since it only adds metadata overhead.
+ * swap_writeout() will put the page back to the active LRU list
+ * in the case.
+ *
+ * It is assumed that any compressor that sets the output length
+ * to 0 or a value >= PAGE_SIZE will also return a negative
+ * error status in @err; i.e, will not return a successful
+ * compression status in @err in this case.
*/
- comp_ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait);
- dlen = acomp_ctx->req->dlen;
+ if (unlikely(err && !wb_enabled))
+ goto compress_error;
/*
- * If a page cannot be compressed into a size smaller than PAGE_SIZE,
- * save the content as is without a compression, to keep the LRU order
- * of writebacks. If writeback is disabled, reject the page since it
- * only adds metadata overhead. swap_writeout() will put the page back
- * to the active LRU list in the case.
+ * For each output SG list in @acomp_ctx->req->sg_table->sgl,
+ * the @sg->length should be set to either the page's compressed
+ * length (success), or it's negative value compression error.
*/
- if (comp_ret || !dlen || dlen >= PAGE_SIZE) {
- if (!wb_enabled) {
- comp_ret = comp_ret ? comp_ret : -EINVAL;
- goto unlock;
+ for_each_sg(acomp_ctx->sg_table->sgl, sg, nr_batch_pages,
+ comp_batch_iter) {
+ batch_iter = batch_start + comp_batch_iter;
+ dst = acomp_ctx->buffers[comp_batch_iter];
+ dlen = sg->length;
+
+ if (unlikely(dlen <= 0 || dlen >= PAGE_SIZE)) {
+ dlen = PAGE_SIZE;
+ dst = kmap_local_folio(folio,
+ (folio_start + batch_iter) * PAGE_SIZE);
}
- comp_ret = 0;
- dlen = PAGE_SIZE;
- dst = kmap_local_folio(folio, index * PAGE_SIZE);
- mapped = true;
+
+ handle = zs_malloc(pool->zs_pool, dlen, gfp, nid);
+
+ if (unlikely(IS_ERR_VALUE(handle))) {
+ if (PTR_ERR((void *)handle) == -ENOSPC)
+ zswap_reject_compress_poor++;
+ else
+ zswap_reject_alloc_fail++;
+
+ if (dst != acomp_ctx->buffers[comp_batch_iter])
+ kunmap_local(dst);
+
+ return false;
+ }
+
+ zs_obj_write(pool->zs_pool, handle, dst, dlen);
+ entries[batch_iter]->handle = handle;
+ entries[batch_iter]->length = dlen;
+ if (dst != acomp_ctx->buffers[comp_batch_iter])
+ kunmap_local(dst);
}
- gfp = GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE;
- handle = zs_malloc(pool->zs_pool, dlen, gfp, folio_nid(folio));
- if (IS_ERR_VALUE(handle)) {
- alloc_ret = PTR_ERR((void *)handle);
- goto unlock;
+ return true;
+
+compress_error:
+ for_each_sg(acomp_ctx->sg_table->sgl, sg, nr_batch_pages,
+ comp_batch_iter) {
+ if ((int)sg->length < 0) {
+ if ((int)sg->length == -ENOSPC)
+ zswap_reject_compress_poor++;
+ else
+ zswap_reject_compress_fail++;
+ }
}
- zs_obj_write(pool->zs_pool, handle, dst, dlen);
- entry->handle = handle;
- entry->length = dlen;
+ return false;
+}
-unlock:
- if (mapped)
- kunmap_local(dst);
- if (comp_ret == -ENOSPC || alloc_ret == -ENOSPC)
- zswap_reject_compress_poor++;
- else if (comp_ret)
- zswap_reject_compress_fail++;
- else if (alloc_ret)
- zswap_reject_alloc_fail++;
+/*
+ * zswap_compress() batching implementation for sequential and batching
+ * compressors.
+ *
+ * Description:
+ * ============
+ * Compress multiple @nr_pages in @folio starting from the @folio_start index
+ * in batches of @nr_batch_pages; the latter representing the lesser of the
+ * @nr_pages to be batch compressed and the compressor's batch size.
+ *
+ * @nr_pages can be in (1, ZSWAP_MAX_BATCH_SIZE] even if the compressor does not
+ * support batching.
+ *
+ * If @nr_batch_pages is 1, each page is processed sequentially.
+ *
+ * If @nr_batch_pages is > 1, the compressor may use batching to optimize
+ * compression.
+ */
+static bool zswap_compress(struct folio *folio,
+ long folio_start,
+ u8 nr_pages,
+ u8 nr_batch_pages,
+ struct zswap_entry *entries[],
+ struct zswap_pool *pool,
+ int nid,
+ bool wb_enabled)
+{
+ struct crypto_acomp_ctx *acomp_ctx;
+ bool ret = true;
+ u8 batch_start;
+
+ /*
+ * Locking the acomp_ctx mutex once per store batch results in better
+ * performance as compared to locking per compress batch.
+ */
+ acomp_ctx = raw_cpu_ptr(pool->acomp_ctx);
+ mutex_lock(&acomp_ctx->mutex);
+
+ /*
+ * Compress the @nr_pages in @folio starting at index @folio_start
+ * in batches of @nr_batch_pages.
+ */
+ for (batch_start = 0; batch_start < nr_pages;
+ batch_start += nr_batch_pages) {
+ /*
+ * Send @nr_batch_pages to crypto_acomp for compression:
+ *
+ * These pages are in @folio's range of indices in the interval
+ * [@folio_start + @batch_start,
+ * @folio_start + @batch_start + @nr_batch_pages).
+ */
+ ret = __zswap_compress(folio, folio_start, batch_start,
+ nr_batch_pages, acomp_ctx, entries,
+ pool, nid, wb_enabled);
+ if (unlikely(!ret))
+ break;
+ }
mutex_unlock(&acomp_ctx->mutex);
- return comp_ret == 0 && alloc_ret == 0;
+ return ret;
}
static bool zswap_decompress(struct zswap_entry *entry, struct folio *folio)
@@ -1519,6 +1641,7 @@ static bool zswap_store_pages(struct folio *folio,
{
struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE];
u8 i, store_fail_idx = 0, nr_pages = end - start;
+ bool ret;
VM_WARN_ON_ONCE(nr_pages <= 0 || nr_pages > ZSWAP_MAX_BATCH_SIZE);
@@ -1545,10 +1668,11 @@ static bool zswap_store_pages(struct folio *folio,
INIT_LIST_HEAD(&entries[i]->lru);
}
- for (i = 0; i < nr_pages; ++i) {
- if (!zswap_compress(folio, start + i, entries[i], pool, wb_enabled))
- goto store_pages_failed;
- }
+ ret = zswap_compress(folio, start, nr_pages,
+ min(nr_pages, pool->compr_batch_size),
+ entries, pool, nid, wb_enabled);
+ if (unlikely(!ret))
+ goto store_pages_failed;
for (i = 0; i < nr_pages; ++i) {
struct zswap_entry *old, *entry = entries[i];
--
2.39.5
^ permalink raw reply related [flat|nested] 11+ messages in thread