[PATCH 0/2] crypto: zstd - avoid initializing the workspace twice

[PATCH 0/2] crypto: zstd - avoid initializing the workspace twice

[Usama Arif]

Both zstd_compress() and zstd_decompress() set up the shared per-CPU
workspace as a C/DStream before walking the request, and then, when the
first source and destination fragments each span the whole request, hand
off to zstd_compress_one()/zstd_decompress_one(), which immediately
overwrite that same ctx->wksp with a CCtx/DCtx. The stream setup is
discarded without a byte having been processed.

That one-shot path is not a corner case: zswap always takes it when
storing, and takes it for a load whenever the stored object lies within a
single zsmalloc page.

These two patches defer the stream initialization to the first walk
iteration that actually streams, guarded by a flag because that iteration
can be reached more than once.

A 4 KiB crypto_acomp benchmark [1], twelve runs of nine 30,000-operation
rounds. Bare metal is an Intel Xeon Platinum 8321HC, turbo off,
performance governor, pinned to one core; the VM is a one-vCPU KVM guest
on a faster host.

                  baseline    patched     delta
  bare metal
    compress      52,283 ns   51,038 ns   1,245 ns   2.4%
    decompress     2,317 ns    1,998 ns     319 ns  13.8%
  one-vCPU KVM
    compress      16,675 ns   15,050 ns   1,625 ns   9.8%
    decompress     3,516 ns    2,265 ns   1,251 ns  35.6%

The guest numbers are larger because the two CPUID instructions in
ZSTD_cpuid() become unconditional VM exits there. 

[1] https://gist.github.com/uarif1/5cf02f0e22c23f0d1b3d84348f12914c
 
Usama Arif (2):
  crypto: zstd - Avoid redundant cstream initialization
  crypto: zstd - Avoid redundant dstream initialization

 crypto/zstd.c | 40 ++++++++++++++++++++++++++++------------
 1 file changed, 28 insertions(+), 12 deletions(-)


base-commit: 4b18edbd8e70f7e6860d56370f13244896d0f95c
-- 
2.53.0-Meta

[PATCH 1/2] crypto: zstd - Avoid redundant cstream initialization

[Usama Arif]

zstd_compress() initializes the shared workspace as a CStream before
entering the walk loop. If the first source and destination fragments
each span the whole request it then hands off to zstd_compress_one(),
which initializes that same ctx->wksp as a CCtx, discarding the CStream
setup without having compressed a byte. zswap always takes this one-shot
path when storing, so every page it stores paid for both.

Neither is cheap: zstd_init_cstream() redoes the cwksp layout, zeroes the
ZSTD_CCtx, probes for BMI2 through ZSTD_cpuid(), then resets the session
and parameters and replays ten validated ZSTD_CCtx_setParameter() calls.

Defer the CStream initialization to the first walk iteration that needs
it, guarded by a flag because that iteration can be reached more than
once. The first inner iteration either takes the one-shot path and
returns or initializes the CStream, so the trailing zstd_end_stream()
cannot pick up the stale context left in ctx->cctx by an earlier request.
Unlike the old call site the new one runs with the walk's fragments
mapped, so it has to release them before failing.

For a 4 KB crypto_acomp benchmark for compression, twelve runs of nine
30K operation rounds, on the bare-metal host the median per-round mean
request time fell from 52,283 ns to 51,038 ns (2.4%). In the one-vCPU
KVM guest it fell from 16,675 ns to 15,050 ns (9.8%).
The larger improvement in guest is because of the pair of CPUID
instructions in ZSTD_cpuid() that the removed initialization runs.

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 crypto/zstd.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/crypto/zstd.c b/crypto/zstd.c
index 556f5d2bdd5fb..d64472f3e11e5 100644
--- a/crypto/zstd.c
+++ b/crypto/zstd.c
@@ -96,6 +96,7 @@ static int zstd_compress_one(struct acomp_req *req, struct zstd_ctx *ctx,
 
 static int zstd_compress(struct acomp_req *req)
 {
+	bool stream_initialized = false;
 	struct crypto_acomp_stream *s;
 	unsigned int pos, scur, dcur;
 	unsigned int total_out = 0;
@@ -115,12 +116,6 @@ static int zstd_compress(struct acomp_req *req)
 	if (ret)
 		goto out;
 
-	ctx->cctx = zstd_init_cstream(&ctx->params, 0, ctx->wksp, ctx->wksp_size);
-	if (!ctx->cctx) {
-		ret = -EINVAL;
-		goto out;
-	}
-
 	do {
 		dcur = acomp_walk_next_dst(&walk);
 		if (!dcur) {
@@ -142,6 +137,19 @@ static int zstd_compress(struct acomp_req *req)
 				goto out;
 			}
 
+			if (!stream_initialized) {
+				ctx->cctx = zstd_init_cstream(&ctx->params, 0,
+							      ctx->wksp, ctx->wksp_size);
+				if (!ctx->cctx) {
+					/* Release in the reverse of the map order. */
+					acomp_walk_done_src(&walk, 0);
+					acomp_walk_done_dst(&walk, 0);
+					ret = -EINVAL;
+					goto out;
+				}
+				stream_initialized = true;
+			}
+
 			if (scur) {
 				inbuf.pos = 0;
 				inbuf.src = walk.src.virt.addr;
-- 
2.53.0-Meta

[PATCH 2/2] crypto: zstd - Avoid redundant dstream initialization

[Usama Arif]

zstd_decompress() initializes the shared workspace as a DStream before
entering the walk loop. If the first source and destination fragments
each span the whole request it then hands off to zstd_decompress_one(),
which initializes that same ctx->wksp as a DCtx, discarding the DStream
setup without having decompressed a byte. The two are in fact the same
routine, as ZSTD_initStaticDStream() is a tail call to
ZSTD_initStaticDCtx() and zstd_init_dstream() discards its
max_window_size argument, so the work was done twice byte for byte.

zswap takes this path whenever the stored object lies within a single
zsmalloc page, which is the common case; an object straddling a page
boundary comes back from zs_obj_read_sg_begin() as a two-entry source
scatterlist and streams instead.

Defer the DStream initialization to the first walk iteration that reaches
the streaming path, guarded by a flag because that iteration can be
reached more than once. Within this function ctx->dctx is read only by
the zstd_decompress_stream() call immediately below, so no stale context
can be picked up. As in the previous patch the new call site runs with
the walk's fragments mapped and has to release them before failing.

For a 4 KB crypto_acomp benchmark for decompression, twelve runs of nine
30K operation rounds, on the bare-metal host the median per-round mean
request time fell from 2,317 ns to 1,998 ns (13.8%).
In the one-vCPU KVM guest it fell from 3,516 ns to 2,265 ns (35.6%).

Signed-off-by: Usama Arif <usama.arif@linux.dev>
---
 crypto/zstd.c | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/crypto/zstd.c b/crypto/zstd.c
index d64472f3e11e5..76bdaca327330 100644
--- a/crypto/zstd.c
+++ b/crypto/zstd.c
@@ -215,6 +215,7 @@ static int zstd_decompress_one(struct acomp_req *req, struct zstd_ctx *ctx,
 
 static int zstd_decompress(struct acomp_req *req)
 {
+	bool stream_initialized = false;
 	struct crypto_acomp_stream *s;
 	unsigned int total_out = 0;
 	unsigned int scur, dcur;
@@ -232,12 +233,6 @@ static int zstd_decompress(struct acomp_req *req)
 	if (ret)
 		goto out;
 
-	ctx->dctx = zstd_init_dstream(ZSTD_MAX_SIZE, ctx->wksp, ctx->wksp_size);
-	if (!ctx->dctx) {
-		ret = -EINVAL;
-		goto out;
-	}
-
 	do {
 		scur = acomp_walk_next_src(&walk);
 		if (scur) {
@@ -263,6 +258,19 @@ static int zstd_decompress(struct acomp_req *req)
 				goto out;
 			}
 
+			if (!stream_initialized) {
+				ctx->dctx = zstd_init_dstream(ZSTD_MAX_SIZE, ctx->wksp,
+							      ctx->wksp_size);
+				if (!ctx->dctx) {
+					/* Release in the reverse of the map order. */
+					acomp_walk_done_dst(&walk, 0);
+					acomp_walk_done_src(&walk, 0);
+					ret = -EINVAL;
+					goto out;
+				}
+				stream_initialized = true;
+			}
+
 			outbuf.pos = 0;
 			outbuf.dst = (u8 *)walk.dst.virt.addr;
 			outbuf.size = dcur;
-- 
2.53.0-Meta

Re: [PATCH 0/2] crypto: zstd - avoid initializing the workspace twice

[Yosry Ahmed]

On Tue, Aug 25, 2026 at 3:06 PM Usama Arif <usama.arif@linux.dev> wrote:
>
> Both zstd_compress() and zstd_decompress() set up the shared per-CPU
> workspace as a C/DStream before walking the request, and then, when the
> first source and destination fragments each span the whole request, hand
> off to zstd_compress_one()/zstd_decompress_one(), which immediately
> overwrite that same ctx->wksp with a CCtx/DCtx. The stream setup is
> discarded without a byte having been processed.
>
> That one-shot path is not a corner case: zswap always takes it when
> storing, and takes it for a load whenever the stored object lies within a
> single zsmalloc page.
>
> These two patches defer the stream initialization to the first walk
> iteration that actually streams, guarded by a flag because that iteration
> can be reached more than once.
>
> A 4 KiB crypto_acomp benchmark [1], twelve runs of nine 30,000-operation
> rounds. Bare metal is an Intel Xeon Platinum 8321HC, turbo off,
> performance governor, pinned to one core; the VM is a one-vCPU KVM guest
> on a faster host.
>
>                   baseline    patched     delta
>   bare metal
>     compress      52,283 ns   51,038 ns   1,245 ns   2.4%
>     decompress     2,317 ns    1,998 ns     319 ns  13.8%
>   one-vCPU KVM
>     compress      16,675 ns   15,050 ns   1,625 ns   9.8%
>     decompress     3,516 ns    2,265 ns   1,251 ns  35.6%

Nice!

>
> The guest numbers are larger because the two CPUID instructions in
> ZSTD_cpuid() become unconditional VM exits there.

I wonder how much benefit we get from just doing CPUID once in
ZSTD_cpuid() and cache the results. There might still be value in
avoiding the overall initialization, but I think ZSTD_cpuid() should
only be executing CPUID once anyway (e.g. in case it's called in other
paths)?

>
> [1] https://gist.github.com/uarif1/5cf02f0e22c23f0d1b3d84348f12914c
>
> Usama Arif (2):
>   crypto: zstd - Avoid redundant cstream initialization
>   crypto: zstd - Avoid redundant dstream initialization
>
>  crypto/zstd.c | 40 ++++++++++++++++++++++++++++------------
>  1 file changed, 28 insertions(+), 12 deletions(-)
>
>
> base-commit: 4b18edbd8e70f7e6860d56370f13244896d0f95c
> --
> 2.53.0-Meta
>