[PATCH v6 00/13] zram: redesign zcomp and rework backends

[PATCH v6 00/13] zram: redesign zcomp and rework backends

[Sergey Senozhatsky]

The series has a number of memory usage optimisations and fixes
that result in savings from double digits KB to triple digits
KB per-context/per-CPU, depending on backend and its configuration.
More details are in the corresponding commit messages.

The zcomp redesign and backend rework happen in the final three
patches.  zcomp part, basically, attempts to address the priority
inversion problem, that is seen on some setups.  In zcomp, we have
always kept Read and Write contexts together in per-CPU streams.
With preemptible zram this becomes a bit of a problem, as a higher
priority Reader can preempt a Writer, which may hold a stream
lock, but since the compression stream is locked by the preempted
Writer, the Reader gets blocked on the very same stream lock.
However, we don't have any reasons to have Read and Write contexts
share and compete for the stream, because Read (decompression) and
Write (compression) paths are completely independent, they never
touch each other's buffers.  Thus, we can split compression streams
into per-CPU Read streams and per-CPU Write streams (Barry Song),
so that Readers and Writers don't block each other anymore.
Decoupling Reads and Writes results in noticeable performance
improvements on synthetic tests.  The Reader vs Reader priority
inversion is still possible.  To attack this problem we introduce
shared decompression stream (Brian Geffon) which can be used when
decompression is stateless.  There, however, stateful backends (zstd,
deflate) that cannot have shared decompression contexts.  For those
backends, we introduce a remote CPU borrowing of idle exclusive
decompression contexts, if an exclusive Reader get preempted on
current CPU.  We also now allocate a per-request (Read) bounce
buffer for linearisation of spanning zsmalloc objects, so stateless
decompression contexts do not block each other.

Reader vs Reader priority inversion tests:

  exclusive zstd (fg reader, average of 3 runs):
  Metric           Base               Patched
  ----------------------------------------------------------
  Read IOPS        391k               1,064k        (+2.7x)
  Read clat avg    49.07 us           9.68 us       (-5.1x)
  Read clat stdev  7,179.16 us        30.64 us      (-234x)
  Read clat max    3,208,867 us       5,880.87 us   (-546x)

  shared lz4 (fg reader, average of 3 runs):
  Metric           Base               Patched
  ----------------------------------------------------------
  Read IOPS        972k               1,466k        (+50.9%)
  Read clat avg    11.56 us           3.29 us       (-3.5x)
  Read clat stdev  2,738.84 us        16.57 us      (-165x)
  Read clat max    2,204,467 us       6,442.50 us   (-342x)

Writer vs Reader priority inversion tests:

  lz4 (fg reader, average of 3 runs):
  Metric           Base               Patched
  ----------------------------------------------------------
  Read IOPS        642k               1,472k        (+2.3x)
  Read clat avg    23.97 us           3.28 us       (-7.3x)
  Read clat stdev  4,303.64 us        14.85 us      (-290x)
  Read clat max    2,393,767 us       5,710.53 us   (-419x)

The final patch builds atop of decoupled Read and Write streams and
reworks the way secondary streams are handled.  Secondary compression
streams are only used from recompression, which is serialized by
device lock, IOW it's single-threaded.  However, we still allocated
secondary streams per-CPU (we needed to permit concurrent Reads
of recompressed objects).  Because we now have dedicated R/W
streams, we can allocate a singleton compression context (for
recompression) yet still have per-CPU decompression contexts.
This saves a notable amount of memory with some backends and
configurations (e.g. zstd, deflate, lz4hc).

v5 -> v6:
- fixed for_each_online_cpu_wrap() argument evaluation bug (Sashiko)
- moved the "reset writeback state" patch up
- updates some commit messages

Sergey Senozhatsky (13):
  zram: prefix backends printk-s
  zram: remove debugfs entry on init failure
  zram: reset writeback state in zram_reset_device()
  zram: zstd: do not allocate empty C/D-dictionaries
  zram: lz4hc: pre-initialize dictionary stream in setup_params
  zram: lz4: use LZ4_decompress_safe_usingDict()
  zram: lz4hc: use LZ4_decompress_safe_usingDict()
  zram: zstd: do not use kvzalloc() for zstd allocations
  zram: split zcomp into separate R/W streams
  zram: introduce shared decompression streams
  zram: allocate bounce buffer for spanning objects
  zram: borrow idle decompression streams in exclusive mode
  zram: use a singleton compression context for recompression

 drivers/block/zram/backend_842.c     |   6 +-
 drivers/block/zram/backend_deflate.c |  98 +++++----
 drivers/block/zram/backend_lz4.c     |  85 ++------
 drivers/block/zram/backend_lz4hc.c   | 105 ++++------
 drivers/block/zram/backend_lzo.c     |   6 +-
 drivers/block/zram/backend_lzorle.c  |   6 +-
 drivers/block/zram/backend_zstd.c    | 133 +++++++-----
 drivers/block/zram/zcomp.c           | 298 +++++++++++++++++++++++----
 drivers/block/zram/zcomp.h           |  53 ++++-
 drivers/block/zram/zram_drv.c        |  55 +++--
 10 files changed, 542 insertions(+), 303 deletions(-)

--
2.56.0.212.g129ede20d9-goog

[PATCH v6 01/13] zram: prefix backends printk-s

[Sergey Senozhatsky]

Currently, zram backends prefix printk-s with their
names, which may look a little confusing, e.g.
"zstd: alloc size 408128".  In addition, prefix
backend names with "zram:" so it's clear that the
message is coming from a zram backend.

Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/backend_842.c     | 2 +-
 drivers/block/zram/backend_deflate.c | 2 +-
 drivers/block/zram/backend_lz4.c     | 2 +-
 drivers/block/zram/backend_lz4hc.c   | 2 +-
 drivers/block/zram/backend_lzo.c     | 2 +-
 drivers/block/zram/backend_lzorle.c  | 2 +-
 drivers/block/zram/backend_zstd.c    | 2 +-
 7 files changed, 7 insertions(+), 7 deletions(-)

diff --git a/drivers/block/zram/backend_842.c b/drivers/block/zram/backend_842.c
index 3846a04c69d7..56ffda201727 100644
--- a/drivers/block/zram/backend_842.c
+++ b/drivers/block/zram/backend_842.c
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-or-later
 
-#define pr_fmt(fmt) "842: " fmt
+#define pr_fmt(fmt) "zram: 842: " fmt
 
 #include <linux/kernel.h>
 #include <linux/slab.h>
diff --git a/drivers/block/zram/backend_deflate.c b/drivers/block/zram/backend_deflate.c
index f71b11bcac78..20685aca9bdc 100644
--- a/drivers/block/zram/backend_deflate.c
+++ b/drivers/block/zram/backend_deflate.c
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-or-later
 
-#define pr_fmt(fmt) "deflate: " fmt
+#define pr_fmt(fmt) "zram: deflate: " fmt
 
 #include <linux/kernel.h>
 #include <linux/slab.h>
diff --git a/drivers/block/zram/backend_lz4.c b/drivers/block/zram/backend_lz4.c
index 1e4ad31d39a6..158fd8b5e8a9 100644
--- a/drivers/block/zram/backend_lz4.c
+++ b/drivers/block/zram/backend_lz4.c
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-or-later
 
-#define pr_fmt(fmt) "lz4: " fmt
+#define pr_fmt(fmt) "zram: lz4: " fmt
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
index d8aa01bb258f..5c009835a684 100644
--- a/drivers/block/zram/backend_lz4hc.c
+++ b/drivers/block/zram/backend_lz4hc.c
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-or-later
 
-#define pr_fmt(fmt) "lz4hc: " fmt
+#define pr_fmt(fmt) "zram: lz4hc: " fmt
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
diff --git a/drivers/block/zram/backend_lzo.c b/drivers/block/zram/backend_lzo.c
index d83f92cf757c..78538479c511 100644
--- a/drivers/block/zram/backend_lzo.c
+++ b/drivers/block/zram/backend_lzo.c
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-or-later
 
-#define pr_fmt(fmt) "lzo: " fmt
+#define pr_fmt(fmt) "zram: lzo: " fmt
 
 #include <linux/kernel.h>
 #include <linux/slab.h>
diff --git a/drivers/block/zram/backend_lzorle.c b/drivers/block/zram/backend_lzorle.c
index 1b120d062c92..011787f1430b 100644
--- a/drivers/block/zram/backend_lzorle.c
+++ b/drivers/block/zram/backend_lzorle.c
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-or-later
 
-#define pr_fmt(fmt) "lzo-rle: " fmt
+#define pr_fmt(fmt) "zram: lzo-rle: " fmt
 
 #include <linux/kernel.h>
 #include <linux/slab.h>
diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
index 08da3810cffd..127976b5455b 100644
--- a/drivers/block/zram/backend_zstd.c
+++ b/drivers/block/zram/backend_zstd.c
@@ -1,6 +1,6 @@
 // SPDX-License-Identifier: GPL-2.0-or-later
 
-#define pr_fmt(fmt) "zstd: " fmt
+#define pr_fmt(fmt) "zram: zstd: " fmt
 
 #include <linux/kernel.h>
 #include <linux/slab.h>
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 02/13] zram: remove debugfs entry on init failure

[Sergey Senozhatsky]

Do not leak the debugfs entry when zram_init() fails to register
the block device.

Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/zram_drv.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index ab064c84b092..593fa4ae47c4 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -3297,6 +3297,7 @@ static int __init zram_init(void)
 	if (zram_major <= 0) {
 		pr_err("Unable to get major number\n");
 		class_unregister(&zram_control_class);
+		zram_debugfs_destroy();
 		cpuhp_remove_multi_state(CPUHP_ZCOMP_PREPARE);
 		return -EBUSY;
 	}
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 03/13] zram: reset writeback state in zram_reset_device()

[Sergey Senozhatsky]

Reset all writeback state to its initial defaults on device reset.
We used to keep writeback related data in zram->stats, which
zram_reset_device() clears, but commit 1d69a3f8ae77 ("zram: idle
writeback fixes and cleanup") moved it (along with the newly added
wb_limit_enable) to struct zram, after which neither of them was
reset anymore.  wb_batch_size and compressed_wb, which were added
later, are not reset either.

The patch is more about just being pedantic.

Acked-by: Brian Geffon <bgeffon@google.com>
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/zram_drv.c | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 593fa4ae47c4..0e1aa0da11aa 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -694,6 +694,7 @@ static void reset_bdev(struct zram *zram)
 	zram->disk->fops = &zram_devops;
 	kvfree(zram->bitmap);
 	zram->bitmap = NULL;
+	zram->nr_pages = 0;
 }
 
 static ssize_t backing_dev_show(struct device *dev,
@@ -2881,6 +2882,12 @@ static void zram_reset_device(struct zram *zram)
 	zram_destroy_comps(zram);
 	memset(&zram->stats, 0, sizeof(zram->stats));
 	reset_bdev(zram);
+#ifdef CONFIG_ZRAM_WRITEBACK
+	zram->wb_limit_enable = false;
+	zram->bd_wb_limit = 0;
+	zram->wb_batch_size = 32;
+	zram->compressed_wb = false;
+#endif
 }
 
 static ssize_t disksize_store(struct device *dev, struct device_attribute *attr,
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 04/13] zram: zstd: do not allocate empty C/D-dictionaries

[Sergey Senozhatsky]

Do not zstd_create_cdict_byreference() / zstd_create_ddict_byreference()
C/D dictionaries when no dictionary was provided in the first place,
because zstd still allocates internal memory for such calls (~96 KB at
default level 3, and ~160 KB at levels 9+ on x86_64).

Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/backend_zstd.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
index 127976b5455b..b367049f131d 100644
--- a/drivers/block/zram/backend_zstd.c
+++ b/drivers/block/zram/backend_zstd.c
@@ -76,6 +76,9 @@ static int zstd_setup_params(struct zcomp_params *params)
 	prm = zstd_get_cparams(params->level, PAGE_SIZE,
 			       params->dict_sz);
 
+	if (!params->dict || !params->dict_sz)
+		return 0;
+
 	zp->cdict = zstd_create_cdict_byreference(params->dict,
 						  params->dict_sz,
 						  prm,
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 05/13] zram: lz4hc: pre-initialize dictionary stream in setup_params

[Sergey Senozhatsky]

Pre-initialize the dictionary stream once in lz4hc_setup_params(), store
it in params->drv_data, and memcpy() it into the per-CPU stream before
LZ4_compress_HC_continue(), matching backend_lz4.  This avoids resetting
and hashing the dictionary on every page compression.

This also rejects dictionaries larger than 64 KB at device init time
(matching backend_lz4), instead of failing every page compression with
-EINVAL.

Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/backend_lz4hc.c | 31 +++++++++++++++++++++++++-----
 1 file changed, 26 insertions(+), 5 deletions(-)

diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
index 5c009835a684..f3656c27af21 100644
--- a/drivers/block/zram/backend_lz4hc.c
+++ b/drivers/block/zram/backend_lz4hc.c
@@ -18,10 +18,20 @@ struct lz4hc_ctx {
 
 static void lz4hc_release_params(struct zcomp_params *params)
 {
+	LZ4_streamHC_t *dict_stream = params->drv_data;
+
+	params->drv_data = NULL;
+	if (!dict_stream)
+		return;
+
+	vfree(dict_stream);
 }
 
 static int lz4hc_setup_params(struct zcomp_params *params)
 {
+	LZ4_streamHC_t *dict_stream;
+	int ret;
+
 	if (params->level == ZCOMP_PARAM_NOT_SET) {
 		params->level = LZ4HC_DEFAULT_CLEVEL;
 	} else if (params->level < 1 || params->level > LZ4HC_MAX_CLEVEL) {
@@ -35,6 +45,21 @@ static int lz4hc_setup_params(struct zcomp_params *params)
 		return -EINVAL;
 	}
 
+	if (!params->dict || !params->dict_sz)
+		return 0;
+
+	dict_stream = vzalloc(sizeof(*dict_stream));
+	if (!dict_stream)
+		return -ENOMEM;
+
+	LZ4_resetStreamHC(dict_stream, params->level);
+	ret = LZ4_loadDictHC(dict_stream, params->dict, params->dict_sz);
+	if (ret != params->dict_sz) {
+		vfree(dict_stream);
+		return -EINVAL;
+	}
+	params->drv_data = dict_stream;
+
 	return 0;
 }
 
@@ -93,11 +118,7 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 				      zctx->mem);
 	} else {
 		/* Cstrm needs to be reset */
-		LZ4_resetStreamHC(zctx->cstrm, params->level);
-		ret = LZ4_loadDictHC(zctx->cstrm, params->dict,
-				     params->dict_sz);
-		if (ret != params->dict_sz)
-			return -EINVAL;
+		memcpy(zctx->cstrm, params->drv_data, sizeof(*zctx->cstrm));
 		ret = LZ4_compress_HC_continue(zctx->cstrm, req->src, req->dst,
 					       req->src_len, req->dst_len);
 	}
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 06/13] zram: lz4: use LZ4_decompress_safe_usingDict()

[Sergey Senozhatsky]

Use LZ4_decompress_safe_usingDict() in lz4_decompress() and drop the
per-CPU dstrm allocation.  Store the compression workspace directly
in ctx->context.

Using vmalloc(LZ4_MEM_COMPRESS) instead of kzalloc() for cstrm and
dict_stream when a dictionary is configured also avoids power-of-2
page rounding (~16 KB rounded up to 32 KB), saving ~12 KB per stream.

Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/backend_lz4.c | 76 ++++++--------------------------
 1 file changed, 14 insertions(+), 62 deletions(-)

diff --git a/drivers/block/zram/backend_lz4.c b/drivers/block/zram/backend_lz4.c
index 158fd8b5e8a9..5f2e27805d75 100644
--- a/drivers/block/zram/backend_lz4.c
+++ b/drivers/block/zram/backend_lz4.c
@@ -4,18 +4,10 @@
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
-#include <linux/slab.h>
 #include <linux/vmalloc.h>
 
 #include "backend_lz4.h"
 
-struct lz4_ctx {
-	void *mem;
-
-	LZ4_streamDecode_t *dstrm;
-	LZ4_stream_t *cstrm;
-};
-
 static void lz4_release_params(struct zcomp_params *params)
 {
 	LZ4_stream_t *dict_stream = params->drv_data;
@@ -24,7 +16,7 @@ static void lz4_release_params(struct zcomp_params *params)
 	if (!dict_stream)
 		return;
 
-	kfree(dict_stream);
+	vfree(dict_stream);
 }
 
 static int lz4_setup_params(struct zcomp_params *params)
@@ -42,14 +34,14 @@ static int lz4_setup_params(struct zcomp_params *params)
 	if (!params->dict || !params->dict_sz)
 		return 0;
 
-	dict_stream = kzalloc_obj(*dict_stream);
+	dict_stream = vzalloc(sizeof(*dict_stream));
 	if (!dict_stream)
 		return -ENOMEM;
 
 	ret = LZ4_loadDict(dict_stream,
 			   params->dict, params->dict_sz);
 	if (ret != params->dict_sz) {
-		kfree(dict_stream);
+		vfree(dict_stream);
 		return -EINVAL;
 	}
 	params->drv_data = dict_stream;
@@ -59,61 +51,32 @@ static int lz4_setup_params(struct zcomp_params *params)
 
 static void lz4_destroy(struct zcomp_ctx *ctx)
 {
-	struct lz4_ctx *zctx = ctx->context;
-
-	if (!zctx)
-		return;
-
-	vfree(zctx->mem);
-	kfree(zctx->dstrm);
-	kfree(zctx->cstrm);
-	kfree(zctx);
+	vfree(ctx->context);
 }
 
 static int lz4_create(struct zcomp_params *params, struct zcomp_ctx *ctx)
 {
-	struct lz4_ctx *zctx;
+	BUILD_BUG_ON(sizeof(LZ4_stream_t) != LZ4_MEM_COMPRESS);
 
-	zctx = kzalloc_obj(*zctx);
-	if (!zctx)
+	ctx->context = vmalloc(LZ4_MEM_COMPRESS);
+	if (!ctx->context)
 		return -ENOMEM;
-
-	ctx->context = zctx;
-	if (params->dict_sz == 0) {
-		zctx->mem = vmalloc(LZ4_MEM_COMPRESS);
-		if (!zctx->mem)
-			goto error;
-	} else {
-		zctx->dstrm = kzalloc_obj(*zctx->dstrm);
-		if (!zctx->dstrm)
-			goto error;
-
-		zctx->cstrm = kzalloc_obj(*zctx->cstrm);
-		if (!zctx->cstrm)
-			goto error;
-	}
-
 	return 0;
-
-error:
-	lz4_destroy(ctx);
-	return -ENOMEM;
 }
 
 static int lz4_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 			struct zcomp_req *req)
 {
-	struct lz4_ctx *zctx = ctx->context;
 	int ret;
 
-	if (!zctx->cstrm) {
+	if (!params->drv_data) {
 		ret = LZ4_compress_fast(req->src, req->dst, req->src_len,
 					req->dst_len, params->level,
-					zctx->mem);
+					ctx->context);
 	} else {
 		/* Cstrm needs to be reset */
-		memcpy(zctx->cstrm, params->drv_data, sizeof(*zctx->cstrm));
-		ret = LZ4_compress_fast_continue(zctx->cstrm, req->src,
+		memcpy(ctx->context, params->drv_data, sizeof(LZ4_stream_t));
+		ret = LZ4_compress_fast_continue(ctx->context, req->src,
 						 req->dst, req->src_len,
 						 req->dst_len, params->level);
 	}
@@ -126,22 +89,11 @@ static int lz4_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 static int lz4_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 			  struct zcomp_req *req)
 {
-	struct lz4_ctx *zctx = ctx->context;
 	int ret;
 
-	if (!zctx->dstrm) {
-		ret = LZ4_decompress_safe(req->src, req->dst, req->src_len,
-					  req->dst_len);
-	} else {
-		/* Dstrm needs to be reset */
-		ret = LZ4_setStreamDecode(zctx->dstrm, params->dict,
-					  params->dict_sz);
-		if (!ret)
-			return -EINVAL;
-		ret = LZ4_decompress_safe_continue(zctx->dstrm, req->src,
-						   req->dst, req->src_len,
-						   req->dst_len);
-	}
+	ret = LZ4_decompress_safe_usingDict(req->src, req->dst, req->src_len,
+					    req->dst_len, params->dict,
+					    params->dict_sz);
 	if (ret < 0)
 		return -EINVAL;
 	return 0;
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 07/13] zram: lz4hc: use LZ4_decompress_safe_usingDict()

[Sergey Senozhatsky]

Use LZ4_decompress_safe_usingDict() in lz4hc_decompress() so we don't
need per-CPU LZ4_streamDecode_t anymore.  This also lets us drop
struct lz4hc_ctx and keep compression workspace in ctx->context, just
like backend_lz4 does.

Using vmalloc(LZ4HC_MEM_COMPRESS) instead of kzalloc() for cstrm when a
dictionary is configured also avoids power-of-2 page rounding (~256 KB
rounded up to 512 KB), saving ~252 KB per CPU.

Acked-by: Brian Geffon <bgeffon@google.com>
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/backend_lz4hc.c | 70 +++++-------------------------
 1 file changed, 11 insertions(+), 59 deletions(-)

diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
index f3656c27af21..b62736b26622 100644
--- a/drivers/block/zram/backend_lz4hc.c
+++ b/drivers/block/zram/backend_lz4hc.c
@@ -4,18 +4,10 @@
 
 #include <linux/kernel.h>
 #include <linux/lz4.h>
-#include <linux/slab.h>
 #include <linux/vmalloc.h>
 
 #include "backend_lz4hc.h"
 
-struct lz4hc_ctx {
-	void *mem;
-
-	LZ4_streamDecode_t *dstrm;
-	LZ4_streamHC_t *cstrm;
-};
-
 static void lz4hc_release_params(struct zcomp_params *params)
 {
 	LZ4_streamHC_t *dict_stream = params->drv_data;
@@ -65,61 +57,32 @@ static int lz4hc_setup_params(struct zcomp_params *params)
 
 static void lz4hc_destroy(struct zcomp_ctx *ctx)
 {
-	struct lz4hc_ctx *zctx = ctx->context;
-
-	if (!zctx)
-		return;
-
-	kfree(zctx->dstrm);
-	kfree(zctx->cstrm);
-	vfree(zctx->mem);
-	kfree(zctx);
+	vfree(ctx->context);
 }
 
 static int lz4hc_create(struct zcomp_params *params, struct zcomp_ctx *ctx)
 {
-	struct lz4hc_ctx *zctx;
+	BUILD_BUG_ON(sizeof(LZ4_streamHC_t) != LZ4HC_MEM_COMPRESS);
 
-	zctx = kzalloc_obj(*zctx);
-	if (!zctx)
+	ctx->context = vmalloc(LZ4HC_MEM_COMPRESS);
+	if (!ctx->context)
 		return -ENOMEM;
-
-	ctx->context = zctx;
-	if (params->dict_sz == 0) {
-		zctx->mem = vmalloc(LZ4HC_MEM_COMPRESS);
-		if (!zctx->mem)
-			goto error;
-	} else {
-		zctx->dstrm = kzalloc_obj(*zctx->dstrm);
-		if (!zctx->dstrm)
-			goto error;
-
-		zctx->cstrm = kzalloc_obj(*zctx->cstrm);
-		if (!zctx->cstrm)
-			goto error;
-	}
-
 	return 0;
-
-error:
-	lz4hc_destroy(ctx);
-	return -EINVAL;
 }
 
 static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 			  struct zcomp_req *req)
 {
-	struct lz4hc_ctx *zctx = ctx->context;
 	int ret;
 
-	if (!zctx->cstrm) {
+	if (!params->drv_data) {
 		ret = LZ4_compress_HC(req->src, req->dst, req->src_len,
 				      req->dst_len, params->level,
-				      zctx->mem);
+				      ctx->context);
 	} else {
 		/* Cstrm needs to be reset */
-		memcpy(zctx->cstrm, params->drv_data, sizeof(*zctx->cstrm));
-		ret = LZ4_compress_HC_continue(zctx->cstrm, req->src, req->dst,
+		memcpy(ctx->context, params->drv_data, sizeof(LZ4_streamHC_t));
+		ret = LZ4_compress_HC_continue(ctx->context, req->src, req->dst,
 					       req->src_len, req->dst_len);
 	}
 	if (!ret)
@@ -131,22 +94,11 @@ static int lz4hc_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 static int lz4hc_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 			    struct zcomp_req *req)
 {
-	struct lz4hc_ctx *zctx = ctx->context;
 	int ret;
 
-	if (!zctx->dstrm) {
-		ret = LZ4_decompress_safe(req->src, req->dst, req->src_len,
-					  req->dst_len);
-	} else {
-		/* Dstrm needs to be reset */
-		ret = LZ4_setStreamDecode(zctx->dstrm, params->dict,
-					  params->dict_sz);
-		if (!ret)
-			return -EINVAL;
-		ret = LZ4_decompress_safe_continue(zctx->dstrm, req->src,
-						   req->dst, req->src_len,
-						   req->dst_len);
-	}
+	ret = LZ4_decompress_safe_usingDict(req->src, req->dst, req->src_len,
+					    req->dst_len, params->dict,
+					    params->dict_sz);
 	if (ret < 0)
 		return -EINVAL;
 	return 0;
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 08/13] zram: zstd: do not use kvzalloc() for zstd allocations

[Sergey Senozhatsky]

When a dictionary is configured, zstd allocates all of its memory (C/D
dictionaries, compression and decompression contexts and compression
workspaces) via zstd_custom_alloc().  zstd_custom_alloc() uses
kvzalloc(), which falls back to vmalloc() only when kmalloc() fails,
and kmalloc() rounds such sizes up to a power-of-two number of pages.

For example, on x86_64 with a 110K dictionary, switching from
kvzalloc() to __vmalloc() reduces allocations from:
- CDict (level 3):            512 KB -> 400 KB
- CDict (levels 12+):        2048 KB -> 1552 KB
- DCtx (per-CPU):             128 KB -> 96 KB
- CCtx workspace (level 3):   128 KB -> 84 KB (per-CPU)
- CCtx workspace (level 14+): 1024 KB -> 764 KB (per-CPU)

Use __vmalloc() for allocations larger than PAGE_SIZE (and kzalloc()
for smaller ones), which is also how zram allocates zstd workspaces
when no dictionary is configured.  zstd_custom_free() already uses
kvfree(), which handles both.

Acked-by: Brian Geffon <bgeffon@google.com>
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/backend_zstd.c | 9 ++++++++-
 1 file changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
index b367049f131d..7a8b5ff186f2 100644
--- a/drivers/block/zram/backend_zstd.c
+++ b/drivers/block/zram/backend_zstd.c
@@ -26,10 +26,17 @@ struct zstd_params {
 /*
  * For C/D dictionaries we need to provide zstd with zstd_custom_mem,
  * which zstd uses internally to allocate/free memory when needed.
+ *
+ * zstd allocations (C/D dictionaries and contexts) are large and
+ * long-lived.  Don't use kvzalloc() for them: when enough physically
+ * contiguous memory is available it serves them from kmalloc(), which
+ * rounds them up to a power-of-two number of pages.
  */
 static void *zstd_custom_alloc(void *opaque, size_t size)
 {
-	return kvzalloc(size, GFP_NOIO | __GFP_NOWARN);
+	if (size <= PAGE_SIZE)
+		return kzalloc(size, GFP_NOIO | __GFP_NOWARN);
+	return __vmalloc(size, GFP_NOIO | __GFP_NOWARN | __GFP_ZERO);
 }
 
 static void zstd_custom_free(void *opaque, void *address)
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Sergey Senozhatsky]

Split struct zcomp_strm into separate compression (Write zcomp_cstrm)
and decompression (Read zcomp_dstrm) streams, and split zcomp_ops
context creation/destruction into create_cctx/destroy_cctx and
optional create_dctx/destroy_dctx callbacks.

This decouples Read (decompression) and Write (compression) paths so
they no longer contend on the same per-CPU stream mutex (Writer vs
Reader priority inversion).  Currently, a (low priority) Writer that
holds the stream mutex can be preempted by a (higher priority) Reader
on the same CPU, which then blocks on that mutex until the Writer gets
to run again and releases it, which can take a long time when there
are other runnable tasks on that CPU.  The good news is that Read and
Write contexts are completely independent - compression uses its own
scratch buffer that decompression never touches, and decompression
uses its own local buffer (for spanning zsmalloc objects) that
compression never touches, so the Read/Write context split doesn't
noticeably increase memory usage.  In addition many S/W backends have
stateless decompression contexts.

A quick synthetic fio test for Writer vs Reader inversion on 24 CPUs
(preempt=full, zstd, 48 low priority (nice 19) writers, 24 normal
priority (nice 0) CPU hogs and 24 high priority (nice -19) readers):

  Metric           Base               Patched
  ----------------------------------------------------------
  Read IOPS        207k               1,150k        (+5.6x)
  Read clat avg    101.50 us          7.76 us       (-13.1x)
  Read clat stdev  5,615.90 us        30.78 us      (-182x)
  Read clat p99.99 258,998 us         111.10 us     (-2331x)
  Read clat max    1,342,600 us       11,217 us     (-120x)

Suggested-by: Barry Song <baohua@kernel.org>
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/backend_842.c     |   4 +-
 drivers/block/zram/backend_deflate.c |  96 ++++++++++-------
 drivers/block/zram/backend_lz4.c     |   7 +-
 drivers/block/zram/backend_lz4hc.c   |   4 +-
 drivers/block/zram/backend_lzo.c     |   4 +-
 drivers/block/zram/backend_lzorle.c  |   4 +-
 drivers/block/zram/backend_zstd.c    | 119 +++++++++++++---------
 drivers/block/zram/zcomp.c           | 147 +++++++++++++++++++++------
 drivers/block/zram/zcomp.h           |  28 +++--
 drivers/block/zram/zram_drv.c        |  38 +++----
 10 files changed, 298 insertions(+), 153 deletions(-)

diff --git a/drivers/block/zram/backend_842.c b/drivers/block/zram/backend_842.c
index 56ffda201727..f911cd1ca72e 100644
--- a/drivers/block/zram/backend_842.c
+++ b/drivers/block/zram/backend_842.c
@@ -63,8 +63,8 @@ static int decompress_842(struct zcomp_params *params, struct zcomp_ctx *ctx,
 const struct zcomp_ops backend_842 = {
 	.compress	= compress_842,
 	.decompress	= decompress_842,
-	.create_ctx	= create_842,
-	.destroy_ctx	= destroy_842,
+	.create_cctx	= create_842,
+	.destroy_cctx	= destroy_842,
 	.setup_params	= setup_params_842,
 	.release_params	= release_params_842,
 	.name		= "842",
diff --git a/drivers/block/zram/backend_deflate.c b/drivers/block/zram/backend_deflate.c
index 20685aca9bdc..5ffdaf80d8cf 100644
--- a/drivers/block/zram/backend_deflate.c
+++ b/drivers/block/zram/backend_deflate.c
@@ -13,11 +13,6 @@
 #define DEFLATE_DEF_WINBITS		(-11)
 #define DEFLATE_DEF_MEMLEVEL		MAX_MEM_LEVEL
 
-struct deflate_ctx {
-	struct z_stream_s cctx;
-	struct z_stream_s dctx;
-};
-
 static void deflate_release_params(struct zcomp_params *params)
 {
 }
@@ -51,70 +46,98 @@ static int deflate_setup_params(struct zcomp_params *params)
 	return 0;
 }
 
-static void deflate_destroy(struct zcomp_ctx *ctx)
+static void deflate_destroy_cctx(struct zcomp_ctx *ctx)
 {
-	struct deflate_ctx *zctx = ctx->context;
+	struct z_stream_s *cctx = ctx->context;
 
-	if (!zctx)
+	if (!cctx)
 		return;
 
-	if (zctx->cctx.workspace) {
-		zlib_deflateEnd(&zctx->cctx);
-		vfree(zctx->cctx.workspace);
-	}
-	if (zctx->dctx.workspace) {
-		zlib_inflateEnd(&zctx->dctx);
-		vfree(zctx->dctx.workspace);
+	if (cctx->workspace) {
+		zlib_deflateEnd(cctx);
+		vfree(cctx->workspace);
 	}
-	kfree(zctx);
+	kfree(cctx);
 }
 
-static int deflate_create(struct zcomp_params *params, struct zcomp_ctx *ctx)
+static int deflate_create_cctx(struct zcomp_params *params,
+			       struct zcomp_ctx *ctx)
 {
-	struct deflate_ctx *zctx;
+	struct z_stream_s *cctx;
 	size_t sz;
 	int ret;
 
-	zctx = kzalloc_obj(*zctx);
-	if (!zctx)
+	cctx = kzalloc_obj(*cctx);
+	if (!cctx)
 		return -ENOMEM;
 
-	ctx->context = zctx;
+	ctx->context = cctx;
 	sz = zlib_deflate_workspacesize(params->deflate.winbits, MAX_MEM_LEVEL);
-	zctx->cctx.workspace = vzalloc(sz);
-	if (!zctx->cctx.workspace)
+	cctx->workspace = vzalloc(sz);
+	if (!cctx->workspace)
 		goto error;
 
-	ret = zlib_deflateInit2(&zctx->cctx, params->level, Z_DEFLATED,
+	ret = zlib_deflateInit2(cctx, params->level, Z_DEFLATED,
 				params->deflate.winbits, DEFLATE_DEF_MEMLEVEL,
 				Z_DEFAULT_STRATEGY);
 	if (ret != Z_OK)
 		goto error;
 
+	return 0;
+
+error:
+	deflate_destroy_cctx(ctx);
+	return -EINVAL;
+}
+
+static void deflate_destroy_dctx(struct zcomp_ctx *ctx)
+{
+	struct z_stream_s *dctx = ctx->context;
+
+	if (!dctx)
+		return;
+
+	if (dctx->workspace) {
+		zlib_inflateEnd(dctx);
+		vfree(dctx->workspace);
+	}
+	kfree(dctx);
+}
+
+static int deflate_create_dctx(struct zcomp_params *params,
+			       struct zcomp_ctx *ctx)
+{
+	struct z_stream_s *dctx;
+	size_t sz;
+	int ret;
+
+	dctx = kzalloc_obj(*dctx);
+	if (!dctx)
+		return -ENOMEM;
+
+	ctx->context = dctx;
 	sz = zlib_inflate_workspacesize();
-	zctx->dctx.workspace = vzalloc(sz);
-	if (!zctx->dctx.workspace)
+	dctx->workspace = vzalloc(sz);
+	if (!dctx->workspace)
 		goto error;
 
-	ret = zlib_inflateInit2(&zctx->dctx, params->deflate.winbits);
+	ret = zlib_inflateInit2(dctx, params->deflate.winbits);
 	if (ret != Z_OK)
 		goto error;
 
 	return 0;
 
 error:
-	deflate_destroy(ctx);
+	deflate_destroy_dctx(ctx);
 	return -EINVAL;
 }
 
 static int deflate_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 			    struct zcomp_req *req)
 {
-	struct deflate_ctx *zctx = ctx->context;
-	struct z_stream_s *deflate;
+	struct z_stream_s *deflate = ctx->context;
 	int ret;
 
-	deflate = &zctx->cctx;
 	ret = zlib_deflateReset(deflate);
 	if (ret != Z_OK)
 		return -EINVAL;
@@ -136,12 +159,9 @@ static int deflate_decompress(struct zcomp_params *params,
 			      struct zcomp_ctx *ctx,
 			      struct zcomp_req *req)
 {
-	struct deflate_ctx *zctx = ctx->context;
-	struct z_stream_s *inflate;
+	struct z_stream_s *inflate = ctx->context;
 	int ret;
 
-	inflate = &zctx->dctx;
-
 	ret = zlib_inflateReset(inflate);
 	if (ret != Z_OK)
 		return -EINVAL;
@@ -161,8 +181,10 @@ static int deflate_decompress(struct zcomp_params *params,
 const struct zcomp_ops backend_deflate = {
 	.compress	= deflate_compress,
 	.decompress	= deflate_decompress,
-	.create_ctx	= deflate_create,
-	.destroy_ctx	= deflate_destroy,
+	.create_cctx	= deflate_create_cctx,
+	.destroy_cctx	= deflate_destroy_cctx,
+	.create_dctx	= deflate_create_dctx,
+	.destroy_dctx	= deflate_destroy_dctx,
 	.setup_params	= deflate_setup_params,
 	.release_params	= deflate_release_params,
 	.name		= "deflate",
diff --git a/drivers/block/zram/backend_lz4.c b/drivers/block/zram/backend_lz4.c
index 5f2e27805d75..9767482a2fea 100644
--- a/drivers/block/zram/backend_lz4.c
+++ b/drivers/block/zram/backend_lz4.c
@@ -38,8 +38,7 @@ static int lz4_setup_params(struct zcomp_params *params)
 	if (!dict_stream)
 		return -ENOMEM;
 
-	ret = LZ4_loadDict(dict_stream,
-			   params->dict, params->dict_sz);
+	ret = LZ4_loadDict(dict_stream, params->dict, params->dict_sz);
 	if (ret != params->dict_sz) {
 		vfree(dict_stream);
 		return -EINVAL;
@@ -102,8 +101,8 @@ static int lz4_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 const struct zcomp_ops backend_lz4 = {
 	.compress	= lz4_compress,
 	.decompress	= lz4_decompress,
-	.create_ctx	= lz4_create,
-	.destroy_ctx	= lz4_destroy,
+	.create_cctx	= lz4_create,
+	.destroy_cctx	= lz4_destroy,
 	.setup_params	= lz4_setup_params,
 	.release_params	= lz4_release_params,
 	.name		= "lz4",
diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
index b62736b26622..2d1364a29b27 100644
--- a/drivers/block/zram/backend_lz4hc.c
+++ b/drivers/block/zram/backend_lz4hc.c
@@ -107,8 +107,8 @@ static int lz4hc_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 const struct zcomp_ops backend_lz4hc = {
 	.compress	= lz4hc_compress,
 	.decompress	= lz4hc_decompress,
-	.create_ctx	= lz4hc_create,
-	.destroy_ctx	= lz4hc_destroy,
+	.create_cctx	= lz4hc_create,
+	.destroy_cctx	= lz4hc_destroy,
 	.setup_params	= lz4hc_setup_params,
 	.release_params	= lz4hc_release_params,
 	.name		= "lz4hc",
diff --git a/drivers/block/zram/backend_lzo.c b/drivers/block/zram/backend_lzo.c
index 78538479c511..e703560d9ecd 100644
--- a/drivers/block/zram/backend_lzo.c
+++ b/drivers/block/zram/backend_lzo.c
@@ -61,8 +61,8 @@ static int lzo_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 const struct zcomp_ops backend_lzo = {
 	.compress	= lzo_compress,
 	.decompress	= lzo_decompress,
-	.create_ctx	= lzo_create,
-	.destroy_ctx	= lzo_destroy,
+	.create_cctx	= lzo_create,
+	.destroy_cctx	= lzo_destroy,
 	.setup_params	= lzo_setup_params,
 	.release_params	= lzo_release_params,
 	.name		= "lzo",
diff --git a/drivers/block/zram/backend_lzorle.c b/drivers/block/zram/backend_lzorle.c
index 011787f1430b..e95026e99202 100644
--- a/drivers/block/zram/backend_lzorle.c
+++ b/drivers/block/zram/backend_lzorle.c
@@ -61,8 +61,8 @@ static int lzorle_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 const struct zcomp_ops backend_lzorle = {
 	.compress	= lzorle_compress,
 	.decompress	= lzorle_decompress,
-	.create_ctx	= lzorle_create,
-	.destroy_ctx	= lzorle_destroy,
+	.create_cctx	= lzorle_create,
+	.destroy_cctx	= lzorle_destroy,
 	.setup_params	= lzorle_setup_params,
 	.release_params	= lzorle_release_params,
 	.name		= "lzo-rle",
diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
index 7a8b5ff186f2..d55615f34300 100644
--- a/drivers/block/zram/backend_zstd.c
+++ b/drivers/block/zram/backend_zstd.c
@@ -10,10 +10,8 @@
 #include "backend_zstd.h"
 
 struct zstd_ctx {
-	zstd_cctx *cctx;
-	zstd_dctx *dctx;
-	void *cctx_mem;
-	void *dctx_mem;
+	void *ctx;
+	void *mem;
 };
 
 struct zstd_params {
@@ -105,7 +103,7 @@ static int zstd_setup_params(struct zcomp_params *params)
 	return -EINVAL;
 }
 
-static void zstd_destroy(struct zcomp_ctx *ctx)
+static void zstd_destroy_cctx(struct zcomp_ctx *ctx)
 {
 	struct zstd_ctx *zctx = ctx->context;
 
@@ -113,30 +111,23 @@ static void zstd_destroy(struct zcomp_ctx *ctx)
 		return;
 
 	/*
-	 * If ->cctx_mem and ->dctx_mem were allocated then we didn't use
-	 * C/D dictionary and ->cctx / ->dctx were "embedded" into these
-	 * buffers.
+	 * If ->mem was allocated then we didn't use C/D dictionary and
+	 * ->ctx was "embedded" into this buffer.
 	 *
-	 * If otherwise then we need to explicitly release ->cctx / ->dctx.
+	 * If otherwise then we need to explicitly release ->ctx.
 	 */
-	if (zctx->cctx_mem)
-		vfree(zctx->cctx_mem);
+	if (zctx->mem)
+		vfree(zctx->mem);
 	else
-		zstd_free_cctx(zctx->cctx);
-
-	if (zctx->dctx_mem)
-		vfree(zctx->dctx_mem);
-	else
-		zstd_free_dctx(zctx->dctx);
+		zstd_free_cctx(zctx->ctx);
 
 	kfree(zctx);
 }
 
-static int zstd_create(struct zcomp_params *params, struct zcomp_ctx *ctx)
+static int zstd_create_cctx(struct zcomp_params *params, struct zcomp_ctx *ctx)
 {
+	struct zstd_params *zp = params->drv_data;
 	struct zstd_ctx *zctx;
-	zstd_parameters prm;
-	size_t sz;
 
 	zctx = kzalloc_obj(*zctx);
 	if (!zctx)
@@ -144,40 +135,74 @@ static int zstd_create(struct zcomp_params *params, struct zcomp_ctx *ctx)
 
 	ctx->context = zctx;
 	if (params->dict_sz == 0) {
-		prm = zstd_get_params(params->level, PAGE_SIZE);
-		sz = zstd_cctx_workspace_bound(&prm.cParams);
-		zctx->cctx_mem = vzalloc(sz);
-		if (!zctx->cctx_mem)
-			goto error;
-
-		zctx->cctx = zstd_init_cctx(zctx->cctx_mem, sz);
-		if (!zctx->cctx)
-			goto error;
+		zstd_parameters prm = zp->cprm;
+		size_t sz = zstd_cctx_workspace_bound(&prm.cParams);
 
-		sz = zstd_dctx_workspace_bound();
-		zctx->dctx_mem = vzalloc(sz);
-		if (!zctx->dctx_mem)
+		zctx->mem = vzalloc(sz);
+		if (!zctx->mem)
 			goto error;
 
-		zctx->dctx = zstd_init_dctx(zctx->dctx_mem, sz);
-		if (!zctx->dctx)
+		zctx->ctx = zstd_init_cctx(zctx->mem, sz);
+		if (!zctx->ctx)
 			goto error;
 	} else {
-		struct zstd_params *zp = params->drv_data;
+		zctx->ctx = zstd_create_cctx_advanced(zp->custom_mem);
+		if (!zctx->ctx)
+			goto error;
+	}
+
+	return 0;
+
+error:
+	zstd_destroy_cctx(ctx);
+	return -EINVAL;
+}
+
+static void zstd_destroy_dctx(struct zcomp_ctx *ctx)
+{
+	struct zstd_ctx *zctx = ctx->context;
+
+	if (!zctx)
+		return;
+
+	if (zctx->mem)
+		vfree(zctx->mem);
+	else
+		zstd_free_dctx(zctx->ctx);
+
+	kfree(zctx);
+}
+
+static int zstd_create_dctx(struct zcomp_params *params, struct zcomp_ctx *ctx)
+{
+	struct zstd_params *zp = params->drv_data;
+	struct zstd_ctx *zctx;
 
-		zctx->cctx = zstd_create_cctx_advanced(zp->custom_mem);
-		if (!zctx->cctx)
+	zctx = kzalloc_obj(*zctx);
+	if (!zctx)
+		return -ENOMEM;
+
+	ctx->context = zctx;
+	if (params->dict_sz == 0) {
+		size_t sz = zstd_dctx_workspace_bound();
+
+		zctx->mem = vzalloc(sz);
+		if (!zctx->mem)
 			goto error;
 
-		zctx->dctx = zstd_create_dctx_advanced(zp->custom_mem);
-		if (!zctx->dctx)
+		zctx->ctx = zstd_init_dctx(zctx->mem, sz);
+		if (!zctx->ctx)
+			goto error;
+	} else {
+		zctx->ctx = zstd_create_dctx_advanced(zp->custom_mem);
+		if (!zctx->ctx)
 			goto error;
 	}
 
 	return 0;
 
 error:
-	zstd_destroy(ctx);
+	zstd_destroy_dctx(ctx);
 	return -EINVAL;
 }
 
@@ -189,10 +214,10 @@ static int zstd_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 	size_t ret;
 
 	if (params->dict_sz == 0)
-		ret = zstd_compress_cctx(zctx->cctx, req->dst, req->dst_len,
+		ret = zstd_compress_cctx(zctx->ctx, req->dst, req->dst_len,
 					 req->src, req->src_len, &zp->cprm);
 	else
-		ret = zstd_compress_using_cdict(zctx->cctx, req->dst,
+		ret = zstd_compress_using_cdict(zctx->ctx, req->dst,
 						req->dst_len, req->src,
 						req->src_len,
 						zp->cdict);
@@ -210,10 +235,10 @@ static int zstd_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 	size_t ret;
 
 	if (params->dict_sz == 0)
-		ret = zstd_decompress_dctx(zctx->dctx, req->dst, req->dst_len,
+		ret = zstd_decompress_dctx(zctx->ctx, req->dst, req->dst_len,
 					   req->src, req->src_len);
 	else
-		ret = zstd_decompress_using_ddict(zctx->dctx, req->dst,
+		ret = zstd_decompress_using_ddict(zctx->ctx, req->dst,
 						  req->dst_len, req->src,
 						  req->src_len, zp->ddict);
 	if (zstd_is_error(ret))
@@ -224,8 +249,10 @@ static int zstd_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
 const struct zcomp_ops backend_zstd = {
 	.compress	= zstd_compress,
 	.decompress	= zstd_decompress,
-	.create_ctx	= zstd_create,
-	.destroy_ctx	= zstd_destroy,
+	.create_cctx	= zstd_create_cctx,
+	.destroy_cctx	= zstd_destroy_cctx,
+	.create_dctx	= zstd_create_dctx,
+	.destroy_dctx	= zstd_destroy_dctx,
 	.setup_params	= zstd_setup_params,
 	.release_params	= zstd_release_params,
 	.name		= "zstd",
diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 028e2f0f587e..1c9f6d5c8cd1 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -45,30 +45,65 @@ static const struct zcomp_ops *backends[] = {
 	NULL
 };
 
-static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm)
+static void zcomp_cstrm_free(struct zcomp *comp, struct zcomp_cstrm *zstrm)
 {
-	comp->ops->destroy_ctx(&zstrm->ctx);
-	vfree(zstrm->local_copy);
+	if (comp->ops->destroy_cctx)
+		comp->ops->destroy_cctx(&zstrm->ctx);
 	vfree(zstrm->buffer);
 	zstrm->buffer = NULL;
+	zstrm->ctx.context = NULL;
 }
 
-static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm)
+static int zcomp_cstrm_init(struct zcomp *comp, struct zcomp_cstrm *zstrm)
 {
 	int ret;
 
-	ret = comp->ops->create_ctx(comp->params, &zstrm->ctx);
-	if (ret)
-		return ret;
+	if (comp->ops->create_cctx) {
+		ret = comp->ops->create_cctx(comp->params, &zstrm->ctx);
+		if (ret) {
+			/* backend has released partially created context */
+			zstrm->ctx.context = NULL;
+			return ret;
+		}
+	}
 
-	zstrm->local_copy = vzalloc(PAGE_SIZE);
 	/*
 	 * allocate 2 pages. 1 for compressed data, plus 1 extra for the
 	 * case when compressed size is larger than the original one
 	 */
 	zstrm->buffer = vzalloc(2 * PAGE_SIZE);
-	if (!zstrm->buffer || !zstrm->local_copy) {
-		zcomp_strm_free(comp, zstrm);
+	if (!zstrm->buffer) {
+		zcomp_cstrm_free(comp, zstrm);
+		return -ENOMEM;
+	}
+	return 0;
+}
+
+static void zcomp_dstrm_free(struct zcomp *comp, struct zcomp_dstrm *zstrm)
+{
+	if (comp->ops->destroy_dctx)
+		comp->ops->destroy_dctx(&zstrm->ctx);
+	vfree(zstrm->local_copy);
+	zstrm->local_copy = NULL;
+	zstrm->ctx.context = NULL;
+}
+
+static int zcomp_dstrm_init(struct zcomp *comp, struct zcomp_dstrm *zstrm)
+{
+	int ret;
+
+	if (comp->ops->create_dctx) {
+		ret = comp->ops->create_dctx(comp->params, &zstrm->ctx);
+		if (ret) {
+			/* backend has released partially created context */
+			zstrm->ctx.context = NULL;
+			return ret;
+		}
+	}
+
+	zstrm->local_copy = vzalloc(PAGE_SIZE);
+	if (!zstrm->local_copy) {
+		zcomp_dstrm_free(comp, zstrm);
 		return -ENOMEM;
 	}
 	return 0;
@@ -114,10 +149,10 @@ ssize_t zcomp_available_show(const char *comp, char *buf, ssize_t at)
 	return at;
 }
 
-struct zcomp_strm *zcomp_stream_get(struct zcomp *comp)
+struct zcomp_cstrm *zcomp_cstrm_get(struct zcomp *comp)
 {
 	for (;;) {
-		struct zcomp_strm *zstrm = raw_cpu_ptr(comp->stream);
+		struct zcomp_cstrm *zstrm = raw_cpu_ptr(comp->cstream);
 
 		/*
 		 * Inspired by zswap
@@ -136,12 +171,29 @@ struct zcomp_strm *zcomp_stream_get(struct zcomp *comp)
 	}
 }
 
-void zcomp_stream_put(struct zcomp_strm *zstrm)
+void zcomp_cstrm_put(struct zcomp_cstrm *zstrm)
+{
+	mutex_unlock(&zstrm->lock);
+}
+
+struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp)
+{
+	for (;;) {
+		struct zcomp_dstrm *zstrm = raw_cpu_ptr(comp->dstream);
+
+		mutex_lock(&zstrm->lock);
+		if (likely(zstrm->local_copy))
+			return zstrm;
+		mutex_unlock(&zstrm->lock);
+	}
+}
+
+void zcomp_dstrm_put(struct zcomp_dstrm *zstrm)
 {
 	mutex_unlock(&zstrm->lock);
 }
 
-int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
+int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm,
 		   const void *src, unsigned int *dst_len)
 {
 	struct zcomp_req req = {
@@ -159,7 +211,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
 	return ret;
 }
 
-int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,
+int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm,
 		     struct scatterlist *sg, unsigned int src_len, void *dst)
 {
 	struct zcomp_req req = {
@@ -191,23 +243,44 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,
 int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node)
 {
 	struct zcomp *comp = hlist_entry(node, struct zcomp, node);
-	struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu);
+	struct zcomp_cstrm *cstrm;
+	struct zcomp_dstrm *dstrm;
 	int ret;
 
-	ret = zcomp_strm_init(comp, zstrm);
-	if (ret)
+	cstrm = per_cpu_ptr(comp->cstream, cpu);
+	ret = zcomp_cstrm_init(comp, cstrm);
+	if (ret) {
 		pr_err("Can't allocate a compression stream\n");
-	return ret;
+		return ret;
+	}
+
+	dstrm = per_cpu_ptr(comp->dstream, cpu);
+	ret = zcomp_dstrm_init(comp, dstrm);
+	if (ret) {
+		pr_err("Can't allocate a decompression stream\n");
+		zcomp_cstrm_free(comp, cstrm);
+		return ret;
+	}
+
+	return 0;
 }
 
 int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node)
 {
 	struct zcomp *comp = hlist_entry(node, struct zcomp, node);
-	struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu);
+	struct zcomp_cstrm *cstrm;
+	struct zcomp_dstrm *dstrm;
+
+	cstrm = per_cpu_ptr(comp->cstream, cpu);
+	mutex_lock(&cstrm->lock);
+	zcomp_cstrm_free(comp, cstrm);
+	mutex_unlock(&cstrm->lock);
+
+	dstrm = per_cpu_ptr(comp->dstream, cpu);
+	mutex_lock(&dstrm->lock);
+	zcomp_dstrm_free(comp, dstrm);
+	mutex_unlock(&dstrm->lock);
 
-	mutex_lock(&zstrm->lock);
-	zcomp_strm_free(comp, zstrm);
-	mutex_unlock(&zstrm->lock);
 	return 0;
 }
 
@@ -215,17 +288,27 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params)
 {
 	int ret, cpu;
 
-	comp->stream = alloc_percpu(struct zcomp_strm);
-	if (!comp->stream)
-		return -ENOMEM;
-
 	comp->params = params;
 	ret = comp->ops->setup_params(comp->params);
 	if (ret)
 		goto cleanup;
 
-	for_each_possible_cpu(cpu)
-		mutex_init(&per_cpu_ptr(comp->stream, cpu)->lock);
+	comp->cstream = alloc_percpu(struct zcomp_cstrm);
+	if (!comp->cstream) {
+		ret = -ENOMEM;
+		goto cleanup;
+	}
+
+	comp->dstream = alloc_percpu(struct zcomp_dstrm);
+	if (!comp->dstream) {
+		ret = -ENOMEM;
+		goto cleanup;
+	}
+
+	for_each_possible_cpu(cpu) {
+		mutex_init(&per_cpu_ptr(comp->cstream, cpu)->lock);
+		mutex_init(&per_cpu_ptr(comp->dstream, cpu)->lock);
+	}
 
 	ret = cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node);
 	if (ret < 0)
@@ -235,7 +318,8 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params)
 
 cleanup:
 	comp->ops->release_params(comp->params);
-	free_percpu(comp->stream);
+	free_percpu(comp->dstream);
+	free_percpu(comp->cstream);
 	return ret;
 }
 
@@ -243,7 +327,8 @@ void zcomp_destroy(struct zcomp *comp)
 {
 	cpuhp_state_remove_instance(CPUHP_ZCOMP_PREPARE, &comp->node);
 	comp->ops->release_params(comp->params);
-	free_percpu(comp->stream);
+	free_percpu(comp->dstream);
+	free_percpu(comp->cstream);
 	kfree(comp);
 }
 
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index f38fd31f9e4e..22be50dc51ba 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -39,10 +39,15 @@ struct zcomp_ctx {
 	void *context;
 };
 
-struct zcomp_strm {
+struct zcomp_cstrm {
 	struct mutex lock;
 	/* compression buffer */
 	void *buffer;
+	struct zcomp_ctx ctx;
+};
+
+struct zcomp_dstrm {
+	struct mutex lock;
 	/* local copy of handle memory */
 	void *local_copy;
 	struct zcomp_ctx ctx;
@@ -62,8 +67,11 @@ struct zcomp_ops {
 	int (*decompress)(struct zcomp_params *params, struct zcomp_ctx *ctx,
 			  struct zcomp_req *req);
 
-	int (*create_ctx)(struct zcomp_params *params, struct zcomp_ctx *ctx);
-	void (*destroy_ctx)(struct zcomp_ctx *ctx);
+	int (*create_cctx)(struct zcomp_params *params, struct zcomp_ctx *ctx);
+	void (*destroy_cctx)(struct zcomp_ctx *ctx);
+
+	int (*create_dctx)(struct zcomp_params *params, struct zcomp_ctx *ctx);
+	void (*destroy_dctx)(struct zcomp_ctx *ctx);
 
 	int (*setup_params)(struct zcomp_params *params);
 	void (*release_params)(struct zcomp_params *params);
@@ -73,7 +81,8 @@ struct zcomp_ops {
 
 /* dynamic per-device compression frontend */
 struct zcomp {
-	struct zcomp_strm __percpu *stream;
+	struct zcomp_cstrm __percpu *cstream;
+	struct zcomp_dstrm __percpu *dstream;
 	const struct zcomp_ops *ops;
 	struct zcomp_params *params;
 	struct hlist_node node;
@@ -87,12 +96,15 @@ const char *zcomp_lookup_backend_name(const char *comp);
 struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params);
 void zcomp_destroy(struct zcomp *comp);
 
-struct zcomp_strm *zcomp_stream_get(struct zcomp *comp);
-void zcomp_stream_put(struct zcomp_strm *zstrm);
+struct zcomp_cstrm *zcomp_cstrm_get(struct zcomp *comp);
+void zcomp_cstrm_put(struct zcomp_cstrm *zstrm);
+
+struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp);
+void zcomp_dstrm_put(struct zcomp_dstrm *zstrm);
 
-int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
+int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm,
 		   const void *src, unsigned int *dst_len);
-int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,
+int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm,
 		     struct scatterlist *sg, unsigned int src_len, void *dst);
 
 #endif /* _ZCOMP_H_ */
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 0e1aa0da11aa..def5266679b0 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -1348,7 +1348,7 @@ static ssize_t writeback_store(struct device *dev,
 static int decompress_bdev_page(struct zram *zram, struct page *page,
 				unsigned long index)
 {
-	struct zcomp_strm *zstrm;
+	struct zcomp_dstrm *zstrm;
 	struct scatterlist sg[1];
 	unsigned int size;
 	int ret, prio;
@@ -1373,7 +1373,7 @@ static int decompress_bdev_page(struct zram *zram, struct page *page,
 	sg_init_table(sg, 1);
 	sg_set_page(sg, page, size, 0);
 
-	zstrm = zcomp_stream_get(zram->comps[prio]);
+	zstrm = zcomp_dstrm_get(zram->comps[prio]);
 	ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size,
 			       zstrm->local_copy);
 	if (!ret) {
@@ -1382,7 +1382,7 @@ static int decompress_bdev_page(struct zram *zram, struct page *page,
 		copy_page(dst, zstrm->local_copy);
 		kunmap_local(dst);
 	}
-	zcomp_stream_put(zstrm);
+	zcomp_dstrm_put(zstrm);
 	slot_unlock(zram, index);
 
 	return ret;
@@ -2115,7 +2115,7 @@ static int read_incompressible_page(struct zram *zram, struct page *page,
 static int read_compressed_page(struct zram *zram, struct page *page,
 				unsigned long index)
 {
-	struct zcomp_strm *zstrm;
+	struct zcomp_dstrm *zstrm;
 	struct scatterlist sg[2];
 	unsigned long handle;
 	unsigned int size;
@@ -2126,13 +2126,13 @@ static int read_compressed_page(struct zram *zram, struct page *page,
 	size = get_slot_size(zram, index);
 	prio = get_slot_comp_priority(zram, index);
 
-	zstrm = zcomp_stream_get(zram->comps[prio]);
+	zstrm = zcomp_dstrm_get(zram->comps[prio]);
 	zs_obj_read_sg_begin(zram->mem_pool, handle, sg, size);
 	dst = kmap_local_page(page);
 	ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size, dst);
 	kunmap_local(dst);
 	zs_obj_read_sg_end(zram->mem_pool, handle);
-	zcomp_stream_put(zstrm);
+	zcomp_dstrm_put(zstrm);
 
 	return ret;
 }
@@ -2300,7 +2300,7 @@ static int zram_write_page(struct zram *zram, struct page *page,
 	unsigned long handle;
 	unsigned int comp_len;
 	void *mem;
-	struct zcomp_strm *zstrm;
+	struct zcomp_cstrm *zstrm;
 	unsigned long element;
 	bool same_filled;
 
@@ -2310,20 +2310,20 @@ static int zram_write_page(struct zram *zram, struct page *page,
 	if (same_filled)
 		return write_same_filled_page(zram, element, index);
 
-	zstrm = zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]);
+	zstrm = zcomp_cstrm_get(zram->comps[ZRAM_PRIMARY_COMP]);
 	mem = kmap_local_page(page);
 	ret = zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm,
 			     mem, &comp_len);
 	kunmap_local(mem);
 
 	if (unlikely(ret)) {
-		zcomp_stream_put(zstrm);
+		zcomp_cstrm_put(zstrm);
 		pr_err("Compression failed! err=%d\n", ret);
 		return ret;
 	}
 
 	if (comp_len >= huge_class_size) {
-		zcomp_stream_put(zstrm);
+		zcomp_cstrm_put(zstrm);
 		return write_incompressible_page(zram, page, index);
 	}
 
@@ -2331,18 +2331,18 @@ static int zram_write_page(struct zram *zram, struct page *page,
 			   GFP_NOIO | __GFP_NOWARN |
 			   __GFP_HIGHMEM | __GFP_MOVABLE, page_to_nid(page));
 	if (IS_ERR_VALUE(handle)) {
-		zcomp_stream_put(zstrm);
+		zcomp_cstrm_put(zstrm);
 		return PTR_ERR((void *)handle);
 	}
 
 	if (!zram_can_store_page(zram)) {
-		zcomp_stream_put(zstrm);
+		zcomp_cstrm_put(zstrm);
 		zs_free(zram->mem_pool, handle);
 		return -ENOMEM;
 	}
 
 	zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len);
-	zcomp_stream_put(zstrm);
+	zcomp_cstrm_put(zstrm);
 
 	slot_lock(zram, index);
 	slot_free(zram, index);
@@ -2451,7 +2451,7 @@ static int recompress_slot(struct zram *zram, unsigned long index,
 			   struct page *page, u64 *num_recomp_pages,
 			   u32 threshold, u32 prio)
 {
-	struct zcomp_strm *zstrm = NULL;
+	struct zcomp_cstrm *zstrm = NULL;
 	unsigned long handle_old;
 	unsigned long handle_new;
 	unsigned int comp_len_old;
@@ -2483,7 +2483,7 @@ static int recompress_slot(struct zram *zram, unsigned long index,
 	 */
 	clear_slot_flag(zram, index, ZRAM_IDLE);
 
-	zstrm = zcomp_stream_get(zram->comps[prio]);
+	zstrm = zcomp_cstrm_get(zram->comps[prio]);
 	src = kmap_local_page(page);
 	ret = zcomp_compress(zram->comps[prio], zstrm, src, &comp_len_new);
 	kunmap_local(src);
@@ -2498,7 +2498,7 @@ static int recompress_slot(struct zram *zram, unsigned long index,
 		*num_recomp_pages -= 1;
 
 	if (ret) {
-		zcomp_stream_put(zstrm);
+		zcomp_cstrm_put(zstrm);
 		return ret;
 	}
 
@@ -2507,7 +2507,7 @@ static int recompress_slot(struct zram *zram, unsigned long index,
 
 	if (class_index_new >= class_index_old ||
 	    (threshold && comp_len_new >= threshold)) {
-		zcomp_stream_put(zstrm);
+		zcomp_cstrm_put(zstrm);
 
 		/*
 		 * Secondary algorithms failed to re-compress the page
@@ -2536,12 +2536,12 @@ static int recompress_slot(struct zram *zram, unsigned long index,
 			       __GFP_HIGHMEM | __GFP_MOVABLE,
 			       page_to_nid(page));
 	if (IS_ERR_VALUE(handle_new)) {
-		zcomp_stream_put(zstrm);
+		zcomp_cstrm_put(zstrm);
 		return PTR_ERR((void *)handle_new);
 	}
 
 	zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new);
-	zcomp_stream_put(zstrm);
+	zcomp_cstrm_put(zstrm);
 
 	slot_free(zram, index);
 	set_slot_handle(zram, index, handle_new);
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 10/13] zram: introduce shared decompression streams

[Sergey Senozhatsky]

Currently, per-CPU decompression streams use a mutex, so there is
a chance for Reader vs Reader priority inversion - a low priority
Reader can get preempted by a higher priority Reader, which simply
will get blocked on the stream mutex.

Switch zcomp_dstrm to permit parallel usage.  However, things are not
completely straightforward.  Many backends have stateless decompression
and don't modify decompression context, so those can be used
concurrently.  But then there are backends that do modify decompression
context.  We also can have compressed object span across physical pages,
which zcomp linearises using stream's ->local_copy buffer before
decompression.  That buffer, clearly, cannot be used concurrently.

Give zcomp_dstrm two locks:
- ->lock (rw_semaphore) pins the stream against CPU hotplug teardown.
  Every decompression request takes it in read mode, and only
  zcomp_cpu_dead() takes it in write mode.
- ->excl_lock (mutex) serializes access to mutable stream state
  (->local_copy and ->ctx).

When the backend has no per-stream decompression context (stateless)
and zsmalloc object fits in a single page (zcomp_shared_decomp()),
the request touches neither ->local_copy nor ->ctx, so it only takes
->lock in read mode (ZCOMP_DSTRM_SHARED).  Otherwise, when ->local_copy
is needed or the backend has a decompression context (zstd, deflate),
additionally take ->excl_lock (ZCOMP_DSTRM_EXCLUSIVE) so exclusive
requests serialize only against each other without blocking shared
readers.

A quick synthetic fio test for Reader vs Reader inversion on 24 CPUs
(preempt=full, lz4, 48 low priority (nice 19) readers, 24 normal
priority (nice 0) CPU hogs and 24 high priority (nice -19) readers):

  Metric           Base               Patched
  ----------------------------------------------------------
  Read IOPS        857k               1,382k        (+61.3%)
  Read clat avg    14.40 us           4.07 us       (-3.5x)
  Read clat stdev  3,270.31 us        671.46 us     (-4.9x)
  Read clat p99.99 1,500 us           24.64 us      (-60.9x)
  Read clat max    2,430,200 us       1,022,758 us  (-2.4x)

Suggested-by: Brian Geffon <bgeffon@google.com>
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/zcomp.c    | 56 ++++++++++++++++++++++++++++-------
 drivers/block/zram/zcomp.h    | 21 +++++++++++--
 drivers/block/zram/zram_drv.c | 12 +++++---
 3 files changed, 71 insertions(+), 18 deletions(-)

diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 1c9f6d5c8cd1..da122bf4dd15 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -165,7 +165,7 @@ struct zcomp_cstrm *zcomp_cstrm_get(struct zcomp *comp)
 		 * so then unlock and re-try on the current CPU.
 		 */
 		mutex_lock(&zstrm->lock);
-		if (likely(zstrm->buffer))
+		if (zstrm->buffer)
 			return zstrm;
 		mutex_unlock(&zstrm->lock);
 	}
@@ -176,21 +176,49 @@ void zcomp_cstrm_put(struct zcomp_cstrm *zstrm)
 	mutex_unlock(&zstrm->lock);
 }
 
-struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp)
+bool zcomp_shared_decomp(struct zcomp *comp, struct scatterlist *sg)
+{
+	/*
+	 * Shared decompression is only safe when the backend has no
+	 * mutable per-stream decompression context and the object is
+	 * contained within a single page (no ->local_copy bounce needed).
+	 */
+	return !comp->ops->create_dctx && sg_is_last(sg);
+}
+
+struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp,
+				    enum zcomp_dstrm_mode mode)
 {
 	for (;;) {
 		struct zcomp_dstrm *zstrm = raw_cpu_ptr(comp->dstream);
 
-		mutex_lock(&zstrm->lock);
-		if (likely(zstrm->local_copy))
-			return zstrm;
-		mutex_unlock(&zstrm->lock);
+		/*
+		 * Pin the stream.  The write side of ->lock is only taken
+		 * by zcomp_cpu_dead(), so this never contends with other
+		 * decompression requests, regardless of their mode.
+		 */
+		down_read(&zstrm->lock);
+		if (!zstrm->local_copy) {
+			/* The stream has been destroyed, re-try */
+			up_read(&zstrm->lock);
+			continue;
+		}
+
+		/*
+		 * Exclusive requests additionally own ->local_copy and
+		 * ->ctx and hence can only wait for each other.
+		 */
+		if (mode == ZCOMP_DSTRM_EXCLUSIVE)
+			mutex_lock(&zstrm->excl_lock);
+		return zstrm;
 	}
 }
 
-void zcomp_dstrm_put(struct zcomp_dstrm *zstrm)
+void zcomp_dstrm_put(struct zcomp_dstrm *zstrm, enum zcomp_dstrm_mode mode)
 {
-	mutex_unlock(&zstrm->lock);
+	if (mode == ZCOMP_DSTRM_EXCLUSIVE)
+		mutex_unlock(&zstrm->excl_lock);
+	up_read(&zstrm->lock);
 }
 
 int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm,
@@ -223,6 +251,9 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm,
 	int ret;
 
 	might_sleep();
+	lockdep_assert_held(&zstrm->lock);
+	if (!zcomp_shared_decomp(comp, sg))
+		lockdep_assert_held(&zstrm->excl_lock);
 
 	if (sg_is_last(sg)) {
 		/* the object is contained within one page, read it in-place */
@@ -277,9 +308,9 @@ int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node)
 	mutex_unlock(&cstrm->lock);
 
 	dstrm = per_cpu_ptr(comp->dstream, cpu);
-	mutex_lock(&dstrm->lock);
+	down_write(&dstrm->lock);
 	zcomp_dstrm_free(comp, dstrm);
-	mutex_unlock(&dstrm->lock);
+	up_write(&dstrm->lock);
 
 	return 0;
 }
@@ -306,8 +337,11 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params)
 	}
 
 	for_each_possible_cpu(cpu) {
+		struct zcomp_dstrm *zstrm = per_cpu_ptr(comp->dstream, cpu);
+
 		mutex_init(&per_cpu_ptr(comp->cstream, cpu)->lock);
-		mutex_init(&per_cpu_ptr(comp->dstream, cpu)->lock);
+		init_rwsem(&zstrm->lock);
+		mutex_init(&zstrm->excl_lock);
 	}
 
 	ret = cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node);
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index 22be50dc51ba..bf41b27958c4 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -4,11 +4,17 @@
 #define _ZCOMP_H_
 
 #include <linux/mutex.h>
+#include <linux/rwsem.h>
 
 struct scatterlist;
 
 #define ZCOMP_PARAM_NOT_SET	INT_MIN
 
+enum zcomp_dstrm_mode {
+	ZCOMP_DSTRM_SHARED,
+	ZCOMP_DSTRM_EXCLUSIVE,
+};
+
 struct deflate_params {
 	s32 winbits;
 };
@@ -47,7 +53,14 @@ struct zcomp_cstrm {
 };
 
 struct zcomp_dstrm {
-	struct mutex lock;
+	/*
+	 * Pins the stream: held in read mode by every user, regardless
+	 * of the decompression mode, and in write mode only by
+	 * zcomp_cpu_dead().
+	 */
+	struct rw_semaphore lock;
+	/* Serializes users of the mutable stream state: ->local_copy, ->ctx */
+	struct mutex excl_lock;
 	/* local copy of handle memory */
 	void *local_copy;
 	struct zcomp_ctx ctx;
@@ -99,8 +112,10 @@ void zcomp_destroy(struct zcomp *comp);
 struct zcomp_cstrm *zcomp_cstrm_get(struct zcomp *comp);
 void zcomp_cstrm_put(struct zcomp_cstrm *zstrm);
 
-struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp);
-void zcomp_dstrm_put(struct zcomp_dstrm *zstrm);
+bool zcomp_shared_decomp(struct zcomp *comp, struct scatterlist *sg);
+struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp,
+				    enum zcomp_dstrm_mode mode);
+void zcomp_dstrm_put(struct zcomp_dstrm *zstrm, enum zcomp_dstrm_mode mode);
 
 int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm,
 		   const void *src, unsigned int *dst_len);
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index def5266679b0..a34859ff0461 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -1373,7 +1373,7 @@ static int decompress_bdev_page(struct zram *zram, struct page *page,
 	sg_init_table(sg, 1);
 	sg_set_page(sg, page, size, 0);
 
-	zstrm = zcomp_dstrm_get(zram->comps[prio]);
+	zstrm = zcomp_dstrm_get(zram->comps[prio], ZCOMP_DSTRM_EXCLUSIVE);
 	ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size,
 			       zstrm->local_copy);
 	if (!ret) {
@@ -1382,7 +1382,7 @@ static int decompress_bdev_page(struct zram *zram, struct page *page,
 		copy_page(dst, zstrm->local_copy);
 		kunmap_local(dst);
 	}
-	zcomp_dstrm_put(zstrm);
+	zcomp_dstrm_put(zstrm, ZCOMP_DSTRM_EXCLUSIVE);
 	slot_unlock(zram, index);
 
 	return ret;
@@ -2115,6 +2115,7 @@ static int read_incompressible_page(struct zram *zram, struct page *page,
 static int read_compressed_page(struct zram *zram, struct page *page,
 				unsigned long index)
 {
+	enum zcomp_dstrm_mode mode = ZCOMP_DSTRM_EXCLUSIVE;
 	struct zcomp_dstrm *zstrm;
 	struct scatterlist sg[2];
 	unsigned long handle;
@@ -2126,13 +2127,16 @@ static int read_compressed_page(struct zram *zram, struct page *page,
 	size = get_slot_size(zram, index);
 	prio = get_slot_comp_priority(zram, index);
 
-	zstrm = zcomp_dstrm_get(zram->comps[prio]);
 	zs_obj_read_sg_begin(zram->mem_pool, handle, sg, size);
+	if (zcomp_shared_decomp(zram->comps[prio], sg))
+		mode = ZCOMP_DSTRM_SHARED;
+
+	zstrm = zcomp_dstrm_get(zram->comps[prio], mode);
 	dst = kmap_local_page(page);
 	ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size, dst);
 	kunmap_local(dst);
+	zcomp_dstrm_put(zstrm, mode);
 	zs_obj_read_sg_end(zram->mem_pool, handle);
-	zcomp_dstrm_put(zstrm);
 
 	return ret;
 }
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 11/13] zram: allocate bounce buffer for spanning objects

[Sergey Senozhatsky]

zsmalloc objects that span two physical pages need to be linearized
before decompression, which zcomp does using stream's ->local_copy
buffer.  That buffer cannot be used concurrently, so such reads take
the stream in exclusive mode, even when the backend has stateless
decompression.  And spanning objects are not rare: Bo Zhang measured
23.3% of reads on lzo-rle going through the exclusive path, causing
contention on ->excl_lock of up to 72-96 ms [1].

Allocate a temporary bounce buffer for spanning objects instead, so
that backends with stateless decompression can always use shared mode.
If the allocation fails we fall back to ->local_copy under ->excl_lock.

Per-request bounce buffer fixes the remaining tail latency, as was
measured by Bo Zhang [2]:

  Metric          Base      kmalloc bounce
  ----------------------------------------
  read contended  ~1,959    0
  shared          76.7%     100%

A quick synthetic fio test for Reader vs Reader inversion on 24 CPUs
(lz4, 48 low priority (nice 19) readers, 24 normal priority CPU hogs
 and 24 high priority readers), average of 3 runs:

  Metric           Base               Patched
  ----------------------------------------------------------
  Read IOPS        1,452k             1,466k        (+1.0%)
  Read clat avg    3.46 us            3.30 us       (-4.9%)
  Read clat stdev  165.37 us          15.69 us      (-10.5x)
  Read clat max    306,679 us         7,060 us      (-43.4x)

An alternative approach is to vm_map_ram() scattered objects
(two pages), which fixes the tail latency in the same way,
but deferred TLB flushes add more cost, so it was rejected.

[1] https://lore.kernel.org/linux-kernel/20261014153603.1365617-1-zhangbo56@xiaomi.com
[2] https://lore.kernel.org/linux-kernel/20261015112440.1491168-1-zhangbo56@xiaomi.com
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/zcomp.c    | 47 +++++++++++++++++++++++++----------
 drivers/block/zram/zcomp.h    |  2 +-
 drivers/block/zram/zram_drv.c |  2 +-
 3 files changed, 36 insertions(+), 15 deletions(-)

diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index da122bf4dd15..4fda13aac896 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -176,14 +176,13 @@ void zcomp_cstrm_put(struct zcomp_cstrm *zstrm)
 	mutex_unlock(&zstrm->lock);
 }
 
-bool zcomp_shared_decomp(struct zcomp *comp, struct scatterlist *sg)
+bool zcomp_shared_decomp(struct zcomp *comp)
 {
 	/*
 	 * Shared decompression is only safe when the backend has no
-	 * mutable per-stream decompression context and the object is
-	 * contained within a single page (no ->local_copy bounce needed).
+	 * mutable per-stream decompression context.
 	 */
-	return !comp->ops->create_dctx && sg_is_last(sg);
+	return !comp->ops->create_dctx;
 }
 
 struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp,
@@ -239,6 +238,15 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm,
 	return ret;
 }
 
+static int zcomp_decompress_buf(struct zcomp *comp, struct zcomp_dstrm *zstrm,
+				struct scatterlist *sg, void *buf,
+				struct zcomp_req *req)
+{
+	sg_copy_to_buffer(sg, 2, buf, req->src_len);
+	req->src = buf;
+	return comp->ops->decompress(comp->params, &zstrm->ctx, req);
+}
+
 int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm,
 		     struct scatterlist *sg, unsigned int src_len, void *dst)
 {
@@ -247,27 +255,40 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm,
 		.src_len = src_len,
 		.dst_len = PAGE_SIZE,
 	};
-	void *src = NULL;
+	void *src;
 	int ret;
 
 	might_sleep();
 	lockdep_assert_held(&zstrm->lock);
-	if (!zcomp_shared_decomp(comp, sg))
+	if (!zcomp_shared_decomp(comp))
 		lockdep_assert_held(&zstrm->excl_lock);
 
 	if (sg_is_last(sg)) {
 		/* the object is contained within one page, read it in-place */
 		src = kmap_local_page(sg_page(sg));
 		req.src = src + sg->offset;
-	} else {
-		/* the object spans two pages, linearize it into local copy */
-		sg_copy_to_buffer(sg, 2, zstrm->local_copy, src_len);
-		req.src = zstrm->local_copy;
+		ret = comp->ops->decompress(comp->params, &zstrm->ctx, &req);
+		kunmap_local(src);
+		return ret;
 	}
 
-	ret = comp->ops->decompress(comp->params, &zstrm->ctx, &req);
-	if (src)
-		kunmap_local(src);
+	/* exclusive mode already holds ->excl_lock */
+	if (!zcomp_shared_decomp(comp))
+		return zcomp_decompress_buf(comp, zstrm, sg, zstrm->local_copy,
+					    &req);
+
+	/* shared mode: try a per-request bounce buffer first */
+	src = kmalloc(src_len, GFP_NOWAIT | __GFP_NOWARN);
+	if (src) {
+		ret = zcomp_decompress_buf(comp, zstrm, sg, src, &req);
+		kfree(src);
+		return ret;
+	}
+
+	/* fall back to ->local_copy serialized by ->excl_lock */
+	mutex_lock(&zstrm->excl_lock);
+	ret = zcomp_decompress_buf(comp, zstrm, sg, zstrm->local_copy, &req);
+	mutex_unlock(&zstrm->excl_lock);
 	return ret;
 }
 
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index bf41b27958c4..362a6da014aa 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -112,7 +112,7 @@ void zcomp_destroy(struct zcomp *comp);
 struct zcomp_cstrm *zcomp_cstrm_get(struct zcomp *comp);
 void zcomp_cstrm_put(struct zcomp_cstrm *zstrm);
 
-bool zcomp_shared_decomp(struct zcomp *comp, struct scatterlist *sg);
+bool zcomp_shared_decomp(struct zcomp *comp);
 struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp,
 				    enum zcomp_dstrm_mode mode);
 void zcomp_dstrm_put(struct zcomp_dstrm *zstrm, enum zcomp_dstrm_mode mode);
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index a34859ff0461..b429c52cb37d 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -2128,7 +2128,7 @@ static int read_compressed_page(struct zram *zram, struct page *page,
 	prio = get_slot_comp_priority(zram, index);
 
 	zs_obj_read_sg_begin(zram->mem_pool, handle, sg, size);
-	if (zcomp_shared_decomp(zram->comps[prio], sg))
+	if (zcomp_shared_decomp(zram->comps[prio]))
 		mode = ZCOMP_DSTRM_SHARED;
 
 	zstrm = zcomp_dstrm_get(zram->comps[prio], mode);
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 12/13] zram: borrow idle decompression streams in exclusive mode

[Sergey Senozhatsky]

Shared decompression streams eliminate Reader vs Reader priority
inversion for backends with stateless decompression.  However,
backends with stateful decompression contexts (zstd, deflate)
still require exclusive access to the stream [1].

When a low priority exclusive stream Reader is preempted by a high
priority exclusive Reader, instead of unconditionally blocking on
the local CPU's ->excl_lock, we can scan online CPUs starting from
the current and try to borrow first free stream.  Only when all
online streams are busy, sleep on a per-zcomp waitqueue until any
CPU releases an exclusive stream, which significantly drops
contention and wait time, per Bo Zhang's measurements [2]:

  Metric              Base      cross-CPU borrow
  ----------------------------------------------
  dexcl_contended     15,919    1,447
  dexcl_wait_ns_total 109.6 s   3.66 s
  dexcl_wait_ns_max   145 ms    26.6 ms
  dexcl_avg_wait      6.88 ms   2.53 ms
  dexcl_acquire       2.21M     2.94M

A quick synthetic fio test for Reader vs Reader inversion on 24 CPUs
(preempt=full, zstd, 48 low priority (nice 19) bg readers, 24 normal
priority (nice 0) CPU hogs and 24 high priority (nice -19) fg readers),
fg reader results (average of 3 runs):

  Metric           Base               Patched
  ----------------------------------------------------------
  Read IOPS        396k               1,064k        (+2.7x)
  Read clat avg    48.32 us           9.66 us       (-5.0x)
  Read clat stdev  7,571.00 us        31.92 us      (-237x)
  Read clat p99.99 5,707.10 us        2,233.68 us   (-2.6x)
  Read clat max    3,555,300 us       6,287.57 us   (-565x)

[1] https://lore.kernel.org/linux-kernel/20261015112440.1491168-1-zhangbo56@xiaomi.com
[2] https://lore.kernel.org/linux-kernel/20261016134847.1620103-1-zhangbo0325@gmail.com
Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/zcomp.c    | 77 +++++++++++++++++++++++++----------
 drivers/block/zram/zcomp.h    |  5 ++-
 drivers/block/zram/zram_drv.c |  4 +-
 3 files changed, 61 insertions(+), 25 deletions(-)

diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 4fda13aac896..72683f91313f 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -185,38 +185,67 @@ bool zcomp_shared_decomp(struct zcomp *comp)
 	return !comp->ops->create_dctx;
 }
 
+static bool zcomp_dstrm_excl_trylock(struct zcomp_dstrm *zstrm)
+{
+	/* Cheap pre-check: don't touch a busy stream's cachelines */
+	if (mutex_is_locked(&zstrm->excl_lock))
+		return false;
+	/*
+	 * Pin the stream.  The write side of ->lock is only taken by
+	 * zcomp_cpu_dead(), which also sets ->local_copy to NULL.
+	 */
+	if (!down_read_trylock(&zstrm->lock))
+		return false;
+	if (zstrm->local_copy && mutex_trylock(&zstrm->excl_lock))
+		return true;
+	up_read(&zstrm->lock);
+	return false;
+}
+
+static struct zcomp_dstrm *zcomp_dstrm_excl_tryget(struct zcomp *comp)
+{
+	unsigned int cpu, start = raw_smp_processor_id();
+	struct zcomp_dstrm *zstrm;
+
+	/* Best-effort start CPU hint; migration during the scan is fine */
+	for_each_online_cpu_wrap(cpu, start) {
+		zstrm = per_cpu_ptr(comp->dstream, cpu);
+		if (zcomp_dstrm_excl_trylock(zstrm))
+			return zstrm;
+	}
+	return NULL;
+}
+
 struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp,
 				    enum zcomp_dstrm_mode mode)
 {
-	for (;;) {
-		struct zcomp_dstrm *zstrm = raw_cpu_ptr(comp->dstream);
-
-		/*
-		 * Pin the stream.  The write side of ->lock is only taken
-		 * by zcomp_cpu_dead(), so this never contends with other
-		 * decompression requests, regardless of their mode.
-		 */
-		down_read(&zstrm->lock);
-		if (!zstrm->local_copy) {
-			/* The stream has been destroyed, re-try */
+	struct zcomp_dstrm *zstrm;
+
+	if (mode == ZCOMP_DSTRM_SHARED) {
+		for (;;) {
+			zstrm = raw_cpu_ptr(comp->dstream);
+			down_read(&zstrm->lock);
+			if (zstrm->local_copy)
+				return zstrm;
 			up_read(&zstrm->lock);
-			continue;
 		}
-
-		/*
-		 * Exclusive requests additionally own ->local_copy and
-		 * ->ctx and hence can only wait for each other.
-		 */
-		if (mode == ZCOMP_DSTRM_EXCLUSIVE)
-			mutex_lock(&zstrm->excl_lock);
-		return zstrm;
 	}
+
+	/* Checks zcomp_dstrm_excl_tryget() before touching the waitq */
+	wait_event(comp->dstrm_waitq, (zstrm = zcomp_dstrm_excl_tryget(comp)));
+	return zstrm;
 }
 
-void zcomp_dstrm_put(struct zcomp_dstrm *zstrm, enum zcomp_dstrm_mode mode)
+void zcomp_dstrm_put(struct zcomp *comp, struct zcomp_dstrm *zstrm,
+		     enum zcomp_dstrm_mode mode)
 {
-	if (mode == ZCOMP_DSTRM_EXCLUSIVE)
+	if (mode == ZCOMP_DSTRM_EXCLUSIVE) {
 		mutex_unlock(&zstrm->excl_lock);
+		up_read(&zstrm->lock);
+		if (wq_has_sleeper(&comp->dstrm_waitq))
+			wake_up(&comp->dstrm_waitq);
+		return;
+	}
 	up_read(&zstrm->lock);
 }
 
@@ -289,6 +318,8 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm,
 	mutex_lock(&zstrm->excl_lock);
 	ret = zcomp_decompress_buf(comp, zstrm, sg, zstrm->local_copy, &req);
 	mutex_unlock(&zstrm->excl_lock);
+	if (wq_has_sleeper(&comp->dstrm_waitq))
+		wake_up(&comp->dstrm_waitq);
 	return ret;
 }
 
@@ -357,6 +388,8 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params)
 		goto cleanup;
 	}
 
+	init_waitqueue_head(&comp->dstrm_waitq);
+
 	for_each_possible_cpu(cpu) {
 		struct zcomp_dstrm *zstrm = per_cpu_ptr(comp->dstream, cpu);
 
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index 362a6da014aa..ca1ebc70ad0a 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -5,6 +5,7 @@
 
 #include <linux/mutex.h>
 #include <linux/rwsem.h>
+#include <linux/wait.h>
 
 struct scatterlist;
 
@@ -96,6 +97,7 @@ struct zcomp_ops {
 struct zcomp {
 	struct zcomp_cstrm __percpu *cstream;
 	struct zcomp_dstrm __percpu *dstream;
+	wait_queue_head_t dstrm_waitq;
 	const struct zcomp_ops *ops;
 	struct zcomp_params *params;
 	struct hlist_node node;
@@ -115,7 +117,8 @@ void zcomp_cstrm_put(struct zcomp_cstrm *zstrm);
 bool zcomp_shared_decomp(struct zcomp *comp);
 struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp,
 				    enum zcomp_dstrm_mode mode);
-void zcomp_dstrm_put(struct zcomp_dstrm *zstrm, enum zcomp_dstrm_mode mode);
+void zcomp_dstrm_put(struct zcomp *comp, struct zcomp_dstrm *zstrm,
+		     enum zcomp_dstrm_mode mode);
 
 int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm,
 		   const void *src, unsigned int *dst_len);
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index b429c52cb37d..52341f05da83 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -1382,7 +1382,7 @@ static int decompress_bdev_page(struct zram *zram, struct page *page,
 		copy_page(dst, zstrm->local_copy);
 		kunmap_local(dst);
 	}
-	zcomp_dstrm_put(zstrm, ZCOMP_DSTRM_EXCLUSIVE);
+	zcomp_dstrm_put(zram->comps[prio], zstrm, ZCOMP_DSTRM_EXCLUSIVE);
 	slot_unlock(zram, index);
 
 	return ret;
@@ -2135,7 +2135,7 @@ static int read_compressed_page(struct zram *zram, struct page *page,
 	dst = kmap_local_page(page);
 	ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size, dst);
 	kunmap_local(dst);
-	zcomp_dstrm_put(zstrm, mode);
+	zcomp_dstrm_put(zram->comps[prio], zstrm, mode);
 	zs_obj_read_sg_end(zram->mem_pool, handle);
 
 	return ret;
-- 
2.56.0.212.g129ede20d9-goog

[PATCH v6 13/13] zram: use a singleton compression context for recompression

[Sergey Senozhatsky]

Secondary compressors are only used by recompress_store(), which is
serialized under zram->dev_lock, so we only ever use one secondary
compressor at a time, yet we still allocate them on each CPU, which is
just a plain memory wastage.  However, we do need per-CPU decompressors
in order to support concurrent reads.

Now that Read and Write contexts are split, we can allocate just a
single compression stream (recomp_cstream) for secondary algorithms
instead of per-CPU streams, avoiding unused per-CPU compression
contexts and 2 * PAGE_SIZE buffers.

For example, on an 8-CPU x86_64 system (4K PAGE_SIZE), this saves:
- zstd (level 3, default):        ~700 KB (~100 KB/CPU)
- zstd (levels 12-22, no dict):   ~1.91 MB (~280 KB/CPU)
- zstd (levels 14-22, 110K dict): ~5.33 MB (~780 KB/CPU, lazy)
- deflate (winbits=-11, default): ~1.04 MB (~152 KB/CPU)
- deflate (winbits=15):           ~1.86 MB (~272 KB/CPU)
- lz4hc:                          ~1.84 MB (~269 KB/CPU)

Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
---
 drivers/block/zram/zcomp.c    | 75 ++++++++++++++++++++++++++---------
 drivers/block/zram/zcomp.h    |  9 +++--
 drivers/block/zram/zram_drv.c |  5 ++-
 3 files changed, 65 insertions(+), 24 deletions(-)

diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
index 72683f91313f..ad8c7dc73922 100644
--- a/drivers/block/zram/zcomp.c
+++ b/drivers/block/zram/zcomp.c
@@ -151,6 +151,11 @@ ssize_t zcomp_available_show(const char *comp, char *buf, ssize_t at)
 
 struct zcomp_cstrm *zcomp_cstrm_get(struct zcomp *comp)
 {
+	if (comp->recomp_cstream) {
+		mutex_lock(&comp->recomp_cstream->lock);
+		return comp->recomp_cstream;
+	}
+
 	for (;;) {
 		struct zcomp_cstrm *zstrm = raw_cpu_ptr(comp->cstream);
 
@@ -326,22 +331,25 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm,
 int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node)
 {
 	struct zcomp *comp = hlist_entry(node, struct zcomp, node);
-	struct zcomp_cstrm *cstrm;
+	struct zcomp_cstrm *cstrm = NULL;
 	struct zcomp_dstrm *dstrm;
 	int ret;
 
-	cstrm = per_cpu_ptr(comp->cstream, cpu);
-	ret = zcomp_cstrm_init(comp, cstrm);
-	if (ret) {
-		pr_err("Can't allocate a compression stream\n");
-		return ret;
+	if (comp->cstream) {
+		cstrm = per_cpu_ptr(comp->cstream, cpu);
+		ret = zcomp_cstrm_init(comp, cstrm);
+		if (ret) {
+			pr_err("Can't allocate a compression stream\n");
+			return ret;
+		}
 	}
 
 	dstrm = per_cpu_ptr(comp->dstream, cpu);
 	ret = zcomp_dstrm_init(comp, dstrm);
 	if (ret) {
 		pr_err("Can't allocate a decompression stream\n");
-		zcomp_cstrm_free(comp, cstrm);
+		if (cstrm)
+			zcomp_cstrm_free(comp, cstrm);
 		return ret;
 	}
 
@@ -354,10 +362,12 @@ int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node)
 	struct zcomp_cstrm *cstrm;
 	struct zcomp_dstrm *dstrm;
 
-	cstrm = per_cpu_ptr(comp->cstream, cpu);
-	mutex_lock(&cstrm->lock);
-	zcomp_cstrm_free(comp, cstrm);
-	mutex_unlock(&cstrm->lock);
+	if (comp->cstream) {
+		cstrm = per_cpu_ptr(comp->cstream, cpu);
+		mutex_lock(&cstrm->lock);
+		zcomp_cstrm_free(comp, cstrm);
+		mutex_unlock(&cstrm->lock);
+	}
 
 	dstrm = per_cpu_ptr(comp->dstream, cpu);
 	down_write(&dstrm->lock);
@@ -367,7 +377,8 @@ int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node)
 	return 0;
 }
 
-static int zcomp_init(struct zcomp *comp, struct zcomp_params *params)
+static int zcomp_init(struct zcomp *comp, struct zcomp_params *params,
+		      bool recomp)
 {
 	int ret, cpu;
 
@@ -376,10 +387,28 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params)
 	if (ret)
 		goto cleanup;
 
-	comp->cstream = alloc_percpu(struct zcomp_cstrm);
-	if (!comp->cstream) {
-		ret = -ENOMEM;
-		goto cleanup;
+	if (recomp) {
+		comp->recomp_cstream = kzalloc_obj(*comp->recomp_cstream);
+		if (!comp->recomp_cstream) {
+			ret = -ENOMEM;
+			goto cleanup;
+		}
+		mutex_init(&comp->recomp_cstream->lock);
+		ret = zcomp_cstrm_init(comp, comp->recomp_cstream);
+		if (ret) {
+			/* zcomp_cstrm_init() cleans up after itself */
+			kfree(comp->recomp_cstream);
+			comp->recomp_cstream = NULL;
+			goto cleanup;
+		}
+	} else {
+		comp->cstream = alloc_percpu(struct zcomp_cstrm);
+		if (!comp->cstream) {
+			ret = -ENOMEM;
+			goto cleanup;
+		}
+		for_each_possible_cpu(cpu)
+			mutex_init(&per_cpu_ptr(comp->cstream, cpu)->lock);
 	}
 
 	comp->dstream = alloc_percpu(struct zcomp_dstrm);
@@ -393,7 +422,6 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params)
 	for_each_possible_cpu(cpu) {
 		struct zcomp_dstrm *zstrm = per_cpu_ptr(comp->dstream, cpu);
 
-		mutex_init(&per_cpu_ptr(comp->cstream, cpu)->lock);
 		init_rwsem(&zstrm->lock);
 		mutex_init(&zstrm->excl_lock);
 	}
@@ -405,6 +433,10 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params)
 	return 0;
 
 cleanup:
+	if (comp->recomp_cstream) {
+		zcomp_cstrm_free(comp, comp->recomp_cstream);
+		kfree(comp->recomp_cstream);
+	}
 	comp->ops->release_params(comp->params);
 	free_percpu(comp->dstream);
 	free_percpu(comp->cstream);
@@ -414,13 +446,18 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params)
 void zcomp_destroy(struct zcomp *comp)
 {
 	cpuhp_state_remove_instance(CPUHP_ZCOMP_PREPARE, &comp->node);
+	if (comp->recomp_cstream) {
+		zcomp_cstrm_free(comp, comp->recomp_cstream);
+		kfree(comp->recomp_cstream);
+	}
 	comp->ops->release_params(comp->params);
 	free_percpu(comp->dstream);
 	free_percpu(comp->cstream);
 	kfree(comp);
 }
 
-struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params)
+struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params,
+			   bool recomp)
 {
 	struct zcomp *comp;
 	int error;
@@ -443,7 +480,7 @@ struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params)
 		return ERR_PTR(-EINVAL);
 	}
 
-	error = zcomp_init(comp, params);
+	error = zcomp_init(comp, params, recomp);
 	if (error) {
 		kfree(comp);
 		return ERR_PTR(error);
diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
index ca1ebc70ad0a..39efefc43a13 100644
--- a/drivers/block/zram/zcomp.h
+++ b/drivers/block/zram/zcomp.h
@@ -40,7 +40,8 @@ struct zcomp_params {
 /*
  * Run-time driver context - scratch buffers, etc. It is modified during
  * request execution (compression/decompression), cannot be shared, so
- * it's in per-CPU area.
+ * it's in per-CPU area (except for recompression, which uses a single
+ * per-device compression stream).
  */
 struct zcomp_ctx {
 	void *context;
@@ -95,9 +96,10 @@ struct zcomp_ops {
 
 /* dynamic per-device compression frontend */
 struct zcomp {
-	struct zcomp_cstrm __percpu *cstream;
 	struct zcomp_dstrm __percpu *dstream;
 	wait_queue_head_t dstrm_waitq;
+	struct zcomp_cstrm __percpu *cstream;
+	struct zcomp_cstrm *recomp_cstream;
 	const struct zcomp_ops *ops;
 	struct zcomp_params *params;
 	struct hlist_node node;
@@ -108,7 +110,8 @@ int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node);
 ssize_t zcomp_available_show(const char *comp, char *buf, ssize_t at);
 const char *zcomp_lookup_backend_name(const char *comp);
 
-struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params);
+struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params,
+			   bool recomp);
 void zcomp_destroy(struct zcomp *comp);
 
 struct zcomp_cstrm *zcomp_cstrm_get(struct zcomp *comp);
diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
index 52341f05da83..079f3723db31 100644
--- a/drivers/block/zram/zram_drv.c
+++ b/drivers/block/zram/zram_drv.c
@@ -2526,7 +2526,7 @@ static int recompress_slot(struct zram *zram, unsigned long index,
 	}
 
 	/*
-	 * We are holding per-CPU stream mutex and entry lock so better
+	 * We are holding compression stream mutex and entry lock so better
 	 * avoid direct reclaim.  Allocation error is not fatal since
 	 * we still have the old object in the mem_pool.
 	 *
@@ -2928,7 +2928,8 @@ static ssize_t disksize_store(struct device *dev, struct device_attribute *attr,
 			continue;
 
 		comp = zcomp_create(zram->comp_algs[prio],
-				    &zram->params[prio]);
+				    &zram->params[prio],
+				    prio != ZRAM_PRIMARY_COMP);
 		if (IS_ERR(comp)) {
 			pr_err("Cannot initialise %s compressing backend\n",
 			       zram->comp_algs[prio]);
-- 
2.56.0.212.g129ede20d9-goog

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Hillf Danton]

On Wed, 21 Oct 2026 11:42:19 +0900 Sergey Senozhatsky wrote:
> Split struct zcomp_strm into separate compression (Write zcomp_cstrm)
> and decompression (Read zcomp_dstrm) streams, and split zcomp_ops
> context creation/destruction into create_cctx/destroy_cctx and
> optional create_dctx/destroy_dctx callbacks.
> 
> This decouples Read (decompression) and Write (compression) paths so
> they no longer contend on the same per-CPU stream mutex (Writer vs
> Reader priority inversion).  Currently, a (low priority) Writer that
> holds the stream mutex can be preempted by a (higher priority) Reader
> on the same CPU, which then blocks on that mutex until the Writer gets
> to run again and releases it, which can take a long time when there
> are other runnable tasks on that CPU.
>
Given lock like mutex, priority inversion happens from Monday to Friday,
but I wonder if you are inventing some wheels in 2026 if proxy excution
works for you.

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Sergey Senozhatsky]

On (26/10/21 12:23), Hillf Danton wrote:
> On Wed, 21 Oct 2026 11:42:19 +0900 Sergey Senozhatsky wrote:
> > Split struct zcomp_strm into separate compression (Write zcomp_cstrm)
> > and decompression (Read zcomp_dstrm) streams, and split zcomp_ops
> > context creation/destruction into create_cctx/destroy_cctx and
> > optional create_dctx/destroy_dctx callbacks.
> > 
> > This decouples Read (decompression) and Write (compression) paths so
> > they no longer contend on the same per-CPU stream mutex (Writer vs
> > Reader priority inversion).  Currently, a (low priority) Writer that
> > holds the stream mutex can be preempted by a (higher priority) Reader
> > on the same CPU, which then blocks on that mutex until the Writer gets
> > to run again and releases it, which can take a long time when there
> > are other runnable tasks on that CPU.
> >
> Given lock like mutex, priority inversion happens from Monday to Friday,
> but I wonder if you are inventing some wheels in 2026 if proxy excution
> works for you.

Reader-Writer split is a foundation for further improvements (like
singleton recompression stream and so on).

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Hillf Danton]

On Wed, 21 Oct 2026 13:56:17 +0900 Sergey Senozhatsky wrote:
>On (26/10/21 12:23), Hillf Danton wrote:
>> On Wed, 21 Oct 2026 11:42:19 +0900 Sergey Senozhatsky wrote:
>> > Split struct zcomp_strm into separate compression (Write zcomp_cstrm)
>> > and decompression (Read zcomp_dstrm) streams, and split zcomp_ops
>> > context creation/destruction into create_cctx/destroy_cctx and
>> > optional create_dctx/destroy_dctx callbacks.
>> > 
>> > This decouples Read (decompression) and Write (compression) paths so
>> > they no longer contend on the same per-CPU stream mutex (Writer vs
>> > Reader priority inversion).  Currently, a (low priority) Writer that
>> > holds the stream mutex can be preempted by a (higher priority) Reader
>> > on the same CPU, which then blocks on that mutex until the Writer gets
>> > to run again and releases it, which can take a long time when there
>> > are other runnable tasks on that CPU.
>> >
>> Given lock like mutex, priority inversion happens from Monday to Friday,
>> but I wonder if you are inventing some wheels in 2026 if proxy excution
>> works for you.
>
> Reader-Writer split is a foundation for further improvements (like
> singleton recompression stream and so on).
>
But that split is not needed at all if PE works, yes?

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Hillf Danton]

On Wed, 21 Oct 2026 15:53:21 +0800 Hillf Danton wrote:
>On Wed, 21 Oct 2026 13:56:17 +0900 Sergey Senozhatsky wrote:
>>On (26/10/21 12:23), Hillf Danton wrote:
>>> On Wed, 21 Oct 2026 11:42:19 +0900 Sergey Senozhatsky wrote:
>>> > Split struct zcomp_strm into separate compression (Write zcomp_cstrm)
>>> > and decompression (Read zcomp_dstrm) streams, and split zcomp_ops
>>> > context creation/destruction into create_cctx/destroy_cctx and
>>> > optional create_dctx/destroy_dctx callbacks.
>>> > 
>>> > This decouples Read (decompression) and Write (compression) paths so
>>> > they no longer contend on the same per-CPU stream mutex (Writer vs
>>> > Reader priority inversion).  Currently, a (low priority) Writer that
>>> > holds the stream mutex can be preempted by a (higher priority) Reader
>>> > on the same CPU, which then blocks on that mutex until the Writer gets
>>> > to run again and releases it, which can take a long time when there
>>> > are other runnable tasks on that CPU.
>>> >
>>> Given lock like mutex, priority inversion happens from Monday to Friday,
>>> but I wonder if you are inventing some wheels in 2026 if proxy excution
>>> works for you.
>>
>> Reader-Writer split is a foundation for further improvements (like
>> singleton recompression stream and so on).
>>
> But that split is not needed at all if PE works, yes?
>
Because PE is good at handling complex priority hiarchy. In a simple case,
task1(priority=1) is the owner of lock1 with two waiters, task2(priority=2)
and task3(priority=3), while task3 is the owner of lock2 also with two waiters,
task4(priority=5) and task5(priority=7).

When the number of locks and tasks grows, I have increasing difficulty even
describing the lock chain that you in combination with Brian are unable to
handle. That is why in general PE is prefered over whatever wheels you can create.

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Barry Song]

On Thu, Oct 22, 2026 at 8:12 AM Hillf Danton <hdanton@sina.com> wrote:
>
> On Wed, 21 Oct 2026 15:53:21 +0800 Hillf Danton wrote:
> >On Wed, 21 Oct 2026 13:56:17 +0900 Sergey Senozhatsky wrote:
> >>On (26/10/21 12:23), Hillf Danton wrote:
> >>> On Wed, 21 Oct 2026 11:42:19 +0900 Sergey Senozhatsky wrote:
> >>> > Split struct zcomp_strm into separate compression (Write zcomp_cstrm)
> >>> > and decompression (Read zcomp_dstrm) streams, and split zcomp_ops
> >>> > context creation/destruction into create_cctx/destroy_cctx and
> >>> > optional create_dctx/destroy_dctx callbacks.
> >>> >
> >>> > This decouples Read (decompression) and Write (compression) paths so
> >>> > they no longer contend on the same per-CPU stream mutex (Writer vs
> >>> > Reader priority inversion).  Currently, a (low priority) Writer that
> >>> > holds the stream mutex can be preempted by a (higher priority) Reader
> >>> > on the same CPU, which then blocks on that mutex until the Writer gets
> >>> > to run again and releases it, which can take a long time when there
> >>> > are other runnable tasks on that CPU.
> >>> >
> >>> Given lock like mutex, priority inversion happens from Monday to Friday,
> >>> but I wonder if you are inventing some wheels in 2026 if proxy excution
> >>> works for you.
> >>
> >> Reader-Writer split is a foundation for further improvements (like
> >> singleton recompression stream and so on).
> >>
> > But that split is not needed at all if PE works, yes?
> >
> Because PE is good at handling complex priority hiarchy. In a simple case,
> task1(priority=1) is the owner of lock1 with two waiters, task2(priority=2)
> and task3(priority=3), while task3 is the owner of lock2 also with two waiters,
> task4(priority=5) and task5(priority=7).
>
> When the number of locks and tasks grows, I have increasing difficulty even
> describing the lock chain that you in combination with Brian are unable to
> handle. That is why in general PE is prefered over whatever wheels you can create.
>

I still believe that reducing lock contention is the most correct way
to address  lock
contention.

Proxy execution and other priority-inheritance mechanisms can help in
some cases,
but they also have downsides. For example, allowing a low-priority
task to run at a
high priority can affect other tasks. When many tasks require proxy execution or
priority inheritance, they still compete for the same resources.

So I'd vote for reducing lock contention as the first option. If that
turns out to be
impractical, we can consider priority inheritance.

Best Regards
Barry

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Sergey Senozhatsky]

On (26/10/22 08:08), Hillf Danton wrote:
[..]
> >>> Given lock like mutex, priority inversion happens from Monday to Friday,
> >>> but I wonder if you are inventing some wheels in 2026 if proxy excution
> >>> works for you.
> >>
> >> Reader-Writer split is a foundation for further improvements (like
> >> singleton recompression stream and so on).
> >>
> > But that split is not needed at all if PE works, yes?
> >
> 
> When the number of locks and tasks grows, I have increasing difficulty even
> describing the lock chain that you in combination with Brian are unable to
> handle. That is why in general PE is prefered over whatever wheels you can create.

Hillf, if we reinvent the wheel then I'm very happy with this reinvention.
But we don't.

There is not a single reason for compression and decompression to be in
the same context - they don't share a single byte.  And there is no
reason whatsoever to serialize them against each other.  Compression and
decompression should have been separate contexts from the beginning.
While serializing decompression with a PE/PI-enabled mutex is cool, not
having decompression serialization at all is way cooler.  In addition,
decoupling compression and decompression leads to a much saner recompression
stream design, which saves considerable amount of per-CPU memory.
That's what this patch does for us.

The commit message probably can be better, but other than that I'm
happy with this series.

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Hillf Danton]

On Thu, 22 Oct 2026 08:19:41 +0800 Barry Song wrote:
>On Thu, Oct 22, 2026 at 8:12 AM Hillf Danton <hdanton@sina.com> wrote:
>> On Wed, 21 Oct 2026 15:53:21 +0800 Hillf Danton wrote:
>> >On Wed, 21 Oct 2026 13:56:17 +0900 Sergey Senozhatsky wrote:
>> >>On (26/10/21 12:23), Hillf Danton wrote:
>> >>> On Wed, 21 Oct 2026 11:42:19 +0900 Sergey Senozhatsky wrote:
>> >>> > Split struct zcomp_strm into separate compression (Write zcomp_cstrm)
>> >>> > and decompression (Read zcomp_dstrm) streams, and split zcomp_ops
>> >>> > context creation/destruction into create_cctx/destroy_cctx and
>> >>> > optional create_dctx/destroy_dctx callbacks.
>> >>> >
>> >>> > This decouples Read (decompression) and Write (compression) paths so
>> >>> > they no longer contend on the same per-CPU stream mutex (Writer vs
>> >>> > Reader priority inversion).  Currently, a (low priority) Writer that
>> >>> > holds the stream mutex can be preempted by a (higher priority) Reader
>> >>> > on the same CPU, which then blocks on that mutex until the Writer gets
>> >>> > to run again and releases it, which can take a long time when there
>> >>> > are other runnable tasks on that CPU.
>> >>> >
>> >>> Given lock like mutex, priority inversion happens from Monday to Friday,
>> >>> but I wonder if you are inventing some wheels in 2026 if proxy excution
>> >>> works for you.
>> >>
>> >> Reader-Writer split is a foundation for further improvements (like
>> >> singleton recompression stream and so on).
>> >>
>> > But that split is not needed at all if PE works, yes?
>> >
>> Because PE is good at handling complex priority hiarchy. In a simple case,
>> task1(priority=1) is the owner of lock1 with two waiters, task2(priority=2)
>> and task3(priority=3), while task3 is the owner of lock2 also with two waiters,
>> task4(priority=5) and task5(priority=7).
>>
>> When the number of locks and tasks grows, I have increasing difficulty even
>> describing the lock chain that you in combination with Brian are unable to
>> handle. That is why in general PE is prefered over whatever wheels you can create.
>>
>
> I still believe that reducing lock contention is the most correct way
> to address  lock
> contention.
>
Given the log message above, I dont think the issue in question is the regular
lock contention, but priority inversion caused by preempt. Due to preempt, the
preempter becomes lock waiter without spinning for lock owner, which I think
creates the issue. Particularly given the spin for owner in mutex, if the lock
is hot then the critical section is supposed to be not well implemented.

What Sergey did not show is -- what if the preemptee gets back on cpu at the
preempter's priority after preempt, with the critical section assumed good.
This can be checked by making the Writer high priority instead.

What I am confused is -- why decompression needs to run faster without boosting
compression up. Where is the compressed material?

> Proxy execution and other priority-inheritance mechanisms can help in
> some cases,
> but they also have downsides. For example, allowing a low-priority
> task to run at a
> high priority can affect other tasks. When many tasks require proxy execution or

No agree. Just because of the high priority lock waiter, low priority lock
owner can run at the high priority, so the priority inversion is effectively
erased, particularly given a well implemented critical section.

> priority inheritance, they still compete for the same resources.
>
Even when lock contension rises, the lock owner runs at the priority as
high as the lock chain can offer.

>So I'd vote for reducing lock contention as the first option. If that
>turns out to be
>impractical, we can consider priority inheritance.
>
>Best Regards
>Barry

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Hillf Danton]

On Thu, 22 Oct 2026 11:06:53 +0900 Sergey Senozhatsky wrote:
> On (26/10/22 08:08), Hillf Danton wrote:
> [..]
> > >>> Given lock like mutex, priority inversion happens from Monday to Friday,
> > >>> but I wonder if you are inventing some wheels in 2026 if proxy excution
> > >>> works for you.
> > >>
> > >> Reader-Writer split is a foundation for further improvements (like
> > >> singleton recompression stream and so on).
> > >>
> > > But that split is not needed at all if PE works, yes?
> > >
> > 
> > When the number of locks and tasks grows, I have increasing difficulty even
> > describing the lock chain that you in combination with Brian are unable to
> > handle. That is why in general PE is prefered over whatever wheels you can create.
> 
> Hillf, if we reinvent the wheel then I'm very happy with this reinvention.
> But we don't.
> 
> There is not a single reason for compression and decompression to be in
> the same context - they don't share a single byte.  And there is no
> reason whatsoever to serialize them against each other.  Compression and
> decompression should have been separate contexts from the beginning.
> While serializing decompression with a PE/PI-enabled mutex is cool, not
> having decompression serialization at all is way cooler.  In addition,

I see compression and decompression are decoupled at the mutex level, but given
the "same CPU" in your log message, nothing changes at the CPU level, reguarding
the micro seconds to complete the two critical sections. So the decouple sounds
like the half baked croissant, HBC, the Eric Dumazet dude sometimes is happy
to deliver to lore.

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Sergey Senozhatsky]

On (26/10/23 07:25), Hillf Danton wrote:
[..]
> So the decouple sounds like the half baked croissant, HBC, the
> Eric Dumazet dude sometimes is happy to deliver to lore.

I take it as a compliment.

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Sergey Senozhatsky]

On (26/10/22 19:11), Hillf Danton wrote:
[..]
> What I am confused is -- why decompression needs to run faster without boosting
> compression up. Where is the compressed material?

In a typical embedded device (e.g. android) zram write is swap-out
(direct, indirect kswapd, or backgroud service swapping out idle apps),
users don't see those.  zram read, on the contrary, is what users
see - because that's swap-in, read bursts when user brings back idle
app (camera app, etc.) which needs to get back to foregound quickly.
So, the read path is more important.

I tested the series against PE.

readers vs readers (no writers) 3 runs

    BASE+PE     IOPS 1,449k  avg 3.48us  sd 155.6us  p99.99 26.8us  max 280,820us
                IOPS 1,435k  avg 3.69us  sd 322.9us  p99.99 24.2us  max 540,512us
                IOPS 1,459k  avg 3.46us  sd  92.8us  p99.99 25.5us  max 172,186us
    PATCHED+PE  IOPS 1,457k  avg 3.31us  sd  15.9us  p99.99 25.5us  max   7,580us
                IOPS 1,475k  avg 3.28us  sd  14.6us  p99.99 24.5us  max   7,327us
                IOPS 1,463k  avg 3.32us  sd  17.2us  p99.99 28.5us  max   6,816us

R/W stream split makes sense for reasons that we already mentioned.
Readers don't need to be blocked on Writers - they don't compete for
the same data.  And, moreover, readers don't need to be serialized at
all for stateful decompression.

writers vs readers 3 runs

    BASE+PE     IOPS 1,271k  avg 5.57us  sd 1,393us  p99.99 27.5us  max 1,581,800us
                IOPS 1,461k  avg 3.43us  sd    78us  p99.99 23.9us  max   227,015us
                IOPS 1,444k  avg 3.48us  sd   222us  p99.99 23.9us  max   592,628us
    PATCHED+PE  IOPS 1,457k  avg 3.32us  sd    18us  p99.99 27.5us  max     5,620us
                IOPS 1,464k  avg 3.31us  sd    17us  p99.99 24.2us  max     5,621us
                IOPS 1,475k  avg 3.29us  sd    16us  p99.99 26.5us  max     5,620us

And, in addition, R/W streams split saves per-CPU memory.

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Hillf Danton]

On Fri, 23 Oct 2026 14:01:32 +0900 Sergey Senozhatsky wrote:
> On (26/10/22 19:11), Hillf Danton wrote:
> [..]
> > What I am confused is -- why decompression needs to run faster without boosting
> > compression up. Where is the compressed material?
> 
> In a typical embedded device (e.g. android) zram write is swap-out
> (direct, indirect kswapd, or backgroud service swapping out idle apps),
> users don't see those.  zram read, on the contrary, is what users
> see - because that's swap-in, read bursts when user brings back idle
> app (camera app, etc.) which needs to get back to foregound quickly.
> So, the read path is more important.
> 
> I tested the series against PE.
> 
Thanks for sharing data.

> readers vs readers (no writers) 3 runs
> 
>     BASE+PE     IOPS 1,449k  avg 3.48us  sd 155.6us  p99.99 26.8us  max 280,820us
>                 IOPS 1,435k  avg 3.69us  sd 322.9us  p99.99 24.2us  max 540,512us
>                 IOPS 1,459k  avg 3.46us  sd  92.8us  p99.99 25.5us  max 172,186us
>     PATCHED+PE  IOPS 1,457k  avg 3.31us  sd  15.9us  p99.99 25.5us  max   7,580us
>                 IOPS 1,475k  avg 3.28us  sd  14.6us  p99.99 24.5us  max   7,327us
>                 IOPS 1,463k  avg 3.32us  sd  17.2us  p99.99 28.5us  max   6,816us
> 
     BASE+PE     IOPS 1,449k
                 IOPS 1,435k
                 IOPS 1,459k
		+)----------
		      4,343
	       /3)----------
		      1,447

     PATCHED+PE  IOPS 1,457k
                 IOPS 1,475k
                 IOPS 1,463k
		+)----------
		      4,395
	       /3)----------
		      1,465

		(1,465 - 1,447) / 1,447 = 18 / 1,447 = 1%


> R/W stream split makes sense for reasons that we already mentioned.
> Readers don't need to be blocked on Writers - they don't compete for
> the same data.  And, moreover, readers don't need to be serialized at
> all for stateful decompression.
> 
> writers vs readers 3 runs
> 
>     BASE+PE     IOPS 1,271k  avg 5.57us  sd 1,393us  p99.99 27.5us  max 1,581,800us
>                 IOPS 1,461k  avg 3.43us  sd    78us  p99.99 23.9us  max   227,015us
>                 IOPS 1,444k  avg 3.48us  sd   222us  p99.99 23.9us  max   592,628us
>     PATCHED+PE  IOPS 1,457k  avg 3.32us  sd    18us  p99.99 27.5us  max     5,620us
>                 IOPS 1,464k  avg 3.31us  sd    17us  p99.99 24.2us  max     5,621us
>                 IOPS 1,475k  avg 3.29us  sd    16us  p99.99 26.5us  max     5,620us
> 
     BASE+PE     IOPS 1,271k
                 IOPS 1,461k
                 IOPS 1,444k
		+)----------
		      4,176
	       /3)----------
		      1,392

     PATCHED+PE  IOPS 1,457k
                 IOPS 1,464k
                 IOPS 1,475k
		+)----------
		      4,396
	       /3)----------
		      1,465

		(1,465 - 1,392) / 1,392 = 73 / 1,392 = 5%


	improvment	within 5%	more than 10%	more than 20%
	(mIngo's) rule	safely ignored	good		standing ovation

More than two miles away from standing ovation.

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Brian Geffon]

On Tue, Oct 20, 2026 at 10:43 PM Sergey Senozhatsky
<senozhatsky@chromium.org> wrote:
>
> Split struct zcomp_strm into separate compression (Write zcomp_cstrm)
> and decompression (Read zcomp_dstrm) streams, and split zcomp_ops
> context creation/destruction into create_cctx/destroy_cctx and
> optional create_dctx/destroy_dctx callbacks.
>
> This decouples Read (decompression) and Write (compression) paths so
> they no longer contend on the same per-CPU stream mutex (Writer vs
> Reader priority inversion).  Currently, a (low priority) Writer that
> holds the stream mutex can be preempted by a (higher priority) Reader
> on the same CPU, which then blocks on that mutex until the Writer gets
> to run again and releases it, which can take a long time when there
> are other runnable tasks on that CPU.  The good news is that Read and
> Write contexts are completely independent - compression uses its own
> scratch buffer that decompression never touches, and decompression
> uses its own local buffer (for spanning zsmalloc objects) that
> compression never touches, so the Read/Write context split doesn't
> noticeably increase memory usage.  In addition many S/W backends have
> stateless decompression contexts.
>
> A quick synthetic fio test for Writer vs Reader inversion on 24 CPUs
> (preempt=full, zstd, 48 low priority (nice 19) writers, 24 normal
> priority (nice 0) CPU hogs and 24 high priority (nice -19) readers):
>
>   Metric           Base               Patched
>   ----------------------------------------------------------
>   Read IOPS        207k               1,150k        (+5.6x)
>   Read clat avg    101.50 us          7.76 us       (-13.1x)
>   Read clat stdev  5,615.90 us        30.78 us      (-182x)
>   Read clat p99.99 258,998 us         111.10 us     (-2331x)
>   Read clat max    1,342,600 us       11,217 us     (-120x)

These results are awesome and thanks for exploring other options. I
agree this sounds like a great path forward.

>
> Suggested-by: Barry Song <baohua@kernel.org>
> Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>

Acked-by: Brian Geffon <bgeffon@google.com>

> ---
>  drivers/block/zram/backend_842.c     |   4 +-
>  drivers/block/zram/backend_deflate.c |  96 ++++++++++-------
>  drivers/block/zram/backend_lz4.c     |   7 +-
>  drivers/block/zram/backend_lz4hc.c   |   4 +-
>  drivers/block/zram/backend_lzo.c     |   4 +-
>  drivers/block/zram/backend_lzorle.c  |   4 +-
>  drivers/block/zram/backend_zstd.c    | 119 +++++++++++++---------
>  drivers/block/zram/zcomp.c           | 147 +++++++++++++++++++++------
>  drivers/block/zram/zcomp.h           |  28 +++--
>  drivers/block/zram/zram_drv.c        |  38 +++----
>  10 files changed, 298 insertions(+), 153 deletions(-)
>
> diff --git a/drivers/block/zram/backend_842.c b/drivers/block/zram/backend_842.c
> index 56ffda201727..f911cd1ca72e 100644
> --- a/drivers/block/zram/backend_842.c
> +++ b/drivers/block/zram/backend_842.c
> @@ -63,8 +63,8 @@ static int decompress_842(struct zcomp_params *params, struct zcomp_ctx *ctx,
>  const struct zcomp_ops backend_842 = {
>         .compress       = compress_842,
>         .decompress     = decompress_842,
> -       .create_ctx     = create_842,
> -       .destroy_ctx    = destroy_842,
> +       .create_cctx    = create_842,
> +       .destroy_cctx   = destroy_842,
>         .setup_params   = setup_params_842,
>         .release_params = release_params_842,
>         .name           = "842",
> diff --git a/drivers/block/zram/backend_deflate.c b/drivers/block/zram/backend_deflate.c
> index 20685aca9bdc..5ffdaf80d8cf 100644
> --- a/drivers/block/zram/backend_deflate.c
> +++ b/drivers/block/zram/backend_deflate.c
> @@ -13,11 +13,6 @@
>  #define DEFLATE_DEF_WINBITS            (-11)
>  #define DEFLATE_DEF_MEMLEVEL           MAX_MEM_LEVEL
>
> -struct deflate_ctx {
> -       struct z_stream_s cctx;
> -       struct z_stream_s dctx;
> -};
> -
>  static void deflate_release_params(struct zcomp_params *params)
>  {
>  }
> @@ -51,70 +46,98 @@ static int deflate_setup_params(struct zcomp_params *params)
>         return 0;
>  }
>
> -static void deflate_destroy(struct zcomp_ctx *ctx)
> +static void deflate_destroy_cctx(struct zcomp_ctx *ctx)
>  {
> -       struct deflate_ctx *zctx = ctx->context;
> +       struct z_stream_s *cctx = ctx->context;
>
> -       if (!zctx)
> +       if (!cctx)
>                 return;
>
> -       if (zctx->cctx.workspace) {
> -               zlib_deflateEnd(&zctx->cctx);
> -               vfree(zctx->cctx.workspace);
> -       }
> -       if (zctx->dctx.workspace) {
> -               zlib_inflateEnd(&zctx->dctx);
> -               vfree(zctx->dctx.workspace);
> +       if (cctx->workspace) {
> +               zlib_deflateEnd(cctx);
> +               vfree(cctx->workspace);
>         }
> -       kfree(zctx);
> +       kfree(cctx);
>  }
>
> -static int deflate_create(struct zcomp_params *params, struct zcomp_ctx *ctx)
> +static int deflate_create_cctx(struct zcomp_params *params,
> +                              struct zcomp_ctx *ctx)
>  {
> -       struct deflate_ctx *zctx;
> +       struct z_stream_s *cctx;
>         size_t sz;
>         int ret;
>
> -       zctx = kzalloc_obj(*zctx);
> -       if (!zctx)
> +       cctx = kzalloc_obj(*cctx);
> +       if (!cctx)
>                 return -ENOMEM;
>
> -       ctx->context = zctx;
> +       ctx->context = cctx;
>         sz = zlib_deflate_workspacesize(params->deflate.winbits, MAX_MEM_LEVEL);
> -       zctx->cctx.workspace = vzalloc(sz);
> -       if (!zctx->cctx.workspace)
> +       cctx->workspace = vzalloc(sz);
> +       if (!cctx->workspace)
>                 goto error;
>
> -       ret = zlib_deflateInit2(&zctx->cctx, params->level, Z_DEFLATED,
> +       ret = zlib_deflateInit2(cctx, params->level, Z_DEFLATED,
>                                 params->deflate.winbits, DEFLATE_DEF_MEMLEVEL,
>                                 Z_DEFAULT_STRATEGY);
>         if (ret != Z_OK)
>                 goto error;
>
> +       return 0;
> +
> +error:
> +       deflate_destroy_cctx(ctx);
> +       return -EINVAL;
> +}
> +
> +static void deflate_destroy_dctx(struct zcomp_ctx *ctx)
> +{
> +       struct z_stream_s *dctx = ctx->context;
> +
> +       if (!dctx)
> +               return;
> +
> +       if (dctx->workspace) {
> +               zlib_inflateEnd(dctx);
> +               vfree(dctx->workspace);
> +       }
> +       kfree(dctx);
> +}
> +
> +static int deflate_create_dctx(struct zcomp_params *params,
> +                              struct zcomp_ctx *ctx)
> +{
> +       struct z_stream_s *dctx;
> +       size_t sz;
> +       int ret;
> +
> +       dctx = kzalloc_obj(*dctx);
> +       if (!dctx)
> +               return -ENOMEM;
> +
> +       ctx->context = dctx;
>         sz = zlib_inflate_workspacesize();
> -       zctx->dctx.workspace = vzalloc(sz);
> -       if (!zctx->dctx.workspace)
> +       dctx->workspace = vzalloc(sz);
> +       if (!dctx->workspace)
>                 goto error;
>
> -       ret = zlib_inflateInit2(&zctx->dctx, params->deflate.winbits);
> +       ret = zlib_inflateInit2(dctx, params->deflate.winbits);
>         if (ret != Z_OK)
>                 goto error;
>
>         return 0;
>
>  error:
> -       deflate_destroy(ctx);
> +       deflate_destroy_dctx(ctx);
>         return -EINVAL;
>  }
>
>  static int deflate_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
>                             struct zcomp_req *req)
>  {
> -       struct deflate_ctx *zctx = ctx->context;
> -       struct z_stream_s *deflate;
> +       struct z_stream_s *deflate = ctx->context;
>         int ret;
>
> -       deflate = &zctx->cctx;
>         ret = zlib_deflateReset(deflate);
>         if (ret != Z_OK)
>                 return -EINVAL;
> @@ -136,12 +159,9 @@ static int deflate_decompress(struct zcomp_params *params,
>                               struct zcomp_ctx *ctx,
>                               struct zcomp_req *req)
>  {
> -       struct deflate_ctx *zctx = ctx->context;
> -       struct z_stream_s *inflate;
> +       struct z_stream_s *inflate = ctx->context;
>         int ret;
>
> -       inflate = &zctx->dctx;
> -
>         ret = zlib_inflateReset(inflate);
>         if (ret != Z_OK)
>                 return -EINVAL;
> @@ -161,8 +181,10 @@ static int deflate_decompress(struct zcomp_params *params,
>  const struct zcomp_ops backend_deflate = {
>         .compress       = deflate_compress,
>         .decompress     = deflate_decompress,
> -       .create_ctx     = deflate_create,
> -       .destroy_ctx    = deflate_destroy,
> +       .create_cctx    = deflate_create_cctx,
> +       .destroy_cctx   = deflate_destroy_cctx,
> +       .create_dctx    = deflate_create_dctx,
> +       .destroy_dctx   = deflate_destroy_dctx,
>         .setup_params   = deflate_setup_params,
>         .release_params = deflate_release_params,
>         .name           = "deflate",
> diff --git a/drivers/block/zram/backend_lz4.c b/drivers/block/zram/backend_lz4.c
> index 5f2e27805d75..9767482a2fea 100644
> --- a/drivers/block/zram/backend_lz4.c
> +++ b/drivers/block/zram/backend_lz4.c
> @@ -38,8 +38,7 @@ static int lz4_setup_params(struct zcomp_params *params)
>         if (!dict_stream)
>                 return -ENOMEM;
>
> -       ret = LZ4_loadDict(dict_stream,
> -                          params->dict, params->dict_sz);
> +       ret = LZ4_loadDict(dict_stream, params->dict, params->dict_sz);
>         if (ret != params->dict_sz) {
>                 vfree(dict_stream);
>                 return -EINVAL;
> @@ -102,8 +101,8 @@ static int lz4_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
>  const struct zcomp_ops backend_lz4 = {
>         .compress       = lz4_compress,
>         .decompress     = lz4_decompress,
> -       .create_ctx     = lz4_create,
> -       .destroy_ctx    = lz4_destroy,
> +       .create_cctx    = lz4_create,
> +       .destroy_cctx   = lz4_destroy,
>         .setup_params   = lz4_setup_params,
>         .release_params = lz4_release_params,
>         .name           = "lz4",
> diff --git a/drivers/block/zram/backend_lz4hc.c b/drivers/block/zram/backend_lz4hc.c
> index b62736b26622..2d1364a29b27 100644
> --- a/drivers/block/zram/backend_lz4hc.c
> +++ b/drivers/block/zram/backend_lz4hc.c
> @@ -107,8 +107,8 @@ static int lz4hc_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
>  const struct zcomp_ops backend_lz4hc = {
>         .compress       = lz4hc_compress,
>         .decompress     = lz4hc_decompress,
> -       .create_ctx     = lz4hc_create,
> -       .destroy_ctx    = lz4hc_destroy,
> +       .create_cctx    = lz4hc_create,
> +       .destroy_cctx   = lz4hc_destroy,
>         .setup_params   = lz4hc_setup_params,
>         .release_params = lz4hc_release_params,
>         .name           = "lz4hc",
> diff --git a/drivers/block/zram/backend_lzo.c b/drivers/block/zram/backend_lzo.c
> index 78538479c511..e703560d9ecd 100644
> --- a/drivers/block/zram/backend_lzo.c
> +++ b/drivers/block/zram/backend_lzo.c
> @@ -61,8 +61,8 @@ static int lzo_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
>  const struct zcomp_ops backend_lzo = {
>         .compress       = lzo_compress,
>         .decompress     = lzo_decompress,
> -       .create_ctx     = lzo_create,
> -       .destroy_ctx    = lzo_destroy,
> +       .create_cctx    = lzo_create,
> +       .destroy_cctx   = lzo_destroy,
>         .setup_params   = lzo_setup_params,
>         .release_params = lzo_release_params,
>         .name           = "lzo",
> diff --git a/drivers/block/zram/backend_lzorle.c b/drivers/block/zram/backend_lzorle.c
> index 011787f1430b..e95026e99202 100644
> --- a/drivers/block/zram/backend_lzorle.c
> +++ b/drivers/block/zram/backend_lzorle.c
> @@ -61,8 +61,8 @@ static int lzorle_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
>  const struct zcomp_ops backend_lzorle = {
>         .compress       = lzorle_compress,
>         .decompress     = lzorle_decompress,
> -       .create_ctx     = lzorle_create,
> -       .destroy_ctx    = lzorle_destroy,
> +       .create_cctx    = lzorle_create,
> +       .destroy_cctx   = lzorle_destroy,
>         .setup_params   = lzorle_setup_params,
>         .release_params = lzorle_release_params,
>         .name           = "lzo-rle",
> diff --git a/drivers/block/zram/backend_zstd.c b/drivers/block/zram/backend_zstd.c
> index 7a8b5ff186f2..d55615f34300 100644
> --- a/drivers/block/zram/backend_zstd.c
> +++ b/drivers/block/zram/backend_zstd.c
> @@ -10,10 +10,8 @@
>  #include "backend_zstd.h"
>
>  struct zstd_ctx {
> -       zstd_cctx *cctx;
> -       zstd_dctx *dctx;
> -       void *cctx_mem;
> -       void *dctx_mem;
> +       void *ctx;
> +       void *mem;
>  };
>
>  struct zstd_params {
> @@ -105,7 +103,7 @@ static int zstd_setup_params(struct zcomp_params *params)
>         return -EINVAL;
>  }
>
> -static void zstd_destroy(struct zcomp_ctx *ctx)
> +static void zstd_destroy_cctx(struct zcomp_ctx *ctx)
>  {
>         struct zstd_ctx *zctx = ctx->context;
>
> @@ -113,30 +111,23 @@ static void zstd_destroy(struct zcomp_ctx *ctx)
>                 return;
>
>         /*
> -        * If ->cctx_mem and ->dctx_mem were allocated then we didn't use
> -        * C/D dictionary and ->cctx / ->dctx were "embedded" into these
> -        * buffers.
> +        * If ->mem was allocated then we didn't use C/D dictionary and
> +        * ->ctx was "embedded" into this buffer.
>          *
> -        * If otherwise then we need to explicitly release ->cctx / ->dctx.
> +        * If otherwise then we need to explicitly release ->ctx.
>          */
> -       if (zctx->cctx_mem)
> -               vfree(zctx->cctx_mem);
> +       if (zctx->mem)
> +               vfree(zctx->mem);
>         else
> -               zstd_free_cctx(zctx->cctx);
> -
> -       if (zctx->dctx_mem)
> -               vfree(zctx->dctx_mem);
> -       else
> -               zstd_free_dctx(zctx->dctx);
> +               zstd_free_cctx(zctx->ctx);
>
>         kfree(zctx);
>  }
>
> -static int zstd_create(struct zcomp_params *params, struct zcomp_ctx *ctx)
> +static int zstd_create_cctx(struct zcomp_params *params, struct zcomp_ctx *ctx)
>  {
> +       struct zstd_params *zp = params->drv_data;
>         struct zstd_ctx *zctx;
> -       zstd_parameters prm;
> -       size_t sz;
>
>         zctx = kzalloc_obj(*zctx);
>         if (!zctx)
> @@ -144,40 +135,74 @@ static int zstd_create(struct zcomp_params *params, struct zcomp_ctx *ctx)
>
>         ctx->context = zctx;
>         if (params->dict_sz == 0) {
> -               prm = zstd_get_params(params->level, PAGE_SIZE);
> -               sz = zstd_cctx_workspace_bound(&prm.cParams);
> -               zctx->cctx_mem = vzalloc(sz);
> -               if (!zctx->cctx_mem)
> -                       goto error;
> -
> -               zctx->cctx = zstd_init_cctx(zctx->cctx_mem, sz);
> -               if (!zctx->cctx)
> -                       goto error;
> +               zstd_parameters prm = zp->cprm;
> +               size_t sz = zstd_cctx_workspace_bound(&prm.cParams);
>
> -               sz = zstd_dctx_workspace_bound();
> -               zctx->dctx_mem = vzalloc(sz);
> -               if (!zctx->dctx_mem)
> +               zctx->mem = vzalloc(sz);
> +               if (!zctx->mem)
>                         goto error;
>
> -               zctx->dctx = zstd_init_dctx(zctx->dctx_mem, sz);
> -               if (!zctx->dctx)
> +               zctx->ctx = zstd_init_cctx(zctx->mem, sz);
> +               if (!zctx->ctx)
>                         goto error;
>         } else {
> -               struct zstd_params *zp = params->drv_data;
> +               zctx->ctx = zstd_create_cctx_advanced(zp->custom_mem);
> +               if (!zctx->ctx)
> +                       goto error;
> +       }
> +
> +       return 0;
> +
> +error:
> +       zstd_destroy_cctx(ctx);
> +       return -EINVAL;
> +}
> +
> +static void zstd_destroy_dctx(struct zcomp_ctx *ctx)
> +{
> +       struct zstd_ctx *zctx = ctx->context;
> +
> +       if (!zctx)
> +               return;
> +
> +       if (zctx->mem)
> +               vfree(zctx->mem);
> +       else
> +               zstd_free_dctx(zctx->ctx);
> +
> +       kfree(zctx);
> +}
> +
> +static int zstd_create_dctx(struct zcomp_params *params, struct zcomp_ctx *ctx)
> +{
> +       struct zstd_params *zp = params->drv_data;
> +       struct zstd_ctx *zctx;
>
> -               zctx->cctx = zstd_create_cctx_advanced(zp->custom_mem);
> -               if (!zctx->cctx)
> +       zctx = kzalloc_obj(*zctx);
> +       if (!zctx)
> +               return -ENOMEM;
> +
> +       ctx->context = zctx;
> +       if (params->dict_sz == 0) {
> +               size_t sz = zstd_dctx_workspace_bound();
> +
> +               zctx->mem = vzalloc(sz);
> +               if (!zctx->mem)
>                         goto error;
>
> -               zctx->dctx = zstd_create_dctx_advanced(zp->custom_mem);
> -               if (!zctx->dctx)
> +               zctx->ctx = zstd_init_dctx(zctx->mem, sz);
> +               if (!zctx->ctx)
> +                       goto error;
> +       } else {
> +               zctx->ctx = zstd_create_dctx_advanced(zp->custom_mem);
> +               if (!zctx->ctx)
>                         goto error;
>         }
>
>         return 0;
>
>  error:
> -       zstd_destroy(ctx);
> +       zstd_destroy_dctx(ctx);
>         return -EINVAL;
>  }
>
> @@ -189,10 +214,10 @@ static int zstd_compress(struct zcomp_params *params, struct zcomp_ctx *ctx,
>         size_t ret;
>
>         if (params->dict_sz == 0)
> -               ret = zstd_compress_cctx(zctx->cctx, req->dst, req->dst_len,
> +               ret = zstd_compress_cctx(zctx->ctx, req->dst, req->dst_len,
>                                          req->src, req->src_len, &zp->cprm);
>         else
> -               ret = zstd_compress_using_cdict(zctx->cctx, req->dst,
> +               ret = zstd_compress_using_cdict(zctx->ctx, req->dst,
>                                                 req->dst_len, req->src,
>                                                 req->src_len,
>                                                 zp->cdict);
> @@ -210,10 +235,10 @@ static int zstd_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
>         size_t ret;
>
>         if (params->dict_sz == 0)
> -               ret = zstd_decompress_dctx(zctx->dctx, req->dst, req->dst_len,
> +               ret = zstd_decompress_dctx(zctx->ctx, req->dst, req->dst_len,
>                                            req->src, req->src_len);
>         else
> -               ret = zstd_decompress_using_ddict(zctx->dctx, req->dst,
> +               ret = zstd_decompress_using_ddict(zctx->ctx, req->dst,
>                                                   req->dst_len, req->src,
>                                                   req->src_len, zp->ddict);
>         if (zstd_is_error(ret))
> @@ -224,8 +249,10 @@ static int zstd_decompress(struct zcomp_params *params, struct zcomp_ctx *ctx,
>  const struct zcomp_ops backend_zstd = {
>         .compress       = zstd_compress,
>         .decompress     = zstd_decompress,
> -       .create_ctx     = zstd_create,
> -       .destroy_ctx    = zstd_destroy,
> +       .create_cctx    = zstd_create_cctx,
> +       .destroy_cctx   = zstd_destroy_cctx,
> +       .create_dctx    = zstd_create_dctx,
> +       .destroy_dctx   = zstd_destroy_dctx,
>         .setup_params   = zstd_setup_params,
>         .release_params = zstd_release_params,
>         .name           = "zstd",
> diff --git a/drivers/block/zram/zcomp.c b/drivers/block/zram/zcomp.c
> index 028e2f0f587e..1c9f6d5c8cd1 100644
> --- a/drivers/block/zram/zcomp.c
> +++ b/drivers/block/zram/zcomp.c
> @@ -45,30 +45,65 @@ static const struct zcomp_ops *backends[] = {
>         NULL
>  };
>
> -static void zcomp_strm_free(struct zcomp *comp, struct zcomp_strm *zstrm)
> +static void zcomp_cstrm_free(struct zcomp *comp, struct zcomp_cstrm *zstrm)
>  {
> -       comp->ops->destroy_ctx(&zstrm->ctx);
> -       vfree(zstrm->local_copy);
> +       if (comp->ops->destroy_cctx)
> +               comp->ops->destroy_cctx(&zstrm->ctx);
>         vfree(zstrm->buffer);
>         zstrm->buffer = NULL;
> +       zstrm->ctx.context = NULL;
>  }
>
> -static int zcomp_strm_init(struct zcomp *comp, struct zcomp_strm *zstrm)
> +static int zcomp_cstrm_init(struct zcomp *comp, struct zcomp_cstrm *zstrm)
>  {
>         int ret;
>
> -       ret = comp->ops->create_ctx(comp->params, &zstrm->ctx);
> -       if (ret)
> -               return ret;
> +       if (comp->ops->create_cctx) {
> +               ret = comp->ops->create_cctx(comp->params, &zstrm->ctx);
> +               if (ret) {
> +                       /* backend has released partially created context */
> +                       zstrm->ctx.context = NULL;
> +                       return ret;
> +               }
> +       }
>
> -       zstrm->local_copy = vzalloc(PAGE_SIZE);
>         /*
>          * allocate 2 pages. 1 for compressed data, plus 1 extra for the
>          * case when compressed size is larger than the original one
>          */
>         zstrm->buffer = vzalloc(2 * PAGE_SIZE);
> -       if (!zstrm->buffer || !zstrm->local_copy) {
> -               zcomp_strm_free(comp, zstrm);
> +       if (!zstrm->buffer) {
> +               zcomp_cstrm_free(comp, zstrm);
> +               return -ENOMEM;
> +       }
> +       return 0;
> +}
> +
> +static void zcomp_dstrm_free(struct zcomp *comp, struct zcomp_dstrm *zstrm)
> +{
> +       if (comp->ops->destroy_dctx)
> +               comp->ops->destroy_dctx(&zstrm->ctx);
> +       vfree(zstrm->local_copy);
> +       zstrm->local_copy = NULL;
> +       zstrm->ctx.context = NULL;
> +}
> +
> +static int zcomp_dstrm_init(struct zcomp *comp, struct zcomp_dstrm *zstrm)
> +{
> +       int ret;
> +
> +       if (comp->ops->create_dctx) {
> +               ret = comp->ops->create_dctx(comp->params, &zstrm->ctx);
> +               if (ret) {
> +                       /* backend has released partially created context */
> +                       zstrm->ctx.context = NULL;
> +                       return ret;
> +               }
> +       }
> +
> +       zstrm->local_copy = vzalloc(PAGE_SIZE);
> +       if (!zstrm->local_copy) {
> +               zcomp_dstrm_free(comp, zstrm);
>                 return -ENOMEM;
>         }
>         return 0;
> @@ -114,10 +149,10 @@ ssize_t zcomp_available_show(const char *comp, char *buf, ssize_t at)
>         return at;
>  }
>
> -struct zcomp_strm *zcomp_stream_get(struct zcomp *comp)
> +struct zcomp_cstrm *zcomp_cstrm_get(struct zcomp *comp)
>  {
>         for (;;) {
> -               struct zcomp_strm *zstrm = raw_cpu_ptr(comp->stream);
> +               struct zcomp_cstrm *zstrm = raw_cpu_ptr(comp->cstream);
>
>                 /*
>                  * Inspired by zswap
> @@ -136,12 +171,29 @@ struct zcomp_strm *zcomp_stream_get(struct zcomp *comp)
>         }
>  }
>
> -void zcomp_stream_put(struct zcomp_strm *zstrm)
> +void zcomp_cstrm_put(struct zcomp_cstrm *zstrm)
> +{
> +       mutex_unlock(&zstrm->lock);
> +}
> +
> +struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp)
> +{
> +       for (;;) {
> +               struct zcomp_dstrm *zstrm = raw_cpu_ptr(comp->dstream);
> +
> +               mutex_lock(&zstrm->lock);
> +               if (likely(zstrm->local_copy))
> +                       return zstrm;
> +               mutex_unlock(&zstrm->lock);
> +       }
> +}
> +
> +void zcomp_dstrm_put(struct zcomp_dstrm *zstrm)
>  {
>         mutex_unlock(&zstrm->lock);
>  }
>
> -int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
> +int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm,
>                    const void *src, unsigned int *dst_len)
>  {
>         struct zcomp_req req = {
> @@ -159,7 +211,7 @@ int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
>         return ret;
>  }
>
> -int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,
> +int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm,
>                      struct scatterlist *sg, unsigned int src_len, void *dst)
>  {
>         struct zcomp_req req = {
> @@ -191,23 +243,44 @@ int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,
>  int zcomp_cpu_up_prepare(unsigned int cpu, struct hlist_node *node)
>  {
>         struct zcomp *comp = hlist_entry(node, struct zcomp, node);
> -       struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu);
> +       struct zcomp_cstrm *cstrm;
> +       struct zcomp_dstrm *dstrm;
>         int ret;
>
> -       ret = zcomp_strm_init(comp, zstrm);
> -       if (ret)
> +       cstrm = per_cpu_ptr(comp->cstream, cpu);
> +       ret = zcomp_cstrm_init(comp, cstrm);
> +       if (ret) {
>                 pr_err("Can't allocate a compression stream\n");
> -       return ret;
> +               return ret;
> +       }
> +
> +       dstrm = per_cpu_ptr(comp->dstream, cpu);
> +       ret = zcomp_dstrm_init(comp, dstrm);
> +       if (ret) {
> +               pr_err("Can't allocate a decompression stream\n");
> +               zcomp_cstrm_free(comp, cstrm);
> +               return ret;
> +       }
> +
> +       return 0;
>  }
>
>  int zcomp_cpu_dead(unsigned int cpu, struct hlist_node *node)
>  {
>         struct zcomp *comp = hlist_entry(node, struct zcomp, node);
> -       struct zcomp_strm *zstrm = per_cpu_ptr(comp->stream, cpu);
> +       struct zcomp_cstrm *cstrm;
> +       struct zcomp_dstrm *dstrm;
> +
> +       cstrm = per_cpu_ptr(comp->cstream, cpu);
> +       mutex_lock(&cstrm->lock);
> +       zcomp_cstrm_free(comp, cstrm);
> +       mutex_unlock(&cstrm->lock);
> +
> +       dstrm = per_cpu_ptr(comp->dstream, cpu);
> +       mutex_lock(&dstrm->lock);
> +       zcomp_dstrm_free(comp, dstrm);
> +       mutex_unlock(&dstrm->lock);
>
> -       mutex_lock(&zstrm->lock);
> -       zcomp_strm_free(comp, zstrm);
> -       mutex_unlock(&zstrm->lock);
>         return 0;
>  }
>
> @@ -215,17 +288,27 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params)
>  {
>         int ret, cpu;
>
> -       comp->stream = alloc_percpu(struct zcomp_strm);
> -       if (!comp->stream)
> -               return -ENOMEM;
> -
>         comp->params = params;
>         ret = comp->ops->setup_params(comp->params);
>         if (ret)
>                 goto cleanup;
>
> -       for_each_possible_cpu(cpu)
> -               mutex_init(&per_cpu_ptr(comp->stream, cpu)->lock);
> +       comp->cstream = alloc_percpu(struct zcomp_cstrm);
> +       if (!comp->cstream) {
> +               ret = -ENOMEM;
> +               goto cleanup;
> +       }
> +
> +       comp->dstream = alloc_percpu(struct zcomp_dstrm);
> +       if (!comp->dstream) {
> +               ret = -ENOMEM;
> +               goto cleanup;
> +       }
> +
> +       for_each_possible_cpu(cpu) {
> +               mutex_init(&per_cpu_ptr(comp->cstream, cpu)->lock);
> +               mutex_init(&per_cpu_ptr(comp->dstream, cpu)->lock);
> +       }
>
>         ret = cpuhp_state_add_instance(CPUHP_ZCOMP_PREPARE, &comp->node);
>         if (ret < 0)
> @@ -235,7 +318,8 @@ static int zcomp_init(struct zcomp *comp, struct zcomp_params *params)
>
>  cleanup:
>         comp->ops->release_params(comp->params);
> -       free_percpu(comp->stream);
> +       free_percpu(comp->dstream);
> +       free_percpu(comp->cstream);
>         return ret;
>  }
>
> @@ -243,7 +327,8 @@ void zcomp_destroy(struct zcomp *comp)
>  {
>         cpuhp_state_remove_instance(CPUHP_ZCOMP_PREPARE, &comp->node);
>         comp->ops->release_params(comp->params);
> -       free_percpu(comp->stream);
> +       free_percpu(comp->dstream);
> +       free_percpu(comp->cstream);
>         kfree(comp);
>  }
>
> diff --git a/drivers/block/zram/zcomp.h b/drivers/block/zram/zcomp.h
> index f38fd31f9e4e..22be50dc51ba 100644
> --- a/drivers/block/zram/zcomp.h
> +++ b/drivers/block/zram/zcomp.h
> @@ -39,10 +39,15 @@ struct zcomp_ctx {
>         void *context;
>  };
>
> -struct zcomp_strm {
> +struct zcomp_cstrm {
>         struct mutex lock;
>         /* compression buffer */
>         void *buffer;
> +       struct zcomp_ctx ctx;
> +};
> +
> +struct zcomp_dstrm {
> +       struct mutex lock;
>         /* local copy of handle memory */
>         void *local_copy;
>         struct zcomp_ctx ctx;
> @@ -62,8 +67,11 @@ struct zcomp_ops {
>         int (*decompress)(struct zcomp_params *params, struct zcomp_ctx *ctx,
>                           struct zcomp_req *req);
>
> -       int (*create_ctx)(struct zcomp_params *params, struct zcomp_ctx *ctx);
> -       void (*destroy_ctx)(struct zcomp_ctx *ctx);
> +       int (*create_cctx)(struct zcomp_params *params, struct zcomp_ctx *ctx);
> +       void (*destroy_cctx)(struct zcomp_ctx *ctx);
> +
> +       int (*create_dctx)(struct zcomp_params *params, struct zcomp_ctx *ctx);
> +       void (*destroy_dctx)(struct zcomp_ctx *ctx);
>
>         int (*setup_params)(struct zcomp_params *params);
>         void (*release_params)(struct zcomp_params *params);
> @@ -73,7 +81,8 @@ struct zcomp_ops {
>
>  /* dynamic per-device compression frontend */
>  struct zcomp {
> -       struct zcomp_strm __percpu *stream;
> +       struct zcomp_cstrm __percpu *cstream;
> +       struct zcomp_dstrm __percpu *dstream;
>         const struct zcomp_ops *ops;
>         struct zcomp_params *params;
>         struct hlist_node node;
> @@ -87,12 +96,15 @@ const char *zcomp_lookup_backend_name(const char *comp);
>  struct zcomp *zcomp_create(const char *alg, struct zcomp_params *params);
>  void zcomp_destroy(struct zcomp *comp);
>
> -struct zcomp_strm *zcomp_stream_get(struct zcomp *comp);
> -void zcomp_stream_put(struct zcomp_strm *zstrm);
> +struct zcomp_cstrm *zcomp_cstrm_get(struct zcomp *comp);
> +void zcomp_cstrm_put(struct zcomp_cstrm *zstrm);
> +
> +struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp);
> +void zcomp_dstrm_put(struct zcomp_dstrm *zstrm);
>
> -int zcomp_compress(struct zcomp *comp, struct zcomp_strm *zstrm,
> +int zcomp_compress(struct zcomp *comp, struct zcomp_cstrm *zstrm,
>                    const void *src, unsigned int *dst_len);
> -int zcomp_decompress(struct zcomp *comp, struct zcomp_strm *zstrm,
> +int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm,
>                      struct scatterlist *sg, unsigned int src_len, void *dst);
>
>  #endif /* _ZCOMP_H_ */
> diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c
> index 0e1aa0da11aa..def5266679b0 100644
> --- a/drivers/block/zram/zram_drv.c
> +++ b/drivers/block/zram/zram_drv.c
> @@ -1348,7 +1348,7 @@ static ssize_t writeback_store(struct device *dev,
>  static int decompress_bdev_page(struct zram *zram, struct page *page,
>                                 unsigned long index)
>  {
> -       struct zcomp_strm *zstrm;
> +       struct zcomp_dstrm *zstrm;
>         struct scatterlist sg[1];
>         unsigned int size;
>         int ret, prio;
> @@ -1373,7 +1373,7 @@ static int decompress_bdev_page(struct zram *zram, struct page *page,
>         sg_init_table(sg, 1);
>         sg_set_page(sg, page, size, 0);
>
> -       zstrm = zcomp_stream_get(zram->comps[prio]);
> +       zstrm = zcomp_dstrm_get(zram->comps[prio]);
>         ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size,
>                                zstrm->local_copy);
>         if (!ret) {
> @@ -1382,7 +1382,7 @@ static int decompress_bdev_page(struct zram *zram, struct page *page,
>                 copy_page(dst, zstrm->local_copy);
>                 kunmap_local(dst);
>         }
> -       zcomp_stream_put(zstrm);
> +       zcomp_dstrm_put(zstrm);
>         slot_unlock(zram, index);
>
>         return ret;
> @@ -2115,7 +2115,7 @@ static int read_incompressible_page(struct zram *zram, struct page *page,
>  static int read_compressed_page(struct zram *zram, struct page *page,
>                                 unsigned long index)
>  {
> -       struct zcomp_strm *zstrm;
> +       struct zcomp_dstrm *zstrm;
>         struct scatterlist sg[2];
>         unsigned long handle;
>         unsigned int size;
> @@ -2126,13 +2126,13 @@ static int read_compressed_page(struct zram *zram, struct page *page,
>         size = get_slot_size(zram, index);
>         prio = get_slot_comp_priority(zram, index);
>
> -       zstrm = zcomp_stream_get(zram->comps[prio]);
> +       zstrm = zcomp_dstrm_get(zram->comps[prio]);
>         zs_obj_read_sg_begin(zram->mem_pool, handle, sg, size);
>         dst = kmap_local_page(page);
>         ret = zcomp_decompress(zram->comps[prio], zstrm, sg, size, dst);
>         kunmap_local(dst);
>         zs_obj_read_sg_end(zram->mem_pool, handle);
> -       zcomp_stream_put(zstrm);
> +       zcomp_dstrm_put(zstrm);
>
>         return ret;
>  }
> @@ -2300,7 +2300,7 @@ static int zram_write_page(struct zram *zram, struct page *page,
>         unsigned long handle;
>         unsigned int comp_len;
>         void *mem;
> -       struct zcomp_strm *zstrm;
> +       struct zcomp_cstrm *zstrm;
>         unsigned long element;
>         bool same_filled;
>
> @@ -2310,20 +2310,20 @@ static int zram_write_page(struct zram *zram, struct page *page,
>         if (same_filled)
>                 return write_same_filled_page(zram, element, index);
>
> -       zstrm = zcomp_stream_get(zram->comps[ZRAM_PRIMARY_COMP]);
> +       zstrm = zcomp_cstrm_get(zram->comps[ZRAM_PRIMARY_COMP]);
>         mem = kmap_local_page(page);
>         ret = zcomp_compress(zram->comps[ZRAM_PRIMARY_COMP], zstrm,
>                              mem, &comp_len);
>         kunmap_local(mem);
>
>         if (unlikely(ret)) {
> -               zcomp_stream_put(zstrm);
> +               zcomp_cstrm_put(zstrm);
>                 pr_err("Compression failed! err=%d\n", ret);
>                 return ret;
>         }
>
>         if (comp_len >= huge_class_size) {
> -               zcomp_stream_put(zstrm);
> +               zcomp_cstrm_put(zstrm);
>                 return write_incompressible_page(zram, page, index);
>         }
>
> @@ -2331,18 +2331,18 @@ static int zram_write_page(struct zram *zram, struct page *page,
>                            GFP_NOIO | __GFP_NOWARN |
>                            __GFP_HIGHMEM | __GFP_MOVABLE, page_to_nid(page));
>         if (IS_ERR_VALUE(handle)) {
> -               zcomp_stream_put(zstrm);
> +               zcomp_cstrm_put(zstrm);
>                 return PTR_ERR((void *)handle);
>         }
>
>         if (!zram_can_store_page(zram)) {
> -               zcomp_stream_put(zstrm);
> +               zcomp_cstrm_put(zstrm);
>                 zs_free(zram->mem_pool, handle);
>                 return -ENOMEM;
>         }
>
>         zs_obj_write(zram->mem_pool, handle, zstrm->buffer, comp_len);
> -       zcomp_stream_put(zstrm);
> +       zcomp_cstrm_put(zstrm);
>
>         slot_lock(zram, index);
>         slot_free(zram, index);
> @@ -2451,7 +2451,7 @@ static int recompress_slot(struct zram *zram, unsigned long index,
>                            struct page *page, u64 *num_recomp_pages,
>                            u32 threshold, u32 prio)
>  {
> -       struct zcomp_strm *zstrm = NULL;
> +       struct zcomp_cstrm *zstrm = NULL;
>         unsigned long handle_old;
>         unsigned long handle_new;
>         unsigned int comp_len_old;
> @@ -2483,7 +2483,7 @@ static int recompress_slot(struct zram *zram, unsigned long index,
>          */
>         clear_slot_flag(zram, index, ZRAM_IDLE);
>
> -       zstrm = zcomp_stream_get(zram->comps[prio]);
> +       zstrm = zcomp_cstrm_get(zram->comps[prio]);
>         src = kmap_local_page(page);
>         ret = zcomp_compress(zram->comps[prio], zstrm, src, &comp_len_new);
>         kunmap_local(src);
> @@ -2498,7 +2498,7 @@ static int recompress_slot(struct zram *zram, unsigned long index,
>                 *num_recomp_pages -= 1;
>
>         if (ret) {
> -               zcomp_stream_put(zstrm);
> +               zcomp_cstrm_put(zstrm);
>                 return ret;
>         }
>
> @@ -2507,7 +2507,7 @@ static int recompress_slot(struct zram *zram, unsigned long index,
>
>         if (class_index_new >= class_index_old ||
>             (threshold && comp_len_new >= threshold)) {
> -               zcomp_stream_put(zstrm);
> +               zcomp_cstrm_put(zstrm);
>
>                 /*
>                  * Secondary algorithms failed to re-compress the page
> @@ -2536,12 +2536,12 @@ static int recompress_slot(struct zram *zram, unsigned long index,
>                                __GFP_HIGHMEM | __GFP_MOVABLE,
>                                page_to_nid(page));
>         if (IS_ERR_VALUE(handle_new)) {
> -               zcomp_stream_put(zstrm);
> +               zcomp_cstrm_put(zstrm);
>                 return PTR_ERR((void *)handle_new);
>         }
>
>         zs_obj_write(zram->mem_pool, handle_new, zstrm->buffer, comp_len_new);
> -       zcomp_stream_put(zstrm);
> +       zcomp_cstrm_put(zstrm);
>
>         slot_free(zram, index);
>         set_slot_handle(zram, index, handle_new);
> --
> 2.56.0.212.g129ede20d9-goog
>

Re: [PATCH v6 09/13] zram: split zcomp into separate R/W streams

[Sergey Senozhatsky]

On (26/10/23 19:40), Hillf Danton wrote:
[..]
> > writers vs readers 3 runs
> > 
> >     BASE+PE     IOPS 1,271k  avg 5.57us  sd 1,393us  p99.99 27.5us  max 1,581,800us
> >                 IOPS 1,461k  avg 3.43us  sd    78us  p99.99 23.9us  max   227,015us
> >                 IOPS 1,444k  avg 3.48us  sd   222us  p99.99 23.9us  max   592,628us
> >     PATCHED+PE  IOPS 1,457k  avg 3.32us  sd    18us  p99.99 27.5us  max     5,620us
> >                 IOPS 1,464k  avg 3.31us  sd    17us  p99.99 24.2us  max     5,621us
> >                 IOPS 1,475k  avg 3.29us  sd    16us  p99.99 26.5us  max     5,620us

BASE+PE looked noisy, so I sat down and tested properly (also tested
zstd and various compression levels).

(PE enabled for all tests)

######## zstd compression level 3 W-vs-R
    BASE     IOPS 1,154k  avg 7.84us  sd   444us  max   697,634us
             IOPS 881k  avg 14.16us  sd 2,242us  max 1,387,000us
             IOPS 704k  avg 20.43us  sd 3,499us  max 2,058,300us
             IOPS 563k  avg 29.02us  sd 3,910us  max 1,649,800us
             IOPS 626k  avg 24.79us  sd 4,013us  max 2,085,800us
             IOPS 976k  avg 11.54us  sd 1,518us  max 1,165,800us
             IOPS 1,178k  avg 7.44us  sd    22us  max     3,103us
             IOPS 1,120k  avg 7.39us  sd    21us  max     7,222us
             IOPS 522k  avg 32.51us  sd 4,311us  max 1,598,300us
             IOPS 581k  avg 27.54us  sd 3,626us  max 1,448,600us
             IOPS 1,160k  avg 7.69us  sd   352us  max   702,099us
             IOPS 1,169k  avg 7.52us  sd   129us  max   281,428us
             IOPS 954k  avg 12.06us  sd 1,705us  max 1,545,600us
             IOPS 1,029k  avg 10.24us  sd 1,156us  max   976,065us
             IOPS 1,169k  avg 7.48us  sd    24us  max     4,305us
             IOPS 1,177k  avg 7.45us  sd    53us  max   118,777us
             IOPS 579k  avg 27.43us  sd 3,661us  max 1,615,800us
             IOPS 1,169k  avg 7.52us  sd   115us  max   228,074us
             IOPS 1,110k  avg 8.59us  sd   985us  max 1,288,000us
             IOPS 1,177k  avg 7.45us  sd    29us  max    72,245us
             IOPS 580k  avg 27.74us  sd 3,755us  max 1,507,000us
    PATCHED  IOPS 1,179k  avg 7.40us  sd    23us  max     7,404us
             IOPS 1,179k  avg 7.41us  sd    22us  max     2,857us
             IOPS 1,176k  avg 7.40us  sd    22us  max     6,615us
             IOPS 1,175k  avg 7.44us  sd    23us  max     5,623us
             IOPS 1,176k  avg 7.42us  sd    24us  max     5,625us
             IOPS 1,173k  avg 7.42us  sd    24us  max     5,632us
             IOPS 1,172k  avg 7.41us  sd    23us  max     6,223us
             IOPS 1,173k  avg 7.40us  sd    25us  max     5,848us
             IOPS 1,173k  avg 7.40us  sd    24us  max     7,563us
             IOPS 1,172k  avg 7.42us  sd    24us  max     5,625us
             IOPS 1,175k  avg 7.42us  sd    23us  max     5,626us
             IOPS 1,177k  avg 7.42us  sd    23us  max     4,243us
             IOPS 1,170k  avg 7.46us  sd    25us  max     5,622us
             IOPS 1,178k  avg 7.40us  sd    22us  max     3,213us
             IOPS 1,167k  avg 7.43us  sd    25us  max     7,593us
             IOPS 1,174k  avg 7.44us  sd    23us  max     6,971us
             IOPS 1,179k  avg 7.38us  sd    23us  max     3,477us
             IOPS 1,180k  avg 7.39us  sd    23us  max     2,894us
             IOPS 1,145k  avg 7.37us  sd    21us  max     4,048us
             IOPS 1,177k  avg 7.38us  sd    22us  max     5,620us
             IOPS 1,177k  avg 7.40us  sd    23us  max     5,623us

    IOPS: BASE 932k (+-260k), PATCHED 1,174k (+-7k), 21 runs each.
          significant difference at 95% confidence: PATCHED is 241k (25.9%) higher; 95% CI [+136k, +353k], Welch's t-test p = 0.000
    max latency: BASE 3.1..2,085.8 ms, PATCHED 2.9..7.6 ms
          runs with max over 100 ms: BASE 17 of 21, PATCHED 0 of 21; over 1 s: BASE 11 of 21, PATCHED 0 of 21

######## zstd compression level 12 W-vs-R
    BASE     IOPS 690k  avg 21.53us  sd 3,088us  max 1,635,500us
             IOPS 478k  avg 36.51us  sd 4,385us  max 1,941,300us
             IOPS 822k  avg 16.13us  sd 2,865us  max 2,263,800us
             IOPS 1,161k  avg 7.73us  sd   114us  max   190,307us
             IOPS 780k  avg 17.56us  sd 2,717us  max 1,550,500us
             IOPS 1,166k  avg 7.66us  sd    40us  max    94,522us
             IOPS 585k  avg 27.51us  sd 3,847us  max 1,550,600us
             IOPS 1,147k  avg 7.97us  sd   344us  max   496,483us
             IOPS 1,166k  avg 7.64us  sd    49us  max   115,243us
             IOPS 1,163k  avg 7.66us  sd    23us  max     7,006us
             IOPS 1,071k  avg 9.42us  sd 1,489us  max 1,915,500us
             IOPS 962k  avg 11.88us  sd 2,168us  max 1,904,500us
             IOPS 607k  avg 26.05us  sd 3,538us  max 1,591,100us
             IOPS 1,167k  avg 7.63us  sd    23us  max     2,903us
             IOPS 1,157k  avg 7.78us  sd   198us  max   418,930us
             IOPS 1,163k  avg 7.66us  sd    40us  max   100,346us
             IOPS 1,162k  avg 7.70us  sd    32us  max    82,602us
             IOPS 488k  avg 35.43us  sd 4,340us  max 1,786,400us
             IOPS 563k  avg 28.85us  sd 3,989us  max 2,209,500us
             IOPS 843k  avg 15.43us  sd 2,490us  max 1,572,100us
             IOPS 604k  avg 26.30us  sd 3,560us  max 1,555,700us
    PATCHED  IOPS 1,166k  avg 7.60us  sd    23us  max     5,623us
             IOPS 1,169k  avg 7.58us  sd    23us  max     5,629us
             IOPS 1,157k  avg 7.67us  sd    27us  max     5,639us
             IOPS 1,166k  avg 7.61us  sd    24us  max     5,644us
             IOPS 1,160k  avg 7.64us  sd    26us  max     7,543us
             IOPS 1,167k  avg 7.60us  sd    24us  max     7,514us
             IOPS 1,160k  avg 7.61us  sd    25us  max     6,643us
             IOPS 1,162k  avg 7.65us  sd    25us  max     7,315us
             IOPS 1,161k  avg 7.66us  sd    24us  max     6,237us
             IOPS 1,158k  avg 7.62us  sd    27us  max     7,688us
             IOPS 1,167k  avg 7.58us  sd    24us  max     5,637us
             IOPS 1,165k  avg 7.59us  sd    24us  max     7,098us
             IOPS 948k  avg 7.58us  sd    23us  max     7,808us
             IOPS 1,166k  avg 7.60us  sd    24us  max     5,621us
             IOPS 1,166k  avg 7.58us  sd    24us  max     5,672us
             IOPS 1,162k  avg 7.65us  sd    25us  max     6,824us
             IOPS 1,165k  avg 7.59us  sd    24us  max     7,078us
             IOPS 1,168k  avg 7.58us  sd    23us  max     6,191us
             IOPS 1,170k  avg 7.56us  sd    22us  max     2,851us
             IOPS 1,171k  avg 7.55us  sd    22us  max     2,834us
             IOPS 1,166k  avg 7.59us  sd    24us  max     5,627us

    IOPS: BASE 902k (+-269k), PATCHED 1,154k (+-47k), 21 runs each.
          significant difference at 95% confidence: PATCHED is 252k (27.9%) higher; 95% CI [+142k, +367k], Welch's t-test p = 0.000
    max latency: BASE 2.9..2,263.8 ms, PATCHED 2.8..7.8 ms
          runs with max over 100 ms: BASE 17 of 21, PATCHED 0 of 21; over 1 s: BASE 12 of 21, PATCHED 0 of 21

######## lz4 W-vs-R
    BASE     IOPS 1,422k  avg 3.84us  sd   418us  max   619,495us
             IOPS 1,450k  avg 3.57us  sd   326us  max   609,723us
             IOPS 1,414k  avg 3.92us  sd   543us  max   879,815us
             IOPS 1,451k  avg 3.56us  sd   253us  max   503,428us
             IOPS 1,460k  avg 3.40us  sd    68us  max   135,825us
             IOPS 1,413k  avg 3.31us  sd    44us  max   188,450us
             IOPS 1,432k  avg 3.73us  sd   503us  max   844,228us
             IOPS 1,387k  avg 4.20us  sd   719us  max   961,827us
             IOPS 1,464k  avg 3.38us  sd    44us  max   133,675us
             IOPS 1,469k  avg 3.38us  sd    22us  max    65,812us
             IOPS 1,465k  avg 3.40us  sd    49us  max   111,788us
             IOPS 1,465k  avg 3.39us  sd    64us  max   174,040us
             IOPS 1,467k  avg 3.38us  sd    36us  max    97,821us
             IOPS 1,445k  avg 3.63us  sd   343us  max   671,026us
             IOPS 1,472k  avg 3.35us  sd    27us  max    86,795us
             IOPS 1,473k  avg 3.33us  sd    44us  max   138,096us
             IOPS 1,439k  avg 3.71us  sd   353us  max   619,815us
             IOPS 1,398k  avg 3.36us  sd    54us  max   206,815us
             IOPS 1,471k  avg 3.36us  sd    15us  max     2,832us
             IOPS 1,407k  avg 4.02us  sd   629us  max 1,074,800us
             IOPS 1,465k  avg 3.40us  sd    63us  max   158,779us
    PATCHED  IOPS 1,471k  avg 3.30us  sd    15us  max     6,946us
             IOPS 1,473k  avg 3.29us  sd    16us  max     5,618us
             IOPS 1,474k  avg 3.29us  sd    15us  max     5,615us
             IOPS 1,473k  avg 3.29us  sd    15us  max     5,625us
             IOPS 1,440k  avg 3.26us  sd    13us  max     2,829us
             IOPS 1,456k  avg 3.30us  sd    15us  max     5,623us
             IOPS 1,476k  avg 3.29us  sd    15us  max     2,835us
             IOPS 1,464k  avg 3.28us  sd    16us  max     6,063us
             IOPS 1,468k  avg 3.32us  sd    19us  max     6,692us
             IOPS 1,465k  avg 3.30us  sd    17us  max     6,834us
             IOPS 1,468k  avg 3.28us  sd    16us  max     7,268us
             IOPS 1,463k  avg 3.31us  sd    17us  max     6,480us
             IOPS 1,454k  avg 3.33us  sd    20us  max     8,418us
             IOPS 1,452k  avg 3.32us  sd    19us  max     5,621us
             IOPS 1,477k  avg 3.27us  sd    14us  max     2,834us
             IOPS 1,468k  avg 3.30us  sd    16us  max     5,617us
             IOPS 1,475k  avg 3.30us  sd    16us  max     3,884us
             IOPS 1,475k  avg 3.30us  sd    16us  max     5,692us
             IOPS 1,471k  avg 3.30us  sd    15us  max     5,616us
             IOPS 1,473k  avg 3.28us  sd    16us  max     7,014us
             IOPS 1,475k  avg 3.28us  sd    15us  max     5,616us

    IOPS: BASE 1,444k (+-27k), PATCHED 1,467k (+-10k), 21 runs each.
          significant difference at 95% confidence: PATCHED is 23k (1.6%) higher; 95% CI [+11k, +35k], Welch's t-test p = 0.001
    max latency: BASE 2.8..1,074.8 ms, PATCHED 2.8..8.4 ms
          runs with max over 100 ms: BASE 17 of 21, PATCHED 0 of 21; over 1 s: BASE 1 of 21, PATCHED 0 of 21