From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793329369; b=PMDZmyNYC7xUGrqAi3usDq0nL9IAyfeIbZvKS8UGeWb5/UpqRrz358smNQ9bzbeUGJaV 4wr3M8g+QshzpHY78c0MbwlTU+RXZ7dq5MLf/LrknHT7yqSq2ocJtcmm/E3p5as7kc8j9 FRVGodSj+jjbIF1l2MnEOhC9djc4Y07gTow5HvsQwc7gfccspIsBhyTFLBcCFtSI/Q9Zq igwWEkiVaD2UiRNCN9VOmxyI89BikbAzBB2VWrppinOb8WzZlTdkaq+qqufYzTXrWDDV7 V+vsOkWm+B3DElUPK7kZDkXQl1cRzINk7BXlGH+lG0l0x40wmg2ps2ohvVRbZpYAg4A== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793329369; h=Received:DKIM-Signature:Date:To:From:Subject:Message-Id; bh=zAZxnLsBGGVCRMUlQybhG7ATue+jA+AQSYE5YAavTY8=; b=MeQIseC2WPAHgnX9wR3D5Em/iyoqHymZqT+Vc/j8HETjaLxLhyL5CiSlRtlSNvCPcYuU AxjeaZE+7gUiFZ7HgQKJSBafw/zGeo4505rM+zmd72+2A5NlZ2S7RshjaNVtAS23+5Km8 uKdnMOf8GKHORmqW2nFzyXT9ZRTlDDLGkBemmfADRMgHHJz3jGUbg9UMFaiHCVYvapl2P E0VWwpMyTOwyn7oeaW2eRdle0zRhrn/3e4NxyF2z0HmCMmD0r0u5iOW4JFUYOThPz75wn jV6A2fPFEopHw64mO9XRsk6uhNMUzUuwvLke9jFoJ2PVP0fwmaY0peikWTu9fUVXnkQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=linux-foundation.org header.i=@linux-foundation.org header.a=rsa-sha256 header.s=korg header.b=B2cOvPz3; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 4AC20606A31E for ; Fri, 30 Oct 2026 03:02:48 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=linux-foundation.org header.i=@linux-foundation.org header.a=rsa-sha256 header.s=korg header.b=B2cOvPz3 Received: by smtp.kernel.org (Postfix) with ESMTPSA id BD5AC1F000C1; Fri, 30 Oct 2026 03:02:48 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=korg; t=1793329368; bh=zAZxnLsBGGVCRMUlQybhG7ATue+jA+AQSYE5YAavTY8=; h=Date:To:From:Subject; b=B2cOvPz3dwuwoJVF5PUQic5yxQdUv9E8vcWShaDEDM0nbSe56/hJsdTNYWjrpb41Q pfd42+Qs+MTTaCitDPFQ+W1hwADfXkfWzM96YARiRn8le7Q6a1gUTuwmxiJbTauBon smosodAqDIKBBCXf4ndmvirCU/zNVwbRQDrciv6U= Date: Thu, 29 Oct 2026 20:02:48 -0700 To: mm-commits@vger.kernel.org,zhangbo56@xiaomi.com,zhangbo0325@gmail.com, richardycc@google.com,minchan@kernel.org,hdanton@sina.com,bgeffon@google.com, baohua@kernel.org,senozhatsky@chromium.org,akpm@linux-foundation.org From: Andrew Morton Subject: + zram-allocate-bounce-buffer-for-spanning-objects.patch added to mm-nonmm-unstable branch Message-Id: <20261030030248.BD5AC1F000C1@smtp.kernel.org> X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list The patch titled Subject: zram: allocate bounce buffer for spanning objects has been added to the -mm mm-nonmm-unstable branch. Its filename is zram-allocate-bounce-buffer-for-spanning-objects.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/zram-allocate-bounce-buffer-for-spanning-objects.patch This patch will later appear in the mm-nonmm-unstable branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via various branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there most days ------------------------------------------------------ From: Sergey Senozhatsky Subject: zram: allocate bounce buffer for spanning objects Date: Sun, 25 Oct 2026 11:18:29 +0900 zsmalloc objects that span two physical pages need to be linearized before decompression, which zcomp does using stream's ->local_copy buffer. That buffer cannot be used concurrently, so such reads take the stream in exclusive mode, even when the backend has stateless decompression. And spanning objects are not rare: Bo Zhang measured 23.3% of reads on lzo-rle going through the exclusive path, causing contention on ->excl_lock of up to 72-96 ms [1]. Allocate a temporary bounce buffer for spanning objects instead, so that backends with stateless decompression can always use shared mode. If the allocation fails we fall back to ->local_copy under ->excl_lock. Per-request bounce buffer fixes the remaining tail latency, as was measured by Bo Zhang [2]: Metric Base kmalloc bounce ---------------------------------------- read contended ~1,959 0 shared 76.7% 100% A quick synthetic fio test for Reader vs Reader inversion on 24 CPUs (lz4, 48 low priority (nice 19) readers, 24 normal priority CPU hogs and 24 high priority readers), average of 3 runs. Base is the previous patch (shared streams without bounce buffer): Metric Base Patched ---------------------------------------------------------- Read IOPS 1,452k 1,466k (+1.0%) Read clat avg 3.46 us 3.30 us (-4.9%) Read clat stdev 165 us 15.69 us (-10.5x) Read clat max 306,679 us 7,060 us (-43.4x) An alternative approach is to vm_map_ram() scattered objects (two pages), which fixes the tail latency in the same way, but deferred TLB flushes add more cost, so it was rejected. Link: https://lore.kernel.org/20261025021839.1542805-14-senozhatsky@chromium.org Link: https://lore.kernel.org/linux-kernel/20261014153603.1365617-1-zhangbo56@xiaomi.com [1] Link: https://lore.kernel.org/linux-kernel/20261016134847.1620103-1-zhangbo0325@gmail.com [2] Signed-off-by: Sergey Senozhatsky Signed-off-by: Andrew Morton Cc: Brian Geffon Cc: Barry Song Cc: Bo Zhang Cc: Minchan Kim Cc: Richard Chang Cc: Bo Zhang Cc: Hillf Danton --- drivers/block/zram/zcomp.c | 47 +++++++++++++++++++++++--------- drivers/block/zram/zcomp.h | 2 - drivers/block/zram/zram_drv.c | 2 - 3 files changed, 36 insertions(+), 15 deletions(-) --- a/drivers/block/zram/zcomp.c~zram-allocate-bounce-buffer-for-spanning-objects +++ a/drivers/block/zram/zcomp.c @@ -176,14 +176,13 @@ void zcomp_cstrm_put(struct zcomp_cstrm mutex_unlock(&zstrm->lock); } -bool zcomp_shared_decomp(struct zcomp *comp, struct scatterlist *sg) +bool zcomp_shared_decomp(struct zcomp *comp) { /* * Shared decompression is only safe when the backend has no - * mutable per-stream decompression context and the object is - * contained within a single page (no ->local_copy bounce needed). + * mutable per-stream decompression context. */ - return !comp->ops->create_dctx && sg_is_last(sg); + return !comp->ops->create_dctx; } struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp, @@ -239,6 +238,15 @@ int zcomp_compress(struct zcomp *comp, s return ret; } +static int zcomp_decompress_buf(struct zcomp *comp, struct zcomp_dstrm *zstrm, + struct scatterlist *sg, void *buf, + struct zcomp_req *req) +{ + sg_copy_to_buffer(sg, 2, buf, req->src_len); + req->src = buf; + return comp->ops->decompress(comp->params, &zstrm->ctx, req); +} + int zcomp_decompress(struct zcomp *comp, struct zcomp_dstrm *zstrm, struct scatterlist *sg, unsigned int src_len, void *dst) { @@ -247,27 +255,40 @@ int zcomp_decompress(struct zcomp *comp, .src_len = src_len, .dst_len = PAGE_SIZE, }; - void *src = NULL; + void *src; int ret; might_sleep(); lockdep_assert_held(&zstrm->lock); - if (!zcomp_shared_decomp(comp, sg)) + if (!zcomp_shared_decomp(comp)) lockdep_assert_held(&zstrm->excl_lock); if (sg_is_last(sg)) { /* the object is contained within one page, read it in-place */ src = kmap_local_page(sg_page(sg)); req.src = src + sg->offset; - } else { - /* the object spans two pages, linearize it into local copy */ - sg_copy_to_buffer(sg, 2, zstrm->local_copy, src_len); - req.src = zstrm->local_copy; + ret = comp->ops->decompress(comp->params, &zstrm->ctx, &req); + kunmap_local(src); + return ret; } - ret = comp->ops->decompress(comp->params, &zstrm->ctx, &req); - if (src) - kunmap_local(src); + /* exclusive mode already holds ->excl_lock */ + if (!zcomp_shared_decomp(comp)) + return zcomp_decompress_buf(comp, zstrm, sg, zstrm->local_copy, + &req); + + /* shared mode: try a per-request bounce buffer first */ + src = kmalloc(src_len, GFP_NOWAIT | __GFP_NOWARN); + if (src) { + ret = zcomp_decompress_buf(comp, zstrm, sg, src, &req); + kfree(src); + return ret; + } + + /* fall back to ->local_copy serialized by ->excl_lock */ + mutex_lock(&zstrm->excl_lock); + ret = zcomp_decompress_buf(comp, zstrm, sg, zstrm->local_copy, &req); + mutex_unlock(&zstrm->excl_lock); return ret; } --- a/drivers/block/zram/zcomp.h~zram-allocate-bounce-buffer-for-spanning-objects +++ a/drivers/block/zram/zcomp.h @@ -112,7 +112,7 @@ void zcomp_destroy(struct zcomp *comp); struct zcomp_cstrm *zcomp_cstrm_get(struct zcomp *comp); void zcomp_cstrm_put(struct zcomp_cstrm *zstrm); -bool zcomp_shared_decomp(struct zcomp *comp, struct scatterlist *sg); +bool zcomp_shared_decomp(struct zcomp *comp); struct zcomp_dstrm *zcomp_dstrm_get(struct zcomp *comp, enum zcomp_dstrm_mode mode); void zcomp_dstrm_put(struct zcomp_dstrm *zstrm, enum zcomp_dstrm_mode mode); --- a/drivers/block/zram/zram_drv.c~zram-allocate-bounce-buffer-for-spanning-objects +++ a/drivers/block/zram/zram_drv.c @@ -2128,7 +2128,7 @@ static int read_compressed_page(struct z prio = get_slot_comp_priority(zram, index); zs_obj_read_sg_begin(zram->mem_pool, handle, sg, size); - if (zcomp_shared_decomp(zram->comps[prio], sg)) + if (zcomp_shared_decomp(zram->comps[prio])) mode = ZCOMP_DSTRM_SHARED; zstrm = zcomp_dstrm_get(zram->comps[prio], mode); _ Patches currently in -mm which might be from senozhatsky@chromium.org are zram-prefix-backends-printk-s.patch zram-remove-debugfs-entry-on-init-failure.patch zram-reset-writeback-state-in-zram_reset_device.patch zram-move-mark_slot_accessed-under-existing-slot-lock-sections.patch zram-remove-stale-warn_on-from-zram_read_page.patch zram-zstd-do-not-allocate-empty-c-d-dictionaries.patch zram-lz4hc-pre-initialize-dictionary-stream-in-setup_params.patch zram-lz4-use-lz4_decompress_safe_usingdict.patch zram-lz4hc-use-lz4_decompress_safe_usingdict.patch zram-zstd-do-not-use-kvzalloc-for-zstd-allocations.patch zram-split-zcomp-into-separate-r-w-streams.patch zram-introduce-shared-decompression-streams.patch zram-allocate-bounce-buffer-for-spanning-objects.patch zram-borrow-idle-decompression-streams-in-exclusive-mode.patch zram-use-a-singleton-compression-context-for-recompression.patch