From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-alma10-1.taild15c8.ts.net [100.103.45.18]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 85720324B33 for ; Sat, 10 Oct 2026 22:22:12 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=100.103.45.18 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791670934; cv=none; b=ublzCPBgMRRLtpDA+Hj832NfapMEzKuhSTtJ6UVW8xCE/mJ10nQ7+lj59YP5FCcngeKXtUsaOf1Sn/+udusL6BGEV6+aXgZLfezTI1hPbjtOIL9kalkD6z3AwTlme/cMP7ZqbainQ+/2OdHIMeGPq3BeafCinoGs61cczfmgFZU= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1791670934; c=relaxed/simple; bh=thFNPnSM2Mntb2bO/ysaAleTnM2y73shUwWDMyMH1iI=; h=Date:To:From:Subject:Message-Id; b=GpVcfO53OpjT7eCN8hAsmVaRjPJlSrkzi/XhI7LOMipUS8gJPmY9oICysPLnqbxVT/LvND6sSrZakWft5lbqxwSd8FZr8qM7STiNncN4fctyZXaVR7UenPwzBRvaxKAWxCZlv/pKXKmTbXQxVOuOyuanqZ6cGQwIfDQ5lyacqFQ= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b=QNf89z+R; arc=none smtp.client-ip=100.103.45.18 Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key) header.d=linux-foundation.org header.i=@linux-foundation.org header.b="QNf89z+R" Received: by smtp.kernel.org (Postfix) with ESMTPSA id DCC401F00893; Sat, 10 Oct 2026 22:22:11 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=korg; t=1791670932; bh=m0NyTyeUClVuHBqZ2ku1880InqeijizrwsxvOaFveNY=; h=Date:To:From:Subject; b=QNf89z+RJpFlJuJFByYj3jHeFJV89sgNNQXou3RrHdFCU8ipKz9PpS9xua3Uu5nA7 w8HrBqXNeI+htsAgtUef0NoX1Nt7S3Tz/+CTVUzIYFlUpHnG+Rg5La3rmoHD1wGqve qtZT/IJUor5tl9Q0trwItuOGpbACF74iOzXkhbu0= Date: Sat, 10 Oct 2026 15:22:11 -0700 To: mm-commits@vger.kernel.org,yosry@kernel.org,terrelln@fb.com,shakeel.butt@linux.dev,senozhatsky@chromium.org,riel@surriel.com,nphamcs@gmail.com,herbert@gondor.apana.org.au,hannes@cmpxchg.org,dsterba@suse.com,davem@davemloft.net,chengming.zhou@linux.dev,alex@ghiti.fr,usama.arif@linux.dev,akpm@linux-foundation.org From: Andrew Morton Subject: + mm-zswap-use-separate-compression-and-decompression-requests.patch added to mm-new branch Message-Id: <20261010222211.DCC401F00893@smtp.kernel.org> Precedence: bulk X-Mailing-List: mm-commits@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: The patch titled Subject: mm: zswap: use separate compression and decompression requests has been added to the -mm mm-new branch. Its filename is mm-zswap-use-separate-compression-and-decompression-requests.patch This patch will shortly appear at https://git.kernel.org/pub/scm/linux/kernel/git/akpm/25-new.git/tree/patches/mm-zswap-use-separate-compression-and-decompression-requests.patch This patch will later appear in the mm-new branch at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm Note, mm-new is a provisional staging ground for work-in-progress patches, and acceptance into mm-new is a notification for others take notice and to finish up reviews. Please do not hesitate to respond to review feedback and post updated versions to replace or incrementally fixup patches in mm-new. The mm-new branch of mm.git is not included in linux-next If a few days of testing in mm-new is successful, the patch will me moved into mm.git's mm-unstable branch, which is included in linux-next Before you just go and hit "reply", please: a) Consider who else should be cc'ed b) Prefer to cc a suitable mailing list as well c) Ideally: find the original patch on the mailing list and do a reply-to-all to that, adding suitable additional cc's *** Remember to use Documentation/process/submit-checklist.rst when testing your code *** The -mm tree is included into linux-next via various branches at git://git.kernel.org/pub/scm/linux/kernel/git/akpm/mm and is updated there most days ------------------------------------------------------ From: Usama Arif Subject: mm: zswap: use separate compression and decompression requests Date: Fri, 09 Oct 2026 09:30:30 -0700 Patch series "mm: zswap: reduce request contention on loads", v2. Stores and loads share a per-CPU acomp request and mutex. A low-priority store can be preempted right after the compressor drops its stream lock, while it still holds the zswap mutex, and a higher-priority load on that CPU then waits for the store to run again. This follows the work from Sergey Senozhatsky's zram series which splits it for the same reason [1]. Patch 1 gives compression and decompression separate requests, waits and mutexes, so loads no longer wait for stores, though they can still wait for each other. Patch 2 decompresses with an on-stack request when the algorithm is synchronous and needs no request context, which covers all in-tree software compressors, so those loads take no zswap lock. Asynchronous algorithms keep the per-CPU request and mutex. For software compressors the series allocates the same number of requests as before; each per-CPU context grows by 72 bytes, and the load path is about 270 bytes deeper on x86-64. The series does not fix two related cases: - Stores still serialize on the compression mutex, so a high-priority task that reclaims (direct reclaim, MADV_PAGEOUT) can still wait for a preempted store. - On PREEMPT_RT the codec stream locks are preemptible, so a load can still wait for a preempted store inside the codec. The numbers below are the slowest read per run, as a median (min-max) of 5 runs. Each run is 12 seconds in a zstd VM with lazy preemption, vm.page-cluster=0 and swap on /dev/ram0. With 1 vCPU, four nice +10 workers page memory out and read it back while a nice 0 task spins. A nice -19 reader pages out its own buffer and measures how long each read of it takes. With 8 vCPUs there are 16 workers, 8 spinning tasks and 8 readers. Before series (ms) With series (ms) 1 vCPU 22.3 (21.6-22.6) 0.97 (0.72-1.4) 8 vCPUs 314 (97-2542) 7.0 (5.0-98) Reads over 10 ms fell from 26-35 per run to none with 1 vCPU, and from 3-18 per run to at most one with 8 vCPUs. The benchmark and test programs were written with the help of an LLM. This patch (of 2): Stores and loads serialize on the same per-CPU acomp request and mutex. A low-priority store can be preempted as soon as the compressor drops its stream lock, while it still holds the mutex. A higher-priority load on that CPU then waits until the store runs again, which can take a long time when other tasks are runnable. Give compression and decompression their own request, completion wait and mutex. Since commit e2c3b6b21c77f ("mm: zswap: use SG list decompression APIs from zsmalloc"), the per-CPU buffer is only used for compression, so group it with the compression request. Independent requests can share the per-CPU transform; the codecs and drivers synchronize their shared state internally. Loads can still wait for each other on the decompression mutex, and stores still serialize on the compression mutex. acomp_request_free() ignores NULL requests, so drop the check in acomp_ctx_free(). This follows the proposal from Sergey Senozhatsky for the same split for zram [1]. Link: https://lore.kernel.org/20261009163146.83112-1-usama.arif@linux.dev Link: https://lore.kernel.org/20261009163146.83112-2-usama.arif@linux.dev Link: https://lore.kernel.org/all/20261005122036.718976-10-senozhatsky@chromium.org/ [1] Signed-off-by: Usama Arif Signed-off-by: Andrew Morton Acked-by: Nhat Pham Cc: Herbert Xu Cc: Cc: David Sterba Cc: Johannes Weiner Cc: Nick Terrell Cc: Yosry Ahmed Cc: Rik van Riel Cc: Shakeel Butt Cc: Cc: Sergey Senozhatsky Cc: David S. Miller --- mm/zswap.c | 108 ++++++++++++++++++++++++++++++--------------------- 1 file changed, 64 insertions(+), 44 deletions(-) --- a/mm/zswap.c~mm-zswap-use-separate-compression-and-decompression-requests +++ a/mm/zswap.c @@ -137,14 +137,25 @@ bool zswap_never_enabled(void) * data structures **********************************/ -struct crypto_acomp_ctx { - struct crypto_acomp *acomp; +struct zswap_acomp_req { struct acomp_req *req; struct crypto_wait wait; - u8 *buffer; struct mutex mutex; }; +/* The compression mutex also protects the output buffer. */ +struct zswap_comp_ctx { + struct zswap_acomp_req areq; + u8 *buffer; +}; + +/* Separate requests, so that decompression does not wait for compression. */ +struct crypto_acomp_ctx { + struct crypto_acomp *acomp; + struct zswap_comp_ctx comp; + struct zswap_acomp_req decomp; +}; + /* * The lock ordering is zswap_tree.lock -> zswap_pool.lru_lock. * The only case where lru_lock is not acquired while holding tree.lock is @@ -270,14 +281,10 @@ static void acomp_ctx_free(struct crypto if (!acomp_ctx) return; - /* - * If there was an error in allocating @acomp_ctx->req, it - * would be set to NULL. - */ - if (acomp_ctx->req) - acomp_request_free(acomp_ctx->req); - - acomp_ctx->req = NULL; + acomp_request_free(acomp_ctx->comp.areq.req); + acomp_ctx->comp.areq.req = NULL; + acomp_request_free(acomp_ctx->decomp.req); + acomp_ctx->decomp.req = NULL; /* * We have to handle both cases here: an error pointer return from @@ -289,8 +296,8 @@ static void acomp_ctx_free(struct crypto acomp_ctx->acomp = NULL; - kfree(acomp_ctx->buffer); - acomp_ctx->buffer = NULL; + kfree(acomp_ctx->comp.buffer); + acomp_ctx->comp.buffer = NULL; } static struct zswap_pool *zswap_pool_create(char *compressor) @@ -796,6 +803,28 @@ static void zswap_entry_free(struct zswa /********************************* * compressed storage functions **********************************/ +static int zswap_acomp_req_init(struct zswap_acomp_req *areq, + struct crypto_acomp *acomp) +{ + /* acomp_request_alloc() returns NULL in case of an error. */ + areq->req = acomp_request_alloc(acomp); + if (!areq->req) + return -ENOMEM; + + crypto_init_wait(&areq->wait); + + /* + * if the backend of acomp is async zip, crypto_req_done() will wakeup + * crypto_wait_req(); if the backend of acomp is scomp, the callback + * won't be called, crypto_wait_req() will return without blocking. + */ + acomp_request_set_callback(areq->req, CRYPTO_TFM_REQ_MAY_BACKLOG, + crypto_req_done, &areq->wait); + + mutex_init(&areq->mutex); + return 0; +} + static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node) { struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node); @@ -811,8 +840,8 @@ static int zswap_cpu_comp_prepare(unsign return 0; } - acomp_ctx->buffer = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); - if (!acomp_ctx->buffer) + acomp_ctx->comp.buffer = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu)); + if (!acomp_ctx->comp.buffer) return ret; /* @@ -827,25 +856,13 @@ static int zswap_cpu_comp_prepare(unsign goto fail; } - /* acomp_request_alloc() returns NULL in case of an error. */ - acomp_ctx->req = acomp_request_alloc(acomp_ctx->acomp); - if (!acomp_ctx->req) { + if (zswap_acomp_req_init(&acomp_ctx->comp.areq, acomp_ctx->acomp) || + zswap_acomp_req_init(&acomp_ctx->decomp, acomp_ctx->acomp)) { pr_err("could not alloc crypto acomp_request %s\n", pool->tfm_name); goto fail; } - crypto_init_wait(&acomp_ctx->wait); - - /* - * if the backend of acomp is async zip, crypto_req_done() will wakeup - * crypto_wait_req(); if the backend of acomp is scomp, the callback - * won't be called, crypto_wait_req() will return without blocking. - */ - acomp_request_set_callback(acomp_ctx->req, CRYPTO_TFM_REQ_MAY_BACKLOG, - crypto_req_done, &acomp_ctx->wait); - - mutex_init(&acomp_ctx->mutex); return 0; fail: @@ -856,7 +873,7 @@ fail: static bool zswap_compress(struct folio *folio, long index, struct zswap_entry *entry, struct zswap_pool *pool) { - struct crypto_acomp_ctx *acomp_ctx; + struct zswap_comp_ctx *comp_ctx; struct scatterlist input, output; int comp_ret = 0, alloc_ret = 0; unsigned int dlen = PAGE_SIZE; @@ -865,15 +882,16 @@ static bool zswap_compress(struct folio u8 *dst; bool mapped = false; - acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); - mutex_lock(&acomp_ctx->mutex); + comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp; + mutex_lock(&comp_ctx->areq.mutex); - dst = acomp_ctx->buffer; + dst = comp_ctx->buffer; sg_init_table(&input, 1); sg_set_folio(&input, folio, PAGE_SIZE, index * PAGE_SIZE); sg_init_one(&output, dst, PAGE_SIZE); - acomp_request_set_params(acomp_ctx->req, &input, &output, PAGE_SIZE, dlen); + acomp_request_set_params(comp_ctx->areq.req, &input, &output, + PAGE_SIZE, dlen); /* * it maybe looks a little bit silly that we send an asynchronous request, @@ -885,10 +903,12 @@ static bool zswap_compress(struct folio * existing method to send the second page before the first page is done * in one thread doing zswap. * but in different threads running on different cpu, we have different - * acomp instance, so multiple threads can do (de)compression in parallel. + * acomp instance, and compression and decompression use separate + * requests, so multiple threads can do (de)compression in parallel. */ - comp_ret = crypto_wait_req(crypto_acomp_compress(acomp_ctx->req), &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; + comp_ret = crypto_wait_req(crypto_acomp_compress(comp_ctx->areq.req), + &comp_ctx->areq.wait); + dlen = comp_ctx->areq.req->dlen; /* * If a page cannot be compressed into a size smaller than PAGE_SIZE, @@ -932,7 +952,7 @@ unlock: else if (alloc_ret) zswap_reject_alloc_fail++; - mutex_unlock(&acomp_ctx->mutex); + mutex_unlock(&comp_ctx->areq.mutex); return comp_ret == 0 && alloc_ret == 0; } @@ -948,7 +968,7 @@ static bool zswap_decompress(struct zswa return false; acomp_ctx = raw_cpu_ptr(pool->acomp_ctx); - mutex_lock(&acomp_ctx->mutex); + mutex_lock(&acomp_ctx->decomp.mutex); zs_obj_read_sg_begin(pool->zs_pool, entry->handle, input, entry->length); /* zswap entries of length PAGE_SIZE are not compressed. */ @@ -965,15 +985,15 @@ static bool zswap_decompress(struct zswa } else { sg_init_table(&output, 1); sg_set_folio(&output, folio, PAGE_SIZE, 0); - acomp_request_set_params(acomp_ctx->req, input, &output, + acomp_request_set_params(acomp_ctx->decomp.req, input, &output, entry->length, PAGE_SIZE); - ret = crypto_acomp_decompress(acomp_ctx->req); - ret = crypto_wait_req(ret, &acomp_ctx->wait); - dlen = acomp_ctx->req->dlen; + ret = crypto_acomp_decompress(acomp_ctx->decomp.req); + ret = crypto_wait_req(ret, &acomp_ctx->decomp.wait); + dlen = acomp_ctx->decomp.req->dlen; } zs_obj_read_sg_end(pool->zs_pool, entry->handle); - mutex_unlock(&acomp_ctx->mutex); + mutex_unlock(&acomp_ctx->decomp.mutex); if (!ret && dlen == PAGE_SIZE) return true; _ Patches currently in -mm which might be from usama.arif@linux.dev are mm-filemap-move-lruvec-accounting-outside-the-xarray-lock.patch mm-remove-the-unused-zone-unaccepted_cleanup.patch mm-zswap-use-separate-compression-and-decompression-requests.patch mm-zswap-use-stack-requests-for-synchronous-decompression.patch