All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH v4 0/8] Batched zswap_store() framework with compression batching
@ 2026-11-20  4:18 Kanchana P. Sridhar
  2026-11-20  4:18 ` [PATCH v4 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
                   ` (8 more replies)
  0 siblings, 9 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20  4:18 UTC (permalink / raw)
  To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
	davem, linux-crypto, kanchanapsridhar2026, linux-kernel
  Cc: rientjes, willy, senozhatsky


v4: Batched zswap_store() framework with compression batching
=============================================================

This series is a spin-off of "v14: zswap compression batching with optimized
iaa_crypto driver" [1], and focuses on the generic crypto_acomp and zswap
compression batching enablers. Resuming work on this series following a gap
since v14 (January 2026). Please note that my email address and affiliation have
changed from kanchana.p.sridhar@intel.com to kanchanapsridhar2026@gmail.com. I
have retained the original Intel sign-offs on prior patches for historical
attribution while signing off v4 of this series with my current address.

The main focus of v4 was to establish a performance measurement methodology to
quantify the benefits to high level metrics by using zswap compression
batching. This was done using an Intel Sapphire Rapids bare-metal instance with
48 physical cores and by running "usemem30", i.e., 30 concurrent usemem
processes, each of which allocates 10G, in a 150G memory.high contrained cgroup,
for 10 iterations. The environment was set up to minimize performance jitter so
we can understand the system-level/workload-level advantages of zswap
compression batching when running workloads at scale.

To summarize the main contribution of this patch series: Batching is introduced
in zswap_store() of large folios: we carve out a large folio into batches of
ZSWAP_MAX_BATCH_SIZE (4 pages) for software compressors, and store each
batch. Moving the fundamental operations that need to be happen to store a page
to now work at the granularity of a batch offers opportunities to optimize them
from these perspectives:

1) Utilize existing kernel API that minimize obtaining locks for bulk
   operations, e.g. kmem_cache_alloc_bulk()/kmem_cache_free_bulk().

2) We keep the working set for each of the core store functionalities
   (alloc/initialize zswap entries, compress pages, store pages in zsmalloc,
   pool/objcg reference increments, objcg charging) to be focused on a
   batch, thereby minimizing cache evictions and benefiting from cache
   locality.

3) Along the lines of (1), the main zswap_compress() batching routine obtains
   the comp_ctx mutex lock once per batch, and releases it after it has
   processed all pages in that batch.

4) Any zswap store functionality that works on a batch can potentially be
   optimized via bulk-processing/parallelized as a next step, to further improve
   performance.

Performance testing: usemem30 with 100% over-commit:
=====================================================
Validation was done with mm-unstable as of 11-17-2026 (Baseline: commit
866f761fd414) and with this patch series.

v4 was validated on an Intel Xeon Sapphire Rapids bare-metal instance with
48 cores.

 Bare-metal instance specs:
 --------------------------
 - 48 physical cores
 - 190G RAM
 - 200G SWAP disk partition

 Workload: 30 concurrent usemem processes, each allocates 10G:
 -------------------------------------------------------------
 The vm-scalability "usemem" test (unmodified version) was run in a
 cgroup with memory.high fixed at 150G. There is no swap limit set for the
 cgroup.

 To quantify the performance benefit of zswap compress batching at scale,
 30 usemem processes were run ("usemem30"), each allocating and writing 10G of
 memory, and sleeping for 10 sec before exiting.

  usemem -w -O -b 5 -s 10 -n 30 10g
  echo 0 > /sys/module/zswap/parameters/shrinker_enabled

 Performance validation methodology:
 -----------------------------------
 These settings were used to minimize variation:

 - All cores were fixed to run at 2.4GHz with "performance" scaling governor.
 - Turbo boost was disabled.
 - CPU power-saving C-states restricted with:

     GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1 processor.max_cstate=1"

 The usemem30 experiments were run with the baseline mm-unstable as of
 11-17-2026 (commit 866f761fd414) and with this patch-series.

 After booting into each kernel:

  1) Constrain the cgroup's memory.high to 150G.

  2) Run 2 warm-up iterations of usemem30. Don't use this data for benchmarking
     since sys time typically skews higher for initial runs.

  3) Next, run 10 benchmarking iterations of usemem30:
     - Log these high-level metrics: Sys time and individual usemem throughput.

  4) Sys time and Throughput metrics were gathered for three mTHP settings:

     - 2048kB
     - 1024kB
     - 64kB

  5) Benchmark summary data includes Sys time and Throughput components from the
     10 iterations of usemem30:

     - Total Samples
     - Mean Value
     - Std Deviation
     - CoV (Variance): Coefficient of variation (relative standard deviation)
       measures spread relative to the mean rather than statistical variance.

 An LLM was used to help generate the benchmarking and data processing scripts.

usemem30 Benchmarking Summary:
==============================

 ---------------------------------------------------
    Improvement with v4 zswap compress batching
 ---------------------------------------------------
  mTHP enabled:       2048kB     1024kB       64kB
 ---------------------------------------------------
      Sys time:      -13.87%     -2.64%     -1.72%
    Throughput:       +3.85%     +0.42%     +1.31%
 ---------------------------------------------------

Detailed usemem30 benchmarking data:
====================================

 1) 2048kB mTHP:
 ===============
 
 ------------------------------------------------------------------
 2048kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)        (sec)
 ------------------------------------------------------------------
 [Run 1/10]  Sys Time:    1581.414s    1356.252s 
 [Run 2/10]  Sys Time:    1642.874s    1478.929s 
 [Run 3/10]  Sys Time:    1594.458s    1411.252s 
 [Run 4/10]  Sys Time:    1580.988s    1380.959s 
 [Run 5/10]  Sys Time:    1579.654s    1380.609s 
 [Run 6/10]  Sys Time:    1627.765s    1345.582s 
 [Run 7/10]  Sys Time:    1593.872s    1334.485s 
 [Run 8/10]  Sys Time:    1623.189s    1405.471s 
 [Run 9/10]  Sys Time:    1661.954s    1433.365s 
 [Run 10/10] Sys Time:    1602.098s    1329.813s
 ------------------------------------------------------------------
 
 ------------------------------------------------------------------
 METRIC: System CPU Time (Sys Time) : 2048kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)            (sec)
 ------------------------------------------------------------------
 Total Samples    :              10               10        
 Mean Value       :        1,608.83         1,385.67
 Std Deviation    :           28.67            47.47
 CoV (Variance)   :           1.78%            3.43%       
 ------------------------------------------------------------------
 METRIC: Per-Process Throughput : 2048kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                             (KB/s)           (KB/s)
 ------------------------------------------------------------------
 Total Samples    :             300              300       
 Mean Value       :         292,261          303,508  
 Std Deviation    :          12,946           20,806  
 CoV (Variance)   :           4.42%            6.86% 
 ------------------------------------------------------------------


 2) 1024kB mTHP:
 ===============
 
 ------------------------------------------------------------------
 1024kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)        (sec)
 ------------------------------------------------------------------
 [Run 1/10]  Sys Time:    1567.639s    1594.696s 
 [Run 2/10]  Sys Time:    1537.577s    1456.113s 
 [Run 3/10]  Sys Time:    1538.562s    1521.887s 
 [Run 4/10]  Sys Time:    1626.176s    1531.962s 
 [Run 5/10]  Sys Time:    1593.955s    1528.733s 
 [Run 6/10]  Sys Time:    1643.519s    1558.482s 
 [Run 7/10]  Sys Time:    1510.613s    1480.177s 
 [Run 8/10]  Sys Time:    1632.018s    1584.093s 
 [Run 9/10]  Sys Time:    1569.424s    1528.915s 
 [Run 10/10] Sys Time:    1552.747s    1570.451s
 ------------------------------------------------------------------
 
 ------------------------------------------------------------------
 METRIC: System CPU Time (Sys Time) : 1024kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)            (sec)
 ------------------------------------------------------------------
 Total Samples    :             10                10        
 Mean Value       :       1,577.22          1,535.55
 Std Deviation    :          45.13             43.79
 CoV (Variance)   :          2.86%             2.85%       
 ------------------------------------------------------------------
 METRIC: Per-Process Throughput : 1024kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                             (KB/s)           (KB/s)
 ------------------------------------------------------------------
 Total Samples    :            300               300       
 Mean Value       :        289,904           291,114  
 Std Deviation    :         15,385            14,553  
 CoV (Variance)   :          5.31%             5.00% 
 ------------------------------------------------------------------


 3) 64kB mTHP:
 =============
 
 ------------------------------------------------------------------
 64kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)        (sec)
 ------------------------------------------------------------------
 [Run 1/10]  Sys Time:    1914.525s    1878.483s 
 [Run 2/10]  Sys Time:    1869.412s    1863.177s 
 [Run 3/10]  Sys Time:    1944.049s    1863.502s 
 [Run 4/10]  Sys Time:    1942.757s    1900.144s 
 [Run 5/10]  Sys Time:    1866.505s    1887.285s 
 [Run 6/10]  Sys Time:    1877.716s    1861.668s 
 [Run 7/10]  Sys Time:    1861.741s    1868.788s 
 [Run 8/10]  Sys Time:    1904.735s    1816.372s 
 [Run 9/10]  Sys Time:    1891.074s    1850.435s 
 [Run 10/10] Sys Time:    1893.366s    1849.747s
 ------------------------------------------------------------------
 
 ------------------------------------------------------------------
 METRIC: System CPU Time (Sys Time) : 64kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)            (sec)
 ------------------------------------------------------------------
 Total Samples    :              10               10        
 Mean Value       :        1,896.59         1,863.96
 Std Deviation    :           29.86            22.96 
 CoV (Variance)   :           1.57%            1.23%       
 ------------------------------------------------------------------
 METRIC: Per-Process Throughput : 64kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                             (KB/s)           (KB/s)
 ------------------------------------------------------------------
 Total Samples    :            300               300       
 Mean Value       :        239,114           242,254   
 Std Deviation    :          9,090             6,419   
 CoV (Variance)   :          3.80%             2.65% 
 ------------------------------------------------------------------


usemem30 vmstats and zswap stats collected after benchmarking runs finish:
==========================================================================

 -----------------------------------------------------------------------
                              2048kB mTHP
 -----------------------------------------------------------------------
                     Baseline mm-unstable   v4 zswap compress batching
 -----------------------------------------------------------------------
 pswpin                                 0                          0
 pswpout                                0                          0
 thp_swpout                             0                          0
 thp_swpout_fallback                    6                          5
 zswpin                              4030                       4342
 zswpout                        430976801                  429712411
 zswpwb                                 0                          0
 pgmajfault                          6198                       6529
 2048kB-alloc                     1540507                    1536772
 2048kB-zswpout                    838683                     832643
 2048kB-swpout                          0                          0
 2048kB-swpout-fallback                 6                          5
 2048kB-swpin                           0                          0
 zswap_pool_total_size            4919296                    4882432
 zswap_pool_limit_hit                   0                          0
 zswap_written_back_pages               0                          0
 zswap_stored_pages                  4047                       3818
 memory.current                         0                          0
 memory.swap.current                    0                          0
 -----------------------------------------------------------------------


 -----------------------------------------------------------------------
                              1024kB mTHP
 -----------------------------------------------------------------------
                      Baseline mm-unstable   v4 zswap compress batching
 -----------------------------------------------------------------------
 pswpin                                  0                       0
 pswpout                                 0                       0
 thp_swpout                              0                       0
 thp_swpout_fallback                     0                       0
 zswpin                               4806                    4691
 zswpout                         429599479               429010497
 zswpwb                                  0                       0
 pgmajfault                           7117                    6993
 1024kB-alloc                      3192334                 3190121
 1024kB-zswpout                    1633939                 1625770
 1024kB-swpout                           0                       0
 1024kB-swpout-fallback                 11                       0
 1024kB-swpin                            0                       0
 zswap_pool_total_size             7577600                 6418432
 zswap_pool_limit_hit                    0                       0
 zswap_written_back_pagess               0                       0
 zswap_stored_incompressible_pages      12                       9
 zswap_stored_pages                   6798                    5175
 memory.current                          0                       0
 memory.swap.current                     0                       0
 ------------------------------------------------------------------


 ------------------------------------------------------------------
                             64kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
 ------------------------------------------------------------------
 pswpin                           0                        0
 pswpout                          0                        0
 thp_swpout                       0                        0
 thp_swpout_fallback              0                        0
 zswpin                        4906                     4444
 zswpout                  431480331                430606606
 zswpwb                           0                        0
 pgmajfault                    7233                     6669
 64kB-alloc                56263335                 56231864
 64kB-zswpout              26950511                 26901497
 64kB-swpout                      0                        0
 64kB-swpout-fallback             0                        0
 64kB-swpin                       0                        0
 zswap_pool_total_size      5648384                  5050368
 zswap_pool_limit_hit             0                        0
 zswap_written_back_pages         0                        0
 zswap_stored_pages            4175                     3652
 memory.current                   0                        0
 memory.swap.current              0                        0
 ------------------------------------------------------------------

Conclusions based on the data:
==============================

 v3 demonstrated that the batching of zswap_store() phases and compress
 batching optimizations benefit zswap_store() and zswap_compress_per_page()
 median and tail latencies for PMD folios and base 4K folios.

 v4 demonstrates how these low-level bptrace savings obtained in v3 translate
 to performance improvements at the system level and workload level. v4's
 performance validation is done at scale with 30 concurrent usemem processes,
 with Sys time and Throughput being the key high-level metrics. We further study
 this for PMD THP, 1024kB and 64kB folios.

 Sys time measured across 10 consecutive iterations of usemem30 improves
 significantly with zswap compress batching, -13.87% lower than mm-unstable for
 PMD THP. Per-process Throughput increases by 3.85% for PMD folios. As this
 patch-series impacts batching within folios, the Sys time savings reduces
 for lower order folios.

 The immediate benefit of zswap compress batching is saving CPU compute
 cycles, that can imply lower power consumption and existing kernel
 optimizations with reduced CPU compute cycles spent in critical
 paths. There are also modest throughput improvements that this
 patch-series delivers.

 As other kernel components related to mTHP become more efficient, the net
 realized performance benefit can only be better with this patch-series than
 without, given the significant Sys time reduction with zswap store and compress
 batching.

 The batching framework established in this series allows us to make
 locking and bulk-processing improvements in each stage of zswap_store(). One of
 the ideas that has shown very promising results is to replace zsmalloc with
 SLUB compressed caches and will be submitted separately. This is expected to
 resolve the p50 batch compress latency issue seen in v3's bpftrace data, since
 it will be directly optimizing zswap_update_entries(), where compressed buffers
 can be binned based on size and bulk allocated from SLUB.
 
 The speedup with zswap store batching of large folios has a propelling effect
 on other parts of mm, especially, by enabling large folios to be swapped out
 without splitting, as the higher-order folio vmstat v4 data shows. This enables
 other mm components that are optimized for large folios, such as munmap(),
 folio refcounts/mapcounts updates, PTE invalidation upon freeing memory,
 zswap_invalidate(); which benefit from xarray offsets/range continuity. As a
 result, subsequent large folio swapouts are more likely to find contiguous swap
 slots, there are fewer large folio splits during reclaim and the cycle
 continues. As the data demonstrates, this directly contributes to more memory
 over-commit opportunity.

 The benefits are attributable to the store batching framework added in patches
 6-7, that is fully utilized to improve zswap_compress() latency by giving it a
 batch of pages to be compressed. As the bpftrace data shows, patches 6-7 alone
 are not sufficient for the gains. Batch compression added in patch 8 benefits
 latency significantly, for these reasons:

  1) Per-CPU sg_table with ZSWAP_MAX_BATCH_SIZE scatterlists that are statically
     mapped to compression dst buffers for that CPU, thereby minimizing local SG
     lists initialization, and using compact new crypto_acomp API introduced in
     patches 1-4 that set up an acomp_req's src/dst to be SG lists.

  2) Patch 6 allocates ZSWAP_MAX_BATCH_SIZE destination buffers in the per-CPU
     acomp_ctx, for an added memory overhead of ~12 KB per-CPU. The trade-off pays
     off in significant latency improvements which can potentially be used to
     drive higher memory over-commit in data center setups.

  3) A one-shot zswap_update_entries() does all the zsmalloc handle allocation
     and copying for the batch. The key concept driving efficiency is that the
     working set is kept focused on batch-centric data pertaining to a handful
     of operations on the batch, to minimize cache evictions and benefit from
     cache locality.

  4) By making zswap_compress() more efficient, we are able to lock the comp
     mutex per batch of ZSWAP_MAX_BATCH_SIZE (4) pages instead of per-page.
     
 The exceptions are p50/p95 batch compress latencies (normalized per-page), even
 though the p99 batch compress latency improves significantly, by 15% over
 mm-unstable. Batch compress latency can be optimized further in subsequent
 patches.

[1]: https://patchwork.kernel.org/project/linux-mm/cover/20260125033537.334628-1-kanchana.p.sridhar@intel.com/


Changes since v3:
=================
1) Rebased to latest mm-unstable as of 11-17-2026, commit 866f761fd414.

2) Sashiko's v3 code review comments addressed in v4:

   - Patch 7:
     - Modified obj_cgroup_charge_zswap() to call obj_cgroup_charge_zswap_many()
       to avoid the charging code duplication.
     - memcontrol.c: Added a stub for obj_cgroup_get_many() in the #else section
       of the #ifdef CONFIG_MEMCG block.
     - We skip the incompressible page updates in zswap_tree_stores() and
       obj_cgroup_charge_zswap_many() if the nr_incomp_pages is 0.

  - Patch 8:
    - Clamp nr_batch_pages to zc->nr_pages - batch_start on each
      iteration over zc->nr_pages in zswap_compress(). Calculate slen
      accordingly, before proceeding to use slen for the subsequent crypto()
      API calls.
    - In the post-crypto_acomp_compress() !wb_enabled block that checks for
      negative errors codes in sg->length, cast sg->length to (int) before
      checking for sg->length <= 0.

3) Gathered high-level metrics (Sys time, throughput) as suggested by Yosry.


Changes since v2:
=================

Resolved v2 code review issues highlighted by Sashiko. Baseline mm-unstable as
of 10-25-2026, commit a2abad0d7f45.

1) Patch 3:
  - Re-wrote the fast path conditional to be !acomp_is_async(tfm) instead of
    !acomp_req_on_stack(req).
  - Moved the state-flag changes from acomp_save_req() back to
    acomp_virt_to_sg().

2) Patch 4:
  - Replaced the ambiguous use of "cra_flags & CRYPTO_ALG_TYPE_ACOMPRESS" with
    directly checkinf for "cra_type == &crypto_acomp_type" to detect an
    acomp_alg.

3) Patch 5:
  - Fixed the off-by-one bitfield width by modifying
    ZSWAP_MAX_COMPR_LENGTH_SHIFT to 19U and the build time assertion check for
    BUILD_BUG_ON(ZSWAP_MAX_COMPR_LENGTH_SHIFT <= PAGE_SHIFT)).

4) Patch 6:
  - Moved the "u8 compr_batch_size" to struct zswap_comp_ctx, which is
    per-CPU to address the "unsynchronized write to pool->compr_batch_size on
    CPU hotplug" issue Sashiko highlighted.

5) Patch 7:
  - The special casing for comp_ret=0 with dlen == PAGE_SIZE (among other
    end-cases) when writeback is disabled, is replicated in the batched
    version of zswap_compress() as well.

6) Patch 8:
  - Added obj_cgroup_charge_zswap_many() to correctly account for MEMCG_ZSWAP*
    counters with bulk-charging total compressed bytes to zswap.
  - Resolved zsmalloc() handles leak issue in case of compress batch failures
    in zswap_update_entries().
  - Deleted zswap_pool_get().


Changes since v1:
=================
1) Rebased to mm-unstable as of 10-25-2026, commit a2abad0d7f45.

2) Addressed Herbert's comments on checking for the req->unit_size being equal
   to the req->slen to identify singleton acomp_reqs and to quickly
   short-circuit with setting the dst SG list length instead of going through
   the request chaining flow. Further, the most common code path is the very
   first check, with the short-circuit, in crypto_acomp_compress().
   
3) Addressed Yosry's comments wrt:
   - struct zswap_entry "length" field build-time assertion to have sufficient
     bit-fields for the architecture's PAGE_SIZE.
   - Use kmem_cache_alloc() instead of kmem_cache_alloc_node() in
     zswap_entry_cache_alloc().

4) Established a reliable and accurate performance validation methodology, based
   on Yosry's comments.

5) Refactored the sub-components of zswap_store_pages() where each routine
   processes a batch. The new zswap_account_stores() gets refs, updates
   statistics and charges cgroup for the batch, rather than for a page at a
   time. zswap_store_pages() is more readable and maintainable. This batching
   framework can be immediately useful for implementing more ideas such as SLUB
   bulk allocation of compressed data, reclaiming a batch of folios from
   shrink_folio_list(), etc.


Changes since [1]:
==================
1) Better cacheline placement of struct zswap_pool and struct zswap_entry
   members for temporal memory access efficiency in zswap store/load hot paths,
   while minimizing holes.

2) Refactored the core compression batching functionality to a separate
   __zswap_compress() function, which is called by zswap_compress(), as per
   Nhat's and Yosry's comments. Hopefully this helps explain the outer "store
   batching" zswap_compress() vis-a-vis the inner "compress batching"
   __zswap_compress() distinction.

3) The acomp_ctx is locked once and is passed to the routine that compresses a
   batch, per the algorithm's batch-size. Similarly, the folio's node id and
   writeback_enabled status are computed once and passed to the routines that
   implement the different store functionalities, to save latency by avoid
   localized recomputes of information that doesn't change.

4) Moved the folio node id being stored in zswap_entry to a separate patch, per
   Yosry's comment.

5) Addressed the kunmap_local() issue flagged by Jixin He in the
   IS_ERR_VALUE(handle) error path
   (https://patchwork.kernel.org/comment/27095925) - thanks!

6) Moved comments inline, per Yosry's suggestions.


Thanks,
Kanchana


Kanchana P. Sridhar (8):
  crypto: acomp - Define a unit_size in struct acomp_req to enable
    batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE.
  crypto: acomp - Add bit to indicate segmentation support.
  crypto: acomp - Add trivial segmentation wrapper.
  crypto: acomp - Add API to get a compression algorithm's batch-size.
  mm: zswap: Store the folio's node id in the zswap_entry.
  mm: zswap: Store large folios in batches.
  mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
  mm: zswap: Batched zswap_compress() for storing large folios in
    batches.

 crypto/acompress.c                  |  48 +-
 include/crypto/acompress.h          |  66 ++-
 include/crypto/algapi.h             |   5 +
 include/crypto/internal/acompress.h |  10 +-
 include/linux/crypto.h              |   5 +-
 include/linux/memcontrol.h          |  12 +
 mm/memcontrol.c                     |  23 +-
 mm/zswap.c                          | 667 +++++++++++++++++++++-------
 8 files changed, 662 insertions(+), 174 deletions(-)

-- 
2.39.5


^ permalink raw reply	[flat|nested] 25+ messages in thread

* [PATCH v4 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE.
  2026-11-20  4:18 [PATCH v4 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
@ 2026-11-20  4:18 ` Kanchana P. Sridhar
  2026-11-20  4:19 ` [PATCH v4 2/8] crypto: acomp - Add bit to indicate segmentation support Kanchana P. Sridhar
                   ` (7 subsequent siblings)
  8 siblings, 0 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20  4:18 UTC (permalink / raw)
  To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
	davem, linux-crypto, kanchanapsridhar2026, linux-kernel
  Cc: rientjes, willy, senozhatsky

We add a new @unit_size data member to struct acomp_req along with a
helper function acomp_request_set_unit_size() for kernel modules to set
the unit size to use while breaking down the request's src/dst
scatterlists.

An acomp_alg can implement batching by using the @req->unit_size to
break down the SG lists passed in via @req->dst and/or @req->src, to
submit individual @req->slen/@req->unit_size compress jobs or
@req->dlen/@req->unit_size decompress jobs, for batch compression and
batch decompression respectively.

In case of batch compression, the folio's pages for the batch can be
retrieved from the @req->src scatterlist by using a struct sg_page_iter
after determining the number of pages as @req->slen/@req->unit_size.

Please note that acomp_request_set_callback() sets the @req->unit_size
to 0.

In zswap_acomp_req_init(), after the call to
zswap_set_acomp_req_callback(), we set the unit size for zswap to
PAGE_SIZE.

      acomp_request_set_unit_size(areq->req, PAGE_SIZE);

Suggested-by: Herbert Xu <herbert@gondor.apana.org.au>
Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
 include/crypto/acompress.h | 49 ++++++++++++++++++++++++++++++++++++++
 mm/zswap.c                 |  6 +++++
 2 files changed, 55 insertions(+)

diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index 5d5358dfab73..6462fc46f4ec 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -79,6 +79,7 @@ struct acomp_req_chain {
  * @dvirt:	Destination virtual address
  * @slen:	Size of the input buffer
  * @dlen:	Size of the output buffer and number of bytes produced
+ * @unit_size:	Unit size for the request for use in batching
  * @chain:	Private API code data, do not use
  * @__ctx:	Start of private context data
  */
@@ -94,6 +95,7 @@ struct acomp_req {
 	};
 	unsigned int slen;
 	unsigned int dlen;
+	unsigned int unit_size;
 
 	struct acomp_req_chain chain;
 
@@ -329,9 +331,56 @@ static inline void acomp_request_set_callback(struct acomp_req *req,
 {
 	flgs &= ~CRYPTO_ACOMP_REQ_PRIVATE;
 	flgs |= req->base.flags & CRYPTO_ACOMP_REQ_PRIVATE;
+	req->unit_size = 0;
 	crypto_request_set_callback(&req->base, flgs, cmpl, data);
 }
 
+/**
+ * acomp_request_set_unit_size() -- Sets the unit size for the request.
+ *
+ * This is a helper function that enables batching for zswap, IPComp, etc.
+ * It allows multiple independent compression (or decompression) operations to
+ * be submitted in a single request's SG lists, where each SG list ("segment")
+ * is processed independently. The unit size helps derive segments from a
+ * single request. crypto_acomp does not expect the segments to be related in
+ * any way.
+ *
+ * Example usage model:
+ *
+ * A module such as zswap that's configured to use a batching compressor, can
+ * accomplish batch compression of "nr_pages" with crypto_acomp by creating an
+ * output SG table for the batch, initialized to contain "nr_pages" SG
+ * lists. Each scatterlist is mapped to the nth destination buffer for the
+ * batch. Depending on whether the @req is used for batch compress/decompress,
+ * zswap must set the @req's source/destination length to be
+ * "nr_pages * @req->unit_size" respectively.
+ *
+ * An acomp_alg can implement batch compression by using the @req->unit_size
+ * to break down the SG lists passed in via @req->dst to submit individual
+ * "@req->slen/@req->unit_size" compress jobs to be processed as a batch.
+ *
+ * Similarly, zswap can implement batch decompression by passing an
+ * SG table with "nr_pages" SG lists via @req->src to process
+ * "@req->dlen/@req->unit_size" decompress jobs as a batch.
+ *
+ * This API must be called after acomp_request_set_callback(),
+ * which sets @req->unit_size to 0. This makes it easy for users of
+ * crypto_acomp to rely on a default of not opting in to batching.
+ * Users such as zswap opt in to batching by defining @req->unit_size
+ * to a non-zero value for use by acomp_algs supporting batching.
+ *
+ * @du would be PAGE_SIZE for zswap, it could be the MTU for IPsec.
+ *
+ * @req:	asynchronous compress/decompress request
+ * @du:		data unit size of the input/output buffer scatterlist.
+ */
+static inline void acomp_request_set_unit_size(struct acomp_req *req,
+					       unsigned int du)
+{
+	req->unit_size = du;
+}
+
+
 /**
  * acomp_request_set_params() -- Sets request parameters
  *
diff --git a/mm/zswap.c b/mm/zswap.c
index a2e0314db1cf..9a323862bd59 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -829,6 +829,12 @@ static int zswap_acomp_req_init(struct zswap_acomp_req *areq,
 	crypto_init_wait(&areq->wait);
 	zswap_set_acomp_req_callback(areq->req, &areq->wait);
 
+	/*
+	 * This should be called after acomp_request_set_callback(),
+	 * which initializes req->unit_size to 0.
+	 */
+	acomp_request_set_unit_size(areq->req, PAGE_SIZE);
+
 	mutex_init(&areq->mutex);
 	return 0;
 }
-- 
2.39.5


^ permalink raw reply related	[flat|nested] 25+ messages in thread

* [PATCH v4 2/8] crypto: acomp - Add bit to indicate segmentation support.
  2026-11-20  4:18 [PATCH v4 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
  2026-11-20  4:18 ` [PATCH v4 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
@ 2026-11-20  4:19 ` Kanchana P. Sridhar
  2026-11-20  4:19 ` [PATCH v4 3/8] crypto: acomp - Add trivial segmentation wrapper Kanchana P. Sridhar
                   ` (6 subsequent siblings)
  8 siblings, 0 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20  4:19 UTC (permalink / raw)
  To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
	davem, linux-crypto, kanchanapsridhar2026, linux-kernel
  Cc: rientjes, willy, senozhatsky

This patch adds segmentation support for compression.

Add a bit to the crypto_alg flags to indicate support for segmentation.
Also add a helper for acomp to test whether a given tfm supports
segmentation.

Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
 include/crypto/algapi.h             | 5 +++++
 include/crypto/internal/acompress.h | 5 +++++
 include/linux/crypto.h              | 3 +++
 3 files changed, 13 insertions(+)

diff --git a/include/crypto/algapi.h b/include/crypto/algapi.h
index 05deea9dac5e..7d406cfe5751 100644
--- a/include/crypto/algapi.h
+++ b/include/crypto/algapi.h
@@ -280,6 +280,11 @@ static inline bool crypto_tfm_req_virt(struct crypto_tfm *tfm)
 	return tfm->__crt_alg->cra_flags & CRYPTO_ALG_REQ_VIRT;
 }
 
+static inline bool crypto_tfm_req_seg(struct crypto_tfm *tfm)
+{
+	return tfm->__crt_alg->cra_flags & CRYPTO_ALG_REQ_SEG;
+}
+
 static inline u32 crypto_request_flags(struct crypto_async_request *req)
 {
 	return req->flags & ~CRYPTO_TFM_REQ_ON_STACK;
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 9cd37df32dc4..93ae59623f13 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -189,6 +189,11 @@ static inline bool crypto_acomp_req_virt(struct crypto_acomp *tfm)
 	return crypto_tfm_req_virt(&tfm->base);
 }
 
+static inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm)
+{
+	return crypto_tfm_req_seg(&tfm->base);
+}
+
 void crypto_acomp_free_streams(struct crypto_acomp_streams *s);
 int crypto_acomp_alloc_streams(struct crypto_acomp_streams *s);
 
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index b7c97f1c47c9..78c699ac3f82 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -139,6 +139,9 @@
 /* Set if the algorithm cannot have a fallback (e.g., phmac). */
 #define CRYPTO_ALG_NO_FALLBACK		0x00080000
 
+/* Set if the algorithm supports segmentation. */
+#define CRYPTO_ALG_REQ_SEG		0x00100000
+
 /* The high bits 0xff000000 are reserved for type-specific flags. */
 
 /*
-- 
2.39.5


^ permalink raw reply related	[flat|nested] 25+ messages in thread

* [PATCH v4 3/8] crypto: acomp - Add trivial segmentation wrapper.
  2026-11-20  4:18 [PATCH v4 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
  2026-11-20  4:18 ` [PATCH v4 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
  2026-11-20  4:19 ` [PATCH v4 2/8] crypto: acomp - Add bit to indicate segmentation support Kanchana P. Sridhar
@ 2026-11-20  4:19 ` Kanchana P. Sridhar
  2026-11-20  4:19 ` [PATCH v4 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
                   ` (5 subsequent siblings)
  8 siblings, 0 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20  4:19 UTC (permalink / raw)
  To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
	davem, linux-crypto, kanchanapsridhar2026, linux-kernel
  Cc: rientjes, willy, senozhatsky

This patch provides a wrapper for existing algorithms so that they can
accept a single segment while returning the compressed length or error
through the dst SG list length.

This trivial segmentation wrapper only supports compression with a
segment count of exactly one.

The reason is that the first user zswap will only allocate the
extra memory if the underlying algorithm supports segmentation,
and otherwise only one segment will be given at a time.

Having this wrapper means that the same calling convention can
be used for all algorithms, regardless of segmentation support.

To gain latency in the zswap_store() hot path impacted by this patch:

1) The most common code path is invoked first in
   crypto_acomp_compress().

2) The helper functions used in (1) are made __always_inline.

Co-developed-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Herbert Xu <herbert@gondor.apana.org.au>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
 crypto/acompress.c                  | 34 ++++++++++++++++++++++++++---
 include/crypto/acompress.h          |  5 +++--
 include/crypto/internal/acompress.h |  4 ++--
 include/linux/crypto.h              |  2 +-
 4 files changed, 37 insertions(+), 8 deletions(-)

diff --git a/crypto/acompress.c b/crypto/acompress.c
index c8b4b8589769..ac4dc52668b8 100644
--- a/crypto/acompress.c
+++ b/crypto/acompress.c
@@ -166,14 +166,18 @@ static void acomp_save_req(struct acomp_req *req, crypto_completion_t cplt)
 
 	state->compl = req->base.complete;
 	state->data = req->base.data;
+	state->unit_size = req->unit_size;
+
 	req->base.complete = cplt;
 	req->base.data = req;
+	req->unit_size = 0;
 }
 
 static void acomp_restore_req(struct acomp_req *req)
 {
 	req->base.complete = req->chain.compl;
 	req->base.data = req->chain.data;
+	req->unit_size = req->chain.unit_size;
 }
 
 static void acomp_reqchain_virt(struct acomp_req *req)
@@ -242,6 +246,10 @@ static int acomp_reqchain_finish(struct acomp_req *req, int err)
 {
 	acomp_reqchain_virt(req);
 	acomp_restore_req(req);
+
+	if (req->unit_size)
+		req->dst->length = unlikely(err) ? err : req->dlen;
+
 	return err;
 }
 
@@ -266,10 +274,13 @@ static int acomp_do_req_chain(struct acomp_req *req, bool comp)
 {
 	int err;
 
+	if (unlikely(req->unit_size && req->slen > req->unit_size))
+		return -ENOSYS;
+
 	acomp_save_req(req, acomp_reqchain_done);
 
 	err = acomp_do_one_req(req, comp);
-	if (err == -EBUSY || err == -EINPROGRESS)
+	if (unlikely(err == -EBUSY || err == -EINPROGRESS))
 		return err;
 
 	return acomp_reqchain_finish(req, err);
@@ -279,10 +290,27 @@ int crypto_acomp_compress(struct acomp_req *req)
 {
 	struct crypto_acomp *tfm = crypto_acomp_reqtfm(req);
 
+	if (likely((req->unit_size == req->slen) && acomp_request_issg(req) &&
+		    !crypto_acomp_req_seg(tfm) && !acomp_is_async(tfm))) {
+		int err = tfm->compress(req);
+
+		req->dst->length = unlikely(err) ? err : req->dlen;
+		return err;
+	}
+
 	if (acomp_req_on_stack(req) && acomp_is_async(tfm))
 		return -EAGAIN;
+
+	if (req->unit_size && acomp_request_issg(req)) {
+		if (!crypto_acomp_req_seg(tfm))
+			return acomp_do_req_chain(req, true);
+
+		return tfm->compress(req);
+	}
+
 	if (crypto_acomp_req_virt(tfm) || acomp_request_issg(req))
-		return crypto_acomp_reqtfm(req)->compress(req);
+		return tfm->compress(req);
+
 	return acomp_do_req_chain(req, true);
 }
 EXPORT_SYMBOL_GPL(crypto_acomp_compress);
@@ -294,7 +322,7 @@ int crypto_acomp_decompress(struct acomp_req *req)
 	if (acomp_req_on_stack(req) && acomp_is_async(tfm))
 		return -EAGAIN;
 	if (crypto_acomp_req_virt(tfm) || acomp_request_issg(req))
-		return crypto_acomp_reqtfm(req)->decompress(req);
+		return tfm->decompress(req);
 	return acomp_do_req_chain(req, false);
 }
 EXPORT_SYMBOL_GPL(crypto_acomp_decompress);
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index 6462fc46f4ec..93fa02ce818b 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -67,6 +67,7 @@ struct acomp_req_chain {
 		struct folio *dfolio;
 	};
 	u32 flags;
+	u32 unit_size;
 };
 
 /**
@@ -199,7 +200,7 @@ static inline void acomp_request_set_tfm(struct acomp_req *req,
 	crypto_request_set_tfm(&req->base, crypto_acomp_tfm(tfm));
 }
 
-static inline bool acomp_is_async(struct crypto_acomp *tfm)
+static __always_inline bool acomp_is_async(struct crypto_acomp *tfm)
 {
 	return crypto_comp_alg_common(tfm)->base.cra_flags &
 	       CRYPTO_ALG_ASYNC;
@@ -294,7 +295,7 @@ static inline void *acomp_request_extra(struct acomp_req *req)
 	return (void *)((char *)req + len);
 }
 
-static inline bool acomp_req_on_stack(struct acomp_req *req)
+static __always_inline bool acomp_req_on_stack(struct acomp_req *req)
 {
 	return crypto_req_on_stack(&req->base);
 }
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 93ae59623f13..0c0d3fc8a9fc 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -146,7 +146,7 @@ void crypto_unregister_acomp(struct acomp_alg *alg);
 int crypto_register_acomps(struct acomp_alg *algs, int count);
 void crypto_unregister_acomps(struct acomp_alg *algs, int count);
 
-static inline bool acomp_request_issg(struct acomp_req *req)
+static __always_inline bool acomp_request_issg(struct acomp_req *req)
 {
 	return !(req->base.flags & (CRYPTO_ACOMP_REQ_SRC_VIRT |
 				    CRYPTO_ACOMP_REQ_DST_VIRT));
@@ -189,7 +189,7 @@ static inline bool crypto_acomp_req_virt(struct crypto_acomp *tfm)
 	return crypto_tfm_req_virt(&tfm->base);
 }
 
-static inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm)
+static __always_inline bool crypto_acomp_req_seg(struct crypto_acomp *tfm)
 {
 	return crypto_tfm_req_seg(&tfm->base);
 }
diff --git a/include/linux/crypto.h b/include/linux/crypto.h
index 78c699ac3f82..0628b62e4946 100644
--- a/include/linux/crypto.h
+++ b/include/linux/crypto.h
@@ -492,7 +492,7 @@ static inline bool crypto_tfm_is_async(struct crypto_tfm *tfm)
 	return tfm->__crt_alg->cra_flags & CRYPTO_ALG_ASYNC;
 }
 
-static inline bool crypto_req_on_stack(struct crypto_async_request *req)
+static __always_inline bool crypto_req_on_stack(struct crypto_async_request *req)
 {
 	return req->flags & CRYPTO_TFM_REQ_ON_STACK;
 }
-- 
2.39.5


^ permalink raw reply related	[flat|nested] 25+ messages in thread

* [PATCH v4 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size.
  2026-11-20  4:18 [PATCH v4 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
                   ` (2 preceding siblings ...)
  2026-11-20  4:19 ` [PATCH v4 3/8] crypto: acomp - Add trivial segmentation wrapper Kanchana P. Sridhar
@ 2026-11-20  4:19 ` Kanchana P. Sridhar
  2026-11-21  4:20   ` sashiko-bot
  2026-11-20  4:19 ` [PATCH v4 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
                   ` (4 subsequent siblings)
  8 siblings, 1 reply; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20  4:19 UTC (permalink / raw)
  To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
	davem, linux-crypto, kanchanapsridhar2026, linux-kernel
  Cc: rientjes, willy, senozhatsky

This commit adds a @batch_size data member to struct acomp_alg.

An acomp_alg compression algorithm that supports batching of
compressions and decompressions must provide a @batch_size greater
than 1, representing the maximum batch-size that the compressor
supports, so that kernel users of crypto_acomp, such as zswap, can
allocate resources for submitting multiple compress/decompress jobs that
can be batched, and invoke batching of [de]compressions.

The new crypto_acomp_batch_size() API queries the crypto_acomp's
acomp_alg for the batch-size. If the acomp_alg has registered a
@batch_size greater than 1, this is returned. If not, a default of "1"
is returned.

zswap can invoke crypto_acomp_batch_size() to query the maximum number
of requests that can be batch [de]compressed. Based on this, zswap
can use the minimum of any zswap-specific upper limits for batch-size
and the compressor's max @batch_size, to allocate batching resources.

This enables zswap to compress/decompress multiple pages as a batch
to improve swapout/swapin performance and memory savings.

Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
 crypto/acompress.c                  | 14 ++++++++++++++
 include/crypto/acompress.h          | 12 ++++++++++++
 include/crypto/internal/acompress.h |  3 +++
 3 files changed, 29 insertions(+)

diff --git a/crypto/acompress.c b/crypto/acompress.c
index ac4dc52668b8..df9e42f519c3 100644
--- a/crypto/acompress.c
+++ b/crypto/acompress.c
@@ -327,6 +327,20 @@ int crypto_acomp_decompress(struct acomp_req *req)
 }
 EXPORT_SYMBOL_GPL(crypto_acomp_decompress);
 
+unsigned int crypto_acomp_batch_size(struct crypto_acomp *tfm)
+{
+	if (acomp_is_async(tfm) &&
+	    (crypto_comp_alg_common(tfm)->base.cra_type == &crypto_acomp_type)) {
+		struct acomp_alg *alg = crypto_acomp_alg(tfm);
+
+		if (alg && alg->batch_size > 1)
+			return alg->batch_size;
+	}
+
+	return 1;
+}
+EXPORT_SYMBOL_GPL(crypto_acomp_batch_size);
+
 void comp_prepare_alg(struct comp_alg_common *alg)
 {
 	struct crypto_alg *base = &alg->base;
diff --git a/include/crypto/acompress.h b/include/crypto/acompress.h
index 93fa02ce818b..3ce4704dc879 100644
--- a/include/crypto/acompress.h
+++ b/include/crypto/acompress.h
@@ -593,6 +593,18 @@ int crypto_acomp_compress(struct acomp_req *req);
  */
 int crypto_acomp_decompress(struct acomp_req *req);
 
+/**
+ * crypto_acomp_batch_size() -- Get the algorithm's batch size
+ *
+ * Function returns the algorithm's batch size for batching operations
+ *
+ * @tfm:	ACOMPRESS tfm handle allocated with crypto_alloc_acomp()
+ *
+ * Return:	@tfm's acomp_alg's @batch_size, if it has defined a
+ *		@batch_size greater than 1; else return 1.
+ */
+unsigned int crypto_acomp_batch_size(struct crypto_acomp *tfm);
+
 static inline struct acomp_req *acomp_request_on_stack_init(
 	char *buf, struct crypto_acomp *tfm)
 {
diff --git a/include/crypto/internal/acompress.h b/include/crypto/internal/acompress.h
index 0c0d3fc8a9fc..2d4d302d1b3f 100644
--- a/include/crypto/internal/acompress.h
+++ b/include/crypto/internal/acompress.h
@@ -28,6 +28,8 @@
  *
  * @compress:	Function performs a compress operation
  * @decompress:	Function performs a de-compress operation
+ * @batch_size:	Maximum batch-size for batching compress/decompress
+ *		operations.
  * @init:	Initialize the cryptographic transformation object.
  *		This function is used to initialize the cryptographic
  *		transformation object. This function is called only once at
@@ -47,6 +49,7 @@
 struct acomp_alg {
 	int (*compress)(struct acomp_req *req);
 	int (*decompress)(struct acomp_req *req);
+	unsigned int batch_size;
 	int (*init)(struct crypto_acomp *tfm);
 	void (*exit)(struct crypto_acomp *tfm);
 
-- 
2.39.5


^ permalink raw reply related	[flat|nested] 25+ messages in thread

* [PATCH v4 5/8] mm: zswap: Store the folio's node id in the zswap_entry.
  2026-11-20  4:18 [PATCH v4 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
                   ` (3 preceding siblings ...)
  2026-11-20  4:19 ` [PATCH v4 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
@ 2026-11-20  4:19 ` Kanchana P. Sridhar
  2026-11-20 22:46   ` Nhat Pham
  2026-11-21  4:20   ` sashiko-bot
  2026-11-20  4:19 ` [PATCH v4 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
                   ` (3 subsequent siblings)
  8 siblings, 2 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20  4:19 UTC (permalink / raw)
  To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
	davem, linux-crypto, kanchanapsridhar2026, linux-kernel
  Cc: rientjes, willy, senozhatsky

When storing a folio in zswap, save the folio's node id in the zswap
entry instead of relying on the zswap_entry being allocated on the same
node. This enables optimizations such as batch allocation of zswap
entries for large folios using kmem_cache_alloc_bulk() that does not
support specifying a node id.

Accordingly, replace the SLUB allocation in zswap_entry_cache_alloc() to
be node-agnostic.

The size of the zswap_entry remains unchanged as 'referenced' is lumped
in with the 'length' (as it doesn't need a full unsigned int anyway).
A build time assertion checks for the zswap_entry's 'length' bitfield
being large enough to represent PAGE_SIZE, which can be 256KB on some
architectures.

Suggested-by: Nhat Pham <nphamcs@gmail.com>
Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
 mm/zswap.c | 33 ++++++++++++++++++---------------
 1 file changed, 18 insertions(+), 15 deletions(-)

diff --git a/mm/zswap.c b/mm/zswap.c
index 9a323862bd59..78d9694f75b5 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -181,6 +181,7 @@ struct zswap_pool {
  */
 #define ZSWAP_FIRST_POOL_ID 1
 #define ZSWAP_MAX_POOL_ID U8_MAX
+#define ZSWAP_MAX_COMPR_LENGTH_SHIFT 19U
 static DEFINE_XARRAY_FLAGS(zswap_pools, XA_FLAGS_ALLOC1 | XA_FLAGS_LOCK_BH);
 static struct zswap_pool __rcu *zswap_current_pool;
 
@@ -207,15 +208,19 @@ static struct shrinker *zswap_shrinker;
  *              logic if referenced is unset. See comments in the shrinker
  *              section for context.
  * pool_idx - id of the zswap_pool that the entry's data is in.
+ * nid - NUMA node id of the page for which this is the zswap entry.
  * handle - zsmalloc allocation handle that stores the compressed page data
  * objcg - the obj_cgroup that the compressed memory is charged to
  * lru - handle to the pool's lru used to evict pages.
  */
 struct zswap_entry {
 	swp_entry_t swpentry;
-	unsigned int length;
-	bool referenced;
-	u8 pool_idx;
+	struct {
+		unsigned int length:ZSWAP_MAX_COMPR_LENGTH_SHIFT;
+		bool referenced:1U;
+		u8 pool_idx;
+	};
+	int nid;
 	struct zs_handle handle;
 	struct obj_cgroup *objcg;
 	struct list_head lru;
@@ -680,14 +685,8 @@ static int zswap_enabled_param_set(const char *val,
 * lru functions
 **********************************/
 
-static inline int entry_to_nid(struct zswap_entry *entry)
-{
-	return page_to_nid(virt_to_page(entry));
-}
-
 static void zswap_lru_add(struct zswap_entry *entry)
 {
-	int nid = entry_to_nid(entry);
 	struct mem_cgroup *memcg;
 
 	/*
@@ -704,19 +703,18 @@ static void zswap_lru_add(struct zswap_entry *entry)
 	rcu_read_lock();
 	memcg = obj_cgroup_memcg(entry->objcg);
 	/* will always succeed */
-	list_lru_add(&zswap_list_lru, &entry->lru, nid, memcg);
+	list_lru_add(&zswap_list_lru, &entry->lru, entry->nid, memcg);
 	rcu_read_unlock();
 }
 
 static void zswap_lru_del(struct zswap_entry *entry)
 {
-	int nid = entry_to_nid(entry);
 	struct mem_cgroup *memcg;
 
 	rcu_read_lock();
 	memcg = obj_cgroup_memcg(entry->objcg);
 	/* will always succeed */
-	list_lru_del(&zswap_list_lru, &entry->lru, nid, memcg);
+	list_lru_del(&zswap_list_lru, &entry->lru, entry->nid, memcg);
 	rcu_read_unlock();
 }
 
@@ -764,10 +762,11 @@ void zswap_memcg_offline_cleanup(struct mem_cgroup *memcg)
 **********************************/
 static struct kmem_cache *zswap_entry_cache;
 
-static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp, int nid)
+static struct zswap_entry *zswap_entry_cache_alloc(gfp_t gfp)
 {
 	struct zswap_entry *entry;
-	entry = kmem_cache_alloc_node(zswap_entry_cache, gfp, nid);
+
+	entry = kmem_cache_alloc(zswap_entry_cache, gfp);
 	if (!entry)
 		return NULL;
 	zs_handle_init(&entry->handle);
@@ -1537,9 +1536,10 @@ static bool zswap_store_page(struct folio *folio, long index,
 {
 	swp_entry_t page_swpentry = folio_swap_entry(folio, index);
 	struct zswap_entry *entry, *old;
+	int nid = folio_nid(folio);
 
 	/* allocate entry */
-	entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio));
+	entry = zswap_entry_cache_alloc(GFP_KERNEL);
 	if (!entry) {
 		zswap_reject_kmemcache_fail++;
 		return false;
@@ -1554,6 +1554,7 @@ static bool zswap_store_page(struct folio *folio, long index,
 	 * to an unrelated live pool.
 	 */
 	entry->pool_idx = pool->idx;
+	entry->nid = nid;
 
 	old = xa_store(swap_zswap_tree(page_swpentry),
 		       swp_offset(page_swpentry),
@@ -1930,6 +1931,8 @@ static int zswap_setup(void)
 	struct zswap_pool *pool;
 	int ret;
 
+	BUILD_BUG_ON(ZSWAP_MAX_COMPR_LENGTH_SHIFT <= PAGE_SHIFT);
+
 	zswap_entry_cache = KMEM_CACHE(zswap_entry, 0);
 	if (!zswap_entry_cache) {
 		pr_err("entry cache creation failed\n");
-- 
2.39.5


^ permalink raw reply related	[flat|nested] 25+ messages in thread

* [PATCH v4 6/8] mm: zswap: Store large folios in batches.
  2026-11-20  4:18 [PATCH v4 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
                   ` (4 preceding siblings ...)
  2026-11-20  4:19 ` [PATCH v4 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
@ 2026-11-20  4:19 ` Kanchana P. Sridhar
  2026-11-20 23:08   ` Nhat Pham
  2026-11-20 23:25   ` Nhat Pham
  2026-11-20  4:19 ` [PATCH v4 7/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations Kanchana P. Sridhar
                   ` (2 subsequent siblings)
  8 siblings, 2 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20  4:19 UTC (permalink / raw)
  To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
	davem, linux-crypto, kanchanapsridhar2026, linux-kernel
  Cc: rientjes, willy, senozhatsky

Use batching when storing large folios in zswap. A new constant,
ZSWAP_MAX_BATCH_SIZE is defined as 4U to support processing batches of
up to 4 pages at a time while storing large folios. This commit
modifies the "store single page" approach to "store a batch of pages".

As part of CPU onlining per-CPU acomp_ctx setup, ZSWAP_MAX_BATCH_SIZE
compression buffers are allocated to eventually be able to compress and
store ZSWAP_MAX_BATCH_SIZE pages. This happens in subsequent commits.

For existing software compressors, the extra overhead is the
the extra 'buffers' pointer, 3 additional elements in 'buffers' and 3
additional 4K space per 'buffers[i]': in all ~12 KB per-CPU on
x86_64. Only the first buffer is currently used.

zswap_store_page() is replaced with zswap_store_pages(), which processes
a batch of pages and allows for batching optimizations. For now, only
optimize allocating entries by using batch allocations from the slab
cache.

Avoid repeatedly calling mem_cgroup_zswap_writeback_enabled() for every
page and only call it once for the folio, since the entire folio is
charged to a single memcg. Similarly, obtain the nid once to use during
each batch store of a folio. To facilitate this, a "struct
zswap_store_context" is introduced to keep function parameter lists
concise.

Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
 mm/zswap.c | 307 +++++++++++++++++++++++++++++++++++++----------------
 1 file changed, 217 insertions(+), 90 deletions(-)

diff --git a/mm/zswap.c b/mm/zswap.c
index 78d9694f75b5..df59ab0c1029 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -85,6 +85,9 @@ static bool zswap_pool_reached_full;
 
 #define ZSWAP_PARAM_UNSET ""
 
+/* Limit the batch size to limit per-CPU memory usage for dst buffers. */
+#define ZSWAP_MAX_BATCH_SIZE 4U
+
 static int zswap_setup(void);
 
 /* Enable/disable zswap */
@@ -144,10 +147,12 @@ struct zswap_acomp_req {
 	struct mutex mutex;
 };
 
-/* The compression mutex also protects the output buffer. */
+/*
+ * The compression mutex also protects the output buffer(s).
+ */
 struct zswap_comp_ctx {
 	struct zswap_acomp_req areq;
-	u8 *buffer;
+	u8 **buffers;
 };
 
 /* Separate requests, so that decompression does not wait for compression. */
@@ -162,6 +167,7 @@ struct crypto_acomp_ctx {
  * The only case where lru_lock is not acquired while holding tree.lock is
  * when a zswap_entry is taken off the lru for writeback, in that case it
  * needs to be verified that it's still valid in the tree.
+ *
  */
 struct zswap_pool {
 	struct zs_pool *zs_pool;
@@ -226,6 +232,15 @@ struct zswap_entry {
 	struct list_head lru;
 };
 
+struct zswap_store_context {
+	struct folio *folio;
+	long folio_start;
+	bool wb_enabled;
+	int nid;
+	u8 nr_pages;
+	struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE];
+};
+
 /*
  * No RCU section is needed around the returned pointer: a stored entry pins
  * its pool via percpu_ref (taken in zswap_store_page()), so the id cannot be
@@ -284,6 +299,8 @@ static void __zswap_pool_empty(struct percpu_ref *ref);
 
 static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
 {
+	u8 i;
+
 	if (!acomp_ctx)
 		return;
 
@@ -302,8 +319,12 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
 
 	acomp_ctx->acomp = NULL;
 
-	kfree(acomp_ctx->comp.buffer);
-	acomp_ctx->comp.buffer = NULL;
+	if (acomp_ctx->comp.buffers) {
+		for (i = 0; i < ZSWAP_MAX_BATCH_SIZE; ++i)
+			kfree(acomp_ctx->comp.buffers[i]);
+		kfree(acomp_ctx->comp.buffers);
+	}
+	acomp_ctx->comp.buffers = NULL;
 }
 
 static struct zswap_pool *zswap_pool_create(char *compressor)
@@ -388,6 +409,7 @@ static struct zswap_pool *zswap_pool_create(char *compressor)
 cpuhp_add_fail:
 	for_each_possible_cpu(cpu)
 		acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu));
+
 error:
 	if (pool->acomp_ctx)
 		free_percpu(pool->acomp_ctx);
@@ -778,6 +800,44 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)
 	kmem_cache_free(zswap_entry_cache, entry);
 }
 
+static inline void zswap_entries_cache_free_batch(
+	struct zswap_entry **entries,
+	u8 nr_entries)
+{
+	/*
+	 * It is okay to use this to free entries allocated separately
+	 * by zswap_entry_cache_alloc().
+	 */
+	kmem_cache_free_bulk(zswap_entry_cache, nr_entries, (void **)entries);
+}
+
+static bool zswap_entries_cache_alloc_batch(
+	struct zswap_store_context *zc,
+	gfp_t gfp)
+{
+	u8 i;
+
+	bool ret = kmem_cache_alloc_bulk(zswap_entry_cache, gfp,
+					 zc->nr_pages, (void **)zc->entries);
+
+	if (likely(ret)) {
+		for (i = 0; i < zc->nr_pages; ++i)
+			zs_handle_init(&zc->entries[i]->handle);
+	} else {
+		for (i = 0; i < zc->nr_pages; ++i) {
+			zc->entries[i] = zswap_entry_cache_alloc(GFP_KERNEL);
+
+			if (unlikely(!zc->entries[i])) {
+				zswap_reject_kmemcache_fail++;
+				zswap_entries_cache_free_batch(zc->entries, i);
+				return false;
+			}
+		}
+	}
+
+	return true;
+}
+
 /*
  * Carries out the common pattern of freeing an entry's zsmalloc allocation,
  * freeing the entry itself, and decrementing the number of stored pages.
@@ -842,7 +902,9 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
 {
 	struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node);
 	struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu);
+	int nid = cpu_to_node(cpu);
 	int ret = -ENOMEM;
+	u8 i;
 
 	/*
 	 * To handle cases where the CPU goes through online-offline-online
@@ -853,15 +915,11 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
 		return 0;
 	}
 
-	acomp_ctx->comp.buffer = kmalloc_node(PAGE_SIZE, GFP_KERNEL, cpu_to_node(cpu));
-	if (!acomp_ctx->comp.buffer)
-		return ret;
-
 	/*
 	 * In case of an error, crypto_alloc_acomp_node() returns an
 	 * error pointer, never NULL.
 	 */
-	acomp_ctx->acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, cpu_to_node(cpu));
+	acomp_ctx->acomp = crypto_alloc_acomp_node(pool->tfm_name, 0, 0, nid);
 	if (IS_ERR(acomp_ctx->acomp)) {
 		pr_err("could not alloc crypto acomp %s : %pe\n",
 				pool->tfm_name, acomp_ctx->acomp);
@@ -873,6 +931,24 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
 	if (ret)
 		goto req_fail;
 
+	/*
+	 * Allocate up to ZSWAP_MAX_BATCH_SIZE dst buffers for all compressors.
+	 */
+	acomp_ctx->comp.buffers = kcalloc_node(ZSWAP_MAX_BATCH_SIZE,
+					       sizeof(u8 *), GFP_KERNEL, nid);
+	if (!acomp_ctx->comp.buffers) {
+		ret = -ENOMEM;
+		goto req_fail;
+	}
+
+	for (i = 0; i < ZSWAP_MAX_BATCH_SIZE; ++i) {
+		acomp_ctx->comp.buffers[i] = kmalloc_node(PAGE_SIZE, GFP_KERNEL, nid);
+		if (!acomp_ctx->comp.buffers[i]) {
+			ret = -ENOMEM;
+			goto req_fail;
+		}
+	}
+
 	/* Skip the per-CPU request when decompression can use the stack. */
 	if (acomp_is_async(acomp_ctx->acomp)) {
 		ret = zswap_acomp_req_init(&acomp_ctx->decomp, acomp_ctx->acomp);
@@ -890,7 +966,8 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
 }
 
 static bool zswap_compress(struct folio *folio, long index,
-			   struct zswap_entry *entry, struct zswap_pool *pool)
+			   struct zswap_entry *entry, struct zswap_pool *pool,
+			   bool wb_enabled)
 {
 	struct zswap_comp_ctx *comp_ctx;
 	struct scatterlist input, output;
@@ -903,7 +980,7 @@ static bool zswap_compress(struct folio *folio, long index,
 	comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
 	mutex_lock(&comp_ctx->areq.mutex);
 
-	dst = comp_ctx->buffer;
+	dst = comp_ctx->buffers[0];
 	sg_init_table(&input, 1);
 	sg_set_folio(&input, folio, PAGE_SIZE, index * PAGE_SIZE);
 
@@ -942,12 +1019,8 @@ static bool zswap_compress(struct folio *folio, long index,
 	 * to the active LRU list in the case.
 	 */
 	if (comp_ret) {
-		rcu_read_lock();
-		if (!mem_cgroup_zswap_writeback_enabled(folio_memcg(folio))) {
-			rcu_read_unlock();
+		if (!wb_enabled)
 			goto unlock;
-		}
-		rcu_read_unlock();
 		comp_ret = 0;
 		dlen = PAGE_SIZE;
 		dst = kmap_local_folio(folio, index * PAGE_SIZE);
@@ -1530,91 +1603,125 @@ static void shrink_worker(struct work_struct *w)
 * main API
 **********************************/
 
-static bool zswap_store_page(struct folio *folio, long index,
-			     struct obj_cgroup *objcg,
-			     struct zswap_pool *pool)
+/*
+ * Store multiple pages in @zc->folio, starting from the page at index @start up
+ * to the page at index @end-1.
+ */
+static bool zswap_store_pages(struct zswap_store_context *zc,
+			      long start,
+			      long end,
+			      struct zswap_pool *pool,
+			      struct obj_cgroup *objcg)
 {
-	swp_entry_t page_swpentry = folio_swap_entry(folio, index);
-	struct zswap_entry *entry, *old;
-	int nid = folio_nid(folio);
+	u8 i, store_fail_idx = 0;
+	bool ret;
 
-	/* allocate entry */
-	entry = zswap_entry_cache_alloc(GFP_KERNEL);
-	if (!entry) {
-		zswap_reject_kmemcache_fail++;
-		return false;
-	}
+	zc->nr_pages = end - start;
+	zc->folio_start = start;
+
+	VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
 
-	if (!zswap_compress(folio, index, entry, pool))
-		goto compress_failed;
+	ret = zswap_entries_cache_alloc_batch(zc, GFP_KERNEL);
+	if (unlikely(!ret))
+		return false;
 
 	/*
-	 * Set pool_idx before the xa_store() below publishes the entry, or a
-	 * concurrent reader could resolve a stale pool_idx left by slab reuse
-	 * to an unrelated live pool.
+	 * We co-locate entry initialization as much as possible here to
+	 * minimize potential cache misses.
 	 */
-	entry->pool_idx = pool->idx;
-	entry->nid = nid;
-
-	old = xa_store(swap_zswap_tree(page_swpentry),
-		       swp_offset(page_swpentry),
-		       entry, GFP_KERNEL);
-	if (xa_is_err(old)) {
-		int err = xa_err(old);
+	for (i = 0; i < zc->nr_pages; ++i) {
+		/*
+		 * Set pool_idx before the xa_store() below publishes the entry, or a
+		 * concurrent reader could resolve a stale pool_idx left by slab reuse
+		 * to an unrelated live pool.
+		 */
+		zc->entries[i]->pool_idx = pool->idx;
+		zc->entries[i]->swpentry = folio_swap_entry(zc->folio,
+							zc->folio_start + i);
+		zc->entries[i]->objcg = objcg;
+		zc->entries[i]->referenced = true;
+		zc->entries[i]->nid = zc->nid;
+		INIT_LIST_HEAD(&zc->entries[i]->lru);
+	}
 
-		WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err);
-		zswap_reject_alloc_fail++;
-		goto store_failed;
+	for (i = 0; i < zc->nr_pages; ++i) {
+		if (!zswap_compress(zc->folio, zc->folio_start + i,
+				    zc->entries[i], pool, zc->wb_enabled))
+			goto store_pages_failed;
 	}
 
-	/*
-	 * We may have had an existing entry that became stale when
-	 * the folio was redirtied and now the new version is being
-	 * swapped out. Get rid of the old.
-	 */
-	if (old)
-		zswap_entry_free(old);
+	for (i = 0; i < zc->nr_pages; ++i) {
+		struct zswap_entry *old, *entry = zc->entries[i];
+
+		old = xa_store(swap_zswap_tree(entry->swpentry),
+			       swp_offset(entry->swpentry),
+			       entry, GFP_KERNEL);
+		if (unlikely(xa_is_err(old))) {
+			int err = xa_err(old);
+
+			WARN_ONCE(err != -ENOMEM, "unexpected xarray error: %d\n", err);
+			zswap_reject_alloc_fail++;
+			/*
+			 * Entries up to this point have been stored in the
+			 * xarray. zswap_store() will erase them from the xarray
+			 * and call zswap_entry_free(). Local cleanup in
+			 * 'store_pages_failed' only needs to happen for
+			 * entries from [@i to @nr_pages).
+			 */
+			store_fail_idx = i;
+			goto store_pages_failed;
+		}
 
-	/*
-	 * The entry is successfully compressed and stored in the tree, there is
-	 * no further possibility of failure. Grab refs to the pool and objcg,
-	 * charge zswap memory, and increment zswap_stored_pages.
-	 * The opposite actions will be performed by zswap_entry_free()
-	 * when the entry is removed from the tree.
-	 */
-	zswap_pool_get(pool);
-	if (objcg) {
-		obj_cgroup_get(objcg);
-		obj_cgroup_charge_zswap(objcg, entry->length);
-	}
-	atomic_long_inc(&zswap_stored_pages);
-	if (entry->length == PAGE_SIZE)
-		atomic_long_inc(&zswap_stored_incompressible_pages);
+		/*
+		 * We may have had an existing entry that became stale when
+		 * the folio was redirtied and now the new version is being
+		 * swapped out. Get rid of the old.
+		 */
+		if (unlikely(old))
+			zswap_entry_free(old);
 
-	/*
-	 * We finish initializing the entry while it's already in xarray.
-	 * This is safe because:
-	 *
-	 * 1. Concurrent stores and invalidations are excluded by folio lock.
-	 *
-	 * 2. Writeback is excluded by the entry not being on the LRU yet.
-	 *    The publishing order matters to prevent writeback from seeing
-	 *    an incoherent entry.
-	 */
-	entry->swpentry = page_swpentry;
-	entry->objcg = objcg;
-	entry->referenced = true;
-	if (entry->length) {
-		INIT_LIST_HEAD(&entry->lru);
-		zswap_lru_add(entry);
+		/*
+		 * The entry is successfully compressed and stored in the tree,
+		 * and further failures will be cleaned up in zswap_store().
+		 * Grab refs to the pool and objcg, charge zswap memory, and
+		 * increment zswap_stored_pages. The opposite actions will be
+		 * performed by zswap_entry_free() when the entry is removed
+		 * from the tree.
+		 */
+		zswap_pool_get(pool);
+		if (objcg) {
+			obj_cgroup_get(objcg);
+			obj_cgroup_charge_zswap(objcg, entry->length);
+		}
+		atomic_long_inc(&zswap_stored_pages);
+		if (unlikely(entry->length == PAGE_SIZE))
+			atomic_long_inc(&zswap_stored_incompressible_pages);
+
+		/*
+		 * We finish by adding the entry to the LRU while it's already
+		 * in xarray. This is safe because:
+		 *
+		 * 1. Concurrent stores and invalidations are excluded by folio lock.
+		 *
+		 * 2. Writeback is excluded by the entry not being on the LRU yet.
+		 *    The publishing order matters to prevent writeback from seeing
+		 *    an incoherent entry.
+		 */
+		if (likely(entry->length))
+			zswap_lru_add(entry);
 	}
 
 	return true;
 
-store_failed:
-	zs_free(pool->zs_pool, &entry->handle);
-compress_failed:
-	zswap_entry_cache_free(entry);
+store_pages_failed:
+	for (i = store_fail_idx; i < zc->nr_pages; ++i) {
+		if (zs_handle_is_allocated(&zc->entries[i]->handle))
+			zs_free(pool->zs_pool, &zc->entries[i]->handle);
+	}
+
+	zswap_entries_cache_free_batch(&zc->entries[store_fail_idx],
+				       zc->nr_pages - store_fail_idx);
+
 	return false;
 }
 
@@ -1624,9 +1731,14 @@ bool zswap_store(struct folio *folio)
 	swp_entry_t swp = folio->swap;
 	struct obj_cgroup *objcg = NULL;
 	struct mem_cgroup *memcg = NULL;
-	struct zswap_pool *pool;
+	int nid = folio_nid(folio);
 	bool ret = false;
-	long index;
+	struct zswap_pool *pool;
+	long start, end;
+	struct zswap_store_context zc = {
+		.folio = folio,
+		.nid = nid,
+	};
 
 	VM_WARN_ON_ONCE(!folio_test_locked(folio));
 	VM_WARN_ON_ONCE(!folio_test_swapcache(folio));
@@ -1660,8 +1772,23 @@ bool zswap_store(struct folio *folio)
 		mem_cgroup_put(memcg);
 	}
 
-	for (index = 0; index < nr_pages; ++index) {
-		if (!zswap_store_page(folio, index, objcg, pool))
+	rcu_read_lock();
+	zc.wb_enabled = mem_cgroup_zswap_writeback_enabled(folio_memcg(folio));
+	rcu_read_unlock();
+
+	/*
+	 * For batching and non-batching compressors, store the folio in
+	 * batches of ZSWAP_MAX_BATCH_SIZE, where the pages in the batch
+	 * are compressed in batches of the compressor's batch_size. This gives
+	 * better performance than invoking zswap_store_pages() per-page,
+	 * due to cache locality of working set structures.
+	 */
+	for (start = 0; start < nr_pages; start += ZSWAP_MAX_BATCH_SIZE) {
+		end = min(start + ZSWAP_MAX_BATCH_SIZE, nr_pages);
+
+		ret = zswap_store_pages(&zc, start, end, pool, objcg);
+
+		if (unlikely(!ret))
 			goto put_pool;
 	}
 
-- 
2.39.5


^ permalink raw reply related	[flat|nested] 25+ messages in thread

* [PATCH v4 7/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
  2026-11-20  4:18 [PATCH v4 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
                   ` (5 preceding siblings ...)
  2026-11-20  4:19 ` [PATCH v4 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
@ 2026-11-20  4:19 ` Kanchana P. Sridhar
  2026-11-20 23:45   ` Nhat Pham
  2026-11-20  4:19 ` [PATCH v4 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches Kanchana P. Sridhar
  2026-11-20 23:16 ` [PATCH v4 0/8] Batched zswap_store() framework with compression batching Nhat Pham
  8 siblings, 1 reply; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20  4:19 UTC (permalink / raw)
  To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
	davem, linux-crypto, kanchanapsridhar2026, linux-kernel
  Cc: rientjes, willy, senozhatsky

The various stages of zswap_store_pages() are broken down into routines
that work on a batch of data when a large folio is being processed:

 zswap_init_entries()
 zswap_tree_stores()

The struct zswap_store_context is used to pass constraints and data
among these batching components. The resulting code is highly simplified
and maintainable, where each of the batching sub-routines of
zswap_store() can be optimized modularly.

For instance, the zswap_tree_stores() now does the zswap atomics
counter increments, getting pool references, getting objcg references
and objcg zswap charging at the batch level. A new
obj_cgroup_charge_zswap_many() is added to enable batch charging of
compressed bytes for multiple pages/incompressible pages in the batch,
in order to accurately update the MEMCG_ZSWAP* counters.

The next commit will implement batching of zswap_compress().

Subsequent patch series will be submitted, that implement ideas building
upon this batching framework:

 1) Replacing zsmalloc with SLUB allocator, comprising of kmem_caches
    for a finite # of compressed size bins. This enables bulk allocation
    of compressed buffers in a batch that belong to the same size
    bin. It gets rid of significant zsmalloc latency overhead (locks,
    compressed data spanning zspages, etc.), without compromising memory
    efficiency.

 2) vmscan reclaim batching of hybrid any-order folios, that will submit
    a batch of folios to zswap_store()'s batching framework to optimize
    storing of batches of pages from different folios, in addition to
    within large folios.

Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
 include/linux/memcontrol.h |  12 +++
 mm/memcontrol.c            |  23 ++++-
 mm/zswap.c                 | 181 +++++++++++++++++++++++++------------
 3 files changed, 153 insertions(+), 63 deletions(-)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 980358bc6f49..93f060818c14 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -1244,6 +1244,11 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css)
 	return NULL;
 }
 
+static inline void obj_cgroup_get_many(struct obj_cgroup *objcg,
+				       unsigned long nr)
+{
+}
+
 static inline void obj_cgroup_get(struct obj_cgroup *objcg)
 {
 }
@@ -1921,6 +1926,8 @@ static inline void mem_cgroup_calculate_protection_path(struct mem_cgroup *root,
 #if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP)
 bool obj_cgroup_may_zswap(struct obj_cgroup *objcg);
 void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size);
+void obj_cgroup_charge_zswap_many(struct obj_cgroup *objcg, size_t size,
+				  u8 nr_pages, u8 nr_incomp_pages);
 void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size);
 bool mem_cgroup_zswap_writeback_enabled(const struct mem_cgroup *memcg);
 #else
@@ -1932,6 +1939,11 @@ static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg,
 					   size_t size)
 {
 }
+static inline void obj_cgroup_charge_zswap_many(struct obj_cgroup *objcg,
+						size_t size, u8 nr_pages,
+						u8 nr_incomp_pages)
+{
+}
 static inline void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg,
 					     size_t size)
 {
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
index 091401cecf39..2236f5ba6304 100644
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -6345,6 +6345,23 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
  * compression and storage in zswap for this cgroup to go ahead.
  */
 void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size)
+{
+	obj_cgroup_charge_zswap_many(objcg, size, 1, (u8)(size == PAGE_SIZE));
+}
+
+/**
+ * obj_cgroup_charge_zswap_many - bulk charge compression backend memory
+ *                                from compressing a batch of pages.
+ * @objcg: the object cgroup
+ * @size: total compressed length of a batch of compressed objects
+ * @nr_pages: # of pages that were compressed to store in zswap
+ * @nr_incomp_pages: # of incompressible pages to store in zswap
+ *
+ * This forces the charge after obj_cgroup_may_zswap() allowed
+ * compression and storage in zswap for this cgroup to go ahead.
+ */
+void obj_cgroup_charge_zswap_many(struct obj_cgroup *objcg, size_t size,
+				  u8 nr_pages, u8 nr_incomp_pages)
 {
 	struct mem_cgroup *memcg;
 
@@ -6363,9 +6380,9 @@ void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size)
 	rcu_read_lock();
 	memcg = obj_cgroup_memcg(objcg);
 	mod_memcg_state(memcg, MEMCG_ZSWAP_B, size);
-	mod_memcg_state(memcg, MEMCG_ZSWAPPED, 1);
-	if (size == PAGE_SIZE)
-		mod_memcg_state(memcg, MEMCG_ZSWAP_INCOMP, 1);
+	mod_memcg_state(memcg, MEMCG_ZSWAPPED, nr_pages);
+	if (nr_incomp_pages)
+		mod_memcg_state(memcg, MEMCG_ZSWAP_INCOMP, nr_incomp_pages);
 	rcu_read_unlock();
 }
 
diff --git a/mm/zswap.c b/mm/zswap.c
index df59ab0c1029..5e1fb5624ffb 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -491,12 +491,12 @@ static int __must_check zswap_pool_tryget(struct zswap_pool *pool)
 }
 
 /* The caller must already have a reference. */
-static void zswap_pool_get(struct zswap_pool *pool)
+static inline void zswap_pool_get_many(struct zswap_pool *pool, u8 nr)
 {
-	percpu_ref_get(&pool->ref);
+	percpu_ref_get_many(&pool->ref, nr);
 }
 
-static void zswap_pool_put(struct zswap_pool *pool)
+static inline void zswap_pool_put(struct zswap_pool *pool)
 {
 	percpu_ref_put(&pool->ref);
 }
@@ -1603,55 +1603,60 @@ static void shrink_worker(struct work_struct *w)
 * main API
 **********************************/
 
+/* Functions called by zswap_store(), that operate on batches of pages. */
+
 /*
- * Store multiple pages in @zc->folio, starting from the page at index @start up
- * to the page at index @end-1.
+ * zswap_init_entries() co-locates most of the zswap entry fields' setup under
+ * the assumption that all phases of store processing will succeed. An important
+ * marker to simplify catch-all cleanup in zswap_store_pages() is done by
+ * initializing each entry's @handle to ERR_PTR(-EINVAL).
  */
-static bool zswap_store_pages(struct zswap_store_context *zc,
-			      long start,
-			      long end,
-			      struct zswap_pool *pool,
-			      struct obj_cgroup *objcg)
+static inline void zswap_init_entries(struct zswap_store_context *zc,
+				      struct zswap_pool *pool,
+				      struct obj_cgroup *objcg)
 {
-	u8 i, store_fail_idx = 0;
-	bool ret;
-
-	zc->nr_pages = end - start;
-	zc->folio_start = start;
-
-	VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
-
-	ret = zswap_entries_cache_alloc_batch(zc, GFP_KERNEL);
-	if (unlikely(!ret))
-		return false;
+	struct zswap_entry *entry;
+	u8 i;
 
 	/*
 	 * We co-locate entry initialization as much as possible here to
 	 * minimize potential cache misses.
 	 */
 	for (i = 0; i < zc->nr_pages; ++i) {
+		entry = zc->entries[i];
 		/*
-		 * Set pool_idx before the xa_store() below publishes the entry, or a
-		 * concurrent reader could resolve a stale pool_idx left by slab reuse
-		 * to an unrelated live pool.
+		 * Set pool_idx before the xa_store() in zswap_tree_stores()
+		 * publishes the entry, or a concurrent reader could resolve a
+		 * stale pool_idx left by slab reuse to an unrelated live pool.
 		 */
-		zc->entries[i]->pool_idx = pool->idx;
-		zc->entries[i]->swpentry = folio_swap_entry(zc->folio,
+		entry->pool_idx = pool->idx;
+		entry->swpentry = folio_swap_entry(zc->folio,
 							zc->folio_start + i);
-		zc->entries[i]->objcg = objcg;
-		zc->entries[i]->referenced = true;
-		zc->entries[i]->nid = zc->nid;
-		INIT_LIST_HEAD(&zc->entries[i]->lru);
+		entry->objcg = objcg;
+		entry->referenced = true;
+		entry->nid = zc->nid;
+		INIT_LIST_HEAD(&entry->lru);
 	}
+}
 
-	for (i = 0; i < zc->nr_pages; ++i) {
-		if (!zswap_compress(zc->folio, zc->folio_start + i,
-				    zc->entries[i], pool, zc->wb_enabled))
-			goto store_pages_failed;
-	}
+/*
+ * zswap_tree_stores() stores the batch entries in xarray and adds them to the
+ * zswap LRU list. If an error occurs during xa_store(), localized state
+ * unwinding is done to erase prior batch entries from the xarray and delete
+ * them from the zswap LRU list. Remaining batch cleanup occurs in
+ * zswap_store_pages(). As batch entries are processed, we maintain aggregate
+ * statistics to do the accounting in one bulk operation.
+ */
+static bool zswap_tree_stores(struct zswap_store_context *zc,
+			      struct zswap_pool *pool,
+			      struct obj_cgroup *objcg)
+{
+	u8 i, store_fail_idx = 0, nr_incomp = 0;
+	size_t total_batch_comp_length = 0;
+	struct zswap_entry *old, *entry;
 
 	for (i = 0; i < zc->nr_pages; ++i) {
-		struct zswap_entry *old, *entry = zc->entries[i];
+		entry = zc->entries[i];
 
 		old = xa_store(swap_zswap_tree(entry->swpentry),
 			       swp_offset(entry->swpentry),
@@ -1663,13 +1668,11 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
 			zswap_reject_alloc_fail++;
 			/*
 			 * Entries up to this point have been stored in the
-			 * xarray. zswap_store() will erase them from the xarray
-			 * and call zswap_entry_free(). Local cleanup in
-			 * 'store_pages_failed' only needs to happen for
-			 * entries from [@i to @nr_pages).
+			 * xarray. Local cleanup in 'tree_store_failed' needs to
+			 * erase entries up until now.
 			 */
 			store_fail_idx = i;
-			goto store_pages_failed;
+			goto tree_store_failed;
 		}
 
 		/*
@@ -1680,22 +1683,10 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
 		if (unlikely(old))
 			zswap_entry_free(old);
 
-		/*
-		 * The entry is successfully compressed and stored in the tree,
-		 * and further failures will be cleaned up in zswap_store().
-		 * Grab refs to the pool and objcg, charge zswap memory, and
-		 * increment zswap_stored_pages. The opposite actions will be
-		 * performed by zswap_entry_free() when the entry is removed
-		 * from the tree.
-		 */
-		zswap_pool_get(pool);
-		if (objcg) {
-			obj_cgroup_get(objcg);
-			obj_cgroup_charge_zswap(objcg, entry->length);
-		}
-		atomic_long_inc(&zswap_stored_pages);
+		total_batch_comp_length += entry->length;
+
 		if (unlikely(entry->length == PAGE_SIZE))
-			atomic_long_inc(&zswap_stored_incompressible_pages);
+			++nr_incomp;
 
 		/*
 		 * We finish by adding the entry to the LRU while it's already
@@ -1711,17 +1702,87 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
 			zswap_lru_add(entry);
 	}
 
+	/*
+	 * Each batch entry is successfully compressed and stored in the tree,
+	 * and further failures will be cleaned up in zswap_store().
+	 * Next, we do bulk accounting for the batch: grab refs to the pool and
+	 * objcg, charge zswap memory, increment zswap_stored_pages and
+	 * zswap_stored_incompressible_pages. The opposite actions will be
+	 * performed by zswap_entry_free() when the entry is removed from the
+	 * tree.
+	 */
+	zswap_pool_get_many(pool, zc->nr_pages);
+	if (objcg) {
+		obj_cgroup_get_many(objcg, zc->nr_pages);
+		obj_cgroup_charge_zswap_many(objcg, total_batch_comp_length,
+					     zc->nr_pages, nr_incomp);
+	}
+	atomic_long_add(zc->nr_pages, &zswap_stored_pages);
+	if (unlikely(nr_incomp))
+		atomic_long_add(nr_incomp, &zswap_stored_incompressible_pages);
+
+	return true;
+
+tree_store_failed:
+	for (i = 0; i < store_fail_idx; ++i) {
+		xa_erase(swap_zswap_tree(zc->entries[i]->swpentry),
+			 swp_offset(zc->entries[i]->swpentry));
+		zswap_lru_del(zc->entries[i]);
+	}
+
+	return false;
+}
+
+/*
+ * Store multiple pages in @zc->folio, starting from the page at index @start up
+ * to the page at index @end-1, i.e. @zc->nr_pages. Each phase of
+ * zswap_store_pages() does batch processing of @zc->nr_pages in the folio.
+ * State unwinding is kept to a minimum as a result of the batch processing: a
+ * batch proceeds to the next phase of store processing only if it passes this
+ * stage. The batch cleanup is consolidated under the @store_pages_failed
+ * label. Errors in storing a batch result in folio level cleanup occurring in
+ * zswap_store().
+ */
+static bool zswap_store_pages(struct zswap_store_context *zc,
+			      long start,
+			      long end,
+			      struct zswap_pool *pool,
+			      struct obj_cgroup *objcg)
+{
+	bool ret;
+	u8 i;
+
+	zc->nr_pages = end - start;
+	zc->folio_start = start;
+
+	VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
+
+	ret = zswap_entries_cache_alloc_batch(zc, GFP_KERNEL);
+	if (unlikely(!ret))
+		return false;
+
+	zswap_init_entries(zc, pool, objcg);
+
+	for (i = 0; i < zc->nr_pages; ++i) {
+		if (!zswap_compress(zc->folio, zc->folio_start + i,
+				    zc->entries[i], pool, zc->wb_enabled))
+			goto store_pages_failed;
+	}
+
+	ret = zswap_tree_stores(zc, pool, objcg);
+
+	if (unlikely(!ret))
+		goto store_pages_failed;
+
 	return true;
 
 store_pages_failed:
-	for (i = store_fail_idx; i < zc->nr_pages; ++i) {
+	for (i = 0; i < zc->nr_pages; ++i) {
 		if (zs_handle_is_allocated(&zc->entries[i]->handle))
 			zs_free(pool->zs_pool, &zc->entries[i]->handle);
 	}
 
-	zswap_entries_cache_free_batch(&zc->entries[store_fail_idx],
-				       zc->nr_pages - store_fail_idx);
-
+	zswap_entries_cache_free_batch(zc->entries, zc->nr_pages);
 	return false;
 }
 
-- 
2.39.5


^ permalink raw reply related	[flat|nested] 25+ messages in thread

* [PATCH v4 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches.
  2026-11-20  4:18 [PATCH v4 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
                   ` (6 preceding siblings ...)
  2026-11-20  4:19 ` [PATCH v4 7/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations Kanchana P. Sridhar
@ 2026-11-20  4:19 ` Kanchana P. Sridhar
  2026-11-20 23:55   ` Nhat Pham
  2026-11-21  4:20   ` sashiko-bot
  2026-11-20 23:16 ` [PATCH v4 0/8] Batched zswap_store() framework with compression batching Nhat Pham
  8 siblings, 2 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20  4:19 UTC (permalink / raw)
  To: hannes, yosry, nphamcs, chengming.zhou, akpm, linux-mm, herbert,
	davem, linux-crypto, kanchanapsridhar2026, linux-kernel
  Cc: rientjes, willy, senozhatsky

A new batching implementation of zswap_compress() is provided for
compressors that do and do not support batching. This eliminates code
duplication and facilitates code maintainability with the introduction
of compress batching.

The vectorized implementation of calling the earlier zswap_compress()
sequentially, one page at a time in zswap_store_pages(), is replaced
with this new version of zswap_compress() that accepts multiple pages to
compress as a batch.

When the crypto_acomp is used to set up the per-CPU acomp_ctx
during CPU onlining, the compressor's batch_size is obtained by calling:

  crypto_acomp_batch_size(acomp_ctx->acomp)

This is bounded by ZSWAP_MAX_BATCH_SIZE to limit the memory
overhead and to reflect the store batch constraint of max # of pages in
a batch. The clamped batch_size is stored in the per-CPU
acomp_ctx->comp.compr_batch_size.

As noted in previous commits in this series, ZSWAP_MAX_BATCH_SIZE
per-CPU compression buffers are available for sequential and batching
compression algorithms.

Regardless of compression batching, we always process large folios in
batches of ZSWAP_MAX_BATCH_SIZE.

If the compressor does not support batching, each page in the batch is
compressed sequentially. For batching compressors, up to
acomp_ctx->comp.compr_batch_size pages in the batch are compressed per
call to crypto_acomp_compress(). The first phase of zswap_compress()
thus only calls crypto_acomp_compress() until all pages in the batch
have been compressed. When this finishes, the compressed buffers and
compressed lengths for the batch are available in the
acomp_ctx->comp.sg_table->sgl scatterlists.

A new zswap_update_entries() then iterates through the (up to)
ZSWAP_MAX_BATCH_SIZE SG lists in acomp_ctx->comp.sg_table->sgl,
allocates zsmalloc memory and copies all compressed buffers to
the compressed pool. All existing functionality on zswap
writeback_enabled (or not) fallbacks under various error conditions
denoted by the crypto_acomp_compress() return value [and with this
commit, also stored in the individual acomp_ctx->comp.sg_table->sgl
sg->lengths in the crypto_acomp_compress() fast path], is essentially
preserved.

A significant zswap architectural change is that the batched
zswap_compress() incorporates Herbert's suggestion for SG lists to
represent the batch's inputs/outputs to interface with the crypto API
[1].

The introduction of zswap_update_entries() presents an opportunity to
optimize the task of storing a batch of compressed buffers. Subsequent
patch series will be submitted to optimize this phase by replacing
zsmalloc with the SLUB allocator to improve zswap_store()/zswap_load()
latencies.

The batching framework introduced in this series improves zswap_store()
latency by a lot.

Performance testing: usemem30 with 100% over-commit:
=====================================================
Validation was done with mm-unstable as of 11-17-2026 (Baseline: commit
866f761fd414) and with this patch series.

v4 was validated on an Intel Xeon Sapphire Rapids bare-metal instance with
48 cores.

 Bare-metal instance specs:
 --------------------------
 - 48 physical cores
 - 190G RAM
 - 200G SWAP disk partition

 Workload: 30 concurrent usemem processes, each allocates 10G:
 -------------------------------------------------------------
 The vm-scalability "usemem" test (unmodified version) was run in a
 cgroup with memory.high fixed at 150G. There is no swap limit set for the
 cgroup.

 To quantify the performance benefit of zswap compress batching at scale,
 30 usemem processes were run ("usemem30"), each allocating and writing 10G of
 memory, and sleeping for 10 sec before exiting.

  usemem -w -O -b 5 -s 10 -n 30 10g
  echo 0 > /sys/module/zswap/parameters/shrinker_enabled

 Performance validation methodology:
 -----------------------------------
 These settings were used to minimize variation:

 - All cores were fixed to run at 2.4GHz with "performance" scaling governor.
 - Turbo boost was disabled.
 - CPU power-saving C-states restricted with:

     GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1 processor.max_cstate=1"

 The usemem30 experiments were run with the baseline mm-unstable as of
 11-17-2026 (commit 866f761fd414) and with this patch-series.

 After booting into each kernel:

  1) Constrain the cgroup's memory.high to 150G.

  2) Run 2 warm-up iterations of usemem30. Don't use this data for benchmarking
     since sys time typically skews higher for initial runs.

  3) Next, run 10 benchmarking iterations of usemem30:
     - Log these high-level metrics: Sys time and individual usemem throughput.

  4) Sys time and Throughput metrics were gathered for three mTHP settings:

     - 2048kB
     - 1024kB
     - 64kB

  5) Benchmark summary data includes Sys time and Throughput components from the
     10 iterations of usemem30:

     - Total Samples
     - Mean Value
     - Std Deviation
     - CoV (Variance): Coefficient of variation (relative standard
       deviation) measures spread relative to the mean rather than
       statistical variance.

 An LLM was used to help generate the benchmarking and data processing scripts.

usemem30 Benchmarking Summary:
==============================

 ---------------------------------------------------
    Improvement with v4 zswap compress batching
 ---------------------------------------------------
  mTHP enabled:       2048kB     1024kB       64kB
 ---------------------------------------------------
      Sys time:      -13.87%     -2.64%     -1.72%
    Throughput:       +3.85%     +0.42%     +1.31%
 ---------------------------------------------------

Detailed usemem30 benchmarking data:
====================================

 1) 2048kB mTHP:
 ===============

 ------------------------------------------------------------------
 2048kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)        (sec)
 ------------------------------------------------------------------
 [Run 1/10]  Sys Time:    1581.414s    1356.252s
 [Run 2/10]  Sys Time:    1642.874s    1478.929s
 [Run 3/10]  Sys Time:    1594.458s    1411.252s
 [Run 4/10]  Sys Time:    1580.988s    1380.959s
 [Run 5/10]  Sys Time:    1579.654s    1380.609s
 [Run 6/10]  Sys Time:    1627.765s    1345.582s
 [Run 7/10]  Sys Time:    1593.872s    1334.485s
 [Run 8/10]  Sys Time:    1623.189s    1405.471s
 [Run 9/10]  Sys Time:    1661.954s    1433.365s
 [Run 10/10] Sys Time:    1602.098s    1329.813s
 ------------------------------------------------------------------

 ------------------------------------------------------------------
 METRIC: System CPU Time (Sys Time) : 2048kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)            (sec)
 ------------------------------------------------------------------
 Total Samples    :              10               10
 Mean Value       :        1,608.83         1,385.67
 Std Deviation    :           28.67            47.47
 CoV (Variance)   :           1.78%            3.43%
 ------------------------------------------------------------------
 METRIC: Per-Process Throughput : 2048kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                             (KB/s)           (KB/s)
 ------------------------------------------------------------------
 Total Samples    :             300              300
 Mean Value       :         292,261          303,508
 Std Deviation    :          12,946           20,806
 CoV (Variance)   :           4.42%            6.86%
 ------------------------------------------------------------------

 2) 1024kB mTHP:
 ===============

 ------------------------------------------------------------------
 1024kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)        (sec)
 ------------------------------------------------------------------
 [Run 1/10]  Sys Time:    1567.639s    1594.696s
 [Run 2/10]  Sys Time:    1537.577s    1456.113s
 [Run 3/10]  Sys Time:    1538.562s    1521.887s
 [Run 4/10]  Sys Time:    1626.176s    1531.962s
 [Run 5/10]  Sys Time:    1593.955s    1528.733s
 [Run 6/10]  Sys Time:    1643.519s    1558.482s
 [Run 7/10]  Sys Time:    1510.613s    1480.177s
 [Run 8/10]  Sys Time:    1632.018s    1584.093s
 [Run 9/10]  Sys Time:    1569.424s    1528.915s
 [Run 10/10] Sys Time:    1552.747s    1570.451s
 ------------------------------------------------------------------

 ------------------------------------------------------------------
 METRIC: System CPU Time (Sys Time) : 1024kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)            (sec)
 ------------------------------------------------------------------
 Total Samples    :             10                10
 Mean Value       :       1,577.22          1,535.55
 Std Deviation    :          45.13             43.79
 CoV (Variance)   :          2.86%             2.85%
 ------------------------------------------------------------------
 METRIC: Per-Process Throughput : 1024kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                             (KB/s)           (KB/s)
 ------------------------------------------------------------------
 Total Samples    :            300               300
 Mean Value       :        289,904           291,114
 Std Deviation    :         15,385            14,553
 CoV (Variance)   :          5.31%             5.00%
 ------------------------------------------------------------------

 3) 64kB mTHP:
 =============

 ------------------------------------------------------------------
 64kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)        (sec)
 ------------------------------------------------------------------
 [Run 1/10]  Sys Time:    1914.525s    1878.483s
 [Run 2/10]  Sys Time:    1869.412s    1863.177s
 [Run 3/10]  Sys Time:    1944.049s    1863.502s
 [Run 4/10]  Sys Time:    1942.757s    1900.144s
 [Run 5/10]  Sys Time:    1866.505s    1887.285s
 [Run 6/10]  Sys Time:    1877.716s    1861.668s
 [Run 7/10]  Sys Time:    1861.741s    1868.788s
 [Run 8/10]  Sys Time:    1904.735s    1816.372s
 [Run 9/10]  Sys Time:    1891.074s    1850.435s
 [Run 10/10] Sys Time:    1893.366s    1849.747s
 ------------------------------------------------------------------

 ------------------------------------------------------------------
 METRIC: System CPU Time (Sys Time) : 64kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                              (sec)            (sec)
 ------------------------------------------------------------------
 Total Samples    :              10               10
 Mean Value       :        1,896.59         1,863.96
 Std Deviation    :           29.86            22.96
 CoV (Variance)   :           1.57%            1.23%
 ------------------------------------------------------------------
 METRIC: Per-Process Throughput : 64kB mTHP
 ------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap compress batching
                             (KB/s)           (KB/s)
 ------------------------------------------------------------------
 Total Samples    :            300               300
 Mean Value       :        239,114           242,254
 Std Deviation    :          9,090             6,419
 CoV (Variance)   :          3.80%             2.65%
 ------------------------------------------------------------------

Conclusions based on the data:
==============================

 v3 demonstrated that the batching of zswap_store() phases and compress
 batching optimizations benefit zswap_store() and
 zswap_compress_per_page() median and tail latencies for PMD folios and
 base 4K folios.

 v4 demonstrates how these low-level bptrace savings obtained in v3
 translate to performance improvements at the system level and workload
 level. v4's performance validation is done at scale with 30 concurrent
 usemem processes, with Sys time and Throughput being the key high-level
 metrics. We further study this for PMD THP, 1024kB and 64kB folios.

 Sys time measured across 10 consecutive iterations of usemem30 improves
 significantly with zswap compress batching, -13.87% lower than
 mm-unstable for PMD THP. Per-process Throughput increases by 3.85% for
 PMD folios. As this patch-series impacts batching within folios, the
 Sys time savings reduces for lower order folios.

 The immediate benefit of zswap compress batching is saving CPU compute
 cycles, that can imply lower power consumption and existing kernel
 optimizations with reduced CPU compute cycles spent in critical
 paths. There are also modest throughput improvements that this
 patch-series delivers.

 As other kernel components related to mTHP become more efficient, the
 net realized performance benefit can only be better with this
 patch-series than without, given the significant Sys time reduction
 with zswap store and compress batching.

 The batching framework established in this series allows us to make
 locking and bulk-processing improvements in each stage of
 zswap_store(). One of the ideas that has shown very promising results
 is to replace zsmalloc with SLUB compressed caches and will be
 submitted separately. This is expected to resolve the p50 batch
 compress latency issue seen in v3's bpftrace data, since it will be
 directly optimizing zswap_update_entries(), where compressed buffers
 can be binned based on size and bulk allocated from SLUB.

 The speedup with zswap store batching of large folios has a propelling
 effect on other parts of mm, especially, by enabling large folios to be
 swapped out without splitting, as the higher-order folio vmstat v4 data
 shows. This enables other mm components that are optimized for large
 folios, such as munmap(), folio refcounts/mapcounts updates, PTE
 invalidation upon freeing memory, zswap_invalidate(); which benefit
 from xarray offsets/range continuity. As a result, subsequent large
 folio swapouts are more likely to find contiguous swap slots, there are
 fewer large folio splits during reclaim and the cycle continues. As the
 data demonstrates, this directly contributes to more memory over-commit
 opportunity.

 The benefits are attributable to the store batching framework added in
 patches 6-7, that is fully utilized to improve zswap_compress() latency
 by giving it a batch of pages to be compressed. As the bpftrace data
 shows, patches 6-7 alone are not sufficient for the gains. Batch
 compression added in patch 8 benefits latency significantly, for these
 reasons:

  1) Per-CPU sg_table with ZSWAP_MAX_BATCH_SIZE scatterlists that are
     statically mapped to compression dst buffers for that CPU, thereby
     minimizing local SG lists initialization, and using compact new
     crypto_acomp API introduced in patches 1-4 that set up an
     acomp_req's src/dst to be SG lists.

  2) Patch 6 allocates ZSWAP_MAX_BATCH_SIZE destination buffers in the
     per-CPU acomp_ctx, for an added memory overhead of ~12 KB
     per-CPU. The trade-off pays off in significant latency improvements
     which can potentially be used to drive higher memory over-commit in
     data center setups.

  3) A one-shot zswap_update_entries() does all the zsmalloc handle
     allocation and copying for the batch. The key concept driving
     efficiency is that the working set is kept focused on batch-centric
     data pertaining to a handful of operations on the batch, to
     minimize cache evictions and benefit from cache locality.

  4) By making zswap_compress() more efficient, we are able to lock the
     comp mutex per batch of ZSWAP_MAX_BATCH_SIZE (4) pages instead of
     per-page.

 The exceptions are p50/p95 batch compress latencies (normalized
 per-page), even though the p99 batch compress latency improves
 significantly, by 15% over mm-unstable. Batch compress latency can be
 optimized further in subsequent patches.

Architectural considerations for the zswap batching framework:
==============================================================
The zswap batching framework is designed to be hardware-agnostic.
It can be leveraged by any hardware accelerator or software-based
compressor.

Potential future clients of the batching framework:
===================================================
This patch-series demonstrates the performance benefits of compression
batching when used in zswap_store() of large folios. Compression
batching can be used for other use cases such as batching compression in
zram, batch compression of different folios during reclaim, kcompressd,
file systems, etc. Decompression batching can be used to improve
efficiency of zswap writeback (Thanks Nhat for this idea), batching
decompressions in zram, etc.

[1]: https://lore.kernel.org/all/aJ7Fk6RpNc815Ivd@gondor.apana.org.au/T/#m99aea2ce3d284e6c5a3253061d97b08c4752a798

Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
---
 mm/zswap.c | 288 +++++++++++++++++++++++++++++++++++++++--------------
 1 file changed, 215 insertions(+), 73 deletions(-)

diff --git a/mm/zswap.c b/mm/zswap.c
index 5e1fb5624ffb..56354a568bad 100644
--- a/mm/zswap.c
+++ b/mm/zswap.c
@@ -149,9 +149,14 @@ struct zswap_acomp_req {
 
 /*
  * The compression mutex also protects the output buffer(s).
+ *
+ * @compr_batch_size: The max batch size of the compression algorithm,
+ *                    bounded by ZSWAP_MAX_BATCH_SIZE.
  */
 struct zswap_comp_ctx {
 	struct zswap_acomp_req areq;
+	u8 compr_batch_size;
+	struct sg_table *sg_table;
 	u8 **buffers;
 };
 
@@ -238,6 +243,8 @@ struct zswap_store_context {
 	bool wb_enabled;
 	int nid;
 	u8 nr_pages;
+	struct zs_pool *zs_pool;
+	struct zswap_comp_ctx *comp_ctx;
 	struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE];
 };
 
@@ -325,6 +332,13 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
 		kfree(acomp_ctx->comp.buffers);
 	}
 	acomp_ctx->comp.buffers = NULL;
+	acomp_ctx->comp.compr_batch_size = 0;
+
+	if (acomp_ctx->comp.sg_table) {
+		sg_free_table(acomp_ctx->comp.sg_table);
+		kfree(acomp_ctx->comp.sg_table);
+	}
+	acomp_ctx->comp.sg_table = NULL;
 }
 
 static struct zswap_pool *zswap_pool_create(char *compressor)
@@ -903,6 +917,7 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
 	struct zswap_pool *pool = hlist_entry(node, struct zswap_pool, node);
 	struct crypto_acomp_ctx *acomp_ctx = per_cpu_ptr(pool->acomp_ctx, cpu);
 	int nid = cpu_to_node(cpu);
+	struct scatterlist *sg;
 	int ret = -ENOMEM;
 	u8 i;
 
@@ -927,6 +942,13 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
 		goto fail;
 	}
 
+	/*
+	 * Constrain the crypto algorithm's batch size by ZSWAP_MAX_BATCH_SIZE,
+	 * store this in the per-CPU zswap_comp_ctx's @compr_batch_size.
+	 */
+	acomp_ctx->comp.compr_batch_size = min(ZSWAP_MAX_BATCH_SIZE,
+					       crypto_acomp_batch_size(acomp_ctx->acomp));
+
 	ret = zswap_acomp_req_init(&acomp_ctx->comp.areq, acomp_ctx->acomp);
 	if (ret)
 		goto req_fail;
@@ -949,6 +971,26 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
 		}
 	}
 
+	acomp_ctx->comp.sg_table = kmalloc(sizeof(*acomp_ctx->comp.sg_table),
+					   GFP_KERNEL);
+	if (!acomp_ctx->comp.sg_table) {
+		ret = -ENOMEM;
+		goto req_fail;
+	}
+
+	if (sg_alloc_table(acomp_ctx->comp.sg_table, ZSWAP_MAX_BATCH_SIZE,
+			   GFP_KERNEL)) {
+		ret = -ENOMEM;
+		goto req_fail;
+	}
+
+	/*
+	 * Statically map the per-CPU destination buffers to the per-CPU
+	 * SG lists; crucial for performance.
+	 */
+	for_each_sg(acomp_ctx->comp.sg_table->sgl, sg, ZSWAP_MAX_BATCH_SIZE, i)
+		sg_set_buf(sg, acomp_ctx->comp.buffers[i], PAGE_SIZE);
+
 	/* Skip the per-CPU request when decompression can use the stack. */
 	if (acomp_is_async(acomp_ctx->acomp)) {
 		ret = zswap_acomp_req_init(&acomp_ctx->decomp, acomp_ctx->acomp);
@@ -965,88 +1007,179 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
 	return ret;
 }
 
-static bool zswap_compress(struct folio *folio, long index,
-			   struct zswap_entry *entry, struct zswap_pool *pool,
-			   bool wb_enabled)
+/*
+ * zswap_update_entries() is called only when all pages in a store batch have
+ * been successfully compressed (subsumes the writeback_enabled fallback in case
+ * of compress errors). The only error that can occur in this phase is zs_malloc
+ * allocation failure, upon which an error status is returned to
+ * zswap_store_pages().
+ */
+static inline bool zswap_update_entries(struct zswap_store_context *zc)
 {
-	struct zswap_comp_ctx *comp_ctx;
-	struct scatterlist input, output;
-	int comp_ret = 0, alloc_ret = 0;
-	unsigned int dlen = PAGE_SIZE;
-	gfp_t gfp;
-	u8 *dst;
-	bool mapped = false;
+	gfp_t gfp = GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE;
+	struct zswap_entry *entry;
+	struct scatterlist *sg;
+	unsigned int dlen;
+	u8 batch_iter;
+	u8 *buf;
+	int err;
+
+	/* Called should ensure that each dst sg->length is > 0. */
+	for_each_sg(zc->comp_ctx->sg_table->sgl, sg, zc->nr_pages, batch_iter) {
+		entry = zc->entries[batch_iter];
+		dlen = sg->length;
+
+		prefetchw(entry);
+
+		if (unlikely(dlen == PAGE_SIZE)) {
+			buf = kmap_local_folio(zc->folio,
+					(zc->folio_start + batch_iter) * PAGE_SIZE);
+			err = zs_store(zc->zs_pool, &entry->handle, buf, dlen, gfp, zc->nid);
+			kunmap_local(buf);
+		} else {
+			buf = zc->comp_ctx->buffers[batch_iter];
+			err = zs_store(zc->zs_pool, &entry->handle, buf, dlen, gfp, zc->nid);
+		}
 
-	comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
-	mutex_lock(&comp_ctx->areq.mutex);
+		if (unlikely(err)) {
+			if (err == -ENOSPC)
+				zswap_reject_compress_poor++;
+			else
+				zswap_reject_alloc_fail++;
 
-	dst = comp_ctx->buffers[0];
-	sg_init_table(&input, 1);
-	sg_set_folio(&input, folio, PAGE_SIZE, index * PAGE_SIZE);
+			return false;
+		}
+
+		entry->length = dlen;
+	}
 
-	sg_init_one(&output, dst, PAGE_SIZE);
-	acomp_request_set_params(comp_ctx->areq.req, &input, &output,
-				 PAGE_SIZE, dlen);
+	return true;
+}
+
+/*
+ * zswap_compress() batching implementation for sequential and batching
+ * compressors.
+ *
+ * Description:
+ * ============
+ * Compress multiple @zc->nr_pages in @zc->folio starting from the
+ * @zc->folio_start index in batches of @nr_batch_pages; the latter
+ * representing the lesser of the @zc->nr_pages to be batch compressed and the
+ * compressor's batch size.
+ *
+ * @zc->nr_pages can be in (1, ZSWAP_MAX_BATCH_SIZE] even if the compressor does
+ * not support batching.
+ *
+ * If @nr_batch_pages is 1, each page is compressed sequentially.
+ *
+ * If @nr_batch_pages is > 1, the compressor may use batching to optimize
+ * compression.
+ *
+ * In all cases, the store processing that needs to happen post-compression
+ * occurs in batches of @zc->nr_pages. Error handling is minimal, mostly to
+ * update zswap statistics, with catch-all cleanup for the batch occuring in the
+ * @store_pages_failed label in zswap_store_pages().
+ */
+static bool zswap_compress(struct zswap_store_context *zc)
+{
+	u8 nr_batch_pages, batch_start, comp_batch_iter;
+	struct scatterlist *sg, *dst_sg;
+	unsigned int slen;
+	int err;
 
 	/*
-	 * it maybe looks a little bit silly that we send an asynchronous request,
-	 * then wait for its completion synchronously. This makes the process look
-	 * synchronous in fact.
-	 * Theoretically, acomp supports users send multiple acomp requests in one
-	 * acomp instance, then get those requests done simultaneously. but in this
-	 * case, zswap actually does store and load page by page, there is no
-	 * existing method to send the second page before the first page is done
-	 * in one thread doing zswap.
-	 * but in different threads running on different cpu, we have different
-	 * acomp instance, and compression and decompression use separate
-	 * requests, so multiple threads can do (de)compression in parallel.
+	 * @nr_batch_pages is constrained by the per-CPU comp_ctx's compressor
+	 * batch size and the @zc->nr_pages in this store batch. @slen indicates
+	 * the total source length bytes for @nr_batch_pages.
 	 */
-	comp_ret = crypto_wait_req(crypto_acomp_compress(comp_ctx->areq.req),
-				   &comp_ctx->areq.wait);
-	dlen = comp_ctx->areq.req->dlen;
-	if (!comp_ret) {
-		if (dlen == PAGE_SIZE)
-			comp_ret = -ENOSPC;
-		else if (WARN_ON_ONCE(!dlen || dlen > PAGE_SIZE))
-			comp_ret = -EINVAL;
-	}
+	nr_batch_pages = min(zc->comp_ctx->compr_batch_size, zc->nr_pages);
 
 	/*
-	 * If a page cannot be compressed into a size smaller than PAGE_SIZE,
-	 * save the content as is without a compression, to keep the LRU order
-	 * of writebacks.  If writeback is disabled, reject the page since it
-	 * only adds metadata overhead.  swap_writeout() will put the page back
-	 * to the active LRU list in the case.
+	 * Compress the @nr_pages in @folio starting at index @folio_start
+	 * in batches of @nr_batch_pages.
 	 */
-	if (comp_ret) {
-		if (!wb_enabled)
-			goto unlock;
-		comp_ret = 0;
-		dlen = PAGE_SIZE;
-		dst = kmap_local_folio(folio, index * PAGE_SIZE);
-		mapped = true;
+	for (batch_start = 0; batch_start < zc->nr_pages;
+	     batch_start += nr_batch_pages) {
+		/*
+		 * Clamp the last iteration's batch size if the compressor's
+		 * batch_size does not evenly divide @zc->nr_pages.
+		 */
+		nr_batch_pages = min(nr_batch_pages, zc->nr_pages - batch_start);
+		slen = nr_batch_pages * PAGE_SIZE;
+
+		/*
+		 * Send @nr_batch_pages to crypto_acomp for compression:
+		 *
+		 * These pages are in @zc->folio's range of indices in the
+		 * interval:
+		 *     [@zc->folio_start + @batch_start,
+		 *      @zc->folio_start + @batch_start + @nr_batch_pages).
+		 *
+		 * The source folio pages in the batch are directly submitted to
+		 * crypto_acomp via acomp_request_set_src_folio().
+		 */
+		acomp_request_set_src_folio(zc->comp_ctx->areq.req, zc->folio,
+				(zc->folio_start + batch_start) * PAGE_SIZE,
+				slen);
+
+		dst_sg = zc->comp_ctx->sg_table->sgl + batch_start;
+		dst_sg->length = slen;
+
+		/*
+		 * The per-CPU @zc->comp_ctx->sg_table scatterlists are statically
+		 * mapped to the per-CPU dst @buffers at pool creation time.
+		 * There are ZSWAP_MAX_BATCH_SIZE per-CPU SG lists and dst
+		 * buffers allocated for all compressors.
+		 */
+		acomp_request_set_dst_sg(zc->comp_ctx->areq.req, dst_sg, slen);
+
+		err = crypto_wait_req(crypto_acomp_compress(zc->comp_ctx->areq.req),
+				      &zc->comp_ctx->areq.wait);
+
+		/*
+		 * If a page cannot be compressed into a size smaller than
+		 * PAGE_SIZE, save the content as is without a compression, to
+		 * keep the LRU order of writebacks.  If writeback is disabled,
+		 * reject the page since it only adds metadata overhead.
+		 * swap_writeout() will put the page back to the active LRU list
+		 * in the case.
+		 */
+		if (err && !zc->wb_enabled)
+			goto compress_error;
+
+		/*
+		 * For each output SG list in @acomp_ctx->req->sg_table->sgl,
+		 * the @sg->length should be set to either the page's compressed
+		 * length (success), or it's negative value compression error.
+		 */
+		for_each_sg(dst_sg, sg, nr_batch_pages, comp_batch_iter) {
+			if (((int)sg->length <= 0) || (sg->length >= PAGE_SIZE)) {
+				if (!zc->wb_enabled) {
+					if (sg->length == PAGE_SIZE)
+						sg->length = -ENOSPC;
+					else if (WARN_ON_ONCE(!sg->length || sg->length > PAGE_SIZE))
+						sg->length = -EINVAL;
+					goto compress_error;
+				}
+
+				sg->length = PAGE_SIZE;
+			}
+		}
 	}
 
-	gfp = GFP_NOWAIT | __GFP_NORETRY | __GFP_HIGHMEM | __GFP_MOVABLE;
-	alloc_ret = zs_store(pool->zs_pool, &entry->handle, dst, dlen, gfp,
-			     folio_nid(folio));
-	if (alloc_ret)
-		goto unlock;
+	return zswap_update_entries(zc);
 
-	entry->length = dlen;
+compress_error:
+	for_each_sg(dst_sg, sg, nr_batch_pages, comp_batch_iter) {
+		if ((int)sg->length < 0) {
+			if ((int)sg->length == -ENOSPC)
+				zswap_reject_compress_poor++;
+			else
+				zswap_reject_compress_fail++;
+		}
+	}
 
-unlock:
-	if (mapped)
-		kunmap_local(dst);
-	if (comp_ret == -ENOSPC || alloc_ret == -ENOSPC)
-		zswap_reject_compress_poor++;
-	else if (comp_ret)
-		zswap_reject_compress_fail++;
-	else if (alloc_ret)
-		zswap_reject_alloc_fail++;
-
-	mutex_unlock(&comp_ctx->areq.mutex);
-	return comp_ret == 0 && alloc_ret == 0;
+	return false;
 }
 
 static bool __zswap_decompress(struct zswap_entry *entry,
@@ -1763,11 +1896,19 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
 
 	zswap_init_entries(zc, pool, objcg);
 
-	for (i = 0; i < zc->nr_pages; ++i) {
-		if (!zswap_compress(zc->folio, zc->folio_start + i,
-				    zc->entries[i], pool, zc->wb_enabled))
-			goto store_pages_failed;
-	}
+	/*
+	 * Locking the comp_ctx mutex once per store batch results in better
+	 * performance as compared to locking per compress batch.
+	 */
+	zc->comp_ctx = &raw_cpu_ptr(pool->acomp_ctx)->comp;
+	mutex_lock(&zc->comp_ctx->areq.mutex);
+
+	ret = zswap_compress(zc);
+
+	mutex_unlock(&zc->comp_ctx->areq.mutex);
+
+	if (unlikely(!ret))
+		goto store_pages_failed;
 
 	ret = zswap_tree_stores(zc, pool, objcg);
 
@@ -1833,6 +1974,7 @@ bool zswap_store(struct folio *folio)
 		mem_cgroup_put(memcg);
 	}
 
+	zc.zs_pool = pool->zs_pool;
 	rcu_read_lock();
 	zc.wb_enabled = mem_cgroup_zswap_writeback_enabled(folio_memcg(folio));
 	rcu_read_unlock();
-- 
2.39.5


^ permalink raw reply related	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 5/8] mm: zswap: Store the folio's node id in the zswap_entry.
  2026-11-20  4:19 ` [PATCH v4 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
@ 2026-11-20 22:46   ` Nhat Pham
  2026-11-20 23:01     ` Kanchana P. Sridhar
  2026-11-21  4:20   ` sashiko-bot
  1 sibling, 1 reply; 25+ messages in thread
From: Nhat Pham @ 2026-11-20 22:46 UTC (permalink / raw)
  To: Kanchana P. Sridhar
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky

On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
<kanchanapsridhar2026@gmail.com> wrote:
>
> When storing a folio in zswap, save the folio's node id in the zswap
> entry instead of relying on the zswap_entry being allocated on the same
> node. This enables optimizations such as batch allocation of zswap
> entries for large folios using kmem_cache_alloc_bulk() that does not
> support specifying a node id.
>
> Accordingly, replace the SLUB allocation in zswap_entry_cache_alloc() to
> be node-agnostic.
>
> The size of the zswap_entry remains unchanged as 'referenced' is lumped
> in with the 'length' (as it doesn't need a full unsigned int anyway).
> A build time assertion checks for the zswap_entry's 'length' bitfield
> being large enough to represent PAGE_SIZE, which can be 256KB on some
> architectures.
>
> Suggested-by: Nhat Pham <nphamcs@gmail.com>
> Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
> Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
> Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>

LGTM for the most part:

Acked-by: Nhat Pham <nphamcs@gmail.com>

> ---
> + * nid - NUMA node id of the page for which this is the zswap entry.

Maybe "NUMA node id of the original page of this zswap entry"?

>   * handle - zsmalloc allocation handle that stores the compressed page data
>   * objcg - the obj_cgroup that the compressed memory is charged to
>   * lru - handle to the pool's lru used to evict pages.
>   */
>  struct zswap_entry {
>         swp_entry_t swpentry;
> -       unsigned int length;
> -       bool referenced;
> -       u8 pool_idx;
> +       struct {
> +               unsigned int length:ZSWAP_MAX_COMPR_LENGTH_SHIFT;

I could have sworn we agreed to just inline this macro.

(btw how many bits do we still have here? )


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 5/8] mm: zswap: Store the folio's node id in the zswap_entry.
  2026-11-20 22:46   ` Nhat Pham
@ 2026-11-20 23:01     ` Kanchana P. Sridhar
  0 siblings, 0 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20 23:01 UTC (permalink / raw)
  To: Nhat Pham
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky,
	Kanchana P. Sridhar

On Fri, Nov 20, 2026 at 2:46 PM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
> <kanchanapsridhar2026@gmail.com> wrote:
> >
> > When storing a folio in zswap, save the folio's node id in the zswap
> > entry instead of relying on the zswap_entry being allocated on the same
> > node. This enables optimizations such as batch allocation of zswap
> > entries for large folios using kmem_cache_alloc_bulk() that does not
> > support specifying a node id.
> >
> > Accordingly, replace the SLUB allocation in zswap_entry_cache_alloc() to
> > be node-agnostic.
> >
> > The size of the zswap_entry remains unchanged as 'referenced' is lumped
> > in with the 'length' (as it doesn't need a full unsigned int anyway).
> > A build time assertion checks for the zswap_entry's 'length' bitfield
> > being large enough to represent PAGE_SIZE, which can be 256KB on some
> > architectures.
> >
> > Suggested-by: Nhat Pham <nphamcs@gmail.com>
> > Suggested-by: Yosry Ahmed <yosry.ahmed@linux.dev>
> > Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
> > Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
>
> LGTM for the most part:
>
> Acked-by: Nhat Pham <nphamcs@gmail.com>

Many thanks, Nhat :) !!

>
> > ---
> > + * nid - NUMA node id of the page for which this is the zswap entry.
>
> Maybe "NUMA node id of the original page of this zswap entry"?

Sure. Will make this change in my next rev/patchlet.

>
> >   * handle - zsmalloc allocation handle that stores the compressed page data
> >   * objcg - the obj_cgroup that the compressed memory is charged to
> >   * lru - handle to the pool's lru used to evict pages.
> >   */
> >  struct zswap_entry {
> >         swp_entry_t swpentry;
> > -       unsigned int length;
> > -       bool referenced;
> > -       u8 pool_idx;
> > +       struct {
> > +               unsigned int length:ZSWAP_MAX_COMPR_LENGTH_SHIFT;
>
> I could have sworn we agreed to just inline this macro.

True, however, per Yosry's earlier suggestion, I added a build time
assertion for whether ZSWAP_MAX_COMPR_LENGTH_SHIFT has enough bits to
represent the architecture's PAGE_SIZE, and could not determine a
sizeof/other API that returns a bitfield's size in bits. Hence I
resorted to the macro.

>
> (btw how many bits do we still have here? )

We now have 4 bits after "referenced":

struct zswap_entry {
        swp_entry_t                swpentry;             /*     0     8 */
        struct {
                unsigned int       length:19;            /*     8: 0  4 */

                /* Bitfield combined with next fields */

                bool               referenced:1;         /*    10: 3  1 */

                /* XXX 4 bits hole, try to pack */

                u8                 pool_idx;             /*    11     1 */
        };                                               /*     8     4 */
        int                        nid;                  /*    12     4 */
        struct zs_handle           handle;               /*    16     8 */
        struct obj_cgroup *        objcg;                /*    24     8 */
        struct list_head           lru;                  /*    32    16 */

        /* size: 48, cachelines: 1, members: 6 */
        /* last cacheline: 48 bytes */
};


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 6/8] mm: zswap: Store large folios in batches.
  2026-11-20  4:19 ` [PATCH v4 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
@ 2026-11-20 23:08   ` Nhat Pham
  2026-11-20 23:40     ` Kanchana P. Sridhar
  2026-11-20 23:25   ` Nhat Pham
  1 sibling, 1 reply; 25+ messages in thread
From: Nhat Pham @ 2026-11-20 23:08 UTC (permalink / raw)
  To: Kanchana P. Sridhar
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky

On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
<kanchanapsridhar2026@gmail.com> wrote:
>
> Use batching when storing large folios in zswap. A new constant,
> ZSWAP_MAX_BATCH_SIZE is defined as 4U to support processing batches of
> up to 4 pages at a time while storing large folios. This commit
> modifies the "store single page" approach to "store a batch of pages".
>
> As part of CPU onlining per-CPU acomp_ctx setup, ZSWAP_MAX_BATCH_SIZE
> compression buffers are allocated to eventually be able to compress and
> store ZSWAP_MAX_BATCH_SIZE pages. This happens in subsequent commits.
>
> For existing software compressors, the extra overhead is the
> the extra 'buffers' pointer, 3 additional elements in 'buffers' and 3
> additional 4K space per 'buffers[i]': in all ~12 KB per-CPU on
> x86_64. Only the first buffer is currently used.

It's not a lot but do we need multiple buffers here for all software
compressors? Other than the hardware accelerated compressors,
compression happens sequentially, no?


>
> zswap_store_page() is replaced with zswap_store_pages(), which processes
> a batch of pages and allows for batching optimizations. For now, only
> optimize allocating entries by using batch allocations from the slab
> cache.
>
> Avoid repeatedly calling mem_cgroup_zswap_writeback_enabled() for every
> page and only call it once for the folio, since the entire folio is
> charged to a single memcg. Similarly, obtain the nid once to use during
> each batch store of a folio. To facilitate this, a "struct
> zswap_store_context" is introduced to keep function parameter lists
> concise.

The wb_enabled stuff sounds like sth you can pull into its own patch?

[...]

> -       /*
> -        * We may have had an existing entry that became stale when
> -        * the folio was redirtied and now the new version is being
> -        * swapped out. Get rid of the old.
> -        */
> -       if (old)
> -               zswap_entry_free(old);
> +       for (i = 0; i < zc->nr_pages; ++i) {
> +               struct zswap_entry *old, *entry = zc->entries[i];
> +
> +               old = xa_store(swap_zswap_tree(entry->swpentry),
> +                              swp_offset(entry->swpentry),
> +                              entry, GFP_KERNEL);

This range is guaranteed to be contiguous and always belong to the
same zswap tree right? Why are we not using xas API here?

(Or maybe we can do this as follow-up/in a different patch?)


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 0/8] Batched zswap_store() framework with compression batching
  2026-11-20  4:18 [PATCH v4 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
                   ` (7 preceding siblings ...)
  2026-11-20  4:19 ` [PATCH v4 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches Kanchana P. Sridhar
@ 2026-11-20 23:16 ` Nhat Pham
  2026-11-21  1:04   ` Kanchana P. Sridhar
  8 siblings, 1 reply; 25+ messages in thread
From: Nhat Pham @ 2026-11-20 23:16 UTC (permalink / raw)
  To: Kanchana P. Sridhar
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky

On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
<kanchanapsridhar2026@gmail.com> wrote:
>
>
> v4: Batched zswap_store() framework with compression batching
> =============================================================
>
> This series is a spin-off of "v14: zswap compression batching with optimized
> iaa_crypto driver" [1], and focuses on the generic crypto_acomp and zswap
> compression batching enablers. Resuming work on this series following a gap
> since v14 (January 2026). Please note that my email address and affiliation have
> changed from kanchana.p.sridhar@intel.com to kanchanapsridhar2026@gmail.com. I
> have retained the original Intel sign-offs on prior patches for historical
> attribution while signing off v4 of this series with my current address.
>
> The main focus of v4 was to establish a performance measurement methodology to
> quantify the benefits to high level metrics by using zswap compression
> batching. This was done using an Intel Sapphire Rapids bare-metal instance with
> 48 physical cores and by running "usemem30", i.e., 30 concurrent usemem
> processes, each of which allocates 10G, in a 150G memory.high contrained cgroup,
> for 10 iterations. The environment was set up to minimize performance jitter so
> we can understand the system-level/workload-level advantages of zswap
> compression batching when running workloads at scale.
>
> To summarize the main contribution of this patch series: Batching is introduced
> in zswap_store() of large folios: we carve out a large folio into batches of
> ZSWAP_MAX_BATCH_SIZE (4 pages) for software compressors, and store each
> batch. Moving the fundamental operations that need to be happen to store a page
> to now work at the granularity of a batch offers opportunities to optimize them
> from these perspectives:
>
> 1) Utilize existing kernel API that minimize obtaining locks for bulk
>    operations, e.g. kmem_cache_alloc_bulk()/kmem_cache_free_bulk().
>
> 2) We keep the working set for each of the core store functionalities
>    (alloc/initialize zswap entries, compress pages, store pages in zsmalloc,
>    pool/objcg reference increments, objcg charging) to be focused on a
>    batch, thereby minimizing cache evictions and benefiting from cache
>    locality.
>
> 3) Along the lines of (1), the main zswap_compress() batching routine obtains
>    the comp_ctx mutex lock once per batch, and releases it after it has
>    processed all pages in that batch.
>
> 4) Any zswap store functionality that works on a batch can potentially be
>    optimized via bulk-processing/parallelized as a next step, to further improve
>    performance.
>
> Performance testing: usemem30 with 100% over-commit:
> =====================================================
> Validation was done with mm-unstable as of 11-17-2026 (Baseline: commit
> 866f761fd414) and with this patch series.
>
> v4 was validated on an Intel Xeon Sapphire Rapids bare-metal instance with
> 48 cores.
>
>  Bare-metal instance specs:
>  --------------------------
>  - 48 physical cores
>  - 190G RAM
>  - 200G SWAP disk partition
>
>  Workload: 30 concurrent usemem processes, each allocates 10G:
>  -------------------------------------------------------------
>  The vm-scalability "usemem" test (unmodified version) was run in a
>  cgroup with memory.high fixed at 150G. There is no swap limit set for the
>  cgroup.
>
>  To quantify the performance benefit of zswap compress batching at scale,
>  30 usemem processes were run ("usemem30"), each allocating and writing 10G of
>  memory, and sleeping for 10 sec before exiting.
>
>   usemem -w -O -b 5 -s 10 -n 30 10g
>   echo 0 > /sys/module/zswap/parameters/shrinker_enabled
>
>  Performance validation methodology:
>  -----------------------------------
>  These settings were used to minimize variation:
>
>  - All cores were fixed to run at 2.4GHz with "performance" scaling governor.
>  - Turbo boost was disabled.
>  - CPU power-saving C-states restricted with:
>
>      GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1 processor.max_cstate=1"
>
>  The usemem30 experiments were run with the baseline mm-unstable as of
>  11-17-2026 (commit 866f761fd414) and with this patch-series.
>
>  After booting into each kernel:
>
>   1) Constrain the cgroup's memory.high to 150G.
>
>   2) Run 2 warm-up iterations of usemem30. Don't use this data for benchmarking
>      since sys time typically skews higher for initial runs.
>
>   3) Next, run 10 benchmarking iterations of usemem30:
>      - Log these high-level metrics: Sys time and individual usemem throughput.
>
>   4) Sys time and Throughput metrics were gathered for three mTHP settings:
>
>      - 2048kB
>      - 1024kB
>      - 64kB
>
>   5) Benchmark summary data includes Sys time and Throughput components from the
>      10 iterations of usemem30:
>
>      - Total Samples
>      - Mean Value
>      - Std Deviation
>      - CoV (Variance): Coefficient of variation (relative standard deviation)
>        measures spread relative to the mean rather than statistical variance.
>
>  An LLM was used to help generate the benchmarking and data processing scripts.
>
> usemem30 Benchmarking Summary:
> ==============================
>
>  ---------------------------------------------------
>     Improvement with v4 zswap compress batching
>  ---------------------------------------------------
>   mTHP enabled:       2048kB     1024kB       64kB
>  ---------------------------------------------------
>       Sys time:      -13.87%     -2.64%     -1.72%
>     Throughput:       +3.85%     +0.42%     +1.31%
>  ---------------------------------------------------

Have you tried perf-ing the store path to see what was the overhead
source (before and after this optimization)? I'm curious.

Mostly LGTM - still taking a look...


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 6/8] mm: zswap: Store large folios in batches.
  2026-11-20  4:19 ` [PATCH v4 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
  2026-11-20 23:08   ` Nhat Pham
@ 2026-11-20 23:25   ` Nhat Pham
  2026-11-20 23:51     ` Kanchana P. Sridhar
  1 sibling, 1 reply; 25+ messages in thread
From: Nhat Pham @ 2026-11-20 23:25 UTC (permalink / raw)
  To: Kanchana P. Sridhar
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky

On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
<kanchanapsridhar2026@gmail.com> wrote:
>
> Use batching when storing large folios in zswap. A new constant,
> ZSWAP_MAX_BATCH_SIZE is defined as 4U to support processing batches of
> up to 4 pages at a time while storing large folios. This commit
> modifies the "store single page" approach to "store a batch of pages".
>
> As part of CPU onlining per-CPU acomp_ctx setup, ZSWAP_MAX_BATCH_SIZE
> compression buffers are allocated to eventually be able to compress and
> store ZSWAP_MAX_BATCH_SIZE pages. This happens in subsequent commits.
>
> For existing software compressors, the extra overhead is the
> the extra 'buffers' pointer, 3 additional elements in 'buffers' and 3
> additional 4K space per 'buffers[i]': in all ~12 KB per-CPU on
> x86_64. Only the first buffer is currently used.
>
> zswap_store_page() is replaced with zswap_store_pages(), which processes
> a batch of pages and allows for batching optimizations. For now, only
> optimize allocating entries by using batch allocations from the slab
> cache.
>
> Avoid repeatedly calling mem_cgroup_zswap_writeback_enabled() for every
> page and only call it once for the folio, since the entire folio is
> charged to a single memcg. Similarly, obtain the nid once to use during
> each batch store of a folio. To facilitate this, a "struct
> zswap_store_context" is introduced to keep function parameter lists
> concise.
>
> Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
> Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
> ---
>  mm/zswap.c | 307 +++++++++++++++++++++++++++++++++++++----------------
>  1 file changed, 217 insertions(+), 90 deletions(-)
>
> diff --git a/mm/zswap.c b/mm/zswap.c
> index 78d9694f75b5..df59ab0c1029 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
> @@ -85,6 +85,9 @@ static bool zswap_pool_reached_full;
>
>  #define ZSWAP_PARAM_UNSET ""
>
> +/* Limit the batch size to limit per-CPU memory usage for dst buffers. */
> +#define ZSWAP_MAX_BATCH_SIZE 4U
> +
>  static int zswap_setup(void);
>
>  /* Enable/disable zswap */
> @@ -144,10 +147,12 @@ struct zswap_acomp_req {
>         struct mutex mutex;
>  };
>
> -/* The compression mutex also protects the output buffer. */
> +/*
> + * The compression mutex also protects the output buffer(s).
> + */
>  struct zswap_comp_ctx {
>         struct zswap_acomp_req areq;
> -       u8 *buffer;
> +       u8 **buffers;
>  };
>
>  /* Separate requests, so that decompression does not wait for compression. */
> @@ -162,6 +167,7 @@ struct crypto_acomp_ctx {
>   * The only case where lru_lock is not acquired while holding tree.lock is
>   * when a zswap_entry is taken off the lru for writeback, in that case it
>   * needs to be verified that it's still valid in the tree.
> + *
>   */
>  struct zswap_pool {
>         struct zs_pool *zs_pool;
> @@ -226,6 +232,15 @@ struct zswap_entry {
>         struct list_head lru;
>  };
>
> +struct zswap_store_context {
> +       struct folio *folio;
> +       long folio_start;
> +       bool wb_enabled;
> +       int nid;
> +       u8 nr_pages;
> +       struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE];
> +};
> +
>  /*
>   * No RCU section is needed around the returned pointer: a stored entry pins
>   * its pool via percpu_ref (taken in zswap_store_page()), so the id cannot be
> @@ -284,6 +299,8 @@ static void __zswap_pool_empty(struct percpu_ref *ref);
>
>  static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
>  {
> +       u8 i;
> +
>         if (!acomp_ctx)
>                 return;
>
> @@ -302,8 +319,12 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
>
>         acomp_ctx->acomp = NULL;
>
> -       kfree(acomp_ctx->comp.buffer);
> -       acomp_ctx->comp.buffer = NULL;
> +       if (acomp_ctx->comp.buffers) {
> +               for (i = 0; i < ZSWAP_MAX_BATCH_SIZE; ++i)
> +                       kfree(acomp_ctx->comp.buffers[i]);
> +               kfree(acomp_ctx->comp.buffers);
> +       }
> +       acomp_ctx->comp.buffers = NULL;
>  }
>
>  static struct zswap_pool *zswap_pool_create(char *compressor)
> @@ -388,6 +409,7 @@ static struct zswap_pool *zswap_pool_create(char *compressor)
>  cpuhp_add_fail:
>         for_each_possible_cpu(cpu)
>                 acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu));
> +
>  error:
>         if (pool->acomp_ctx)
>                 free_percpu(pool->acomp_ctx);
> @@ -778,6 +800,44 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)
>         kmem_cache_free(zswap_entry_cache, entry);
>  }
>
> +static inline void zswap_entries_cache_free_batch(
> +       struct zswap_entry **entries,
> +       u8 nr_entries)
> +{
> +       /*
> +        * It is okay to use this to free entries allocated separately
> +        * by zswap_entry_cache_alloc().
> +        */
> +       kmem_cache_free_bulk(zswap_entry_cache, nr_entries, (void **)entries);
> +}
> +
> +static bool zswap_entries_cache_alloc_batch(
> +       struct zswap_store_context *zc,
> +       gfp_t gfp)
> +{
> +       u8 i;
> +
> +       bool ret = kmem_cache_alloc_bulk(zswap_entry_cache, gfp,
> +                                        zc->nr_pages, (void **)zc->entries);
> +
> +       if (likely(ret)) {
> +               for (i = 0; i < zc->nr_pages; ++i)
> +                       zs_handle_init(&zc->entries[i]->handle);
> +       } else {
> +               for (i = 0; i < zc->nr_pages; ++i) {
> +                       zc->entries[i] = zswap_entry_cache_alloc(GFP_KERNEL);

Oh and why are we using GFP_KERNEL in the fallback path, but passing a
gfp to kmem_cache_alloc_bulk above?

(and it seems like the only caller of
zswap_entries_cache_alloc_batch() passed GFP_KERNEL for gfp anyway?).

Let's just drop gfp and use GFP_KERNEL if that's the intention?

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 6/8] mm: zswap: Store large folios in batches.
  2026-11-20 23:08   ` Nhat Pham
@ 2026-11-20 23:40     ` Kanchana P. Sridhar
  0 siblings, 0 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20 23:40 UTC (permalink / raw)
  To: Nhat Pham
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky,
	Kanchana P. Sridhar

On Fri, Nov 20, 2026 at 3:08 PM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
> <kanchanapsridhar2026@gmail.com> wrote:
> >
> > Use batching when storing large folios in zswap. A new constant,
> > ZSWAP_MAX_BATCH_SIZE is defined as 4U to support processing batches of
> > up to 4 pages at a time while storing large folios. This commit
> > modifies the "store single page" approach to "store a batch of pages".
> >
> > As part of CPU onlining per-CPU acomp_ctx setup, ZSWAP_MAX_BATCH_SIZE
> > compression buffers are allocated to eventually be able to compress and
> > store ZSWAP_MAX_BATCH_SIZE pages. This happens in subsequent commits.
> >
> > For existing software compressors, the extra overhead is the
> > the extra 'buffers' pointer, 3 additional elements in 'buffers' and 3
> > additional 4K space per 'buffers[i]': in all ~12 KB per-CPU on
> > x86_64. Only the first buffer is currently used.
>
> It's not a lot but do we need multiple buffers here for all software
> compressors? Other than the hardware accelerated compressors,
> compression happens sequentially, no?

That's true, compression happens sequentially. However, multiple
buffers here helps software compressors because it allows the
"batch-centric" compute and lean working set concept, which results in
the sys time improvements (cache locality, fewer cache evictions).
Hence, I think multiple buffers here benefit latency.

>
>
> >
> > zswap_store_page() is replaced with zswap_store_pages(), which processes
> > a batch of pages and allows for batching optimizations. For now, only
> > optimize allocating entries by using batch allocations from the slab
> > cache.
> >
> > Avoid repeatedly calling mem_cgroup_zswap_writeback_enabled() for every
> > page and only call it once for the folio, since the entire folio is
> > charged to a single memcg. Similarly, obtain the nid once to use during
> > each batch store of a folio. To facilitate this, a "struct
> > zswap_store_context" is introduced to keep function parameter lists
> > concise.
>
> The wb_enabled stuff sounds like sth you can pull into its own patch?

Do you mean we should obtain wb_enabled status once and pass it to
zswap_store_page(), and factor this into a patch before patch 6 in
this series? Sure, I can do this.

>
> [...]
>
> > -       /*
> > -        * We may have had an existing entry that became stale when
> > -        * the folio was redirtied and now the new version is being
> > -        * swapped out. Get rid of the old.
> > -        */
> > -       if (old)
> > -               zswap_entry_free(old);
> > +       for (i = 0; i < zc->nr_pages; ++i) {
> > +               struct zswap_entry *old, *entry = zc->entries[i];
> > +
> > +               old = xa_store(swap_zswap_tree(entry->swpentry),
> > +                              swp_offset(entry->swpentry),
> > +                              entry, GFP_KERNEL);
>
> This range is guaranteed to be contiguous and always belong to the
> same zswap tree right? Why are we not using xas API here?
>
> (Or maybe we can do this as follow-up/in a different patch?)

Great point. I was looking into this based on your earlier comment,
and was studying the repercussions of obtaining the xa_lock once,
followed by __xa_store() or xas() API, but still needed to figure out
a few things, hence tabled this for a subsequent patch:

 - Impact on zswap loads from being locked out for significantly more
time, potentially causing page fault handling errors.
 - Impact of holding the xa_lock for 4 entries at a time during zswap
stores: would it result in errors due to the task blocked in an atomic
path.
 - Ascertaining if the assumption is true: contiguous range of offsets
is guaranteed to always belong to the same xarray.
 - Comprehend tree rebalancing with xas approach, would it change the
above assumption.

This seemed to require some more focus, and deserving of a patch of
its own. I agree, we should pursue this in a follow-up patch.

Thanks,
Kanchana

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 7/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
  2026-11-20  4:19 ` [PATCH v4 7/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations Kanchana P. Sridhar
@ 2026-11-20 23:45   ` Nhat Pham
  2026-11-21  1:36     ` Kanchana P. Sridhar
  0 siblings, 1 reply; 25+ messages in thread
From: Nhat Pham @ 2026-11-20 23:45 UTC (permalink / raw)
  To: Kanchana P. Sridhar
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky

On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
<kanchanapsridhar2026@gmail.com> wrote:
>
> Subsequent patch series will be submitted, that implement ideas building
> upon this batching framework:
>
>  1) Replacing zsmalloc with SLUB allocator, comprising of kmem_caches
>     for a finite # of compressed size bins. This enables bulk allocation
>     of compressed buffers in a batch that belong to the same size
>     bin. It gets rid of significant zsmalloc latency overhead (locks,
>     compressed data spanning zspages, etc.), without compromising memory
>     efficiency.

I don't think this will be a good idea, because it will make
compressed memory unmovable.

>
>  2) vmscan reclaim batching of hybrid any-order folios, that will submit
>     a batch of folios to zswap_store()'s batching framework to optimize
>     storing of batches of pages from different folios, in addition to
>     within large folios.

This one might be more complicated than its worth, but I'm not sure.

>
> Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>

Code mostly LGTM.

> ---
>  include/linux/memcontrol.h |  12 +++
>  mm/memcontrol.c            |  23 ++++-
>  mm/zswap.c                 | 181 +++++++++++++++++++++++++------------
>  3 files changed, 153 insertions(+), 63 deletions(-)
>
> diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> index 980358bc6f49..93f060818c14 100644
> --- a/include/linux/memcontrol.h
> +++ b/include/linux/memcontrol.h
> @@ -1244,6 +1244,11 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css)
>         return NULL;
>  }
>
> +static inline void obj_cgroup_get_many(struct obj_cgroup *objcg,
> +                                      unsigned long nr)
> +{
> +}
> +
>  static inline void obj_cgroup_get(struct obj_cgroup *objcg)
>  {
>  }
> @@ -1921,6 +1926,8 @@ static inline void mem_cgroup_calculate_protection_path(struct mem_cgroup *root,
>  #if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP)
>  bool obj_cgroup_may_zswap(struct obj_cgroup *objcg);
>  void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size);
> +void obj_cgroup_charge_zswap_many(struct obj_cgroup *objcg, size_t size,
> +                                 u8 nr_pages, u8 nr_incomp_pages);
>  void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size);
>  bool mem_cgroup_zswap_writeback_enabled(const struct mem_cgroup *memcg);
>  #else
> @@ -1932,6 +1939,11 @@ static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg,
>                                            size_t size)
>  {
>  }

Why do we still need obj_cgroup_charge_zswap() at all?

> +static inline void obj_cgroup_charge_zswap_many(struct obj_cgroup *objcg,
> +                                               size_t size, u8 nr_pages,
> +                                               u8 nr_incomp_pages)
> +{
> +}
>  static inline void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg,
>                                              size_t size)
>  {
> diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> index 091401cecf39..2236f5ba6304 100644
> --- a/mm/memcontrol.c
> +++ b/mm/memcontrol.c
> @@ -6345,6 +6345,23 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
>   * compression and storage in zswap for this cgroup to go ahead.
>   */
>  void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size)
> +{
> +       obj_cgroup_charge_zswap_many(objcg, size, 1, (u8)(size == PAGE_SIZE));
> +}
> +
> +/**
> + * obj_cgroup_charge_zswap_many - bulk charge compression backend memory
> + *                                from compressing a batch of pages.
> + * @objcg: the object cgroup
> + * @size: total compressed length of a batch of compressed objects
> + * @nr_pages: # of pages that were compressed to store in zswap
> + * @nr_incomp_pages: # of incompressible pages to store in zswap
> + *
> + * This forces the charge after obj_cgroup_may_zswap() allowed
> + * compression and storage in zswap for this cgroup to go ahead.
> + */
> +void obj_cgroup_charge_zswap_many(struct obj_cgroup *objcg, size_t size,
> +                                 u8 nr_pages, u8 nr_incomp_pages)
>  {
>         struct mem_cgroup *memcg;
>
> @@ -6363,9 +6380,9 @@ void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size)
>         rcu_read_lock();
>         memcg = obj_cgroup_memcg(objcg);
>         mod_memcg_state(memcg, MEMCG_ZSWAP_B, size);
> -       mod_memcg_state(memcg, MEMCG_ZSWAPPED, 1);
> -       if (size == PAGE_SIZE)
> -               mod_memcg_state(memcg, MEMCG_ZSWAP_INCOMP, 1);
> +       mod_memcg_state(memcg, MEMCG_ZSWAPPED, nr_pages);
> +       if (nr_incomp_pages)
> +               mod_memcg_state(memcg, MEMCG_ZSWAP_INCOMP, nr_incomp_pages);
>         rcu_read_unlock();
>  }
>
> diff --git a/mm/zswap.c b/mm/zswap.c
> index df59ab0c1029..5e1fb5624ffb 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
> @@ -491,12 +491,12 @@ static int __must_check zswap_pool_tryget(struct zswap_pool *pool)
>  }
>
>  /* The caller must already have a reference. */
> -static void zswap_pool_get(struct zswap_pool *pool)
> +static inline void zswap_pool_get_many(struct zswap_pool *pool, u8 nr)
>  {
> -       percpu_ref_get(&pool->ref);
> +       percpu_ref_get_many(&pool->ref, nr);
>  }
>
> -static void zswap_pool_put(struct zswap_pool *pool)
> +static inline void zswap_pool_put(struct zswap_pool *pool)
>  {
>         percpu_ref_put(&pool->ref);
>  }
> @@ -1603,55 +1603,60 @@ static void shrink_worker(struct work_struct *w)
>  * main API
>  **********************************/
>
> +/* Functions called by zswap_store(), that operate on batches of pages. */
> +
>  /*
> - * Store multiple pages in @zc->folio, starting from the page at index @start up
> - * to the page at index @end-1.
> + * zswap_init_entries() co-locates most of the zswap entry fields' setup under
> + * the assumption that all phases of store processing will succeed. An important
> + * marker to simplify catch-all cleanup in zswap_store_pages() is done by
> + * initializing each entry's @handle to ERR_PTR(-EINVAL).
>   */
> -static bool zswap_store_pages(struct zswap_store_context *zc,
> -                             long start,
> -                             long end,
> -                             struct zswap_pool *pool,
> -                             struct obj_cgroup *objcg)
> +static inline void zswap_init_entries(struct zswap_store_context *zc,
> +                                     struct zswap_pool *pool,
> +                                     struct obj_cgroup *objcg)
>  {
> -       u8 i, store_fail_idx = 0;
> -       bool ret;
> -
> -       zc->nr_pages = end - start;
> -       zc->folio_start = start;
> -
> -       VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
> -
> -       ret = zswap_entries_cache_alloc_batch(zc, GFP_KERNEL);
> -       if (unlikely(!ret))
> -               return false;
> +       struct zswap_entry *entry;
> +       u8 i;
>
>         /*
>          * We co-locate entry initialization as much as possible here to
>          * minimize potential cache misses.
>          */
>         for (i = 0; i < zc->nr_pages; ++i) {
> +               entry = zc->entries[i];
>                 /*
> -                * Set pool_idx before the xa_store() below publishes the entry, or a
> -                * concurrent reader could resolve a stale pool_idx left by slab reuse
> -                * to an unrelated live pool.
> +                * Set pool_idx before the xa_store() in zswap_tree_stores()
> +                * publishes the entry, or a concurrent reader could resolve a
> +                * stale pool_idx left by slab reuse to an unrelated live pool.
>                  */
> -               zc->entries[i]->pool_idx = pool->idx;
> -               zc->entries[i]->swpentry = folio_swap_entry(zc->folio,
> +               entry->pool_idx = pool->idx;
> +               entry->swpentry = folio_swap_entry(zc->folio,
>                                                         zc->folio_start + i);
> -               zc->entries[i]->objcg = objcg;
> -               zc->entries[i]->referenced = true;
> -               zc->entries[i]->nid = zc->nid;
> -               INIT_LIST_HEAD(&zc->entries[i]->lru);
> +               entry->objcg = objcg;
> +               entry->referenced = true;
> +               entry->nid = zc->nid;
> +               INIT_LIST_HEAD(&entry->lru);
>         }
> +}
>
> -       for (i = 0; i < zc->nr_pages; ++i) {
> -               if (!zswap_compress(zc->folio, zc->folio_start + i,
> -                                   zc->entries[i], pool, zc->wb_enabled))
> -                       goto store_pages_failed;
> -       }
> +/*
> + * zswap_tree_stores() stores the batch entries in xarray and adds them to the
> + * zswap LRU list. If an error occurs during xa_store(), localized state
> + * unwinding is done to erase prior batch entries from the xarray and delete
> + * them from the zswap LRU list. Remaining batch cleanup occurs in
> + * zswap_store_pages(). As batch entries are processed, we maintain aggregate
> + * statistics to do the accounting in one bulk operation.
> + */
> +static bool zswap_tree_stores(struct zswap_store_context *zc,
> +                             struct zswap_pool *pool,
> +                             struct obj_cgroup *objcg)
> +{
> +       u8 i, store_fail_idx = 0, nr_incomp = 0;
> +       size_t total_batch_comp_length = 0;
> +       struct zswap_entry *old, *entry;
>
>         for (i = 0; i < zc->nr_pages; ++i) {
> -               struct zswap_entry *old, *entry = zc->entries[i];
> +               entry = zc->entries[i];
>
>                 old = xa_store(swap_zswap_tree(entry->swpentry),
>                                swp_offset(entry->swpentry),
> @@ -1663,13 +1668,11 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
>                         zswap_reject_alloc_fail++;
>                         /*
>                          * Entries up to this point have been stored in the
> -                        * xarray. zswap_store() will erase them from the xarray
> -                        * and call zswap_entry_free(). Local cleanup in
> -                        * 'store_pages_failed' only needs to happen for
> -                        * entries from [@i to @nr_pages).
> +                        * xarray. Local cleanup in 'tree_store_failed' needs to
> +                        * erase entries up until now.
>                          */
>                         store_fail_idx = i;
> -                       goto store_pages_failed;
> +                       goto tree_store_failed;
>                 }
>
>                 /*
> @@ -1680,22 +1683,10 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
>                 if (unlikely(old))
>                         zswap_entry_free(old);
>
> -               /*
> -                * The entry is successfully compressed and stored in the tree,
> -                * and further failures will be cleaned up in zswap_store().
> -                * Grab refs to the pool and objcg, charge zswap memory, and
> -                * increment zswap_stored_pages. The opposite actions will be
> -                * performed by zswap_entry_free() when the entry is removed
> -                * from the tree.
> -                */
> -               zswap_pool_get(pool);
> -               if (objcg) {
> -                       obj_cgroup_get(objcg);
> -                       obj_cgroup_charge_zswap(objcg, entry->length);
> -               }
> -               atomic_long_inc(&zswap_stored_pages);
> +               total_batch_comp_length += entry->length;
> +
>                 if (unlikely(entry->length == PAGE_SIZE))
> -                       atomic_long_inc(&zswap_stored_incompressible_pages);
> +                       ++nr_incomp;
>
>                 /*
>                  * We finish by adding the entry to the LRU while it's already
> @@ -1711,17 +1702,87 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
>                         zswap_lru_add(entry);
>         }
>
> +       /*
> +        * Each batch entry is successfully compressed and stored in the tree,
> +        * and further failures will be cleaned up in zswap_store().
> +        * Next, we do bulk accounting for the batch: grab refs to the pool and
> +        * objcg, charge zswap memory, increment zswap_stored_pages and
> +        * zswap_stored_incompressible_pages. The opposite actions will be
> +        * performed by zswap_entry_free() when the entry is removed from the
> +        * tree.
> +        */
> +       zswap_pool_get_many(pool, zc->nr_pages);
> +       if (objcg) {
> +               obj_cgroup_get_many(objcg, zc->nr_pages);
> +               obj_cgroup_charge_zswap_many(objcg, total_batch_comp_length,
> +                                            zc->nr_pages, nr_incomp);
> +       }
> +       atomic_long_add(zc->nr_pages, &zswap_stored_pages);
> +       if (unlikely(nr_incomp))
> +               atomic_long_add(nr_incomp, &zswap_stored_incompressible_pages);
> +
> +       return true;
> +

Can you add a comment that we only clean up the current batch here,
and the clean up of other batches in the folio are deferred to
zswap_store() (or whereever up the stack it is handled?).

> +tree_store_failed:
> +       for (i = 0; i < store_fail_idx; ++i) {
> +               xa_erase(swap_zswap_tree(zc->entries[i]->swpentry),
> +                        swp_offset(zc->entries[i]->swpentry));

xas API?

> +               zswap_lru_del(zc->entries[i]);
> +       }
> +
> +       return false;
> +}
> +
> +/*


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 6/8] mm: zswap: Store large folios in batches.
  2026-11-20 23:25   ` Nhat Pham
@ 2026-11-20 23:51     ` Kanchana P. Sridhar
  0 siblings, 0 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-20 23:51 UTC (permalink / raw)
  To: Nhat Pham
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky,
	Kanchana P. Sridhar

On Fri, Nov 20, 2026 at 3:25 PM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
> <kanchanapsridhar2026@gmail.com> wrote:
> >
> > Use batching when storing large folios in zswap. A new constant,
> > ZSWAP_MAX_BATCH_SIZE is defined as 4U to support processing batches of
> > up to 4 pages at a time while storing large folios. This commit
> > modifies the "store single page" approach to "store a batch of pages".
> >
> > As part of CPU onlining per-CPU acomp_ctx setup, ZSWAP_MAX_BATCH_SIZE
> > compression buffers are allocated to eventually be able to compress and
> > store ZSWAP_MAX_BATCH_SIZE pages. This happens in subsequent commits.
> >
> > For existing software compressors, the extra overhead is the
> > the extra 'buffers' pointer, 3 additional elements in 'buffers' and 3
> > additional 4K space per 'buffers[i]': in all ~12 KB per-CPU on
> > x86_64. Only the first buffer is currently used.
> >
> > zswap_store_page() is replaced with zswap_store_pages(), which processes
> > a batch of pages and allows for batching optimizations. For now, only
> > optimize allocating entries by using batch allocations from the slab
> > cache.
> >
> > Avoid repeatedly calling mem_cgroup_zswap_writeback_enabled() for every
> > page and only call it once for the folio, since the entire folio is
> > charged to a single memcg. Similarly, obtain the nid once to use during
> > each batch store of a folio. To facilitate this, a "struct
> > zswap_store_context" is introduced to keep function parameter lists
> > concise.
> >
> > Signed-off-by: Kanchana P Sridhar <kanchana.p.sridhar@intel.com>
> > Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
> > ---
> >  mm/zswap.c | 307 +++++++++++++++++++++++++++++++++++++----------------
> >  1 file changed, 217 insertions(+), 90 deletions(-)
> >
> > diff --git a/mm/zswap.c b/mm/zswap.c
> > index 78d9694f75b5..df59ab0c1029 100644
> > --- a/mm/zswap.c
> > +++ b/mm/zswap.c
> > @@ -85,6 +85,9 @@ static bool zswap_pool_reached_full;
> >
> >  #define ZSWAP_PARAM_UNSET ""
> >
> > +/* Limit the batch size to limit per-CPU memory usage for dst buffers. */
> > +#define ZSWAP_MAX_BATCH_SIZE 4U
> > +
> >  static int zswap_setup(void);
> >
> >  /* Enable/disable zswap */
> > @@ -144,10 +147,12 @@ struct zswap_acomp_req {
> >         struct mutex mutex;
> >  };
> >
> > -/* The compression mutex also protects the output buffer. */
> > +/*
> > + * The compression mutex also protects the output buffer(s).
> > + */
> >  struct zswap_comp_ctx {
> >         struct zswap_acomp_req areq;
> > -       u8 *buffer;
> > +       u8 **buffers;
> >  };
> >
> >  /* Separate requests, so that decompression does not wait for compression. */
> > @@ -162,6 +167,7 @@ struct crypto_acomp_ctx {
> >   * The only case where lru_lock is not acquired while holding tree.lock is
> >   * when a zswap_entry is taken off the lru for writeback, in that case it
> >   * needs to be verified that it's still valid in the tree.
> > + *
> >   */
> >  struct zswap_pool {
> >         struct zs_pool *zs_pool;
> > @@ -226,6 +232,15 @@ struct zswap_entry {
> >         struct list_head lru;
> >  };
> >
> > +struct zswap_store_context {
> > +       struct folio *folio;
> > +       long folio_start;
> > +       bool wb_enabled;
> > +       int nid;
> > +       u8 nr_pages;
> > +       struct zswap_entry *entries[ZSWAP_MAX_BATCH_SIZE];
> > +};
> > +
> >  /*
> >   * No RCU section is needed around the returned pointer: a stored entry pins
> >   * its pool via percpu_ref (taken in zswap_store_page()), so the id cannot be
> > @@ -284,6 +299,8 @@ static void __zswap_pool_empty(struct percpu_ref *ref);
> >
> >  static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
> >  {
> > +       u8 i;
> > +
> >         if (!acomp_ctx)
> >                 return;
> >
> > @@ -302,8 +319,12 @@ static void acomp_ctx_free(struct crypto_acomp_ctx *acomp_ctx)
> >
> >         acomp_ctx->acomp = NULL;
> >
> > -       kfree(acomp_ctx->comp.buffer);
> > -       acomp_ctx->comp.buffer = NULL;
> > +       if (acomp_ctx->comp.buffers) {
> > +               for (i = 0; i < ZSWAP_MAX_BATCH_SIZE; ++i)
> > +                       kfree(acomp_ctx->comp.buffers[i]);
> > +               kfree(acomp_ctx->comp.buffers);
> > +       }
> > +       acomp_ctx->comp.buffers = NULL;
> >  }
> >
> >  static struct zswap_pool *zswap_pool_create(char *compressor)
> > @@ -388,6 +409,7 @@ static struct zswap_pool *zswap_pool_create(char *compressor)
> >  cpuhp_add_fail:
> >         for_each_possible_cpu(cpu)
> >                 acomp_ctx_free(per_cpu_ptr(pool->acomp_ctx, cpu));
> > +
> >  error:
> >         if (pool->acomp_ctx)
> >                 free_percpu(pool->acomp_ctx);
> > @@ -778,6 +800,44 @@ static void zswap_entry_cache_free(struct zswap_entry *entry)
> >         kmem_cache_free(zswap_entry_cache, entry);
> >  }
> >
> > +static inline void zswap_entries_cache_free_batch(
> > +       struct zswap_entry **entries,
> > +       u8 nr_entries)
> > +{
> > +       /*
> > +        * It is okay to use this to free entries allocated separately
> > +        * by zswap_entry_cache_alloc().
> > +        */
> > +       kmem_cache_free_bulk(zswap_entry_cache, nr_entries, (void **)entries);
> > +}
> > +
> > +static bool zswap_entries_cache_alloc_batch(
> > +       struct zswap_store_context *zc,
> > +       gfp_t gfp)
> > +{
> > +       u8 i;
> > +
> > +       bool ret = kmem_cache_alloc_bulk(zswap_entry_cache, gfp,
> > +                                        zc->nr_pages, (void **)zc->entries);
> > +
> > +       if (likely(ret)) {
> > +               for (i = 0; i < zc->nr_pages; ++i)
> > +                       zs_handle_init(&zc->entries[i]->handle);
> > +       } else {
> > +               for (i = 0; i < zc->nr_pages; ++i) {
> > +                       zc->entries[i] = zswap_entry_cache_alloc(GFP_KERNEL);
>
> Oh and why are we using GFP_KERNEL in the fallback path, but passing a
> gfp to kmem_cache_alloc_bulk above?
>
> (and it seems like the only caller of
> zswap_entries_cache_alloc_batch() passed GFP_KERNEL for gfp anyway?).
>
> Let's just drop gfp and use GFP_KERNEL if that's the intention?

Good catch! Makes sense, I will drop the gfp and use GFP_KERNEL in the next rev.

Thanks,
Kanchana

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches.
  2026-11-20  4:19 ` [PATCH v4 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches Kanchana P. Sridhar
@ 2026-11-20 23:55   ` Nhat Pham
  2026-11-21  1:17     ` Kanchana P. Sridhar
  2026-11-21  4:20   ` sashiko-bot
  1 sibling, 1 reply; 25+ messages in thread
From: Nhat Pham @ 2026-11-20 23:55 UTC (permalink / raw)
  To: Kanchana P. Sridhar
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky

On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
<kanchanapsridhar2026@gmail.com> wrote:
>
> A new batching implementation of zswap_compress() is provided for
> compressors that do and do not support batching. This eliminates code
> duplication and facilitates code maintainability with the introduction
> of compress batching.
>
> The vectorized implementation of calling the earlier zswap_compress()
> sequentially, one page at a time in zswap_store_pages(), is replaced
> with this new version of zswap_compress() that accepts multiple pages to
> compress as a batch.
>
> When the crypto_acomp is used to set up the per-CPU acomp_ctx
> during CPU onlining, the compressor's batch_size is obtained by calling:
>
>   crypto_acomp_batch_size(acomp_ctx->acomp)
>
> This is bounded by ZSWAP_MAX_BATCH_SIZE to limit the memory
> overhead and to reflect the store batch constraint of max # of pages in
> a batch. The clamped batch_size is stored in the per-CPU
> acomp_ctx->comp.compr_batch_size.
>
> As noted in previous commits in this series, ZSWAP_MAX_BATCH_SIZE
> per-CPU compression buffers are available for sequential and batching
> compression algorithms.
>
> Regardless of compression batching, we always process large folios in
> batches of ZSWAP_MAX_BATCH_SIZE.
>
> If the compressor does not support batching, each page in the batch is
> compressed sequentially. For batching compressors, up to
> acomp_ctx->comp.compr_batch_size pages in the batch are compressed per
> call to crypto_acomp_compress(). The first phase of zswap_compress()
> thus only calls crypto_acomp_compress() until all pages in the batch
> have been compressed. When this finishes, the compressed buffers and
> compressed lengths for the batch are available in the
> acomp_ctx->comp.sg_table->sgl scatterlists.
>
> A new zswap_update_entries() then iterates through the (up to)
> ZSWAP_MAX_BATCH_SIZE SG lists in acomp_ctx->comp.sg_table->sgl,
> allocates zsmalloc memory and copies all compressed buffers to
> the compressed pool. All existing functionality on zswap
> writeback_enabled (or not) fallbacks under various error conditions
> denoted by the crypto_acomp_compress() return value [and with this
> commit, also stored in the individual acomp_ctx->comp.sg_table->sgl
> sg->lengths in the crypto_acomp_compress() fast path], is essentially
> preserved.
>
> A significant zswap architectural change is that the batched
> zswap_compress() incorporates Herbert's suggestion for SG lists to
> represent the batch's inputs/outputs to interface with the crypto API
> [1].
>
> The introduction of zswap_update_entries() presents an opportunity to
> optimize the task of storing a batch of compressed buffers. Subsequent
> patch series will be submitted to optimize this phase by replacing
> zsmalloc with the SLUB allocator to improve zswap_store()/zswap_load()
> latencies.
>
> The batching framework introduced in this series improves zswap_store()
> latency by a lot.
>
> Performance testing: usemem30 with 100% over-commit:
> =====================================================
> Validation was done with mm-unstable as of 11-17-2026 (Baseline: commit
> 866f761fd414) and with this patch series.
>
> v4 was validated on an Intel Xeon Sapphire Rapids bare-metal instance with
> 48 cores.
>
>  Bare-metal instance specs:
>  --------------------------
>  - 48 physical cores
>  - 190G RAM
>  - 200G SWAP disk partition
>
>  Workload: 30 concurrent usemem processes, each allocates 10G:
>  -------------------------------------------------------------
>  The vm-scalability "usemem" test (unmodified version) was run in a
>  cgroup with memory.high fixed at 150G. There is no swap limit set for the
>  cgroup.
>
>  To quantify the performance benefit of zswap compress batching at scale,
>  30 usemem processes were run ("usemem30"), each allocating and writing 10G of
>  memory, and sleeping for 10 sec before exiting.
>
>   usemem -w -O -b 5 -s 10 -n 30 10g
>   echo 0 > /sys/module/zswap/parameters/shrinker_enabled
>
>  Performance validation methodology:
>  -----------------------------------
>  These settings were used to minimize variation:
>
>  - All cores were fixed to run at 2.4GHz with "performance" scaling governor.
>  - Turbo boost was disabled.
>  - CPU power-saving C-states restricted with:
>
>      GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1 processor.max_cstate=1"
>
>  The usemem30 experiments were run with the baseline mm-unstable as of
>  11-17-2026 (commit 866f761fd414) and with this patch-series.
>
>  After booting into each kernel:
>
>   1) Constrain the cgroup's memory.high to 150G.
>
>   2) Run 2 warm-up iterations of usemem30. Don't use this data for benchmarking
>      since sys time typically skews higher for initial runs.
>
>   3) Next, run 10 benchmarking iterations of usemem30:
>      - Log these high-level metrics: Sys time and individual usemem throughput.
>
>   4) Sys time and Throughput metrics were gathered for three mTHP settings:
>
>      - 2048kB
>      - 1024kB
>      - 64kB
>
>   5) Benchmark summary data includes Sys time and Throughput components from the
>      10 iterations of usemem30:
>
>      - Total Samples
>      - Mean Value
>      - Std Deviation
>      - CoV (Variance): Coefficient of variation (relative standard
>        deviation) measures spread relative to the mean rather than
>        statistical variance.
>
>  An LLM was used to help generate the benchmarking and data processing scripts.
>
> usemem30 Benchmarking Summary:
> ==============================
>
>  ---------------------------------------------------
>     Improvement with v4 zswap compress batching
>  ---------------------------------------------------
>   mTHP enabled:       2048kB     1024kB       64kB
>  ---------------------------------------------------
>       Sys time:      -13.87%     -2.64%     -1.72%
>     Throughput:       +3.85%     +0.42%     +1.31%
>  ---------------------------------------------------
>
> Detailed usemem30 benchmarking data:
> ====================================
>
>  1) 2048kB mTHP:
>  ===============
>
>  ------------------------------------------------------------------
>  2048kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
>  ------------------------------------------------------------------
>                Baseline mm-unstable   v4 zswap compress batching
>                               (sec)        (sec)
>  ------------------------------------------------------------------
>  [Run 1/10]  Sys Time:    1581.414s    1356.252s
>  [Run 2/10]  Sys Time:    1642.874s    1478.929s
>  [Run 3/10]  Sys Time:    1594.458s    1411.252s
>  [Run 4/10]  Sys Time:    1580.988s    1380.959s
>  [Run 5/10]  Sys Time:    1579.654s    1380.609s
>  [Run 6/10]  Sys Time:    1627.765s    1345.582s
>  [Run 7/10]  Sys Time:    1593.872s    1334.485s
>  [Run 8/10]  Sys Time:    1623.189s    1405.471s
>  [Run 9/10]  Sys Time:    1661.954s    1433.365s
>  [Run 10/10] Sys Time:    1602.098s    1329.813s
>  ------------------------------------------------------------------
>
>  ------------------------------------------------------------------
>  METRIC: System CPU Time (Sys Time) : 2048kB mTHP
>  ------------------------------------------------------------------
>                Baseline mm-unstable   v4 zswap compress batching
>                               (sec)            (sec)
>  ------------------------------------------------------------------
>  Total Samples    :              10               10
>  Mean Value       :        1,608.83         1,385.67
>  Std Deviation    :           28.67            47.47
>  CoV (Variance)   :           1.78%            3.43%
>  ------------------------------------------------------------------
>  METRIC: Per-Process Throughput : 2048kB mTHP
>  ------------------------------------------------------------------
>                Baseline mm-unstable   v4 zswap compress batching
>                              (KB/s)           (KB/s)
>  ------------------------------------------------------------------
>  Total Samples    :             300              300
>  Mean Value       :         292,261          303,508
>  Std Deviation    :          12,946           20,806
>  CoV (Variance)   :           4.42%            6.86%
>  ------------------------------------------------------------------
>
>  2) 1024kB mTHP:
>  ===============
>
>  ------------------------------------------------------------------
>  1024kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
>  ------------------------------------------------------------------
>                Baseline mm-unstable   v4 zswap compress batching
>                               (sec)        (sec)
>  ------------------------------------------------------------------
>  [Run 1/10]  Sys Time:    1567.639s    1594.696s
>  [Run 2/10]  Sys Time:    1537.577s    1456.113s
>  [Run 3/10]  Sys Time:    1538.562s    1521.887s
>  [Run 4/10]  Sys Time:    1626.176s    1531.962s
>  [Run 5/10]  Sys Time:    1593.955s    1528.733s
>  [Run 6/10]  Sys Time:    1643.519s    1558.482s
>  [Run 7/10]  Sys Time:    1510.613s    1480.177s
>  [Run 8/10]  Sys Time:    1632.018s    1584.093s
>  [Run 9/10]  Sys Time:    1569.424s    1528.915s
>  [Run 10/10] Sys Time:    1552.747s    1570.451s
>  ------------------------------------------------------------------
>
>  ------------------------------------------------------------------
>  METRIC: System CPU Time (Sys Time) : 1024kB mTHP
>  ------------------------------------------------------------------
>                Baseline mm-unstable   v4 zswap compress batching
>                               (sec)            (sec)
>  ------------------------------------------------------------------
>  Total Samples    :             10                10
>  Mean Value       :       1,577.22          1,535.55
>  Std Deviation    :          45.13             43.79
>  CoV (Variance)   :          2.86%             2.85%
>  ------------------------------------------------------------------
>  METRIC: Per-Process Throughput : 1024kB mTHP
>  ------------------------------------------------------------------
>                Baseline mm-unstable   v4 zswap compress batching
>                              (KB/s)           (KB/s)
>  ------------------------------------------------------------------
>  Total Samples    :            300               300
>  Mean Value       :        289,904           291,114
>  Std Deviation    :         15,385            14,553
>  CoV (Variance)   :          5.31%             5.00%
>  ------------------------------------------------------------------
>
>  3) 64kB mTHP:
>  =============
>
>  ------------------------------------------------------------------
>  64kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
>  ------------------------------------------------------------------
>                Baseline mm-unstable   v4 zswap compress batching
>                               (sec)        (sec)
>  ------------------------------------------------------------------
>  [Run 1/10]  Sys Time:    1914.525s    1878.483s
>  [Run 2/10]  Sys Time:    1869.412s    1863.177s
>  [Run 3/10]  Sys Time:    1944.049s    1863.502s
>  [Run 4/10]  Sys Time:    1942.757s    1900.144s
>  [Run 5/10]  Sys Time:    1866.505s    1887.285s
>  [Run 6/10]  Sys Time:    1877.716s    1861.668s
>  [Run 7/10]  Sys Time:    1861.741s    1868.788s
>  [Run 8/10]  Sys Time:    1904.735s    1816.372s
>  [Run 9/10]  Sys Time:    1891.074s    1850.435s
>  [Run 10/10] Sys Time:    1893.366s    1849.747s
>  ------------------------------------------------------------------
>
>  ------------------------------------------------------------------
>  METRIC: System CPU Time (Sys Time) : 64kB mTHP
>  ------------------------------------------------------------------
>                Baseline mm-unstable   v4 zswap compress batching
>                               (sec)            (sec)
>  ------------------------------------------------------------------
>  Total Samples    :              10               10
>  Mean Value       :        1,896.59         1,863.96
>  Std Deviation    :           29.86            22.96
>  CoV (Variance)   :           1.57%            1.23%
>  ------------------------------------------------------------------
>  METRIC: Per-Process Throughput : 64kB mTHP
>  ------------------------------------------------------------------
>                Baseline mm-unstable   v4 zswap compress batching
>                              (KB/s)           (KB/s)
>  ------------------------------------------------------------------
>  Total Samples    :            300               300
>  Mean Value       :        239,114           242,254
>  Std Deviation    :          9,090             6,419
>  CoV (Variance)   :          3.80%             2.65%
>  ------------------------------------------------------------------

The numbers look nice, but what's the zswap setup here? What's the
compressor? And how much of the win comes from this patch v.s the last
patch?

This patch series, and this patch in particular, seem like a lot of
complexity. I'd like to understand better the win before adding all
this complexity to the codebase.

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 0/8] Batched zswap_store() framework with compression batching
  2026-11-20 23:16 ` [PATCH v4 0/8] Batched zswap_store() framework with compression batching Nhat Pham
@ 2026-11-21  1:04   ` Kanchana P. Sridhar
  0 siblings, 0 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-21  1:04 UTC (permalink / raw)
  To: Nhat Pham
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky,
	Kanchana P. Sridhar

On Fri, Nov 20, 2026 at 3:16 PM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
> <kanchanapsridhar2026@gmail.com> wrote:
> >
> >
> > v4: Batched zswap_store() framework with compression batching
> > =============================================================
> >
> > This series is a spin-off of "v14: zswap compression batching with optimized
> > iaa_crypto driver" [1], and focuses on the generic crypto_acomp and zswap
> > compression batching enablers. Resuming work on this series following a gap
> > since v14 (January 2026). Please note that my email address and affiliation have
> > changed from kanchana.p.sridhar@intel.com to kanchanapsridhar2026@gmail.com. I
> > have retained the original Intel sign-offs on prior patches for historical
> > attribution while signing off v4 of this series with my current address.
> >
> > The main focus of v4 was to establish a performance measurement methodology to
> > quantify the benefits to high level metrics by using zswap compression
> > batching. This was done using an Intel Sapphire Rapids bare-metal instance with
> > 48 physical cores and by running "usemem30", i.e., 30 concurrent usemem
> > processes, each of which allocates 10G, in a 150G memory.high contrained cgroup,
> > for 10 iterations. The environment was set up to minimize performance jitter so
> > we can understand the system-level/workload-level advantages of zswap
> > compression batching when running workloads at scale.
> >
> > To summarize the main contribution of this patch series: Batching is introduced
> > in zswap_store() of large folios: we carve out a large folio into batches of
> > ZSWAP_MAX_BATCH_SIZE (4 pages) for software compressors, and store each
> > batch. Moving the fundamental operations that need to be happen to store a page
> > to now work at the granularity of a batch offers opportunities to optimize them
> > from these perspectives:
> >
> > 1) Utilize existing kernel API that minimize obtaining locks for bulk
> >    operations, e.g. kmem_cache_alloc_bulk()/kmem_cache_free_bulk().
> >
> > 2) We keep the working set for each of the core store functionalities
> >    (alloc/initialize zswap entries, compress pages, store pages in zsmalloc,
> >    pool/objcg reference increments, objcg charging) to be focused on a
> >    batch, thereby minimizing cache evictions and benefiting from cache
> >    locality.
> >
> > 3) Along the lines of (1), the main zswap_compress() batching routine obtains
> >    the comp_ctx mutex lock once per batch, and releases it after it has
> >    processed all pages in that batch.
> >
> > 4) Any zswap store functionality that works on a batch can potentially be
> >    optimized via bulk-processing/parallelized as a next step, to further improve
> >    performance.
> >
> > Performance testing: usemem30 with 100% over-commit:
> > =====================================================
> > Validation was done with mm-unstable as of 11-17-2026 (Baseline: commit
> > 866f761fd414) and with this patch series.
> >
> > v4 was validated on an Intel Xeon Sapphire Rapids bare-metal instance with
> > 48 cores.
> >
> >  Bare-metal instance specs:
> >  --------------------------
> >  - 48 physical cores
> >  - 190G RAM
> >  - 200G SWAP disk partition
> >
> >  Workload: 30 concurrent usemem processes, each allocates 10G:
> >  -------------------------------------------------------------
> >  The vm-scalability "usemem" test (unmodified version) was run in a
> >  cgroup with memory.high fixed at 150G. There is no swap limit set for the
> >  cgroup.
> >
> >  To quantify the performance benefit of zswap compress batching at scale,
> >  30 usemem processes were run ("usemem30"), each allocating and writing 10G of
> >  memory, and sleeping for 10 sec before exiting.
> >
> >   usemem -w -O -b 5 -s 10 -n 30 10g
> >   echo 0 > /sys/module/zswap/parameters/shrinker_enabled
> >
> >  Performance validation methodology:
> >  -----------------------------------
> >  These settings were used to minimize variation:
> >
> >  - All cores were fixed to run at 2.4GHz with "performance" scaling governor.
> >  - Turbo boost was disabled.
> >  - CPU power-saving C-states restricted with:
> >
> >      GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1 processor.max_cstate=1"
> >
> >  The usemem30 experiments were run with the baseline mm-unstable as of
> >  11-17-2026 (commit 866f761fd414) and with this patch-series.
> >
> >  After booting into each kernel:
> >
> >   1) Constrain the cgroup's memory.high to 150G.
> >
> >   2) Run 2 warm-up iterations of usemem30. Don't use this data for benchmarking
> >      since sys time typically skews higher for initial runs.
> >
> >   3) Next, run 10 benchmarking iterations of usemem30:
> >      - Log these high-level metrics: Sys time and individual usemem throughput.
> >
> >   4) Sys time and Throughput metrics were gathered for three mTHP settings:
> >
> >      - 2048kB
> >      - 1024kB
> >      - 64kB
> >
> >   5) Benchmark summary data includes Sys time and Throughput components from the
> >      10 iterations of usemem30:
> >
> >      - Total Samples
> >      - Mean Value
> >      - Std Deviation
> >      - CoV (Variance): Coefficient of variation (relative standard deviation)
> >        measures spread relative to the mean rather than statistical variance.
> >
> >  An LLM was used to help generate the benchmarking and data processing scripts.
> >
> > usemem30 Benchmarking Summary:
> > ==============================
> >
> >  ---------------------------------------------------
> >     Improvement with v4 zswap compress batching
> >  ---------------------------------------------------
> >   mTHP enabled:       2048kB     1024kB       64kB
> >  ---------------------------------------------------
> >       Sys time:      -13.87%     -2.64%     -1.72%
> >     Throughput:       +3.85%     +0.42%     +1.31%
> >  ---------------------------------------------------
>
> Have you tried perf-ing the store path to see what was the overhead
> source (before and after this optimization)? I'm curious.
>
> Mostly LGTM - still taking a look...

Thanks Nhat, for reviewing the code and providing valuable feedback!
Thanks also for your time.

Forgot to mention: ZSTD is the zswap compressor used. I will also
reply to your comment in the other email, along with more setup
details.
All zswap parameters (max pool percent, shrinker) are defaults.

I did analyze the performance of the store path using bpftrace to
understand what was the overhead source before and after this
optimization. My experiments used PMD folios to maximize the zswap
store/compress batching samples so the traces are helpful in
understanding how the optimizations are eliminating overhead.
Although, this was done with v3 and my earlier, memory constrained
QEMU setup running on Chromebook/Crostini.

These are the bpftrace p50/p95/p99 latencies. The labels mean:

 "zswap_store_higher_order_folio":
    zswap_store() of PMD folios.

 "zswap_store_order0_folio":
    zswap_store() of order-0 folios.

 "zswap_batch_compress_per_page":
    zswap_compress() per-page latency of store batches generated from PMD
    folios.

 "zswap_single_compress_per_page":
    zswap_compress() per-page latency of order-0 folios.

 =============================================================================
   usemem PMD THP BPFTRACE
 =============================================================================
           Baseline           zswap           zswap          Change    Change
        mm-unstable           store        compress            with      with
         10-25-2026        batching        batching        compress  compress
                        (patch 6-7)       (patch 8)        batching  batching
                                                               (ns)       %
 =============================================================================
  zswap_store_order0_folio (ns):
 =============================================================================
  p50        39,631          37,538          37,765          -1,866    -4.71%
  p95        53,088          49,690          49,349          -3,739    -7.04%
  p99        65,065          63,529          57,404          -7,661   -11.77%
 =============================================================================
  zswap_store_higher_order_folio (ns):
 =============================================================================
  p50    21,460,366      21,201,411      20,383,233      -1,077,133    -5.02%
  p95    25,975,341      27,558,253      25,045,350        -929,991    -3.58%
  p99    29,538,493      61,926,648      27,184,835      -2,353,658    -7.97%
 =============================================================================
  zswap_single_compress_per_page (ns):
 =============================================================================
  p50        37,545          39,320          37,218            -327    -0.87%
  p95        51,549          66,503          51,481             -68    -0.13%
  p99        89,866         134,854          87,091          -2,775    -3.09%
 =============================================================================
  zswap_batch_compress_per_page (ns):
 =============================================================================
  p50                                        40,493           2,948     7.85%
  p95                                        56,944           5,395    10.47%
  p99                                        76,242         -13,624   -15.16%
 =============================================================================

Analysis:
=========

 The batching of zswap_store() phases, and compress batching
optimizations benefit zswap_store() and zswap_compress_per_page()
median and tail latencies
 for PMD folios and base 4K folios. The benefits are attributable
firstly to the store batching framework added in patches 6-7, that is
fully utilized to improve
 zswap_compress() latency by giving it a batch of pages to be compressed.

Gains seen from store batching alone (patches 6-7) are due to:

1) Bulk allocating the zswap entries for the batch,
2) Minimizing computes by getting the wb_enabled once per PMD folio,
rather than per-page,
3) Fewer function calls: zswap_store_pages() called once every 4
pages, rather than per-page.

Furthermore, the batch compression added in patch 8 improves latency
significantly, even more than that obtained by store batching alone,
for these reasons:

The overall approach for making the zswap_store() code more efficient, has been:

a) Minimize computes, especially conditionals. For e.g. crypto API
that set things up for a batch, rather than for a page.
b) The working set is kept focused on batch-centric data pertaining to
a handful of operations on the batch, to minimize cache evictions and
benefit from cache locality.
c) Each block of compute in zswap_compress() is doing checks/state
changes for a batch, which I believe simplifies the architecture and
allows us to build further optimizations.

Specifically:

1) Per-CPU sg_table with ZSWAP_MAX_BATCH_SIZE scatterlists that are
statically mapped to compression dst buffers for that CPU, thereby
minimizing local SG lists initialization, and using compact new
crypto_acomp API introduced in patches 1-4 that set up an acomp_req's
src/dst to be SG lists.

2) Patch 6 allocates ZSWAP_MAX_BATCH_SIZE destination buffers in the
per-CPU acomp_ctx, for an added memory overhead of ~12 KB per-CPU.
However, it allows us to not have to immediately call zs_store() with
the compressed buffer, but instead continue to compress the next page.
I haven't quantified this with perf event counters, but my guess is
this minimizes cache evictions because we are focused on running a
focused set of computes on a specific data set.

3) A one-shot zswap_update_entries() does all the zsmalloc handle
allocation and copying for the batch.

4) Bulk accounting for the batch instead of per-page.

5) Single point error-handling in zswap_store_pages() to avoid
conditionals, so as to not wash away batching gains.

6) By making zswap_compress() more efficient, we are able to lock the
comp mutex per batch of ZSWAP_MAX_BATCH_SIZE (4) pages instead of
per-page.

7) Optimize most-likely-to-succeed fast paths, move conditionals and
error processing to less-likely code (which is also consolidated to a
single error handling label).

8) Don't proceed to next stage of batch processing until this stage is
successful for its specific functionality scope. This minimizes
computes by avoiding if-then-else checks per-page.

The bpftraces highlight exceptions in p50/p95 batch compress latencies
(normalized per-page), even though the p99 batch compress latency
improves by 15% over mm-unstable. Batch compress latency can be
optimized further in subsequent patches. The batching framework
established in this series allows us to make locking and
bulk-processing improvements in each stage of zswap_store().

Please feel free to share any additional insights you may have as you
review the code!

Thanks,
Kanchana

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches.
  2026-11-20 23:55   ` Nhat Pham
@ 2026-11-21  1:17     ` Kanchana P. Sridhar
  2026-11-21  3:24       ` Kanchana P. Sridhar
  0 siblings, 1 reply; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-21  1:17 UTC (permalink / raw)
  To: Nhat Pham
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky,
	Kanchana P. Sridhar

On Fri, Nov 20, 2026 at 3:55 PM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
> <kanchanapsridhar2026@gmail.com> wrote:
> >
> > A new batching implementation of zswap_compress() is provided for
> > compressors that do and do not support batching. This eliminates code
> > duplication and facilitates code maintainability with the introduction
> > of compress batching.
> >
> > The vectorized implementation of calling the earlier zswap_compress()
> > sequentially, one page at a time in zswap_store_pages(), is replaced
> > with this new version of zswap_compress() that accepts multiple pages to
> > compress as a batch.
> >
> > When the crypto_acomp is used to set up the per-CPU acomp_ctx
> > during CPU onlining, the compressor's batch_size is obtained by calling:
> >
> >   crypto_acomp_batch_size(acomp_ctx->acomp)
> >
> > This is bounded by ZSWAP_MAX_BATCH_SIZE to limit the memory
> > overhead and to reflect the store batch constraint of max # of pages in
> > a batch. The clamped batch_size is stored in the per-CPU
> > acomp_ctx->comp.compr_batch_size.
> >
> > As noted in previous commits in this series, ZSWAP_MAX_BATCH_SIZE
> > per-CPU compression buffers are available for sequential and batching
> > compression algorithms.
> >
> > Regardless of compression batching, we always process large folios in
> > batches of ZSWAP_MAX_BATCH_SIZE.
> >
> > If the compressor does not support batching, each page in the batch is
> > compressed sequentially. For batching compressors, up to
> > acomp_ctx->comp.compr_batch_size pages in the batch are compressed per
> > call to crypto_acomp_compress(). The first phase of zswap_compress()
> > thus only calls crypto_acomp_compress() until all pages in the batch
> > have been compressed. When this finishes, the compressed buffers and
> > compressed lengths for the batch are available in the
> > acomp_ctx->comp.sg_table->sgl scatterlists.
> >
> > A new zswap_update_entries() then iterates through the (up to)
> > ZSWAP_MAX_BATCH_SIZE SG lists in acomp_ctx->comp.sg_table->sgl,
> > allocates zsmalloc memory and copies all compressed buffers to
> > the compressed pool. All existing functionality on zswap
> > writeback_enabled (or not) fallbacks under various error conditions
> > denoted by the crypto_acomp_compress() return value [and with this
> > commit, also stored in the individual acomp_ctx->comp.sg_table->sgl
> > sg->lengths in the crypto_acomp_compress() fast path], is essentially
> > preserved.
> >
> > A significant zswap architectural change is that the batched
> > zswap_compress() incorporates Herbert's suggestion for SG lists to
> > represent the batch's inputs/outputs to interface with the crypto API
> > [1].
> >
> > The introduction of zswap_update_entries() presents an opportunity to
> > optimize the task of storing a batch of compressed buffers. Subsequent
> > patch series will be submitted to optimize this phase by replacing
> > zsmalloc with the SLUB allocator to improve zswap_store()/zswap_load()
> > latencies.
> >
> > The batching framework introduced in this series improves zswap_store()
> > latency by a lot.
> >
> > Performance testing: usemem30 with 100% over-commit:
> > =====================================================
> > Validation was done with mm-unstable as of 11-17-2026 (Baseline: commit
> > 866f761fd414) and with this patch series.
> >
> > v4 was validated on an Intel Xeon Sapphire Rapids bare-metal instance with
> > 48 cores.
> >
> >  Bare-metal instance specs:
> >  --------------------------
> >  - 48 physical cores
> >  - 190G RAM
> >  - 200G SWAP disk partition
> >
> >  Workload: 30 concurrent usemem processes, each allocates 10G:
> >  -------------------------------------------------------------
> >  The vm-scalability "usemem" test (unmodified version) was run in a
> >  cgroup with memory.high fixed at 150G. There is no swap limit set for the
> >  cgroup.
> >
> >  To quantify the performance benefit of zswap compress batching at scale,
> >  30 usemem processes were run ("usemem30"), each allocating and writing 10G of
> >  memory, and sleeping for 10 sec before exiting.
> >
> >   usemem -w -O -b 5 -s 10 -n 30 10g
> >   echo 0 > /sys/module/zswap/parameters/shrinker_enabled
> >
> >  Performance validation methodology:
> >  -----------------------------------
> >  These settings were used to minimize variation:
> >
> >  - All cores were fixed to run at 2.4GHz with "performance" scaling governor.
> >  - Turbo boost was disabled.
> >  - CPU power-saving C-states restricted with:
> >
> >      GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1 processor.max_cstate=1"
> >
> >  The usemem30 experiments were run with the baseline mm-unstable as of
> >  11-17-2026 (commit 866f761fd414) and with this patch-series.
> >
> >  After booting into each kernel:
> >
> >   1) Constrain the cgroup's memory.high to 150G.
> >
> >   2) Run 2 warm-up iterations of usemem30. Don't use this data for benchmarking
> >      since sys time typically skews higher for initial runs.
> >
> >   3) Next, run 10 benchmarking iterations of usemem30:
> >      - Log these high-level metrics: Sys time and individual usemem throughput.
> >
> >   4) Sys time and Throughput metrics were gathered for three mTHP settings:
> >
> >      - 2048kB
> >      - 1024kB
> >      - 64kB
> >
> >   5) Benchmark summary data includes Sys time and Throughput components from the
> >      10 iterations of usemem30:
> >
> >      - Total Samples
> >      - Mean Value
> >      - Std Deviation
> >      - CoV (Variance): Coefficient of variation (relative standard
> >        deviation) measures spread relative to the mean rather than
> >        statistical variance.
> >
> >  An LLM was used to help generate the benchmarking and data processing scripts.
> >
> > usemem30 Benchmarking Summary:
> > ==============================
> >
> >  ---------------------------------------------------
> >     Improvement with v4 zswap compress batching
> >  ---------------------------------------------------
> >   mTHP enabled:       2048kB     1024kB       64kB
> >  ---------------------------------------------------
> >       Sys time:      -13.87%     -2.64%     -1.72%
> >     Throughput:       +3.85%     +0.42%     +1.31%
> >  ---------------------------------------------------
> >
> > Detailed usemem30 benchmarking data:
> > ====================================
> >
> >  1) 2048kB mTHP:
> >  ===============
> >
> >  ------------------------------------------------------------------
> >  2048kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
> >  ------------------------------------------------------------------
> >                Baseline mm-unstable   v4 zswap compress batching
> >                               (sec)        (sec)
> >  ------------------------------------------------------------------
> >  [Run 1/10]  Sys Time:    1581.414s    1356.252s
> >  [Run 2/10]  Sys Time:    1642.874s    1478.929s
> >  [Run 3/10]  Sys Time:    1594.458s    1411.252s
> >  [Run 4/10]  Sys Time:    1580.988s    1380.959s
> >  [Run 5/10]  Sys Time:    1579.654s    1380.609s
> >  [Run 6/10]  Sys Time:    1627.765s    1345.582s
> >  [Run 7/10]  Sys Time:    1593.872s    1334.485s
> >  [Run 8/10]  Sys Time:    1623.189s    1405.471s
> >  [Run 9/10]  Sys Time:    1661.954s    1433.365s
> >  [Run 10/10] Sys Time:    1602.098s    1329.813s
> >  ------------------------------------------------------------------
> >
> >  ------------------------------------------------------------------
> >  METRIC: System CPU Time (Sys Time) : 2048kB mTHP
> >  ------------------------------------------------------------------
> >                Baseline mm-unstable   v4 zswap compress batching
> >                               (sec)            (sec)
> >  ------------------------------------------------------------------
> >  Total Samples    :              10               10
> >  Mean Value       :        1,608.83         1,385.67
> >  Std Deviation    :           28.67            47.47
> >  CoV (Variance)   :           1.78%            3.43%
> >  ------------------------------------------------------------------
> >  METRIC: Per-Process Throughput : 2048kB mTHP
> >  ------------------------------------------------------------------
> >                Baseline mm-unstable   v4 zswap compress batching
> >                              (KB/s)           (KB/s)
> >  ------------------------------------------------------------------
> >  Total Samples    :             300              300
> >  Mean Value       :         292,261          303,508
> >  Std Deviation    :          12,946           20,806
> >  CoV (Variance)   :           4.42%            6.86%
> >  ------------------------------------------------------------------
> >
> >  2) 1024kB mTHP:
> >  ===============
> >
> >  ------------------------------------------------------------------
> >  1024kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
> >  ------------------------------------------------------------------
> >                Baseline mm-unstable   v4 zswap compress batching
> >                               (sec)        (sec)
> >  ------------------------------------------------------------------
> >  [Run 1/10]  Sys Time:    1567.639s    1594.696s
> >  [Run 2/10]  Sys Time:    1537.577s    1456.113s
> >  [Run 3/10]  Sys Time:    1538.562s    1521.887s
> >  [Run 4/10]  Sys Time:    1626.176s    1531.962s
> >  [Run 5/10]  Sys Time:    1593.955s    1528.733s
> >  [Run 6/10]  Sys Time:    1643.519s    1558.482s
> >  [Run 7/10]  Sys Time:    1510.613s    1480.177s
> >  [Run 8/10]  Sys Time:    1632.018s    1584.093s
> >  [Run 9/10]  Sys Time:    1569.424s    1528.915s
> >  [Run 10/10] Sys Time:    1552.747s    1570.451s
> >  ------------------------------------------------------------------
> >
> >  ------------------------------------------------------------------
> >  METRIC: System CPU Time (Sys Time) : 1024kB mTHP
> >  ------------------------------------------------------------------
> >                Baseline mm-unstable   v4 zswap compress batching
> >                               (sec)            (sec)
> >  ------------------------------------------------------------------
> >  Total Samples    :             10                10
> >  Mean Value       :       1,577.22          1,535.55
> >  Std Deviation    :          45.13             43.79
> >  CoV (Variance)   :          2.86%             2.85%
> >  ------------------------------------------------------------------
> >  METRIC: Per-Process Throughput : 1024kB mTHP
> >  ------------------------------------------------------------------
> >                Baseline mm-unstable   v4 zswap compress batching
> >                              (KB/s)           (KB/s)
> >  ------------------------------------------------------------------
> >  Total Samples    :            300               300
> >  Mean Value       :        289,904           291,114
> >  Std Deviation    :         15,385            14,553
> >  CoV (Variance)   :          5.31%             5.00%
> >  ------------------------------------------------------------------
> >
> >  3) 64kB mTHP:
> >  =============
> >
> >  ------------------------------------------------------------------
> >  64kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
> >  ------------------------------------------------------------------
> >                Baseline mm-unstable   v4 zswap compress batching
> >                               (sec)        (sec)
> >  ------------------------------------------------------------------
> >  [Run 1/10]  Sys Time:    1914.525s    1878.483s
> >  [Run 2/10]  Sys Time:    1869.412s    1863.177s
> >  [Run 3/10]  Sys Time:    1944.049s    1863.502s
> >  [Run 4/10]  Sys Time:    1942.757s    1900.144s
> >  [Run 5/10]  Sys Time:    1866.505s    1887.285s
> >  [Run 6/10]  Sys Time:    1877.716s    1861.668s
> >  [Run 7/10]  Sys Time:    1861.741s    1868.788s
> >  [Run 8/10]  Sys Time:    1904.735s    1816.372s
> >  [Run 9/10]  Sys Time:    1891.074s    1850.435s
> >  [Run 10/10] Sys Time:    1893.366s    1849.747s
> >  ------------------------------------------------------------------
> >
> >  ------------------------------------------------------------------
> >  METRIC: System CPU Time (Sys Time) : 64kB mTHP
> >  ------------------------------------------------------------------
> >                Baseline mm-unstable   v4 zswap compress batching
> >                               (sec)            (sec)
> >  ------------------------------------------------------------------
> >  Total Samples    :              10               10
> >  Mean Value       :        1,896.59         1,863.96
> >  Std Deviation    :           29.86            22.96
> >  CoV (Variance)   :           1.57%            1.23%
> >  ------------------------------------------------------------------
> >  METRIC: Per-Process Throughput : 64kB mTHP
> >  ------------------------------------------------------------------
> >                Baseline mm-unstable   v4 zswap compress batching
> >                              (KB/s)           (KB/s)
> >  ------------------------------------------------------------------
> >  Total Samples    :            300               300
> >  Mean Value       :        239,114           242,254
> >  Std Deviation    :          9,090             6,419
> >  CoV (Variance)   :          3.80%             2.65%
> >  ------------------------------------------------------------------
>
> The numbers look nice, but what's the zswap setup here? What's the
> compressor? And how much of the win comes from this patch v.s the last
> patch?
>
> This patch series, and this patch in particular, seem like a lot of
> complexity. I'd like to understand better the win before adding all
> this complexity to the codebase.

Hi Nhat,

ZSTD is the zswap compressor. My apologies for forgetting to mention
this in the performance testing section.

All reclaim and zswap parameters are the defaults in the kernel build
(I also configure ZSTD as my default zswap compressor when I build the
kernel):

vm.swappiness = 60
/sys/module/zswap/parameters/max_pool_percent: 20
/sys/module/zswap/parameters/shrinker_enabled: N

/sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled: [always]
inherit madvise never
/sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never
/sys/kernel/mm/transparent_hugepage/defrag: always defer defer+madvise
[madvise] never

I just shared my analysis based on bpftrace data in response to your
cover letter comments, to try and explain the wins with zswap store
batching alone (patches 6-7) and then with zswap compress batching
(patch 8). Please do let me know if you have follow-up
questions/comments.

I will also gather and post usemem30 numbers before/after patch 8.

Thanks,
Kanchana


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 7/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations.
  2026-11-20 23:45   ` Nhat Pham
@ 2026-11-21  1:36     ` Kanchana P. Sridhar
  0 siblings, 0 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-21  1:36 UTC (permalink / raw)
  To: Nhat Pham
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky,
	Kanchana P. Sridhar

On Fri, Nov 20, 2026 at 3:46 PM Nhat Pham <nphamcs@gmail.com> wrote:
>
> On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
> <kanchanapsridhar2026@gmail.com> wrote:
> >
> > Subsequent patch series will be submitted, that implement ideas building
> > upon this batching framework:
> >
> >  1) Replacing zsmalloc with SLUB allocator, comprising of kmem_caches
> >     for a finite # of compressed size bins. This enables bulk allocation
> >     of compressed buffers in a batch that belong to the same size
> >     bin. It gets rid of significant zsmalloc latency overhead (locks,
> >     compressed data spanning zspages, etc.), without compromising memory
> >     efficiency.
>
> I don't think this will be a good idea, because it will make
> compressed memory unmovable.

Yes, SLUB comes with the constraint of the compressed memory being
unmovable. However, I want to better understand the trade off between
compressed memory migration vs. potentially representing it as SLUB
unmovable memory that needs to be written back. I was hoping to
include a discussion of my understanding of the pros-and-cons when I
submit this RFC, to get your thoughts.

>
> >
> >  2) vmscan reclaim batching of hybrid any-order folios, that will submit
> >     a batch of folios to zswap_store()'s batching framework to optimize
> >     storing of batches of pages from different folios, in addition to
> >     within large folios.
>
> This one might be more complicated than its worth, but I'm not sure.

This does benefit latency a lot! But, one step at a time :)

>
> >
> > Signed-off-by: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>
>
> Code mostly LGTM.

Thanks for reviewing and providing valuable comments.

>
> > ---
> >  include/linux/memcontrol.h |  12 +++
> >  mm/memcontrol.c            |  23 ++++-
> >  mm/zswap.c                 | 181 +++++++++++++++++++++++++------------
> >  3 files changed, 153 insertions(+), 63 deletions(-)
> >
> > diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
> > index 980358bc6f49..93f060818c14 100644
> > --- a/include/linux/memcontrol.h
> > +++ b/include/linux/memcontrol.h
> > @@ -1244,6 +1244,11 @@ struct mem_cgroup *mem_cgroup_from_css(struct cgroup_subsys_state *css)
> >         return NULL;
> >  }
> >
> > +static inline void obj_cgroup_get_many(struct obj_cgroup *objcg,
> > +                                      unsigned long nr)
> > +{
> > +}
> > +
> >  static inline void obj_cgroup_get(struct obj_cgroup *objcg)
> >  {
> >  }
> > @@ -1921,6 +1926,8 @@ static inline void mem_cgroup_calculate_protection_path(struct mem_cgroup *root,
> >  #if defined(CONFIG_MEMCG) && defined(CONFIG_ZSWAP)
> >  bool obj_cgroup_may_zswap(struct obj_cgroup *objcg);
> >  void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size);
> > +void obj_cgroup_charge_zswap_many(struct obj_cgroup *objcg, size_t size,
> > +                                 u8 nr_pages, u8 nr_incomp_pages);
> >  void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg, size_t size);
> >  bool mem_cgroup_zswap_writeback_enabled(const struct mem_cgroup *memcg);
> >  #else
> > @@ -1932,6 +1939,11 @@ static inline void obj_cgroup_charge_zswap(struct obj_cgroup *objcg,
> >                                            size_t size)
> >  {
> >  }
>
> Why do we still need obj_cgroup_charge_zswap() at all?

Ok, I will delete it in next rev.

>
> > +static inline void obj_cgroup_charge_zswap_many(struct obj_cgroup *objcg,
> > +                                               size_t size, u8 nr_pages,
> > +                                               u8 nr_incomp_pages)
> > +{
> > +}
> >  static inline void obj_cgroup_uncharge_zswap(struct obj_cgroup *objcg,
> >                                              size_t size)
> >  {
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index 091401cecf39..2236f5ba6304 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -6345,6 +6345,23 @@ bool obj_cgroup_may_zswap(struct obj_cgroup *objcg)
> >   * compression and storage in zswap for this cgroup to go ahead.
> >   */
> >  void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size)
> > +{
> > +       obj_cgroup_charge_zswap_many(objcg, size, 1, (u8)(size == PAGE_SIZE));
> > +}
> > +
> > +/**
> > + * obj_cgroup_charge_zswap_many - bulk charge compression backend memory
> > + *                                from compressing a batch of pages.
> > + * @objcg: the object cgroup
> > + * @size: total compressed length of a batch of compressed objects
> > + * @nr_pages: # of pages that were compressed to store in zswap
> > + * @nr_incomp_pages: # of incompressible pages to store in zswap
> > + *
> > + * This forces the charge after obj_cgroup_may_zswap() allowed
> > + * compression and storage in zswap for this cgroup to go ahead.
> > + */
> > +void obj_cgroup_charge_zswap_many(struct obj_cgroup *objcg, size_t size,
> > +                                 u8 nr_pages, u8 nr_incomp_pages)
> >  {
> >         struct mem_cgroup *memcg;
> >
> > @@ -6363,9 +6380,9 @@ void obj_cgroup_charge_zswap(struct obj_cgroup *objcg, size_t size)
> >         rcu_read_lock();
> >         memcg = obj_cgroup_memcg(objcg);
> >         mod_memcg_state(memcg, MEMCG_ZSWAP_B, size);
> > -       mod_memcg_state(memcg, MEMCG_ZSWAPPED, 1);
> > -       if (size == PAGE_SIZE)
> > -               mod_memcg_state(memcg, MEMCG_ZSWAP_INCOMP, 1);
> > +       mod_memcg_state(memcg, MEMCG_ZSWAPPED, nr_pages);
> > +       if (nr_incomp_pages)
> > +               mod_memcg_state(memcg, MEMCG_ZSWAP_INCOMP, nr_incomp_pages);
> >         rcu_read_unlock();
> >  }
> >
> > diff --git a/mm/zswap.c b/mm/zswap.c
> > index df59ab0c1029..5e1fb5624ffb 100644
> > --- a/mm/zswap.c
> > +++ b/mm/zswap.c
> > @@ -491,12 +491,12 @@ static int __must_check zswap_pool_tryget(struct zswap_pool *pool)
> >  }
> >
> >  /* The caller must already have a reference. */
> > -static void zswap_pool_get(struct zswap_pool *pool)
> > +static inline void zswap_pool_get_many(struct zswap_pool *pool, u8 nr)
> >  {
> > -       percpu_ref_get(&pool->ref);
> > +       percpu_ref_get_many(&pool->ref, nr);
> >  }
> >
> > -static void zswap_pool_put(struct zswap_pool *pool)
> > +static inline void zswap_pool_put(struct zswap_pool *pool)
> >  {
> >         percpu_ref_put(&pool->ref);
> >  }
> > @@ -1603,55 +1603,60 @@ static void shrink_worker(struct work_struct *w)
> >  * main API
> >  **********************************/
> >
> > +/* Functions called by zswap_store(), that operate on batches of pages. */
> > +
> >  /*
> > - * Store multiple pages in @zc->folio, starting from the page at index @start up
> > - * to the page at index @end-1.
> > + * zswap_init_entries() co-locates most of the zswap entry fields' setup under
> > + * the assumption that all phases of store processing will succeed. An important
> > + * marker to simplify catch-all cleanup in zswap_store_pages() is done by
> > + * initializing each entry's @handle to ERR_PTR(-EINVAL).
> >   */
> > -static bool zswap_store_pages(struct zswap_store_context *zc,
> > -                             long start,
> > -                             long end,
> > -                             struct zswap_pool *pool,
> > -                             struct obj_cgroup *objcg)
> > +static inline void zswap_init_entries(struct zswap_store_context *zc,
> > +                                     struct zswap_pool *pool,
> > +                                     struct obj_cgroup *objcg)
> >  {
> > -       u8 i, store_fail_idx = 0;
> > -       bool ret;
> > -
> > -       zc->nr_pages = end - start;
> > -       zc->folio_start = start;
> > -
> > -       VM_WARN_ON_ONCE(zc->nr_pages <= 0 || zc->nr_pages > ZSWAP_MAX_BATCH_SIZE);
> > -
> > -       ret = zswap_entries_cache_alloc_batch(zc, GFP_KERNEL);
> > -       if (unlikely(!ret))
> > -               return false;
> > +       struct zswap_entry *entry;
> > +       u8 i;
> >
> >         /*
> >          * We co-locate entry initialization as much as possible here to
> >          * minimize potential cache misses.
> >          */
> >         for (i = 0; i < zc->nr_pages; ++i) {
> > +               entry = zc->entries[i];
> >                 /*
> > -                * Set pool_idx before the xa_store() below publishes the entry, or a
> > -                * concurrent reader could resolve a stale pool_idx left by slab reuse
> > -                * to an unrelated live pool.
> > +                * Set pool_idx before the xa_store() in zswap_tree_stores()
> > +                * publishes the entry, or a concurrent reader could resolve a
> > +                * stale pool_idx left by slab reuse to an unrelated live pool.
> >                  */
> > -               zc->entries[i]->pool_idx = pool->idx;
> > -               zc->entries[i]->swpentry = folio_swap_entry(zc->folio,
> > +               entry->pool_idx = pool->idx;
> > +               entry->swpentry = folio_swap_entry(zc->folio,
> >                                                         zc->folio_start + i);
> > -               zc->entries[i]->objcg = objcg;
> > -               zc->entries[i]->referenced = true;
> > -               zc->entries[i]->nid = zc->nid;
> > -               INIT_LIST_HEAD(&zc->entries[i]->lru);
> > +               entry->objcg = objcg;
> > +               entry->referenced = true;
> > +               entry->nid = zc->nid;
> > +               INIT_LIST_HEAD(&entry->lru);
> >         }
> > +}
> >
> > -       for (i = 0; i < zc->nr_pages; ++i) {
> > -               if (!zswap_compress(zc->folio, zc->folio_start + i,
> > -                                   zc->entries[i], pool, zc->wb_enabled))
> > -                       goto store_pages_failed;
> > -       }
> > +/*
> > + * zswap_tree_stores() stores the batch entries in xarray and adds them to the
> > + * zswap LRU list. If an error occurs during xa_store(), localized state
> > + * unwinding is done to erase prior batch entries from the xarray and delete
> > + * them from the zswap LRU list. Remaining batch cleanup occurs in
> > + * zswap_store_pages(). As batch entries are processed, we maintain aggregate
> > + * statistics to do the accounting in one bulk operation.
> > + */
> > +static bool zswap_tree_stores(struct zswap_store_context *zc,
> > +                             struct zswap_pool *pool,
> > +                             struct obj_cgroup *objcg)
> > +{
> > +       u8 i, store_fail_idx = 0, nr_incomp = 0;
> > +       size_t total_batch_comp_length = 0;
> > +       struct zswap_entry *old, *entry;
> >
> >         for (i = 0; i < zc->nr_pages; ++i) {
> > -               struct zswap_entry *old, *entry = zc->entries[i];
> > +               entry = zc->entries[i];
> >
> >                 old = xa_store(swap_zswap_tree(entry->swpentry),
> >                                swp_offset(entry->swpentry),
> > @@ -1663,13 +1668,11 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
> >                         zswap_reject_alloc_fail++;
> >                         /*
> >                          * Entries up to this point have been stored in the
> > -                        * xarray. zswap_store() will erase them from the xarray
> > -                        * and call zswap_entry_free(). Local cleanup in
> > -                        * 'store_pages_failed' only needs to happen for
> > -                        * entries from [@i to @nr_pages).
> > +                        * xarray. Local cleanup in 'tree_store_failed' needs to
> > +                        * erase entries up until now.
> >                          */
> >                         store_fail_idx = i;
> > -                       goto store_pages_failed;
> > +                       goto tree_store_failed;
> >                 }
> >
> >                 /*
> > @@ -1680,22 +1683,10 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
> >                 if (unlikely(old))
> >                         zswap_entry_free(old);
> >
> > -               /*
> > -                * The entry is successfully compressed and stored in the tree,
> > -                * and further failures will be cleaned up in zswap_store().
> > -                * Grab refs to the pool and objcg, charge zswap memory, and
> > -                * increment zswap_stored_pages. The opposite actions will be
> > -                * performed by zswap_entry_free() when the entry is removed
> > -                * from the tree.
> > -                */
> > -               zswap_pool_get(pool);
> > -               if (objcg) {
> > -                       obj_cgroup_get(objcg);
> > -                       obj_cgroup_charge_zswap(objcg, entry->length);
> > -               }
> > -               atomic_long_inc(&zswap_stored_pages);
> > +               total_batch_comp_length += entry->length;
> > +
> >                 if (unlikely(entry->length == PAGE_SIZE))
> > -                       atomic_long_inc(&zswap_stored_incompressible_pages);
> > +                       ++nr_incomp;
> >
> >                 /*
> >                  * We finish by adding the entry to the LRU while it's already
> > @@ -1711,17 +1702,87 @@ static bool zswap_store_pages(struct zswap_store_context *zc,
> >                         zswap_lru_add(entry);
> >         }
> >
> > +       /*
> > +        * Each batch entry is successfully compressed and stored in the tree,
> > +        * and further failures will be cleaned up in zswap_store().
> > +        * Next, we do bulk accounting for the batch: grab refs to the pool and
> > +        * objcg, charge zswap memory, increment zswap_stored_pages and
> > +        * zswap_stored_incompressible_pages. The opposite actions will be
> > +        * performed by zswap_entry_free() when the entry is removed from the
> > +        * tree.
> > +        */
> > +       zswap_pool_get_many(pool, zc->nr_pages);
> > +       if (objcg) {
> > +               obj_cgroup_get_many(objcg, zc->nr_pages);
> > +               obj_cgroup_charge_zswap_many(objcg, total_batch_comp_length,
> > +                                            zc->nr_pages, nr_incomp);
> > +       }
> > +       atomic_long_add(zc->nr_pages, &zswap_stored_pages);
> > +       if (unlikely(nr_incomp))
> > +               atomic_long_add(nr_incomp, &zswap_stored_incompressible_pages);
> > +
> > +       return true;
> > +
>
> Can you add a comment that we only clean up the current batch here,
> and the clean up of other batches in the folio are deferred to
> zswap_store() (or whereever up the stack it is handled?).

Sure, will do.

>
> > +tree_store_failed:
> > +       for (i = 0; i < store_fail_idx; ++i) {
> > +               xa_erase(swap_zswap_tree(zc->entries[i]->swpentry),
> > +                        swp_offset(zc->entries[i]->swpentry));
>
> xas API?

Sure, I need to spend time on it, thinking best if it is a subsequent patch.

>
> > +               zswap_lru_del(zc->entries[i]);
> > +       }
> > +
> > +       return false;
> > +}
> > +
> > +/*


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches.
  2026-11-21  1:17     ` Kanchana P. Sridhar
@ 2026-11-21  3:24       ` Kanchana P. Sridhar
  0 siblings, 0 replies; 25+ messages in thread
From: Kanchana P. Sridhar @ 2026-11-21  3:24 UTC (permalink / raw)
  To: Nhat Pham
  Cc: hannes, yosry, chengming.zhou, akpm, linux-mm, herbert, davem,
	linux-crypto, linux-kernel, rientjes, willy, senozhatsky,
	Kanchana P. Sridhar

On Fri, Nov 20, 2026 at 5:17 PM Kanchana P. Sridhar
<kanchanapsridhar2026@gmail.com> wrote:
>
> On Fri, Nov 20, 2026 at 3:55 PM Nhat Pham <nphamcs@gmail.com> wrote:
> >
> > On Thu, Nov 19, 2026 at 8:19 PM Kanchana P. Sridhar
> > <kanchanapsridhar2026@gmail.com> wrote:
> > >
> > > A new batching implementation of zswap_compress() is provided for
> > > compressors that do and do not support batching. This eliminates code
> > > duplication and facilitates code maintainability with the introduction
> > > of compress batching.
> > >
> > > The vectorized implementation of calling the earlier zswap_compress()
> > > sequentially, one page at a time in zswap_store_pages(), is replaced
> > > with this new version of zswap_compress() that accepts multiple pages to
> > > compress as a batch.
> > >
> > > When the crypto_acomp is used to set up the per-CPU acomp_ctx
> > > during CPU onlining, the compressor's batch_size is obtained by calling:
> > >
> > >   crypto_acomp_batch_size(acomp_ctx->acomp)
> > >
> > > This is bounded by ZSWAP_MAX_BATCH_SIZE to limit the memory
> > > overhead and to reflect the store batch constraint of max # of pages in
> > > a batch. The clamped batch_size is stored in the per-CPU
> > > acomp_ctx->comp.compr_batch_size.
> > >
> > > As noted in previous commits in this series, ZSWAP_MAX_BATCH_SIZE
> > > per-CPU compression buffers are available for sequential and batching
> > > compression algorithms.
> > >
> > > Regardless of compression batching, we always process large folios in
> > > batches of ZSWAP_MAX_BATCH_SIZE.
> > >
> > > If the compressor does not support batching, each page in the batch is
> > > compressed sequentially. For batching compressors, up to
> > > acomp_ctx->comp.compr_batch_size pages in the batch are compressed per
> > > call to crypto_acomp_compress(). The first phase of zswap_compress()
> > > thus only calls crypto_acomp_compress() until all pages in the batch
> > > have been compressed. When this finishes, the compressed buffers and
> > > compressed lengths for the batch are available in the
> > > acomp_ctx->comp.sg_table->sgl scatterlists.
> > >
> > > A new zswap_update_entries() then iterates through the (up to)
> > > ZSWAP_MAX_BATCH_SIZE SG lists in acomp_ctx->comp.sg_table->sgl,
> > > allocates zsmalloc memory and copies all compressed buffers to
> > > the compressed pool. All existing functionality on zswap
> > > writeback_enabled (or not) fallbacks under various error conditions
> > > denoted by the crypto_acomp_compress() return value [and with this
> > > commit, also stored in the individual acomp_ctx->comp.sg_table->sgl
> > > sg->lengths in the crypto_acomp_compress() fast path], is essentially
> > > preserved.
> > >
> > > A significant zswap architectural change is that the batched
> > > zswap_compress() incorporates Herbert's suggestion for SG lists to
> > > represent the batch's inputs/outputs to interface with the crypto API
> > > [1].
> > >
> > > The introduction of zswap_update_entries() presents an opportunity to
> > > optimize the task of storing a batch of compressed buffers. Subsequent
> > > patch series will be submitted to optimize this phase by replacing
> > > zsmalloc with the SLUB allocator to improve zswap_store()/zswap_load()
> > > latencies.
> > >
> > > The batching framework introduced in this series improves zswap_store()
> > > latency by a lot.
> > >
> > > Performance testing: usemem30 with 100% over-commit:
> > > =====================================================
> > > Validation was done with mm-unstable as of 11-17-2026 (Baseline: commit
> > > 866f761fd414) and with this patch series.
> > >
> > > v4 was validated on an Intel Xeon Sapphire Rapids bare-metal instance with
> > > 48 cores.
> > >
> > >  Bare-metal instance specs:
> > >  --------------------------
> > >  - 48 physical cores
> > >  - 190G RAM
> > >  - 200G SWAP disk partition
> > >
> > >  Workload: 30 concurrent usemem processes, each allocates 10G:
> > >  -------------------------------------------------------------
> > >  The vm-scalability "usemem" test (unmodified version) was run in a
> > >  cgroup with memory.high fixed at 150G. There is no swap limit set for the
> > >  cgroup.
> > >
> > >  To quantify the performance benefit of zswap compress batching at scale,
> > >  30 usemem processes were run ("usemem30"), each allocating and writing 10G of
> > >  memory, and sleeping for 10 sec before exiting.
> > >
> > >   usemem -w -O -b 5 -s 10 -n 30 10g
> > >   echo 0 > /sys/module/zswap/parameters/shrinker_enabled
> > >
> > >  Performance validation methodology:
> > >  -----------------------------------
> > >  These settings were used to minimize variation:
> > >
> > >  - All cores were fixed to run at 2.4GHz with "performance" scaling governor.
> > >  - Turbo boost was disabled.
> > >  - CPU power-saving C-states restricted with:
> > >
> > >      GRUB_CMDLINE_LINUX_DEFAULT="intel_idle.max_cstate=1 processor.max_cstate=1"
> > >
> > >  The usemem30 experiments were run with the baseline mm-unstable as of
> > >  11-17-2026 (commit 866f761fd414) and with this patch-series.
> > >
> > >  After booting into each kernel:
> > >
> > >   1) Constrain the cgroup's memory.high to 150G.
> > >
> > >   2) Run 2 warm-up iterations of usemem30. Don't use this data for benchmarking
> > >      since sys time typically skews higher for initial runs.
> > >
> > >   3) Next, run 10 benchmarking iterations of usemem30:
> > >      - Log these high-level metrics: Sys time and individual usemem throughput.
> > >
> > >   4) Sys time and Throughput metrics were gathered for three mTHP settings:
> > >
> > >      - 2048kB
> > >      - 1024kB
> > >      - 64kB
> > >
> > >   5) Benchmark summary data includes Sys time and Throughput components from the
> > >      10 iterations of usemem30:
> > >
> > >      - Total Samples
> > >      - Mean Value
> > >      - Std Deviation
> > >      - CoV (Variance): Coefficient of variation (relative standard
> > >        deviation) measures spread relative to the mean rather than
> > >        statistical variance.
> > >
> > >  An LLM was used to help generate the benchmarking and data processing scripts.
> > >
> > > usemem30 Benchmarking Summary:
> > > ==============================
> > >
> > >  ---------------------------------------------------
> > >     Improvement with v4 zswap compress batching
> > >  ---------------------------------------------------
> > >   mTHP enabled:       2048kB     1024kB       64kB
> > >  ---------------------------------------------------
> > >       Sys time:      -13.87%     -2.64%     -1.72%
> > >     Throughput:       +3.85%     +0.42%     +1.31%
> > >  ---------------------------------------------------
> > >
> > > Detailed usemem30 benchmarking data:
> > > ====================================
> > >
> > >  1) 2048kB mTHP:
> > >  ===============
> > >
> > >  ------------------------------------------------------------------
> > >  2048kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
> > >  ------------------------------------------------------------------
> > >                Baseline mm-unstable   v4 zswap compress batching
> > >                               (sec)        (sec)
> > >  ------------------------------------------------------------------
> > >  [Run 1/10]  Sys Time:    1581.414s    1356.252s
> > >  [Run 2/10]  Sys Time:    1642.874s    1478.929s
> > >  [Run 3/10]  Sys Time:    1594.458s    1411.252s
> > >  [Run 4/10]  Sys Time:    1580.988s    1380.959s
> > >  [Run 5/10]  Sys Time:    1579.654s    1380.609s
> > >  [Run 6/10]  Sys Time:    1627.765s    1345.582s
> > >  [Run 7/10]  Sys Time:    1593.872s    1334.485s
> > >  [Run 8/10]  Sys Time:    1623.189s    1405.471s
> > >  [Run 9/10]  Sys Time:    1661.954s    1433.365s
> > >  [Run 10/10] Sys Time:    1602.098s    1329.813s
> > >  ------------------------------------------------------------------
> > >
> > >  ------------------------------------------------------------------
> > >  METRIC: System CPU Time (Sys Time) : 2048kB mTHP
> > >  ------------------------------------------------------------------
> > >                Baseline mm-unstable   v4 zswap compress batching
> > >                               (sec)            (sec)
> > >  ------------------------------------------------------------------
> > >  Total Samples    :              10               10
> > >  Mean Value       :        1,608.83         1,385.67
> > >  Std Deviation    :           28.67            47.47
> > >  CoV (Variance)   :           1.78%            3.43%
> > >  ------------------------------------------------------------------
> > >  METRIC: Per-Process Throughput : 2048kB mTHP
> > >  ------------------------------------------------------------------
> > >                Baseline mm-unstable   v4 zswap compress batching
> > >                              (KB/s)           (KB/s)
> > >  ------------------------------------------------------------------
> > >  Total Samples    :             300              300
> > >  Mean Value       :         292,261          303,508
> > >  Std Deviation    :          12,946           20,806
> > >  CoV (Variance)   :           4.42%            6.86%
> > >  ------------------------------------------------------------------
> > >
> > >  2) 1024kB mTHP:
> > >  ===============
> > >
> > >  ------------------------------------------------------------------
> > >  1024kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
> > >  ------------------------------------------------------------------
> > >                Baseline mm-unstable   v4 zswap compress batching
> > >                               (sec)        (sec)
> > >  ------------------------------------------------------------------
> > >  [Run 1/10]  Sys Time:    1567.639s    1594.696s
> > >  [Run 2/10]  Sys Time:    1537.577s    1456.113s
> > >  [Run 3/10]  Sys Time:    1538.562s    1521.887s
> > >  [Run 4/10]  Sys Time:    1626.176s    1531.962s
> > >  [Run 5/10]  Sys Time:    1593.955s    1528.733s
> > >  [Run 6/10]  Sys Time:    1643.519s    1558.482s
> > >  [Run 7/10]  Sys Time:    1510.613s    1480.177s
> > >  [Run 8/10]  Sys Time:    1632.018s    1584.093s
> > >  [Run 9/10]  Sys Time:    1569.424s    1528.915s
> > >  [Run 10/10] Sys Time:    1552.747s    1570.451s
> > >  ------------------------------------------------------------------
> > >
> > >  ------------------------------------------------------------------
> > >  METRIC: System CPU Time (Sys Time) : 1024kB mTHP
> > >  ------------------------------------------------------------------
> > >                Baseline mm-unstable   v4 zswap compress batching
> > >                               (sec)            (sec)
> > >  ------------------------------------------------------------------
> > >  Total Samples    :             10                10
> > >  Mean Value       :       1,577.22          1,535.55
> > >  Std Deviation    :          45.13             43.79
> > >  CoV (Variance)   :          2.86%             2.85%
> > >  ------------------------------------------------------------------
> > >  METRIC: Per-Process Throughput : 1024kB mTHP
> > >  ------------------------------------------------------------------
> > >                Baseline mm-unstable   v4 zswap compress batching
> > >                              (KB/s)           (KB/s)
> > >  ------------------------------------------------------------------
> > >  Total Samples    :            300               300
> > >  Mean Value       :        289,904           291,114
> > >  Std Deviation    :         15,385            14,553
> > >  CoV (Variance)   :          5.31%             5.00%
> > >  ------------------------------------------------------------------
> > >
> > >  3) 64kB mTHP:
> > >  =============
> > >
> > >  ------------------------------------------------------------------
> > >  64kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
> > >  ------------------------------------------------------------------
> > >                Baseline mm-unstable   v4 zswap compress batching
> > >                               (sec)        (sec)
> > >  ------------------------------------------------------------------
> > >  [Run 1/10]  Sys Time:    1914.525s    1878.483s
> > >  [Run 2/10]  Sys Time:    1869.412s    1863.177s
> > >  [Run 3/10]  Sys Time:    1944.049s    1863.502s
> > >  [Run 4/10]  Sys Time:    1942.757s    1900.144s
> > >  [Run 5/10]  Sys Time:    1866.505s    1887.285s
> > >  [Run 6/10]  Sys Time:    1877.716s    1861.668s
> > >  [Run 7/10]  Sys Time:    1861.741s    1868.788s
> > >  [Run 8/10]  Sys Time:    1904.735s    1816.372s
> > >  [Run 9/10]  Sys Time:    1891.074s    1850.435s
> > >  [Run 10/10] Sys Time:    1893.366s    1849.747s
> > >  ------------------------------------------------------------------
> > >
> > >  ------------------------------------------------------------------
> > >  METRIC: System CPU Time (Sys Time) : 64kB mTHP
> > >  ------------------------------------------------------------------
> > >                Baseline mm-unstable   v4 zswap compress batching
> > >                               (sec)            (sec)
> > >  ------------------------------------------------------------------
> > >  Total Samples    :              10               10
> > >  Mean Value       :        1,896.59         1,863.96
> > >  Std Deviation    :           29.86            22.96
> > >  CoV (Variance)   :           1.57%            1.23%
> > >  ------------------------------------------------------------------
> > >  METRIC: Per-Process Throughput : 64kB mTHP
> > >  ------------------------------------------------------------------
> > >                Baseline mm-unstable   v4 zswap compress batching
> > >                              (KB/s)           (KB/s)
> > >  ------------------------------------------------------------------
> > >  Total Samples    :            300               300
> > >  Mean Value       :        239,114           242,254
> > >  Std Deviation    :          9,090             6,419
> > >  CoV (Variance)   :          3.80%             2.65%
> > >  ------------------------------------------------------------------
> >
> > The numbers look nice, but what's the zswap setup here? What's the
> > compressor? And how much of the win comes from this patch v.s the last
> > patch?
> >
> > This patch series, and this patch in particular, seem like a lot of
> > complexity. I'd like to understand better the win before adding all
> > this complexity to the codebase.
>
> Hi Nhat,
>
> ZSTD is the zswap compressor. My apologies for forgetting to mention
> this in the performance testing section.
>
> All reclaim and zswap parameters are the defaults in the kernel build
> (I also configure ZSTD as my default zswap compressor when I build the
> kernel):
>
> vm.swappiness = 60
> /sys/module/zswap/parameters/max_pool_percent: 20
> /sys/module/zswap/parameters/shrinker_enabled: N
>
> /sys/kernel/mm/transparent_hugepage/hugepages-2048kB/enabled: [always]
> inherit madvise never
> /sys/kernel/mm/transparent_hugepage/enabled: [always] madvise never
> /sys/kernel/mm/transparent_hugepage/defrag: always defer defer+madvise
> [madvise] never
>
> I just shared my analysis based on bpftrace data in response to your
> cover letter comments, to try and explain the wins with zswap store
> batching alone (patches 6-7) and then with zswap compress batching
> (patch 8). Please do let me know if you have follow-up
> questions/comments.
>
> I will also gather and post usemem30 numbers before/after patch 8.

These are the data summaries/details with v4's zswap store batching
only (patches 6-7), and with zswap compress batching added in patch 8,
to help us better understand the wins.

Batch compression of PMD THP with ZSTD yields an additional 6.56% Sys
time savings and additional 1.12% Throughput improvement over zswap
store batching.
As expected, PMD THP demonstrates the most gains overall with
store+compress batching.

As mentioned in the cover letter, as other kernel components related
to mTHP become more efficient, the net realized performance benefit
can only be better with patch 8 than without.

usemem30 ZSTD Benchmarking Summary:
===================================

 ------------------------------------------------------------------
    Improvement over mm-unstable with v4 zswap store batching only
 ------------------------------------------------------------------
  mTHP enabled:       2048kB     1024kB       64kB
 ------------------------------------------------------------------
      Sys time:      -7.31%      +2.86%     -2.05%
    Throughput:      +2.73%      +0.96%     +2.03%
 ------------------------------------------------------------------

 ------------------------------------------------------------------
    Improvement over mm-unstable with v4 zswap store batching
                                  and v4 zswap compress batching
 ------------------------------------------------------------------
  mTHP enabled:       2048kB     1024kB       64kB
 ------------------------------------------------------------------
      Sys time:      -13.87%     -2.64%     -1.72%
    Throughput:       +3.85%     +0.42%     +1.31%
 ------------------------------------------------------------------

Detailed usemem30 ZSTD benchmarking data:
=========================================

 1) 2048kB mTHP:
 ===============

 ------------------------------------------------------------------------
 2048kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
 ------------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap store   v4 zswap compress
                              (sec)   batching (sec)   batching (sec)
 ------------------------------------------------------------------------
 [Run 1/10]  Sys Time:    1581.414s       1479.949s       1356.252s
 [Run 2/10]  Sys Time:    1642.874s       1436.926s       1478.929s
 [Run 3/10]  Sys Time:    1594.458s       1502.354s       1411.252s
 [Run 4/10]  Sys Time:    1580.988s       1466.870s       1380.959s
 [Run 5/10]  Sys Time:    1579.654s       1567.244s       1380.609s
 [Run 6/10]  Sys Time:    1627.765s       1471.919s       1345.582s
 [Run 7/10]  Sys Time:    1593.872s       1581.152s       1334.485s
 [Run 8/10]  Sys Time:    1623.189s       1440.385s       1405.471s
 [Run 9/10]  Sys Time:    1661.954s       1485.601s       1433.365s
 [Run 10/10] Sys Time:    1602.098s       1480.110s       1329.813s
 ------------------------------------------------------------------------

 ------------------------------------------------------------------------
 METRIC: System CPU Time (Sys Time) : 2048kB mTHP
 ------------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap store   v4 zswap compress
                              (sec)   batching (sec)   batching (sec)
 ------------------------------------------------------------------------
 Total Samples    :              10               10                10
 Mean Value       :        1,608.83         1,491.25          1,385.67
 Std Deviation    :           28.67            48.04             47.47
 CoV (Variance)   :           1.78%            3.22%             3.43%
 ------------------------------------------------------------------------
 METRIC: Per-Process Throughput : 2048kB mTHP
 ------------------------------------------------------------------------
               Baseline mm-unstable    v4 zswap store   v4 zswap compress
                             (KB/s)   batching (KB/s)   batching (KB/s)
 ------------------------------------------------------------------------
 Total Samples    :             300               300               300
 Mean Value       :         292,261           300,233           303,508
 Std Deviation    :          12,946            14,777            20,806
 CoV (Variance)   :           4.42%             4.92%             6.86%
 ------------------------------------------------------------------------


 2) 1024kB mTHP:
 ===============

 ------------------------------------------------------------------------
 1024kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
 ------------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap store   v4 zswap compress
                              (sec)   batching (sec)   batching (sec)
 ------------------------------------------------------------------------
 [Run 1/10]  Sys Time:    1567.639s       1608.238s       1594.696s
 [Run 2/10]  Sys Time:    1537.577s       1608.420s       1456.113s
 [Run 3/10]  Sys Time:    1538.562s       1649.323s       1521.887s
 [Run 4/10]  Sys Time:    1626.176s       1618.451s       1531.962s
 [Run 5/10]  Sys Time:    1593.955s       1550.748s       1528.733s
 [Run 6/10]  Sys Time:    1643.519s       1679.910s       1558.482s
 [Run 7/10]  Sys Time:    1510.613s       1607.291s       1480.177s
 [Run 8/10]  Sys Time:    1632.018s       1636.487s       1584.093s
 [Run 9/10]  Sys Time:    1569.424s       1626.656s       1528.915s
 [Run 10/10] Sys Time:    1552.747s       1638.399s       1570.451s
 ------------------------------------------------------------------------

 ------------------------------------------------------------------------
 METRIC: System CPU Time (Sys Time) : 1024kB mTHP
 ------------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap store   v4 zswap compress
                              (sec)   batching (sec)   batching (sec)
 ------------------------------------------------------------------------
 Total Samples    :             10                10                10
 Mean Value       :       1,577.22          1,622.39          1,535.55
 Std Deviation    :          45.13             33.79             43.79
 CoV (Variance)   :          2.86%             2.08%             2.85%
 ------------------------------------------------------------------------
 METRIC: Per-Process Throughput : 1024kB mTHP
 ------------------------------------------------------------------------
               Baseline mm-unstable    v4 zswap store   v4 zswap compress
                             (KB/s)   batching (KB/s)   batching (KB/s)
 ------------------------------------------------------------------------
 Total Samples    :            300                300               300
 Mean Value       :        289,904            292,673           291,114
 Std Deviation    :         15,385             13,727            14,553
 CoV (Variance)   :          5.31%              4.69%             5.00%
 ------------------------------------------------------------------------


 3) 64kB mTHP:
 =============

 ------------------------------------------------------------------------
 64kB mTHP usemem30 benchmark suite: 10 runs (Limit: 150G).
 ------------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap store   v4 zswap compress
                              (sec)   batching (sec)   batching (sec)
 ------------------------------------------------------------------------
 [Run 1/10]  Sys Time:    1914.525s       1892.620s       1878.483s
 [Run 2/10]  Sys Time:    1869.412s       1845.021s       1863.177s
 [Run 3/10]  Sys Time:    1944.049s       1874.024s       1863.502s
 [Run 4/10]  Sys Time:    1942.757s       1841.728s       1900.144s
 [Run 5/10]  Sys Time:    1866.505s       1819.562s       1887.285s
 [Run 6/10]  Sys Time:    1877.716s       1885.175s       1861.668s
 [Run 7/10]  Sys Time:    1861.741s       1849.232s       1868.788s
 [Run 8/10]  Sys Time:    1904.735s       1861.398s       1816.372s
 [Run 9/10]  Sys Time:    1891.074s       1887.040s       1850.435s
 [Run 10/10] Sys Time:    1893.366s       1821.599s       1849.747s
 ------------------------------------------------------------------------

 ------------------------------------------------------------------------
 METRIC: System CPU Time (Sys Time) : 64kB mTHP
 ------------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap store   v4 zswap compress
                              (sec)   batching (sec)   batching (sec)
 ------------------------------------------------------------------------
 Total Samples    :              10              10                 10
 Mean Value       :        1,896.59        1,857.74           1,863.96
 Std Deviation    :           29.86           26.58              22.96
 CoV (Variance)   :           1.57%           1.43%              1.23%
 ------------------------------------------------------------------------
 METRIC: Per-Process Throughput : 64kB mTHP
 ------------------------------------------------------------------------
               Baseline mm-unstable   v4 zswap store   v4 zswap compress
                             (KB/s)   batching (KB/s)   batching (KB/s)
 ------------------------------------------------------------------------
 Total Samples    :            300                300               300
 Mean Value       :        239,114            243,970           242,254
 Std Deviation    :          9,090              9,971             6,419
 CoV (Variance)   :          3.80%              4.09%             2.65%
 ------------------------------------------------------------------------

Thanks,
Kanchana


^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size.
  2026-11-20  4:19 ` [PATCH v4 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
@ 2026-11-21  4:20   ` sashiko-bot
  0 siblings, 0 replies; 25+ messages in thread
From: sashiko-bot @ 2026-11-21  4:20 UTC (permalink / raw)
  To: Kanchana P. Sridhar, hannes, yosry, nphamcs, chengming.zhou, akpm,
	linux-mm, herbert, davem, linux-crypto
  Cc: rientjes, willy, senozhatsky

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] crypto: acomp: crypto_acomp_batch_size() checks async instead
  of req_seg
--

commit 4f8164105102abaa98a8641d32dd35c9cb223ee0
Author: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>

crypto: acomp - Add API to get a compression algorithm's batch-size.

This commit adds a batch_size field to struct acomp_alg and introduces
crypto_acomp_batch_size() so callers can query the maximum batch size supported
by an acomp algorithm.

> diff --git a/crypto/acompress.c b/crypto/acompress.c
> index ac4dc52668b89..df9e42f519c35 100644
> --- a/crypto/acompress.c
> +++ b/crypto/acompress.c
> @@ -327,6 +327,20 @@ int crypto_acomp_decompress(struct acomp_req *req)
>  }
>  EXPORT_SYMBOL_GPL(crypto_acomp_decompress);
>  
> +unsigned int crypto_acomp_batch_size(struct crypto_acomp *tfm)
> +{
> +	if (acomp_is_async(tfm) &&
> +	    (crypto_comp_alg_common(tfm)->base.cra_type == &crypto_acomp_type)) {
> +		struct acomp_alg *alg = crypto_acomp_alg(tfm);
> +
> +		if (alg && alg->batch_size > 1)
> +			return alg->batch_size;
> +	}
> +
> +	return 1;
> +}
> +EXPORT_SYMBOL_GPL(crypto_acomp_batch_size);

[Severity: Medium]
Should crypto_acomp_batch_size() check crypto_acomp_req_seg(tfm) instead of
acomp_is_async(tfm)?

crypto_acomp_compress() only dispatches segmented requests (where
req->unit_size is set) to tfm->compress(req) when crypto_acomp_req_seg(tfm) is
true, and otherwise sends them to acomp_do_req_chain(), which rejects
multi-segment requests (req->slen > req->unit_size) with -ENOSYS:

crypto/acompress.c:crypto_acomp_compress() {
    ...
	if (req->unit_size && acomp_request_issg(req)) {
		if (!crypto_acomp_req_seg(tfm))
			return acomp_do_req_chain(req, true);

		return tfm->compress(req);
	}
    ...
}

If an asynchronous acomp_alg sets batch_size > 1 without CRYPTO_ALG_REQ_SEG,
crypto_acomp_batch_size() returns alg->batch_size and callers such as zswap
will submit multi-unit requests that fail with -ENOSYS.  Conversely, because
base.cra_type == &crypto_acomp_type already distinguishes struct acomp_alg from
struct scomp_alg, requiring acomp_is_async(tfm) forces
crypto_acomp_batch_size() to return 1 for a synchronous acomp_alg even when it
sets CRYPTO_ALG_REQ_SEG and batch_size > 1.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261120041906.5785-1-kanchanapsridhar2026@gmail.com?part=4

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 5/8] mm: zswap: Store the folio's node id in the zswap_entry.
  2026-11-20  4:19 ` [PATCH v4 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
  2026-11-20 22:46   ` Nhat Pham
@ 2026-11-21  4:20   ` sashiko-bot
  1 sibling, 0 replies; 25+ messages in thread
From: sashiko-bot @ 2026-11-21  4:20 UTC (permalink / raw)
  To: Kanchana P. Sridhar, hannes, yosry, nphamcs, chengming.zhou, akpm,
	linux-mm, herbert, davem, linux-crypto
  Cc: rientjes, willy, senozhatsky

Thank you for your contribution! Sashiko AI review found 1 potential
issue to consider:
- [Medium] mm: zswap: shrink_memcg() skips non-N_NORMAL_MEMORY nodes in
  zswap_list_lru
--

commit e4e47a21782dc39867fe81289fb7e9ed7f2692b4
Author: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>

mm: zswap: Store the folio's node id in the zswap_entry.

This commit stores the folio's NUMA node ID in struct zswap_entry and uses
it when adding and removing entries on zswap_list_lru, instead of deriving
the node ID from the slab page backing the zswap_entry.

> diff --git a/mm/zswap.c b/mm/zswap.c
> index 9a323862bd59d..78d9694f75b5e 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
[ ... ]
> @@ -680,14 +685,8 @@ static int zswap_enabled_param_set(const char *val,
>  * lru functions
>  **********************************/
>  
> -static inline int entry_to_nid(struct zswap_entry *entry)
> -{
> -	return page_to_nid(virt_to_page(entry));
> -}
> -
>  static void zswap_lru_add(struct zswap_entry *entry)
>  {
> -	int nid = entry_to_nid(entry);
>  	struct mem_cgroup *memcg;
>  
>  	/*
> @@ -704,19 +703,18 @@ static void zswap_lru_add(struct zswap_entry *entry)
>  	rcu_read_lock();
>  	memcg = obj_cgroup_memcg(entry->objcg);
>  	/* will always succeed */
> -	list_lru_add(&zswap_list_lru, &entry->lru, nid, memcg);
> +	list_lru_add(&zswap_list_lru, &entry->lru, entry->nid, memcg);
>  	rcu_read_unlock();
>  }
[ ... ]
> @@ -1537,9 +1536,10 @@ static bool zswap_store_page(struct folio *folio, long index,
>  {
>  	swp_entry_t page_swpentry = folio_swap_entry(folio, index);
>  	struct zswap_entry *entry, *old;
> +	int nid = folio_nid(folio);
>  
>  	/* allocate entry */
> -	entry = zswap_entry_cache_alloc(GFP_KERNEL, folio_nid(folio));
> +	entry = zswap_entry_cache_alloc(GFP_KERNEL);
>  	if (!entry) {
>  		zswap_reject_kmemcache_fail++;
>  		return false;
> @@ -1554,6 +1554,7 @@ static bool zswap_store_page(struct folio *folio, long index,
>  	 * to an unrelated live pool.
>  	 */
>  	entry->pool_idx = pool->idx;
> +	entry->nid = nid;

[Severity: Medium]
Can shrink_memcg() miss zswap_list_lru entries when a folio resides on a
node outside N_NORMAL_MEMORY, such as a ZONE_MOVABLE-only CXL or
hotplugged memory node?

Previously, entry_to_nid() took the node ID from the GFP_KERNEL slab page
backing struct zswap_entry, which was always in N_NORMAL_MEMORY.  Now
entry->nid is folio_nid(folio), which for GFP_HIGHUSER_MOVABLE folios can
be any node in N_MEMORY, while shrink_memcg() only walks
N_NORMAL_MEMORY:

mm/zswap.c:shrink_memcg() {
	...
	for_each_node_state(nid, N_NORMAL_MEMORY) {
		unsigned long nr_to_walk = SWAP_CLUSTER_MAX;

		shrunk += list_lru_walk_one(&zswap_list_lru, nid, memcg,
					    &shrink_memcg_cb, &ctl, &nr_to_walk);
		scanned += SWAP_CLUSTER_MAX - nr_to_walk;
	}
	...
}

Entries stored from non-N_NORMAL_MEMORY nodes will never be scanned or
written back by synchronous cgroup limit reclaim or shrink_worker().
Should shrink_memcg() iterate over N_MEMORY instead?

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261120041906.5785-1-kanchanapsridhar2026@gmail.com?part=5

^ permalink raw reply	[flat|nested] 25+ messages in thread

* Re: [PATCH v4 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches.
  2026-11-20  4:19 ` [PATCH v4 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches Kanchana P. Sridhar
  2026-11-20 23:55   ` Nhat Pham
@ 2026-11-21  4:20   ` sashiko-bot
  1 sibling, 0 replies; 25+ messages in thread
From: sashiko-bot @ 2026-11-21  4:20 UTC (permalink / raw)
  To: Kanchana P. Sridhar, hannes, yosry, nphamcs, chengming.zhou, akpm,
	linux-mm, herbert, davem, linux-crypto
  Cc: rientjes, willy, senozhatsky

Thank you for your contribution! Sashiko AI review found 3 potential
issues to consider:
- [High] mm: zswap: invalid and unreset dst scatterlist lengths in
  zswap_compress()
- [High] mm: zswap: unterminated dst scatterlist slice passed in
  zswap_compress()
- [Medium] mm: zswap: unsigned sg->length check and error accounting in
  zswap_compress()
--

commit a77cc3e636d5134b6371aa977bb1025c031583c2
Author: Kanchana P. Sridhar <kanchanapsridhar2026@gmail.com>

mm: zswap: Batched zswap_compress() for storing large folios in batches.

This commit replaces the per-page zswap_compress() call in
zswap_store_pages() with a batched zswap_compress() and
zswap_update_entries() that compress up to ZSWAP_MAX_BATCH_SIZE pages per
batch using a per-CPU scatterlist table mapped to destination buffers.

> diff --git a/mm/zswap.c b/mm/zswap.c
> index 5e1fb5624ffb8..56354a568bad0 100644
> --- a/mm/zswap.c
> +++ b/mm/zswap.c
[ ... ]
> @@ -949,6 +971,26 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
>  		}
>  	}
>  
> +	acomp_ctx->comp.sg_table = kmalloc(sizeof(*acomp_ctx->comp.sg_table),
> +					   GFP_KERNEL);
> +	if (!acomp_ctx->comp.sg_table) {
> +		ret = -ENOMEM;
> +		goto req_fail;
> +	}
> +
> +	if (sg_alloc_table(acomp_ctx->comp.sg_table, ZSWAP_MAX_BATCH_SIZE,
> +			   GFP_KERNEL)) {
> +		ret = -ENOMEM;
> +		goto req_fail;
> +	}
> +
> +	/*
> +	 * Statically map the per-CPU destination buffers to the per-CPU
> +	 * SG lists; crucial for performance.
> +	 */
> +	for_each_sg(acomp_ctx->comp.sg_table->sgl, sg, ZSWAP_MAX_BATCH_SIZE, i)
> +		sg_set_buf(sg, acomp_ctx->comp.buffers[i], PAGE_SIZE);
> +
[ ... ]
> @@ -965,88 +1007,179 @@ static int zswap_cpu_comp_prepare(unsigned int cpu, struct hlist_node *node)
[ ... ]
> +static bool zswap_compress(struct zswap_store_context *zc)
> +{
> +	u8 nr_batch_pages, batch_start, comp_batch_iter;
> +	struct scatterlist *sg, *dst_sg;
> +	unsigned int slen;
> +	int err;
>  
>  	/*
> +	 * @nr_batch_pages is constrained by the per-CPU comp_ctx's compressor
> +	 * batch size and the @zc->nr_pages in this store batch. @slen indicates
> +	 * the total source length bytes for @nr_batch_pages.
>  	 */
> +	nr_batch_pages = min(zc->comp_ctx->compr_batch_size, zc->nr_pages);
>  
>  	/*
> +	 * Compress the @nr_pages in @folio starting at index @folio_start
> +	 * in batches of @nr_batch_pages.
>  	 */
> +	for (batch_start = 0; batch_start < zc->nr_pages;
> +	     batch_start += nr_batch_pages) {
> +		/*
> +		 * Clamp the last iteration's batch size if the compressor's
> +		 * batch_size does not evenly divide @zc->nr_pages.
> +		 */
> +		nr_batch_pages = min(nr_batch_pages, zc->nr_pages - batch_start);
> +		slen = nr_batch_pages * PAGE_SIZE;
> +
> +		/*
> +		 * Send @nr_batch_pages to crypto_acomp for compression:
> +		 *
> +		 * These pages are in @zc->folio's range of indices in the
> +		 * interval:
> +		 *     [@zc->folio_start + @batch_start,
> +		 *      @zc->folio_start + @batch_start + @nr_batch_pages).
> +		 *
> +		 * The source folio pages in the batch are directly submitted to
> +		 * crypto_acomp via acomp_request_set_src_folio().
> +		 */
> +		acomp_request_set_src_folio(zc->comp_ctx->areq.req, zc->folio,
> +				(zc->folio_start + batch_start) * PAGE_SIZE,
> +				slen);
> +
> +		dst_sg = zc->comp_ctx->sg_table->sgl + batch_start;
> +		dst_sg->length = slen;

[Severity: High]
Does setting only dst_sg->length = slen overrun acomp_ctx->comp.buffers[]
and leave dst_sg[1 .. nr_batch_pages - 1].length uninitialized when
nr_batch_pages > 1?

Each entry in acomp_ctx->comp.sg_table->sgl is mapped to a single
PAGE_SIZE buffer in acomp_ctx->comp.buffers[i], and earlier runs overwrite
each entry's sg->length with compressed lengths or negative error codes.
When a batching compressor runs with nr_batch_pages > 1, dst_sg[0].length
is set to nr_batch_pages * PAGE_SIZE (exceeding its PAGE_SIZE buffer),
while dst_sg[1 .. nr_batch_pages - 1].length are not reset to PAGE_SIZE.
If the compressor fails when zc->wb_enabled is true without updating
dst_sg[1 .. nr_batch_pages - 1].length, any entry still holding a stale
compressed length (0 < sg->length < PAGE_SIZE) from a previous call will
skip the PAGE_SIZE fallback and cause zswap_update_entries() to store
stale data from the previous folio.

Each of the nr_batch_pages scatterlist entries in dst_sg needs its length
reset to PAGE_SIZE before crypto_acomp_compress().

> +
> +		/*
> +		 * The per-CPU @zc->comp_ctx->sg_table scatterlists are statically
> +		 * mapped to the per-CPU dst @buffers at pool creation time.
> +		 * There are ZSWAP_MAX_BATCH_SIZE per-CPU SG lists and dst
> +		 * buffers allocated for all compressors.
> +		 */
> +		acomp_request_set_dst_sg(zc->comp_ctx->areq.req, dst_sg, slen);

[Severity: High]
Isn't dst_sg an unterminated scatterlist slice whenever
batch_start + nr_batch_pages < ZSWAP_MAX_BATCH_SIZE?

sg_alloc_table() only sets SG_END on sgl[ZSWAP_MAX_BATCH_SIZE - 1], so
passing dst_sg as req->dst leaves sg_nents(req->dst) equal to
ZSWAP_MAX_BATCH_SIZE - batch_start (4, 3, or 2 instead of nr_batch_pages).
In-tree hardware acomp drivers rely on sg_nents(req->dst):
iaa_comp_acompress() falls back to software deflate whenever
sg_nents(req->dst) > 1 (affecting all order-0 folios and 3 of every 4
pages in a batch), while __qat_bl_sgl_to_bufl() and
hisi_acc_sg_buf_map_to_hw_sgl() walk and DMA-map all sg_nents(req->dst)
entries up to sgl[3] using their sg->length fields, which can still hold a
negative errno such as (unsigned int)-ENOSPC (~4 GiB) left behind by a
prior failed batch.

> +
> +		err = crypto_wait_req(crypto_acomp_compress(zc->comp_ctx->areq.req),
> +				      &zc->comp_ctx->areq.wait);
> +
> +		/*
> +		 * If a page cannot be compressed into a size smaller than
> +		 * PAGE_SIZE, save the content as is without a compression, to
> +		 * keep the LRU order of writebacks.  If writeback is disabled,
> +		 * reject the page since it only adds metadata overhead.
> +		 * swap_writeout() will put the page back to the active LRU list
> +		 * in the case.
> +		 */
> +		if (err && !zc->wb_enabled)
> +			goto compress_error;
> +
> +		/*
> +		 * For each output SG list in @acomp_ctx->req->sg_table->sgl,
> +		 * the @sg->length should be set to either the page's compressed
> +		 * length (success), or it's negative value compression error.
> +		 */
> +		for_each_sg(dst_sg, sg, nr_batch_pages, comp_batch_iter) {
> +			if (((int)sg->length <= 0) || (sg->length >= PAGE_SIZE)) {
> +				if (!zc->wb_enabled) {
> +					if (sg->length == PAGE_SIZE)
> +						sg->length = -ENOSPC;
> +					else if (WARN_ON_ONCE(!sg->length || sg->length > PAGE_SIZE))
> +						sg->length = -EINVAL;
> +					goto compress_error;
> +				}
> +
> +				sg->length = PAGE_SIZE;
> +			}
> +		}
>  	}
>  
> +	return zswap_update_entries(zc);
>  
> +compress_error:
> +	for_each_sg(dst_sg, sg, nr_batch_pages, comp_batch_iter) {
> +		if ((int)sg->length < 0) {
> +			if ((int)sg->length == -ENOSPC)
> +				zswap_reject_compress_poor++;
> +			else
> +				zswap_reject_compress_fail++;
> +		}
> +	}
>  
> +	return false;
>  }

[Severity: Medium]
Won't evaluating sg->length > PAGE_SIZE on an unsigned int trigger
WARN_ON_ONCE() whenever sg->length holds a negative error code?

Because sg->length is unsigned, a negative error code in sg->length (such
as -ENOSPC from a failed page in a batch) satisfies sg->length > PAGE_SIZE
when !zc->wb_enabled, firing a spurious WARN_ON_ONCE() and overwriting the
error code with -EINVAL so compress_error increments
zswap_reject_compress_fail instead of zswap_reject_compress_poor.
Conversely, if crypto_acomp_compress() returns an early request-level
error in err before writing negative error codes into dst_sg->length,
jumping to compress_error leaves all sg->length values positive and
returns false without incrementing either rejection counter.

-- 
Sashiko AI review · https://sashiko.dev/#/patchset/20261120041906.5785-1-kanchanapsridhar2026@gmail.com?part=8

^ permalink raw reply	[flat|nested] 25+ messages in thread

end of thread, other threads:[~2026-11-21  4:20 UTC | newest]

Thread overview: 25+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-20  4:18 [PATCH v4 0/8] Batched zswap_store() framework with compression batching Kanchana P. Sridhar
2026-11-20  4:18 ` [PATCH v4 1/8] crypto: acomp - Define a unit_size in struct acomp_req to enable batching. mm: zswap: Set the unit size for zswap to PAGE_SIZE Kanchana P. Sridhar
2026-11-20  4:19 ` [PATCH v4 2/8] crypto: acomp - Add bit to indicate segmentation support Kanchana P. Sridhar
2026-11-20  4:19 ` [PATCH v4 3/8] crypto: acomp - Add trivial segmentation wrapper Kanchana P. Sridhar
2026-11-20  4:19 ` [PATCH v4 4/8] crypto: acomp - Add API to get a compression algorithm's batch-size Kanchana P. Sridhar
2026-11-21  4:20   ` sashiko-bot
2026-11-20  4:19 ` [PATCH v4 5/8] mm: zswap: Store the folio's node id in the zswap_entry Kanchana P. Sridhar
2026-11-20 22:46   ` Nhat Pham
2026-11-20 23:01     ` Kanchana P. Sridhar
2026-11-21  4:20   ` sashiko-bot
2026-11-20  4:19 ` [PATCH v4 6/8] mm: zswap: Store large folios in batches Kanchana P. Sridhar
2026-11-20 23:08   ` Nhat Pham
2026-11-20 23:40     ` Kanchana P. Sridhar
2026-11-20 23:25   ` Nhat Pham
2026-11-20 23:51     ` Kanchana P. Sridhar
2026-11-20  4:19 ` [PATCH v4 7/8] mm: zswap: Refactor and optimize zswap_store_pages() batch operations Kanchana P. Sridhar
2026-11-20 23:45   ` Nhat Pham
2026-11-21  1:36     ` Kanchana P. Sridhar
2026-11-20  4:19 ` [PATCH v4 8/8] mm: zswap: Batched zswap_compress() for storing large folios in batches Kanchana P. Sridhar
2026-11-20 23:55   ` Nhat Pham
2026-11-21  1:17     ` Kanchana P. Sridhar
2026-11-21  3:24       ` Kanchana P. Sridhar
2026-11-21  4:20   ` sashiko-bot
2026-11-20 23:16 ` [PATCH v4 0/8] Batched zswap_store() framework with compression batching Nhat Pham
2026-11-21  1:04   ` Kanchana P. Sridhar

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.