[PATCH v2 0/4] crypto: skcipher - per-tfm multi-data-unit batching

[Leonid Ravich <lravich@amazon.com>]

This is v2 of the multi-data-unit skcipher request series, addressing
review feedback from Mikulas Patocka on v1.

v1: https://lore.kernel.org/linux-crypto/20260519115955.27267-1-lravich@amazon.com/

The series adds a per-tfm "data unit size" to the skcipher API so a
caller can submit several data units in one crypto request, mirroring
the data_unit_size concept already exposed by struct blk_crypto_config
for inline encryption hardware.  The first user is dm-crypt, which
today issues one skcipher request per sector and so pays a per-sector
cost in request allocation, callback dispatch, completion handling,
and scatterlist setup.

Proof-of-concept performance numbers from the RFC reply [1]: +19%
throughput / -40% CPU on a single-core arm64 system with a hardware
XTS-AES-256 accelerator running fio 4 KiB sequential writes through
dm-crypt, when an out-of-tree arm64 xts driver advertises the new
flag.  This series itself does not include arch enablement.

[1] https://lore.kernel.org/linux-crypto/20260428101225.24316-1-lravich@amazon.com/

Changes since v1
----------------

Patch 4 (dm-crypt) only.  Patches 1-3 are unchanged from v1.

  - Multi-DU scatterlist allocation now uses
    GFP_NOIO | __GFP_NORETRY | __GFP_NOWARN, so the allocator does
    not loop forever waiting for memory that won't come on the
    swap-out-to-dm-crypt path.  (Mikulas)

  - On scatterlist allocation failure, return -EAGAIN instead of
    -ENOMEM.  crypt_convert() handles -EAGAIN by clearing its
    local multi_du flag and re-entering the per-sector path for
    the rest of this crypt_convert() invocation.  The per-tfm
    data_unit_size on the cipher remains set, so subsequent bios
    (which start a fresh crypt_convert() and re-read cipher_flags)
    get to try multi-DU again once memory pressure eases.

    This gives forward progress under total memory exhaustion: the
    per-sector path uses only cc->req_pool (a mempool with
    reservoir set up at table-load time) and the inline
    dmreq->sg_in[]/sg_out[] arrays, never doing any allocation
    that could fail.  The previous v1 mapping of -ENOMEM to
    BLK_STS_DEV_RESOURCE could loop indefinitely on swap, since
    the bio retry would try the same multi-DU allocation again.
    (Mikulas)

  - Walk the bio with __bio_for_each_bvec instead of
    __bio_for_each_segment.  __bio_for_each_segment splits each
    bvec at PAGE_SIZE boundaries; __bio_for_each_bvec keeps
    multi-page bvecs as single units, which is faster with folios
    and produces fewer scatterlist entries.  (Mikulas)

Design overview (unchanged from v1)
-----------------------------------

* Patch 1 adds an `unsigned int data_unit_size` field to
  `struct crypto_skcipher` (per-tfm: invariant for the consumer's
  lifetime, set once via `crypto_skcipher_set_data_unit_size()`),
  plus a capability flag CRYPTO_ALG_SKCIPHER_MULTI_DATA_UNIT in
  `cra_flags` (type-specific high-byte range, mirroring the
  CRYPTO_AHASH_ALG_BLOCK_ONLY precedent).  `crypto_skcipher_encrypt()`
  and `crypto_skcipher_decrypt()` validate that `cryptlen` is a
  positive multiple of `data_unit_size`.  The setter rejects
  sub-blocksize values; algorithm registration rejects the flag
  for algorithms with `ivsize != 16`.

  Also exposes `skcipher_walk_data_units()` in
  <crypto/internal/skcipher.h> as a default per-DU dispatcher for
  drivers that don't want to roll their own.

* Patch 2 lets the generic `xts(...)` template advertise the flag
  when the inner cipher is synchronous.  This is the in-tree
  software producer of the new capability.

* Patch 3 extends `testmgr` with a self-comparison test that fires
  automatically for every alg advertising the flag.

* Patch 4 turns dm-crypt on automatically when all of the
  following hold at table load: skcipher (not aead), tfms_count
  == 1, IV mode is plain or plain64, no per-sector
  iv_gen_ops->post() hook, no dm-integrity stacking, and the
  underlying cipher advertises the capability.

This series intentionally does NOT add the capability flag to any
arch crypto driver.  Arch maintainers can opt in independently in
follow-up patches by wrapping their xts(aes) entry points with
skcipher_walk_data_units() or, for hardware engines, by submitting
one HW command for the whole multi-DU request.

Verification
------------

* checkpatch.pl --strict: clean on all 4 patches.
* Builds clean on x86_64 (defconfig + DM_CRYPT + CRYPTO_AES_NI_INTEL)
  and arm64 (cross-compile, defconfig + DM_CRYPT +
  CRYPTO_AES_ARM64_CE_BLK + CRYPTO_AES_ARM64_NEON_BLK) on top of
  axboe/for-next (a8cafdf8c949).
* QEMU boots; existing xts-aes-aesni / xts-aes-ce / xts-aes-neon
  crypto self-tests pass.
* In-kernel testmgr self-comparison passes for any algorithm
  advertising the flag.
* dm-crypt round-trip with plain64: PASS on x86 and arm64.
* dm-crypt round-trip with essiv:sha256 (single-DU path): PASS.
* dm-crypt large-bio: PASS.
* dm-crypt activation gating: plain -> enabled, plain64 ->
  enabled, essiv:sha256 -> fallback, plain64be -> fallback.
* Byte-equivalence: 256 MB of ciphertext written through the
  multi-DU path is bit-identical to ciphertext written on an
  unpatched axboe/for-next baseline (sha256
  4913910b1aa6f8859fcb8f4adec20230274993a3ade8f4dd0140a323dc43efc0).
* Low-memory boot (mem=128M): PASS — no regression in the
  per-sector path under tight memory.

The OOM-fallback path (multi-DU helper returns -EAGAIN, caller
reverts to per-sector) is verified by inspection: the fallback is
two lines in crypt_convert(), the per-sector path uses only the
existing mempool reserve and the inline dmreq SG arrays (no
allocation that could fail), and there is no shared state between
the two paths that could deadlock.

Leonid Ravich (4):
  crypto: skcipher - add per-tfm data_unit_size for batched requests
  crypto: xts - support multiple data units per request in template
  crypto: testmgr - exercise multi-data-unit path for skcipher
  dm crypt: batch all sectors of a bio per crypto request

 crypto/skcipher.c                  | 120 +++++++++++++
 crypto/testmgr.c                   | 129 ++++++++++++++
 crypto/xts.c                       |  25 ++-
 drivers/md/dm-crypt.c              | 272 ++++++++++++++++++++++++++++-
 include/crypto/internal/skcipher.h |  34 ++++
 include/crypto/skcipher.h          |  85 +++++++++
 6 files changed, 656 insertions(+), 9 deletions(-)


base-commit: a8cafdf8c949f17c92eca0045532e88ac0dac30d
-- 
2.47.3