[PATCH 0/4] crypto: skcipher - per-tfm multi-data-unit batching

[Leonid Ravich <lravich@amazon.com>]

This implements the multi-data-unit skcipher request flow proposed in
the RFC thread [1], following Herbert's ack of the IPsec-friendly
shape and the proof-of-concept performance numbers I posted in [2]
(+19% throughput / -40% CPU on a single-core arm64 system with a
hardware XTS-AES-256 accelerator running fio 4 KiB sequential writes
through dm-crypt).

The series adds a per-tfm "data unit size" to the skcipher API so a
caller can submit several data units in one crypto request, mirroring
the data_unit_size concept already exposed by struct blk_crypto_config
for inline encryption hardware.

The first user is dm-crypt, which today issues one skcipher request
per sector and so pays a per-sector cost in request allocation,
callback dispatch, completion handling, and scatterlist setup.
Allowing the cipher to consume a whole bio per request removes that
overhead.  As shown in [2], the per-sector cost dominates the profile
(~25% of CPU cycles) on a hardware accelerator where AES rounds
themselves are nearly free.

[1] https://lore.kernel.org/linux-crypto/...  (RFC: crypto: skcipher
    multi-data-unit requests for dm-crypt)
[2] Message-Id: 20260428101225.24316-1-lravich@amazon.com

Design overview
---------------

* Patch 1 adds an `unsigned int data_unit_size` field to
  `struct crypto_skcipher` (per-tfm: invariant for the consumer's
  lifetime, set once via `crypto_skcipher_set_data_unit_size()`),
  plus a capability flag CRYPTO_ALG_SKCIPHER_MULTI_DATA_UNIT in
  `cra_flags` (type-specific high-byte range, mirroring the
  CRYPTO_AHASH_ALG_BLOCK_ONLY precedent).  `crypto_skcipher_encrypt()`
  and `crypto_skcipher_decrypt()` validate that `cryptlen` is a
  positive multiple of `data_unit_size`.  The setter rejects
  sub-blocksize values; algorithm registration rejects the flag for
  algorithms with `ivsize != 16`.

  Also exposes `skcipher_walk_data_units()` in
  <crypto/internal/skcipher.h> as a default per-DU dispatcher for
  drivers that don't want to roll their own.

* Patch 2 lets the generic `xts(...)` template advertise the flag
  when the inner cipher is synchronous.  This is the in-tree
  software producer of the new capability.

* Patch 3 extends `testmgr` with a self-comparison test that fires
  automatically for every alg advertising the flag.  The test
  encrypts random plaintext two ways - one batched request vs N
  back-to-back single-DU requests with derived IVs - and rejects
  the algorithm if the ciphertexts differ.

* Patch 4 turns dm-crypt on automatically when all of the following
  hold at table load: skcipher (not aead), `tfms_count == 1`, IV
  mode is plain or plain64, no per-sector `iv_gen_ops->post()`, no
  dm-integrity stacking, and the underlying cipher advertises the
  capability.  Heap-allocated scatterlists are stashed in
  `dm_crypt_request` and freed in `crypt_free_req_skcipher()`,
  initialised to NULL on every request alloc to keep the free path
  safe on the per-sector code path that does not use them.

This series intentionally does NOT add the capability flag to any
arch crypto driver.  Arch maintainers can opt in independently by
wrapping their xts(aes) entry points with skcipher_walk_data_units()
or, for hardware engines, by submitting one HW command for the whole
multi-DU request.  The contract documented in
crypto_skcipher_set_data_unit_size() is the only obligation.

Why per-tfm and why cra_flags
-----------------------------

`data_unit_size` is invariant for the tfm's lifetime in every
plausible consumer.  dm-crypt picks one sector size per mapped
target at table load.  fscrypt would pick one per master key.
IPsec would pick one per SA.  Putting the field on
`crypto_skcipher` (rather than on every `skcipher_request`) avoids
growing a hot per-request struct used by fscrypt, IPsec ESP,
AF_ALG, etc.  It also lets the driver validate the value once in
`setkey()` and keeps the encrypt/decrypt fast path single-branch
(`likely(!data_unit_size)`).

The capability lives in `cra_flags` for consistency with existing
skcipher capabilities, so it surfaces in `/proc/crypto` and templates
can OR it into derived algorithms.

IV semantics
------------

The contract documented in `crypto_skcipher_set_data_unit_size()`:
the algorithm treats the caller-supplied IV as a 128-bit
little-endian counter and adds the data-unit index for each
subsequent data unit.  This is what dm-crypt's plain and plain64
generators already produce, so no IV translation is needed at the
boundary.  For modes that don't fit (essiv, lmk, tcw, eboiv,
plain64be, random, null, benbi, elephant) dm-crypt falls back to the
existing per-sector path.

Verification
------------

* checkpatch.pl --strict: clean on all 4 patches.
* Builds clean on x86_64 and arm64.
* QEMU boots; existing xts-aes-aesni / xts-aes-ce / xts-aes-neon
  crypto self-tests pass.
* In-kernel testmgr self-comparison passes for any algorithm
  advertising the flag.
* dm-crypt round-trip with plain64: encrypt+decrypt produces correct
  data through both the existing per-sector path and the multi-DU
  path (the latter exercised against an out-of-tree arm64 / x86 xts
  enablement during development).
* dm-crypt activation gating: plain -> enabled, plain64 -> enabled,
  essiv:sha256 -> fallback (correctly rejected), plain64be ->
  fallback.
* Byte-equivalence: 256 MB of ciphertext written through the
  multi-DU path is bit-identical to ciphertext written through the
  per-sector path (sha256
  4913910b1aa6f8859fcb8f4adec20230274993a3ade8f4dd0140a323dc43efc0
  on plain64+xts-aes).  The on-disk format is unchanged.

Leonid Ravich (4):
  crypto: skcipher - add per-tfm data_unit_size for batched requests
  crypto: xts - support multiple data units per request in template
  crypto: testmgr - exercise multi-data-unit path for skcipher
  dm crypt: batch all sectors of a bio per crypto request

 crypto/skcipher.c                  | 120 ++++++++++++++
 crypto/testmgr.c                   | 129 +++++++++++++++
 crypto/xts.c                       |  25 ++-
 drivers/md/dm-crypt.c              | 248 ++++++++++++++++++++++++++++-
 include/crypto/internal/skcipher.h |  34 ++++
 include/crypto/skcipher.h          |  85 ++++++++++
 6 files changed, 632 insertions(+), 9 deletions(-)

-- 
2.47.3