[RFC] crypto: skcipher multi-data-unit requests for dm-crypt

[Leonid Ravich <lravich@amazon.com>]

dm-crypt submits one skcipher request per sector. For XTS mode with
512-byte sectors, a large bio can contain hundreds of sectors, each
requiring a separate crypto_skcipher_encrypt() call with its own
request allocation, IV generation, and async callback.

On systems with asynchronous hardware crypto accelerators, the
actual encryption is fast and this per-request setup overhead
becomes the bottleneck. Reducing the number of crypto API calls
per bio significantly improves throughput for these configurations.

This problem was discussed in December 2016 by Binoy Jayan, Milan
Broz, and Herbert Xu:

  https://lkml.indiana.edu/hypermail/linux/kernel/1612.2/01912.html

Herbert suggested moving IV generation into the crypto API so that
dm-crypt could submit larger blocks. The essiv template (by Ard
Biesheuvel, merged in 5.4) was the first step. The multi-data-unit
request concept was never implemented.

Existing hardware support
=========================

Several upstream crypto drivers already have hardware support for
per-data-unit tweak management in XTS mode but cannot use it
because the crypto API has no way to communicate the data unit
size:

NXP CAAM (drivers/crypto/caam/) has a hardware sector_size
register in its XTS shared descriptor. The driver currently
hardcodes it to 32768 bytes with this comment:

  "Set sector size to a big value, practically disabling
   sector size segmentation in xts implementation. We cannot
   take full advantage of this HW feature with existing
   crypto API / dm-crypt SW architecture."

Arm CryptoCell (drivers/crypto/ccree/) programs the hardware
data unit size via set_xex_data_unit_size() in its HW descriptor.
The driver template structure already has a data_unit field,
currently unused.

HiSilicon SEC2 (drivers/crypto/hisilicon/sec2/) submits the full
cryptlen to hardware with internal tweak management and SG DMA
via hardware scatter-gather lists.

Intel QAT (drivers/crypto/intel/qat/) submits full cryptlen to
hardware XTS mode in one operation with SG buffer lists.

Proposal
========

Add a data_unit_size field to struct skcipher_request:

  struct skcipher_request {
      unsigned int cryptlen;
      u8 *iv;
      struct scatterlist *src;
      struct scatterlist *dst;
+     unsigned int data_unit_size;
      struct crypto_async_request base;
      void *__ctx[] CRYPTO_MINALIGN_ATTR;
  };

When data_unit_size is 0, behavior is unchanged (cryptlen is one
data unit). When data_unit_size is nonzero, cryptlen must be a
multiple of data_unit_size. The IV applies to the first data unit.
The crypto driver is responsible for incrementing the tweak per
data unit according to the mode.

This mirrors the data_unit_size concept already present in struct
blk_crypto_config for inline encryption. In blk-crypto the size
is a property of the key configuration. Here it is per-request
because dm-crypt may use different sector sizes across different
device-mapper tables sharing the same tfm.

Required changes
=================

1. crypto: skcipher - add data_unit_size to skcipher_request
   as described above. The skcipher layer validates that
   cryptlen is a multiple of data_unit_size before dispatching
   to the driver.

2. crypto: xts - handle data_unit_size > 0 in the generic
   software XTS template by looping internally per data unit.
   This provides a universal fallback so every xts(...)
   instantiation supports multi-data-unit. The tweak increment
   (gf128mul_x_ble) is already implemented in the template.
   Hardware drivers override this with native support.

3. crypto: testmgr - add multi-data-unit XTS test vectors
   that cross-validate against individual per-unit encryption.

4. crypto: drivers - CAAM, CryptoCell, and other drivers with
   hardware data-unit support can read req->data_unit_size and
   program their hardware registers accordingly. For CAAM this
   means setting the sector_size register to the actual value
   instead of the current 32768 workaround.

5. dm-crypt - build a multi-entry scatterlist from the bio's
   bio_vecs, generate the IV for the first sector, set
   data_unit_size to the sector size, and submit one request.
   The existing per-sector path remains for IV modes that
   require post-processing (lmk, tcw, elephant) and for AEAD
   integrity modes.

Thanks,

Leonid Ravich