* [PATCH v19 00/14] crypto/dmaengine: qce: introduce BAM locking and use DMA for register I/O
From: Bartosz Golaszewski @ 2026-05-26 13:10 UTC (permalink / raw)
To: Vinod Koul, Jonathan Corbet, Thara Gopinath, Herbert Xu,
David S. Miller, Udit Tiwari, Md Sadre Alam, Dmitry Baryshkov,
Manivannan Sadhasivam, Stephan Gerhold, Bjorn Andersson,
Peter Ujfalusi, Michal Simek, Frank Li, Andy Gross,
Neil Armstrong
Cc: dmaengine, linux-doc, linux-kernel, linux-arm-msm, linux-crypto,
linux-arm-kernel, brgl, Bartosz Golaszewski, Bartosz Golaszewski,
Dmitry Baryshkov, Konrad Dybcio
I feel like I fell into the trap of trying to address pre-existing
issues reported by sashiko and in the process provoking more reports so
let this be the last iteration where I do this. Vinod can we get this
queued for v7.2 now and iron out any previously existing problems in
tree?
Merging strategy: there are build-time dependencies between the crypto
and DMA patches so the best approach is for Vinod to create an immutable
branch with the DMA part pulled in by the crypto tree.
This iteration continues to build on top of v12 but uses the BAM's NWD
bit on data descriptors as suggested by Stephan. To that end, there are
some more changes like reversing the order of command and data
descriptors queuedy by the QCE driver.
Currently the QCE crypto driver accesses the crypto engine registers
directly via CPU. Trust Zone may perform crypto operations simultaneously
resulting in a race condition. To remedy that, let's introduce support
for BAM locking/unlocking to the driver. The BAM driver will now wrap
any existing issued descriptor chains with additional descriptors
performing the locking when the client starts the transaction
(dmaengine_issue_pending()). The client wanting to profit from locking
needs to switch to performing register I/O over DMA and communicate the
address to which to perform the dummy writes via a call to
dmaengine_desc_attach_metadata().
In the specific case of the BAM DMA this translates to sending command
descriptors performing dummy writes with the relevant flags set. The BAM
will then lock all other pipes not related to the current pipe group, and
keep handling the current pipe only until it sees the the unlock bit.
In order for the locking to work correctly, we also need to switch to
using DMA for all register I/O.
On top of this, the series contains some additional tweaks and
refactoring.
The goal of this is not to improve the performance but to prepare the
driver for supporting decryption into secure buffers in the future.
Tested with tcrypt.ko, kcapi and cryptsetup.
Signed-off-by: Bartosz Golaszewski <bartosz.golaszewski@linaro.org>
Signed-off-by: Bartosz Golaszewski <bartosz.golaszewski@oss.qualcomm.com>
---
Changes in v19:
- Fix more potential issues in remove path (sashiko)
- Remove unneeded return value check for vchan_tx_prep() as it can never
fail
- Link to v18: https://patch.msgid.link/20260522-qcom-qce-cmd-descr-v18-0-99103926bafc@oss.qualcomm.com
Changes in v18:
- Free the BAM interrupt before disabling the clock in remove() path too
- convert the size assigned to command descriptors to little endian
- don't pass DMA mapping attributes to dma_map_sg() in bam_dma when
setting up command descriptors
- Cancel the QCE workqueue *after* any outstanding DMA transfer
completes
- When mapping the scatterlist for command descriptors: use the actual
number of mapped segments for dmaengine_prep_slave_sg()
- Drop the leftover read_buf field from struct qce_device
- Unmap command descriptors only after terminating the RX transfer
- Pass the actual size of the metadata struct to
dmaengine_desc_attach_metadata(), this is not really required for our
use-case but let's do this for correctness and make sashiko happy
- Drop double assignment of bam_ce_idx in qce_clear_bam_transaction()
- Remove unused QCE_MAX_REG_READ
- Link to v17: https://patch.msgid.link/20260519-qcom-qce-cmd-descr-v17-0-53a595414b79@oss.qualcomm.com
Changes in v17:
- New patch: free the interrupt before disabling the clock in error path
in probe()
- New patch: cancel the QCE work on device detach
- Hold the channel lock when attaching the metadata
- Reorder the operations in devm_qce_dma_request() to avoid freeing
memory that may still be used by the DMA channel
- Register algorithms as the last step in QCE's probe() to avoid making
the resources available to the system before the DMA is fully set up
- Fix error paths in algo request handlers
- Don't pass dmaengine attributes to map_sg_attrs() as it expects
dma-mapping attribute flags
- Fix a dma mapping leak for command descriptors
- Rebase on top of v7.1-rc4
- Link to v16: https://patch.msgid.link/20260427-qcom-qce-cmd-descr-v16-0-945fd1cafbbc@oss.qualcomm.com
Changes in v16:
- Fix a reported race between dma_map_sg() called with spinlock taken
and the corresponding dma_unmap_sg() called without it by moving the
descriptor locking data into the descriptor struct
- Also queue the TX data descriptors before the command descriptors to
match what downstream is doing
- Tweak commit messages
- Rebase on top of v7.1-rc1
- Link to v15: https://patch.msgid.link/20260402-qcom-qce-cmd-descr-v15-0-98b5361f7ed7@oss.qualcomm.com
Changes in v15:
- Extend the descriptor metadata struct to also carry the channel's
transfer direction and stop using dmaengine_slave_config() for that
- Link to v14: https://patch.msgid.link/20260323-qcom-qce-cmd-descr-v14-0-f323af411274@oss.qualcomm.com
Changes in v14:
- Don't return an error to a client which wants to use locking on BAM
that doesn't support it
- Add a comment describing the DMA descriptor metadata structure
- Fix memory leaks
- Remove leftovers from previous iterations
- Propagate errors from dma_cookie_assign() when setting up lock
descriptors
- Link to v13: https://patch.msgid.link/20260317-qcom-qce-cmd-descr-v13-0-0968eb4f8c40@oss.qualcomm.com
Changes in v13:
- As part of the DMA changes in the QCE driver: reverse the order of
queueing the descriptors in the QCE driver: queue command descriptors
with all the register writes first, followed by all the data descriptors,
this is in line with the recommandations from the BAM HPG
- Set the NWD (notify-when-done) bit (DMA_PREP_FENCE in dmaengine
parlance) on the data descriptors to ensure that the UNLOCK descriptor
will not be processed until after they have been processed by the
engine. While technically the NWD bit is only needed on the final data
descriptor, it's hard to tell which one *will* be the last from the
driver's point-of-view and both the downstream driver as well as
the Qualcomm TZ against which we want to synchronize sets NWD on every
data descriptor,
- Revert to creating the LOCK/UNLOCK command descriptor pair in one
place now that the NWD bit is in place,
- Link to v12: https://patch.msgid.link/20260310-qcom-qce-cmd-descr-v12-0-398f37f26ef0@oss.qualcomm.com
Changes in v12:
- Wait until the transaction is done before queueing the UNLOCK command
descriptor
- Use descriptor metadata for communicating the scratchpad address to
the BAM driver
- To that end: reverse the order of the series (first BAM, then QCE) to
maintain bisectability
- Unmap buffers used for dummy writes after the transaction
- Link to v11: https://patch.msgid.link/20260302-qcom-qce-cmd-descr-v11-0-4bf1f5db4802@oss.qualcomm.com
Changes in v11:
- Use new approach, not requiring the client to be involved in locking.
- Add a patch constifying dma_descriptor_metadata_ops
- Rebase on top of v7.0-rc1
- Link to v10: https://lore.kernel.org/r/20251219-qcom-qce-cmd-descr-v10-0-ff7e4bf7dad4@oss.qualcomm.com
Changes in v10:
- Move DESC_FLAG_(UN)LOCK BIT definitions from patch 2 to 3
- Add a patch constifying the dma engine metadata as the first in the
series
- Use the VERSION register for dummy lock/unlock writes
- Link to v9: https://lore.kernel.org/r/20251128-qcom-qce-cmd-descr-v9-0-9a5f72b89722@linaro.org
Changes in v9:
- Drop the global, generic LOCK/UNLOCK flags and instead use DMA
descriptor metadata ops to pass BAM-specific information from the QCE
to the DMA engine
- Link to v8: https://lore.kernel.org/r/20251106-qcom-qce-cmd-descr-v8-0-ecddca23ca26@linaro.org
Changes in v8:
- Rework the command descriptor logic and drop a lot of unneeded code
- Use the physical address for BAM command descriptor access, not the
mapped DMA address
- Fix the problems with iommu faults on newer platforms
- Generalize the LOCK/UNLOCK flags in dmaengine and reword the docs and
commit messages
- Make the BAM locking logic stricter in the DMA engine driver
- Add some additional minor QCE driver refactoring changes to the series
- Lots of small reworks and tweaks to rebase on current mainline and fix
previous issues
- Link to v7: https://lore.kernel.org/all/20250311-qce-cmd-descr-v7-0-db613f5d9c9f@linaro.org/
Changes in v7:
- remove unused code: writing to multiple registers was not used in v6,
neither were the functions for reading registers over BAM DMA-
- remove
- don't read the SW_VERSION register needlessly in the BAM driver,
instead: encode the information on whether the IP supports BAM locking
in device match data
- shrink code where possible with logic modifications (for instance:
change the implementation of qce_write() instead of replacing it
everywhere with a new symbol)
- remove duplicated error messages
- rework commit messages
- a lot of shuffling code around for easier review and a more
streamlined series
- Link to v6: https://lore.kernel.org/all/20250115103004.3350561-1-quic_mdalam@quicinc.com/
Changes in v6:
- change "BAM" to "DMA"
- Ensured this series is compilable with the current Linux-next tip of
the tree (TOT).
Changes in v5:
- Added DMA_PREP_LOCK and DMA_PREP_UNLOCK flag support in separate patch
- Removed DMA_PREP_LOCK & DMA_PREP_UNLOCK flag
- Added FIELD_GET and GENMASK macro to extract major and minor version
Changes in v4:
- Added feature description and test hardware
with test command
- Fixed patch version numbering
- Dropped dt-binding patch
- Dropped device tree changes
- Added BAM_SW_VERSION register read
- Handled the error path for the api dma_map_resource()
in probe
- updated the commit messages for batter redability
- Squash the change where qce_bam_acquire_lock() and
qce_bam_release_lock() api got introduce to the change where
the lock/unlock flag get introced
- changed cover letter subject heading to
"dmaengine: qcom: bam_dma: add cmd descriptor support"
- Added the very initial post for BAM lock/unlock patch link
as v1 to track this feature
Changes in v3:
- https://lore.kernel.org/lkml/183d4f5e-e00a-8ef6-a589-f5704bc83d4a@quicinc.com/
- Addressed all the comments from v2
- Added the dt-binding
- Fix alignment issue
- Removed type casting from qce_write_reg_dma()
and qce_read_reg_dma()
- Removed qce_bam_txn = dma->qce_bam_txn; line from
qce_alloc_bam_txn() api and directly returning
dma->qce_bam_txn
Changes in v2:
- https://lore.kernel.org/lkml/20231214114239.2635325-1-quic_mdalam@quicinc.com/
- Initial set of patches for cmd descriptor support
- Add client driver to use BAM lock/unlock feature
- Added register read/write via BAM in QCE Crypto driver
to use BAM lock/unlock feature
---
Bartosz Golaszewski (14):
dmaengine: constify struct dma_descriptor_metadata_ops
dmaengine: qcom: bam_dma: free interrupt before the clock in error path
dmaengine: qcom: bam_dma: convert tasklet to a BH workqueue
dmaengine: qcom: bam_dma: Extend the driver's device match data
dmaengine: qcom: bam_dma: Add pipe_lock_supported flag support
dmaengine: qcom: bam_dma: add support for BAM locking
crypto: qce - Cancel work on device detach
crypto: qce - Include algapi.h in the core.h header
crypto: qce - Remove unused ignore_buf
crypto: qce - Simplify arguments of devm_qce_dma_request()
crypto: qce - Use existing devres APIs in devm_qce_dma_request()
crypto: qce - Map crypto memory for DMA
crypto: qce - Add BAM DMA support for crypto register I/O
crypto: qce - Communicate the base physical address to the dmaengine
drivers/crypto/qce/aead.c | 10 +-
drivers/crypto/qce/common.c | 20 ++--
drivers/crypto/qce/core.c | 39 ++++++-
drivers/crypto/qce/core.h | 7 ++
drivers/crypto/qce/dma.c | 168 ++++++++++++++++++++++++-----
drivers/crypto/qce/dma.h | 11 +-
drivers/crypto/qce/sha.c | 10 +-
drivers/crypto/qce/skcipher.c | 10 +-
drivers/dma/qcom/bam_dma.c | 227 +++++++++++++++++++++++++++++++++------
drivers/dma/ti/k3-udma.c | 2 +-
drivers/dma/xilinx/xilinx_dma.c | 2 +-
include/linux/dma/qcom_bam_dma.h | 14 +++
include/linux/dmaengine.h | 2 +-
13 files changed, 427 insertions(+), 95 deletions(-)
---
base-commit: def113ae602a35ab7a1dc42a6c43188e180287be
change-id: 20251103-qcom-qce-cmd-descr-c5e9b11fe609
Best regards,
--
Bartosz Golaszewski <bartosz.golaszewski@oss.qualcomm.com>
^ permalink raw reply
* Re: [PATCH net-next 0/6] Remove unused support for crypto tfm cloning
From: Paolo Abeni @ 2026-05-26 10:22 UTC (permalink / raw)
To: Eric Biggers, netdev
Cc: linux-crypto, linux-kernel, Eric Dumazet, Neal Cardwell,
Kuniyuki Iwashima, David S . Miller, David Ahern, Jakub Kicinski,
Simon Horman, Ard Biesheuvel, Jason A . Donenfeld, Herbert Xu,
Dmitry Safonov
In-Reply-To: <20260522053028.91165-1-ebiggers@kernel.org>
On 5/22/26 7:30 AM, Eric Biggers wrote:
> This series is targeting net-next because it depends on
> "net/tcp: Remove tcp_sigpool". So far no commits in cryptodev conflict
> with this, so I suggest that this be taken through net-next for 7.2.
Makes sense, and series LGTM, but waiting a bit for an explicit ack from
Herbert.
/P
^ permalink raw reply
* [PATCH crypto 1/1] crypto: chacha20poly1305: validate poly1305 template argument
From: Ren Wei @ 2026-05-26 10:11 UTC (permalink / raw)
To: linux-crypto
Cc: herbert, davem, yuantan098, zcliangcn, bird, tr0jan,
ngochuongbui67, n05ec
In-Reply-To: <cover.1779777598.git.ngochuongbui67@gmail.com>
From: Xiaonan Zhao <ngochuongbui67@gmail.com>
chachapoly_create() still accepts the compatibility poly1305 parameter
in the template name, but it assumes the second template argument is
always present and immediately passes it to strcmp().
When the argument is missing, crypto_attr_alg_name() returns an error
pointer. Check for that before comparing the name so malformed template
instantiations fail with an error instead of dereferencing the error
pointer in strcmp().
This matches the surrounding Crypto API template pattern where
crypto_attr_alg_name() results are validated before string-specific use.
Fixes: a298765e28ad ("crypto: chacha20poly1305 - Use lib/crypto poly1305")
Cc: stable@kernel.org
Reported-by: Yuan Tan <yuantan098@gmail.com>
Reported-by: Zhengchuan Liang <zcliangcn@gmail.com>
Reported-by: Xin Liu <bird@lzu.edu.cn>
Co-developed-by: Luxing Yin <tr0jan@lzu.edu.cn>
Signed-off-by: Luxing Yin <tr0jan@lzu.edu.cn>
Signed-off-by: Xiaonan Zhao <ngochuongbui67@gmail.com>
Signed-off-by: Ren Wei <n05ec@lzu.edu.cn>
---
crypto/chacha20poly1305.c | 11 +++++++++--
1 file changed, 9 insertions(+), 2 deletions(-)
diff --git a/crypto/chacha20poly1305.c b/crypto/chacha20poly1305.c
index b4b5a7198d84..27df9e1eb058 100644
--- a/crypto/chacha20poly1305.c
+++ b/crypto/chacha20poly1305.c
@@ -375,6 +375,7 @@ static int chachapoly_create(struct crypto_template *tmpl, struct rtattr **tb,
struct aead_instance *inst;
struct chachapoly_instance_ctx *ctx;
struct skcipher_alg_common *chacha;
+ const char *poly_name;
int err;
if (ivsize > CHACHAPOLY_IV_SIZE)
@@ -396,9 +397,15 @@ static int chachapoly_create(struct crypto_template *tmpl, struct rtattr **tb,
goto err_free_inst;
chacha = crypto_spawn_skcipher_alg_common(&ctx->chacha);
+ poly_name = crypto_attr_alg_name(tb[2]);
+ if (IS_ERR(poly_name)) {
+ err = PTR_ERR(poly_name);
+ goto err_free_inst;
+ }
+
err = -EINVAL;
- if (strcmp(crypto_attr_alg_name(tb[2]), "poly1305") &&
- strcmp(crypto_attr_alg_name(tb[2]), "poly1305-generic"))
+ if (strcmp(poly_name, "poly1305") &&
+ strcmp(poly_name, "poly1305-generic"))
goto err_free_inst;
/* Need 16-byte IV size, including Initial Block Counter value */
if (chacha->ivsize != CHACHA_IV_SIZE)
--
2.47.3
^ permalink raw reply related
* [PATCH v4 3/3] crypto: ti - Add support for HMAC in DTHEv2 Hashing Engine driver
From: T Pratham @ 2026-05-26 9:43 UTC (permalink / raw)
To: T Pratham, Herbert Xu, David S. Miller
Cc: Manorit Chawdhry, Kamlesh Gurudasani, Shiva Tripathi,
Kavitha Malarvizhi, Vishal Mahaveer, Praneeth Bajjuri,
linux-crypto, linux-kernel
In-Reply-To: <20260526094355.555712-1-t-pratham@ti.com>
Add support for HMAC-SHA512/384/256/224 and HMAC-MD5 algorithms in the
hashing engine of the DTHEv2 hardware cryptographic engine.
Signed-off-by: T Pratham <t-pratham@ti.com>
---
drivers/crypto/ti/Kconfig | 1 +
drivers/crypto/ti/dthev2-common.h | 10 +-
drivers/crypto/ti/dthev2-hash.c | 346 +++++++++++++++++++++++++++++-
3 files changed, 351 insertions(+), 6 deletions(-)
diff --git a/drivers/crypto/ti/Kconfig b/drivers/crypto/ti/Kconfig
index 9c2aa50cfbfbe..68dccf92f5382 100644
--- a/drivers/crypto/ti/Kconfig
+++ b/drivers/crypto/ti/Kconfig
@@ -13,6 +13,7 @@ config CRYPTO_DEV_TI_DTHEV2
select CRYPTO_SHA256
select CRYPTO_SHA512
select CRYPTO_MD5
+ select CRYPTO_HMAC
select SG_SPLIT
help
This enables support for the TI DTHE V2 hw cryptography engine
diff --git a/drivers/crypto/ti/dthev2-common.h b/drivers/crypto/ti/dthev2-common.h
index 24799007ea81f..847804ed74736 100644
--- a/drivers/crypto/ti/dthev2-common.h
+++ b/drivers/crypto/ti/dthev2-common.h
@@ -31,9 +31,9 @@
#define DTHE_DMA_TIMEOUT_MS 2000
/*
* Size of largest possible key (of all algorithms) to be stored in dthe_tfm_ctx
- * This is currently the keysize of XTS-AES-256 which is 512 bits (64 bytes)
+ * This is currently the keysize of HMAC-SHA512 which is 1024 bits (128 bytes)
*/
-#define DTHE_MAX_KEYSIZE (AES_MAX_KEY_SIZE * 2)
+#define DTHE_MAX_KEYSIZE (SHA512_BLOCK_SIZE)
enum dthe_hash_alg_sel {
DTHE_HASH_MD5 = 0,
@@ -93,9 +93,9 @@ struct dthe_list {
/**
* struct dthe_tfm_ctx - Transform ctx struct containing ctx for all sub-components of DTHE V2
* @dev_data: Device data struct pointer
- * @keylen: AES key length
+ * @keylen: Key length for algorithms that use a key
* @authsize: Authentication size for modes with authentication
- * @key: AES key
+ * @key: Buffer storing the key
* @aes_mode: AES mode
* @hash_mode: Hashing Engine mode
* @phash_size: partial hash size of the hash algorithm selected
@@ -135,6 +135,7 @@ struct dthe_aes_req_ctx {
* @phash: buffer to store a partial hash from a previous operation
* @digestcnt: stores the digest count from a previous operation; currently hardware only provides
* a single 32-bit value even for SHA384/512
+ * @odigest: buffer to store the outer digest from a previous operation
* @phash_available: flag indicating if a partial hash from a previous operation is available
* @flags: flags for internal use
* @padding: padding buffer for handling unaligned data
@@ -143,6 +144,7 @@ struct dthe_aes_req_ctx {
struct dthe_hash_req_ctx {
u32 phash[SHA512_DIGEST_SIZE / sizeof(u32)];
u64 digestcnt[2];
+ u32 odigest[SHA512_DIGEST_SIZE / sizeof(u32)];
u8 phash_available;
u8 flags;
u8 padding[SHA512_BLOCK_SIZE];
diff --git a/drivers/crypto/ti/dthev2-hash.c b/drivers/crypto/ti/dthev2-hash.c
index 4b5df4fdcaa5f..62a42e63c4772 100644
--- a/drivers/crypto/ti/dthev2-hash.c
+++ b/drivers/crypto/ti/dthev2-hash.c
@@ -8,6 +8,7 @@
#include <crypto/algapi.h>
#include <crypto/hash.h>
+#include <crypto/hmac.h>
#include <crypto/internal/hash.h>
#include <crypto/md5.h>
#include <crypto/sha2.h>
@@ -23,6 +24,7 @@
/* Registers */
#define DTHE_P_HASH_BASE 0x5000
+#define DTHE_P_HASH512_ODIGEST_A 0x0200
#define DTHE_P_HASH512_IDIGEST_A 0x0240
#define DTHE_P_HASH512_DIGEST_COUNT 0x0280
#define DTHE_P_HASH512_MODE 0x0284
@@ -45,6 +47,13 @@
#define DTHE_HASH_MODE_USE_ALG_CONST BIT(3)
#define DTHE_HASH_MODE_CLOSE_HASH BIT(4)
+#define DTHE_HASH_MODE_HMAC_KEY_PROCESSING BIT(5)
+#define DTHE_HASH_MODE_HMAC_OUTER_HASH BIT(7)
+
+/* Misc */
+#define DTHE_HMAC_SHA512_MAX_KEYSIZE (SHA512_BLOCK_SIZE)
+#define DTHE_HMAC_SHA256_MAX_KEYSIZE (SHA256_BLOCK_SIZE)
+#define DTHE_HMAC_MD5_MAX_KEYSIZE (MD5_BLOCK_SIZE)
enum dthe_hash_op {
DTHE_HASH_OP_UPDATE = 0,
@@ -74,6 +83,19 @@ static void dthe_hash_write_zero_message(enum dthe_hash_alg_sel mode, void *dst)
}
}
+static int dthe_hmac_write_zero_message(struct ahash_request *req)
+{
+ HASH_FBREQ_ON_STACK(fbreq, req);
+ int ret;
+
+ ahash_request_set_crypt(fbreq, req->src, req->result,
+ req->nbytes);
+
+ ret = crypto_ahash_digest(fbreq);
+ HASH_REQUEST_ZERO(fbreq);
+ return ret;
+}
+
static enum dthe_hash_alg_sel dthe_hash_get_hash_mode(struct crypto_ahash *tfm)
{
unsigned int ds = crypto_ahash_digestsize(tfm);
@@ -131,6 +153,18 @@ static unsigned int dthe_hash_get_phash_size(struct dthe_tfm_ctx *ctx)
return phash_size;
}
+static const char *dthe_hash_get_alg_name(struct dthe_tfm_ctx *ctx)
+{
+ switch (ctx->hash_mode) {
+ case DTHE_HASH_SHA224: return "sha224";
+ case DTHE_HASH_SHA256: return "sha256";
+ case DTHE_HASH_SHA384: return "sha384";
+ case DTHE_HASH_SHA512: return "sha512";
+ case DTHE_HASH_MD5: return "md5";
+ default: return NULL;
+ }
+}
+
static int dthe_hash_init_tfm(struct crypto_ahash *tfm)
{
struct dthe_tfm_ctx *ctx = crypto_ahash_ctx(tfm);
@@ -184,6 +218,7 @@ static int dthe_hash_dma_start(struct ahash_request *req, struct scatterlist *sr
enum dma_data_direction src_dir = DMA_TO_DEVICE;
u32 hash_mode;
int ds = crypto_ahash_digestsize(tfm);
+ bool is_hmac = (ctx->keylen > 0);
int ret = 0;
u32 *dst;
u32 dst_len;
@@ -229,8 +264,11 @@ static int dthe_hash_dma_start(struct ahash_request *req, struct scatterlist *sr
hash_mode = ctx->hash_mode;
- if (rctx->flags == DTHE_HASH_OP_FINUP)
+ if (rctx->flags == DTHE_HASH_OP_FINUP) {
hash_mode |= DTHE_HASH_MODE_CLOSE_HASH;
+ if (is_hmac)
+ hash_mode |= DTHE_HASH_MODE_HMAC_OUTER_HASH;
+ }
if (rctx->phash_available) {
for (int i = 0; i < ctx->phash_size / sizeof(u32); ++i)
@@ -238,9 +276,28 @@ static int dthe_hash_dma_start(struct ahash_request *req, struct scatterlist *sr
sha_base_reg +
DTHE_P_HASH512_IDIGEST_A +
(DTHE_REG_SIZE * i));
+ if (is_hmac) {
+ for (int i = 0; i < ctx->phash_size / sizeof(u32); ++i)
+ writel_relaxed(rctx->odigest[i],
+ sha_base_reg +
+ DTHE_P_HASH512_ODIGEST_A +
+ (DTHE_REG_SIZE * i));
+ }
writel_relaxed(rctx->digestcnt[0],
sha_base_reg + DTHE_P_HASH512_DIGEST_COUNT);
+ } else if (is_hmac) {
+ hash_mode |= DTHE_HASH_MODE_HMAC_KEY_PROCESSING;
+
+ for (int i = 0; i < (ctx->keylen / 2) / sizeof(u32); ++i)
+ writel_relaxed(ctx->key[i], sha_base_reg +
+ DTHE_P_HASH512_ODIGEST_A +
+ (DTHE_REG_SIZE * i));
+ for (int i = 0; i < (ctx->keylen / 2) / sizeof(u32); ++i)
+ writel_relaxed(ctx->key[i + (ctx->keylen / 2) / sizeof(u32)],
+ sha_base_reg +
+ DTHE_P_HASH512_IDIGEST_A +
+ (DTHE_REG_SIZE * i));
} else {
hash_mode |= DTHE_HASH_MODE_USE_ALG_CONST;
}
@@ -275,6 +332,12 @@ static int dthe_hash_dma_start(struct ahash_request *req, struct scatterlist *sr
dst[i] = readl_relaxed(sha_base_reg +
DTHE_P_HASH512_IDIGEST_A +
(DTHE_REG_SIZE * i));
+ if (is_hmac) {
+ for (int i = 0; i < dst_len; ++i)
+ rctx->odigest[i] = readl_relaxed(sha_base_reg +
+ DTHE_P_HASH512_ODIGEST_A +
+ (DTHE_REG_SIZE * i));
+ }
rctx->digestcnt[0] = readl_relaxed(sha_base_reg + DTHE_P_HASH512_DIGEST_COUNT);
rctx->phash_available = 1;
@@ -399,6 +462,10 @@ static int dthe_hash_final(struct ahash_request *req)
return crypto_transfer_hash_request_to_engine(engine, req);
}
+ if (ctx->keylen > 0)
+ /* HMAC with zero-length message */
+ return dthe_hmac_write_zero_message(req);
+
dthe_hash_write_zero_message(ctx->hash_mode, req->result);
return 0;
@@ -458,6 +525,53 @@ static const void *dthe_hash_get_init_state(struct dthe_tfm_ctx *ctx)
}
}
+/*
+ * Compute the HMAC inner or outer pad state (the intermediate hash state after
+ * processing one block of key XOR pad_byte) and write it to @out. Only the
+ * first ctx->phash_size bytes are written, which is always the raw hash state
+ * at the start of the shash export struct.
+ */
+static int dthe_hmac_compute_pad_state(struct dthe_tfm_ctx *ctx, u8 pad_byte,
+ void *out)
+{
+ const char *alg_name = dthe_hash_get_alg_name(ctx);
+ struct crypto_shash *ktfm;
+ u8 data[SHA512_BLOCK_SIZE];
+ unsigned int ss;
+ u8 *state;
+ int ret;
+
+ ktfm = crypto_alloc_shash(alg_name, 0, 0);
+ if (IS_ERR(ktfm))
+ return PTR_ERR(ktfm);
+
+ ss = crypto_shash_statesize(ktfm);
+ state = kmalloc(ss, GFP_KERNEL);
+ if (!state) {
+ crypto_free_shash(ktfm);
+ return -ENOMEM;
+ }
+
+ SHASH_DESC_ON_STACK(desc, ktfm);
+
+ desc->tfm = ktfm;
+ for (int i = 0; i < ctx->keylen; i++)
+ data[i] = ((u8 *)ctx->key)[i] ^ pad_byte;
+
+ ret = crypto_shash_init(desc) ?:
+ crypto_shash_update(desc, data, ctx->keylen) ?:
+ crypto_shash_export(desc, state);
+
+ if (!ret)
+ memcpy(out, state, ctx->phash_size);
+
+ memzero_explicit(state, ss);
+ kfree(state);
+ crypto_free_shash(ktfm);
+ memzero_explicit(data, sizeof(data));
+ return ret;
+}
+
static int dthe_hash_export(struct ahash_request *req, void *out)
{
struct dthe_hash_req_ctx *rctx = ahash_request_ctx(req);
@@ -467,9 +581,12 @@ static int dthe_hash_export(struct ahash_request *req, void *out)
u8 *u8;
u64 *u64;
} p = { .u8 = out };
+ int ret = 0;
if (rctx->phash_available)
memcpy(out, rctx->phash, ctx->phash_size);
+ else if (ctx->keylen > 0)
+ ret = dthe_hmac_compute_pad_state(ctx, HMAC_IPAD_VALUE, out);
else
memcpy(out, dthe_hash_get_init_state(ctx), ctx->phash_size);
@@ -478,7 +595,12 @@ static int dthe_hash_export(struct ahash_request *req, void *out)
if (ctx->phash_size >= SHA512_DIGEST_SIZE)
put_unaligned(rctx->digestcnt[1], p.u64++);
- return 0;
+ if (ctx->keylen > 0) {
+ memcpy(p.u8, rctx->odigest, ctx->phash_size);
+ p.u8 += ctx->phash_size;
+ }
+
+ return ret;
}
static int dthe_hash_import(struct ahash_request *req, const void *in)
@@ -498,9 +620,59 @@ static int dthe_hash_import(struct ahash_request *req, const void *in)
rctx->digestcnt[1] = get_unaligned(p.u64++);
rctx->phash_available = ((rctx->digestcnt[0]) ? 1 : 0);
+ if (ctx->keylen > 0) {
+ memcpy(rctx->odigest, p.u8, ctx->phash_size);
+ p.u8 += ctx->phash_size;
+ }
+
return 0;
}
+static int dthe_hmac_setkey(struct crypto_ahash *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ struct dthe_tfm_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct crypto_ahash *fb = crypto_ahash_fb(tfm);
+ const char *hash_alg_name = dthe_hash_get_alg_name(ctx);
+ unsigned int max_keysize;
+
+ memzero_explicit(ctx->key, sizeof(ctx->key));
+
+ switch (ctx->hash_mode) {
+ case DTHE_HASH_SHA512:
+ case DTHE_HASH_SHA384:
+ max_keysize = DTHE_HMAC_SHA512_MAX_KEYSIZE;
+ break;
+ case DTHE_HASH_SHA256:
+ case DTHE_HASH_SHA224:
+ max_keysize = DTHE_HMAC_SHA256_MAX_KEYSIZE;
+ break;
+ case DTHE_HASH_MD5:
+ max_keysize = DTHE_HMAC_MD5_MAX_KEYSIZE;
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ if (keylen > max_keysize) {
+ struct crypto_shash *ktfm = crypto_alloc_shash(hash_alg_name, 0, 0);
+ SHASH_DESC_ON_STACK(desc, ktfm);
+ int err;
+
+ desc->tfm = ktfm;
+ err = crypto_shash_digest(desc, key, keylen, (u8 *)ctx->key);
+ crypto_free_shash(ktfm);
+ if (err)
+ return err;
+ } else {
+ memcpy(ctx->key, key, keylen);
+ }
+
+ ctx->keylen = max_keysize;
+
+ return crypto_ahash_setkey(fb, key, keylen);
+}
+
static struct ahash_engine_alg hash_algs[] = {
{
.base.init_tfm = dthe_hash_init_tfm,
@@ -667,6 +839,176 @@ static struct ahash_engine_alg hash_algs[] = {
},
.op.do_one_request = dthe_hash_run,
},
+ {
+ .base.init_tfm = dthe_hash_init_tfm,
+ .base.init = dthe_hash_init,
+ .base.update = dthe_hash_update,
+ .base.final = dthe_hash_final,
+ .base.finup = dthe_hash_finup,
+ .base.digest = dthe_hash_digest,
+ .base.export = dthe_hash_export,
+ .base.import = dthe_hash_import,
+ .base.setkey = dthe_hmac_setkey,
+ .base.halg = {
+ .digestsize = SHA512_DIGEST_SIZE,
+ .statesize = sizeof(struct dthe_hash_req_ctx),
+ .base = {
+ .cra_name = "hmac(sha512)",
+ .cra_driver_name = "hmac-sha512-dthev2",
+ .cra_priority = 299,
+ .cra_flags = CRYPTO_ALG_TYPE_AHASH |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_NEED_FALLBACK |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_ALLOCATES_MEMORY |
+ CRYPTO_AHASH_ALG_BLOCK_ONLY |
+ CRYPTO_AHASH_ALG_FINAL_NONZERO |
+ CRYPTO_AHASH_ALG_FINUP_MAX |
+ CRYPTO_AHASH_ALG_NO_EXPORT_CORE,
+ .cra_blocksize = SHA512_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct dthe_tfm_ctx),
+ .cra_reqsize = sizeof(struct dthe_hash_req_ctx),
+ .cra_module = THIS_MODULE,
+ }
+ },
+ .op.do_one_request = dthe_hash_run,
+ },
+ {
+ .base.init_tfm = dthe_hash_init_tfm,
+ .base.init = dthe_hash_init,
+ .base.update = dthe_hash_update,
+ .base.final = dthe_hash_final,
+ .base.finup = dthe_hash_finup,
+ .base.digest = dthe_hash_digest,
+ .base.export = dthe_hash_export,
+ .base.import = dthe_hash_import,
+ .base.setkey = dthe_hmac_setkey,
+ .base.halg = {
+ .digestsize = SHA384_DIGEST_SIZE,
+ .statesize = sizeof(struct dthe_hash_req_ctx),
+ .base = {
+ .cra_name = "hmac(sha384)",
+ .cra_driver_name = "hmac-sha384-dthev2",
+ .cra_priority = 299,
+ .cra_flags = CRYPTO_ALG_TYPE_AHASH |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_NEED_FALLBACK |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_ALLOCATES_MEMORY |
+ CRYPTO_AHASH_ALG_BLOCK_ONLY |
+ CRYPTO_AHASH_ALG_FINAL_NONZERO |
+ CRYPTO_AHASH_ALG_FINUP_MAX |
+ CRYPTO_AHASH_ALG_NO_EXPORT_CORE,
+ .cra_blocksize = SHA384_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct dthe_tfm_ctx),
+ .cra_reqsize = sizeof(struct dthe_hash_req_ctx),
+ .cra_module = THIS_MODULE,
+ }
+ },
+ .op.do_one_request = dthe_hash_run,
+ },
+ {
+ .base.init_tfm = dthe_hash_init_tfm,
+ .base.init = dthe_hash_init,
+ .base.update = dthe_hash_update,
+ .base.final = dthe_hash_final,
+ .base.finup = dthe_hash_finup,
+ .base.digest = dthe_hash_digest,
+ .base.export = dthe_hash_export,
+ .base.import = dthe_hash_import,
+ .base.setkey = dthe_hmac_setkey,
+ .base.halg = {
+ .digestsize = SHA256_DIGEST_SIZE,
+ .statesize = sizeof(struct dthe_hash_req_ctx),
+ .base = {
+ .cra_name = "hmac(sha256)",
+ .cra_driver_name = "hmac-sha256-dthev2",
+ .cra_priority = 299,
+ .cra_flags = CRYPTO_ALG_TYPE_AHASH |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_NEED_FALLBACK |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_ALLOCATES_MEMORY |
+ CRYPTO_AHASH_ALG_BLOCK_ONLY |
+ CRYPTO_AHASH_ALG_FINAL_NONZERO |
+ CRYPTO_AHASH_ALG_FINUP_MAX |
+ CRYPTO_AHASH_ALG_NO_EXPORT_CORE,
+ .cra_blocksize = SHA256_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct dthe_tfm_ctx),
+ .cra_reqsize = sizeof(struct dthe_hash_req_ctx),
+ .cra_module = THIS_MODULE,
+ }
+ },
+ .op.do_one_request = dthe_hash_run,
+ },
+ {
+ .base.init_tfm = dthe_hash_init_tfm,
+ .base.init = dthe_hash_init,
+ .base.update = dthe_hash_update,
+ .base.final = dthe_hash_final,
+ .base.finup = dthe_hash_finup,
+ .base.digest = dthe_hash_digest,
+ .base.export = dthe_hash_export,
+ .base.import = dthe_hash_import,
+ .base.setkey = dthe_hmac_setkey,
+ .base.halg = {
+ .digestsize = SHA224_DIGEST_SIZE,
+ .statesize = sizeof(struct dthe_hash_req_ctx),
+ .base = {
+ .cra_name = "hmac(sha224)",
+ .cra_driver_name = "hmac-sha224-dthev2",
+ .cra_priority = 299,
+ .cra_flags = CRYPTO_ALG_TYPE_AHASH |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_NEED_FALLBACK |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_ALLOCATES_MEMORY |
+ CRYPTO_AHASH_ALG_BLOCK_ONLY |
+ CRYPTO_AHASH_ALG_FINAL_NONZERO |
+ CRYPTO_AHASH_ALG_FINUP_MAX |
+ CRYPTO_AHASH_ALG_NO_EXPORT_CORE,
+ .cra_blocksize = SHA224_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct dthe_tfm_ctx),
+ .cra_reqsize = sizeof(struct dthe_hash_req_ctx),
+ .cra_module = THIS_MODULE,
+ }
+ },
+ .op.do_one_request = dthe_hash_run,
+ },
+ {
+ .base.init_tfm = dthe_hash_init_tfm,
+ .base.init = dthe_hash_init,
+ .base.update = dthe_hash_update,
+ .base.final = dthe_hash_final,
+ .base.finup = dthe_hash_finup,
+ .base.digest = dthe_hash_digest,
+ .base.export = dthe_hash_export,
+ .base.import = dthe_hash_import,
+ .base.setkey = dthe_hmac_setkey,
+ .base.halg = {
+ .digestsize = MD5_DIGEST_SIZE,
+ .statesize = sizeof(struct dthe_hash_req_ctx),
+ .base = {
+ .cra_name = "hmac(md5)",
+ .cra_driver_name = "hmac-md5-dthev2",
+ .cra_priority = 299,
+ .cra_flags = CRYPTO_ALG_TYPE_AHASH |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_NEED_FALLBACK |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_ALLOCATES_MEMORY |
+ CRYPTO_AHASH_ALG_BLOCK_ONLY |
+ CRYPTO_AHASH_ALG_FINAL_NONZERO |
+ CRYPTO_AHASH_ALG_FINUP_MAX |
+ CRYPTO_AHASH_ALG_NO_EXPORT_CORE,
+ .cra_blocksize = MD5_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct dthe_tfm_ctx),
+ .cra_reqsize = sizeof(struct dthe_hash_req_ctx),
+ .cra_module = THIS_MODULE,
+ }
+ },
+ .op.do_one_request = dthe_hash_run,
+ },
};
int dthe_register_hash_algs(void)
--
2.34.1
^ permalink raw reply related
* [PATCH v4 2/3] crypto: ti - Add support for MD5 in DTHEv2 Hashing Engine driver
From: T Pratham @ 2026-05-26 9:43 UTC (permalink / raw)
To: T Pratham, Herbert Xu, David S. Miller
Cc: Manorit Chawdhry, Kamlesh Gurudasani, Shiva Tripathi,
Kavitha Malarvizhi, Vishal Mahaveer, Praneeth Bajjuri,
linux-crypto, linux-kernel
In-Reply-To: <20260526094355.555712-1-t-pratham@ti.com>
Add support for MD5 algorithm in the hashing engine of DTHEv2 hardware
cryptographic engine.
Signed-off-by: T Pratham <t-pratham@ti.com>
---
drivers/crypto/ti/Kconfig | 1 +
drivers/crypto/ti/dthev2-common.h | 1 +
drivers/crypto/ti/dthev2-hash.c | 49 +++++++++++++++++++++++++++++++
3 files changed, 51 insertions(+)
diff --git a/drivers/crypto/ti/Kconfig b/drivers/crypto/ti/Kconfig
index 90af2c7cb1c55..9c2aa50cfbfbe 100644
--- a/drivers/crypto/ti/Kconfig
+++ b/drivers/crypto/ti/Kconfig
@@ -12,6 +12,7 @@ config CRYPTO_DEV_TI_DTHEV2
select CRYPTO_CCM
select CRYPTO_SHA256
select CRYPTO_SHA512
+ select CRYPTO_MD5
select SG_SPLIT
help
This enables support for the TI DTHE V2 hw cryptography engine
diff --git a/drivers/crypto/ti/dthev2-common.h b/drivers/crypto/ti/dthev2-common.h
index bfe7f2de4415c..24799007ea81f 100644
--- a/drivers/crypto/ti/dthev2-common.h
+++ b/drivers/crypto/ti/dthev2-common.h
@@ -17,6 +17,7 @@
#include <crypto/internal/aead.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/skcipher.h>
+#include <crypto/md5.h>
#include <crypto/sha2.h>
#include <linux/delay.h>
diff --git a/drivers/crypto/ti/dthev2-hash.c b/drivers/crypto/ti/dthev2-hash.c
index 204637ab72374..4b5df4fdcaa5f 100644
--- a/drivers/crypto/ti/dthev2-hash.c
+++ b/drivers/crypto/ti/dthev2-hash.c
@@ -9,6 +9,7 @@
#include <crypto/algapi.h>
#include <crypto/hash.h>
#include <crypto/internal/hash.h>
+#include <crypto/md5.h>
#include <crypto/sha2.h>
#include "dthev2-common.h"
@@ -65,6 +66,9 @@ static void dthe_hash_write_zero_message(enum dthe_hash_alg_sel mode, void *dst)
case DTHE_HASH_SHA224:
memcpy(dst, sha224_zero_message_hash, SHA224_DIGEST_SIZE);
break;
+ case DTHE_HASH_MD5:
+ memcpy(dst, md5_zero_message_hash, MD5_DIGEST_SIZE);
+ break;
default:
break;
}
@@ -93,6 +97,9 @@ static enum dthe_hash_alg_sel dthe_hash_get_hash_mode(struct crypto_ahash *tfm)
case SHA224_DIGEST_SIZE:
hash_mode = DTHE_HASH_SHA224;
break;
+ case MD5_DIGEST_SIZE:
+ hash_mode = DTHE_HASH_MD5;
+ break;
default:
hash_mode = DTHE_HASH_ERR;
break;
@@ -114,6 +121,9 @@ static unsigned int dthe_hash_get_phash_size(struct dthe_tfm_ctx *ctx)
case DTHE_HASH_SHA224:
phash_size = SHA256_DIGEST_SIZE;
break;
+ case DTHE_HASH_MD5:
+ phash_size = MD5_DIGEST_SIZE;
+ break;
default:
break;
}
@@ -426,6 +436,10 @@ static const u64 sha512_init_state[SHA512_DIGEST_SIZE / sizeof(u64)] = {
SHA512_H4, SHA512_H5, SHA512_H6, SHA512_H7
};
+static const u32 md5_init_state[MD5_DIGEST_SIZE / sizeof(u32)] = {
+ MD5_H0, MD5_H1, MD5_H2, MD5_H3
+};
+
static const void *dthe_hash_get_init_state(struct dthe_tfm_ctx *ctx)
{
switch (ctx->hash_mode) {
@@ -437,6 +451,8 @@ static const void *dthe_hash_get_init_state(struct dthe_tfm_ctx *ctx)
return sha384_init_state;
case DTHE_HASH_SHA512:
return sha512_init_state;
+ case DTHE_HASH_MD5:
+ return md5_init_state;
default:
return NULL;
}
@@ -618,6 +634,39 @@ static struct ahash_engine_alg hash_algs[] = {
},
.op.do_one_request = dthe_hash_run,
},
+ {
+ .base.init_tfm = dthe_hash_init_tfm,
+ .base.init = dthe_hash_init,
+ .base.update = dthe_hash_update,
+ .base.final = dthe_hash_final,
+ .base.finup = dthe_hash_finup,
+ .base.digest = dthe_hash_digest,
+ .base.export = dthe_hash_export,
+ .base.import = dthe_hash_import,
+ .base.halg = {
+ .digestsize = MD5_DIGEST_SIZE,
+ .statesize = sizeof(struct dthe_hash_req_ctx),
+ .base = {
+ .cra_name = "md5",
+ .cra_driver_name = "md5-dthev2",
+ .cra_priority = 299,
+ .cra_flags = CRYPTO_ALG_TYPE_AHASH |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_OPTIONAL_KEY |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_ALLOCATES_MEMORY |
+ CRYPTO_AHASH_ALG_BLOCK_ONLY |
+ CRYPTO_AHASH_ALG_FINAL_NONZERO |
+ CRYPTO_AHASH_ALG_FINUP_MAX |
+ CRYPTO_AHASH_ALG_NO_EXPORT_CORE,
+ .cra_blocksize = MD5_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct dthe_tfm_ctx),
+ .cra_reqsize = sizeof(struct dthe_hash_req_ctx),
+ .cra_module = THIS_MODULE,
+ }
+ },
+ .op.do_one_request = dthe_hash_run,
+ },
};
int dthe_register_hash_algs(void)
--
2.34.1
^ permalink raw reply related
* [PATCH v4 1/3] crypto: ti - Add support for SHA224/256/384/512 in DTHEv2 driver
From: T Pratham @ 2026-05-26 9:43 UTC (permalink / raw)
To: T Pratham, Herbert Xu, David S. Miller
Cc: Manorit Chawdhry, Kamlesh Gurudasani, Shiva Tripathi,
Kavitha Malarvizhi, Vishal Mahaveer, Praneeth Bajjuri,
linux-kernel, linux-crypto
In-Reply-To: <20260526094355.555712-1-t-pratham@ti.com>
Add support for SHA224, SHA256, SHA384, SHA512 algorithms in the Hashing
Engine of the DTHEv2 hardware cryptographic engine.
Signed-off-by: T Pratham <t-pratham@ti.com>
---
drivers/crypto/ti/Kconfig | 2 +
drivers/crypto/ti/Makefile | 2 +-
drivers/crypto/ti/dthev2-aes.c | 6 +-
drivers/crypto/ti/dthev2-common.c | 43 +-
drivers/crypto/ti/dthev2-common.h | 47 ++-
drivers/crypto/ti/dthev2-hash.c | 631 ++++++++++++++++++++++++++++++
6 files changed, 717 insertions(+), 14 deletions(-)
create mode 100644 drivers/crypto/ti/dthev2-hash.c
diff --git a/drivers/crypto/ti/Kconfig b/drivers/crypto/ti/Kconfig
index 1a3a571ac8cef..90af2c7cb1c55 100644
--- a/drivers/crypto/ti/Kconfig
+++ b/drivers/crypto/ti/Kconfig
@@ -10,6 +10,8 @@ config CRYPTO_DEV_TI_DTHEV2
select CRYPTO_XTS
select CRYPTO_GCM
select CRYPTO_CCM
+ select CRYPTO_SHA256
+ select CRYPTO_SHA512
select SG_SPLIT
help
This enables support for the TI DTHE V2 hw cryptography engine
diff --git a/drivers/crypto/ti/Makefile b/drivers/crypto/ti/Makefile
index b883078f203d7..a90bc97a52321 100644
--- a/drivers/crypto/ti/Makefile
+++ b/drivers/crypto/ti/Makefile
@@ -1,3 +1,3 @@
# SPDX-License-Identifier: GPL-2.0-only
obj-$(CONFIG_CRYPTO_DEV_TI_DTHEV2) += dthev2.o
-dthev2-objs := dthev2-common.o dthev2-aes.o
+dthev2-objs := dthev2-common.o dthev2-aes.o dthev2-hash.o
diff --git a/drivers/crypto/ti/dthev2-aes.c b/drivers/crypto/ti/dthev2-aes.c
index eb5cd902dfb59..164e72c07ee3f 100644
--- a/drivers/crypto/ti/dthev2-aes.c
+++ b/drivers/crypto/ti/dthev2-aes.c
@@ -512,7 +512,7 @@ static int dthe_aes_run(struct crypto_engine *engine, void *areq)
}
local_bh_disable();
- crypto_finalize_skcipher_request(dev_data->engine, req, ret);
+ crypto_finalize_skcipher_request(engine, req, ret);
local_bh_enable();
return 0;
}
@@ -547,7 +547,7 @@ static int dthe_aes_crypt(struct skcipher_request *req)
return 0;
}
- engine = dev_data->engine;
+ engine = dev_data->aes_engine;
return crypto_transfer_skcipher_request_to_engine(engine, req);
}
@@ -1183,7 +1183,7 @@ static int dthe_aead_crypt(struct aead_request *req)
(ctx->aes_mode == DTHE_AES_CCM && !is_zero_ctr))
return dthe_aead_do_fallback(req);
- engine = dev_data->engine;
+ engine = dev_data->aes_engine;
return crypto_transfer_aead_request_to_engine(engine, req);
}
diff --git a/drivers/crypto/ti/dthev2-common.c b/drivers/crypto/ti/dthev2-common.c
index a2ad79bec105a..88d0ccd5d0227 100644
--- a/drivers/crypto/ti/dthev2-common.c
+++ b/drivers/crypto/ti/dthev2-common.c
@@ -96,6 +96,11 @@ static int dthe_dma_init(struct dthe_data *dev_data)
goto err_dma_sha_tx;
}
+ /*
+ * Do AES Rx and Tx channel config here because it is invariant of AES mode
+ * SHA Tx channel config is done before DMA transfer depending on hashing algorithm
+ */
+
memzero_explicit(&cfg, sizeof(cfg));
cfg.src_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
@@ -130,11 +135,21 @@ static int dthe_dma_init(struct dthe_data *dev_data)
static int dthe_register_algs(void)
{
- return dthe_register_aes_algs();
+ int ret = 0;
+
+ ret = dthe_register_hash_algs();
+ if (ret)
+ return ret;
+ ret = dthe_register_aes_algs();
+ if (ret)
+ dthe_unregister_hash_algs();
+
+ return ret;
}
static void dthe_unregister_algs(void)
{
+ dthe_unregister_hash_algs();
dthe_unregister_aes_algs();
}
@@ -163,15 +178,26 @@ static int dthe_probe(struct platform_device *pdev)
if (ret)
goto probe_dma_err;
- dev_data->engine = crypto_engine_alloc_init(dev, 1);
- if (!dev_data->engine) {
+ dev_data->aes_engine = crypto_engine_alloc_init(dev, 1);
+ if (!dev_data->aes_engine) {
ret = -ENOMEM;
goto probe_engine_err;
}
+ dev_data->hash_engine = crypto_engine_alloc_init(dev, 1);
+ if (!dev_data->hash_engine) {
+ ret = -ENOMEM;
+ goto probe_hash_engine_err;
+ }
+
+ ret = crypto_engine_start(dev_data->aes_engine);
+ if (ret) {
+ dev_err(dev, "Failed to start crypto engine for AES\n");
+ goto probe_engine_start_err;
+ }
- ret = crypto_engine_start(dev_data->engine);
+ ret = crypto_engine_start(dev_data->hash_engine);
if (ret) {
- dev_err(dev, "Failed to start crypto engine\n");
+ dev_err(dev, "Failed to start crypto engine for hash\n");
goto probe_engine_start_err;
}
@@ -184,7 +210,9 @@ static int dthe_probe(struct platform_device *pdev)
return 0;
probe_engine_start_err:
- crypto_engine_exit(dev_data->engine);
+ crypto_engine_exit(dev_data->hash_engine);
+probe_hash_engine_err:
+ crypto_engine_exit(dev_data->aes_engine);
probe_engine_err:
dma_release_channel(dev_data->dma_aes_rx);
dma_release_channel(dev_data->dma_aes_tx);
@@ -207,7 +235,8 @@ static void dthe_remove(struct platform_device *pdev)
dthe_unregister_algs();
- crypto_engine_exit(dev_data->engine);
+ crypto_engine_exit(dev_data->aes_engine);
+ crypto_engine_exit(dev_data->hash_engine);
dma_release_channel(dev_data->dma_aes_rx);
dma_release_channel(dev_data->dma_aes_tx);
diff --git a/drivers/crypto/ti/dthev2-common.h b/drivers/crypto/ti/dthev2-common.h
index d4a3b9c18bbc1..bfe7f2de4415c 100644
--- a/drivers/crypto/ti/dthev2-common.h
+++ b/drivers/crypto/ti/dthev2-common.h
@@ -17,6 +17,7 @@
#include <crypto/internal/aead.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/skcipher.h>
+#include <crypto/sha2.h>
#include <linux/delay.h>
#include <linux/dmaengine.h>
@@ -33,6 +34,16 @@
*/
#define DTHE_MAX_KEYSIZE (AES_MAX_KEY_SIZE * 2)
+enum dthe_hash_alg_sel {
+ DTHE_HASH_MD5 = 0,
+ DTHE_HASH_SHA1 = BIT(1),
+ DTHE_HASH_SHA224 = BIT(2),
+ DTHE_HASH_SHA256 = BIT(1) | BIT(2),
+ DTHE_HASH_SHA384 = BIT(0),
+ DTHE_HASH_SHA512 = BIT(0) | BIT(1),
+ DTHE_HASH_ERR = BIT(0) | BIT(1) | BIT(2),
+};
+
enum dthe_aes_mode {
DTHE_AES_ECB = 0,
DTHE_AES_CBC,
@@ -49,7 +60,8 @@ enum dthe_aes_mode {
* @dev: Device pointer
* @regs: Base address of the register space
* @list: list node for dev
- * @engine: Crypto engine instance
+ * @aes_engine: Crypto engine instance for AES Engine
+ * @hash_engine: Crypto engine instance for Hashing Engine
* @dma_aes_rx: AES Rx DMA Channel
* @dma_aes_tx: AES Tx DMA Channel
* @dma_sha_tx: SHA Tx DMA Channel
@@ -58,7 +70,8 @@ struct dthe_data {
struct device *dev;
void __iomem *regs;
struct list_head list;
- struct crypto_engine *engine;
+ struct crypto_engine *aes_engine;
+ struct crypto_engine *hash_engine;
struct dma_chan *dma_aes_rx;
struct dma_chan *dma_aes_tx;
@@ -83,6 +96,8 @@ struct dthe_list {
* @authsize: Authentication size for modes with authentication
* @key: AES key
* @aes_mode: AES mode
+ * @hash_mode: Hashing Engine mode
+ * @phash_size: partial hash size of the hash algorithm selected
* @aead_fb: Fallback crypto aead handle
* @skcipher_fb: Fallback crypto skcipher handle for AES-XTS mode
*/
@@ -91,7 +106,11 @@ struct dthe_tfm_ctx {
unsigned int keylen;
unsigned int authsize;
u32 key[DTHE_MAX_KEYSIZE / sizeof(u32)];
- enum dthe_aes_mode aes_mode;
+ union {
+ enum dthe_aes_mode aes_mode;
+ enum dthe_hash_alg_sel hash_mode;
+ };
+ unsigned int phash_size;
union {
struct crypto_sync_aead *aead_fb;
struct crypto_sync_skcipher *skcipher_fb;
@@ -110,6 +129,25 @@ struct dthe_aes_req_ctx {
struct completion aes_compl;
};
+/**
+ * struct dthe_hash_req_ctx - Hashing engine ctx struct
+ * @phash: buffer to store a partial hash from a previous operation
+ * @digestcnt: stores the digest count from a previous operation; currently hardware only provides
+ * a single 32-bit value even for SHA384/512
+ * @phash_available: flag indicating if a partial hash from a previous operation is available
+ * @flags: flags for internal use
+ * @padding: padding buffer for handling unaligned data
+ * @hash_compl: Completion variable for use in manual completion in case of DMA callback failure
+ */
+struct dthe_hash_req_ctx {
+ u32 phash[SHA512_DIGEST_SIZE / sizeof(u32)];
+ u64 digestcnt[2];
+ u8 phash_available;
+ u8 flags;
+ u8 padding[SHA512_BLOCK_SIZE];
+ struct completion hash_compl;
+};
+
/* Struct definitions end */
struct dthe_data *dthe_get_dev(struct dthe_tfm_ctx *ctx);
@@ -131,4 +169,7 @@ struct scatterlist *dthe_copy_sg(struct scatterlist *dst,
int dthe_register_aes_algs(void);
void dthe_unregister_aes_algs(void);
+int dthe_register_hash_algs(void);
+void dthe_unregister_hash_algs(void);
+
#endif
diff --git a/drivers/crypto/ti/dthev2-hash.c b/drivers/crypto/ti/dthev2-hash.c
new file mode 100644
index 0000000000000..204637ab72374
--- /dev/null
+++ b/drivers/crypto/ti/dthev2-hash.c
@@ -0,0 +1,631 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * K3 DTHE V2 crypto accelerator driver
+ *
+ * Copyright (C) Texas Instruments 2025 - https://www.ti.com
+ * Author: T Pratham <t-pratham@ti.com>
+ */
+
+#include <crypto/algapi.h>
+#include <crypto/hash.h>
+#include <crypto/internal/hash.h>
+#include <crypto/sha2.h>
+
+#include "dthev2-common.h"
+
+#include <linux/delay.h>
+#include <linux/dmaengine.h>
+#include <linux/dma-mapping.h>
+#include <linux/io.h>
+#include <linux/scatterlist.h>
+
+/* Registers */
+
+#define DTHE_P_HASH_BASE 0x5000
+#define DTHE_P_HASH512_IDIGEST_A 0x0240
+#define DTHE_P_HASH512_DIGEST_COUNT 0x0280
+#define DTHE_P_HASH512_MODE 0x0284
+#define DTHE_P_HASH512_LENGTH 0x0288
+#define DTHE_P_HASH512_DATA_IN_START 0x0080
+#define DTHE_P_HASH512_DATA_IN_END 0x00FC
+
+#define DTHE_P_HASH_SYSCONFIG 0x0110
+#define DTHE_P_HASH_IRQSTATUS 0x0118
+#define DTHE_P_HASH_IRQENABLE 0x011C
+
+/* Register write values and macros */
+#define DTHE_HASH_SYSCONFIG_INT_EN BIT(2)
+#define DTHE_HASH_SYSCONFIG_DMA_EN BIT(3)
+#define DTHE_HASH_IRQENABLE_EN_ALL GENMASK(3, 0)
+#define DTHE_HASH_IRQSTATUS_OP_READY BIT(0)
+#define DTHE_HASH_IRQSTATUS_IP_READY BIT(1)
+#define DTHE_HASH_IRQSTATUS_PH_READY BIT(2)
+#define DTHE_HASH_IRQSTATUS_CTX_READY BIT(3)
+
+#define DTHE_HASH_MODE_USE_ALG_CONST BIT(3)
+#define DTHE_HASH_MODE_CLOSE_HASH BIT(4)
+
+enum dthe_hash_op {
+ DTHE_HASH_OP_UPDATE = 0,
+ DTHE_HASH_OP_FINUP,
+};
+
+static void dthe_hash_write_zero_message(enum dthe_hash_alg_sel mode, void *dst)
+{
+ switch (mode) {
+ case DTHE_HASH_SHA512:
+ memcpy(dst, sha512_zero_message_hash, SHA512_DIGEST_SIZE);
+ break;
+ case DTHE_HASH_SHA384:
+ memcpy(dst, sha384_zero_message_hash, SHA384_DIGEST_SIZE);
+ break;
+ case DTHE_HASH_SHA256:
+ memcpy(dst, sha256_zero_message_hash, SHA256_DIGEST_SIZE);
+ break;
+ case DTHE_HASH_SHA224:
+ memcpy(dst, sha224_zero_message_hash, SHA224_DIGEST_SIZE);
+ break;
+ default:
+ break;
+ }
+}
+
+static enum dthe_hash_alg_sel dthe_hash_get_hash_mode(struct crypto_ahash *tfm)
+{
+ unsigned int ds = crypto_ahash_digestsize(tfm);
+ enum dthe_hash_alg_sel hash_mode;
+
+ /*
+ * Currently, all hash algorithms supported by DTHEv2 have unique digest sizes.
+ * So we can do this. Otherwise, we would have to get the algorithm from the
+ * alg_name and do a strcmp.
+ */
+ switch (ds) {
+ case SHA512_DIGEST_SIZE:
+ hash_mode = DTHE_HASH_SHA512;
+ break;
+ case SHA384_DIGEST_SIZE:
+ hash_mode = DTHE_HASH_SHA384;
+ break;
+ case SHA256_DIGEST_SIZE:
+ hash_mode = DTHE_HASH_SHA256;
+ break;
+ case SHA224_DIGEST_SIZE:
+ hash_mode = DTHE_HASH_SHA224;
+ break;
+ default:
+ hash_mode = DTHE_HASH_ERR;
+ break;
+ }
+
+ return hash_mode;
+}
+
+static unsigned int dthe_hash_get_phash_size(struct dthe_tfm_ctx *ctx)
+{
+ unsigned int phash_size = 0;
+
+ switch (ctx->hash_mode) {
+ case DTHE_HASH_SHA512:
+ case DTHE_HASH_SHA384:
+ phash_size = SHA512_DIGEST_SIZE;
+ break;
+ case DTHE_HASH_SHA256:
+ case DTHE_HASH_SHA224:
+ phash_size = SHA256_DIGEST_SIZE;
+ break;
+ default:
+ break;
+ }
+
+ return phash_size;
+}
+
+static int dthe_hash_init_tfm(struct crypto_ahash *tfm)
+{
+ struct dthe_tfm_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct dthe_data *dev_data = dthe_get_dev(ctx);
+
+ if (!dev_data)
+ return -ENODEV;
+
+ ctx->dev_data = dev_data;
+
+ ctx->hash_mode = dthe_hash_get_hash_mode(tfm);
+ if (ctx->hash_mode == DTHE_HASH_ERR)
+ return -EINVAL;
+
+ ctx->phash_size = dthe_hash_get_phash_size(ctx);
+
+ return 0;
+}
+
+static int dthe_hash_config_dma_chan(struct dma_chan *chan, struct crypto_ahash *tfm)
+{
+ struct dma_slave_config cfg;
+ int bs = crypto_ahash_blocksize(tfm);
+
+ memzero_explicit(&cfg, sizeof(cfg));
+
+ cfg.dst_addr_width = DMA_SLAVE_BUSWIDTH_4_BYTES;
+ cfg.dst_maxburst = bs / 4;
+
+ return dmaengine_slave_config(chan, &cfg);
+}
+
+static void dthe_hash_dma_in_callback(void *data)
+{
+ struct ahash_request *req = (struct ahash_request *)data;
+ struct dthe_hash_req_ctx *rctx = ahash_request_ctx(req);
+
+ complete(&rctx->hash_compl);
+}
+
+static int dthe_hash_dma_start(struct ahash_request *req, struct scatterlist *src,
+ int src_nents, size_t len)
+{
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct dthe_tfm_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct dthe_hash_req_ctx *rctx = ahash_request_ctx(req);
+ struct dthe_data *dev_data = dthe_get_dev(ctx);
+ struct device *tx_dev;
+ struct dma_async_tx_descriptor *desc_out;
+ int mapped_nents;
+ enum dma_data_direction src_dir = DMA_TO_DEVICE;
+ u32 hash_mode;
+ int ds = crypto_ahash_digestsize(tfm);
+ int ret = 0;
+ u32 *dst;
+ u32 dst_len;
+ void __iomem *sha_base_reg = dev_data->regs + DTHE_P_HASH_BASE;
+
+ u32 hash_sysconfig_val = DTHE_HASH_SYSCONFIG_INT_EN | DTHE_HASH_SYSCONFIG_DMA_EN;
+ u32 hash_irqenable_val = DTHE_HASH_IRQENABLE_EN_ALL;
+
+ writel_relaxed(hash_sysconfig_val, sha_base_reg + DTHE_P_HASH_SYSCONFIG);
+ writel_relaxed(hash_irqenable_val, sha_base_reg + DTHE_P_HASH_IRQENABLE);
+
+ /* Config SHA DMA channel as per SHA mode */
+ ret = dthe_hash_config_dma_chan(dev_data->dma_sha_tx, tfm);
+ if (ret) {
+ dev_err(dev_data->dev, "Can't configure sha_tx dmaengine slave: %d\n", ret);
+ goto hash_err;
+ }
+
+ tx_dev = dmaengine_get_dma_device(dev_data->dma_sha_tx);
+ if (!tx_dev) {
+ ret = -ENODEV;
+ goto hash_err;
+ }
+
+ mapped_nents = dma_map_sg(tx_dev, src, src_nents, src_dir);
+ if (mapped_nents == 0) {
+ ret = -EINVAL;
+ goto hash_err;
+ }
+
+ desc_out = dmaengine_prep_slave_sg(dev_data->dma_sha_tx, src, mapped_nents,
+ DMA_MEM_TO_DEV, DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
+ if (!desc_out) {
+ dev_err(dev_data->dev, "OUT prep_slave_sg() failed\n");
+ ret = -EINVAL;
+ goto hash_prep_err;
+ }
+
+ desc_out->callback = dthe_hash_dma_in_callback;
+ desc_out->callback_param = req;
+
+ init_completion(&rctx->hash_compl);
+
+ hash_mode = ctx->hash_mode;
+
+ if (rctx->flags == DTHE_HASH_OP_FINUP)
+ hash_mode |= DTHE_HASH_MODE_CLOSE_HASH;
+
+ if (rctx->phash_available) {
+ for (int i = 0; i < ctx->phash_size / sizeof(u32); ++i)
+ writel_relaxed(rctx->phash[i],
+ sha_base_reg +
+ DTHE_P_HASH512_IDIGEST_A +
+ (DTHE_REG_SIZE * i));
+
+ writel_relaxed(rctx->digestcnt[0],
+ sha_base_reg + DTHE_P_HASH512_DIGEST_COUNT);
+ } else {
+ hash_mode |= DTHE_HASH_MODE_USE_ALG_CONST;
+ }
+
+ writel_relaxed(hash_mode, sha_base_reg + DTHE_P_HASH512_MODE);
+ writel_relaxed(len, sha_base_reg + DTHE_P_HASH512_LENGTH);
+
+ dmaengine_submit(desc_out);
+
+ dma_async_issue_pending(dev_data->dma_sha_tx);
+
+ ret = wait_for_completion_timeout(&rctx->hash_compl,
+ msecs_to_jiffies(DTHE_DMA_TIMEOUT_MS));
+ if (!ret) {
+ dmaengine_terminate_sync(dev_data->dma_sha_tx);
+ ret = -ETIMEDOUT;
+ } else {
+ ret = 0;
+ }
+
+ if (rctx->flags == DTHE_HASH_OP_UPDATE) {
+ /* If coming from update, we need to read the phash and store it for future */
+ dst = rctx->phash;
+ dst_len = ctx->phash_size / sizeof(u32);
+ } else {
+ /* If coming from finup or final, we need to read the final digest */
+ dst = (u32 *)req->result;
+ dst_len = ds / sizeof(u32);
+ }
+
+ for (int i = 0; i < dst_len; ++i)
+ dst[i] = readl_relaxed(sha_base_reg +
+ DTHE_P_HASH512_IDIGEST_A +
+ (DTHE_REG_SIZE * i));
+
+ rctx->digestcnt[0] = readl_relaxed(sha_base_reg + DTHE_P_HASH512_DIGEST_COUNT);
+ rctx->phash_available = 1;
+
+hash_prep_err:
+ dma_unmap_sg(tx_dev, src, src_nents, src_dir);
+hash_err:
+ return ret;
+}
+
+static int dthe_hash_run(struct crypto_engine *engine, void *areq)
+{
+ struct ahash_request *req = container_of(areq, struct ahash_request, base);
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct dthe_hash_req_ctx *rctx = ahash_request_ctx(req);
+
+ struct scatterlist *src, *sg;
+ int src_nents = 0;
+ unsigned int bs = crypto_ahash_blocksize(tfm);
+ unsigned int tot_len = req->nbytes;
+ unsigned int len_to_process;
+ unsigned int len_to_buffer;
+ unsigned int pad_len = 0;
+ u8 *pad_buf = rctx->padding;
+ int ret = 0;
+
+ if (rctx->flags == DTHE_HASH_OP_UPDATE) {
+ len_to_process = tot_len - (tot_len % bs);
+ len_to_buffer = tot_len % bs;
+
+ if (len_to_process == 0) {
+ ret = len_to_buffer;
+ goto hash_buf_all;
+ }
+ } else {
+ len_to_process = tot_len;
+ len_to_buffer = 0;
+ }
+
+ src_nents = sg_nents_for_len(req->src, len_to_process);
+
+ /*
+ * Certain DMA restrictions forced us to send data in multiples of BLOCK_SIZE
+ * bytes. So, add a padding 0s at the end of src scatterlist if data is not a
+ * multiple of block_size bytes (Can only happen in final or finup). The extra
+ * data is ignored by the DTHE hardware.
+ */
+ if (len_to_process % bs) {
+ pad_len = bs - (len_to_process % bs);
+ src_nents++;
+ }
+
+ src = kcalloc(src_nents, sizeof(*src), GFP_KERNEL);
+ if (!src) {
+ ret = -ENOMEM;
+ goto hash_buf_all;
+ }
+
+ sg_init_table(src, src_nents);
+ sg = dthe_copy_sg(src, req->src, len_to_process);
+ if (pad_len > 0) {
+ memset(pad_buf, 0, pad_len);
+ sg_set_buf(sg, pad_buf, pad_len);
+ }
+
+ ret = dthe_hash_dma_start(req, src, src_nents, len_to_process);
+ if (!ret)
+ ret = len_to_buffer;
+
+ kfree(src);
+
+hash_buf_all:
+ local_bh_disable();
+ crypto_finalize_hash_request(engine, req, ret);
+ local_bh_enable();
+ return 0;
+}
+
+static int dthe_hash_init(struct ahash_request *req)
+{
+ struct dthe_hash_req_ctx *rctx = ahash_request_ctx(req);
+
+ rctx->phash_available = 0;
+ rctx->digestcnt[0] = 0;
+ rctx->digestcnt[1] = 0;
+
+ return 0;
+}
+
+static int dthe_hash_update(struct ahash_request *req)
+{
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct dthe_tfm_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct dthe_hash_req_ctx *rctx = ahash_request_ctx(req);
+ struct dthe_data *dev_data = dthe_get_dev(ctx);
+ struct crypto_engine *engine = dev_data->hash_engine;
+
+ if (req->nbytes == 0)
+ return 0;
+
+ rctx->flags = DTHE_HASH_OP_UPDATE;
+
+ return crypto_transfer_hash_request_to_engine(engine, req);
+}
+
+static int dthe_hash_final(struct ahash_request *req)
+{
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct dthe_tfm_ctx *ctx = crypto_ahash_ctx(tfm);
+ struct dthe_hash_req_ctx *rctx = ahash_request_ctx(req);
+ struct dthe_data *dev_data = dthe_get_dev(ctx);
+ struct crypto_engine *engine = dev_data->hash_engine;
+
+ /**
+ * We are always buffering data in update, except when nbytes = 0.
+ * So, either we get the buffered data here (nbytes > 0) or
+ * it is the case that we got zero message to begin with
+ */
+ if (req->nbytes > 0) {
+ rctx->flags = DTHE_HASH_OP_FINUP;
+
+ return crypto_transfer_hash_request_to_engine(engine, req);
+ }
+
+ dthe_hash_write_zero_message(ctx->hash_mode, req->result);
+
+ return 0;
+}
+
+static int dthe_hash_finup(struct ahash_request *req)
+{
+ /* With AHASH_ALG_BLOCK_ONLY, final becomes same as finup. */
+ return dthe_hash_final(req);
+}
+
+static int dthe_hash_digest(struct ahash_request *req)
+{
+ dthe_hash_init(req);
+ return dthe_hash_finup(req);
+}
+
+static const u32 sha224_init_state[SHA256_DIGEST_SIZE / sizeof(u32)] = {
+ SHA224_H0, SHA224_H1, SHA224_H2, SHA224_H3,
+ SHA224_H4, SHA224_H5, SHA224_H6, SHA224_H7
+};
+
+static const u32 sha256_init_state[SHA256_DIGEST_SIZE / sizeof(u32)] = {
+ SHA256_H0, SHA256_H1, SHA256_H2, SHA256_H3,
+ SHA256_H4, SHA256_H5, SHA256_H6, SHA256_H7
+};
+
+static const u64 sha384_init_state[SHA512_DIGEST_SIZE / sizeof(u64)] = {
+ SHA384_H0, SHA384_H1, SHA384_H2, SHA384_H3,
+ SHA384_H4, SHA384_H5, SHA384_H6, SHA384_H7
+};
+
+static const u64 sha512_init_state[SHA512_DIGEST_SIZE / sizeof(u64)] = {
+ SHA512_H0, SHA512_H1, SHA512_H2, SHA512_H3,
+ SHA512_H4, SHA512_H5, SHA512_H6, SHA512_H7
+};
+
+static const void *dthe_hash_get_init_state(struct dthe_tfm_ctx *ctx)
+{
+ switch (ctx->hash_mode) {
+ case DTHE_HASH_SHA224:
+ return sha224_init_state;
+ case DTHE_HASH_SHA256:
+ return sha256_init_state;
+ case DTHE_HASH_SHA384:
+ return sha384_init_state;
+ case DTHE_HASH_SHA512:
+ return sha512_init_state;
+ default:
+ return NULL;
+ }
+}
+
+static int dthe_hash_export(struct ahash_request *req, void *out)
+{
+ struct dthe_hash_req_ctx *rctx = ahash_request_ctx(req);
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct dthe_tfm_ctx *ctx = crypto_ahash_ctx(tfm);
+ union {
+ u8 *u8;
+ u64 *u64;
+ } p = { .u8 = out };
+
+ if (rctx->phash_available)
+ memcpy(out, rctx->phash, ctx->phash_size);
+ else
+ memcpy(out, dthe_hash_get_init_state(ctx), ctx->phash_size);
+
+ p.u8 += ctx->phash_size;
+ put_unaligned(rctx->digestcnt[0], p.u64++);
+ if (ctx->phash_size >= SHA512_DIGEST_SIZE)
+ put_unaligned(rctx->digestcnt[1], p.u64++);
+
+ return 0;
+}
+
+static int dthe_hash_import(struct ahash_request *req, const void *in)
+{
+ struct dthe_hash_req_ctx *rctx = ahash_request_ctx(req);
+ struct crypto_ahash *tfm = crypto_ahash_reqtfm(req);
+ struct dthe_tfm_ctx *ctx = crypto_ahash_ctx(tfm);
+ union {
+ const u8 *u8;
+ const u64 *u64;
+ } p = { .u8 = in };
+
+ memcpy(rctx->phash, in, ctx->phash_size);
+ p.u8 += ctx->phash_size;
+ rctx->digestcnt[0] = get_unaligned(p.u64++);
+ if (ctx->phash_size >= SHA512_DIGEST_SIZE)
+ rctx->digestcnt[1] = get_unaligned(p.u64++);
+ rctx->phash_available = ((rctx->digestcnt[0]) ? 1 : 0);
+
+ return 0;
+}
+
+static struct ahash_engine_alg hash_algs[] = {
+ {
+ .base.init_tfm = dthe_hash_init_tfm,
+ .base.init = dthe_hash_init,
+ .base.update = dthe_hash_update,
+ .base.final = dthe_hash_final,
+ .base.finup = dthe_hash_finup,
+ .base.digest = dthe_hash_digest,
+ .base.export = dthe_hash_export,
+ .base.import = dthe_hash_import,
+ .base.halg = {
+ .digestsize = SHA512_DIGEST_SIZE,
+ .statesize = sizeof(struct dthe_hash_req_ctx),
+ .base = {
+ .cra_name = "sha512",
+ .cra_driver_name = "sha512-dthev2",
+ .cra_priority = 299,
+ .cra_flags = CRYPTO_ALG_TYPE_AHASH |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_OPTIONAL_KEY |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_ALLOCATES_MEMORY |
+ CRYPTO_AHASH_ALG_BLOCK_ONLY |
+ CRYPTO_AHASH_ALG_FINAL_NONZERO |
+ CRYPTO_AHASH_ALG_FINUP_MAX |
+ CRYPTO_AHASH_ALG_NO_EXPORT_CORE,
+ .cra_blocksize = SHA512_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct dthe_tfm_ctx),
+ .cra_reqsize = sizeof(struct dthe_hash_req_ctx),
+ .cra_module = THIS_MODULE,
+ }
+ },
+ .op.do_one_request = dthe_hash_run,
+ },
+ {
+ .base.init_tfm = dthe_hash_init_tfm,
+ .base.init = dthe_hash_init,
+ .base.update = dthe_hash_update,
+ .base.final = dthe_hash_final,
+ .base.finup = dthe_hash_finup,
+ .base.digest = dthe_hash_digest,
+ .base.export = dthe_hash_export,
+ .base.import = dthe_hash_import,
+ .base.halg = {
+ .digestsize = SHA384_DIGEST_SIZE,
+ .statesize = sizeof(struct dthe_hash_req_ctx),
+ .base = {
+ .cra_name = "sha384",
+ .cra_driver_name = "sha384-dthev2",
+ .cra_priority = 299,
+ .cra_flags = CRYPTO_ALG_TYPE_AHASH |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_OPTIONAL_KEY |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_ALLOCATES_MEMORY |
+ CRYPTO_AHASH_ALG_BLOCK_ONLY |
+ CRYPTO_AHASH_ALG_FINAL_NONZERO |
+ CRYPTO_AHASH_ALG_FINUP_MAX |
+ CRYPTO_AHASH_ALG_NO_EXPORT_CORE,
+ .cra_blocksize = SHA384_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct dthe_tfm_ctx),
+ .cra_reqsize = sizeof(struct dthe_hash_req_ctx),
+ .cra_module = THIS_MODULE,
+ }
+ },
+ .op.do_one_request = dthe_hash_run,
+ },
+ {
+ .base.init_tfm = dthe_hash_init_tfm,
+ .base.init = dthe_hash_init,
+ .base.update = dthe_hash_update,
+ .base.final = dthe_hash_final,
+ .base.finup = dthe_hash_finup,
+ .base.digest = dthe_hash_digest,
+ .base.export = dthe_hash_export,
+ .base.import = dthe_hash_import,
+ .base.halg = {
+ .digestsize = SHA256_DIGEST_SIZE,
+ .statesize = sizeof(struct dthe_hash_req_ctx),
+ .base = {
+ .cra_name = "sha256",
+ .cra_driver_name = "sha256-dthev2",
+ .cra_priority = 299,
+ .cra_flags = CRYPTO_ALG_TYPE_AHASH |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_OPTIONAL_KEY |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_ALLOCATES_MEMORY |
+ CRYPTO_AHASH_ALG_BLOCK_ONLY |
+ CRYPTO_AHASH_ALG_FINAL_NONZERO |
+ CRYPTO_AHASH_ALG_FINUP_MAX |
+ CRYPTO_AHASH_ALG_NO_EXPORT_CORE,
+ .cra_blocksize = SHA256_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct dthe_tfm_ctx),
+ .cra_reqsize = sizeof(struct dthe_hash_req_ctx),
+ .cra_module = THIS_MODULE,
+ }
+ },
+ .op.do_one_request = dthe_hash_run,
+ },
+ {
+ .base.init_tfm = dthe_hash_init_tfm,
+ .base.init = dthe_hash_init,
+ .base.update = dthe_hash_update,
+ .base.final = dthe_hash_final,
+ .base.finup = dthe_hash_finup,
+ .base.digest = dthe_hash_digest,
+ .base.export = dthe_hash_export,
+ .base.import = dthe_hash_import,
+ .base.halg = {
+ .digestsize = SHA224_DIGEST_SIZE,
+ .statesize = sizeof(struct dthe_hash_req_ctx),
+ .base = {
+ .cra_name = "sha224",
+ .cra_driver_name = "sha224-dthev2",
+ .cra_priority = 299,
+ .cra_flags = CRYPTO_ALG_TYPE_AHASH |
+ CRYPTO_ALG_ASYNC |
+ CRYPTO_ALG_OPTIONAL_KEY |
+ CRYPTO_ALG_KERN_DRIVER_ONLY |
+ CRYPTO_ALG_ALLOCATES_MEMORY |
+ CRYPTO_AHASH_ALG_BLOCK_ONLY |
+ CRYPTO_AHASH_ALG_FINAL_NONZERO |
+ CRYPTO_AHASH_ALG_FINUP_MAX |
+ CRYPTO_AHASH_ALG_NO_EXPORT_CORE,
+ .cra_blocksize = SHA224_BLOCK_SIZE,
+ .cra_ctxsize = sizeof(struct dthe_tfm_ctx),
+ .cra_reqsize = sizeof(struct dthe_hash_req_ctx),
+ .cra_module = THIS_MODULE,
+ }
+ },
+ .op.do_one_request = dthe_hash_run,
+ },
+};
+
+int dthe_register_hash_algs(void)
+{
+ return crypto_engine_register_ahashes(hash_algs, ARRAY_SIZE(hash_algs));
+}
+
+void dthe_unregister_hash_algs(void)
+{
+ crypto_engine_unregister_ahashes(hash_algs, ARRAY_SIZE(hash_algs));
+}
--
2.34.1
^ permalink raw reply related
* [PATCH v4 0/3] Add support for hashing algorithms in TI DTHE V2
From: T Pratham @ 2026-05-26 9:43 UTC (permalink / raw)
To: Herbert Xu, David S . Miller
Cc: T Pratham, linux-crypto, linux-kernel, Manorit Chawdhry,
Kamlesh Gurudasani, Shiva Tripathi, Kavitha Malarvizhi,
Vishal Mahaveer, Praneeth Bajjuri
DTHEv2 is a new cryptography engine introduced in TI AM62L SoC. The
features of DTHEv2 were detailed in [1]. Additional hardware details
available in SoC TRM [2]. DTHEv2 includes, among its various
sub-components, a hashing engine which is compliant with various IETF
and NIST standards, including FIPS 180-3 and FIPS PUB 198.
This patch series adds support for the hashing algorithms
SHA224/256/384/512 and MD5 as well as their HMAC counnterparts for the
hashing engine of Texas Instruments DTHE V2 crypto driver.
The driver is tested using full kernel crypto selftests
(CRYPTO_SELFTESTS_FULL) which all pass successfully [3].
Signed-off-by: T Pratham <t-pratham@ti.com>
---
[1]: [PATCH v7 0/2] Add support for Texas Instruments DTHEv2 Crypto Engine
Link: https://lore.kernel.org/all/20250820092710.3510788-1-t-pratham@ti.com/
[2]: Section 14.6.3 (DMA Control Registers -> DMASS_DTHE)
Link: https://www.ti.com/lit/ug/sprujb4/sprujb4.pdf
[3]: DTHEv2 kernel self-tests logs
Link: https://gist.github.com/Pratham-T/fd15b8e0ee815bcb420a60d451a0cf18
Changelog:
v4:
- Export initial state when phash_available = 0
v3:
- Check for error in registration individualy for hash and aes driver
algos.
- Fixed comment style in dthev2-common.c
v2:
- Completely revamped the driver to use the new CRYPTO_AHASH_BLOCK_ONLY
framework for handling partial blocks and crypto_engine for queueing.
- Added HMAC as well.
Link to previous versions:
v3: https://lore.kernel.org/all/20260226131103.3560884-1-t-pratham@ti.com/
v2: https://lore.kernel.org/all/20260121132408.743777-1-t-pratham@ti.com/
v1: https://lore.kernel.org/all/20250218104943.2304730-1-t-pratham@ti.com/
---
T Pratham (3):
crypto: ti - Add support for SHA224/256/384/512 in DTHEv2 driver
crypto: ti - Add support for MD5 in DTHEv2 Hashing Engine driver
crypto: ti - Add support for HMAC in DTHEv2 Hashing Engine driver
drivers/crypto/ti/Kconfig | 4 +
drivers/crypto/ti/Makefile | 2 +-
drivers/crypto/ti/dthev2-aes.c | 6 +-
drivers/crypto/ti/dthev2-common.c | 43 +-
drivers/crypto/ti/dthev2-common.h | 58 +-
drivers/crypto/ti/dthev2-hash.c | 1022 +++++++++++++++++++++++++++++
6 files changed, 1117 insertions(+), 18 deletions(-)
create mode 100644 drivers/crypto/ti/dthev2-hash.c
--
2.34.1
^ permalink raw reply
* [PATCH] crypto: qat - add KPT support for GEN6 devices
From: nitesh.venkatesh @ 2026-05-26 9:28 UTC (permalink / raw)
To: herbert
Cc: linux-crypto, qat-linux, Junyuan Wang, Nitesh Venkatesh,
Giovanni Cabiddu, Ahsan Atta
From: Junyuan Wang <junyuan.wang@intel.com>
Add support for Intel Key Protection Technology (KPT) on QAT GEN6
devices.
KPT protects private keys from exposure by keeping them wrapped
(encrypted) while in use, in-flight, and at rest. Keys remain in wrapped
form and are not exposed in plaintext in host memory. This feature
operates outside of the Linux crypto framework and kernel keyring.
Extend the firmware admin interface to enable and configure KPT. During
device initialisation, if KPT is enabled, the driver sends an admin
message to firmware to enable KPT mode and configure parameters such as
the maximum number of SWK (Symmetric Wrapping Key) slots and the SWK
time-to-live (TTL).
Expose KPT configuration via a new sysfs attribute group, "qat_kpt", and
add ABI documentation.
Co-developed-by: Nitesh Venkatesh <nitesh.venkatesh@intel.com>
Signed-off-by: Nitesh Venkatesh <nitesh.venkatesh@intel.com>
Signed-off-by: Junyuan Wang <junyuan.wang@intel.com>
Reviewed-by: Giovanni Cabiddu <giovanni.cabiddu@intel.com>
Reviewed-by: Ahsan Atta <ahsan.atta@intel.com>
---
.../ABI/testing/sysfs-driver-qat_kpt | 97 ++++++
.../intel/qat/qat_6xxx/adf_6xxx_hw_data.c | 21 +-
.../intel/qat/qat_6xxx/adf_6xxx_hw_data.h | 9 +
drivers/crypto/intel/qat/qat_6xxx/adf_drv.c | 6 +
drivers/crypto/intel/qat/qat_common/Makefile | 2 +
.../intel/qat/qat_common/adf_accel_devices.h | 2 +
.../crypto/intel/qat/qat_common/adf_admin.c | 39 +++
.../crypto/intel/qat/qat_common/adf_admin.h | 2 +
.../crypto/intel/qat/qat_common/adf_init.c | 8 +
drivers/crypto/intel/qat/qat_common/adf_kpt.c | 56 ++++
drivers/crypto/intel/qat/qat_common/adf_kpt.h | 29 ++
.../intel/qat/qat_common/adf_sysfs_kpt.c | 296 ++++++++++++++++++
.../intel/qat/qat_common/adf_sysfs_kpt.h | 10 +
.../qat/qat_common/icp_qat_fw_init_admin.h | 8 +
.../crypto/intel/qat/qat_common/icp_qat_hw.h | 3 +-
15 files changed, 586 insertions(+), 2 deletions(-)
create mode 100644 Documentation/ABI/testing/sysfs-driver-qat_kpt
create mode 100644 drivers/crypto/intel/qat/qat_common/adf_kpt.c
create mode 100644 drivers/crypto/intel/qat/qat_common/adf_kpt.h
create mode 100644 drivers/crypto/intel/qat/qat_common/adf_sysfs_kpt.c
create mode 100644 drivers/crypto/intel/qat/qat_common/adf_sysfs_kpt.h
diff --git a/Documentation/ABI/testing/sysfs-driver-qat_kpt b/Documentation/ABI/testing/sysfs-driver-qat_kpt
new file mode 100644
index 000000000000..c6480ea1fa4f
--- /dev/null
+++ b/Documentation/ABI/testing/sysfs-driver-qat_kpt
@@ -0,0 +1,97 @@
+What: /sys/bus/pci/devices/<BDF>/qat_kpt/
+Date: August 2026
+KernelVersion: 7.2
+Contact: qat-linux@intel.com
+Description:
+ Directory containing attributes related to the QAT Key Protection
+ Technology (KPT) feature. KPT allows cryptographic keys to be used
+ by the accelerator without being exposed in plaintext to the host.
+
+What: /sys/bus/pci/devices/<BDF>/qat_kpt/enable
+Date: August 2026
+KernelVersion: 7.2
+Contact: qat-linux@intel.com
+Description:
+ (RW) Enables or disables Key Protection Technology (KPT).
+
+ Write 1 to enable KPT, or 0 to disable it.
+
+ Example usage::
+
+ # cat /sys/bus/pci/devices/<BDF>/qat_kpt/enable
+ 0
+ # echo 1 > /sys/bus/pci/devices/<BDF>/qat_kpt/enable
+
+ This attribute is only available on devices that support KPT.
+
+What: /sys/bus/pci/devices/<BDF>/qat_kpt/swk_cnt_per_fn
+Date: August 2026
+KernelVersion: 7.2
+Contact: qat-linux@intel.com
+Description:
+ (RW) Configures the maximum number of KPT symmetric wrapping keys
+ (SWKs) that a Virtual Function (VF) may be associated with.
+
+ Valid values range from 0 to 128. A value of 0 indicates no limit.
+
+ Example usage::
+
+ # cat /sys/bus/pci/devices/<BDF>/qat_kpt/swk_cnt_per_fn
+ 128
+ # echo 128 > /sys/bus/pci/devices/<BDF>/qat_kpt/swk_cnt_per_fn
+
+ This attribute is only available on devices that support KPT.
+
+What: /sys/bus/pci/devices/<BDF>/qat_kpt/swk_cnt_per_pasid
+Date: August 2026
+KernelVersion: 7.2
+Contact: qat-linux@intel.com
+Description:
+ (RW) Configures the maximum number of KPT symmetric wrapping keys
+ (SWKs) per Process Address Space ID (PASID).
+
+ Valid values range from 0 to 128. A value of 0 indicates no limit.
+
+ Example usage::
+
+ # cat /sys/bus/pci/devices/<BDF>/qat_kpt/swk_cnt_per_pasid
+ 128
+ # echo 128 > /sys/bus/pci/devices/<BDF>/qat_kpt/swk_cnt_per_pasid
+
+ This attribute is only available on devices that support KPT.
+
+What: /sys/bus/pci/devices/<BDF>/qat_kpt/swk_max_ttl
+Date: August 2026
+KernelVersion: 7.2
+Contact: qat-linux@intel.com
+Description:
+ (RW) Configures the maximum Time To Live (TTL) for KPT symmetric
+ wrapping keys (SWK).
+
+ Valid values range from 0 to 31536000 seconds. A value of 0
+ indicates that the SWK TTL is unlimited.
+
+ Example usage::
+
+ # cat /sys/bus/pci/devices/<BDF>/qat_kpt/swk_max_ttl
+ 1000
+ # echo 1000 > /sys/bus/pci/devices/<BDF>/qat_kpt/swk_max_ttl
+
+ This attribute is only available on devices that support KPT.
+
+What: /sys/bus/pci/devices/<BDF>/qat_kpt/swk_shared
+Date: August 2026
+KernelVersion: 7.2
+Contact: qat-linux@intel.com
+Description:
+ (RW) Controls shared mode for KPT symmetric wrapping keys (SWK).
+
+ Write 1 to enable shared mode, or 0 to disable it (non-shared mode).
+
+ Example usage::
+
+ # cat /sys/bus/pci/devices/<BDF>/qat_kpt/swk_shared
+ 0
+ # echo 1 > /sys/bus/pci/devices/<BDF>/qat_kpt/swk_shared
+
+ This attribute is only available on devices that support KPT.
diff --git a/drivers/crypto/intel/qat/qat_6xxx/adf_6xxx_hw_data.c b/drivers/crypto/intel/qat/qat_6xxx/adf_6xxx_hw_data.c
index 205680797e2c..388087e64540 100644
--- a/drivers/crypto/intel/qat/qat_6xxx/adf_6xxx_hw_data.c
+++ b/drivers/crypto/intel/qat/qat_6xxx/adf_6xxx_hw_data.c
@@ -752,10 +752,12 @@ static u32 get_accel_cap(struct adf_accel_dev *accel_dev)
unsigned long mask;
u32 caps = 0;
u32 fusectl1;
+ u32 fusectl0;
if (adf_6xxx_is_wcy(GET_HW_DATA(accel_dev)))
return get_accel_cap_wcy(accel_dev);
+ fusectl0 = GET_HW_DATA(accel_dev)->fuses[ADF_FUSECTL0];
fusectl1 = GET_HW_DATA(accel_dev)->fuses[ADF_FUSECTL1];
/* Read accelerator capabilities mask */
@@ -785,7 +787,8 @@ static u32 get_accel_cap(struct adf_accel_dev *accel_dev)
capabilities_asym = ICP_ACCEL_CAPABILITIES_CRYPTO_ASYMMETRIC |
ICP_ACCEL_CAPABILITIES_SM2 |
- ICP_ACCEL_CAPABILITIES_ECEDMONT;
+ ICP_ACCEL_CAPABILITIES_ECEDMONT |
+ ICP_ACCEL_CAPABILITIES_KPT;
if (fusectl1 & ICP_ACCEL_GEN6_MASK_PKE_SLICE) {
capabilities_asym &= ~ICP_ACCEL_CAPABILITIES_CRYPTO_ASYMMETRIC;
@@ -793,6 +796,9 @@ static u32 get_accel_cap(struct adf_accel_dev *accel_dev)
capabilities_asym &= ~ICP_ACCEL_CAPABILITIES_ECEDMONT;
}
+ if (fusectl0 & ADF_GEN6_KPT_FUSE_BIT)
+ capabilities_asym &= ~ICP_ACCEL_CAPABILITIES_KPT;
+
capabilities_dc = ICP_ACCEL_CAPABILITIES_COMPRESSION |
ICP_ACCEL_CAPABILITIES_LZ4_COMPRESSION |
ICP_ACCEL_CAPABILITIES_LZ4S_COMPRESSION |
@@ -960,6 +966,18 @@ static int dev_config(struct adf_accel_dev *accel_dev)
return ret;
}
+static void adf_gen6_init_kpt(struct adf_kpt_hw_data *kpt_data)
+{
+ kpt_data->max_swk_cnt_per_fn_pasid = ADF_6XXX_KPT_MAX_SWK_COUNT_PER_FNPASID;
+ kpt_data->max_swk_ttl = ADF_6XXX_KPT_MAX_SWK_TTL;
+
+ kpt_data->user_input.enable = false;
+ kpt_data->user_input.swk_shared = ADF_6XXX_KPT_DEFAULT_SWK_SHARED_MODE;
+ kpt_data->user_input.swk_max_ttl = ADF_6XXX_KPT_DEFAULT_SWK_TTL;
+ kpt_data->user_input.swk_cnt_per_fn = ADF_6XXX_KPT_DEFAULT_SWK_CNT_PER_FN;
+ kpt_data->user_input.swk_cnt_per_pasid = ADF_6XXX_KPT_DEFAULT_SWK_CNT_PER_PASID;
+}
+
static void adf_gen6_init_rl_data(struct adf_rl_hw_data *rl_data)
{
rl_data->pciout_tb_offset = ADF_GEN6_RL_TOKEN_PCIEOUT_BUCKET_OFFSET;
@@ -1057,6 +1075,7 @@ void adf_init_hw_data_6xxx(struct adf_hw_device_data *hw_data)
adf_gen6_init_tl_data(&hw_data->tl_data);
adf_gen6_init_rl_data(&hw_data->rl_data);
adf_gen6_init_anti_rb(&hw_data->anti_rb_data);
+ adf_gen6_init_kpt(&hw_data->kpt_data);
}
void adf_clean_hw_data_6xxx(struct adf_hw_device_data *hw_data)
diff --git a/drivers/crypto/intel/qat/qat_6xxx/adf_6xxx_hw_data.h b/drivers/crypto/intel/qat/qat_6xxx/adf_6xxx_hw_data.h
index e4d433bdd379..2719ddbc1e05 100644
--- a/drivers/crypto/intel/qat/qat_6xxx/adf_6xxx_hw_data.h
+++ b/drivers/crypto/intel/qat/qat_6xxx/adf_6xxx_hw_data.h
@@ -58,6 +58,7 @@
/* Fuse bits */
#define ADF_GEN6_ANTI_RB_FUSE_BIT BIT(24)
+#define ADF_GEN6_KPT_FUSE_BIT BIT(16)
/*
* Watchdog timers
@@ -164,6 +165,14 @@
/* Clock frequency */
#define ADF_6XXX_AE_FREQ (1000 * HZ_PER_MHZ)
+/* KPT */
+#define ADF_6XXX_KPT_MAX_SWK_COUNT_PER_FNPASID 128
+#define ADF_6XXX_KPT_MAX_SWK_TTL 31536000
+#define ADF_6XXX_KPT_DEFAULT_SWK_SHARED_MODE 1
+#define ADF_6XXX_KPT_DEFAULT_SWK_TTL 0
+#define ADF_6XXX_KPT_DEFAULT_SWK_CNT_PER_FN 0
+#define ADF_6XXX_KPT_DEFAULT_SWK_CNT_PER_PASID 0
+
enum icp_qat_gen6_slice_mask {
ICP_ACCEL_GEN6_MASK_UCS_SLICE = BIT(0),
ICP_ACCEL_GEN6_MASK_AUTH_SLICE = BIT(1),
diff --git a/drivers/crypto/intel/qat/qat_6xxx/adf_drv.c b/drivers/crypto/intel/qat/qat_6xxx/adf_drv.c
index ab62b91ecb51..319b4251ad46 100644
--- a/drivers/crypto/intel/qat/qat_6xxx/adf_drv.c
+++ b/drivers/crypto/intel/qat/qat_6xxx/adf_drv.c
@@ -13,6 +13,7 @@
#include <adf_cfg.h>
#include <adf_common_drv.h>
#include <adf_dbgfs.h>
+#include <adf_sysfs_kpt.h>
#include "adf_gen6_shared.h"
#include "adf_6xxx_hw_data.h"
@@ -217,6 +218,11 @@ static int adf_probe(struct pci_dev *pdev, const struct pci_device_id *ent)
return ret;
ret = adf_sysfs_init(accel_dev);
+ if (ret)
+ return ret;
+
+ if (!(hw_data->fuses[ADF_FUSECTL0] & ADF_GEN6_KPT_FUSE_BIT))
+ ret = adf_sysfs_init_kpt(accel_dev);
return ret;
}
diff --git a/drivers/crypto/intel/qat/qat_common/Makefile b/drivers/crypto/intel/qat/qat_common/Makefile
index 2b1649001518..306ec71bb6fa 100644
--- a/drivers/crypto/intel/qat/qat_common/Makefile
+++ b/drivers/crypto/intel/qat/qat_common/Makefile
@@ -25,12 +25,14 @@ intel_qat-y := adf_accel_engine.o \
adf_hw_arbiter.o \
adf_init.o \
adf_isr.o \
+ adf_kpt.o \
adf_module.o \
adf_mstate_mgr.o \
adf_rl_admin.o \
adf_rl.o \
adf_sysfs.o \
adf_sysfs_anti_rb.o \
+ adf_sysfs_kpt.o \
adf_sysfs_ras_counters.o \
adf_sysfs_rl.o \
adf_timer.o \
diff --git a/drivers/crypto/intel/qat/qat_common/adf_accel_devices.h b/drivers/crypto/intel/qat/qat_common/adf_accel_devices.h
index d9b2a1cf474e..f2d70641c377 100644
--- a/drivers/crypto/intel/qat/qat_common/adf_accel_devices.h
+++ b/drivers/crypto/intel/qat/qat_common/adf_accel_devices.h
@@ -14,6 +14,7 @@
#include "adf_anti_rb.h"
#include "adf_cfg_common.h"
#include "adf_dc.h"
+#include "adf_kpt.h"
#include "adf_rl.h"
#include "adf_telemetry.h"
#include "adf_pfvf_msg.h"
@@ -335,6 +336,7 @@ struct adf_hw_device_data {
struct adf_rl_hw_data rl_data;
struct adf_tl_hw_data tl_data;
struct adf_anti_rb_hw_data anti_rb_data;
+ struct adf_kpt_hw_data kpt_data;
struct qat_migdev_ops vfmig_ops;
const char *fw_name;
const char *fw_mmp_name;
diff --git a/drivers/crypto/intel/qat/qat_common/adf_admin.c b/drivers/crypto/intel/qat/qat_common/adf_admin.c
index 841aa802c79e..1eea74c2a8a5 100644
--- a/drivers/crypto/intel/qat/qat_common/adf_admin.c
+++ b/drivers/crypto/intel/qat/qat_common/adf_admin.c
@@ -13,6 +13,7 @@
#include "adf_common_drv.h"
#include "adf_cfg.h"
#include "adf_heartbeat.h"
+#include "adf_kpt.h"
#include "icp_qat_fw_init_admin.h"
#define ADF_ADMIN_MAILBOX_STRIDE 0x1000
@@ -606,6 +607,44 @@ int adf_send_admin_arb_commit(struct adf_accel_dev *accel_dev)
return adf_send_admin_svn(accel_dev, ICP_QAT_FW_SVN_COMMIT, &resp);
}
+static_assert(sizeof(struct icp_qat_fw_init_admin_kpt_cfg) < PAGE_SIZE);
+
+static void adf_cfg_kpt_config(struct adf_accel_dev *accel_dev,
+ struct icp_qat_fw_init_admin_kpt_cfg *kpt_config)
+{
+ struct adf_kpt_interface_data *user_data = GET_KPT_USER_DATA(accel_dev);
+
+ kpt_config->swk_cnt_per_fn = user_data->swk_cnt_per_fn;
+ kpt_config->swk_cnt_per_pasid = user_data->swk_cnt_per_pasid;
+ kpt_config->swk_ttl_in_secs = user_data->swk_max_ttl;
+ kpt_config->swk_shared_disable = !user_data->swk_shared;
+}
+
+int adf_send_admin_kpt_init(struct adf_accel_dev *accel_dev, void *init_cfg,
+ size_t init_cfg_sz, dma_addr_t init_ptr)
+{
+ struct icp_qat_fw_init_admin_kpt_cfg *kpt_config = init_cfg;
+ u32 ae_mask = GET_HW_DATA(accel_dev)->admin_ae_mask;
+ struct icp_qat_fw_init_admin_resp resp = { };
+ struct icp_qat_fw_init_admin_req req = { };
+ int ret;
+
+ if (!kpt_config || init_cfg_sz < sizeof(*kpt_config))
+ return -EINVAL;
+
+ adf_cfg_kpt_config(accel_dev, kpt_config);
+
+ req.cmd_id = ICP_QAT_FW_KPT_ENABLE;
+ req.init_cfg_ptr = init_ptr;
+ req.init_cfg_sz = sizeof(*kpt_config);
+
+ ret = adf_send_admin(accel_dev, &req, &resp, ae_mask);
+ if (ret)
+ dev_err(&GET_DEV(accel_dev), "Failed to send KPT init admin message\n");
+
+ return ret;
+}
+
int adf_init_admin_comms(struct adf_accel_dev *accel_dev)
{
struct adf_admin_comms *admin;
diff --git a/drivers/crypto/intel/qat/qat_common/adf_admin.h b/drivers/crypto/intel/qat/qat_common/adf_admin.h
index 9704219f2eb7..44eb6dc92e45 100644
--- a/drivers/crypto/intel/qat/qat_common/adf_admin.h
+++ b/drivers/crypto/intel/qat/qat_common/adf_admin.h
@@ -29,5 +29,7 @@ int adf_send_admin_tl_start(struct adf_accel_dev *accel_dev,
int adf_send_admin_tl_stop(struct adf_accel_dev *accel_dev);
int adf_send_admin_arb_query(struct adf_accel_dev *accel_dev, int cmd, u8 *svn);
int adf_send_admin_arb_commit(struct adf_accel_dev *accel_dev);
+int adf_send_admin_kpt_init(struct adf_accel_dev *accel_dev, void *init_cfg,
+ size_t init_cfg_sz, dma_addr_t init_ptr);
#endif
diff --git a/drivers/crypto/intel/qat/qat_common/adf_init.c b/drivers/crypto/intel/qat/qat_common/adf_init.c
index 1c7f9e49914d..3e39c53814de 100644
--- a/drivers/crypto/intel/qat/qat_common/adf_init.c
+++ b/drivers/crypto/intel/qat/qat_common/adf_init.c
@@ -10,6 +10,7 @@
#include "adf_dbgfs.h"
#include "adf_heartbeat.h"
#include "adf_rl.h"
+#include "adf_kpt.h"
#include "adf_sysfs_anti_rb.h"
#include "adf_sysfs_ras_counters.h"
#include "adf_telemetry.h"
@@ -219,6 +220,13 @@ static int adf_dev_start(struct adf_accel_dev *accel_dev)
return -EFAULT;
}
+ /* Enable Key Protection Technology (KPT) */
+ ret = adf_enable_kpt(accel_dev);
+ if (ret) {
+ dev_err(&GET_DEV(accel_dev), "Failed to enable KPT\n");
+ return ret;
+ }
+
if (hw_data->start_timer) {
ret = hw_data->start_timer(accel_dev);
if (ret) {
diff --git a/drivers/crypto/intel/qat/qat_common/adf_kpt.c b/drivers/crypto/intel/qat/qat_common/adf_kpt.c
new file mode 100644
index 000000000000..a0430141ea0e
--- /dev/null
+++ b/drivers/crypto/intel/qat/qat_common/adf_kpt.c
@@ -0,0 +1,56 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2026 Intel Corporation */
+#include <linux/dma-mapping.h>
+
+#include "adf_admin.h"
+#include "adf_cfg_services.h"
+#include "adf_common_drv.h"
+#include "adf_kpt.h"
+
+static bool adf_kpt_supported(struct adf_accel_dev *accel_dev)
+{
+ struct adf_hw_device_data *hw_data = GET_HW_DATA(accel_dev);
+
+ return hw_data->accel_capabilities_mask & ICP_ACCEL_CAPABILITIES_KPT;
+}
+
+int adf_enable_kpt(struct adf_accel_dev *accel_dev)
+{
+ struct adf_kpt_interface_data *user_data = GET_KPT_USER_DATA(accel_dev);
+ struct adf_hw_device_data *hw_data = GET_HW_DATA(accel_dev);
+ dma_addr_t paddr;
+ void *vaddr;
+ int ret;
+ int svc;
+
+ /* Return 0 if KPT is not supported by the hardware */
+ if (!adf_kpt_supported(accel_dev))
+ return 0;
+
+ if (!user_data->enable) {
+ /* Disable KPT capability if user has not enabled it */
+ hw_data->accel_capabilities_mask &= ~ICP_ACCEL_CAPABILITIES_KPT;
+ return 0;
+ }
+
+ svc = adf_get_service_enabled(accel_dev);
+ if (svc < 0)
+ return svc;
+
+ if (svc != SVC_ASYM) {
+ dev_err(&GET_DEV(accel_dev),
+ "KPT can only be enabled when service is configured as 'asym'\n");
+ return -EINVAL;
+ }
+
+ vaddr = dma_alloc_coherent(&GET_DEV(accel_dev), PAGE_SIZE, &paddr,
+ GFP_KERNEL);
+ if (!vaddr)
+ return -ENOMEM;
+
+ ret = adf_send_admin_kpt_init(accel_dev, vaddr, PAGE_SIZE, paddr);
+
+ dma_free_coherent(&GET_DEV(accel_dev), PAGE_SIZE, vaddr, paddr);
+
+ return ret;
+}
diff --git a/drivers/crypto/intel/qat/qat_common/adf_kpt.h b/drivers/crypto/intel/qat/qat_common/adf_kpt.h
new file mode 100644
index 000000000000..17d07d24a319
--- /dev/null
+++ b/drivers/crypto/intel/qat/qat_common/adf_kpt.h
@@ -0,0 +1,29 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2026 Intel Corporation */
+#ifndef ADF_KPT_H_
+#define ADF_KPT_H_
+
+#include <linux/types.h>
+
+#define GET_KPT_CFG_DATA(accel_dev) (&(accel_dev)->hw_device->kpt_data)
+#define GET_KPT_USER_DATA(accel_dev) (&(accel_dev)->hw_device->kpt_data.user_input)
+
+struct adf_accel_dev;
+
+struct adf_kpt_interface_data {
+ bool enable;
+ bool swk_shared;
+ unsigned int swk_cnt_per_fn;
+ unsigned int swk_cnt_per_pasid;
+ unsigned int swk_max_ttl;
+};
+
+struct adf_kpt_hw_data {
+ unsigned int max_swk_cnt_per_fn_pasid;
+ unsigned int max_swk_ttl;
+ struct adf_kpt_interface_data user_input;
+};
+
+int adf_enable_kpt(struct adf_accel_dev *accel_dev);
+
+#endif /* ADF_KPT_H_ */
diff --git a/drivers/crypto/intel/qat/qat_common/adf_sysfs_kpt.c b/drivers/crypto/intel/qat/qat_common/adf_sysfs_kpt.c
new file mode 100644
index 000000000000..1f733ae80bdf
--- /dev/null
+++ b/drivers/crypto/intel/qat/qat_common/adf_sysfs_kpt.c
@@ -0,0 +1,296 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/* Copyright(c) 2026 Intel Corporation */
+#include <linux/sysfs.h>
+#include <linux/types.h>
+
+#include "adf_cfg.h"
+#include "adf_cfg_services.h"
+#include "adf_common_drv.h"
+#include "adf_kpt.h"
+#include "adf_sysfs_kpt.h"
+
+static ssize_t enable_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct adf_kpt_interface_data *user_data;
+ struct adf_accel_dev *accel_dev;
+
+ accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
+ if (!accel_dev)
+ return -EINVAL;
+
+ user_data = GET_KPT_USER_DATA(accel_dev);
+
+ return sysfs_emit(buf, "%d\n", user_data->enable);
+}
+
+static ssize_t enable_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct adf_kpt_interface_data *user_data;
+ struct adf_hw_device_data *hw_data;
+ struct adf_accel_dev *accel_dev;
+ bool enable;
+ int ret;
+
+ accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
+ if (!accel_dev)
+ return -EINVAL;
+
+ if (adf_dev_started(accel_dev)) {
+ dev_info(dev, "Device qat_dev%d must be down before enabling KPT\n",
+ accel_dev->accel_id);
+ return -EINVAL;
+ }
+
+ if (adf_get_service_enabled(accel_dev) != SVC_ASYM) {
+ dev_info(dev, "KPT can only be enabled when the asymmetric service is enabled\n");
+ return -EINVAL;
+ }
+
+ hw_data = GET_HW_DATA(accel_dev);
+
+ /*
+ * Restore the KPT capability bit in the device's capabilities mask
+ * before processing user input, as the bit may have been cleared if
+ * KPT was previously disabled by the user.
+ */
+ hw_data->accel_capabilities_mask = hw_data->get_accel_cap(accel_dev);
+ if (!hw_data->accel_capabilities_mask)
+ return -EINVAL;
+
+ ret = kstrtobool(buf, &enable);
+ if (ret)
+ return ret;
+
+ user_data = GET_KPT_USER_DATA(accel_dev);
+ user_data->enable = enable;
+
+ return count;
+}
+static DEVICE_ATTR_RW(enable);
+
+static ssize_t swk_shared_show(struct device *dev,
+ struct device_attribute *attr, char *buf)
+{
+ struct adf_kpt_interface_data *user_data;
+ struct adf_accel_dev *accel_dev;
+
+ accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
+ if (!accel_dev)
+ return -EINVAL;
+
+ user_data = GET_KPT_USER_DATA(accel_dev);
+
+ return sysfs_emit(buf, "%d\n", user_data->swk_shared);
+}
+
+static ssize_t swk_shared_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct adf_kpt_interface_data *user_data;
+ struct adf_accel_dev *accel_dev;
+ bool swk_shared;
+ int ret;
+
+ accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
+ if (!accel_dev)
+ return -EINVAL;
+
+ if (adf_dev_started(accel_dev)) {
+ dev_info(dev, "Device qat_dev%d must be down before setting swk_shared\n",
+ accel_dev->accel_id);
+ return -EINVAL;
+ }
+
+ ret = kstrtobool(buf, &swk_shared);
+ if (ret)
+ return ret;
+
+ user_data = GET_KPT_USER_DATA(accel_dev);
+ user_data->swk_shared = swk_shared;
+
+ return count;
+}
+static DEVICE_ATTR_RW(swk_shared);
+
+static ssize_t swk_max_ttl_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct adf_kpt_interface_data *user_data;
+ struct adf_accel_dev *accel_dev;
+
+ accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
+ if (!accel_dev)
+ return -EINVAL;
+
+ user_data = GET_KPT_USER_DATA(accel_dev);
+
+ return sysfs_emit(buf, "%u\n", user_data->swk_max_ttl);
+}
+
+static ssize_t swk_max_ttl_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct adf_kpt_hw_data *kpt_data;
+ struct adf_accel_dev *accel_dev;
+ unsigned int swk_max_ttl;
+ int ret;
+
+ accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
+ if (!accel_dev)
+ return -EINVAL;
+
+ if (adf_dev_started(accel_dev)) {
+ dev_info(dev, "Device qat_dev%d must be down before setting swk_max_ttl\n",
+ accel_dev->accel_id);
+ return -EINVAL;
+ }
+
+ ret = kstrtouint(buf, 10, &swk_max_ttl);
+ if (ret)
+ return ret;
+
+ kpt_data = GET_KPT_CFG_DATA(accel_dev);
+
+ if (swk_max_ttl > kpt_data->max_swk_ttl) {
+ dev_info(dev, "Configuration value is out of range (%u - %u)\n",
+ 0, kpt_data->max_swk_ttl);
+ return -EINVAL;
+ }
+
+ kpt_data->user_input.swk_max_ttl = swk_max_ttl;
+
+ return count;
+}
+static DEVICE_ATTR_RW(swk_max_ttl);
+
+static ssize_t swk_cnt_per_fn_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct adf_kpt_interface_data *user_data;
+ struct adf_accel_dev *accel_dev;
+
+ accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
+ if (!accel_dev)
+ return -EINVAL;
+
+ user_data = GET_KPT_USER_DATA(accel_dev);
+
+ return sysfs_emit(buf, "%u\n", user_data->swk_cnt_per_fn);
+}
+
+static ssize_t swk_cnt_per_fn_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct adf_kpt_hw_data *kpt_data;
+ struct adf_accel_dev *accel_dev;
+ unsigned int swk_cnt_per_fn;
+ int ret;
+
+ accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
+ if (!accel_dev)
+ return -EINVAL;
+
+ if (adf_dev_started(accel_dev)) {
+ dev_info(dev, "Device qat_dev%d must be down before setting swk_cnt_per_fn\n",
+ accel_dev->accel_id);
+ return -EINVAL;
+ }
+
+ ret = kstrtouint(buf, 10, &swk_cnt_per_fn);
+ if (ret)
+ return ret;
+
+ kpt_data = GET_KPT_CFG_DATA(accel_dev);
+
+ if (swk_cnt_per_fn > kpt_data->max_swk_cnt_per_fn_pasid) {
+ dev_info(dev, "swk_cnt_per_fn: value out of range (0 - %u)\n",
+ kpt_data->max_swk_cnt_per_fn_pasid);
+ return -EINVAL;
+ }
+
+ kpt_data->user_input.swk_cnt_per_fn = swk_cnt_per_fn;
+
+ return count;
+}
+static DEVICE_ATTR_RW(swk_cnt_per_fn);
+
+static ssize_t swk_cnt_per_pasid_show(struct device *dev, struct device_attribute *attr,
+ char *buf)
+{
+ struct adf_kpt_interface_data *user_data;
+ struct adf_accel_dev *accel_dev;
+
+ accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
+ if (!accel_dev)
+ return -EINVAL;
+
+ user_data = GET_KPT_USER_DATA(accel_dev);
+
+ return sysfs_emit(buf, "%u\n", user_data->swk_cnt_per_pasid);
+}
+
+static ssize_t swk_cnt_per_pasid_store(struct device *dev, struct device_attribute *attr,
+ const char *buf, size_t count)
+{
+ struct adf_kpt_hw_data *kpt_data;
+ struct adf_accel_dev *accel_dev;
+ unsigned int swk_cnt_per_pasid;
+ int ret;
+
+ accel_dev = adf_devmgr_pci_to_accel_dev(to_pci_dev(dev));
+ if (!accel_dev)
+ return -EINVAL;
+
+ if (adf_dev_started(accel_dev)) {
+ dev_info(dev, "Device qat_dev%d must be down before setting swk_cnt_per_pasid\n",
+ accel_dev->accel_id);
+ return -EINVAL;
+ }
+
+ ret = kstrtouint(buf, 10, &swk_cnt_per_pasid);
+ if (ret)
+ return ret;
+
+ kpt_data = GET_KPT_CFG_DATA(accel_dev);
+
+ if (swk_cnt_per_pasid > kpt_data->max_swk_cnt_per_fn_pasid) {
+ dev_info(dev, "swk_cnt_per_pasid: value out of range (0 - %u)\n",
+ kpt_data->max_swk_cnt_per_fn_pasid);
+ return -EINVAL;
+ }
+
+ kpt_data->user_input.swk_cnt_per_pasid = swk_cnt_per_pasid;
+
+ return count;
+}
+static DEVICE_ATTR_RW(swk_cnt_per_pasid);
+
+static struct attribute *qat_kpt_attrs[] = {
+ &dev_attr_enable.attr,
+ &dev_attr_swk_shared.attr,
+ &dev_attr_swk_max_ttl.attr,
+ &dev_attr_swk_cnt_per_fn.attr,
+ &dev_attr_swk_cnt_per_pasid.attr,
+ NULL,
+};
+
+static const struct attribute_group qat_kpt_group = {
+ .attrs = qat_kpt_attrs,
+ .name = "qat_kpt",
+};
+
+int adf_sysfs_init_kpt(struct adf_accel_dev *accel_dev)
+{
+ int ret;
+
+ ret = devm_device_add_group(&GET_DEV(accel_dev), &qat_kpt_group);
+ if (ret) {
+ dev_err(&GET_DEV(accel_dev), "Failed to create qat_kpt attribute group\n");
+ return ret;
+ }
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(adf_sysfs_init_kpt);
diff --git a/drivers/crypto/intel/qat/qat_common/adf_sysfs_kpt.h b/drivers/crypto/intel/qat/qat_common/adf_sysfs_kpt.h
new file mode 100644
index 000000000000..d9d065f75114
--- /dev/null
+++ b/drivers/crypto/intel/qat/qat_common/adf_sysfs_kpt.h
@@ -0,0 +1,10 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/* Copyright(c) 2026 Intel Corporation */
+#ifndef ADF_SYSFS_KPT_H_
+#define ADF_SYSFS_KPT_H_
+
+struct adf_accel_dev;
+
+int adf_sysfs_init_kpt(struct adf_accel_dev *accel_dev);
+
+#endif /* ADF_SYSFS_KPT_H_ */
diff --git a/drivers/crypto/intel/qat/qat_common/icp_qat_fw_init_admin.h b/drivers/crypto/intel/qat/qat_common/icp_qat_fw_init_admin.h
index 6b0f0d100cb9..95f7f514cb63 100644
--- a/drivers/crypto/intel/qat/qat_common/icp_qat_fw_init_admin.h
+++ b/drivers/crypto/intel/qat/qat_common/icp_qat_fw_init_admin.h
@@ -31,6 +31,7 @@ enum icp_qat_fw_init_admin_cmd_id {
ICP_QAT_FW_RL_REMOVE = 136,
ICP_QAT_FW_TL_START = 137,
ICP_QAT_FW_TL_STOP = 138,
+ ICP_QAT_FW_KPT_ENABLE = 144,
ICP_QAT_FW_SVN_READ = 146,
ICP_QAT_FW_SVN_COMMIT = 147,
};
@@ -212,4 +213,11 @@ struct icp_qat_fw_init_admin_pm_info {
__u32 resvrd3[6];
};
+struct icp_qat_fw_init_admin_kpt_cfg {
+ __u32 swk_cnt_per_fn;
+ __u32 swk_cnt_per_pasid;
+ __u32 swk_ttl_in_secs;
+ __u32 swk_shared_disable;
+};
+
#endif
diff --git a/drivers/crypto/intel/qat/qat_common/icp_qat_hw.h b/drivers/crypto/intel/qat/qat_common/icp_qat_hw.h
index 16ef6d98fa42..e38115d3cd75 100644
--- a/drivers/crypto/intel/qat/qat_common/icp_qat_hw.h
+++ b/drivers/crypto/intel/qat/qat_common/icp_qat_hw.h
@@ -114,7 +114,8 @@ enum icp_qat_capabilities_mask {
ICP_ACCEL_CAPABILITIES_LZ4_COMPRESSION = BIT(24),
ICP_ACCEL_CAPABILITIES_LZ4S_COMPRESSION = BIT(25),
ICP_ACCEL_CAPABILITIES_AES_V2 = BIT(26),
- /* Bits 27-28 are currently reserved */
+ ICP_ACCEL_CAPABILITIES_KPT = BIT(27),
+ /* Bit 28 is currently reserved */
ICP_ACCEL_CAPABILITIES_ZUC_256 = BIT(29),
ICP_ACCEL_CAPABILITIES_WIRELESS_CRYPTO_EXT = BIT(30),
};
base-commit: 3797750c3b271f24f98cad54c2932a39a8f99e49
--
2.52.0
^ permalink raw reply related
* Re: [PATCH 1/6] sock: add sock_kzalloc helper
From: Herbert Xu @ 2026-05-26 9:14 UTC (permalink / raw)
To: Thorsten Blum
Cc: David S. Miller, Eric Dumazet, Kuniyuki Iwashima, Paolo Abeni,
Willem de Bruijn, Jakub Kicinski, Simon Horman, linux-crypto,
linux-kernel
In-Reply-To: <ahBRCxCXbCq5LeCc@linux.dev>
On Fri, May 22, 2026 at 02:50:19PM +0200, Thorsten Blum wrote:
> Hi Herbert,
>
> On Mon, Apr 27, 2026 at 12:41:30PM +0200, Thorsten Blum wrote:
> > Add sock_kzalloc() helper - the sock equivalent to kzalloc().
> >
> > Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
> > ---
> > include/net/sock.h | 5 +++++
> > 1 file changed, 5 insertions(+)
> >
> > diff --git a/include/net/sock.h b/include/net/sock.h
> > index dccd3738c368..20bf406dff2d 100644
> > --- a/include/net/sock.h
> > +++ b/include/net/sock.h
> > @@ -1904,6 +1904,11 @@ void sock_kfree_s(struct sock *sk, void *mem, int size);
> > void sock_kzfree_s(struct sock *sk, void *mem, int size);
> > void sk_send_sigurg(struct sock *sk);
> >
> > +static inline void *sock_kzalloc(struct sock *sk, int size, gfp_t priority)
> > +{
> > + return sock_kmalloc(sk, size, priority | __GFP_ZERO);
> > +}
> > +
> > static inline void sock_replace_proto(struct sock *sk, struct proto *proto)
> > {
> > if (sk->sk_socket)
>
> Can you take this series or should I resend this to net-next?
This patch needs an ack from the netdev maintainers.
Thanks,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply
* Re: [PATCH] MAINTAINERS: powerpc: update VMX AES entries
From: Madhavan Srinivasan @ 2026-05-26 2:00 UTC (permalink / raw)
To: Eric Biggers, Thorsten Blum
Cc: Herbert Xu, David S. Miller, Breno Leitão, Nayna Jain,
Paulo Flabiano Smorigo, Ard Biesheuvel, linux-crypto,
linuxppc-dev, linux-kernel
In-Reply-To: <20260524213525.GA112327@quark>
On 5/25/26 3:05 AM, Eric Biggers wrote:
> On Sun, May 24, 2026 at 11:29:45PM +0200, Thorsten Blum wrote:
>> Commit 7cf2082e74ce ("lib/crypto: powerpc/aes: Migrate POWER8 optimized
>> code into library") removed arch/powerpc/crypto/aes.c and moved
>> arch/powerpc/crypto/aesp8-ppc.pl to lib/crypto/powerpc/.
>>
>> However, the "IBM Power VMX Cryptographic instructions" entry still
>> references the removed file and no longer covers the moved aesp8-ppc.pl.
>>
>> Remove the stale entry, add lib/crypto/powerpc/aesp8-ppc.pl, and tighten
>> the arch/powerpc/crypto/aesp8-ppc.* pattern to match the remaining
>> header only.
>>
>> Fixes: 7cf2082e74ce ("lib/crypto: powerpc/aes: Migrate POWER8 optimized code into library")
>> Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
> Acked-by: Eric Biggers <ebiggers@kernel.org>
>
> If this doesn't get picked up through the powerpc tree, I can take this
> through libcrypto-next.
>
> - Eric
I can take this via ppc tree.
Maddy
^ permalink raw reply
* [PATCH 5/5] lib/crypto: xwing: Add KUnit tests for X-Wing KEM
From: Eric Biggers @ 2026-05-25 18:44 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, Ard Biesheuvel, Jason A . Donenfeld, Herbert Xu,
Ryan Appel, Chris Leech, Eric Biggers
In-Reply-To: <20260525184403.101818-1-ebiggers@kernel.org>
Add a KUnit test suite for the X-Wing key encapsulation mechanism.
This includes:
- Test key generation, encapsulation, and decapsulation against the
test vectors from the X-Wing specification
- Test encapsulation/decapsulation round trips
- Benchmark key generation, encapsulation, and decapsulation
This isn't intended to fully test the underlying KEMs. Those already
have their own KUnit tests.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
lib/crypto/.kunitconfig | 1 +
lib/crypto/tests/Kconfig | 9 ++
lib/crypto/tests/Makefile | 1 +
lib/crypto/tests/xwing-testvecs.h | 138 ++++++++++++++++++++++++
lib/crypto/tests/xwing_kunit.c | 129 ++++++++++++++++++++++
scripts/crypto/import-xwing-testvecs.py | 111 +++++++++++++++++++
6 files changed, 389 insertions(+)
create mode 100644 lib/crypto/tests/xwing-testvecs.h
create mode 100644 lib/crypto/tests/xwing_kunit.c
create mode 100755 scripts/crypto/import-xwing-testvecs.py
diff --git a/lib/crypto/.kunitconfig b/lib/crypto/.kunitconfig
index 32e5b4471da8..b42d8aaed244 100644
--- a/lib/crypto/.kunitconfig
+++ b/lib/crypto/.kunitconfig
@@ -17,5 +17,6 @@ CONFIG_CRYPTO_LIB_POLYVAL_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_SHA1_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_SHA256_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_SHA512_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_SHA3_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_SM3_KUNIT_TEST=y
+CONFIG_CRYPTO_LIB_XWING_KUNIT_TEST=y
diff --git a/lib/crypto/tests/Kconfig b/lib/crypto/tests/Kconfig
index 0a110a0733d2..485b5fd81539 100644
--- a/lib/crypto/tests/Kconfig
+++ b/lib/crypto/tests/Kconfig
@@ -147,10 +147,18 @@ config CRYPTO_LIB_SM3_KUNIT_TEST
default KUNIT_ALL_TESTS
select CRYPTO_LIB_BENCHMARK_VISIBLE
help
KUnit tests for the SM3 cryptographic hash function.
+config CRYPTO_LIB_XWING_KUNIT_TEST
+ tristate "KUnit tests for X-Wing" if !KUNIT_ALL_TESTS
+ depends on KUNIT && CRYPTO_LIB_XWING
+ default KUNIT_ALL_TESTS
+ select CRYPTO_LIB_BENCHMARK_VISIBLE
+ help
+ KUnit tests for the X-Wing key encapsulation mechanism.
+
config CRYPTO_LIB_ENABLE_ALL_FOR_KUNIT
tristate "Enable all crypto library code for KUnit tests"
depends on KUNIT
select CRYPTO_LIB_AES_CBC_MACS
select CRYPTO_LIB_BLAKE2B
@@ -165,10 +173,11 @@ config CRYPTO_LIB_ENABLE_ALL_FOR_KUNIT
select CRYPTO_LIB_SHA1
select CRYPTO_LIB_SHA256
select CRYPTO_LIB_SHA512
select CRYPTO_LIB_SHA3
select CRYPTO_LIB_SM3
+ select CRYPTO_LIB_XWING
help
Enable all the crypto library code that has KUnit tests.
Enable this only if you'd like to test all the crypto library code,
even code that wouldn't otherwise need to be built.
diff --git a/lib/crypto/tests/Makefile b/lib/crypto/tests/Makefile
index 3a73d2f33f75..acc8ddc19978 100644
--- a/lib/crypto/tests/Makefile
+++ b/lib/crypto/tests/Makefile
@@ -15,5 +15,6 @@ obj-$(CONFIG_CRYPTO_LIB_POLYVAL_KUNIT_TEST) += polyval_kunit.o
obj-$(CONFIG_CRYPTO_LIB_SHA1_KUNIT_TEST) += sha1_kunit.o
obj-$(CONFIG_CRYPTO_LIB_SHA256_KUNIT_TEST) += sha224_kunit.o sha256_kunit.o
obj-$(CONFIG_CRYPTO_LIB_SHA512_KUNIT_TEST) += sha384_kunit.o sha512_kunit.o
obj-$(CONFIG_CRYPTO_LIB_SHA3_KUNIT_TEST) += sha3_kunit.o
obj-$(CONFIG_CRYPTO_LIB_SM3_KUNIT_TEST) += sm3_kunit.o
+obj-$(CONFIG_CRYPTO_LIB_XWING_KUNIT_TEST) += xwing_kunit.o
diff --git a/lib/crypto/tests/xwing-testvecs.h b/lib/crypto/tests/xwing-testvecs.h
new file mode 100644
index 000000000000..057ca37c7ef7
--- /dev/null
+++ b/lib/crypto/tests/xwing-testvecs.h
@@ -0,0 +1,138 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* This file was generated by import-xwing-testvecs.py */
+
+static const struct xwing_testvec {
+ u8 seed[XWING_SEED_BYTES];
+ u8 pk_hash[SHA3_256_DIGEST_SIZE];
+ u8 sk_hash[SHA3_256_DIGEST_SIZE];
+ u8 eseed[XWING_ESEED_BYTES];
+ u8 ct_hash[SHA3_256_DIGEST_SIZE];
+ u8 ss[XWING_SHARED_SECRET_BYTES];
+} xwing_testvecs[] = {
+ {
+ .seed = {
+ 0x7f, 0x9c, 0x2b, 0xa4, 0xe8, 0x8f, 0x82, 0x7d,
+ 0x61, 0x60, 0x45, 0x50, 0x76, 0x05, 0x85, 0x3e,
+ 0xd7, 0x3b, 0x80, 0x93, 0xf6, 0xef, 0xbc, 0x88,
+ 0xeb, 0x1a, 0x6e, 0xac, 0xfa, 0x66, 0xef, 0x26,
+ },
+ .pk_hash = {
+ 0x51, 0x21, 0x74, 0x59, 0x04, 0x64, 0x3a, 0xd9,
+ 0xdf, 0xac, 0xca, 0x78, 0x69, 0x29, 0x2c, 0x19,
+ 0xa8, 0xa6, 0x95, 0x33, 0xb5, 0x3e, 0x60, 0x66,
+ 0x6b, 0x7d, 0xb9, 0x10, 0xb4, 0xad, 0x63, 0x67,
+ },
+ .sk_hash = {
+ 0x1f, 0x32, 0xc4, 0xb1, 0xa9, 0x65, 0xfa, 0x90,
+ 0x33, 0xb7, 0x85, 0xc2, 0xa2, 0x4b, 0x81, 0xdc,
+ 0xd1, 0xbc, 0x81, 0xe9, 0xbe, 0x0c, 0x20, 0x4f,
+ 0x20, 0xfd, 0xd1, 0x50, 0x92, 0xbb, 0x6d, 0x4a,
+ },
+ .eseed = {
+ 0x3c, 0xb1, 0xee, 0xa9, 0x88, 0x00, 0x4b, 0x93,
+ 0x10, 0x3c, 0xfb, 0x0a, 0xee, 0xfd, 0x2a, 0x68,
+ 0x6e, 0x01, 0xfa, 0x4a, 0x58, 0xe8, 0xa3, 0x63,
+ 0x9c, 0xa8, 0xa1, 0xe3, 0xf9, 0xae, 0x57, 0xe2,
+ 0x35, 0xb8, 0xcc, 0x87, 0x3c, 0x23, 0xdc, 0x62,
+ 0xb8, 0xd2, 0x60, 0x16, 0x9a, 0xfa, 0x2f, 0x75,
+ 0xab, 0x91, 0x6a, 0x58, 0xd9, 0x74, 0x91, 0x88,
+ 0x35, 0xd2, 0x5e, 0x6a, 0x43, 0x50, 0x85, 0xb2,
+ },
+ .ct_hash = {
+ 0xc0, 0xab, 0xd1, 0x49, 0xf8, 0x3f, 0x45, 0x32,
+ 0x4a, 0xc3, 0xa7, 0xdd, 0xc7, 0x60, 0x6c, 0x71,
+ 0xf2, 0x57, 0xe5, 0xea, 0x86, 0x11, 0x35, 0x22,
+ 0x83, 0x4a, 0x0e, 0xe1, 0xbc, 0xb3, 0x4e, 0x3e,
+ },
+ .ss = {
+ 0xd2, 0xdf, 0x05, 0x22, 0x12, 0x8f, 0x09, 0xdd,
+ 0x8e, 0x2c, 0x92, 0xb1, 0xe9, 0x05, 0xc7, 0x93,
+ 0xd8, 0xf5, 0x7a, 0x54, 0xc3, 0xda, 0x25, 0x86,
+ 0x1f, 0x10, 0xbf, 0x4c, 0xa6, 0x13, 0xe3, 0x84,
+ },
+ },
+ {
+ .seed = {
+ 0xba, 0xdf, 0xd6, 0xdf, 0xaa, 0xc3, 0x59, 0xa5,
+ 0xef, 0xbb, 0x7b, 0xcc, 0x4b, 0x59, 0xd5, 0x38,
+ 0xdf, 0x9a, 0x04, 0x30, 0x2e, 0x10, 0xc8, 0xbc,
+ 0x1c, 0xbf, 0x1a, 0x0b, 0x3a, 0x51, 0x20, 0xea,
+ },
+ .pk_hash = {
+ 0x79, 0x9b, 0x60, 0x16, 0xe5, 0xda, 0xa5, 0x6f,
+ 0xfa, 0x1b, 0x5e, 0x79, 0xf7, 0xca, 0xf7, 0x34,
+ 0x13, 0xce, 0xec, 0xd6, 0xdf, 0x42, 0x86, 0x42,
+ 0x40, 0x4c, 0xac, 0x41, 0xdd, 0xee, 0x48, 0x53,
+ },
+ .sk_hash = {
+ 0x1d, 0x31, 0x12, 0xef, 0x3c, 0xb6, 0x68, 0x10,
+ 0xd3, 0xad, 0x64, 0x18, 0x57, 0x9a, 0x5c, 0x8c,
+ 0xd9, 0xd1, 0xfa, 0x13, 0xf6, 0xdd, 0xc7, 0xc1,
+ 0x29, 0x7c, 0x09, 0x63, 0xe8, 0xbb, 0xb0, 0x24,
+ },
+ .eseed = {
+ 0x17, 0xcd, 0xa7, 0xcf, 0xad, 0x76, 0x5f, 0x56,
+ 0x23, 0x47, 0x4d, 0x36, 0x8c, 0xcc, 0xa8, 0xaf,
+ 0x00, 0x07, 0xcd, 0x9f, 0x5e, 0x4c, 0x84, 0x9f,
+ 0x16, 0x7a, 0x58, 0x0b, 0x14, 0xaa, 0xbd, 0xef,
+ 0xae, 0xe7, 0xee, 0xf4, 0x7c, 0xb0, 0xfc, 0xa9,
+ 0x76, 0x7b, 0xe1, 0xfd, 0xa6, 0x94, 0x19, 0xdf,
+ 0xb9, 0x27, 0xe9, 0xdf, 0x07, 0x34, 0x8b, 0x19,
+ 0x66, 0x91, 0xab, 0xae, 0xb5, 0x80, 0xb3, 0x2d,
+ },
+ .ct_hash = {
+ 0x76, 0x80, 0xb7, 0xba, 0x47, 0xae, 0x09, 0xba,
+ 0xc4, 0xb4, 0x30, 0x01, 0xed, 0xce, 0xf9, 0xd9,
+ 0x8e, 0x50, 0xdf, 0x20, 0x02, 0x6e, 0x70, 0xba,
+ 0x6a, 0x42, 0x44, 0x47, 0xe1, 0xf2, 0xb9, 0x61,
+ },
+ .ss = {
+ 0xf2, 0xe8, 0x62, 0x41, 0xc6, 0x4d, 0x60, 0xf6,
+ 0x64, 0x9f, 0xbc, 0x6c, 0x5b, 0x7d, 0x17, 0x18,
+ 0x0b, 0x78, 0x0a, 0x3f, 0x34, 0x35, 0x5e, 0x64,
+ 0xa8, 0x57, 0x49, 0x94, 0x9c, 0x45, 0xf1, 0x50,
+ },
+ },
+ {
+ .seed = {
+ 0xef, 0x58, 0x53, 0x8b, 0x8d, 0x23, 0xf8, 0x77,
+ 0x32, 0xea, 0x63, 0xb0, 0x2b, 0x4f, 0xa0, 0xf4,
+ 0x87, 0x33, 0x60, 0xe2, 0x84, 0x19, 0x28, 0xcd,
+ 0x60, 0xdd, 0x4c, 0xee, 0x8c, 0xc0, 0xd4, 0xc9,
+ },
+ .pk_hash = {
+ 0x1e, 0xf0, 0xc9, 0x9a, 0x06, 0x02, 0x64, 0x50,
+ 0x56, 0x49, 0x57, 0xa5, 0x40, 0x2a, 0x78, 0x8f,
+ 0xef, 0xfb, 0xbe, 0xfd, 0xcc, 0xe5, 0x5d, 0x25,
+ 0xde, 0x25, 0x4d, 0x0a, 0x49, 0xeb, 0x09, 0x5a,
+ },
+ .sk_hash = {
+ 0x75, 0x1b, 0x92, 0x96, 0x9e, 0xc1, 0x7a, 0x6a,
+ 0x5b, 0x4f, 0x95, 0xc0, 0xd2, 0xd9, 0x7d, 0xbc,
+ 0x0d, 0xa7, 0xb0, 0x98, 0x96, 0xaf, 0x69, 0xbd,
+ 0x69, 0xac, 0x7a, 0xb6, 0x19, 0x9e, 0x40, 0x3d,
+ },
+ .eseed = {
+ 0x22, 0xa9, 0x61, 0x88, 0xd0, 0x32, 0x67, 0x5c,
+ 0x8a, 0xc8, 0x50, 0x93, 0x3c, 0x7a, 0xff, 0x15,
+ 0x33, 0xb9, 0x4c, 0x83, 0x4a, 0xdb, 0xb6, 0x9c,
+ 0x61, 0x15, 0xba, 0xd4, 0x69, 0x2d, 0x86, 0x19,
+ 0xf9, 0x0b, 0x0c, 0xdf, 0x8a, 0x7b, 0x9c, 0x26,
+ 0x40, 0x29, 0xac, 0x18, 0x5b, 0x70, 0xb8, 0x3f,
+ 0x28, 0x01, 0xf2, 0xf4, 0xb3, 0xf7, 0x0c, 0x59,
+ 0x3e, 0xa3, 0xae, 0xeb, 0x61, 0x3a, 0x7f, 0x1b,
+ },
+ .ct_hash = {
+ 0x30, 0x88, 0x68, 0x8d, 0x63, 0x20, 0x1d, 0x5d,
+ 0x84, 0x41, 0x70, 0xb7, 0x9f, 0x14, 0x8b, 0x27,
+ 0x91, 0xc1, 0x5f, 0x34, 0x6f, 0xf6, 0xf8, 0xbd,
+ 0x55, 0x98, 0x07, 0xfb, 0xd4, 0x42, 0xf9, 0x1f,
+ },
+ .ss = {
+ 0x95, 0x3f, 0x7f, 0x4e, 0x8c, 0x5b, 0x50, 0x49,
+ 0xbd, 0xc7, 0x71, 0xd1, 0xdf, 0xfa, 0xda, 0x0d,
+ 0xd9, 0x61, 0x47, 0x7d, 0x1a, 0x2a, 0xe0, 0x98,
+ 0x8b, 0xaa, 0x7e, 0xa6, 0x89, 0x8d, 0x89, 0x3f,
+ },
+ },
+};
diff --git a/lib/crypto/tests/xwing_kunit.c b/lib/crypto/tests/xwing_kunit.c
new file mode 100644
index 000000000000..81c4f41e9cab
--- /dev/null
+++ b/lib/crypto/tests/xwing_kunit.c
@@ -0,0 +1,129 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * KUnit tests for X-Wing
+ *
+ * This should be run together with the mlkem and curve25519 KUnit tests.
+ *
+ * Copyright 2026 Google LLC
+ */
+#include <crypto/sha3.h>
+#include <crypto/xwing.h>
+#include <kunit/test.h>
+
+#include "xwing-testvecs.h"
+
+struct xwing_bufs {
+ u8 pk[XWING_PUBLIC_KEY_BYTES];
+ u8 sk[XWING_SECRET_KEY_BYTES];
+ u8 ct[XWING_CIPHERTEXT_BYTES];
+ u8 ss[XWING_SHARED_SECRET_BYTES];
+ u8 pk_hash[SHA3_256_DIGEST_SIZE];
+ u8 sk_hash[SHA3_256_DIGEST_SIZE];
+ u8 ct_hash[SHA3_256_DIGEST_SIZE];
+};
+
+static struct xwing_bufs *alloc_bufs(struct kunit *test)
+{
+ struct xwing_bufs *bufs =
+ kunit_kmalloc(test, sizeof(*bufs), GFP_KERNEL);
+
+ KUNIT_ASSERT_NOT_NULL(test, bufs);
+ return bufs;
+}
+
+static void test_xwing_rfc_testvecs(struct kunit *test)
+{
+ struct xwing_bufs *bufs = alloc_bufs(test);
+
+ for (size_t i = 0; i < ARRAY_SIZE(xwing_testvecs); i++) {
+ const struct xwing_testvec *tv = &xwing_testvecs[i];
+
+ KUNIT_ASSERT_EQ(test, 0,
+ xwing_keygen_internal(bufs->pk, bufs->sk,
+ tv->seed));
+ sha3_256(bufs->pk, sizeof(bufs->pk), bufs->pk_hash);
+ sha3_256(bufs->sk, sizeof(bufs->sk), bufs->sk_hash);
+ KUNIT_ASSERT_MEMEQ(test, tv->pk_hash, bufs->pk_hash,
+ sizeof(tv->pk_hash));
+ KUNIT_ASSERT_MEMEQ(test, tv->sk_hash, bufs->sk_hash,
+ sizeof(tv->sk_hash));
+
+ KUNIT_ASSERT_EQ(test, 0,
+ xwing_encaps_internal(bufs->ct, bufs->ss,
+ bufs->pk, tv->eseed));
+ sha3_256(bufs->ct, sizeof(bufs->ct), bufs->ct_hash);
+ KUNIT_ASSERT_MEMEQ(test, tv->ct_hash, bufs->ct_hash,
+ sizeof(tv->ct_hash));
+ KUNIT_ASSERT_MEMEQ(test, tv->ss, bufs->ss, sizeof(bufs->ss));
+
+ memset(bufs->ss, 0xff, sizeof(bufs->ss));
+ KUNIT_ASSERT_EQ(test, 0,
+ xwing_decaps(bufs->ss, bufs->ct, bufs->sk));
+ KUNIT_ASSERT_MEMEQ(test, tv->ss, bufs->ss, sizeof(bufs->ss));
+ }
+}
+
+static void test_xwing_round_trip(struct kunit *test)
+{
+ struct xwing_bufs *bufs = alloc_bufs(test);
+ u8 ss2[XWING_SHARED_SECRET_BYTES];
+
+ for (int i = 0; i < 20; i++) {
+ KUNIT_ASSERT_EQ(test, 0, xwing_keygen(bufs->pk, bufs->sk));
+ KUNIT_ASSERT_EQ(test, 0,
+ xwing_encaps(bufs->ct, bufs->ss, bufs->pk));
+ KUNIT_ASSERT_EQ(test, 0, xwing_decaps(ss2, bufs->ct, bufs->sk));
+ KUNIT_ASSERT_MEMEQ(test, bufs->ss, ss2, sizeof(bufs->ss));
+ }
+}
+
+/* Benchmark X-Wing performance. */
+static void benchmark_xwing(struct kunit *test)
+{
+ struct xwing_bufs *bufs = alloc_bufs(test);
+ const int iterations = 100;
+ ktime_t start, end;
+
+ if (!IS_ENABLED(CONFIG_CRYPTO_LIB_BENCHMARK))
+ kunit_skip(test, "not enabled");
+
+ start = ktime_get();
+ for (int i = 0; i < iterations; i++)
+ KUNIT_ASSERT_EQ(test, 0, xwing_keygen(bufs->pk, bufs->sk));
+ end = ktime_get();
+ kunit_info(test, "XWing_KeyGen: %llu ns/op\n",
+ div64_u64(ktime_to_ns(ktime_sub(end, start)), iterations));
+
+ start = ktime_get();
+ for (int i = 0; i < iterations; i++)
+ KUNIT_ASSERT_EQ(test, 0,
+ xwing_encaps(bufs->ct, bufs->ss, bufs->pk));
+ end = ktime_get();
+ kunit_info(test, "XWing_Encaps: %llu ns/op\n",
+ div64_u64(ktime_to_ns(ktime_sub(end, start)), iterations));
+
+ start = ktime_get();
+ for (int i = 0; i < iterations; i++)
+ KUNIT_ASSERT_EQ(test, 0,
+ xwing_decaps(bufs->ss, bufs->ct, bufs->sk));
+ end = ktime_get();
+ kunit_info(test, "XWing_Decaps: %llu ns/op\n",
+ div64_u64(ktime_to_ns(ktime_sub(end, start)), iterations));
+}
+
+static struct kunit_case xwing_test_cases[] = {
+ KUNIT_CASE(test_xwing_rfc_testvecs),
+ KUNIT_CASE(test_xwing_round_trip),
+ KUNIT_CASE(benchmark_xwing),
+ {}
+};
+
+static struct kunit_suite xwing_test_suite = {
+ .name = "xwing",
+ .test_cases = xwing_test_cases,
+};
+kunit_test_suite(xwing_test_suite);
+
+MODULE_DESCRIPTION("KUnit tests for X-Wing");
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
+MODULE_LICENSE("GPL");
diff --git a/scripts/crypto/import-xwing-testvecs.py b/scripts/crypto/import-xwing-testvecs.py
new file mode 100755
index 000000000000..380685212838
--- /dev/null
+++ b/scripts/crypto/import-xwing-testvecs.py
@@ -0,0 +1,111 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# This script imports test vectors from the X-Wing draft. To use:
+#
+# wget https://www.ietf.org/archive/id/draft-connolly-cfrg-xwing-kem-10.txt
+# $PATH_TO_THIS_SCRIPT draft-connolly-cfrg-xwing-kem-10.txt > lib/crypto/tests/xwing-testvecs.h
+
+import hashlib
+import os
+import sys
+
+SCRIPT_NAME = os.path.basename(__file__)
+FIELD_NAMES = set(["seed", "sk", "pk", "eseed", "ct", "ss"])
+FIRST_FIELD = "seed"
+
+XWING_SEED_BYTES = 32
+XWING_PUBLIC_KEY_BYTES = 1216
+XWING_SECRET_KEY_BYTES = 32
+XWING_ESEED_BYTES = 64
+XWING_CIPHERTEXT_BYTES = 1120
+XWING_SHARED_SECRET_BYTES = 32
+
+
+def check_length(val, expected_len):
+ assert len(val) == expected_len
+ return val
+
+
+class Testvec:
+ def __init__(self, tv):
+ self.seed = check_length(tv["seed"], XWING_SEED_BYTES)
+ self.pk = check_length(tv["pk"], XWING_PUBLIC_KEY_BYTES)
+ self.sk = check_length(tv["sk"], XWING_SECRET_KEY_BYTES)
+ self.eseed = check_length(tv["eseed"], XWING_ESEED_BYTES)
+ self.ct = check_length(tv["ct"], XWING_CIPHERTEXT_BYTES)
+ self.ss = check_length(tv["ss"], XWING_SHARED_SECRET_BYTES)
+
+
+def load_testvecs(file):
+ testvecs = []
+ with open(file) as file:
+ in_appendix = False
+ tv = None
+ cur_field = None
+ for line in file:
+ if line.startswith("Appendix C."):
+ in_appendix = True
+ elif line.startswith("Appendix D."):
+ break
+ fields = line.split()
+ if len(fields) == 0 or not in_appendix:
+ continue
+ if fields[0].strip() in FIELD_NAMES:
+ cur_field = fields[0].strip()
+ if cur_field == FIRST_FIELD:
+ if tv:
+ testvecs.append(Testvec(tv))
+ tv = {}
+ tv[cur_field] = b""
+ if len(fields) == 1:
+ continue
+ possible_data = fields[1].strip()
+ elif not cur_field:
+ continue
+ else:
+ possible_data = fields[0].strip()
+ try:
+ data = bytes.fromhex(possible_data)
+ tv[cur_field] += data
+ except ValueError:
+ pass
+ testvecs.append(Testvec(tv))
+ return testvecs
+
+
+def print_bytes(prefix, value, bytes_per_line):
+ for i in range(0, len(value), bytes_per_line):
+ line = prefix + "".join(f"0x{b:02x}, " for b in value[i : i + bytes_per_line])
+ print(f"{line.rstrip()}")
+
+
+def print_c_struct_u8_array_field(name, value):
+ print(f"\t\t.{name} = {{")
+ print_bytes("\t\t\t", value, 8)
+ print("\t\t},")
+
+
+testvecs = load_testvecs(sys.argv[1])
+
+print(f"""/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* This file was generated by {SCRIPT_NAME} */
+
+static const struct xwing_testvec {{
+\tu8 seed[XWING_SEED_BYTES];
+\tu8 pk_hash[SHA3_256_DIGEST_SIZE];
+\tu8 sk_hash[SHA3_256_DIGEST_SIZE];
+\tu8 eseed[XWING_ESEED_BYTES];
+\tu8 ct_hash[SHA3_256_DIGEST_SIZE];
+\tu8 ss[XWING_SHARED_SECRET_BYTES];
+}} xwing_testvecs[] = {{""")
+for tv in testvecs:
+ print("\t{")
+ print_c_struct_u8_array_field("seed", tv.seed)
+ print_c_struct_u8_array_field("pk_hash", hashlib.sha3_256(tv.pk).digest())
+ print_c_struct_u8_array_field("sk_hash", hashlib.sha3_256(tv.sk).digest())
+ print_c_struct_u8_array_field("eseed", tv.eseed)
+ print_c_struct_u8_array_field("ct_hash", hashlib.sha3_256(tv.ct).digest())
+ print_c_struct_u8_array_field("ss", tv.ss)
+ print("\t},")
+print("};")
--
2.54.0
^ permalink raw reply related
* [PATCH 4/5] lib/crypto: xwing: Add support for X-Wing KEM
From: Eric Biggers @ 2026-05-25 18:44 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, Ard Biesheuvel, Jason A . Donenfeld, Herbert Xu,
Ryan Appel, Chris Leech, Eric Biggers
In-Reply-To: <20260525184403.101818-1-ebiggers@kernel.org>
Add support for X-Wing, a general-purpose post-quantum/traditional
hybrid key encapsulation mechanism using X25519 and ML-KEM-768. X-Wing
is the recommended KEM for new applications. X-Wing is specified at
https://datatracker.ietf.org/doc/html/draft-connolly-cfrg-xwing-kem-10
This is a proof-of-concept that won't be merged until there's an
in-kernel user. Potential users include in-kernel users of classical
key agreement schemes (Bluetooth, NVMe auth, WireGuard).
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
Documentation/crypto/libcrypto-asymmetric.rst | 7 +
include/crypto/xwing.h | 84 +++++++
lib/crypto/Kconfig | 9 +
lib/crypto/Makefile | 5 +
lib/crypto/xwing.c | 237 ++++++++++++++++++
5 files changed, 342 insertions(+)
create mode 100644 include/crypto/xwing.h
create mode 100644 lib/crypto/xwing.c
diff --git a/Documentation/crypto/libcrypto-asymmetric.rst b/Documentation/crypto/libcrypto-asymmetric.rst
index 6e71c5ce823f..d44ee74f6a46 100644
--- a/Documentation/crypto/libcrypto-asymmetric.rst
+++ b/Documentation/crypto/libcrypto-asymmetric.rst
@@ -16,5 +16,12 @@ ML-KEM
Support for the ML-KEM key encapsulation mechanism.
This shall be used as part of a hybrid scheme such as X-Wing, not by itself.
.. kernel-doc:: include/crypto/mlkem.h
+
+X-Wing
+------
+
+Support for the X-Wing key encapsulation mechanism.
+
+.. kernel-doc:: include/crypto/xwing.h
diff --git a/include/crypto/xwing.h b/include/crypto/xwing.h
new file mode 100644
index 000000000000..d1ad7b1e6596
--- /dev/null
+++ b/include/crypto/xwing.h
@@ -0,0 +1,84 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright 2026 Google LLC
+ */
+
+/**
+ * DOC: X-Wing key encapsulation mechanism
+ *
+ * Implementation of X-Wing, a general-purpose post-quantum/traditional hybrid
+ * key encapsulation mechanism using X25519 and ML-KEM-768. X-Wing is the
+ * recommended KEM for new applications. X-Wing is specified at
+ * https://datatracker.ietf.org/doc/html/draft-connolly-cfrg-xwing-kem-10
+ */
+
+#ifndef _CRYPTO_XWING_H
+#define _CRYPTO_XWING_H
+
+#include <linux/types.h>
+
+#define XWING_SEED_BYTES 32 /* Length of seed for KeyGen */
+#define XWING_PUBLIC_KEY_BYTES 1216
+#define XWING_SECRET_KEY_BYTES 32
+#define XWING_ESEED_BYTES 64 /* Length of seed for Encaps */
+#define XWING_CIPHERTEXT_BYTES 1120
+#define XWING_SHARED_SECRET_BYTES 32
+
+/**
+ * xwing_keygen() - Generate an X-Wing key pair
+ * @pk: (output) The public key (encapsulation key)
+ * @sk: (output) The secret key (decapsulation key)
+ *
+ * Context: Might sleep
+ *
+ * Return: 0 on success, or -ENOMEM if out of memory.
+ */
+int xwing_keygen(u8 pk[XWING_PUBLIC_KEY_BYTES], u8 sk[XWING_SECRET_KEY_BYTES]);
+
+/**
+ * xwing_encaps() - Generate and encapsulate shared secret with X-Wing
+ * @ct: (output) The ciphertext
+ * @ss: (output) The generated shared secret
+ * @pk: The public key (encapsulation key)
+ *
+ * Context: Might sleep
+ *
+ * Return:
+ * * 0 on success
+ * * -EBADMSG if the public key is malformed
+ * * -ENOMEM if out of memory
+ */
+int xwing_encaps(u8 ct[XWING_CIPHERTEXT_BYTES],
+ u8 ss[XWING_SHARED_SECRET_BYTES],
+ const u8 pk[XWING_PUBLIC_KEY_BYTES]);
+
+/**
+ * xwing_decaps() - Decapsulate shared secret with X-Wing
+ * @ss: (output) The decapsulated shared secret
+ * @ct: The ciphertext
+ * @sk: The secret key (decapsulation key)
+ *
+ * Context: Might sleep
+ *
+ * Return:
+ * * 0 on success, including the implicit rejection cases where the ciphertext
+ * is invalid and a randomized shared secret is returned
+ * * -EBADMSG if the secret key is malformed
+ * * -ENOMEM if out of memory
+ */
+int xwing_decaps(u8 ss[XWING_SHARED_SECRET_BYTES],
+ const u8 ct[XWING_CIPHERTEXT_BYTES],
+ const u8 sk[XWING_SECRET_KEY_BYTES]);
+
+#if IS_ENABLED(CONFIG_KUNIT)
+/* Functions taking explicit seeds, only for KUnit testing */
+int xwing_keygen_internal(u8 pk[XWING_PUBLIC_KEY_BYTES],
+ u8 sk[XWING_SECRET_KEY_BYTES],
+ const u8 seed[XWING_SEED_BYTES]);
+int xwing_encaps_internal(u8 ct[XWING_CIPHERTEXT_BYTES],
+ u8 ss[XWING_SHARED_SECRET_BYTES],
+ const u8 pk[XWING_PUBLIC_KEY_BYTES],
+ const u8 eseed[XWING_ESEED_BYTES]);
+#endif /* CONFIG_KUNIT */
+
+#endif /* _CRYPTO_XWING_H */
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index acaa64af4c6d..2ce1867eeb9e 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -285,5 +285,14 @@ config CRYPTO_LIB_SM3_ARCH
depends on CRYPTO_LIB_SM3 && !UML
default y if ARM64
default y if RISCV && 64BIT && TOOLCHAIN_HAS_VECTOR_CRYPTO && \
RISCV_EFFICIENT_VECTOR_UNALIGNED_ACCESS
default y if X86_64
+
+config CRYPTO_LIB_XWING
+ tristate
+ select CRYPTO_LIB_CURVE25519
+ select CRYPTO_LIB_MLKEM
+ select CRYPTO_LIB_SHA3
+ help
+ The X-Wing library functions. Select this if your module uses any of
+ the functions from <crypto/xwing.h>.
diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile
index 94cef4e574cd..a20ae074cbfa 100644
--- a/lib/crypto/Makefile
+++ b/lib/crypto/Makefile
@@ -384,10 +384,15 @@ libsm3-$(CONFIG_RISCV) += riscv/sm3-riscv64-zvksh-zvkb.o
libsm3-$(CONFIG_X86) += x86/sm3-avx-asm_64.o
endif # CONFIG_CRYPTO_LIB_SM3_ARCH
################################################################################
+obj-$(CONFIG_CRYPTO_LIB_XWING) += libxwing.o
+libxwing-y := xwing.o
+
+################################################################################
+
obj-$(CONFIG_MPILIB) += mpi/
obj-$(CONFIG_CRYPTO_SELFTESTS_FULL) += simd.o
# clean-files must be defined unconditionally
diff --git a/lib/crypto/xwing.c b/lib/crypto/xwing.c
new file mode 100644
index 000000000000..deffa46b900e
--- /dev/null
+++ b/lib/crypto/xwing.c
@@ -0,0 +1,237 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * X-Wing key encapsulation mechanism
+ *
+ * See include/crypto/xwing.h for the documentation.
+ *
+ * Copyright 2026 Google LLC
+ */
+
+#include <crypto/curve25519.h>
+#include <crypto/mlkem.h>
+#include <crypto/sha3.h>
+#include <crypto/xwing.h>
+#include <kunit/visibility.h>
+#include <linux/export.h>
+#include <linux/module.h>
+#include <linux/slab.h>
+
+/* pk = pk_mlkem || pk_x25519 */
+static_assert(XWING_PUBLIC_KEY_BYTES ==
+ MLKEM768_PUBLIC_KEY_BYTES + CURVE25519_KEY_SIZE);
+
+/* sk = seed */
+static_assert(XWING_SECRET_KEY_BYTES == XWING_SEED_BYTES);
+
+/* ct = ct_mlkem || ct_x25519 */
+static_assert(XWING_CIPHERTEXT_BYTES ==
+ MLKEM768_CIPHERTEXT_BYTES + CURVE25519_KEY_SIZE);
+
+/* expanded_sk = sk_mlkem || sk_x25519 || pk_mlkem || pk_x25519 */
+#define XWING_EXPANDED_SECRET_KEY_BYTES \
+ (MLKEM768_SECRET_KEY_BYTES + CURVE25519_KEY_SIZE + \
+ MLKEM768_PUBLIC_KEY_BYTES + CURVE25519_KEY_SIZE)
+
+static int xwing_expand_sk(u8 expanded_sk[XWING_EXPANDED_SECRET_KEY_BYTES],
+ const u8 sk[XWING_SECRET_KEY_BYTES])
+{
+ u8 *sk_mlkem = &expanded_sk[0];
+ u8 *sk_x25519 = &sk_mlkem[MLKEM768_SECRET_KEY_BYTES];
+ u8 *pk_mlkem = &sk_x25519[CURVE25519_KEY_SIZE];
+ u8 *pk_x25519 = &pk_mlkem[MLKEM768_PUBLIC_KEY_BYTES];
+ u8 seed_mlkem[MLKEM_SEED_BYTES];
+ struct shake_ctx shake;
+ int err;
+
+ shake256_init(&shake);
+ shake_update(&shake, sk, XWING_SECRET_KEY_BYTES);
+ shake_squeeze(&shake, seed_mlkem, sizeof(seed_mlkem));
+
+ err = mlkem768_keygen_internal(pk_mlkem, sk_mlkem, seed_mlkem);
+ if (err) /* can only be -ENOMEM */
+ goto out;
+ shake_squeeze(&shake, sk_x25519, CURVE25519_KEY_SIZE);
+ curve25519_clamp_secret(sk_x25519);
+ if (unlikely(!curve25519_generate_public(pk_x25519, sk_x25519)))
+ err = -EAGAIN;
+out:
+ shake_zeroize_ctx(&shake);
+ memzero_explicit(seed_mlkem, sizeof(seed_mlkem));
+ return err;
+}
+
+VISIBLE_IF_KUNIT int xwing_keygen_internal(u8 pk[XWING_PUBLIC_KEY_BYTES],
+ u8 sk[XWING_SECRET_KEY_BYTES],
+ const u8 seed[XWING_SEED_BYTES])
+{
+ u8 *expanded_sk __free(kfree_sensitive) =
+ kmalloc(XWING_EXPANDED_SECRET_KEY_BYTES, GFP_KERNEL);
+ int err;
+
+ if (!expanded_sk)
+ return -ENOMEM;
+
+ err = xwing_expand_sk(expanded_sk, seed);
+ if (err)
+ return err;
+ /* pk = pk_mlkem || pk_x25519 */
+ memcpy(pk,
+ &expanded_sk[MLKEM768_SECRET_KEY_BYTES + CURVE25519_KEY_SIZE],
+ XWING_PUBLIC_KEY_BYTES);
+ /* sk = seed */
+ memcpy(sk, seed, XWING_SECRET_KEY_BYTES);
+ return 0;
+}
+EXPORT_SYMBOL_IF_KUNIT(xwing_keygen_internal);
+
+int xwing_keygen(u8 pk[XWING_PUBLIC_KEY_BYTES], u8 sk[XWING_SECRET_KEY_BYTES])
+{
+ u8 seed[XWING_SEED_BYTES];
+ int err;
+
+ do {
+ get_random_bytes(seed, sizeof(seed));
+ err = xwing_keygen_internal(pk, sk, seed);
+ } while (err == -EAGAIN); /* curve25519_null_point case */
+ memzero_explicit(seed, sizeof(seed));
+ return err;
+}
+EXPORT_SYMBOL_GPL(xwing_keygen);
+
+static void xwing_combine(u8 ss[XWING_SHARED_SECRET_BYTES],
+ const u8 ss_mlkem[MLKEM_SHARED_SECRET_BYTES],
+ const u8 ss_x25519[CURVE25519_KEY_SIZE],
+ const u8 ct_x25519[CURVE25519_KEY_SIZE],
+ const u8 pk_x25519[CURVE25519_KEY_SIZE])
+{
+ static const u8 xwing_label[6] = { 0x5c, 0x2e, 0x2f, 0x2f, 0x5e, 0x5c };
+ struct sha3_ctx ctx;
+
+ sha3_256_init(&ctx);
+ sha3_update(&ctx, ss_mlkem, MLKEM_SHARED_SECRET_BYTES);
+ sha3_update(&ctx, ss_x25519, CURVE25519_KEY_SIZE);
+ sha3_update(&ctx, ct_x25519, CURVE25519_KEY_SIZE);
+ sha3_update(&ctx, pk_x25519, CURVE25519_KEY_SIZE);
+ sha3_update(&ctx, xwing_label, sizeof(xwing_label));
+ sha3_final(&ctx, ss);
+}
+
+VISIBLE_IF_KUNIT int xwing_encaps_internal(u8 ct[XWING_CIPHERTEXT_BYTES],
+ u8 ss[XWING_SHARED_SECRET_BYTES],
+ const u8 pk[XWING_PUBLIC_KEY_BYTES],
+ const u8 eseed[XWING_ESEED_BYTES])
+{
+ const u8 *pk_mlkem = &pk[0];
+ const u8 *pk_x25519 = &pk[MLKEM768_PUBLIC_KEY_BYTES];
+ const u8 *eseed_mlkem = &eseed[0];
+ const u8 *eseed_x25519 = &eseed[MLKEM_ESEED_BYTES];
+ u8 eph_sk_x25519[CURVE25519_KEY_SIZE];
+ u8 *ct_mlkem = &ct[0];
+ u8 *ct_x25519 = &ct[MLKEM768_CIPHERTEXT_BYTES];
+ u8 ss_mlkem[MLKEM_SHARED_SECRET_BYTES];
+ u8 ss_x25519[CURVE25519_KEY_SIZE];
+ int err;
+
+ err = mlkem768_encaps_internal(ct_mlkem, ss_mlkem, pk_mlkem,
+ eseed_mlkem);
+ if (err)
+ goto out;
+ memcpy(eph_sk_x25519, eseed_x25519, CURVE25519_KEY_SIZE);
+ curve25519_clamp_secret(eph_sk_x25519);
+ if (!curve25519_generate_public(ct_x25519, eph_sk_x25519)) {
+ err = -EAGAIN;
+ goto out;
+ }
+ if (!curve25519(ss_x25519, eph_sk_x25519, pk_x25519)) {
+ err = -EBADMSG;
+ goto out;
+ }
+ xwing_combine(ss, ss_mlkem, ss_x25519, ct_x25519, pk_x25519);
+ err = 0;
+out:
+ if (err) {
+ get_random_bytes(ct, XWING_CIPHERTEXT_BYTES);
+ get_random_bytes(ss, XWING_SHARED_SECRET_BYTES);
+ }
+ memzero_explicit(eph_sk_x25519, sizeof(eph_sk_x25519));
+ memzero_explicit(ss_mlkem, sizeof(ss_mlkem));
+ memzero_explicit(ss_x25519, sizeof(ss_x25519));
+ return err;
+}
+EXPORT_SYMBOL_IF_KUNIT(xwing_encaps_internal);
+
+int xwing_encaps(u8 ct[XWING_CIPHERTEXT_BYTES],
+ u8 ss[XWING_SHARED_SECRET_BYTES],
+ const u8 pk[XWING_PUBLIC_KEY_BYTES])
+{
+ u8 eseed[XWING_ESEED_BYTES];
+ int err;
+
+ do {
+ get_random_bytes(eseed, sizeof(eseed));
+ err = xwing_encaps_internal(ct, ss, pk, eseed);
+ } while (err == -EAGAIN); /* curve25519_null_point case */
+ memzero_explicit(eseed, sizeof(eseed));
+ return err;
+}
+EXPORT_SYMBOL_GPL(xwing_encaps);
+
+int xwing_decaps(u8 ss[XWING_SHARED_SECRET_BYTES],
+ const u8 ct[XWING_CIPHERTEXT_BYTES],
+ const u8 sk[XWING_SECRET_KEY_BYTES])
+{
+ u8 *expanded_sk __free(kfree_sensitive) =
+ kmalloc(XWING_EXPANDED_SECRET_KEY_BYTES, GFP_KERNEL);
+ u8 *sk_mlkem, *sk_x25519, *pk_mlkem, *pk_x25519;
+ const u8 *ct_mlkem = &ct[0];
+ const u8 *ct_x25519 = &ct[MLKEM768_CIPHERTEXT_BYTES];
+ u8 ss_mlkem[MLKEM_SHARED_SECRET_BYTES];
+ u8 ss_x25519[CURVE25519_KEY_SIZE];
+ int err;
+
+ if (!expanded_sk) {
+ err = -ENOMEM;
+ goto out;
+ }
+ err = xwing_expand_sk(expanded_sk, sk);
+ if (err) {
+ if (err == -EAGAIN) /* curve25519_null_point case */
+ err = -EBADMSG;
+ goto out;
+ }
+ sk_mlkem = &expanded_sk[0];
+ sk_x25519 = &sk_mlkem[MLKEM768_SECRET_KEY_BYTES];
+ pk_mlkem = &sk_x25519[CURVE25519_KEY_SIZE];
+ pk_x25519 = &pk_mlkem[MLKEM768_PUBLIC_KEY_BYTES];
+
+ err = mlkem768_decaps(ss_mlkem, ct_mlkem, sk_mlkem);
+ if (err) {
+ /*
+ * This is either -ENOMEM, or -EBADMSG for a malformed secret
+ * key. This case is *not* reached if the ciphertext is
+ * invalid, as implicit rejection is used.
+ */
+ goto out;
+ }
+ if (!curve25519(ss_x25519, sk_x25519, ct_x25519)) {
+ /*
+ * ss_x25519 is curve25519_null_point, which can happen if the
+ * ciphertext is invalid. In this case the correct behavior is
+ * to continue anyway and implicitly reject.
+ */
+ }
+
+ xwing_combine(ss, ss_mlkem, ss_x25519, ct_x25519, pk_x25519);
+ err = 0;
+out:
+ if (err)
+ get_random_bytes(ss, XWING_SHARED_SECRET_BYTES);
+ memzero_explicit(ss_mlkem, sizeof(ss_mlkem));
+ memzero_explicit(ss_x25519, sizeof(ss_x25519));
+ return err;
+}
+EXPORT_SYMBOL_GPL(xwing_decaps);
+
+MODULE_DESCRIPTION("X-Wing key encapsulation mechanism");
+MODULE_IMPORT_NS("CRYPTO_INTERNAL");
+MODULE_LICENSE("GPL");
--
2.54.0
^ permalink raw reply related
* [PATCH 3/5] lib/crypto: mlkem: Add FIPS 140-3 tests
From: Eric Biggers @ 2026-05-25 18:44 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, Ard Biesheuvel, Jason A . Donenfeld, Herbert Xu,
Ryan Appel, Chris Leech, Eric Biggers
In-Reply-To: <20260525184403.101818-1-ebiggers@kernel.org>
Add the ML-KEM self-tests required by FIPS 140-3:
- Pairwise Consistency Test (PCT) at KeyGen time
- Cryptographic Algorithm Self-Test (CAST) at initialization time
As with the other crypto algorithms, these are implemented separately
from the KUnit test suite, due to the very different requirements.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
lib/crypto/fips-mlkem.h | 523 ++++++++++++++++++++++++
lib/crypto/mlkem.c | 115 ++++++
scripts/crypto/import-mlkem-testvecs.py | 62 +++
3 files changed, 700 insertions(+)
create mode 100644 lib/crypto/fips-mlkem.h
diff --git a/lib/crypto/fips-mlkem.h b/lib/crypto/fips-mlkem.h
new file mode 100644
index 000000000000..bb91c97d4b28
--- /dev/null
+++ b/lib/crypto/fips-mlkem.h
@@ -0,0 +1,523 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* This file was generated by import-mlkem-testvecs.py */
+/* clang-format off */
+
+static const u8 fips_test_mlkem768_seed[MLKEM_SEED_BYTES] __initconst __maybe_unused = {
+ 0x7f, 0x9c, 0x2b, 0xa4, 0xe8, 0x8f, 0x82, 0x7d, 0x61, 0x60, 0x45, 0x50,
+ 0x76, 0x05, 0x85, 0x3e, 0xd7, 0x3b, 0x80, 0x93, 0xf6, 0xef, 0xbc, 0x88,
+ 0xeb, 0x1a, 0x6e, 0xac, 0xfa, 0x66, 0xef, 0x26, 0x3c, 0xb1, 0xee, 0xa9,
+ 0x88, 0x00, 0x4b, 0x93, 0x10, 0x3c, 0xfb, 0x0a, 0xee, 0xfd, 0x2a, 0x68,
+ 0x6e, 0x01, 0xfa, 0x4a, 0x58, 0xe8, 0xa3, 0x63, 0x9c, 0xa8, 0xa1, 0xe3,
+ 0xf9, 0xae, 0x57, 0xe2,
+};
+
+static const u8 fips_test_mlkem768_eseed[MLKEM_ESEED_BYTES] __initconst __maybe_unused = {
+ 0x35, 0xb8, 0xcc, 0x87, 0x3c, 0x23, 0xdc, 0x62, 0xb8, 0xd2, 0x60, 0x16,
+ 0x9a, 0xfa, 0x2f, 0x75, 0xab, 0x91, 0x6a, 0x58, 0xd9, 0x74, 0x91, 0x88,
+ 0x35, 0xd2, 0x5e, 0x6a, 0x43, 0x50, 0x85, 0xb2,
+};
+
+static const u8 fips_test_mlkem768_pk[MLKEM768_PUBLIC_KEY_BYTES] __initconst __maybe_unused = {
+ 0x78, 0x20, 0x32, 0x02, 0x30, 0x23, 0x8e, 0x44, 0x7a, 0xcf, 0xa9, 0x9b,
+ 0x63, 0x32, 0xb7, 0x53, 0x1c, 0x7c, 0xe5, 0x42, 0x03, 0x1b, 0x93, 0xca,
+ 0x14, 0x25, 0x8f, 0x5f, 0x98, 0xb3, 0x0c, 0x87, 0x3f, 0x6d, 0x01, 0xb2,
+ 0xc5, 0x85, 0x3f, 0x34, 0x02, 0x69, 0x18, 0x04, 0x80, 0x47, 0x08, 0xa7,
+ 0xec, 0xfa, 0x5b, 0xa5, 0x98, 0x0f, 0xef, 0x88, 0x36, 0xce, 0xf5, 0xa8,
+ 0x2d, 0xac, 0x83, 0x75, 0xe0, 0x62, 0x34, 0xcb, 0xbe, 0xd6, 0x4c, 0xc5,
+ 0xa4, 0xb2, 0xac, 0x06, 0x84, 0xa0, 0xb4, 0xd9, 0x86, 0x3e, 0x66, 0x44,
+ 0xa8, 0x33, 0x48, 0x49, 0xc0, 0xcf, 0xdb, 0xfb, 0x85, 0x1b, 0x84, 0xb0,
+ 0xe4, 0x8c, 0x6f, 0xb0, 0x48, 0x01, 0xd0, 0x52, 0x2f, 0xd5, 0x76, 0x68,
+ 0x9b, 0x16, 0x44, 0x67, 0x60, 0x4a, 0xb4, 0xda, 0x2e, 0x34, 0xab, 0xcf,
+ 0x76, 0xdc, 0x92, 0x44, 0x67, 0x4c, 0x4e, 0x57, 0x30, 0xf5, 0x78, 0x7c,
+ 0x67, 0x90, 0x21, 0x42, 0x4c, 0x13, 0x40, 0x15, 0x4d, 0x24, 0x49, 0x33,
+ 0x0d, 0xbc, 0x94, 0x0e, 0xba, 0x7b, 0xe1, 0x03, 0x99, 0xec, 0x94, 0x0a,
+ 0x6b, 0x35, 0x40, 0x60, 0x14, 0x99, 0x66, 0x50, 0xb1, 0x48, 0xa1, 0x89,
+ 0x43, 0x47, 0x7f, 0x5e, 0x95, 0x0f, 0x41, 0xa0, 0x1a, 0xb6, 0x34, 0x88,
+ 0xfa, 0xf2, 0x06, 0xf0, 0xf2, 0x25, 0xa7, 0x31, 0x09, 0x67, 0x77, 0x84,
+ 0x79, 0x1b, 0x6d, 0xec, 0x98, 0x0e, 0x60, 0xeb, 0x8f, 0x96, 0xbb, 0xbf,
+ 0x47, 0x22, 0x22, 0xb0, 0x24, 0x8a, 0x12, 0xf9, 0x61, 0x88, 0xa1, 0x7d,
+ 0x68, 0xb8, 0x3e, 0x52, 0xc9, 0x82, 0x7b, 0xa2, 0xc6, 0xbc, 0x82, 0x75,
+ 0x0b, 0x31, 0xae, 0x46, 0xb9, 0x82, 0x7a, 0x28, 0x77, 0xfc, 0xb3, 0x38,
+ 0x00, 0x3a, 0x68, 0xb8, 0xc1, 0xaa, 0x9c, 0x96, 0x6b, 0x55, 0xd2, 0x03,
+ 0x78, 0x74, 0x11, 0x57, 0x77, 0x05, 0xf6, 0xdb, 0x15, 0x59, 0xf9, 0xb4,
+ 0x25, 0x8a, 0x57, 0x18, 0x20, 0x50, 0x5d, 0x83, 0xc1, 0x00, 0x9a, 0xa2,
+ 0xb2, 0x84, 0xb3, 0x75, 0xa7, 0xac, 0x61, 0x6a, 0x1b, 0x29, 0x42, 0xa2,
+ 0xb4, 0x94, 0xc4, 0x3e, 0xb3, 0x70, 0xc1, 0xb9, 0x4a, 0xcd, 0x52, 0x2b,
+ 0x20, 0x8c, 0x17, 0xc7, 0x35, 0x07, 0x7e, 0x60, 0x2f, 0x41, 0x41, 0x0c,
+ 0xfb, 0x1b, 0x3e, 0xa1, 0x09, 0x21, 0x58, 0xc3, 0xb6, 0xea, 0x43, 0xcc,
+ 0x7a, 0xa7, 0x03, 0x09, 0xa6, 0x0f, 0xa9, 0x64, 0x47, 0x21, 0x33, 0x2a,
+ 0x0d, 0xc4, 0x55, 0xbd, 0xb5, 0x63, 0xd5, 0xe6, 0x6e, 0xe2, 0x12, 0x10,
+ 0x87, 0x2a, 0x59, 0x8d, 0x13, 0xa9, 0x51, 0x83, 0x11, 0x9a, 0x50, 0x7a,
+ 0x76, 0x34, 0x3b, 0x45, 0x4b, 0xb3, 0x20, 0x4b, 0xad, 0x83, 0x46, 0x9b,
+ 0x56, 0x27, 0x52, 0xa7, 0x3a, 0x00, 0xe5, 0xe8, 0xc5, 0xa6, 0xd9, 0x87,
+ 0x0a, 0xca, 0x93, 0x78, 0x69, 0x05, 0xc2, 0x0b, 0x85, 0x02, 0xf4, 0x5d,
+ 0x91, 0x14, 0xa8, 0xd2, 0xc7, 0xbd, 0x2e, 0xe0, 0x08, 0x7f, 0xb7, 0x80,
+ 0xfa, 0x50, 0x78, 0x58, 0x8b, 0x02, 0x6f, 0x14, 0x19, 0x6c, 0xc0, 0x68,
+ 0xf6, 0xa7, 0x5e, 0x7b, 0x4a, 0x44, 0x3a, 0xea, 0x41, 0x42, 0x5c, 0xaf,
+ 0x2d, 0xa2, 0x12, 0xa3, 0xe6, 0xad, 0xf6, 0x44, 0x51, 0x64, 0xc2, 0x90,
+ 0x8f, 0x40, 0x9d, 0x43, 0x96, 0x5e, 0xe4, 0x0b, 0x89, 0xd6, 0x54, 0xc8,
+ 0x41, 0x30, 0x97, 0x48, 0x6a, 0x9b, 0x90, 0xe2, 0xb0, 0x44, 0x74, 0x38,
+ 0x91, 0x18, 0x36, 0xb7, 0xca, 0x84, 0xe7, 0x6b, 0x7a, 0x8e, 0xe6, 0x00,
+ 0xa1, 0x3b, 0x55, 0xa8, 0xb8, 0x3d, 0xa0, 0xb2, 0x08, 0x25, 0x38, 0x2b,
+ 0xda, 0xc3, 0xbe, 0x25, 0x04, 0x60, 0x24, 0xc7, 0x88, 0x76, 0x40, 0x53,
+ 0xf1, 0x92, 0x2c, 0xfa, 0xa2, 0xa1, 0xb8, 0x16, 0x22, 0x30, 0xa3, 0x6d,
+ 0xbb, 0x36, 0xcf, 0x8e, 0x62, 0xc8, 0xa1, 0xb1, 0x5f, 0x9d, 0x51, 0x90,
+ 0x07, 0x0a, 0xb4, 0xf5, 0x70, 0xa3, 0xba, 0xf8, 0x63, 0x7c, 0x04, 0xad,
+ 0x19, 0x51, 0xa4, 0xb4, 0x61, 0x0b, 0x5b, 0x7a, 0xb2, 0x33, 0x56, 0xc3,
+ 0x9e, 0x36, 0xa8, 0x12, 0x85, 0x58, 0xa2, 0x59, 0x14, 0xe1, 0xa8, 0xaf,
+ 0x76, 0xe4, 0x77, 0x12, 0xf8, 0x54, 0x38, 0xb1, 0x9e, 0x8d, 0x04, 0xce,
+ 0x92, 0x35, 0x20, 0xd3, 0xab, 0x0a, 0xc4, 0xe4, 0x08, 0x58, 0xf2, 0x36,
+ 0x2d, 0xd4, 0x9d, 0xb4, 0x46, 0xb7, 0xed, 0xf7, 0xa0, 0xdd, 0x4c, 0x83,
+ 0xaa, 0x30, 0x7e, 0xfa, 0x87, 0x76, 0xb3, 0xb8, 0xc7, 0xa1, 0x0a, 0x1f,
+ 0xdf, 0xe6, 0x8e, 0xda, 0x1b, 0x9c, 0xd4, 0x2a, 0xc0, 0xce, 0xc4, 0x76,
+ 0x0c, 0x25, 0x93, 0x18, 0x08, 0x2c, 0x18, 0x41, 0x5d, 0x2c, 0xc2, 0x93,
+ 0x67, 0xf6, 0x06, 0xee, 0xea, 0x1c, 0x01, 0x45, 0x38, 0x9e, 0xa2, 0x88,
+ 0xf3, 0x83, 0xa6, 0x8d, 0xd6, 0x20, 0x12, 0x7a, 0xac, 0xa8, 0x8b, 0xcb,
+ 0x6b, 0x42, 0x45, 0xa6, 0x64, 0x4f, 0x05, 0xa6, 0xb9, 0xf1, 0x16, 0x3d,
+ 0x38, 0xe1, 0x7e, 0x57, 0xcb, 0x27, 0xcc, 0x5c, 0x46, 0x7b, 0x2c, 0xca,
+ 0x94, 0xf9, 0x74, 0xcf, 0xd8, 0x15, 0x0d, 0x52, 0x94, 0xe1, 0x52, 0x20,
+ 0x0b, 0x39, 0xba, 0xfa, 0x46, 0x9b, 0x9c, 0xfa, 0xc3, 0xac, 0xeb, 0x7b,
+ 0x56, 0xb4, 0x7b, 0x13, 0xf1, 0x71, 0x81, 0xea, 0x95, 0x87, 0x78, 0x10,
+ 0xa7, 0x77, 0xae, 0x9d, 0x15, 0x60, 0xed, 0x51, 0x19, 0xab, 0x09, 0xb7,
+ 0x6a, 0xb1, 0x20, 0xe3, 0xc6, 0x19, 0x61, 0xd5, 0x8c, 0x3a, 0x93, 0x16,
+ 0x69, 0xa0, 0x5f, 0xd2, 0x88, 0x2f, 0x51, 0x97, 0xb7, 0xc3, 0xb0, 0x47,
+ 0xa1, 0x97, 0xaa, 0x82, 0x0c, 0x20, 0x8a, 0x5c, 0xb5, 0xbc, 0x40, 0xbe,
+ 0xb8, 0x25, 0x88, 0xf5, 0x61, 0x0b, 0x0b, 0x35, 0x4e, 0xb8, 0x50, 0x39,
+ 0xb0, 0xa4, 0x1d, 0xfd, 0x39, 0x24, 0x8f, 0x16, 0x5a, 0xbb, 0x02, 0x7a,
+ 0xfd, 0x68, 0x17, 0xe0, 0x0c, 0x8d, 0xad, 0xaa, 0xcb, 0x29, 0xcb, 0x25,
+ 0xe4, 0x24, 0x75, 0x09, 0x32, 0xae, 0x27, 0x75, 0x6a, 0x98, 0xb7, 0xb7,
+ 0x49, 0x28, 0x53, 0x90, 0x9c, 0xa9, 0x5a, 0x56, 0x92, 0x2a, 0x13, 0x36,
+ 0x90, 0x6c, 0x02, 0xb8, 0x12, 0x05, 0xbb, 0x58, 0x69, 0x4d, 0xe8, 0xc0,
+ 0x2a, 0x00, 0x5c, 0x23, 0x26, 0x09, 0xc8, 0x79, 0x97, 0xb1, 0x6a, 0xb6,
+ 0xbc, 0x79, 0x0d, 0xd3, 0x29, 0x30, 0x03, 0xb9, 0xb4, 0x86, 0x7b, 0x2b,
+ 0x6e, 0x04, 0xc4, 0x56, 0x18, 0x50, 0x35, 0x94, 0xaa, 0x26, 0x9c, 0x9d,
+ 0x4f, 0xc3, 0x9a, 0xa1, 0xec, 0x36, 0x3f, 0xbc, 0x57, 0xc7, 0x02, 0x41,
+ 0x31, 0x0b, 0x1d, 0x6b, 0x44, 0x98, 0xe8, 0xa5, 0x56, 0xc4, 0x87, 0xb1,
+ 0x6e, 0x45, 0xaa, 0x9b, 0xb9, 0xb1, 0xa6, 0xa9, 0xbb, 0x2a, 0x44, 0x6f,
+ 0x6e, 0xe3, 0x09, 0xed, 0xec, 0x1d, 0x55, 0x32, 0x99, 0x7a, 0x86, 0x94,
+ 0xdd, 0xb9, 0x91, 0xe3, 0xa6, 0x77, 0xf8, 0x95, 0x88, 0x5f, 0xa9, 0x34,
+ 0x81, 0xea, 0x21, 0x7e, 0xba, 0x1c, 0x47, 0x4b, 0xc4, 0x81, 0x88, 0x14,
+ 0x7f, 0xdc, 0x03, 0x22, 0x81, 0xcc, 0xa5, 0xc3, 0x0d, 0x5b, 0x1c, 0x3e,
+ 0xd9, 0x09, 0x32, 0xbd, 0xe1, 0x48, 0x55, 0x92, 0x9f, 0x5e, 0x22, 0x4f,
+ 0x27, 0x38, 0x75, 0x36, 0x4b, 0x60, 0x0c, 0x7c, 0x91, 0x6b, 0xd3, 0x1f,
+ 0x66, 0xd7, 0xbb, 0xc4, 0x70, 0xc5, 0x36, 0xe6, 0x00, 0xe8, 0xda, 0x7f,
+ 0x31, 0xfb, 0x3b, 0x99, 0x45, 0x73, 0xe7, 0x77, 0x96, 0xc5, 0x39, 0x69,
+ 0x37, 0x1a, 0x5e, 0x30, 0xc3, 0x71, 0xa0, 0x32, 0x50, 0xe4, 0x26, 0xc2,
+ 0xdb, 0xda, 0x9e, 0xee, 0xd3, 0x9b, 0x6b, 0x65, 0x03, 0x38, 0x80, 0x66,
+ 0x05, 0xd3, 0x5e, 0x64, 0xd5, 0x59, 0xd0, 0x91, 0x0d, 0x2e, 0xf7, 0x51,
+ 0xaf, 0xa8, 0x33, 0x00, 0x2c, 0x90, 0x5d, 0xb5, 0x5e, 0xe0, 0x09, 0x81,
+ 0x75, 0xf2, 0x11, 0x11, 0x82, 0x2f, 0x6a, 0x9c, 0x1a, 0xab, 0xcb, 0x97,
+ 0x71, 0xcc, 0x15, 0x3b, 0xb4, 0xb6, 0x19, 0xd7, 0x35, 0xfa, 0x82, 0x23,
+ 0xd5, 0xf0, 0x6a, 0x8b, 0x2b, 0x79, 0x08, 0xe3, 0x2d, 0x8f, 0x7b, 0x82,
+ 0x9d, 0xac, 0x88, 0xc9, 0xca, 0xa4, 0x74, 0x2a, 0x12, 0xa2, 0xa4, 0x4b,
+ 0xc6, 0x14, 0x20, 0xf7, 0xd2, 0x6b, 0x87, 0xc3, 0xc8, 0x33, 0xc5, 0x6c,
+ 0xb5, 0x1a, 0xcc, 0x1b, 0xd4, 0x8c, 0x9a, 0x64, 0x8b, 0x4f, 0x60, 0x8e,
+ 0x4f, 0xeb, 0x37, 0x8e, 0x65, 0x54, 0xb7, 0x3c, 0xa8, 0xe3, 0x7b, 0x9d,
+ 0x5f, 0xa7, 0x6b, 0x44, 0x1c, 0x03, 0x97, 0xf8, 0xc0, 0xaa, 0x39, 0xb2,
+ 0x34, 0x8a, 0xbc, 0x3c, 0x6d, 0x0d, 0x13, 0xa6, 0xbd, 0x91, 0xd6, 0x2a,
+ 0x43, 0xc6, 0x51, 0x60, 0x63, 0xdf, 0xd5, 0xbf, 0x77, 0xf0, 0xf1, 0x08,
+ 0x42, 0x24, 0x43, 0x7b, 0x47, 0xa8, 0x2b, 0xa2,
+};
+
+static const u8 fips_test_mlkem768_sk[MLKEM768_SECRET_KEY_BYTES] __initconst __maybe_unused = {
+ 0xe7, 0xfa, 0x46, 0xcd, 0x07, 0xc1, 0xb4, 0xda, 0x48, 0xcb, 0x48, 0xa3,
+ 0xaf, 0xe1, 0x47, 0x9f, 0x96, 0x71, 0x1b, 0xac, 0x5c, 0xd2, 0x62, 0xc4,
+ 0x3d, 0xaa, 0xc5, 0xbc, 0x68, 0x66, 0x2d, 0x5c, 0x5d, 0x0b, 0x83, 0x55,
+ 0x1b, 0xda, 0xb9, 0xd1, 0x48, 0xa4, 0xd7, 0xe4, 0x3d, 0xc6, 0x16, 0x56,
+ 0x1f, 0xb2, 0x7f, 0x58, 0x42, 0x83, 0x8c, 0x71, 0x47, 0x22, 0x45, 0x6c,
+ 0x4c, 0x7c, 0x44, 0xb2, 0x48, 0xa8, 0xe0, 0x66, 0x37, 0xad, 0x40, 0x30,
+ 0xe5, 0x39, 0x5b, 0xd2, 0xa7, 0x81, 0x36, 0x1a, 0x9e, 0x23, 0x61, 0xc1,
+ 0x19, 0x38, 0x39, 0x04, 0x87, 0x81, 0x2e, 0xdb, 0x21, 0xbd, 0x04, 0x8c,
+ 0x45, 0xe9, 0x83, 0xb2, 0x33, 0x94, 0x98, 0x5c, 0x5a, 0x2b, 0x60, 0x7e,
+ 0xfc, 0x97, 0x50, 0x57, 0x3c, 0x7e, 0x9f, 0x74, 0x90, 0xce, 0x42, 0x66,
+ 0x79, 0x59, 0x92, 0x97, 0xc5, 0x8c, 0xe6, 0x42, 0x0f, 0x2a, 0x67, 0x33,
+ 0xb9, 0x01, 0x81, 0x9a, 0xe2, 0x59, 0x9e, 0x50, 0x70, 0xf9, 0xf0, 0x77,
+ 0xb9, 0xe0, 0xb9, 0x94, 0x73, 0x80, 0x55, 0x2c, 0xc4, 0xb6, 0x24, 0x2b,
+ 0x30, 0xf2, 0x88, 0x46, 0xd5, 0x0e, 0x62, 0x38, 0xbd, 0xcb, 0x0b, 0x74,
+ 0x2f, 0x17, 0x5c, 0xfe, 0xbb, 0xab, 0xc6, 0x47, 0xa2, 0xf0, 0x32, 0x93,
+ 0x99, 0x8c, 0x7d, 0x96, 0x41, 0x13, 0xdd, 0x5a, 0x79, 0xe4, 0x7b, 0x40,
+ 0x7c, 0x6b, 0x9c, 0x23, 0x25, 0x2d, 0xa4, 0xf3, 0x40, 0x1e, 0xb5, 0x02,
+ 0x1f, 0x3c, 0xc1, 0xfd, 0x43, 0xc7, 0x87, 0x01, 0xb9, 0x72, 0x87, 0x71,
+ 0x00, 0x12, 0xb9, 0x85, 0x44, 0xac, 0x0f, 0x35, 0xc0, 0x78, 0x8b, 0x7a,
+ 0xa5, 0x14, 0x94, 0x5d, 0x41, 0x80, 0xce, 0x44, 0x9d, 0x12, 0x28, 0x01,
+ 0x77, 0x79, 0xb7, 0xf3, 0xb5, 0x16, 0xf1, 0x97, 0x58, 0x5a, 0x75, 0x29,
+ 0x90, 0x2a, 0x57, 0xd3, 0x08, 0xc0, 0x00, 0xc1, 0x42, 0xe3, 0x96, 0x08,
+ 0xd3, 0xba, 0x70, 0x81, 0x21, 0x8b, 0x3a, 0x1a, 0x29, 0xe4, 0xe3, 0x9e,
+ 0x4f, 0x21, 0x0c, 0x27, 0xa9, 0x2c, 0x30, 0xd0, 0xbf, 0x61, 0x1a, 0x28,
+ 0x03, 0x3a, 0x44, 0xc7, 0x13, 0xa1, 0x6a, 0xb2, 0x0a, 0x59, 0x71, 0x7d,
+ 0xa1, 0xe7, 0x1a, 0xae, 0x78, 0x67, 0x5b, 0xfc, 0x66, 0xa3, 0x97, 0x6d,
+ 0xfb, 0xac, 0x7e, 0xab, 0xfc, 0x89, 0x61, 0x4a, 0x24, 0xe7, 0xe0, 0x9e,
+ 0xf7, 0xe4, 0x48, 0xef, 0xc5, 0x28, 0xa8, 0x23, 0x30, 0xc2, 0xcb, 0xa2,
+ 0x12, 0x8b, 0xc8, 0x17, 0x2a, 0xb6, 0x97, 0x88, 0xba, 0xee, 0x2c, 0x8f,
+ 0x72, 0x1a, 0x30, 0x81, 0xb3, 0xb7, 0x83, 0x5c, 0x9d, 0xfd, 0xf9, 0xc8,
+ 0x37, 0x25, 0xa9, 0x42, 0xfc, 0xb0, 0xe4, 0x06, 0x45, 0xeb, 0x4b, 0x25,
+ 0x8d, 0x62, 0x2c, 0x7c, 0xd5, 0x78, 0x0b, 0x27, 0x09, 0x14, 0x42, 0x69,
+ 0xeb, 0x82, 0x3f, 0xa8, 0xa8, 0x55, 0x40, 0xa2, 0x74, 0x9a, 0xa1, 0x71,
+ 0x31, 0x85, 0x11, 0x94, 0xbc, 0x1d, 0xd9, 0x07, 0x08, 0xf8, 0x09, 0xc3,
+ 0x0d, 0xd8, 0x99, 0x66, 0x82, 0x23, 0xdd, 0x20, 0x06, 0xaf, 0x02, 0x42,
+ 0x39, 0x50, 0x0c, 0x7d, 0x85, 0x6f, 0xef, 0xb7, 0x16, 0xcd, 0x98, 0x8a,
+ 0x2d, 0xea, 0x8a, 0x2f, 0xc1, 0x03, 0x6a, 0x22, 0x3e, 0x24, 0x01, 0x80,
+ 0x99, 0x91, 0x22, 0x65, 0x5a, 0xcc, 0xee, 0xa5, 0x12, 0xc1, 0x42, 0xb2,
+ 0xf9, 0xd7, 0xc1, 0xd3, 0x92, 0x1a, 0x34, 0xe9, 0x49, 0x04, 0xf0, 0x82,
+ 0x97, 0xd8, 0x9d, 0xd9, 0xf6, 0x3f, 0xd3, 0x6b, 0xa8, 0x03, 0xf9, 0x9b,
+ 0x1a, 0xf5, 0x19, 0x71, 0xf5, 0xaf, 0xde, 0xf3, 0x09, 0x61, 0x8a, 0x8b,
+ 0x9c, 0xc2, 0x5f, 0xf4, 0x10, 0x90, 0x0a, 0x43, 0x71, 0x8a, 0xb1, 0xbc,
+ 0x4a, 0xc2, 0xac, 0x9a, 0x57, 0x61, 0x7f, 0x15, 0x0e, 0x2a, 0xc8, 0x3a,
+ 0x06, 0x03, 0x2e, 0x37, 0x3a, 0x92, 0xf1, 0x5a, 0xc4, 0xaa, 0xa8, 0x70,
+ 0x76, 0xe7, 0x2c, 0xa5, 0x21, 0x12, 0xda, 0x95, 0xc0, 0x0c, 0x93, 0x28,
+ 0xea, 0xfa, 0x70, 0x89, 0x01, 0x32, 0x4a, 0xa3, 0x4d, 0xe9, 0xc3, 0x92,
+ 0x78, 0x83, 0x49, 0x2d, 0x79, 0x1a, 0x97, 0x45, 0x2d, 0x9d, 0x18, 0x3c,
+ 0x37, 0xb4, 0x45, 0x6c, 0x84, 0x21, 0x13, 0xac, 0x6c, 0xfe, 0x59, 0x7e,
+ 0x51, 0xe4, 0x82, 0x34, 0xe0, 0x99, 0x7e, 0xe9, 0xcf, 0x45, 0x6a, 0x5f,
+ 0xa3, 0x26, 0x40, 0xe1, 0xfb, 0x1e, 0xa6, 0x49, 0x14, 0xff, 0x62, 0x68,
+ 0xe1, 0x2b, 0x65, 0x75, 0xd5, 0xb8, 0x2e, 0x2b, 0x81, 0x7a, 0x50, 0x1d,
+ 0x2b, 0x07, 0x4f, 0x91, 0xfb, 0x3c, 0xa2, 0xac, 0x6e, 0x2a, 0x39, 0x3e,
+ 0xa6, 0xc1, 0x52, 0x7f, 0x95, 0x0c, 0xdd, 0xa9, 0x2d, 0xe8, 0xa1, 0xa7,
+ 0x60, 0x10, 0x36, 0xa7, 0x33, 0xb5, 0x6d, 0x4b, 0x51, 0xfc, 0xac, 0xc7,
+ 0x03, 0xd1, 0x25, 0xdc, 0x87, 0x12, 0x68, 0xc5, 0x93, 0xb1, 0xb7, 0x97,
+ 0x47, 0x00, 0x15, 0xe7, 0xba, 0x5c, 0x6d, 0x07, 0x5a, 0x0d, 0x93, 0x56,
+ 0xac, 0xb0, 0x5a, 0x1a, 0x80, 0x62, 0x5e, 0xb7, 0xbb, 0x0e, 0x6b, 0xc4,
+ 0xdb, 0xe0, 0xb4, 0x6e, 0x2c, 0x81, 0x53, 0x5c, 0xcd, 0x9b, 0xb2, 0xa5,
+ 0x4f, 0x02, 0x9d, 0xb1, 0x8b, 0x3c, 0xf1, 0x6c, 0xc3, 0x03, 0x00, 0xbe,
+ 0x16, 0x73, 0x0e, 0x39, 0x49, 0x57, 0xcf, 0xe7, 0xa3, 0x88, 0xfc, 0x07,
+ 0xf8, 0x58, 0x3e, 0xac, 0xb0, 0x1f, 0x22, 0x91, 0xc0, 0xf3, 0xf9, 0x13,
+ 0xb3, 0x6a, 0x0e, 0x3f, 0x85, 0x43, 0x6d, 0x78, 0x43, 0xcc, 0x49, 0x23,
+ 0x1e, 0x09, 0x16, 0x70, 0x3b, 0x79, 0x1c, 0x72, 0x0e, 0xed, 0xa5, 0x5b,
+ 0x47, 0xaa, 0xbb, 0xcc, 0x4c, 0x76, 0x72, 0x82, 0x84, 0xee, 0x39, 0x3b,
+ 0xb7, 0x81, 0xab, 0x3e, 0xbc, 0x59, 0x60, 0x94, 0x96, 0xac, 0xbc, 0x44,
+ 0x22, 0x9a, 0x61, 0xf5, 0xa1, 0x6c, 0x9d, 0xec, 0x6d, 0x0f, 0x5c, 0x81,
+ 0xb9, 0x9c, 0x0f, 0xbe, 0xab, 0x47, 0xe6, 0xdc, 0x92, 0xb7, 0xa7, 0x92,
+ 0xa7, 0xa1, 0x15, 0xc5, 0xf3, 0x8b, 0x22, 0x3b, 0x3e, 0x3e, 0xa1, 0x43,
+ 0xd2, 0x09, 0x83, 0xff, 0x1b, 0x8b, 0xd3, 0x75, 0x1e, 0x27, 0x6a, 0x35,
+ 0x95, 0xc3, 0x36, 0xfc, 0x38, 0x9d, 0x46, 0x87, 0xb1, 0xf8, 0x03, 0x24,
+ 0x37, 0x98, 0xca, 0x37, 0x35, 0x80, 0x9f, 0x06, 0x14, 0xe1, 0xd8, 0x6d,
+ 0x1d, 0xab, 0x06, 0xce, 0xe8, 0x95, 0xaa, 0xa7, 0x46, 0xeb, 0xb5, 0x96,
+ 0xce, 0x12, 0x68, 0xa5, 0xc3, 0xcd, 0xbf, 0xb2, 0x57, 0xe9, 0xfc, 0xa7,
+ 0x52, 0x13, 0x38, 0x70, 0x33, 0x0c, 0x8e, 0x7c, 0x22, 0x29, 0xe4, 0x21,
+ 0x29, 0xe1, 0x58, 0x60, 0xb5, 0xc7, 0xe9, 0x17, 0xaf, 0x4b, 0x30, 0x42,
+ 0x9c, 0xdc, 0xb6, 0x86, 0x57, 0x54, 0x6c, 0x06, 0x73, 0xe5, 0xa8, 0x46,
+ 0xec, 0xa7, 0x64, 0xcc, 0x1c, 0xb1, 0x77, 0xb1, 0x3b, 0x9d, 0x33, 0x21,
+ 0x3e, 0x20, 0x39, 0xe8, 0xf1, 0x8b, 0x80, 0xd2, 0x8e, 0xc0, 0x73, 0x7c,
+ 0xa0, 0x5c, 0x10, 0x14, 0x5b, 0x00, 0xd0, 0x44, 0x6e, 0x49, 0xf3, 0x0e,
+ 0xfa, 0x23, 0x9e, 0xef, 0x00, 0x7f, 0x37, 0x82, 0x85, 0x92, 0x75, 0x04,
+ 0xa3, 0x25, 0x89, 0xf4, 0x3a, 0x77, 0x72, 0xe3, 0x6c, 0x52, 0x05, 0x9c,
+ 0xad, 0xe9, 0x54, 0x08, 0xf8, 0x17, 0x9f, 0x89, 0x4e, 0xc2, 0xf5, 0xc7,
+ 0xef, 0xda, 0x05, 0x14, 0x37, 0x58, 0x09, 0xf6, 0x2d, 0x93, 0x11, 0xa7,
+ 0x79, 0xbb, 0xb1, 0xec, 0x50, 0x35, 0x5a, 0x87, 0x75, 0xa6, 0xc7, 0x8b,
+ 0xe5, 0x63, 0xc5, 0x3f, 0xfb, 0x51, 0x9b, 0xa4, 0x9f, 0x60, 0x21, 0xc5,
+ 0xd7, 0xca, 0x5f, 0x94, 0x13, 0x70, 0x00, 0x48, 0x40, 0x58, 0x4a, 0x7b,
+ 0x58, 0x78, 0xa3, 0xfc, 0x38, 0x49, 0x3a, 0xd0, 0x72, 0x99, 0xb5, 0x58,
+ 0x81, 0x62, 0x83, 0xd3, 0x7a, 0x03, 0x08, 0x76, 0xb3, 0xcf, 0x27, 0xa9,
+ 0x88, 0xc8, 0xae, 0x35, 0xdb, 0x0d, 0x50, 0x6a, 0xa8, 0xa2, 0x31, 0x4f,
+ 0x3a, 0x6b, 0x5f, 0x04, 0x37, 0x54, 0xbe, 0x51, 0xb8, 0x92, 0xb7, 0x02,
+ 0x5a, 0x06, 0x50, 0xee, 0x40, 0xa5, 0xff, 0x76, 0x6c, 0x63, 0x09, 0xc9,
+ 0x26, 0x5c, 0x16, 0x61, 0xca, 0x5a, 0xe5, 0xb1, 0xaa, 0xe1, 0xf3, 0x84,
+ 0xd3, 0x29, 0x25, 0x9a, 0xf7, 0x4c, 0xed, 0xc6, 0x0f, 0x29, 0xfc, 0x04,
+ 0xb2, 0x28, 0x24, 0xe3, 0x78, 0x86, 0xed, 0x08, 0x49, 0x5c, 0xd6, 0xb5,
+ 0x50, 0x4c, 0xbe, 0x0e, 0x13, 0xcc, 0x94, 0x71, 0xcf, 0x99, 0x04, 0x3e,
+ 0x6e, 0x75, 0x07, 0x83, 0x2c, 0x1b, 0x47, 0xcb, 0xac, 0x22, 0x68, 0xa8,
+ 0x78, 0x20, 0x32, 0x02, 0x30, 0x23, 0x8e, 0x44, 0x7a, 0xcf, 0xa9, 0x9b,
+ 0x63, 0x32, 0xb7, 0x53, 0x1c, 0x7c, 0xe5, 0x42, 0x03, 0x1b, 0x93, 0xca,
+ 0x14, 0x25, 0x8f, 0x5f, 0x98, 0xb3, 0x0c, 0x87, 0x3f, 0x6d, 0x01, 0xb2,
+ 0xc5, 0x85, 0x3f, 0x34, 0x02, 0x69, 0x18, 0x04, 0x80, 0x47, 0x08, 0xa7,
+ 0xec, 0xfa, 0x5b, 0xa5, 0x98, 0x0f, 0xef, 0x88, 0x36, 0xce, 0xf5, 0xa8,
+ 0x2d, 0xac, 0x83, 0x75, 0xe0, 0x62, 0x34, 0xcb, 0xbe, 0xd6, 0x4c, 0xc5,
+ 0xa4, 0xb2, 0xac, 0x06, 0x84, 0xa0, 0xb4, 0xd9, 0x86, 0x3e, 0x66, 0x44,
+ 0xa8, 0x33, 0x48, 0x49, 0xc0, 0xcf, 0xdb, 0xfb, 0x85, 0x1b, 0x84, 0xb0,
+ 0xe4, 0x8c, 0x6f, 0xb0, 0x48, 0x01, 0xd0, 0x52, 0x2f, 0xd5, 0x76, 0x68,
+ 0x9b, 0x16, 0x44, 0x67, 0x60, 0x4a, 0xb4, 0xda, 0x2e, 0x34, 0xab, 0xcf,
+ 0x76, 0xdc, 0x92, 0x44, 0x67, 0x4c, 0x4e, 0x57, 0x30, 0xf5, 0x78, 0x7c,
+ 0x67, 0x90, 0x21, 0x42, 0x4c, 0x13, 0x40, 0x15, 0x4d, 0x24, 0x49, 0x33,
+ 0x0d, 0xbc, 0x94, 0x0e, 0xba, 0x7b, 0xe1, 0x03, 0x99, 0xec, 0x94, 0x0a,
+ 0x6b, 0x35, 0x40, 0x60, 0x14, 0x99, 0x66, 0x50, 0xb1, 0x48, 0xa1, 0x89,
+ 0x43, 0x47, 0x7f, 0x5e, 0x95, 0x0f, 0x41, 0xa0, 0x1a, 0xb6, 0x34, 0x88,
+ 0xfa, 0xf2, 0x06, 0xf0, 0xf2, 0x25, 0xa7, 0x31, 0x09, 0x67, 0x77, 0x84,
+ 0x79, 0x1b, 0x6d, 0xec, 0x98, 0x0e, 0x60, 0xeb, 0x8f, 0x96, 0xbb, 0xbf,
+ 0x47, 0x22, 0x22, 0xb0, 0x24, 0x8a, 0x12, 0xf9, 0x61, 0x88, 0xa1, 0x7d,
+ 0x68, 0xb8, 0x3e, 0x52, 0xc9, 0x82, 0x7b, 0xa2, 0xc6, 0xbc, 0x82, 0x75,
+ 0x0b, 0x31, 0xae, 0x46, 0xb9, 0x82, 0x7a, 0x28, 0x77, 0xfc, 0xb3, 0x38,
+ 0x00, 0x3a, 0x68, 0xb8, 0xc1, 0xaa, 0x9c, 0x96, 0x6b, 0x55, 0xd2, 0x03,
+ 0x78, 0x74, 0x11, 0x57, 0x77, 0x05, 0xf6, 0xdb, 0x15, 0x59, 0xf9, 0xb4,
+ 0x25, 0x8a, 0x57, 0x18, 0x20, 0x50, 0x5d, 0x83, 0xc1, 0x00, 0x9a, 0xa2,
+ 0xb2, 0x84, 0xb3, 0x75, 0xa7, 0xac, 0x61, 0x6a, 0x1b, 0x29, 0x42, 0xa2,
+ 0xb4, 0x94, 0xc4, 0x3e, 0xb3, 0x70, 0xc1, 0xb9, 0x4a, 0xcd, 0x52, 0x2b,
+ 0x20, 0x8c, 0x17, 0xc7, 0x35, 0x07, 0x7e, 0x60, 0x2f, 0x41, 0x41, 0x0c,
+ 0xfb, 0x1b, 0x3e, 0xa1, 0x09, 0x21, 0x58, 0xc3, 0xb6, 0xea, 0x43, 0xcc,
+ 0x7a, 0xa7, 0x03, 0x09, 0xa6, 0x0f, 0xa9, 0x64, 0x47, 0x21, 0x33, 0x2a,
+ 0x0d, 0xc4, 0x55, 0xbd, 0xb5, 0x63, 0xd5, 0xe6, 0x6e, 0xe2, 0x12, 0x10,
+ 0x87, 0x2a, 0x59, 0x8d, 0x13, 0xa9, 0x51, 0x83, 0x11, 0x9a, 0x50, 0x7a,
+ 0x76, 0x34, 0x3b, 0x45, 0x4b, 0xb3, 0x20, 0x4b, 0xad, 0x83, 0x46, 0x9b,
+ 0x56, 0x27, 0x52, 0xa7, 0x3a, 0x00, 0xe5, 0xe8, 0xc5, 0xa6, 0xd9, 0x87,
+ 0x0a, 0xca, 0x93, 0x78, 0x69, 0x05, 0xc2, 0x0b, 0x85, 0x02, 0xf4, 0x5d,
+ 0x91, 0x14, 0xa8, 0xd2, 0xc7, 0xbd, 0x2e, 0xe0, 0x08, 0x7f, 0xb7, 0x80,
+ 0xfa, 0x50, 0x78, 0x58, 0x8b, 0x02, 0x6f, 0x14, 0x19, 0x6c, 0xc0, 0x68,
+ 0xf6, 0xa7, 0x5e, 0x7b, 0x4a, 0x44, 0x3a, 0xea, 0x41, 0x42, 0x5c, 0xaf,
+ 0x2d, 0xa2, 0x12, 0xa3, 0xe6, 0xad, 0xf6, 0x44, 0x51, 0x64, 0xc2, 0x90,
+ 0x8f, 0x40, 0x9d, 0x43, 0x96, 0x5e, 0xe4, 0x0b, 0x89, 0xd6, 0x54, 0xc8,
+ 0x41, 0x30, 0x97, 0x48, 0x6a, 0x9b, 0x90, 0xe2, 0xb0, 0x44, 0x74, 0x38,
+ 0x91, 0x18, 0x36, 0xb7, 0xca, 0x84, 0xe7, 0x6b, 0x7a, 0x8e, 0xe6, 0x00,
+ 0xa1, 0x3b, 0x55, 0xa8, 0xb8, 0x3d, 0xa0, 0xb2, 0x08, 0x25, 0x38, 0x2b,
+ 0xda, 0xc3, 0xbe, 0x25, 0x04, 0x60, 0x24, 0xc7, 0x88, 0x76, 0x40, 0x53,
+ 0xf1, 0x92, 0x2c, 0xfa, 0xa2, 0xa1, 0xb8, 0x16, 0x22, 0x30, 0xa3, 0x6d,
+ 0xbb, 0x36, 0xcf, 0x8e, 0x62, 0xc8, 0xa1, 0xb1, 0x5f, 0x9d, 0x51, 0x90,
+ 0x07, 0x0a, 0xb4, 0xf5, 0x70, 0xa3, 0xba, 0xf8, 0x63, 0x7c, 0x04, 0xad,
+ 0x19, 0x51, 0xa4, 0xb4, 0x61, 0x0b, 0x5b, 0x7a, 0xb2, 0x33, 0x56, 0xc3,
+ 0x9e, 0x36, 0xa8, 0x12, 0x85, 0x58, 0xa2, 0x59, 0x14, 0xe1, 0xa8, 0xaf,
+ 0x76, 0xe4, 0x77, 0x12, 0xf8, 0x54, 0x38, 0xb1, 0x9e, 0x8d, 0x04, 0xce,
+ 0x92, 0x35, 0x20, 0xd3, 0xab, 0x0a, 0xc4, 0xe4, 0x08, 0x58, 0xf2, 0x36,
+ 0x2d, 0xd4, 0x9d, 0xb4, 0x46, 0xb7, 0xed, 0xf7, 0xa0, 0xdd, 0x4c, 0x83,
+ 0xaa, 0x30, 0x7e, 0xfa, 0x87, 0x76, 0xb3, 0xb8, 0xc7, 0xa1, 0x0a, 0x1f,
+ 0xdf, 0xe6, 0x8e, 0xda, 0x1b, 0x9c, 0xd4, 0x2a, 0xc0, 0xce, 0xc4, 0x76,
+ 0x0c, 0x25, 0x93, 0x18, 0x08, 0x2c, 0x18, 0x41, 0x5d, 0x2c, 0xc2, 0x93,
+ 0x67, 0xf6, 0x06, 0xee, 0xea, 0x1c, 0x01, 0x45, 0x38, 0x9e, 0xa2, 0x88,
+ 0xf3, 0x83, 0xa6, 0x8d, 0xd6, 0x20, 0x12, 0x7a, 0xac, 0xa8, 0x8b, 0xcb,
+ 0x6b, 0x42, 0x45, 0xa6, 0x64, 0x4f, 0x05, 0xa6, 0xb9, 0xf1, 0x16, 0x3d,
+ 0x38, 0xe1, 0x7e, 0x57, 0xcb, 0x27, 0xcc, 0x5c, 0x46, 0x7b, 0x2c, 0xca,
+ 0x94, 0xf9, 0x74, 0xcf, 0xd8, 0x15, 0x0d, 0x52, 0x94, 0xe1, 0x52, 0x20,
+ 0x0b, 0x39, 0xba, 0xfa, 0x46, 0x9b, 0x9c, 0xfa, 0xc3, 0xac, 0xeb, 0x7b,
+ 0x56, 0xb4, 0x7b, 0x13, 0xf1, 0x71, 0x81, 0xea, 0x95, 0x87, 0x78, 0x10,
+ 0xa7, 0x77, 0xae, 0x9d, 0x15, 0x60, 0xed, 0x51, 0x19, 0xab, 0x09, 0xb7,
+ 0x6a, 0xb1, 0x20, 0xe3, 0xc6, 0x19, 0x61, 0xd5, 0x8c, 0x3a, 0x93, 0x16,
+ 0x69, 0xa0, 0x5f, 0xd2, 0x88, 0x2f, 0x51, 0x97, 0xb7, 0xc3, 0xb0, 0x47,
+ 0xa1, 0x97, 0xaa, 0x82, 0x0c, 0x20, 0x8a, 0x5c, 0xb5, 0xbc, 0x40, 0xbe,
+ 0xb8, 0x25, 0x88, 0xf5, 0x61, 0x0b, 0x0b, 0x35, 0x4e, 0xb8, 0x50, 0x39,
+ 0xb0, 0xa4, 0x1d, 0xfd, 0x39, 0x24, 0x8f, 0x16, 0x5a, 0xbb, 0x02, 0x7a,
+ 0xfd, 0x68, 0x17, 0xe0, 0x0c, 0x8d, 0xad, 0xaa, 0xcb, 0x29, 0xcb, 0x25,
+ 0xe4, 0x24, 0x75, 0x09, 0x32, 0xae, 0x27, 0x75, 0x6a, 0x98, 0xb7, 0xb7,
+ 0x49, 0x28, 0x53, 0x90, 0x9c, 0xa9, 0x5a, 0x56, 0x92, 0x2a, 0x13, 0x36,
+ 0x90, 0x6c, 0x02, 0xb8, 0x12, 0x05, 0xbb, 0x58, 0x69, 0x4d, 0xe8, 0xc0,
+ 0x2a, 0x00, 0x5c, 0x23, 0x26, 0x09, 0xc8, 0x79, 0x97, 0xb1, 0x6a, 0xb6,
+ 0xbc, 0x79, 0x0d, 0xd3, 0x29, 0x30, 0x03, 0xb9, 0xb4, 0x86, 0x7b, 0x2b,
+ 0x6e, 0x04, 0xc4, 0x56, 0x18, 0x50, 0x35, 0x94, 0xaa, 0x26, 0x9c, 0x9d,
+ 0x4f, 0xc3, 0x9a, 0xa1, 0xec, 0x36, 0x3f, 0xbc, 0x57, 0xc7, 0x02, 0x41,
+ 0x31, 0x0b, 0x1d, 0x6b, 0x44, 0x98, 0xe8, 0xa5, 0x56, 0xc4, 0x87, 0xb1,
+ 0x6e, 0x45, 0xaa, 0x9b, 0xb9, 0xb1, 0xa6, 0xa9, 0xbb, 0x2a, 0x44, 0x6f,
+ 0x6e, 0xe3, 0x09, 0xed, 0xec, 0x1d, 0x55, 0x32, 0x99, 0x7a, 0x86, 0x94,
+ 0xdd, 0xb9, 0x91, 0xe3, 0xa6, 0x77, 0xf8, 0x95, 0x88, 0x5f, 0xa9, 0x34,
+ 0x81, 0xea, 0x21, 0x7e, 0xba, 0x1c, 0x47, 0x4b, 0xc4, 0x81, 0x88, 0x14,
+ 0x7f, 0xdc, 0x03, 0x22, 0x81, 0xcc, 0xa5, 0xc3, 0x0d, 0x5b, 0x1c, 0x3e,
+ 0xd9, 0x09, 0x32, 0xbd, 0xe1, 0x48, 0x55, 0x92, 0x9f, 0x5e, 0x22, 0x4f,
+ 0x27, 0x38, 0x75, 0x36, 0x4b, 0x60, 0x0c, 0x7c, 0x91, 0x6b, 0xd3, 0x1f,
+ 0x66, 0xd7, 0xbb, 0xc4, 0x70, 0xc5, 0x36, 0xe6, 0x00, 0xe8, 0xda, 0x7f,
+ 0x31, 0xfb, 0x3b, 0x99, 0x45, 0x73, 0xe7, 0x77, 0x96, 0xc5, 0x39, 0x69,
+ 0x37, 0x1a, 0x5e, 0x30, 0xc3, 0x71, 0xa0, 0x32, 0x50, 0xe4, 0x26, 0xc2,
+ 0xdb, 0xda, 0x9e, 0xee, 0xd3, 0x9b, 0x6b, 0x65, 0x03, 0x38, 0x80, 0x66,
+ 0x05, 0xd3, 0x5e, 0x64, 0xd5, 0x59, 0xd0, 0x91, 0x0d, 0x2e, 0xf7, 0x51,
+ 0xaf, 0xa8, 0x33, 0x00, 0x2c, 0x90, 0x5d, 0xb5, 0x5e, 0xe0, 0x09, 0x81,
+ 0x75, 0xf2, 0x11, 0x11, 0x82, 0x2f, 0x6a, 0x9c, 0x1a, 0xab, 0xcb, 0x97,
+ 0x71, 0xcc, 0x15, 0x3b, 0xb4, 0xb6, 0x19, 0xd7, 0x35, 0xfa, 0x82, 0x23,
+ 0xd5, 0xf0, 0x6a, 0x8b, 0x2b, 0x79, 0x08, 0xe3, 0x2d, 0x8f, 0x7b, 0x82,
+ 0x9d, 0xac, 0x88, 0xc9, 0xca, 0xa4, 0x74, 0x2a, 0x12, 0xa2, 0xa4, 0x4b,
+ 0xc6, 0x14, 0x20, 0xf7, 0xd2, 0x6b, 0x87, 0xc3, 0xc8, 0x33, 0xc5, 0x6c,
+ 0xb5, 0x1a, 0xcc, 0x1b, 0xd4, 0x8c, 0x9a, 0x64, 0x8b, 0x4f, 0x60, 0x8e,
+ 0x4f, 0xeb, 0x37, 0x8e, 0x65, 0x54, 0xb7, 0x3c, 0xa8, 0xe3, 0x7b, 0x9d,
+ 0x5f, 0xa7, 0x6b, 0x44, 0x1c, 0x03, 0x97, 0xf8, 0xc0, 0xaa, 0x39, 0xb2,
+ 0x34, 0x8a, 0xbc, 0x3c, 0x6d, 0x0d, 0x13, 0xa6, 0xbd, 0x91, 0xd6, 0x2a,
+ 0x43, 0xc6, 0x51, 0x60, 0x63, 0xdf, 0xd5, 0xbf, 0x77, 0xf0, 0xf1, 0x08,
+ 0x42, 0x24, 0x43, 0x7b, 0x47, 0xa8, 0x2b, 0xa2, 0x28, 0xb8, 0x74, 0x69,
+ 0xd4, 0xee, 0x89, 0x06, 0xec, 0x34, 0xdb, 0xa7, 0x6c, 0x68, 0xd8, 0xa8,
+ 0x22, 0x8d, 0xf3, 0x3c, 0xcf, 0x3a, 0x80, 0xbf, 0x15, 0x6b, 0x29, 0x53,
+ 0xa5, 0x31, 0x26, 0x9f, 0x3c, 0xb1, 0xee, 0xa9, 0x88, 0x00, 0x4b, 0x93,
+ 0x10, 0x3c, 0xfb, 0x0a, 0xee, 0xfd, 0x2a, 0x68, 0x6e, 0x01, 0xfa, 0x4a,
+ 0x58, 0xe8, 0xa3, 0x63, 0x9c, 0xa8, 0xa1, 0xe3, 0xf9, 0xae, 0x57, 0xe2,
+};
+
+static const u8 fips_test_mlkem768_ct[MLKEM768_CIPHERTEXT_BYTES] __initconst __maybe_unused = {
+ 0x1d, 0x3b, 0xe0, 0x4a, 0x6a, 0x14, 0xb4, 0x98, 0xe3, 0x65, 0xde, 0x61,
+ 0xa7, 0x00, 0xb2, 0x3a, 0x20, 0x01, 0xdb, 0xc3, 0xdc, 0xd3, 0x87, 0xcf,
+ 0xa1, 0xf1, 0x69, 0x5f, 0x11, 0x1a, 0xad, 0x47, 0x4b, 0x34, 0x22, 0x3e,
+ 0x10, 0x0a, 0x29, 0xe5, 0x3c, 0x3d, 0x16, 0xa5, 0xd7, 0x69, 0xe1, 0xa6,
+ 0x4f, 0x50, 0xbe, 0x02, 0x9a, 0x09, 0x84, 0x54, 0x23, 0x67, 0x49, 0xa2,
+ 0xc7, 0x03, 0xc0, 0x32, 0xfb, 0x19, 0x9d, 0x2d, 0xee, 0x02, 0x50, 0x90,
+ 0xbb, 0xbc, 0x60, 0x9a, 0x14, 0x28, 0x4c, 0x13, 0x5f, 0xf2, 0xf8, 0xfe,
+ 0xa9, 0x99, 0x3a, 0x36, 0x68, 0x82, 0x37, 0xb0, 0x38, 0x54, 0x7e, 0x77,
+ 0xa7, 0xd5, 0xca, 0xc0, 0x43, 0x11, 0x44, 0xde, 0x34, 0xf7, 0x79, 0x01,
+ 0xfe, 0x25, 0x27, 0xf0, 0x98, 0xd4, 0xca, 0xc1, 0x59, 0x67, 0x4b, 0x99,
+ 0x04, 0xe6, 0xa7, 0x21, 0x90, 0xc8, 0x86, 0xd1, 0x41, 0x0c, 0x5a, 0x53,
+ 0x07, 0x8b, 0xb7, 0xc1, 0x8d, 0x74, 0x9c, 0x21, 0x3d, 0xe4, 0x2d, 0xab,
+ 0xe6, 0xc8, 0x3b, 0x8c, 0x1f, 0xfe, 0xc8, 0xac, 0xb9, 0x32, 0xd0, 0x47,
+ 0x52, 0xa5, 0xe1, 0x6e, 0x0e, 0x58, 0x71, 0x1b, 0xf3, 0xc7, 0xe7, 0x01,
+ 0x19, 0x9f, 0x49, 0x57, 0x45, 0xe0, 0x74, 0x40, 0x93, 0xa8, 0xdd, 0x88,
+ 0x6c, 0xa0, 0x6b, 0xa5, 0xbb, 0x4e, 0x09, 0x36, 0x5f, 0x58, 0xe9, 0x9e,
+ 0x08, 0xe6, 0xb6, 0x90, 0x2a, 0x0c, 0x1b, 0x06, 0x42, 0xe8, 0xfd, 0x42,
+ 0x71, 0x41, 0x6a, 0x25, 0x12, 0x2e, 0x21, 0x9a, 0x19, 0xa8, 0xcd, 0xce,
+ 0x6e, 0x66, 0xaa, 0x62, 0xff, 0x30, 0x2d, 0xf8, 0x57, 0xb4, 0x65, 0x54,
+ 0x79, 0xcf, 0x29, 0x2b, 0xe9, 0xa7, 0xea, 0x31, 0x2b, 0xa4, 0x7c, 0xb7,
+ 0x7e, 0x73, 0xad, 0x9b, 0x2d, 0x4b, 0x56, 0xea, 0xca, 0x24, 0x0c, 0x35,
+ 0x27, 0x16, 0x96, 0xcd, 0xad, 0x56, 0x0f, 0x75, 0xb1, 0xb0, 0x64, 0x16,
+ 0x0c, 0x3c, 0xf5, 0xe6, 0x2f, 0x93, 0x03, 0xab, 0x95, 0x92, 0x44, 0x76,
+ 0x80, 0xd6, 0x7c, 0xc7, 0x57, 0x15, 0x92, 0xef, 0xdf, 0xb4, 0xe3, 0x05,
+ 0xc4, 0x4d, 0x5e, 0x4f, 0xe4, 0x65, 0x5e, 0x7c, 0xf9, 0xed, 0x78, 0xcf,
+ 0x70, 0xb6, 0xea, 0x82, 0x3d, 0x69, 0x24, 0x91, 0x66, 0x62, 0x0d, 0x19,
+ 0xf5, 0xe0, 0x33, 0x46, 0xf0, 0x6b, 0x9b, 0x9a, 0xef, 0x26, 0x59, 0x84,
+ 0xdd, 0x61, 0xf0, 0x2b, 0x94, 0xc6, 0xb0, 0x69, 0xdd, 0x4a, 0x44, 0x66,
+ 0x2f, 0xfd, 0x55, 0x02, 0x12, 0x68, 0x5a, 0x48, 0x3e, 0x9e, 0xd7, 0x01,
+ 0x3e, 0xc3, 0x67, 0x73, 0xaa, 0xc6, 0xdc, 0x45, 0xce, 0x1d, 0xa8, 0x5b,
+ 0x14, 0x04, 0x5c, 0x13, 0xd2, 0x59, 0x32, 0x94, 0x0e, 0xc1, 0xfd, 0xbf,
+ 0xac, 0xaa, 0x7f, 0x66, 0x03, 0xdc, 0x1a, 0xef, 0xf9, 0xf5, 0x2f, 0xae,
+ 0xdb, 0x87, 0x2f, 0xac, 0x14, 0x73, 0xf5, 0xcb, 0x8f, 0xb2, 0x0c, 0xff,
+ 0x71, 0x24, 0x85, 0x59, 0x79, 0x38, 0x90, 0x3a, 0x7d, 0x1e, 0x48, 0x95,
+ 0x22, 0x62, 0x62, 0x94, 0xd4, 0x77, 0x3b, 0x71, 0xcb, 0xdc, 0x01, 0x79,
+ 0xc8, 0x13, 0x08, 0x75, 0x5f, 0x3f, 0x3f, 0x11, 0x5b, 0xc1, 0x78, 0xc2,
+ 0x95, 0xdc, 0xfb, 0x1c, 0xcf, 0xbf, 0xb5, 0x7d, 0x3d, 0x33, 0xb6, 0x97,
+ 0x41, 0xa8, 0x2d, 0xa6, 0x13, 0x8f, 0x73, 0x15, 0x92, 0x19, 0xbd, 0x61,
+ 0x50, 0x20, 0x22, 0x6a, 0x24, 0xb5, 0x8f, 0xac, 0x81, 0x26, 0x6e, 0x46,
+ 0x8e, 0x6a, 0x6c, 0xca, 0x6b, 0xe2, 0xa5, 0x8d, 0xec, 0x46, 0xa3, 0xf7,
+ 0x42, 0xc7, 0xc4, 0x0c, 0x8d, 0x2b, 0xe6, 0x6f, 0xb0, 0x2c, 0x32, 0x4c,
+ 0x3b, 0x58, 0x5d, 0x9a, 0xd0, 0xa4, 0x60, 0x4d, 0x6c, 0xb6, 0xc9, 0xe9,
+ 0xf6, 0x79, 0x06, 0xcb, 0x29, 0x60, 0x6d, 0x1a, 0x77, 0x41, 0xbe, 0x34,
+ 0xeb, 0x24, 0xff, 0xdc, 0x1f, 0x63, 0x4b, 0xc7, 0x4f, 0x76, 0xcf, 0xca,
+ 0x0d, 0xd5, 0x7e, 0xdf, 0x62, 0x62, 0xc8, 0xe7, 0xf6, 0x1c, 0xd0, 0xcc,
+ 0xcf, 0x73, 0x91, 0xbd, 0x2b, 0xd6, 0x9b, 0xef, 0xdf, 0x35, 0xfa, 0xf3,
+ 0x69, 0x33, 0xa3, 0x9b, 0x1d, 0xf4, 0xf6, 0x94, 0x98, 0x4d, 0xbb, 0xd2,
+ 0xd4, 0x77, 0xc3, 0x8e, 0xd3, 0x29, 0xc0, 0x55, 0x8d, 0x0b, 0xc8, 0xa5,
+ 0x71, 0x6f, 0x75, 0xb4, 0x3b, 0xff, 0x17, 0xdc, 0x80, 0xa5, 0x99, 0x9f,
+ 0x39, 0x7a, 0x2f, 0x52, 0xd3, 0xdd, 0x4f, 0x7a, 0x9c, 0x6b, 0x39, 0x43,
+ 0x5b, 0x0c, 0x35, 0x73, 0xeb, 0x65, 0xf5, 0x51, 0x4a, 0xdd, 0xbf, 0x49,
+ 0x89, 0x65, 0xf2, 0x70, 0xa6, 0xc1, 0x9b, 0xaf, 0xce, 0x46, 0x6e, 0x48,
+ 0x36, 0x35, 0xf1, 0x0a, 0x90, 0x68, 0x22, 0x06, 0xa0, 0x5c, 0x0d, 0x24,
+ 0x99, 0x4b, 0x39, 0x79, 0x34, 0xfe, 0xcc, 0x8f, 0x08, 0x44, 0x63, 0x26,
+ 0xa4, 0x9d, 0x3e, 0xba, 0xec, 0x72, 0x22, 0xcd, 0xcf, 0x76, 0xb2, 0x99,
+ 0x80, 0x07, 0x52, 0x82, 0x97, 0xb4, 0xfe, 0x04, 0xa2, 0x1d, 0x8b, 0xfe,
+ 0x76, 0x27, 0xec, 0xe4, 0xbe, 0xd4, 0x89, 0xde, 0x02, 0x70, 0x57, 0x89,
+ 0xda, 0xc5, 0xe6, 0x00, 0x26, 0x48, 0x6f, 0x79, 0xde, 0x2f, 0x93, 0x14,
+ 0x76, 0xd8, 0x09, 0x9e, 0xe2, 0x7d, 0x2d, 0xd8, 0x75, 0x86, 0xb9, 0xe2,
+ 0xeb, 0x0d, 0xa2, 0x24, 0x19, 0xe9, 0x48, 0xc1, 0x87, 0x99, 0x8e, 0xde,
+ 0x7f, 0xac, 0xb0, 0x6a, 0xc9, 0x2b, 0xdc, 0xfc, 0xe4, 0x13, 0x82, 0xb2,
+ 0x9a, 0xf1, 0xcb, 0x62, 0xb4, 0x1a, 0x16, 0xd4, 0xd8, 0xd9, 0xe5, 0x95,
+ 0x80, 0xfe, 0x03, 0x3b, 0x0b, 0xee, 0xd4, 0x5e, 0x23, 0xb0, 0xee, 0x74,
+ 0xc4, 0x0c, 0xa9, 0x00, 0x4d, 0x79, 0x3a, 0x09, 0x68, 0xc9, 0xda, 0x52,
+ 0xbe, 0xad, 0x79, 0x30, 0x9c, 0x4f, 0x84, 0xb4, 0xe8, 0x52, 0x15, 0x14,
+ 0xca, 0xa0, 0x16, 0x30, 0x8d, 0x74, 0x72, 0x6f, 0x89, 0x9f, 0xbc, 0x91,
+ 0x00, 0xe3, 0x52, 0x7d, 0xd9, 0x2e, 0xca, 0xd0, 0xce, 0x35, 0xbe, 0xf8,
+ 0x69, 0x9a, 0x91, 0x1c, 0x36, 0xec, 0xf1, 0x61, 0xe7, 0xff, 0xa7, 0xa5,
+ 0x3e, 0x03, 0xa1, 0xf9, 0xe0, 0x28, 0xe3, 0xa3, 0x15, 0x5b, 0x98, 0xe8,
+ 0x19, 0x86, 0xee, 0x84, 0x90, 0x2f, 0x94, 0x3a, 0x80, 0xa4, 0x7b, 0x20,
+ 0xaa, 0xff, 0x91, 0xb2, 0x19, 0x3d, 0xd1, 0xea, 0x97, 0xf2, 0xc3, 0x8e,
+ 0x89, 0x56, 0xf0, 0x3b, 0xe2, 0x06, 0xd4, 0xb9, 0x06, 0xc7, 0x67, 0xbf,
+ 0x11, 0xfd, 0xc2, 0x96, 0xfa, 0xc2, 0x03, 0x60, 0x5c, 0x1b, 0x18, 0xe1,
+ 0x8c, 0x01, 0xa2, 0x51, 0xc7, 0x42, 0x78, 0x10, 0x12, 0x45, 0xd5, 0x8b,
+ 0xfb, 0x31, 0x98, 0x04, 0x06, 0x7d, 0xe2, 0x97, 0x65, 0xd1, 0xd4, 0x37,
+ 0x74, 0x6b, 0xa3, 0x41, 0x2c, 0x6d, 0xda, 0xd7, 0xcb, 0x01, 0xc2, 0xf3,
+ 0xad, 0x5e, 0x26, 0x51, 0x3b, 0x25, 0xb2, 0x69, 0xfc, 0xa9, 0xeb, 0x3f,
+ 0x84, 0xce, 0x48, 0xa9, 0xe9, 0x01, 0x86, 0xe3, 0xa8, 0x51, 0x50, 0x8b,
+ 0xee, 0xa2, 0x29, 0x2d, 0x8d, 0x3b, 0xb9, 0x6b, 0x5b, 0xa1, 0xf7, 0x35,
+ 0xf0, 0xd4, 0x15, 0x9a, 0x27, 0x95, 0x2c, 0x20, 0x66, 0x45, 0x67, 0x1d,
+ 0x45, 0xf2, 0xd3, 0xa7, 0x95, 0x5e, 0xe7, 0x72, 0x9b, 0x7f, 0xd4, 0xb2,
+ 0x0d, 0xba, 0x88, 0x1f, 0xac, 0x39, 0xbf, 0xa7, 0x5b, 0xfa, 0xf7, 0x60,
+ 0x39, 0xc8, 0xe3, 0x2d, 0x36, 0x38, 0x4e, 0x67, 0x18, 0xbe, 0xab, 0x6f,
+ 0xce, 0xf8, 0x38, 0xf3, 0x91, 0x3a, 0x34, 0xe0, 0x05, 0xfb, 0x4c, 0x19,
+ 0xa8, 0x6a, 0x9c, 0xff, 0x2f, 0x52, 0xff, 0xc6, 0x69, 0x4b, 0xd7, 0x72,
+ 0x14, 0x40, 0x51, 0x86, 0x7c, 0x2c, 0x21, 0x6e, 0xe7, 0x12, 0xe5, 0x9d,
+ 0x8a, 0x02, 0xaa, 0x37, 0x3e, 0xab, 0x4a, 0x13, 0x17, 0x8e, 0xa3, 0xe6,
+ 0x8b, 0xff, 0x5f, 0x96, 0xbf, 0x77, 0x68, 0x6d, 0xa2, 0xf9, 0x27, 0x14,
+ 0x34, 0xf9, 0xcc, 0x7e, 0x38, 0xfd, 0xcd, 0x06, 0x47, 0x58, 0xd6, 0x39,
+ 0xe7, 0xa1, 0xbd, 0x8a, 0x89, 0x7d, 0xf5, 0xfa, 0xb7, 0x57, 0x18, 0xe4,
+ 0x2f, 0x85, 0x8f, 0xe9, 0xba, 0x52, 0x56, 0x79,
+};
+
+static const u8 fips_test_mlkem768_ss[MLKEM_SHARED_SECRET_BYTES] __initconst __maybe_unused = {
+ 0xfe, 0x62, 0x76, 0x21, 0xfe, 0x29, 0x61, 0x86, 0xfc, 0xe3, 0x22, 0x43,
+ 0xdd, 0x55, 0x4b, 0xdd, 0xa3, 0x89, 0x71, 0xb4, 0x7f, 0x18, 0x46, 0x1f,
+ 0x21, 0x32, 0x37, 0x82, 0xdf, 0xe5, 0xff, 0x89,
+};
+
+static const u8 fips_test_mlkem768_ct_invalid[MLKEM768_CIPHERTEXT_BYTES] __initconst __maybe_unused = {
+ 0xba, 0xdf, 0xd6, 0xdf, 0xaa, 0xc3, 0x59, 0xa5, 0xef, 0xbb, 0x7b, 0xcc,
+ 0x4b, 0x59, 0xd5, 0x38, 0xdf, 0x9a, 0x04, 0x30, 0x2e, 0x10, 0xc8, 0xbc,
+ 0x1c, 0xbf, 0x1a, 0x0b, 0x3a, 0x51, 0x20, 0xea, 0x17, 0xcd, 0xa7, 0xcf,
+ 0xad, 0x76, 0x5f, 0x56, 0x23, 0x47, 0x4d, 0x36, 0x8c, 0xcc, 0xa8, 0xaf,
+ 0x00, 0x07, 0xcd, 0x9f, 0x5e, 0x4c, 0x84, 0x9f, 0x16, 0x7a, 0x58, 0x0b,
+ 0x14, 0xaa, 0xbd, 0xef, 0xae, 0xe7, 0xee, 0xf4, 0x7c, 0xb0, 0xfc, 0xa9,
+ 0x76, 0x7b, 0xe1, 0xfd, 0xa6, 0x94, 0x19, 0xdf, 0xb9, 0x27, 0xe9, 0xdf,
+ 0x07, 0x34, 0x8b, 0x19, 0x66, 0x91, 0xab, 0xae, 0xb5, 0x80, 0xb3, 0x2d,
+ 0xef, 0x58, 0x53, 0x8b, 0x8d, 0x23, 0xf8, 0x77, 0x32, 0xea, 0x63, 0xb0,
+ 0x2b, 0x4f, 0xa0, 0xf4, 0x87, 0x33, 0x60, 0xe2, 0x84, 0x19, 0x28, 0xcd,
+ 0x60, 0xdd, 0x4c, 0xee, 0x8c, 0xc0, 0xd4, 0xc9, 0x22, 0xa9, 0x61, 0x88,
+ 0xd0, 0x32, 0x67, 0x5c, 0x8a, 0xc8, 0x50, 0x93, 0x3c, 0x7a, 0xff, 0x15,
+ 0x33, 0xb9, 0x4c, 0x83, 0x4a, 0xdb, 0xb6, 0x9c, 0x61, 0x15, 0xba, 0xd4,
+ 0x69, 0x2d, 0x86, 0x19, 0xf9, 0x0b, 0x0c, 0xdf, 0x8a, 0x7b, 0x9c, 0x26,
+ 0x40, 0x29, 0xac, 0x18, 0x5b, 0x70, 0xb8, 0x3f, 0x28, 0x01, 0xf2, 0xf4,
+ 0xb3, 0xf7, 0x0c, 0x59, 0x3e, 0xa3, 0xae, 0xeb, 0x61, 0x3a, 0x7f, 0x1b,
+ 0x1d, 0xe3, 0x3f, 0xd7, 0x50, 0x81, 0xf5, 0x92, 0x30, 0x5f, 0x2e, 0x45,
+ 0x26, 0xed, 0xc0, 0x96, 0x31, 0xb1, 0x09, 0x58, 0xf4, 0x64, 0xd8, 0x89,
+ 0xf3, 0x1b, 0xa0, 0x10, 0x25, 0x0f, 0xda, 0x7f, 0x13, 0x68, 0xec, 0x29,
+ 0x67, 0xfc, 0x84, 0xef, 0x2a, 0xe9, 0xaf, 0xf2, 0x68, 0xe0, 0xb1, 0x70,
+ 0x0a, 0xff, 0xc6, 0x82, 0x0b, 0x52, 0x3a, 0x3d, 0x91, 0x71, 0x35, 0xf2,
+ 0xdf, 0xf2, 0xee, 0x06, 0xbf, 0xe7, 0x2b, 0x31, 0x24, 0x72, 0x1d, 0x4a,
+ 0x26, 0xc0, 0x4e, 0x53, 0xa7, 0x5e, 0x30, 0xe7, 0x3a, 0x7a, 0x9c, 0x4a,
+ 0x95, 0xd9, 0x1c, 0x55, 0xd4, 0x95, 0xe9, 0xf5, 0x1d, 0xd0, 0xb5, 0xe9,
+ 0xd8, 0x3c, 0x6d, 0x5e, 0x8c, 0xe8, 0x03, 0xaa, 0x62, 0xb8, 0xd6, 0x54,
+ 0xdb, 0x53, 0xd0, 0x9b, 0x8d, 0xcf, 0xf2, 0x73, 0xcd, 0xfe, 0xb5, 0x73,
+ 0xfa, 0xd8, 0xbc, 0xd4, 0x55, 0x78, 0xbe, 0xc2, 0xe7, 0x70, 0xd0, 0x1e,
+ 0xfd, 0xe8, 0x6e, 0x72, 0x1a, 0x3f, 0x7c, 0x6c, 0xce, 0x27, 0x5d, 0xab,
+ 0xe6, 0xe2, 0x14, 0x3f, 0x1a, 0xf1, 0x8d, 0xa7, 0xef, 0xdd, 0xc4, 0xc7,
+ 0xb7, 0x0b, 0x5e, 0x34, 0x5d, 0xb9, 0x3c, 0xc9, 0x36, 0xbe, 0xa3, 0x23,
+ 0x49, 0x1c, 0xcb, 0x38, 0xa3, 0x88, 0xf5, 0x46, 0xa9, 0xff, 0x00, 0xdd,
+ 0x4e, 0x13, 0x00, 0xb9, 0xb2, 0x15, 0x3d, 0x20, 0x41, 0xd2, 0x05, 0xb4,
+ 0x43, 0xe4, 0x1b, 0x45, 0xa6, 0x53, 0xf2, 0xa5, 0xc4, 0x49, 0x2c, 0x1a,
+ 0xdd, 0x54, 0x45, 0x12, 0xdd, 0xa2, 0x52, 0x98, 0x33, 0x46, 0x2b, 0x71,
+ 0xa4, 0x1a, 0x45, 0xbe, 0x97, 0x29, 0x0b, 0x6f, 0x4c, 0xff, 0xda, 0x2c,
+ 0xf9, 0x90, 0x05, 0x16, 0x34, 0xa4, 0xb1, 0xed, 0xf6, 0x11, 0x4f, 0xb4,
+ 0x90, 0x83, 0xc1, 0xfa, 0x3b, 0x30, 0x2e, 0xe0, 0x97, 0xf0, 0x51, 0x26,
+ 0x6b, 0xe6, 0x9d, 0xc7, 0x16, 0xfd, 0xee, 0xf9, 0x1b, 0x0d, 0x4a, 0xb2,
+ 0xde, 0x52, 0x55, 0x50, 0xbf, 0x80, 0xdc, 0x8a, 0x68, 0x4b, 0xc3, 0xb5,
+ 0xa4, 0xd4, 0x6b, 0x7e, 0xfa, 0xe7, 0xaf, 0xdc, 0x62, 0x92, 0x98, 0x8d,
+ 0xc9, 0xac, 0xae, 0x03, 0xf8, 0x63, 0x44, 0x86, 0xc1, 0xab, 0xe2, 0x78,
+ 0x1a, 0xae, 0x4c, 0x02, 0xf3, 0x46, 0x0d, 0x2c, 0xd4, 0xe6, 0xa4, 0x63,
+ 0xa2, 0xba, 0x95, 0x62, 0xee, 0x62, 0x3c, 0xf0, 0xe9, 0xf8, 0x2a, 0xb4,
+ 0xd0, 0xb5, 0xc9, 0xd0, 0x40, 0xa2, 0x69, 0x36, 0x64, 0x79, 0xdf, 0xf0,
+ 0x03, 0x8a, 0xbf, 0xaf, 0x2e, 0x0f, 0xf2, 0x1f, 0x36, 0x96, 0x89, 0x72,
+ 0xe3, 0xf1, 0x04, 0xdd, 0xcb, 0xe1, 0xeb, 0x83, 0x1a, 0x87, 0xc2, 0x13,
+ 0x16, 0x2e, 0x29, 0xb3, 0x4a, 0xdf, 0xa5, 0x64, 0xd1, 0x21, 0xe9, 0xf6,
+ 0xe7, 0x72, 0x9f, 0x42, 0x03, 0xfc, 0x5c, 0x6c, 0x22, 0xfa, 0x7a, 0x73,
+ 0x50, 0xaf, 0xdd, 0xb6, 0x20, 0x92, 0x3a, 0x4a, 0x12, 0x9b, 0x8a, 0xcb,
+ 0x19, 0xea, 0x10, 0xf8, 0x18, 0xc3, 0x0e, 0x3b, 0x5b, 0x1c, 0x57, 0x1f,
+ 0xa7, 0x9e, 0x57, 0xee, 0x30, 0x43, 0x88, 0x31, 0x6a, 0x02, 0xfc, 0xd9,
+ 0x3a, 0x0d, 0x8e, 0xe0, 0x2b, 0xb8, 0x57, 0x01, 0xee, 0x4f, 0xf0, 0x97,
+ 0x53, 0x4b, 0x50, 0x2c, 0x1b, 0x12, 0xfb, 0xb9, 0x5c, 0x8c, 0xcb, 0x2f,
+ 0x54, 0x89, 0x21, 0xd9, 0x9c, 0xc7, 0xc9, 0xfe, 0x17, 0xac, 0x99, 0x1b,
+ 0x67, 0x5e, 0x63, 0x11, 0x44, 0x42, 0x3e, 0xef, 0x7a, 0x58, 0x69, 0x16,
+ 0x8d, 0xa6, 0x3d, 0x1f, 0x4c, 0x21, 0xf6, 0x50, 0xc0, 0x29, 0x23, 0xbf,
+ 0xd3, 0x96, 0xca, 0x6a, 0x5d, 0xb5, 0x41, 0x06, 0x86, 0x24, 0xcb, 0xc5,
+ 0xff, 0xe2, 0x08, 0xc0, 0xd1, 0xa7, 0x4e, 0x1a, 0x29, 0x61, 0x8d, 0x0b,
+ 0xb6, 0x00, 0x36, 0xf5, 0x24, 0x9a, 0xbf, 0xa8, 0x88, 0x98, 0xe3, 0x93,
+ 0x71, 0x8d, 0x6e, 0xfa, 0xb0, 0x5b, 0xb4, 0x12, 0x79, 0xef, 0xcd, 0x4c,
+ 0x5a, 0x0c, 0xc8, 0x37, 0xcc, 0xfc, 0x22, 0xbe, 0x4f, 0x72, 0x5c, 0x08,
+ 0x1f, 0x6a, 0xa0, 0x90, 0x74, 0x9d, 0xba, 0x70, 0x77, 0xba, 0xe8, 0xd4,
+ 0x1a, 0xf3, 0xfe, 0xc5, 0xa6, 0xee, 0x1b, 0x8a, 0xdc, 0xd2, 0x5e, 0x72,
+ 0xde, 0x36, 0x43, 0x45, 0x84, 0xef, 0x56, 0x7c, 0x64, 0x3d, 0x34, 0x42,
+ 0x94, 0xe8, 0xb2, 0x08, 0x6b, 0x87, 0xf6, 0x9c, 0x3b, 0xdc, 0x0d, 0x59,
+ 0x69, 0x85, 0x70, 0x82, 0x98, 0x7c, 0xa1, 0xc6, 0x3b, 0x71, 0x82, 0xe8,
+ 0x68, 0x98, 0xfb, 0x9b, 0x80, 0x39, 0xe7, 0x5e, 0xda, 0x21, 0x9e, 0x28,
+ 0x93, 0x31, 0x61, 0x03, 0x69, 0x27, 0x18, 0x67, 0xb1, 0x45, 0xb2, 0x90,
+ 0x82, 0x93, 0x96, 0x3c, 0xd6, 0x77, 0xc9, 0xa1, 0xae, 0x6c, 0xeb, 0x28,
+ 0x28, 0x9b, 0x25, 0x4c, 0xde, 0xb7, 0x6b, 0x12, 0xf3, 0x3c, 0xe5, 0xcf,
+ 0x37, 0x43, 0x13, 0x1b, 0xfb, 0x55, 0x0f, 0x01, 0x97, 0xbf, 0xe1, 0x6a,
+ 0xff, 0x92, 0x36, 0x72, 0x27, 0xad, 0xc5, 0x07, 0x4f, 0xe3, 0xdc, 0x0d,
+ 0x8d, 0x11, 0x62, 0x53, 0x98, 0x0a, 0x38, 0x63, 0x6b, 0xc9, 0xd2, 0x9f,
+ 0x79, 0x9b, 0xbb, 0x2d, 0x76, 0xa0, 0xa5, 0xf1, 0x38, 0xb8, 0xc7, 0x3b,
+ 0xa4, 0x84, 0xd6, 0x58, 0x87, 0x64, 0xe3, 0x31, 0xd7, 0x0c, 0x37, 0x8c,
+ 0x06, 0x41, 0xf2, 0xd9, 0xb6, 0xfd, 0x7c, 0x09, 0x0d, 0xf5, 0xa7, 0x46,
+ 0x04, 0xa1, 0x32, 0x4b, 0xa0, 0xcc, 0x5c, 0x44, 0x7b, 0x2d, 0xca, 0x64,
+ 0x4a, 0x50, 0xf1, 0xad, 0x04, 0x77, 0xa7, 0x01, 0xb9, 0x05, 0x2e, 0xe9,
+ 0xbe, 0xf2, 0x88, 0x33, 0x47, 0x63, 0x43, 0xc8, 0x2a, 0xf2, 0x9f, 0xf3,
+ 0xa9, 0xb1, 0xc4, 0xcf, 0x12, 0xde, 0x55, 0x9c, 0xb9, 0xd9, 0x41, 0x1f,
+ 0x62, 0xbe, 0xc8, 0x38, 0x12, 0x1f, 0xd7, 0x4b, 0xc1, 0xfa, 0x71, 0x2d,
+ 0x8a, 0xdd, 0x51, 0x50, 0x5c, 0x55, 0xe8, 0x9a, 0x35, 0xde, 0xaf, 0x7a,
+ 0x69, 0xdc, 0x0a, 0x18, 0xad, 0x27, 0x39, 0x60, 0x29, 0xcb, 0xf8, 0x9f,
+ 0x51, 0x3e, 0x1b, 0x8f, 0x48, 0xbc, 0x01, 0x78, 0x3d, 0x68, 0x49, 0xfb,
+ 0x32, 0xf2, 0x11, 0xa4, 0xc8, 0x7e, 0x16, 0xbc, 0xce, 0x0c, 0x41, 0x24,
+ 0x0a, 0x22, 0x3b, 0xa6, 0xd6, 0x9e, 0x0c, 0x51, 0x56, 0x9f, 0x73, 0xcb,
+ 0x10, 0x7e, 0xad, 0x84, 0xd1, 0x4d, 0xee, 0x92, 0x70, 0x2e, 0x3a, 0x95,
+ 0xeb, 0x84, 0x4c, 0x71, 0x6a, 0xec, 0x98, 0x29, 0xd0, 0x65, 0x91, 0xeb,
+ 0xd2, 0x50, 0x1a, 0x32, 0x83, 0xcc, 0x0f, 0xfc, 0x0f, 0xdc, 0xc0, 0x31,
+ 0xfe, 0x8d, 0x86, 0x5e, 0x77, 0xfa, 0xe5, 0xd6, 0xbb, 0x73, 0x81, 0x5d,
+ 0x9a, 0xe3, 0x76, 0x00, 0x6d, 0x0a, 0xe3, 0x20,
+};
+
+static const u8 fips_test_mlkem768_ss_rejected[MLKEM_SHARED_SECRET_BYTES] __initconst __maybe_unused = {
+ 0xb8, 0x77, 0xda, 0x79, 0x2d, 0x89, 0xf2, 0x80, 0x49, 0xb5, 0x90, 0x12,
+ 0x16, 0x01, 0x20, 0x2d, 0x2b, 0xc8, 0xf5, 0xf1, 0xaf, 0x83, 0x82, 0xbf,
+ 0x4f, 0x39, 0x41, 0x05, 0x0d, 0xd5, 0x17, 0x2b,
+};
diff --git a/lib/crypto/mlkem.c b/lib/crypto/mlkem.c
index 5e499e5de636..d2183a526956 100644
--- a/lib/crypto/mlkem.c
+++ b/lib/crypto/mlkem.c
@@ -10,15 +10,17 @@
#include <crypto/mlkem.h>
#include <crypto/sha3.h>
#include <crypto/utils.h>
#include <kunit/visibility.h>
#include <linux/export.h>
+#include <linux/fips.h>
#include <linux/module.h>
#include <linux/random.h>
#include <linux/slab.h>
#include <linux/string.h>
#include <linux/unaligned.h>
+#include "fips-mlkem.h"
#define Q 3329 /* The prime q = 2^8 * 13 + 1 */
#define N 256 /* Number of coefficients per ring element */
#define MAX_K 4 /* Max matrix dimension (k parameter) for any parameter set */
#define MAX_CIPHERTEXT_BYTES 1568 /* Max ciphertext length for any param set */
@@ -627,10 +629,24 @@ static int k_pke_decrypt(u8 m[32], const u8 *sk, const u8 *ct,
/* Decode the plaintext m from the polynomial w. */
compress_and_encode(m, &ws->tmp, 1);
return 0;
}
+static int mlkem_encaps_internal(u8 *ct, u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 *pk,
+ const u8 eseed[MLKEM_ESEED_BYTES],
+ const struct mlkem_parameter_set *params);
+static int mlkem_decaps(u8 ss[MLKEM_SHARED_SECRET_BYTES], const u8 *ct,
+ const u8 *sk, const struct mlkem_parameter_set *params);
+
+struct mlkem_fips140_pct_workspace {
+ u8 eseed[MLKEM_ESEED_BYTES];
+ u8 ct[MLKEM1024_CIPHERTEXT_BYTES];
+ u8 ss1[MLKEM_SHARED_SECRET_BYTES];
+ u8 ss2[MLKEM_SHARED_SECRET_BYTES];
+};
+
/*
* Generate an ML-KEM key pair (@pk, @sk) from the random seed @seed.
* The lengths of @pk and @sk are determined by the chosen @params.
*
* Reference: FIPS 203 Algorithm 16, ML-KEM.KeyGen_internal. Formally it takes
@@ -660,10 +676,31 @@ static int mlkem_keygen_internal(u8 *pk, u8 *sk,
sk_ptr += params->pk_len;
sha3_256(pk, params->pk_len, sk_ptr);
sk_ptr += SHA3_256_DIGEST_SIZE;
memcpy(sk_ptr, &seed[32], 32);
+ if (fips_enabled) {
+ /* Do the Pairwise Consistency Test required by FIPS 140-3. */
+ struct mlkem_fips140_pct_workspace *ws __free(kfree_sensitive) =
+ kmalloc_obj(*ws);
+ if (!ws)
+ return -ENOMEM; /* Out of memory, not a PCT failure */
+ get_random_bytes(ws->eseed, sizeof(ws->eseed));
+ err = mlkem_encaps_internal(ws->ct, ws->ss1, pk, ws->eseed,
+ params);
+ if (err == -ENOMEM)
+ return -ENOMEM; /* Out of memory, not a PCT failure */
+ if (err)
+ panic("mlkem: FIPS PCT failed (encaps): %d", err);
+ err = mlkem_decaps(ws->ss2, ws->ct, sk, params);
+ if (err == -ENOMEM)
+ return -ENOMEM; /* Out of memory, not a PCT failure */
+ if (err)
+ panic("mlkem: FIPS PCT failed (decaps): %d", err);
+ if (crypto_memneq(ws->ss1, ws->ss2, MLKEM_SHARED_SECRET_BYTES))
+ panic("mlkem: FIPS PCT failed (compare)");
+ }
return 0;
}
int mlkem768_keygen_internal(u8 pk[MLKEM768_PUBLIC_KEY_BYTES],
u8 sk[MLKEM768_SECRET_KEY_BYTES],
@@ -889,10 +926,88 @@ int mlkem1024_decaps(u8 ss[MLKEM_SHARED_SECRET_BYTES],
{
return mlkem_decaps(ss, ct, sk, &mlkem1024);
}
EXPORT_SYMBOL_NS_GPL(mlkem1024_decaps, "CRYPTO_INTERNAL");
+#ifdef CONFIG_CRYPTO_FIPS
+/*
+ * This function implements the ML-KEM cryptographic algorithm self-tests
+ * required by FIPS 140-3, or rather the FIPS 140-3 Implementation Guidance
+ * document which is where they are actually specified. The requirement is that
+ * for one ML-KEM parameter set, one test be done for keygen and encapsulation,
+ * and two tests be done for decapsulation to exercise both the normal case and
+ * the implicit rejection case. It's worded in such a way that the tests don't
+ * necessarily have to be "known-answer tests", but those are what we use.
+ *
+ * This is just for FIPS compliance. Normal testing done by kernel developers
+ * and integrators should use the much more comprehensive KUnit test suite.
+ */
+static int __init mlkem_mod_init(void)
+{
+ struct {
+ u8 pk[MLKEM768_PUBLIC_KEY_BYTES];
+ u8 sk[MLKEM768_SECRET_KEY_BYTES];
+ u8 ct[MLKEM768_CIPHERTEXT_BYTES];
+ u8 ss[MLKEM_SHARED_SECRET_BYTES];
+ } *bufs __free(kfree_sensitive) = NULL;
+ int err;
+
+ if (!fips_enabled)
+ return 0;
+
+ bufs = kmalloc_obj(*bufs);
+ if (!bufs)
+ panic("mlkem: fips test failed: ENOMEM");
+
+ err = mlkem768_keygen_internal(bufs->pk, bufs->sk,
+ fips_test_mlkem768_seed);
+ if (err)
+ panic("mlkem: fips test failed: keygen failed with err=%d",
+ err);
+ if (crypto_memneq(bufs->pk, fips_test_mlkem768_pk, sizeof(bufs->pk)))
+ panic("mlkem: fips test failed: wrong public key");
+ if (crypto_memneq(bufs->sk, fips_test_mlkem768_sk, sizeof(bufs->sk)))
+ panic("mlkem: fips test failed: wrong secret key");
+
+ err = mlkem768_encaps_internal(bufs->ct, bufs->ss, bufs->pk,
+ fips_test_mlkem768_eseed);
+ if (err)
+ panic("mlkem: fips test failed: encaps failed with err=%d",
+ err);
+
+ if (crypto_memneq(bufs->ct, fips_test_mlkem768_ct, sizeof(bufs->ct)))
+ panic("mlkem: fips test failed: wrong ciphertext");
+ if (crypto_memneq(bufs->ss, fips_test_mlkem768_ss, sizeof(bufs->ss)))
+ panic("mlkem: fips test failed: wrong shared secret after encaps");
+
+ memset(bufs->ss, 0, sizeof(bufs->ss));
+ err = mlkem768_decaps(bufs->ss, bufs->ct, bufs->sk);
+ if (err)
+ panic("mlkem: fips test failed: valid decaps failed with err=%d",
+ err);
+ if (crypto_memneq(bufs->ss, fips_test_mlkem768_ss, sizeof(bufs->ss)))
+ panic("mlkem: fips test failed: wrong shared secret after valid decaps");
+
+ err = mlkem768_decaps(bufs->ss, fips_test_mlkem768_ct_invalid,
+ bufs->sk);
+ if (err)
+ panic("mlkem: fips test failed: invalid decaps failed with err=%d",
+ err);
+ if (crypto_memneq(bufs->ss, fips_test_mlkem768_ss_rejected,
+ sizeof(bufs->ss)))
+ panic("mlkem: fips test failed: wrong shared secret after invalid decaps");
+
+ return 0;
+}
+subsys_initcall(mlkem_mod_init);
+
+static void __exit mlkem_mod_exit(void)
+{
+}
+module_exit(mlkem_mod_exit);
+#endif /* CONFIG_CRYPTO_FIPS */
+
#if IS_ENABLED(CONFIG_CRYPTO_LIB_MLKEM_KUNIT_TEST)
u16 mlkem_reduce_once(u16 x)
{
return reduce_once(x);
}
diff --git a/scripts/crypto/import-mlkem-testvecs.py b/scripts/crypto/import-mlkem-testvecs.py
index 1d5681ccfee1..ebc76ab218e4 100755
--- a/scripts/crypto/import-mlkem-testvecs.py
+++ b/scripts/crypto/import-mlkem-testvecs.py
@@ -11,10 +11,11 @@
# $PATH_TO_THIS_SCRIPT ./test/
#
# This script generates the following files:
#
# lib/crypto/tests/mlkem-testvecs.h
+# lib/crypto/fips-mlkem.h
import hashlib
import os
import sys
@@ -32,10 +33,13 @@ MLKEM_LENGTHS = {
"sk_len": 3168,
"ct_len": 1568,
},
}
+MLKEM_SEED_BYTES = 64
+MLKEM_ESEED_BYTES = 32
+
def print_header(file):
print("/* SPDX-License-Identifier: GPL-2.0-or-later */", file=file)
print(f"/* This file was generated by {SCRIPT_NAME} */", file=file)
print(f"/* clang-format off */", file=file)
@@ -97,10 +101,66 @@ def hash_testvecs(testvecs):
h.update(tv.ss) # From decapsulation
h.update(tv.ss_rejected)
return h.digest(length=32)
+def gen_fips_file(tv):
+ with open(LINUX_DIR + "/lib/crypto/fips-mlkem.h", "w") as file:
+ ct_len = MLKEM_LENGTHS[768]["ct_len"]
+ print_header(file)
+ attribs = " __initconst __maybe_unused"
+
+ # The test vectors were generated deterministically using the SHAKE128
+ # of an empty string as the randomness. Take the first MLKEM_SEED_BYTES
+ # + MLKEM_ESEED_BYTES to get the seeds used by the first test vector.
+ # Take an additional ct_len bytes to get the invalid ciphertext.
+ rnd_data = hashlib.shake_128().digest(
+ MLKEM_SEED_BYTES + MLKEM_ESEED_BYTES + ct_len
+ )
+
+ print_static_u8_array_definition(
+ file,
+ f"fips_test_mlkem768_seed[MLKEM_SEED_BYTES]{attribs}",
+ rnd_data[:MLKEM_SEED_BYTES],
+ )
+ print_static_u8_array_definition(
+ file,
+ f"fips_test_mlkem768_eseed[MLKEM_ESEED_BYTES]{attribs}",
+ rnd_data[MLKEM_SEED_BYTES : MLKEM_SEED_BYTES + MLKEM_ESEED_BYTES],
+ )
+ print_static_u8_array_definition(
+ file,
+ f"fips_test_mlkem768_pk[MLKEM768_PUBLIC_KEY_BYTES]{attribs}",
+ tv.pk,
+ )
+ print_static_u8_array_definition(
+ file,
+ f"fips_test_mlkem768_sk[MLKEM768_SECRET_KEY_BYTES]{attribs}",
+ tv.sk,
+ )
+ print_static_u8_array_definition(
+ file,
+ f"fips_test_mlkem768_ct[MLKEM768_CIPHERTEXT_BYTES]{attribs}",
+ tv.ct,
+ )
+ print_static_u8_array_definition(
+ file,
+ f"fips_test_mlkem768_ss[MLKEM_SHARED_SECRET_BYTES]{attribs}",
+ tv.ss,
+ )
+ print_static_u8_array_definition(
+ file,
+ f"fips_test_mlkem768_ct_invalid[MLKEM768_CIPHERTEXT_BYTES]{attribs}",
+ rnd_data[MLKEM_SEED_BYTES + MLKEM_ESEED_BYTES :],
+ )
+ print_static_u8_array_definition(
+ file,
+ f"fips_test_mlkem768_ss_rejected[MLKEM_SHARED_SECRET_BYTES]{attribs}",
+ tv.ss_rejected,
+ )
+
+
if len(sys.argv) != 2:
sys.stderr.write(f"Usage: {SCRIPT_NAME} TESTVECS_DIR\n")
sys.exit(2)
testvecs_dir = sys.argv[1]
@@ -113,5 +173,7 @@ with open(LINUX_DIR + "/lib/crypto/tests/mlkem-testvecs.h", "w") as file:
num_testvecs = min(len(testvecs), 1000)
testvecs = testvecs[:num_testvecs]
hash = hash_testvecs(testvecs)
print(f"\n#define MLKEM{paramset}_NUM_TESTVECS {num_testvecs}", file=file)
print_static_u8_array_definition(file, f"mlkem{paramset}_hash[32]", hash)
+ if paramset == 768: # FIPS only requires tests for one parameter set
+ gen_fips_file(testvecs[0])
--
2.54.0
^ permalink raw reply related
* [PATCH 2/5] lib/crypto: mlkem: Add KUnit tests for ML-KEM
From: Eric Biggers @ 2026-05-25 18:44 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, Ard Biesheuvel, Jason A . Donenfeld, Herbert Xu,
Ryan Appel, Chris Leech, Eric Biggers
In-Reply-To: <20260525184403.101818-1-ebiggers@kernel.org>
Add a KUnit test suite for ML-KEM.
For each supported ML-KEM parameter set, it includes:
- Test key generation, encapsulation, and decapsulation against the
first 1000 test vectors from the reference implementation
- Test encapsulation/decapsulation round trips
- Test validation of malformed keys
- Test that every byte of the ciphertext is validated
- Test the reduce_once(), reduce(), compress_d(), and decompress_d()
functions with all allowed inputs
- Benchmark key generation, encapsulation, and decapsulation
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
include/crypto/mlkem.h | 8 +
lib/crypto/.kunitconfig | 1 +
lib/crypto/mlkem.c | 27 ++
lib/crypto/tests/Kconfig | 9 +
lib/crypto/tests/Makefile | 1 +
lib/crypto/tests/mlkem-testvecs.h | 19 +
lib/crypto/tests/mlkem_kunit.c | 520 ++++++++++++++++++++++++
scripts/crypto/import-mlkem-testvecs.py | 117 ++++++
8 files changed, 702 insertions(+)
create mode 100644 lib/crypto/tests/mlkem-testvecs.h
create mode 100644 lib/crypto/tests/mlkem_kunit.c
create mode 100755 scripts/crypto/import-mlkem-testvecs.py
diff --git a/include/crypto/mlkem.h b/include/crypto/mlkem.h
index e33d065c5442..679bd47c8c0b 100644
--- a/include/crypto/mlkem.h
+++ b/include/crypto/mlkem.h
@@ -146,6 +146,14 @@ int mlkem1024_keygen_internal(u8 pk[MLKEM1024_PUBLIC_KEY_BYTES],
int mlkem1024_encaps_internal(u8 ct[MLKEM1024_CIPHERTEXT_BYTES],
u8 ss[MLKEM_SHARED_SECRET_BYTES],
const u8 pk[MLKEM1024_PUBLIC_KEY_BYTES],
const u8 eseed[MLKEM_ESEED_BYTES]);
+#if IS_ENABLED(CONFIG_CRYPTO_LIB_MLKEM_KUNIT_TEST)
+/* Functions exported for KUnit testing only */
+u16 mlkem_reduce_once(u16 x);
+u16 mlkem_reduce(u32 x);
+u16 mlkem_compress_d(u16 x, int d);
+u16 mlkem_decompress_d(u16 x, int d);
+#endif
+
#endif /* _CRYPTO_MLKEM_H */
diff --git a/lib/crypto/.kunitconfig b/lib/crypto/.kunitconfig
index 3efc854a2c08..32e5b4471da8 100644
--- a/lib/crypto/.kunitconfig
+++ b/lib/crypto/.kunitconfig
@@ -8,10 +8,11 @@ CONFIG_CRYPTO_LIB_BLAKE2S_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_CHACHA20POLY1305_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_CURVE25519_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_GHASH_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_MD5_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_MLDSA_KUNIT_TEST=y
+CONFIG_CRYPTO_LIB_MLKEM_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_NH_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_POLY1305_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_POLYVAL_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_SHA1_KUNIT_TEST=y
CONFIG_CRYPTO_LIB_SHA256_KUNIT_TEST=y
diff --git a/lib/crypto/mlkem.c b/lib/crypto/mlkem.c
index b800ecb49f24..5e499e5de636 100644
--- a/lib/crypto/mlkem.c
+++ b/lib/crypto/mlkem.c
@@ -8,10 +8,11 @@
*/
#include <crypto/mlkem.h>
#include <crypto/sha3.h>
#include <crypto/utils.h>
+#include <kunit/visibility.h>
#include <linux/export.h>
#include <linux/module.h>
#include <linux/random.h>
#include <linux/slab.h>
#include <linux/string.h>
@@ -888,7 +889,33 @@ int mlkem1024_decaps(u8 ss[MLKEM_SHARED_SECRET_BYTES],
{
return mlkem_decaps(ss, ct, sk, &mlkem1024);
}
EXPORT_SYMBOL_NS_GPL(mlkem1024_decaps, "CRYPTO_INTERNAL");
+#if IS_ENABLED(CONFIG_CRYPTO_LIB_MLKEM_KUNIT_TEST)
+u16 mlkem_reduce_once(u16 x)
+{
+ return reduce_once(x);
+}
+EXPORT_SYMBOL_IF_KUNIT(mlkem_reduce_once);
+
+u16 mlkem_reduce(u32 x)
+{
+ return reduce(x);
+}
+EXPORT_SYMBOL_IF_KUNIT(mlkem_reduce);
+
+u16 mlkem_compress_d(u16 x, int d)
+{
+ return compress_d(x, d);
+}
+EXPORT_SYMBOL_IF_KUNIT(mlkem_compress_d);
+
+u16 mlkem_decompress_d(u16 x, int d)
+{
+ return decompress_d(x, d);
+}
+EXPORT_SYMBOL_IF_KUNIT(mlkem_decompress_d);
+#endif /* CONFIG_CRYPTO_LIB_MLKEM_KUNIT_TEST */
+
MODULE_DESCRIPTION("ML-KEM (Module-Lattice-Based Key Encapsulation Mechanism)");
MODULE_LICENSE("GPL");
diff --git a/lib/crypto/tests/Kconfig b/lib/crypto/tests/Kconfig
index 9409c1a935c3..0a110a0733d2 100644
--- a/lib/crypto/tests/Kconfig
+++ b/lib/crypto/tests/Kconfig
@@ -67,10 +67,18 @@ config CRYPTO_LIB_MLDSA_KUNIT_TEST
default KUNIT_ALL_TESTS
select CRYPTO_LIB_BENCHMARK_VISIBLE
help
KUnit tests for the ML-DSA digital signature algorithm.
+config CRYPTO_LIB_MLKEM_KUNIT_TEST
+ tristate "KUnit tests for ML-KEM" if !KUNIT_ALL_TESTS
+ depends on KUNIT && CRYPTO_LIB_MLKEM
+ default KUNIT_ALL_TESTS
+ select CRYPTO_LIB_BENCHMARK_VISIBLE
+ help
+ KUnit tests for the ML-KEM key encapsulation mechanism.
+
config CRYPTO_LIB_NH_KUNIT_TEST
tristate "KUnit tests for NH" if !KUNIT_ALL_TESTS
depends on KUNIT && CRYPTO_LIB_NH
default KUNIT_ALL_TESTS
help
@@ -149,10 +157,11 @@ config CRYPTO_LIB_ENABLE_ALL_FOR_KUNIT
select CRYPTO_LIB_CHACHA20POLY1305
select CRYPTO_LIB_CURVE25519
select CRYPTO_LIB_GF128HASH
select CRYPTO_LIB_MD5
select CRYPTO_LIB_MLDSA
+ select CRYPTO_LIB_MLKEM
select CRYPTO_LIB_NH
select CRYPTO_LIB_POLY1305
select CRYPTO_LIB_SHA1
select CRYPTO_LIB_SHA256
select CRYPTO_LIB_SHA512
diff --git a/lib/crypto/tests/Makefile b/lib/crypto/tests/Makefile
index a739413500b6..3a73d2f33f75 100644
--- a/lib/crypto/tests/Makefile
+++ b/lib/crypto/tests/Makefile
@@ -6,10 +6,11 @@ obj-$(CONFIG_CRYPTO_LIB_BLAKE2S_KUNIT_TEST) += blake2s_kunit.o
obj-$(CONFIG_CRYPTO_LIB_CHACHA20POLY1305_KUNIT_TEST) += chacha20poly1305_kunit.o
obj-$(CONFIG_CRYPTO_LIB_CURVE25519_KUNIT_TEST) += curve25519_kunit.o
obj-$(CONFIG_CRYPTO_LIB_GHASH_KUNIT_TEST) += ghash_kunit.o
obj-$(CONFIG_CRYPTO_LIB_MD5_KUNIT_TEST) += md5_kunit.o
obj-$(CONFIG_CRYPTO_LIB_MLDSA_KUNIT_TEST) += mldsa_kunit.o
+obj-$(CONFIG_CRYPTO_LIB_MLKEM_KUNIT_TEST) += mlkem_kunit.o
obj-$(CONFIG_CRYPTO_LIB_NH_KUNIT_TEST) += nh_kunit.o
obj-$(CONFIG_CRYPTO_LIB_POLY1305_KUNIT_TEST) += poly1305_kunit.o
obj-$(CONFIG_CRYPTO_LIB_POLYVAL_KUNIT_TEST) += polyval_kunit.o
obj-$(CONFIG_CRYPTO_LIB_SHA1_KUNIT_TEST) += sha1_kunit.o
obj-$(CONFIG_CRYPTO_LIB_SHA256_KUNIT_TEST) += sha224_kunit.o sha256_kunit.o
diff --git a/lib/crypto/tests/mlkem-testvecs.h b/lib/crypto/tests/mlkem-testvecs.h
new file mode 100644
index 000000000000..207bf38529b2
--- /dev/null
+++ b/lib/crypto/tests/mlkem-testvecs.h
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/* This file was generated by import-mlkem-testvecs.py */
+/* clang-format off */
+
+#define MLKEM768_NUM_TESTVECS 1000
+
+static const u8 mlkem768_hash[32] = {
+ 0x81, 0xa6, 0x72, 0x68, 0x1c, 0x57, 0xaf, 0xc5, 0x1b, 0xdb, 0xc2, 0xfc,
+ 0x08, 0xc0, 0x4a, 0xbe, 0x01, 0xd9, 0xad, 0x5f, 0x1c, 0x77, 0x89, 0xa4,
+ 0xa5, 0x98, 0xdf, 0x9d, 0xb2, 0x91, 0xa6, 0x56,
+};
+
+#define MLKEM1024_NUM_TESTVECS 1000
+
+static const u8 mlkem1024_hash[32] = {
+ 0x64, 0x4e, 0x58, 0xca, 0x6c, 0xf9, 0xb6, 0x08, 0x07, 0x50, 0xa8, 0x76,
+ 0xe6, 0xae, 0xf5, 0xe0, 0x44, 0xce, 0xe1, 0xd9, 0x42, 0x36, 0xb6, 0xc1,
+ 0x14, 0xc7, 0x5f, 0x2c, 0x8b, 0xd7, 0x73, 0x77,
+};
diff --git a/lib/crypto/tests/mlkem_kunit.c b/lib/crypto/tests/mlkem_kunit.c
new file mode 100644
index 000000000000..2e9a9203e9a2
--- /dev/null
+++ b/lib/crypto/tests/mlkem_kunit.c
@@ -0,0 +1,520 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * KUnit tests and benchmark for ML-KEM
+ *
+ * Copyright 2026 Google LLC
+ */
+#include <crypto/mlkem.h>
+#include <crypto/sha3.h>
+#include <kunit/test.h>
+#include "mlkem-testvecs.h"
+
+#define Q 3329
+
+enum mlkem_paramset {
+ MLKEM768,
+ MLKEM1024,
+};
+
+struct mlkem_bufs {
+ u8 *pk, *sk, *ct;
+ size_t pk_len, sk_len, ct_len;
+ u8 ss[MLKEM_SHARED_SECRET_BYTES];
+ u8 seed[MLKEM_SEED_BYTES];
+ u8 eseed[MLKEM_ESEED_BYTES];
+};
+
+static const struct {
+ const char *name;
+ int k;
+ size_t pk_len;
+ size_t sk_len;
+ size_t ct_len;
+} mlkem_paramsets[] = {
+ [MLKEM768] = {
+ .name = "ML-KEM-768",
+ .k = 3,
+ .pk_len = MLKEM768_PUBLIC_KEY_BYTES,
+ .sk_len = MLKEM768_SECRET_KEY_BYTES,
+ .ct_len = MLKEM768_CIPHERTEXT_BYTES,
+ },
+ [MLKEM1024] = {
+ .name = "ML-KEM-1024",
+ .k = 4,
+ .pk_len = MLKEM1024_PUBLIC_KEY_BYTES,
+ .sk_len = MLKEM1024_SECRET_KEY_BYTES,
+ .ct_len = MLKEM1024_CIPHERTEXT_BYTES,
+ },
+};
+
+static struct mlkem_bufs *alloc_bufs(struct kunit *test,
+ enum mlkem_paramset paramset)
+{
+ struct mlkem_bufs *bufs =
+ kunit_kmalloc(test, sizeof(*bufs), GFP_KERNEL);
+
+ KUNIT_ASSERT_NOT_NULL(test, bufs);
+
+ bufs->pk_len = mlkem_paramsets[paramset].pk_len;
+ bufs->pk = kunit_kmalloc(test, bufs->pk_len, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, bufs->pk);
+
+ bufs->sk_len = mlkem_paramsets[paramset].sk_len;
+ bufs->sk = kunit_kmalloc(test, bufs->sk_len, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, bufs->sk);
+
+ bufs->ct_len = mlkem_paramsets[paramset].ct_len;
+ bufs->ct = kunit_kmalloc(test, bufs->ct_len, GFP_KERNEL);
+ KUNIT_ASSERT_NOT_NULL(test, bufs->ct);
+
+ return bufs;
+}
+
+static int keygen(u8 *pk, u8 *sk, enum mlkem_paramset paramset)
+{
+ switch (paramset) {
+ case MLKEM768:
+ return mlkem768_keygen(pk, sk);
+ case MLKEM1024:
+ return mlkem1024_keygen(pk, sk);
+ default:
+ WARN_ON_ONCE(1);
+ return -EOPNOTSUPP;
+ }
+}
+
+static int keygen_internal(u8 *pk, u8 *sk, const u8 seed[MLKEM_SEED_BYTES],
+ enum mlkem_paramset paramset)
+{
+ switch (paramset) {
+ case MLKEM768:
+ return mlkem768_keygen_internal(pk, sk, seed);
+ case MLKEM1024:
+ return mlkem1024_keygen_internal(pk, sk, seed);
+ default:
+ WARN_ON_ONCE(1);
+ return -EOPNOTSUPP;
+ }
+}
+
+static int encaps(u8 *ct, u8 ss[MLKEM_SHARED_SECRET_BYTES], const u8 *pk,
+ enum mlkem_paramset paramset)
+{
+ switch (paramset) {
+ case MLKEM768:
+ return mlkem768_encaps(ct, ss, pk);
+ case MLKEM1024:
+ return mlkem1024_encaps(ct, ss, pk);
+ default:
+ WARN_ON_ONCE(1);
+ return -EOPNOTSUPP;
+ }
+}
+
+static int encaps_internal(u8 *ct, u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 *pk, const u8 eseed[MLKEM_ESEED_BYTES],
+ enum mlkem_paramset paramset)
+{
+ switch (paramset) {
+ case MLKEM768:
+ return mlkem768_encaps_internal(ct, ss, pk, eseed);
+ case MLKEM1024:
+ return mlkem1024_encaps_internal(ct, ss, pk, eseed);
+ default:
+ WARN_ON_ONCE(1);
+ return -EOPNOTSUPP;
+ }
+}
+
+static int decaps(u8 ss[MLKEM_SHARED_SECRET_BYTES], const u8 *ct, const u8 *sk,
+ enum mlkem_paramset paramset)
+{
+ switch (paramset) {
+ case MLKEM768:
+ return mlkem768_decaps(ss, ct, sk);
+ case MLKEM1024:
+ return mlkem1024_decaps(ss, ct, sk);
+ default:
+ WARN_ON_ONCE(1);
+ return -EOPNOTSUPP;
+ }
+}
+
+/*
+ * Test the ML-KEM implementation against the first 1000 test vectors from the
+ * reference implementation.
+ *
+ * To do this without explicitly including all these test vectors, which would
+ * result in a massive source and binary size, we take advantage of the fact
+ * that the reference test vectors are generated deterministically (by
+ * kyber/ref/tests/test_vectors.c). We just regenerate them at runtime using
+ * the same algorithm. We hash all the outputs, then verify that hash against
+ * @expected_cumulative_hash, which proves that all the outputs were correct.
+ */
+static void
+test_mlkem_against_ref_testvecs(struct kunit *test, size_t num_testvecs,
+ const u8 expected_cumulative_hash[32],
+ enum mlkem_paramset paramset)
+{
+ struct mlkem_bufs *bufs = alloc_bufs(test, paramset);
+ struct shake_ctx cumulative_hash_ctx;
+ struct shake_ctx seed_ctx;
+ u8 cumulative_hash[32];
+
+ shake128_init(&cumulative_hash_ctx);
+ shake128_init(&seed_ctx);
+ for (size_t i = 0; i < num_testvecs; i++) {
+ /*
+ * Deterministically generate the next seeds using the same
+ * algorithm as the reference code's test_vectors.c.
+ */
+ shake_squeeze(&seed_ctx, bufs->seed, sizeof(bufs->seed));
+ shake_squeeze(&seed_ctx, bufs->eseed, sizeof(bufs->eseed));
+
+ /* KeyGen, then update with (pk, sk) */
+ KUNIT_ASSERT_EQ(test, 0,
+ keygen_internal(bufs->pk, bufs->sk, bufs->seed,
+ paramset));
+ shake_update(&cumulative_hash_ctx, bufs->pk, bufs->pk_len);
+ shake_update(&cumulative_hash_ctx, bufs->sk, bufs->sk_len);
+
+ /* Encaps, then update with (ct, ss) */
+ KUNIT_ASSERT_EQ(test, 0,
+ encaps_internal(bufs->ct, bufs->ss, bufs->pk,
+ bufs->eseed, paramset));
+ shake_update(&cumulative_hash_ctx, bufs->ct, bufs->ct_len);
+ shake_update(&cumulative_hash_ctx, bufs->ss, sizeof(bufs->ss));
+
+ /* Decaps, then update with ss */
+ memset(bufs->ss, 0xff, sizeof(bufs->ss));
+ KUNIT_ASSERT_EQ(test, 0,
+ decaps(bufs->ss, bufs->ct, bufs->sk, paramset));
+ shake_update(&cumulative_hash_ctx, bufs->ss, sizeof(bufs->ss));
+
+ /*
+ * Deterministically generate an invalid ciphertext, using the
+ * same algorithm as test_vectors.c. Then do Decaps and update
+ * with ss_rejected. This tests the implicit rejection case.
+ */
+ shake_squeeze(&seed_ctx, bufs->ct, bufs->ct_len);
+ KUNIT_ASSERT_EQ(test, 0,
+ decaps(bufs->ss, bufs->ct, bufs->sk, paramset));
+ shake_update(&cumulative_hash_ctx, bufs->ss, sizeof(bufs->ss));
+ }
+ /*
+ * Finalize and verify the cumulative hash. This verifies that every
+ * (pk, sk, ct, ss, ss, ss_rejected) tuple was correct.
+ */
+ shake_squeeze(&cumulative_hash_ctx, cumulative_hash,
+ sizeof(cumulative_hash));
+ KUNIT_ASSERT_MEMEQ(test, expected_cumulative_hash, cumulative_hash,
+ sizeof(cumulative_hash));
+}
+
+static void test_mlkem_round_trip(struct kunit *test,
+ enum mlkem_paramset paramset)
+{
+ struct mlkem_bufs *bufs = alloc_bufs(test, paramset);
+ u8 ss2[MLKEM_SHARED_SECRET_BYTES];
+
+ for (int i = 0; i < 20; i++) {
+ KUNIT_ASSERT_EQ(test, 0, keygen(bufs->pk, bufs->sk, paramset));
+ KUNIT_ASSERT_EQ(test, 0,
+ encaps(bufs->ct, bufs->ss, bufs->pk, paramset));
+ KUNIT_ASSERT_EQ(test, 0,
+ decaps(ss2, bufs->ct, bufs->sk, paramset));
+ KUNIT_ASSERT_MEMEQ(test, bufs->ss, ss2, sizeof(bufs->ss));
+ }
+}
+
+/*
+ * Test that changing any part of the ciphertext results in a different shared
+ * secret due to implicit rejection.
+ */
+static void test_mlkem_rejection(struct kunit *test,
+ enum mlkem_paramset paramset)
+{
+ struct mlkem_bufs *bufs = alloc_bufs(test, paramset);
+ u8 ss2[MLKEM_SHARED_SECRET_BYTES];
+
+ KUNIT_ASSERT_EQ(test, 0, keygen(bufs->pk, bufs->sk, paramset));
+ KUNIT_ASSERT_EQ(test, 0,
+ encaps(bufs->ct, bufs->ss, bufs->pk, paramset));
+
+ /* Decapsulate a valid ciphertext. */
+ KUNIT_ASSERT_EQ(test, 0, decaps(ss2, bufs->ct, bufs->sk, paramset));
+ KUNIT_ASSERT_MEMEQ(test, bufs->ss, ss2, sizeof(bufs->ss));
+
+ for (size_t i = 0; i < bufs->ct_len; i++) {
+ /* Corrupt byte i of the ciphertext. */
+ bufs->ct[i] ^= 1;
+
+ /* Decapsulate an invalid ciphertext and assert ss differs. */
+ KUNIT_ASSERT_EQ(test, 0,
+ decaps(ss2, bufs->ct, bufs->sk, paramset));
+ KUNIT_ASSERT_MEMNEQ(test, bufs->ss, ss2, sizeof(bufs->ss));
+ /* Undo the ciphertext corruption. */
+ bufs->ct[i] ^= 1;
+ }
+}
+
+/*
+ * Test that the encapsulation function returns -EBADMSG if a coefficient in
+ * NTT(t) in the public key is outside the interval [0, Q - 1].
+ */
+static void test_mlkem_invalid_pk(struct kunit *test,
+ enum mlkem_paramset paramset)
+{
+ struct mlkem_bufs *bufs = alloc_bufs(test, paramset);
+ const size_t ntt_t_len = 384 * mlkem_paramsets[paramset].k;
+
+ for (int i = 0; i < 4; i++) {
+ u16 c;
+
+ KUNIT_ASSERT_EQ(test, 0, keygen(bufs->pk, bufs->sk, paramset));
+ KUNIT_ASSERT_EQ(test, 0,
+ encaps(bufs->ct, bufs->ss, bufs->pk, paramset));
+ /*
+ * Corrupt a coefficient of NTT(t), which is an array of 256*k
+ * 12-bit coefficients starting at the beginning of pk.
+ */
+ if (i % 2 == 0)
+ c = Q; /* Low end of invalid range */
+ else
+ c = 0xfff; /* High end of invalid range */
+ if (i < 2) {
+ /* Corrupt the first 12-bit coefficient in NTT(t). */
+ bufs->pk[0] = (c & 0xff);
+ bufs->pk[1] = (bufs->pk[1] & 0xf0) | (c >> 8);
+ } else {
+ /* Corrupt the last 12-bit coefficient in NTT(t). */
+ bufs->pk[ntt_t_len - 2] =
+ (bufs->pk[ntt_t_len - 2] & 0xf) |
+ ((c & 0xf) << 4);
+ bufs->pk[ntt_t_len - 1] = c >> 4;
+ }
+ KUNIT_ASSERT_EQ(test, -EBADMSG,
+ encaps(bufs->ct, bufs->ss, bufs->pk, paramset));
+ }
+}
+
+/*
+ * Test that the decapsulation function returns -EBADMSG if either:
+ *
+ * - H(pk) is corrupt
+ * - A coefficient in NTT(s) is outside the interval [0, Q - 1]
+ */
+static void test_mlkem_invalid_sk(struct kunit *test,
+ enum mlkem_paramset paramset)
+{
+ struct mlkem_bufs *bufs = alloc_bufs(test, paramset);
+ const size_t ntt_s_len = 384 * mlkem_paramsets[paramset].k;
+
+ KUNIT_ASSERT_EQ(test, 0, keygen(bufs->pk, bufs->sk, paramset));
+ KUNIT_ASSERT_EQ(test, 0,
+ encaps(bufs->ct, bufs->ss, bufs->pk, paramset));
+ KUNIT_ASSERT_EQ(test, 0,
+ decaps(bufs->ss, bufs->ct, bufs->sk, paramset));
+
+ /* Corrupt H(pk) in the sk. */
+ bufs->sk[bufs->sk_len - 33] ^= 0x80;
+ KUNIT_ASSERT_EQ(test, -EBADMSG,
+ decaps(bufs->ss, bufs->ct, bufs->sk, paramset));
+
+ for (int i = 0; i < 4; i++) {
+ u16 c;
+
+ KUNIT_ASSERT_EQ(test, 0, keygen(bufs->pk, bufs->sk, paramset));
+ KUNIT_ASSERT_EQ(test, 0,
+ encaps(bufs->ct, bufs->ss, bufs->pk, paramset));
+ KUNIT_ASSERT_EQ(test, 0,
+ decaps(bufs->ss, bufs->ct, bufs->sk, paramset));
+
+ /*
+ * Corrupt a coefficient of NTT(s), which is an array of 256*k
+ * 12-bit coefficients starting at the beginning of sk.
+ */
+ if (i % 2 == 0)
+ c = Q; /* Low end of invalid range */
+ else
+ c = 0xfff; /* High end of invalid range */
+ if (i < 2) {
+ /* Corrupt the first 12-bit coefficient in NTT(s). */
+ bufs->sk[0] = (c & 0xff);
+ bufs->sk[1] = (bufs->sk[1] & 0xf0) | (c >> 8);
+ } else {
+ /* Corrupt the last 12-bit coefficient in NTT(s). */
+ bufs->sk[ntt_s_len - 2] =
+ (bufs->sk[ntt_s_len - 2] & 0xf) |
+ ((c & 0xf) << 4);
+ bufs->sk[ntt_s_len - 1] = c >> 4;
+ }
+ KUNIT_ASSERT_EQ(test, -EBADMSG,
+ decaps(bufs->ss, bufs->ct, bufs->sk, paramset));
+ }
+}
+
+static void test_mlkem(struct kunit *test, size_t num_testvecs,
+ const u8 expected_cumulative_hash[32],
+ enum mlkem_paramset paramset)
+{
+ test_mlkem_against_ref_testvecs(test, num_testvecs,
+ expected_cumulative_hash, paramset);
+ test_mlkem_round_trip(test, paramset);
+ test_mlkem_rejection(test, paramset);
+ test_mlkem_invalid_pk(test, paramset);
+ test_mlkem_invalid_sk(test, paramset);
+}
+
+static void test_mlkem768(struct kunit *test)
+{
+ test_mlkem(test, MLKEM768_NUM_TESTVECS, mlkem768_hash, MLKEM768);
+}
+
+static void test_mlkem1024(struct kunit *test)
+{
+ test_mlkem(test, MLKEM1024_NUM_TESTVECS, mlkem1024_hash, MLKEM1024);
+}
+
+static u16 mod_q(s32 x)
+{
+ x %= Q;
+ if (x < 0)
+ x += Q;
+ return x;
+}
+
+/*
+ * Test that mlkem_reduce_once() and mlkem_reduce() produce the correct output
+ * for every supported input.
+ */
+static void test_mlkem_reduce(struct kunit *test)
+{
+ /* mlkem_reduce_once() supports 0 <= x < 2*Q */
+ for (u16 x = 0; x < 2 * Q; x++)
+ KUNIT_ASSERT_EQ(test, mod_q(x), mlkem_reduce_once(x));
+
+ /* mlkem_reduce() supports 0 <= x < Q + 2*Q*Q */
+ for (u32 x = 0; x < Q + 2 * Q * Q; x++)
+ KUNIT_ASSERT_EQ(test, mod_q(x), mlkem_reduce(x));
+}
+
+/* round((2^d / Q) * x) mod 2^d */
+static u16 compress_d_ref(u16 x, int d)
+{
+ u64 quotient, remainder;
+
+ quotient = div64_u64_rem((u64)x << d, Q, &remainder);
+ if (remainder >= (Q + 1) / 2)
+ quotient++;
+ return quotient & ((1 << d) - 1);
+}
+
+/* round((Q / 2^d) * y) */
+static u16 decompress_d_ref(u16 y, int d)
+{
+ u64 quotient, remainder;
+
+ quotient = div64_u64_rem((u64)y * Q, 1 << d, &remainder);
+ if (remainder >= 1 << (d - 1))
+ quotient++;
+ return quotient;
+}
+
+/*
+ * Test that mlkem_compress_d() produces the correct output for every supported
+ * input.
+ */
+static void test_mlkem_compress(struct kunit *test)
+{
+ /* compress_d() supports 0 <= x < Q and 1 <= d <= 11. */
+ for (int d = 1; d <= 11; d++) {
+ for (int x = 0; x < Q; x++) {
+ KUNIT_ASSERT_EQ(test, compress_d_ref(x, d),
+ mlkem_compress_d(x, d));
+ }
+ }
+}
+
+/*
+ * Test that mlkem_decompress_d() produces the correct output for every
+ * supported input.
+ */
+static void test_mlkem_decompress(struct kunit *test)
+{
+ for (int d = 1; d <= 11; d++) {
+ for (int y = 0; y < (1 << d); y++) {
+ KUNIT_ASSERT_EQ(test, decompress_d_ref(y, d),
+ mlkem_decompress_d(y, d));
+ }
+ }
+}
+
+/* Benchmark ML-KEM performance. */
+static void benchmark_mlkem(struct kunit *test, enum mlkem_paramset paramset)
+{
+ const char *name = mlkem_paramsets[paramset].name;
+ struct mlkem_bufs *bufs = alloc_bufs(test, paramset);
+ const int iterations = 100;
+ ktime_t start, end;
+
+ if (!IS_ENABLED(CONFIG_CRYPTO_LIB_BENCHMARK))
+ kunit_skip(test, "not enabled");
+
+ start = ktime_get();
+ for (int i = 0; i < iterations; i++)
+ KUNIT_ASSERT_EQ(test, 0, keygen(bufs->pk, bufs->sk, paramset));
+ end = ktime_get();
+ kunit_info(test, "%s_KeyGen: %llu ns/op\n", name,
+ div64_u64(ktime_to_ns(ktime_sub(end, start)), iterations));
+
+ start = ktime_get();
+ for (int i = 0; i < iterations; i++)
+ KUNIT_ASSERT_EQ(test, 0,
+ encaps(bufs->ct, bufs->ss, bufs->pk, paramset));
+ end = ktime_get();
+ kunit_info(test, "%s_Encaps: %llu ns/op\n", name,
+ div64_u64(ktime_to_ns(ktime_sub(end, start)), iterations));
+
+ start = ktime_get();
+ for (int i = 0; i < iterations; i++)
+ KUNIT_ASSERT_EQ(test, 0,
+ decaps(bufs->ss, bufs->ct, bufs->sk, paramset));
+ end = ktime_get();
+ kunit_info(test, "%s_Decaps: %llu ns/op\n", name,
+ div64_u64(ktime_to_ns(ktime_sub(end, start)), iterations));
+}
+
+static void benchmark_mlkem768(struct kunit *test)
+{
+ benchmark_mlkem(test, MLKEM768);
+}
+
+static void benchmark_mlkem1024(struct kunit *test)
+{
+ benchmark_mlkem(test, MLKEM1024);
+}
+
+/* clang-format off */
+static struct kunit_case mlkem_test_cases[] = {
+ KUNIT_CASE(test_mlkem768),
+ KUNIT_CASE(test_mlkem1024),
+ KUNIT_CASE(test_mlkem_reduce),
+ KUNIT_CASE(test_mlkem_compress),
+ KUNIT_CASE(test_mlkem_decompress),
+ KUNIT_CASE(benchmark_mlkem768),
+ KUNIT_CASE(benchmark_mlkem1024),
+ {},
+};
+/* clang-format on */
+
+static struct kunit_suite mlkem_test_suite = {
+ .name = "mlkem",
+ .test_cases = mlkem_test_cases,
+};
+kunit_test_suite(mlkem_test_suite);
+
+MODULE_DESCRIPTION("KUnit tests and benchmark for ML-KEM");
+MODULE_IMPORT_NS("CRYPTO_INTERNAL");
+MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
+MODULE_LICENSE("GPL");
diff --git a/scripts/crypto/import-mlkem-testvecs.py b/scripts/crypto/import-mlkem-testvecs.py
new file mode 100755
index 000000000000..1d5681ccfee1
--- /dev/null
+++ b/scripts/crypto/import-mlkem-testvecs.py
@@ -0,0 +1,117 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0-or-later
+#
+# This script imports test vectors from the Kyber reference code. To use:
+#
+# git clone https://github.com/pq-crystals/kyber
+# cd kyber/ref
+# make
+# ./test/test_vectors768 > ./test/tvecs768
+# ./test/test_vectors1024 > ./test/tvecs1024
+# $PATH_TO_THIS_SCRIPT ./test/
+#
+# This script generates the following files:
+#
+# lib/crypto/tests/mlkem-testvecs.h
+
+import hashlib
+import os
+import sys
+
+SCRIPT_NAME = os.path.basename(__file__)
+LINUX_DIR = os.path.dirname(os.path.realpath(__file__)) + "/../.."
+
+MLKEM_LENGTHS = {
+ 768: {
+ "pk_len": 1184,
+ "sk_len": 2400,
+ "ct_len": 1088,
+ },
+ 1024: {
+ "pk_len": 1568,
+ "sk_len": 3168,
+ "ct_len": 1568,
+ },
+}
+
+
+def print_header(file):
+ print("/* SPDX-License-Identifier: GPL-2.0-or-later */", file=file)
+ print(f"/* This file was generated by {SCRIPT_NAME} */", file=file)
+ print(f"/* clang-format off */", file=file)
+
+
+def hex_to_bytes(hex_string, expected_bin_len):
+ res = bytes.fromhex(hex_string)
+ assert len(res) == expected_bin_len
+ return res
+
+
+def print_bytes(file, prefix, value, bytes_per_line):
+ for i in range(0, len(value), bytes_per_line):
+ line = prefix + "".join(f"0x{b:02x}, " for b in value[i : i + bytes_per_line])
+ print(f"{line.rstrip()}", file=file)
+
+
+def print_static_u8_array_definition(file, name, value):
+ print("", file=file)
+ print(f"static const u8 {name} = {{", file=file)
+ print_bytes(file, "\t", value, 12)
+ print("};", file=file)
+
+
+class Testvec:
+ def __init__(self, dict, paramset):
+ lens = MLKEM_LENGTHS[paramset]
+ self.pk = hex_to_bytes(dict["Public Key"], lens["pk_len"])
+ self.sk = hex_to_bytes(dict["Secret Key"], lens["sk_len"])
+ self.ct = hex_to_bytes(dict["Ciphertext"], lens["ct_len"])
+ assert dict["Shared Secret A"] == dict["Shared Secret B"]
+ self.ss = hex_to_bytes(dict["Shared Secret A"], 32)
+ self.ss_rejected = hex_to_bytes(dict["Pseudorandom shared Secret A"], 32)
+
+
+def load_testvecs(tvecs_file, paramset):
+ testvecs = []
+ cur = None
+ with open(tvecs_file) as f:
+ for line in f:
+ (name, value) = line.split(":")
+ if name == "Public Key":
+ if cur:
+ testvecs.append(Testvec(cur, paramset))
+ cur = {}
+ cur[name] = value.strip()
+ if cur:
+ testvecs.append(Testvec(cur, paramset))
+ return testvecs
+
+
+def hash_testvecs(testvecs):
+ h = hashlib.shake_128()
+ for tv in testvecs:
+ h.update(tv.pk)
+ h.update(tv.sk)
+ h.update(tv.ct)
+ h.update(tv.ss) # From encapsulation
+ h.update(tv.ss) # From decapsulation
+ h.update(tv.ss_rejected)
+ return h.digest(length=32)
+
+
+if len(sys.argv) != 2:
+ sys.stderr.write(f"Usage: {SCRIPT_NAME} TESTVECS_DIR\n")
+ sys.exit(2)
+testvecs_dir = sys.argv[1]
+
+with open(LINUX_DIR + "/lib/crypto/tests/mlkem-testvecs.h", "w") as file:
+ print_header(file)
+ for paramset in [768, 1024]:
+ testvecs = load_testvecs(testvecs_dir + f"/tvecs{paramset}", paramset)
+ # There are 10000 test vectors, which take a bit long for a KUnit test.
+ # Trim them down to just the first 1000.
+ num_testvecs = min(len(testvecs), 1000)
+ testvecs = testvecs[:num_testvecs]
+ hash = hash_testvecs(testvecs)
+ print(f"\n#define MLKEM{paramset}_NUM_TESTVECS {num_testvecs}", file=file)
+ print_static_u8_array_definition(file, f"mlkem{paramset}_hash[32]", hash)
--
2.54.0
^ permalink raw reply related
* [PATCH 1/5] lib/crypto: mlkem: Add ML-KEM-768 and ML-KEM-1024 support
From: Eric Biggers @ 2026-05-25 18:43 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, Ard Biesheuvel, Jason A . Donenfeld, Herbert Xu,
Ryan Appel, Chris Leech, Eric Biggers
In-Reply-To: <20260525184403.101818-1-ebiggers@kernel.org>
Add support for ML-KEM-768 and ML-KEM-1024 to lib/crypto/. This is a
proof-of-concept that won't be merged until there's an in-kernel user.
ML-KEM is a "post-quantum" key encapsulation mechanism that is specified
in FIPS 203 and is based on CRYSTALS-Kyber. As a key encapsulation
mechanism (KEM), it consists of a set of algorithms that can be used to
establish a shared secret key over a public channel. As a
"post-quantum" algorithm, it's conjectured to be secure even against
adversaries in possession of a large-scale quantum computer.
Current users of classical key agreement schemes in the kernel include
NVMe in-band authentication, Bluetooth, and WireGuard. It's likely that
at least some of these will eventually be upgraded to a hybrid KEM that
uses ML-KEM, such as X-Wing which uses X25519 and ML-KEM-768.
This commit just gets things started by implementing ML-KEM-768 and
ML-KEM-1024. A later commit adds X-Wing support on top of it.
ML-KEM-512 has been omitted, since it has seen limited adoption and
remains somewhat controversial. It wouldn't be that much work to
support it too, but there's no clear need to do so yet.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
Documentation/crypto/libcrypto-asymmetric.rst | 20 +
Documentation/crypto/libcrypto-signature.rst | 11 -
Documentation/crypto/libcrypto.rst | 2 +-
include/crypto/mlkem.h | 151 +++
lib/crypto/Kconfig | 8 +
lib/crypto/Makefile | 5 +
lib/crypto/mlkem.c | 894 ++++++++++++++++++
7 files changed, 1079 insertions(+), 12 deletions(-)
create mode 100644 Documentation/crypto/libcrypto-asymmetric.rst
delete mode 100644 Documentation/crypto/libcrypto-signature.rst
create mode 100644 include/crypto/mlkem.h
create mode 100644 lib/crypto/mlkem.c
diff --git a/Documentation/crypto/libcrypto-asymmetric.rst b/Documentation/crypto/libcrypto-asymmetric.rst
new file mode 100644
index 000000000000..6e71c5ce823f
--- /dev/null
+++ b/Documentation/crypto/libcrypto-asymmetric.rst
@@ -0,0 +1,20 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later
+
+Asymmetric cryptography
+=======================
+
+ML-DSA
+------
+
+Support for the ML-DSA digital signature algorithm.
+
+.. kernel-doc:: include/crypto/mldsa.h
+
+ML-KEM
+------
+
+Support for the ML-KEM key encapsulation mechanism.
+
+This shall be used as part of a hybrid scheme such as X-Wing, not by itself.
+
+.. kernel-doc:: include/crypto/mlkem.h
diff --git a/Documentation/crypto/libcrypto-signature.rst b/Documentation/crypto/libcrypto-signature.rst
deleted file mode 100644
index e80d59fa51b6..000000000000
--- a/Documentation/crypto/libcrypto-signature.rst
+++ /dev/null
@@ -1,11 +0,0 @@
-.. SPDX-License-Identifier: GPL-2.0-or-later
-
-Digital signature algorithms
-============================
-
-ML-DSA
-------
-
-Support for the ML-DSA digital signature algorithm.
-
-.. kernel-doc:: include/crypto/mldsa.h
diff --git a/Documentation/crypto/libcrypto.rst b/Documentation/crypto/libcrypto.rst
index a1557d45b0e5..8c61b6513e06 100644
--- a/Documentation/crypto/libcrypto.rst
+++ b/Documentation/crypto/libcrypto.rst
@@ -156,10 +156,10 @@ API documentation
=================
.. toctree::
:maxdepth: 2
+ libcrypto-asymmetric
libcrypto-blockcipher
libcrypto-hash
- libcrypto-signature
libcrypto-utils
sha3
diff --git a/include/crypto/mlkem.h b/include/crypto/mlkem.h
new file mode 100644
index 000000000000..e33d065c5442
--- /dev/null
+++ b/include/crypto/mlkem.h
@@ -0,0 +1,151 @@
+/* SPDX-License-Identifier: GPL-2.0-or-later */
+/*
+ * Copyright 2026 Google LLC
+ */
+
+/**
+ * DOC: ML-KEM (Module-Lattice-Based Key Encapsulation Mechanism)
+ *
+ * This is an implementation of ML-KEM, a "post-quantum" key encapsulation
+ * mechanism that is specified in FIPS 203 and is based on CRYSTALS-Kyber.
+ *
+ * Specifically, the ML-KEM-768 and ML-KEM-1024 parameter sets are supported.
+ *
+ * This shall be used as part of a hybrid scheme such as X-Wing, not by itself.
+ *
+ * This implementation is designed to be constant-time, compact, and
+ * memory-efficient, and to reuse the kernel's SHA-3 routines. For simplicity,
+ * it stores integers mod Q as their standard representatives in the interval
+ * [0, Q - 1] across function boundaries. (This makes it more similar to e.g.
+ * BoringSSL than to the Kyber reference code, which uses a slightly more
+ * optimized but harder-to-understand approach.)
+ */
+
+#ifndef _CRYPTO_MLKEM_H
+#define _CRYPTO_MLKEM_H
+
+#include <linux/types.h>
+
+#define MLKEM_SEED_BYTES 64 /* Length of seed for KeyGen */
+#define MLKEM_ESEED_BYTES 32 /* Length of seed for Encaps */
+#define MLKEM_SHARED_SECRET_BYTES 32
+
+#define MLKEM768_PUBLIC_KEY_BYTES 1184
+#define MLKEM768_SECRET_KEY_BYTES 2400
+#define MLKEM768_CIPHERTEXT_BYTES 1088
+
+/**
+ * mlkem768_keygen() - Generate an ML-KEM-768 key pair
+ * @pk: (output) The public key (encapsulation key)
+ * @sk: (output) The secret key (decapsulation key)
+ *
+ * Context: Might sleep
+ *
+ * Return: 0 on success, or -ENOMEM if out of memory.
+ */
+int mlkem768_keygen(u8 pk[MLKEM768_PUBLIC_KEY_BYTES],
+ u8 sk[MLKEM768_SECRET_KEY_BYTES]);
+
+/**
+ * mlkem768_encaps() - Generate and encapsulate shared secret with ML-KEM-768
+ * @ct: (output) The ciphertext
+ * @ss: (output) The generated shared secret
+ * @pk: The public key (encapsulation key)
+ *
+ * Context: Might sleep
+ *
+ * Return:
+ * * 0 on success
+ * * -EBADMSG if the public key is malformed
+ * * -ENOMEM if out of memory
+ */
+int mlkem768_encaps(u8 ct[MLKEM768_CIPHERTEXT_BYTES],
+ u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 pk[MLKEM768_PUBLIC_KEY_BYTES]);
+
+/**
+ * mlkem768_decaps() - Decapsulate shared secret with ML-KEM-768
+ * @ss: (output) The decapsulated shared secret
+ * @ct: The ciphertext
+ * @sk: The secret key (decapsulation key)
+ *
+ * Context: Might sleep
+ *
+ * Return:
+ * * 0 on success, including the "implicit rejection" case where the ciphertext
+ * is invalid and a randomized shared secret is returned
+ * * -EBADMSG if the secret key is malformed
+ * * -ENOMEM if out of memory
+ */
+int mlkem768_decaps(u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 ct[MLKEM768_CIPHERTEXT_BYTES],
+ const u8 sk[MLKEM768_SECRET_KEY_BYTES]);
+
+#define MLKEM1024_PUBLIC_KEY_BYTES 1568
+#define MLKEM1024_SECRET_KEY_BYTES 3168
+#define MLKEM1024_CIPHERTEXT_BYTES 1568
+
+/**
+ * mlkem1024_keygen() - Generate an ML-KEM-1024 key pair
+ * @pk: (output) The public key (encapsulation key)
+ * @sk: (output) The secret key (decapsulation key)
+ *
+ * Context: Might sleep
+ *
+ * Return: 0 on success, or -ENOMEM if out of memory.
+ */
+int mlkem1024_keygen(u8 pk[MLKEM1024_PUBLIC_KEY_BYTES],
+ u8 sk[MLKEM1024_SECRET_KEY_BYTES]);
+
+/**
+ * mlkem1024_encaps() - Generate and encapsulate shared secret with ML-KEM-1024
+ * @ct: (output) The ciphertext
+ * @ss: (output) The generated shared secret
+ * @pk: The public key (encapsulation key)
+ *
+ * Context: Might sleep
+ *
+ * Return:
+ * * 0 on success
+ * * -EBADMSG if the public key is malformed
+ * * -ENOMEM if out of memory
+ */
+int mlkem1024_encaps(u8 ct[MLKEM1024_CIPHERTEXT_BYTES],
+ u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 pk[MLKEM1024_PUBLIC_KEY_BYTES]);
+
+/**
+ * mlkem1024_decaps() - Decapsulate shared secret with ML-KEM-1024
+ * @ss: (output) The decapsulated shared secret
+ * @ct: The ciphertext
+ * @sk: The secret key (decapsulation key)
+ *
+ * Context: Might sleep
+ *
+ * Return:
+ * * 0 on success, including the "implicit rejection" case where the ciphertext
+ * is invalid and a randomized shared secret is returned
+ * * -EBADMSG if the secret key is malformed
+ * * -ENOMEM if out of memory
+ */
+int mlkem1024_decaps(u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 ct[MLKEM1024_CIPHERTEXT_BYTES],
+ const u8 sk[MLKEM1024_SECRET_KEY_BYTES]);
+
+/* Functions taking explicit seeds, only for KUnit testing and hybrid KEMs */
+int mlkem768_keygen_internal(u8 pk[MLKEM768_PUBLIC_KEY_BYTES],
+ u8 sk[MLKEM768_SECRET_KEY_BYTES],
+ const u8 seed[MLKEM_SEED_BYTES]);
+int mlkem768_encaps_internal(u8 ct[MLKEM768_CIPHERTEXT_BYTES],
+ u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 pk[MLKEM768_PUBLIC_KEY_BYTES],
+ const u8 eseed[MLKEM_ESEED_BYTES]);
+int mlkem1024_keygen_internal(u8 pk[MLKEM1024_PUBLIC_KEY_BYTES],
+ u8 sk[MLKEM1024_SECRET_KEY_BYTES],
+ const u8 seed[MLKEM_SEED_BYTES]);
+int mlkem1024_encaps_internal(u8 ct[MLKEM1024_CIPHERTEXT_BYTES],
+ u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 pk[MLKEM1024_PUBLIC_KEY_BYTES],
+ const u8 eseed[MLKEM_ESEED_BYTES]);
+
+#endif /* _CRYPTO_MLKEM_H */
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index d3904b72dae7..acaa64af4c6d 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -141,10 +141,18 @@ config CRYPTO_LIB_MLDSA
select CRYPTO_LIB_SHA3
help
The ML-DSA library functions. Select this if your module uses any of
the functions from <crypto/mldsa.h>.
+config CRYPTO_LIB_MLKEM
+ tristate
+ select CRYPTO_LIB_SHA3
+ select CRYPTO_LIB_UTILS
+ help
+ The ML-KEM library functions. Select this if your module uses any of
+ the functions from <crypto/mlkem.h>.
+
config CRYPTO_LIB_NH
tristate
help
Implementation of the NH almost-universal hash function, specifically
the variant of NH used in Adiantum.
diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile
index 4ad91f390038..94cef4e574cd 100644
--- a/lib/crypto/Makefile
+++ b/lib/crypto/Makefile
@@ -197,10 +197,15 @@ endif # CONFIG_CRYPTO_LIB_MD5_ARCH
obj-$(CONFIG_CRYPTO_LIB_MLDSA) += libmldsa.o
libmldsa-y := mldsa.o
################################################################################
+obj-$(CONFIG_CRYPTO_LIB_MLKEM) += libmlkem.o
+libmlkem-y := mlkem.o
+
+################################################################################
+
obj-$(CONFIG_CRYPTO_LIB_NH) += libnh.o
libnh-y := nh.o
ifeq ($(CONFIG_CRYPTO_LIB_NH_ARCH),y)
CFLAGS_nh.o += -I$(src)/$(SRCARCH)
libnh-$(CONFIG_ARM) += arm/nh-neon-core.o
diff --git a/lib/crypto/mlkem.c b/lib/crypto/mlkem.c
new file mode 100644
index 000000000000..b800ecb49f24
--- /dev/null
+++ b/lib/crypto/mlkem.c
@@ -0,0 +1,894 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * ML-KEM (Module-Lattice-Based Key Encapsulation Mechanism)
+ *
+ * See include/crypto/mlkem.h for the documentation.
+ *
+ * Copyright 2026 Google LLC
+ */
+
+#include <crypto/mlkem.h>
+#include <crypto/sha3.h>
+#include <crypto/utils.h>
+#include <linux/export.h>
+#include <linux/module.h>
+#include <linux/random.h>
+#include <linux/slab.h>
+#include <linux/string.h>
+#include <linux/unaligned.h>
+
+#define Q 3329 /* The prime q = 2^8 * 13 + 1 */
+#define N 256 /* Number of coefficients per ring element */
+#define MAX_K 4 /* Max matrix dimension (k parameter) for any parameter set */
+#define MAX_CIPHERTEXT_BYTES 1568 /* Max ciphertext length for any param set */
+#define ETA 2 /* Value of eta_1 and eta_2 for ML-KEM-768 and ML-KEM-1024 */
+
+/*
+ * This array contains zeta^BitRev_7(i) for 0 <= i < 128.
+ * Generated by the following Python code:
+ * [pow(17, int(f'{i:07b}'[::-1], 2), 3329) for i in range(128)]
+ * Also matches the first table in FIPS 203 Appendix A.
+ */
+static const u16 zetas[128] = {
+ 1, 1729, 2580, 3289, 2642, 630, 1897, 848, 1062, 1919, 193, 797,
+ 2786, 3260, 569, 1746, 296, 2447, 1339, 1476, 3046, 56, 2240, 1333,
+ 1426, 2094, 535, 2882, 2393, 2879, 1974, 821, 289, 331, 3253, 1756,
+ 1197, 2304, 2277, 2055, 650, 1977, 2513, 632, 2865, 33, 1320, 1915,
+ 2319, 1435, 807, 452, 1438, 2868, 1534, 2402, 2647, 2617, 1481, 648,
+ 2474, 3110, 1227, 910, 17, 2761, 583, 2649, 1637, 723, 2288, 1100,
+ 1409, 2662, 3281, 233, 756, 2156, 3015, 3050, 1703, 1651, 2789, 1789,
+ 1847, 952, 1461, 2687, 939, 2308, 2437, 2388, 733, 2337, 268, 641,
+ 1584, 2298, 2037, 3220, 375, 2549, 2090, 1645, 1063, 319, 2773, 757,
+ 2099, 561, 2466, 2594, 2804, 1092, 403, 1026, 1143, 2150, 2775, 886,
+ 1722, 1212, 1874, 1029, 2110, 2935, 885, 2154,
+};
+
+/*
+ * This array contains zeta^(2*BitRev_7(i)+1) for 0 <= i < 128.
+ * Generated by the following Python code:
+ * [pow(17, 2*int(f'{i:07b}'[::-1], 2)+1, 3329) for i in range(128)]
+ * Also matches the second table in FIPS 203 Appendix A, with the values
+ * canonicalized to their standard representatives in [0, Q - 1].
+ */
+static const u16 gammas[128] = {
+ 17, 3312, 2761, 568, 583, 2746, 2649, 680, 1637, 1692, 723, 2606,
+ 2288, 1041, 1100, 2229, 1409, 1920, 2662, 667, 3281, 48, 233, 3096,
+ 756, 2573, 2156, 1173, 3015, 314, 3050, 279, 1703, 1626, 1651, 1678,
+ 2789, 540, 1789, 1540, 1847, 1482, 952, 2377, 1461, 1868, 2687, 642,
+ 939, 2390, 2308, 1021, 2437, 892, 2388, 941, 733, 2596, 2337, 992,
+ 268, 3061, 641, 2688, 1584, 1745, 2298, 1031, 2037, 1292, 3220, 109,
+ 375, 2954, 2549, 780, 2090, 1239, 1645, 1684, 1063, 2266, 319, 3010,
+ 2773, 556, 757, 2572, 2099, 1230, 561, 2768, 2466, 863, 2594, 735,
+ 2804, 525, 1092, 2237, 403, 2926, 1026, 2303, 1143, 2186, 2150, 1179,
+ 2775, 554, 886, 2443, 1722, 1607, 1212, 2117, 1874, 1455, 1029, 2300,
+ 2110, 1219, 2935, 394, 885, 2444, 2154, 1175
+};
+
+struct mlkem_parameter_set {
+ u8 k; /* Rank of the module (number of polynomials in vectors) */
+ u8 du; /* Compression parameter for vector u */
+ u8 dv; /* Compression parameter for polynomial v */
+ u16 pk_len; /* Length of public keys in bytes */
+ u16 sk_len; /* Length of secret keys in bytes */
+ u16 ct_len; /* Length of ciphertexts in bytes */
+};
+
+/* ML-KEM-768 parameters. Reference: FIPS 203 Section 8, "Parameter Sets" */
+static const struct mlkem_parameter_set mlkem768 = {
+ .k = 3,
+ .du = 10,
+ .dv = 4,
+ .pk_len = MLKEM768_PUBLIC_KEY_BYTES,
+ .sk_len = MLKEM768_SECRET_KEY_BYTES,
+ .ct_len = MLKEM768_CIPHERTEXT_BYTES,
+};
+
+/* ML-KEM-1024 parameters. Reference: FIPS 203 Section 8, "Parameter Sets" */
+static const struct mlkem_parameter_set mlkem1024 = {
+ .k = 4,
+ .du = 11,
+ .dv = 5,
+ .pk_len = MLKEM1024_PUBLIC_KEY_BYTES,
+ .sk_len = MLKEM1024_SECRET_KEY_BYTES,
+ .ct_len = MLKEM1024_CIPHERTEXT_BYTES,
+};
+
+/*
+ * An element of the ring R_q (normal form) or the ring T_q (NTT form). In both
+ * cases it consists of N integers in the interval [0, Q - 1]. In R_q, these
+ * are the coefficients of one polynomial of maximum degree N - 1. In T_q,
+ * these are the coefficients of N / 2 polynomials of maximum degree 1.
+ */
+struct mlkem_ring_elem {
+ u16 x[N];
+};
+
+struct k_pke_keygen_workspace {
+ union { /* These aren't used at the same time, so they can overlap. */
+ struct shake_ctx shake;
+ struct sha3_ctx sha3;
+ };
+ union {
+ struct {
+ u8 rho[32];
+ u8 sigma[32];
+ };
+ u8 rho_and_sigma[64];
+ };
+ u8 block[MAX(64 * ETA, SHAKE128_BLOCK_SIZE)];
+ struct mlkem_ring_elem tmp, t;
+ struct mlkem_ring_elem s[MAX_K];
+};
+
+struct k_pke_encrypt_workspace {
+ struct shake_ctx shake;
+ u8 block[MAX(64 * ETA, SHAKE128_BLOCK_SIZE)];
+ struct mlkem_ring_elem tmp, y[MAX_K];
+ union { /* These aren't used at the same time, so they can overlap. */
+ struct mlkem_ring_elem u, v;
+ };
+};
+
+struct k_pke_decrypt_workspace {
+ struct mlkem_ring_elem su, s, tmp;
+};
+
+struct mlkem_encap_workspace {
+ union { /* These aren't used at the same time, so they can overlap. */
+ struct sha3_ctx sha3;
+ struct k_pke_encrypt_workspace k_pke_enc;
+ };
+ union {
+ struct {
+ u8 K[32];
+ u8 r[32];
+ };
+ u8 K_and_r[64];
+ };
+ u8 pk_hash[SHA3_256_DIGEST_SIZE];
+};
+
+struct mlkem_decap_workspace {
+ union { /* These aren't used at the same time, so they can overlap. */
+ struct shake_ctx shake;
+ struct sha3_ctx sha3;
+ struct k_pke_encrypt_workspace k_pke_enc;
+ struct k_pke_decrypt_workspace k_pke_dec;
+ };
+ union {
+ struct {
+ u8 K_prime[32];
+ u8 r_prime[32];
+ };
+ u8 K_prime_and_r_prime[64];
+ };
+ u8 h[SHA3_256_DIGEST_SIZE];
+ u8 m_prime[32];
+ u8 K_bar[32];
+ u8 ct_prime[MAX_CIPHERTEXT_BYTES];
+};
+
+/* Reduce @x to its standard representative mod Q, where 0 <= @x < 2*Q */
+static u16 reduce_once(u16 x)
+{
+ u16 x_minus_q, mask;
+
+ x_minus_q = x - Q; /* This wraps around, setting bit 15, if @x < Q */
+ OPTIMIZER_HIDE_VAR(x_minus_q);
+ mask = (s16)x_minus_q >> 15; /* mask = 0xffff if @x < Q, else 0 */
+ OPTIMIZER_HIDE_VAR(mask);
+
+ return (mask & x) | (~mask & x_minus_q);
+}
+
+/* Reduce @x to its standard representative mod Q, where 0 <= @x < Q + 2*Q*Q */
+static u16 reduce(u32 x)
+{
+ /* Barrett reduction: x - floor((x * floor(2^24 / Q)) / 2^24) * Q */
+ u64 product = (u64)x * 5039;
+ u32 quotient = (u32)(product >> 24);
+ u32 remainder = x - quotient * Q; /* 0 <= remainder < 2*Q */
+
+ return reduce_once(remainder);
+}
+
+/*
+ * Convert @f to its number-theoretically-transformed representation in-place.
+ * Reference: FIPS 203 Algorithm 9, NTT
+ */
+static void ntt(struct mlkem_ring_elem *f)
+{
+ int i = 1; /* index in zetas */
+
+ for (int len = 128; len >= 2; len /= 2) {
+ for (int start = 0; start < 256; start += 2 * len) {
+ u32 zeta = zetas[i++];
+
+ for (int j = start; j < start + len; j++) {
+ u16 t = reduce(zeta * f->x[j + len]);
+
+ f->x[j + len] = reduce_once(Q + f->x[j] - t);
+ f->x[j] = reduce_once(f->x[j] + t);
+ }
+ }
+ }
+}
+
+/*
+ * Convert @f from its number-theoretically-transformed representation in-place.
+ *
+ * Reference: FIPS 203 Algorithm 10, NTT^-1
+ */
+static void invntt(struct mlkem_ring_elem *f)
+{
+ int i = 127; /* index in zetas */
+
+ for (int len = 2; len <= 128; len *= 2) {
+ for (int start = 0; start < 256; start += 2 * len) {
+ u32 zeta = zetas[i--];
+
+ for (int j = start; j < start + len; j++) {
+ u16 t = f->x[j];
+
+ f->x[j] = reduce_once(t + f->x[j + len]);
+ f->x[j + len] =
+ reduce(zeta * (Q + f->x[j + len] - t));
+ }
+ }
+ }
+
+ /* Multiply by 128^-1 = 3303 mod Q. */
+ for (int j = 0; j < 256; j++)
+ f->x[j] = reduce((u32)f->x[j] * 3303);
+}
+
+/* Compute @a += @b, where @a and @b are both elements of either R_q or T_q. */
+static void poly_add(struct mlkem_ring_elem *a, const struct mlkem_ring_elem *b)
+{
+ for (int i = 0; i < N; i++)
+ a->x[i] = reduce_once(a->x[i] + b->x[i]);
+}
+
+/* Compute @a -= @b, where @a and @b are both elements of either R_q or T_q. */
+static void poly_sub(struct mlkem_ring_elem *a, const struct mlkem_ring_elem *b)
+{
+ for (int i = 0; i < N; i++)
+ a->x[i] = reduce_once(Q + a->x[i] - b->x[i]);
+}
+
+/*
+ * Compute @h += @f * @g in the ring T_q.
+ *
+ * Reference: FIPS 203 Algorithm 11, MultiplyNTTs
+ */
+static void poly_mul_acc(struct mlkem_ring_elem *h,
+ const struct mlkem_ring_elem *f,
+ const struct mlkem_ring_elem *g)
+{
+ for (int i = 0; i < N / 2; i++) {
+ u32 a0 = f->x[2 * i];
+ u32 a1 = f->x[2 * i + 1];
+ u32 b0 = g->x[2 * i];
+ u32 b1 = g->x[2 * i + 1];
+ u16 c0 = reduce(a0 * b0 + reduce(a1 * b1) * (u32)gammas[i]);
+ u16 c1 = reduce(a0 * b1 + a1 * b0);
+
+ h->x[2 * i] = reduce_once(h->x[2 * i] + c0);
+ h->x[2 * i + 1] = reduce_once(h->x[2 * i + 1] + c1);
+ }
+}
+
+/*
+ * "Compress" @x in the interval [0, Q - 1] into the smaller interval [0, 2^d
+ * - 1] for the given 1 <= @d <= 11 by computing:
+ *
+ * round((2^d / Q) * x) mod 2^d
+ * = floor((x * 2^d + floor(Q / 2)) / Q) mod 2^d
+ *
+ * Reference: FIPS 203 Section 4.2.1, "Conversion and Compression Algorithms"
+ */
+static u16 compress_d(u16 x, int d)
+{
+ u32 t = ((u32)x << d) + (Q / 2);
+
+ /*
+ * t = floor(t / Q). Avoid potentially variable-time division by using
+ * the equivalent (for the input ranges) reciprocal multiplication
+ * floor((t * ceil(2^32 / Q)) / 2^32). 2^32 is chosen because it's
+ * efficient and provides enough precision for all allowed inputs.
+ */
+ OPTIMIZER_HIDE_VAR(t);
+ t = ((u64)t * 1290168) >> 32;
+ OPTIMIZER_HIDE_VAR(t);
+
+ return t & ((1 << d) - 1);
+}
+
+/*
+ * "Decompress" @y in the interval [0, 2^d - 1] into the larger interval [0, Q
+ * - 1] for the given 1 <= @d <= 11 by computing:
+ *
+ * round((Q / 2^d) * y)
+ * = floor((y * Q + 2^(d-1)) / 2^d)
+ *
+ * Reference: FIPS 203 Section 4.2.1, "Conversion and Compression Algorithms"
+ */
+static u16 decompress_d(u16 y, int d)
+{
+ u32 t;
+
+ OPTIMIZER_HIDE_VAR(y);
+ t = (u32)y * Q;
+ OPTIMIZER_HIDE_VAR(t);
+ return (t + (1 << (d - 1))) >> d;
+}
+
+/*
+ * Encode the ring element @in into 32 * @d bytes at @out.
+ *
+ * Reference: FIPS 203 Algorithm 5, ByteEncode_d
+ */
+static void byte_encode(u8 *out, const struct mlkem_ring_elem *in, int d)
+{
+ u32 acc = 0;
+ int bits = 0;
+
+ for (int i = 0; i < 256; i++) {
+ acc |= (u32)in->x[i] << bits;
+ bits += d;
+ for (; bits >= 8; bits -= 8, acc >>= 8)
+ *out++ = (u8)acc;
+ }
+}
+
+/*
+ * Decode the 32 * @d bytes at @in into a ring element @out with coefficients in
+ * the interval [0, 2^d - 1]. Note that when @d >= 12, decoded coefficients can
+ * be out of range (i.e. >= Q) and need to be validated afterwards.
+ *
+ * Reference: FIPS 203 Algorithm 6, ByteDecode_d
+ */
+static void byte_decode(struct mlkem_ring_elem *out, const u8 *in, int d)
+{
+ const u32 mask = (1 << d) - 1;
+ u32 acc = 0;
+ int bits = 0;
+
+ for (int i = 0; i < 256; i++) {
+ for (; bits < d; bits += 8)
+ acc |= (u32)(*in++) << bits;
+ out->x[i] = acc & mask;
+ acc >>= d;
+ bits -= d;
+ }
+}
+
+static void compress_and_encode(u8 *out, struct mlkem_ring_elem *f, int d)
+{
+ for (int i = 0; i < N; i++)
+ f->x[i] = compress_d(f->x[i], d);
+ byte_encode(out, f, d);
+}
+
+static void decode_and_decompress(struct mlkem_ring_elem *out, const u8 *in,
+ int d)
+{
+ byte_decode(out, in, d);
+ for (int i = 0; i < N; i++)
+ out->x[i] = decompress_d(out->x[i], d);
+}
+
+/*
+ * Check in constant time whether any coefficients in @f are outside the
+ * interval [0, Q - 1]. This is needed after calling byte_decode() with d=12.
+ */
+static bool is_out_of_range(const struct mlkem_ring_elem *f)
+{
+ u32 bad = 0;
+
+ for (int i = 0; i < N; i++) {
+ OPTIMIZER_HIDE_VAR(bad);
+ bad |= Q - 1 - f->x[i]; /* Set bit 31 if f->x[i] >= Q. */
+ }
+ OPTIMIZER_HIDE_VAR(bad);
+ return bad >> 31;
+}
+
+/*
+ * Expand the 33 input bytes (@s, @b) into a random polynomial @out with
+ * coefficients in [-ETA, ETA] using a Centered Binomial
+ * Distribution (CBD). @shake and @block are used as temporary space.
+ *
+ * This is FIPS 203 Algorithm 8 "SamplePolyCBD_eta", composed with PRF_eta.
+ * (SamplePolyCBD_eta is invoked only with the output of PRF_eta.)
+ */
+static void sample_poly_cbd(struct mlkem_ring_elem *out, const u8 s[32], u8 b,
+ struct shake_ctx *shake,
+ u8 block[at_least 64 * ETA])
+{
+ /* Expand (s, b) into '64 * ETA' bytes. */
+ shake256_init(shake);
+ shake_update(shake, s, 32);
+ shake_update(shake, &b, 1);
+ shake_squeeze(shake, block, 64 * ETA);
+
+ /*
+ * Generate the coefficients by counting the number of 1 bits in each
+ * ETA-bit group, then subtracting the counts of adjacent pairs.
+ */
+ static_assert(ETA == 2);
+ for (int i = 0; i < 256; i += 16) {
+ const u64 mask = 0x5555555555555555;
+ u64 t = get_unaligned_le64(&block[i / 2]);
+ u64 popcounts = (t & mask) + ((t >> 1) & mask);
+
+ for (int j = 0; j < 16; j++) {
+ u16 x = (popcounts >> (4 * j)) & 0x3;
+ u16 y = (popcounts >> (4 * j + 2)) & 0x3;
+
+ out->x[i + j] = reduce_once(Q + x - y);
+ }
+ }
+}
+
+/*
+ * Generate the element of the matrix NTT(A) at @row and @column from the seed
+ * @rho. The output, which is an element of the ring T_q, is written to @out.
+ * @shake and @block are used as temporary space.
+ *
+ * Reference: FIPS 203 Algorithm 7, SampleNTT
+ */
+static void sample_ntt(struct mlkem_ring_elem *out, const u8 rho[32], u8 row,
+ u8 column, struct shake_ctx *shake,
+ u8 block[at_least SHAKE128_BLOCK_SIZE])
+{
+ shake128_init(shake);
+ shake_update(shake, rho, 32);
+ shake_update(shake, &column, 1);
+ shake_update(shake, &row, 1);
+
+ for (int i = 0; i < 256;) {
+ /*
+ * Squeeze the next block, then use rejection sampling to
+ * generate up to two coefficients from each 3-byte group.
+ */
+ static_assert(SHAKE128_BLOCK_SIZE % 3 == 0);
+ shake_squeeze(shake, block, SHAKE128_BLOCK_SIZE);
+
+ for (int j = 0; j < SHAKE128_BLOCK_SIZE && i < 256; j += 3) {
+ u16 d1 = block[j] | ((block[j + 1] & 0xf) << 8);
+ u16 d2 = (block[j + 1] >> 4) | (block[j + 2] << 4);
+
+ if (d1 < Q)
+ out->x[i++] = d1;
+ if (i < N && d2 < Q)
+ out->x[i++] = d2;
+ }
+ }
+}
+
+/*
+ * Generate a K-PKE key pair (@pk, @sk) from the random seed @d. @pk is 384*k +
+ * 32 bytes, and @sk is 384*k bytes.
+ *
+ * Reference: FIPS 203 Algorithm 13, K-PKE.KeyGen
+ */
+static int k_pke_keygen(u8 *pk, u8 *sk, const u8 d[32],
+ const struct mlkem_parameter_set *params)
+{
+ struct k_pke_keygen_workspace *ws __free(kfree_sensitive) =
+ kmalloc_obj(*ws);
+ const u8 k = params->k;
+
+ if (!ws)
+ return -ENOMEM;
+
+ /* rho || sigma = G(d || k) */
+ sha3_512_init(&ws->sha3);
+ sha3_update(&ws->sha3, d, 32);
+ sha3_update(&ws->sha3, &k, 1);
+ sha3_final(&ws->sha3, ws->rho_and_sigma);
+
+ /* Generate and encode the secret key NTT(s). */
+ for (int i = 0; i < k; i++) {
+ sample_poly_cbd(&ws->s[i], ws->sigma, i, &ws->shake, ws->block);
+ ntt(&ws->s[i]);
+ byte_encode(&sk[384 * i], &ws->s[i], 12);
+ }
+
+ /*
+ * Generate and encode the public key pk = NTT(t) || rho.
+ * To save memory, do it one row at a time.
+ */
+ for (int i = 0; i < k; i++) { /* For each row in A */
+ /* NTT(t) = NTT(A) * NTT(s) + NTT(e) */
+ ws->t = (struct mlkem_ring_elem){};
+ for (int j = 0; j < k; j++) { /* For each column in A */
+ sample_ntt(&ws->tmp, ws->rho, i, j, &ws->shake,
+ ws->block);
+ poly_mul_acc(&ws->t, &ws->tmp, &ws->s[j]);
+ }
+ sample_poly_cbd(&ws->tmp, ws->sigma, k + i, &ws->shake,
+ ws->block);
+ ntt(&ws->tmp);
+ poly_add(&ws->t, &ws->tmp);
+ byte_encode(&pk[384 * i], &ws->t, 12);
+ }
+ memcpy(&pk[384 * k], ws->rho, 32);
+ return 0;
+}
+
+/*
+ * Encrypt the message @m using the public key @pk and randomness @r.
+ *
+ * Reference: FIPS 203 Algorithm 14, K-PKE.Encrypt
+ */
+static int k_pke_encrypt(u8 *ct, const u8 *pk, const u8 m[32], const u8 r[32],
+ struct k_pke_encrypt_workspace *ws,
+ const struct mlkem_parameter_set *params)
+{
+ const u8 k = params->k;
+ const u8 *rho = &pk[384 * k];
+
+ /* Generate the vector NTT(y) from r. */
+ for (int i = 0; i < k; i++) {
+ sample_poly_cbd(&ws->y[i], r, i, &ws->shake, ws->block);
+ ntt(&ws->y[i]);
+ }
+
+ /*
+ * Compute, compress, and encode u = NTT^-1( NTT(A^T) * NTT(y) ) + e_1.
+ * To save memory, do it one row at a time.
+ */
+ for (int i = 0; i < k; i++) { /* For each row in A^T (column in A) */
+ /* u = NTT(A^T) * NTT(y) */
+ ws->u = (struct mlkem_ring_elem){};
+ for (int j = 0; j < k;
+ j++) { /* For each column in A^T (row in A) */
+ sample_ntt(&ws->tmp, rho, j, i, &ws->shake, ws->block);
+ poly_mul_acc(&ws->u, &ws->tmp, &ws->y[j]);
+ }
+ /* u = NTT^-1(u) */
+ invntt(&ws->u);
+
+ /* u += e_1, where e_1 is generated from r */
+ sample_poly_cbd(&ws->tmp, r, k + i, &ws->shake, ws->block);
+ poly_add(&ws->u, &ws->tmp);
+
+ /* Compress and encode u into the ciphertext ct. */
+ compress_and_encode(&ct[32 * params->du * i], &ws->u,
+ params->du);
+ }
+
+ /* v = NTT^-1( NTT(t^T) * NTT(y) ) */
+ ws->v = (struct mlkem_ring_elem){};
+ for (int i = 0; i < k; i++) {
+ /* Decode next element of NTT(t) from pk. */
+ byte_decode(&ws->tmp, &pk[384 * i], 12);
+ if (is_out_of_range(&ws->tmp)) {
+ memzero_explicit(ct, params->ct_len);
+ return -EBADMSG; /* pk is malformed. */
+ }
+ poly_mul_acc(&ws->v, &ws->tmp, &ws->y[i]);
+ }
+ invntt(&ws->v);
+
+ /* v += e_2, where e_2 is generated from r */
+ sample_poly_cbd(&ws->tmp, r, 2 * k, &ws->shake, ws->block);
+ poly_add(&ws->v, &ws->tmp);
+
+ /* Add the message m, after decompressing from 1 bit to [0, Q - 1]. */
+ for (int i = 0; i < N; i++)
+ ws->v.x[i] =
+ reduce_once(ws->v.x[i] +
+ decompress_d((m[i / 8] >> (i & 7)) & 1, 1));
+
+ /* Compress and encode v into the ciphertext ct. */
+ compress_and_encode(&ct[32 * params->du * k], &ws->v, params->dv);
+ return 0;
+}
+
+/*
+ * Decrypt the ciphertext @ct using the secret key @sk.
+ *
+ * Reference: FIPS 203 Algorithm 15, K-PKE.Decrypt
+ */
+static int k_pke_decrypt(u8 m[32], const u8 *sk, const u8 *ct,
+ struct k_pke_decrypt_workspace *ws,
+ const struct mlkem_parameter_set *params)
+{
+ /* Compute s^T * u' */
+ ws->su = (struct mlkem_ring_elem){};
+ for (int i = 0; i < params->k; i++) {
+ /* Decode next element of NTT(s) from sk. */
+ byte_decode(&ws->s, &sk[384 * i], 12);
+ if (is_out_of_range(&ws->s))
+ return -EBADMSG; /* sk is malformed. */
+
+ /* Decode and decompress next element of u' from ct. */
+ decode_and_decompress(&ws->tmp, &ct[32 * params->du * i],
+ params->du);
+ /* NTT(u') [one element] */
+ ntt(&ws->tmp);
+
+ /* NTT(s^T) * NTT(u') [one element] */
+ poly_mul_acc(&ws->su, &ws->s, &ws->tmp);
+ }
+ invntt(&ws->su);
+
+ /* Decode and decompress v' from the ciphertext ct. */
+ decode_and_decompress(&ws->tmp, &ct[32 * params->du * params->k],
+ params->dv);
+
+ /* w = v' - s^T * u' */
+ poly_sub(&ws->tmp, &ws->su);
+
+ /* Decode the plaintext m from the polynomial w. */
+ compress_and_encode(m, &ws->tmp, 1);
+ return 0;
+}
+
+/*
+ * Generate an ML-KEM key pair (@pk, @sk) from the random seed @seed.
+ * The lengths of @pk and @sk are determined by the chosen @params.
+ *
+ * Reference: FIPS 203 Algorithm 16, ML-KEM.KeyGen_internal. Formally it takes
+ * two 32-byte seeds 'd' and 'z'; here @seed is d || z.
+ */
+static int mlkem_keygen_internal(u8 *pk, u8 *sk,
+ const u8 seed[MLKEM_SEED_BYTES],
+ const struct mlkem_parameter_set *params)
+{
+ u8 *sk_ptr;
+ int err;
+
+ /* Generate a key pair for the public key encryption scheme K-PKE. */
+ err = k_pke_keygen(pk, sk, &seed[0], params);
+ if (err)
+ return err;
+
+ /*
+ * Generate the ML-KEM key pair as
+ * Public key (encapsulation key): pk = pk_pke
+ * Secret key (decapsulation key): sk = sk_pke || pk || H(pk) || z
+ *
+ * For pk there's nothing to add. This just handles the sk.
+ */
+ sk_ptr = &sk[384 * params->k]; /* end of sk_pke */
+ memcpy(sk_ptr, pk, params->pk_len);
+ sk_ptr += params->pk_len;
+ sha3_256(pk, params->pk_len, sk_ptr);
+ sk_ptr += SHA3_256_DIGEST_SIZE;
+ memcpy(sk_ptr, &seed[32], 32);
+
+ return 0;
+}
+
+int mlkem768_keygen_internal(u8 pk[MLKEM768_PUBLIC_KEY_BYTES],
+ u8 sk[MLKEM768_SECRET_KEY_BYTES],
+ const u8 seed[MLKEM_SEED_BYTES])
+{
+ return mlkem_keygen_internal(pk, sk, seed, &mlkem768);
+}
+EXPORT_SYMBOL_NS_GPL(mlkem768_keygen_internal, "CRYPTO_INTERNAL");
+
+int mlkem768_keygen(u8 pk[MLKEM768_PUBLIC_KEY_BYTES],
+ u8 sk[MLKEM768_SECRET_KEY_BYTES])
+{
+ u8 seed[MLKEM_SEED_BYTES];
+ int err;
+
+ get_random_bytes(seed, sizeof(seed));
+ err = mlkem768_keygen_internal(pk, sk, seed);
+ memzero_explicit(seed, sizeof(seed));
+ return err;
+}
+EXPORT_SYMBOL_NS_GPL(mlkem768_keygen, "CRYPTO_INTERNAL");
+
+int mlkem1024_keygen_internal(u8 pk[MLKEM1024_PUBLIC_KEY_BYTES],
+ u8 sk[MLKEM1024_SECRET_KEY_BYTES],
+ const u8 seed[MLKEM_SEED_BYTES])
+{
+ return mlkem_keygen_internal(pk, sk, seed, &mlkem1024);
+}
+EXPORT_SYMBOL_NS_GPL(mlkem1024_keygen_internal, "CRYPTO_INTERNAL");
+
+int mlkem1024_keygen(u8 pk[MLKEM1024_PUBLIC_KEY_BYTES],
+ u8 sk[MLKEM1024_SECRET_KEY_BYTES])
+{
+ u8 seed[MLKEM_SEED_BYTES];
+ int err;
+
+ get_random_bytes(seed, sizeof(seed));
+ err = mlkem1024_keygen_internal(pk, sk, seed);
+ memzero_explicit(seed, sizeof(seed));
+ return err;
+}
+EXPORT_SYMBOL_NS_GPL(mlkem1024_keygen, "CRYPTO_INTERNAL");
+
+/*
+ * Generate and encapsulate a shared secret, using the random seed @eseed.
+ *
+ * Reference: FIPS 203 Algorithm 17, ML-KEM.Encaps_internal
+ */
+static int mlkem_encaps_internal(u8 *ct, u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 *pk,
+ const u8 eseed[MLKEM_ESEED_BYTES],
+ const struct mlkem_parameter_set *params)
+{
+ struct mlkem_encap_workspace *ws __free(kfree_sensitive) =
+ kmalloc_obj(*ws);
+ int err;
+
+ if (!ws)
+ return -ENOMEM;
+
+ /* Derived shared secret key K and randomness r. */
+ sha3_256(pk, params->pk_len, ws->pk_hash);
+ sha3_512_init(&ws->sha3);
+ sha3_update(&ws->sha3, eseed, MLKEM_ESEED_BYTES);
+ sha3_update(&ws->sha3, ws->pk_hash, SHA3_256_DIGEST_SIZE);
+ sha3_final(&ws->sha3, ws->K_and_r);
+
+ /* Encrypt eseed using K-PKE with randomness r. */
+ err = k_pke_encrypt(ct, pk, eseed, ws->r, &ws->k_pke_enc, params);
+ if (err == 0)
+ memcpy(ss, ws->K, MLKEM_SHARED_SECRET_BYTES);
+ return err;
+}
+
+int mlkem768_encaps_internal(u8 ct[MLKEM768_CIPHERTEXT_BYTES],
+ u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 pk[MLKEM768_PUBLIC_KEY_BYTES],
+ const u8 eseed[MLKEM_ESEED_BYTES])
+{
+ return mlkem_encaps_internal(ct, ss, pk, eseed, &mlkem768);
+}
+EXPORT_SYMBOL_NS_GPL(mlkem768_encaps_internal, "CRYPTO_INTERNAL");
+
+int mlkem768_encaps(u8 ct[MLKEM768_CIPHERTEXT_BYTES],
+ u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 pk[MLKEM768_PUBLIC_KEY_BYTES])
+{
+ u8 eseed[MLKEM_ESEED_BYTES];
+ int err;
+
+ get_random_bytes(eseed, sizeof(eseed));
+ err = mlkem768_encaps_internal(ct, ss, pk, eseed);
+ memzero_explicit(eseed, sizeof(eseed));
+ return err;
+}
+EXPORT_SYMBOL_NS_GPL(mlkem768_encaps, "CRYPTO_INTERNAL");
+
+int mlkem1024_encaps_internal(u8 ct[MLKEM1024_CIPHERTEXT_BYTES],
+ u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 pk[MLKEM1024_PUBLIC_KEY_BYTES],
+ const u8 eseed[MLKEM_ESEED_BYTES])
+{
+ return mlkem_encaps_internal(ct, ss, pk, eseed, &mlkem1024);
+}
+EXPORT_SYMBOL_NS_GPL(mlkem1024_encaps_internal, "CRYPTO_INTERNAL");
+
+int mlkem1024_encaps(u8 ct[MLKEM1024_CIPHERTEXT_BYTES],
+ u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 pk[MLKEM1024_PUBLIC_KEY_BYTES])
+{
+ u8 eseed[MLKEM_ESEED_BYTES];
+ int err;
+
+ get_random_bytes(eseed, sizeof(eseed));
+ err = mlkem1024_encaps_internal(ct, ss, pk, eseed);
+ memzero_explicit(eseed, sizeof(eseed));
+ return err;
+}
+EXPORT_SYMBOL_NS_GPL(mlkem1024_encaps, "CRYPTO_INTERNAL");
+
+/* Reference: FIPS 203 Algorithm 21, ML-KEM.Decaps */
+static int mlkem_decaps(u8 ss[MLKEM_SHARED_SECRET_BYTES], const u8 *ct,
+ const u8 *sk, const struct mlkem_parameter_set *params)
+{
+ struct mlkem_decap_workspace *ws __free(kfree_sensitive) =
+ kmalloc_obj(*ws);
+ const u8 *sk_pke, *pk_pke, *h, *z;
+ u32 mask;
+ int err;
+
+ if (!ws) {
+ err = -ENOMEM;
+ goto decaps_failed;
+ }
+
+ /* Extract the parts of the secret key. */
+ sk_pke = sk; /* PKE secret key */
+ pk_pke = &sk_pke[384 * params->k]; /* PKE public key */
+ h = &pk_pke[params->pk_len]; /* Hash of PKE public key */
+ z = &h[SHA3_256_DIGEST_SIZE]; /* Implicit rejection value */
+
+ /* Validate h, as specified in the "Hash check" in FIPS 203. */
+ sha3_256(pk_pke, params->pk_len, ws->h);
+ if (crypto_memneq(h, ws->h, SHA3_256_DIGEST_SIZE)) {
+ /*
+ * This is technically a branch on the contents of the secret
+ * key, but it only indicates whether it's malformed or not.
+ */
+ err = -EBADMSG;
+ goto decaps_failed;
+ }
+
+ /* m' = K-PKE.Decrypt(sk_pke, ct) */
+ err = k_pke_decrypt(ws->m_prime, sk_pke, ct, &ws->k_pke_dec, params);
+ if (err) {
+ /*
+ * This is technically a branch on the contents of the secret
+ * key, but it only indicates whether it's malformed or not.
+ */
+ goto decaps_failed;
+ }
+
+ /* (K', r') = G(m' || h) */
+ sha3_512_init(&ws->sha3);
+ sha3_update(&ws->sha3, ws->m_prime, 32);
+ sha3_update(&ws->sha3, h, SHA3_256_DIGEST_SIZE);
+ sha3_final(&ws->sha3, ws->K_prime_and_r_prime);
+
+ /* K_bar = J(z || ct) */
+ shake256_init(&ws->shake);
+ shake_update(&ws->shake, z, 32);
+ shake_update(&ws->shake, ct, params->ct_len);
+ shake_squeeze(&ws->shake, ws->K_bar, 32);
+
+ /* ct' = K-PKE.Encrypt(pk_pke, m', r') */
+ err = k_pke_encrypt(ws->ct_prime, pk_pke, ws->m_prime, ws->r_prime,
+ &ws->k_pke_enc, params);
+ if (err)
+ goto decaps_failed;
+
+ /*
+ * Return the shared secret: K_bar if ct != ct', else K'.
+ *
+ * This entire section must be constant-time with respect to not only
+ * the contents of ct, ct', K', and K_bar, but also whether K' or K_bar
+ * is chosen. crypto_memneq() isn't necessarily safe to use here, as it
+ * checks its result with a ternary expression.
+ */
+ mask = 0;
+ for (int i = 0; i < params->ct_len; i++) {
+ OPTIMIZER_HIDE_VAR(mask);
+ mask |= ct[i] ^ ws->ct_prime[i];
+ }
+ OPTIMIZER_HIDE_VAR(mask);
+ mask = -mask; /* bit 31 is 1 if ct != ct', else 0 */
+ OPTIMIZER_HIDE_VAR(mask);
+ mask = (s32)mask >> 31; /* mask = 0xffffffff if ct != ct', else 0 */
+ OPTIMIZER_HIDE_VAR(mask);
+ for (int i = 0; i < 32; i++)
+ ss[i] = (ws->K_bar[i] & mask) | (ws->K_prime[i] & ~mask);
+ return 0;
+
+decaps_failed:
+ /*
+ * On error, return a random shared secret. This is a safer default in
+ * the case where the caller forgets to check the return value.
+ */
+ get_random_bytes(ss, MLKEM_SHARED_SECRET_BYTES);
+ return err;
+}
+
+int mlkem768_decaps(u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 ct[MLKEM768_CIPHERTEXT_BYTES],
+ const u8 sk[MLKEM768_SECRET_KEY_BYTES])
+{
+ return mlkem_decaps(ss, ct, sk, &mlkem768);
+}
+EXPORT_SYMBOL_NS_GPL(mlkem768_decaps, "CRYPTO_INTERNAL");
+
+int mlkem1024_decaps(u8 ss[MLKEM_SHARED_SECRET_BYTES],
+ const u8 ct[MLKEM1024_CIPHERTEXT_BYTES],
+ const u8 sk[MLKEM1024_SECRET_KEY_BYTES])
+{
+ return mlkem_decaps(ss, ct, sk, &mlkem1024);
+}
+EXPORT_SYMBOL_NS_GPL(mlkem1024_decaps, "CRYPTO_INTERNAL");
+
+MODULE_DESCRIPTION("ML-KEM (Module-Lattice-Based Key Encapsulation Mechanism)");
+MODULE_LICENSE("GPL");
--
2.54.0
^ permalink raw reply related
* [PATCH 0/5] ML-KEM and X-Wing support
From: Eric Biggers @ 2026-05-25 18:43 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, Ard Biesheuvel, Jason A . Donenfeld, Herbert Xu,
Ryan Appel, Chris Leech, Eric Biggers
This series applies to v7.1-rc5. It is a proof-of-concept that won't be
merged until there is an in-kernel user. Multiple people have been
asking about this though, so I wanted to get ahead of the curve and
provide something that people can experiment with if needed.
This series adds support for "post-quantum" (i.e. quantum-resistant) key
encapsulation to the kernel's crypto library. Specifically this
includes ML-KEM-768 and ML-KEM-1024, and the X-Wing hybrid KEM built on
top of it. The ML-KEM functions are put in the CRYPTO_INTERNAL
namespace, as they will be used only as a component of hybrid KEMs.
It's likely this will eventually be useful for at least one of the
in-kernel users of classical key agreement schemes (currently NVMe
authentication, Bluetooth, and WireGuard). However, the details of the
upgrade to "post-quantum" will be up to the protocol authors in each
case. I suggest that X-Wing be chosen when possible.
Eric Biggers (5):
lib/crypto: mlkem: Add ML-KEM-768 and ML-KEM-1024 support
lib/crypto: mlkem: Add KUnit tests for ML-KEM
lib/crypto: mlkem: Add FIPS 140-3 tests
lib/crypto: xwing: Add support for X-Wing KEM
lib/crypto: xwing: Add KUnit tests for X-Wing KEM
Documentation/crypto/libcrypto-asymmetric.rst | 27 +
Documentation/crypto/libcrypto-signature.rst | 11 -
Documentation/crypto/libcrypto.rst | 2 +-
include/crypto/mlkem.h | 159 +++
include/crypto/xwing.h | 84 ++
lib/crypto/.kunitconfig | 2 +
lib/crypto/Kconfig | 17 +
lib/crypto/Makefile | 10 +
lib/crypto/fips-mlkem.h | 523 +++++++++
lib/crypto/mlkem.c | 1036 +++++++++++++++++
lib/crypto/tests/Kconfig | 18 +
lib/crypto/tests/Makefile | 2 +
lib/crypto/tests/mlkem-testvecs.h | 19 +
lib/crypto/tests/mlkem_kunit.c | 520 +++++++++
lib/crypto/tests/xwing-testvecs.h | 138 +++
lib/crypto/tests/xwing_kunit.c | 129 ++
lib/crypto/xwing.c | 237 ++++
scripts/crypto/import-mlkem-testvecs.py | 179 +++
scripts/crypto/import-xwing-testvecs.py | 111 ++
19 files changed, 3212 insertions(+), 12 deletions(-)
create mode 100644 Documentation/crypto/libcrypto-asymmetric.rst
delete mode 100644 Documentation/crypto/libcrypto-signature.rst
create mode 100644 include/crypto/mlkem.h
create mode 100644 include/crypto/xwing.h
create mode 100644 lib/crypto/fips-mlkem.h
create mode 100644 lib/crypto/mlkem.c
create mode 100644 lib/crypto/tests/mlkem-testvecs.h
create mode 100644 lib/crypto/tests/mlkem_kunit.c
create mode 100644 lib/crypto/tests/xwing-testvecs.h
create mode 100644 lib/crypto/tests/xwing_kunit.c
create mode 100644 lib/crypto/xwing.c
create mode 100755 scripts/crypto/import-mlkem-testvecs.py
create mode 100755 scripts/crypto/import-xwing-testvecs.py
base-commit: e7ae89a0c97ce2b68b0983cd01eda67cf373517d
--
2.54.0
^ permalink raw reply
* Re: [PATCH] MAINTAINERS: powerpc: update VMX AES entries
From: Breno Leitao @ 2026-05-25 15:30 UTC (permalink / raw)
To: Thorsten Blum
Cc: Herbert Xu, David S. Miller, Nayna Jain, Paulo Flabiano Smorigo,
Eric Biggers, Ard Biesheuvel, linux-crypto, linuxppc-dev,
linux-kernel
In-Reply-To: <20260524212943.799757-3-thorsten.blum@linux.dev>
On Sun, May 24, 2026 at 11:29:45PM +0000, Thorsten Blum wrote:
> Commit 7cf2082e74ce ("lib/crypto: powerpc/aes: Migrate POWER8 optimized
> code into library") removed arch/powerpc/crypto/aes.c and moved
> arch/powerpc/crypto/aesp8-ppc.pl to lib/crypto/powerpc/.
>
> However, the "IBM Power VMX Cryptographic instructions" entry still
> references the removed file and no longer covers the moved aesp8-ppc.pl.
>
> Remove the stale entry, add lib/crypto/powerpc/aesp8-ppc.pl, and tighten
> the arch/powerpc/crypto/aesp8-ppc.* pattern to match the remaining
> header only.
>
> Fixes: 7cf2082e74ce ("lib/crypto: powerpc/aes: Migrate POWER8 optimized code into library")
> Signed-off-by: Thorsten Blum <thorsten.blum@linux.dev>
Acked-by: Breno Leitao <leitao@debian.org>
^ permalink raw reply
* Re: [PATCH 0/3] Add support for qcrypto on shikra
From: Eric Biggers @ 2026-05-25 15:26 UTC (permalink / raw)
To: Kuldeep Singh
Cc: Thara Gopinath, Herbert Xu, David S. Miller, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson, Konrad Dybcio,
Vinod Koul, Frank Li, Andy Gross, linux-arm-msm, linux-crypto,
devicetree, linux-kernel, dmaengine, Bartosz Golaszewski,
Bartosz Golaszewski, Gaurav Kashyap, Neeraj Soni
In-Reply-To: <20260525142843.GA2018@quark>
On Mon, May 25, 2026 at 09:28:43AM -0500, Eric Biggers wrote:
> ARMv8 Crypto Extensions are "hardware" as well, just in the CPU. They
> provide constant-time execution, for example.
>
> Granted, they don't protect from power analysis and electromagnetic
> emanation attacks. Does QCE actually provide those protections, though?
>
> Either way, it doesn't really matter in this case. There are multiple
> aspects to security, and before even considering these advanced
> protections, the basics of security need to be absolutely solid. That
> is, the driver needs to always compute the crypto algorithms correctly,
> and it needs to be completely robust when fuzzed by unprivileged
> userspace (because it can accessed in that way).
Looks like these protections are not even present either. From
https://csrc.nist.gov/CSRC/media/projects/cryptographic-module-validation-program/documents/security-policies/140sp5077.pdf :
> The Qualcomm Crypto Engine Core does not support any non-invasive
> security techniques. Therefore, this section is not applicable.
[...]
> The Qualcomm Crypto Engine Core does not implement security
> mechanisms to mitigate other attacks.
- Eric
^ permalink raw reply
* Re: [PATCH] crypto: loongson - Select CRYPTO_RNG
From: Eric Biggers @ 2026-05-25 14:59 UTC (permalink / raw)
To: Qunqin Zhao
Cc: Huacai Chen, linux-crypto, Herbert Xu, linux-kernel, loongarch,
Yinggang Gu, Lee Jones, kernel test robot, stable
In-Reply-To: <05c794a7-f82d-5454-8df9-0ac543f8f8f7@loongson.cn>
On Mon, May 25, 2026 at 04:17:25PM +0800, Qunqin Zhao wrote:
>
> 在 2026/5/25 上午11:20, Eric Biggers 写道:
> > On Mon, May 25, 2026 at 10:45:14AM +0800, Qunqin Zhao wrote:
> > > > > To be honest, I previously assumed that the `hw_random` was designed
> > > > > strictly and exclusively for the TRNG mode.
> > > > >
> > > > > Is it architecturally acceptable or common practice for a PRNG mode to
> > > > > utilize `hw_random` as well?
> > > > >
> > > > > Thanks,
> > > > So the Loongson RNG is a PRNG? Where does it get its entropy from, and
> > > > what is its security strength?
> > > Loongson's hardware supports both TRNG and PRNG simultaneously.
> > >
> > > We can locate a reseed function within loongson-rng.c, which clearly
> > > indicates that it is a PRNG driver.
> > That reseed function gets called with entropy from the Linux RNG. So,
> > it seems it's really just a PRNG seeded from the Linux RNG. What value
> > does that provide over just using the Linux RNG directly?
>
> Alternatively,the reseed function can serve as a stirring mechanism, where
> the primary entropy comes from the internal hardware TRNG.
>
> Or simply ignore the entropy from the Linux RNG entirely, trigger a
> reseeding internal.
>
>
> The driver merely forwards the seed to the firmware; how it is utilized and
> what kind of random numbers are returned are entirely determined by the
> firmware implementation.
>
> >
> > > So the core issue here is whether a PRNG driver can utilize the crypto
> > > interface.
> > If you're asking about crypto_rng, it can. But the crypto_rng interface
> > is also kind of useless. If you're asking about hwrng, it does look
> > like it's designed for TRNGs. Would it be possible for this driver to
> > use the TRNG mode?
>
> I mean crypto_rng.
>
> We might use the hwrng interface to add support for the TRNG in this driver.
If you can actually provide fresh entropy on each call, then yes you
should implement hwrng.
> >
> > > If it cannot, does that imply the drivers listed below serve no practical
> > > purpose? (7.1-rc1)
> > >
> > > loongson@loongson:~/upstream/linux/drivers/crypto$ grep crypto_register_rng
> > Most of the drivers in drivers/crypto/ are added by the hardware
> > manufacturer without any regard for whether they're useful or not.
>
> If we are dropping crypto-rng drivers entirely,
We should, since they have no real point.
> I am fine with removing the Loongson driver along with the others.
>
> However, targeting the Loongson driver alone is unacceptable.
We just happen to be digging into the details on this driver right now,
since we're having to spend time fixing it. Thanks for confirming that
this is supported purely for parity with other drivers.
- Eric
^ permalink raw reply
* Re: [PATCH 0/3] Add support for qcrypto on shikra
From: Eric Biggers @ 2026-05-25 14:45 UTC (permalink / raw)
To: Dmitry Baryshkov
Cc: Kuldeep Singh, Thara Gopinath, Herbert Xu, David S. Miller,
Rob Herring, Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson,
Konrad Dybcio, Vinod Koul, Frank Li, Andy Gross, linux-arm-msm,
linux-crypto, devicetree, linux-kernel, dmaengine
In-Reply-To: <s5u3vlc3r2blg5hniwuqizazooldebr42n7hr2f4jw2ybbq3oe@cdmrv2etrmj2>
On Mon, May 25, 2026 at 01:07:49PM +0300, Dmitry Baryshkov wrote:
> Are other harware crypto drivers faster or slower than the CPU
> implementation? What about the CAAM (sorry, it's just the driver that I
> worked with few years ago). Or Xilinx? My guess would be that for the
> most of the modern ARM64 hardware the NEON implementation is faster than
> the "hw IP" one.
Yes, QCE is hardly unique here. It's just the one we're discussing now.
> My assumtion has always been that we support crypto IP
> for the sake of security (i.e. making sure that the key can't be found
> in the cleartext in memory dumps or that it's impossible to tamper with
> the hash values before singing/verification). From this point of view,
> using priorities is expected and logical: most of the users will need a
> quickest implementation. Some users will need to use protected keys or
> other hw-only features.
>
> Note, I'm not commenting on the driver being buggy. If the issues are
> not fixed in a timely manner, it should be marked with 'depends on
> BROKEN' and further removed if the issues contine to be non-fixed.
Only a few drivers support protected keys ("paes", "phmac"); QCE is
*not* one of them. There are also no explicit users of protected keys
in the kernel, so even if supported by the driver, it's almost never
used in practice in Linux. The only way this feature could potentially
be used in Linux is if one of these drivers is present *and* userspace
explicitly chooses to use it with one of the few kernel features that
might implicitly support it, e.g. dm-crypt. AFAIK that's extremely
rare, and at least in Linux it's really just a checkbox feature.
(HW-wrapped inline crypto keys do get used, but those don't use the
crypto API or these drivers at all.)
As for making it "impossible to tamper with the hash values before
signing/verification", these drivers don't provide anything there.
- Eric
^ permalink raw reply
* Re: [PATCH 0/3] Add support for qcrypto on shikra
From: Eric Biggers @ 2026-05-25 14:28 UTC (permalink / raw)
To: Kuldeep Singh
Cc: Thara Gopinath, Herbert Xu, David S. Miller, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson, Konrad Dybcio,
Vinod Koul, Frank Li, Andy Gross, linux-arm-msm, linux-crypto,
devicetree, linux-kernel, dmaengine, Bartosz Golaszewski,
Bartosz Golaszewski, Gaurav Kashyap, Neeraj Soni
In-Reply-To: <c1697372-54ec-4f57-85d9-ad375ff1a44d@oss.qualcomm.com>
On Mon, May 25, 2026 at 11:10:10AM +0530, Kuldeep Singh wrote:
> > It sounds like you don't actually have an answer to my questions, then.
> >
> > Performance tests (e.g.
> > https://lore.kernel.org/r/20250615031807.GA81869@sol/) have clearly
> > shown that this driver is an order of magnitude slower than the CPU.
> >
> > This driver has historically been quite harmful. People were using it
> > accidentally and encountering very bad performance, as well as bugs such
> > as crashes and filesystem hangs. We fixed that by lowering its
> > cra_priority. But for the same reason, even when enabled on a platform,
> > it's not actually used. Linux would be better without this driver.
> >
>
> +Bartosz, Gaurav, Neeraj
>
> Hi Eric,
>
> GPCE is relevant in terms of providing hardware security.
> There are multiple usecases coming up for example to handle DRM/secure
> buffer usecases to improve overall throughput for secure content.
>
> Regarding performance, it's currently slower compared to arm CE but
> provides an edge by giving hardware security which is considered more
> secure.
>
> Btw, there's been performance improvement with new targets and we are
> expecting to achieve far more better performance with new SoCs family.
> Pakala: GPCE - 550MBps, ARMv8 - 8GBps
> Kaanapali: GPCE - 3GBps, ARMv8 - 10GBps
>
> Please note, there's almost 5x improvement in kaanapali compared to
> pakala. Though overall is still slower compared to arm but as mentioned,
> expecting better performance with hardware improvements as we progress.
>
> Also, currently qce driver exhibit stability issues and that's what we
> are putting effort in stabilizing the software on immediate basis.
>
> There's parallel effort ongoing by Bartosz to introduce baseline for
> secure buffer usecases.
> https://lore.kernel.org/lkml/20260522-qcom-qce-cmd-descr-v18-0-99103926bafc@oss.qualcomm.com/
> There's active development ongoing and i believe lowering cra_priority
> for qce is fine as of now and can scale values once qce becomes
> performance efficient.
>
> Please share your thoughts. Thanks!
ARMv8 Crypto Extensions are "hardware" as well, just in the CPU. They
provide constant-time execution, for example.
Granted, they don't protect from power analysis and electromagnetic
emanation attacks. Does QCE actually provide those protections, though?
Either way, it doesn't really matter in this case. There are multiple
aspects to security, and before even considering these advanced
protections, the basics of security need to be absolutely solid. That
is, the driver needs to always compute the crypto algorithms correctly,
and it needs to be completely robust when fuzzed by unprivileged
userspace (because it can accessed in that way).
Yet, this driver "exhibits stability issues", fails the self-tests, and
doesn't even have exclusive access to the hardware! These are all
security bugs. That very much defeats the claimed point. (Plus, due to
the performance issues no one wants to use it in Linux anyway.)
As for "decrypting into secure buffers": if added that would be a new
feature, separate from the driver's current features. It's not even
supported by the crypto API. So regardless of whether this would be a
useful feature or not (it's unclear it would be), using it as an
argument that the current features of the driver are useful is nonsense.
As for performance getting higher over time, it's still irrelevant when
it's still much slower than the CPU, especially in practical conditions.
Yet, somehow this driver is documented as an "accelerator":
config CRYPTO_DEV_QCE
tristate "Qualcomm crypto engine accelerator"
The CPU is just much a better approach performance-wise. It has no
overhead in setting up DMA, waking/notifying the hardware, and context
switching. The CPU has a lot of room to improve too, via further
optimizations to hardware and the ISA, and in some cases software
optimizations such as interleaving. We've already seen this play out on
x86_64, where Vector AES boosted the AES throughput by a further 2-4x
from its already-super-fast performance.
- Eric
^ permalink raw reply
* Re: [PATCH 4/4] dm crypt: batch all sectors of a bio per crypto request
From: Mikulas Patocka @ 2026-05-25 12:02 UTC (permalink / raw)
To: Leonid Ravich
Cc: Herbert Xu, David S . Miller, Mike Snitzer, Alasdair Kergon,
Ard Biesheuvel, Eric Biggers, Jens Axboe, Horia Geanta,
Gilad Ben-Yossef, linux-crypto, dm-devel, linux-block
In-Reply-To: <20260519120002.27267-5-lravich@amazon.com>
[-- Attachment #1: Type: text/plain, Size: 15091 bytes --]
Hi
On Tue, 19 May 2026, Leonid Ravich wrote:
> When the underlying skcipher driver advertises support for multiple
> data units in a single request (CRYPTO_ALG_SKCIPHER_MULTI_DATA_UNIT),
> configure the cipher with cc->sector_size as data_unit_size and
> submit one request per bio instead of one request per sector. This
> removes per-sector overhead in the crypto API hot path: request
> allocation, callback dispatch, completion handling, and SG setup.
>
> The optimisation is enabled automatically at table load when all
> of the following hold:
>
> - the cipher is non-aead (i.e. skcipher);
> - tfms_count is 1 (interleaved per-sector keys would break batching);
> - the IV mode is plain or plain64 (the only modes whose generator
> produces a sequential 64-bit little-endian counter that the cipher
> can extend by adding the data-unit index, matching the convention
> documented in crypto_skcipher_set_data_unit_size());
> - the iv_gen_ops->post() hook is unset (lmk and tcw use it; both are
> already excluded by the IV-mode test, but the explicit check makes
> the assumption durable against future IV modes);
> - dm-integrity is not stacked (no integrity tag or integrity IV);
> - the cipher driver advertises multi-data-unit support.
>
> A new CRYPT_MULTI_DATA_UNIT cipher_flag, set once at construction
> time, gates the multi-data-unit path. The existing per-sector path
> in crypt_convert_block_skcipher() is unchanged; the new
> crypt_convert_block_skcipher_multi() is reached from a small dispatch
> in crypt_convert() and shares the same backlog/-EBUSY/-EINPROGRESS
> flow control with the per-sector path.
>
> Heap-allocated scatterlists are stashed in dm_crypt_request and freed
> in crypt_free_req_skcipher() to avoid races between the synchronous-
> success free path and async-completion reuse from the request pool.
>
> On -ENOMEM during scatterlist allocation, the bio is requeued via
> BLK_STS_DEV_RESOURCE rather than failed, matching the behaviour of
> the existing -ENOMEM path for crypto request allocation.
You should make sure that you do not attempt to use the multi-data-unit
mode when you retry the bio, otherwise it could loop indefinitely. Note
that there are people who swap to dm-crypt - and so, it must work even if
the memory is totally exhausted.
You should also use GFP_NOIO | __GFP_NORETRY instead of GFP_NOIO, so that
the code doesn't loop in the allocator forever.
Perhaps __bio_for_each_bvec would be better than __bio_for_each_segment,
so that it works faster with folios.
Mikulas
> Verified end-to-end with a byte-equivalence test: encrypted output of
> plain64 dm-crypt with the multi-data-unit path matches output of the
> single-data-unit path bit-for-bit over a 256 MB device.
>
> Signed-off-by: Leonid Ravich <lravich@amazon.com>
> ---
> drivers/md/dm-crypt.c | 248 ++++++++++++++++++++++++++++++++++++++++--
> 1 file changed, 241 insertions(+), 7 deletions(-)
>
> diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c
> index 5ef43231fe77..b35831d43f0e 100644
> --- a/drivers/md/dm-crypt.c
> +++ b/drivers/md/dm-crypt.c
> @@ -98,6 +98,14 @@ struct dm_crypt_request {
> struct scatterlist sg_in[4];
> struct scatterlist sg_out[4];
> u64 iv_sector;
> + /*
> + * Heap-allocated scatterlists used by the multi-data-unit path
> + * when one bio is processed in a single skcipher request. NULL
> + * when the inline sg_in[]/sg_out[] arrays above are sufficient
> + * (single-data-unit path). Freed in crypt_free_req_skcipher().
> + */
> + struct scatterlist *sg_in_ext;
> + struct scatterlist *sg_out_ext;
> };
>
> struct crypt_config;
> @@ -149,6 +157,7 @@ enum cipher_flags {
> CRYPT_IV_LARGE_SECTORS, /* Calculate IV from sector_size, not 512B sectors */
> CRYPT_ENCRYPT_PREPROCESS, /* Must preprocess data for encryption (elephant) */
> CRYPT_KEY_MAC_SIZE_SET, /* The integrity_key_size option was used */
> + CRYPT_MULTI_DATA_UNIT, /* Batch all sectors of a bio per crypto request */
> };
>
> /*
> @@ -1501,12 +1510,139 @@ static int crypt_convert_block_skcipher(struct crypt_config *cc,
> return r;
> }
>
> +/*
> + * Multi-data-unit variant of crypt_convert_block_skcipher. Submits all
> + * remaining sectors of the current bio in one skcipher request whose
> + * data_unit_size is cc->sector_size. The cipher walks the IV between
> + * data units (see crypto_skcipher_set_data_unit_size()).
> + *
> + * Returns the same set of values as crypt_convert_block_skcipher:
> + * 0 on synchronous success (full chunk processed),
> + * -EINPROGRESS / -EBUSY on asynchronous dispatch,
> + * -ENOMEM if scatterlist allocation fails (caller maps to
> + * BLK_STS_DEV_RESOURCE so the bio is requeued, not failed),
> + * negative errno otherwise.
> + *
> + * On success the bio iterators have been advanced by the chunk size.
> + */
> +static int crypt_convert_block_skcipher_multi(struct crypt_config *cc,
> + struct convert_context *ctx,
> + struct skcipher_request *req,
> + unsigned int *out_processed)
> +{
> + const unsigned int sector_size = cc->sector_size;
> + unsigned int total_in = ctx->iter_in.bi_size;
> + unsigned int total_out = ctx->iter_out.bi_size;
> + unsigned int total = min(total_in, total_out);
> + unsigned int n_sectors;
> + unsigned int n_sg_in = 0, n_sg_out = 0;
> + struct dm_crypt_request *dmreq = dmreq_of_req(cc, req);
> + struct scatterlist *sg_in = NULL, *sg_out = NULL;
> + struct bvec_iter iter_in, iter_out;
> + struct bio_vec bv;
> + u8 *iv, *org_iv;
> + int r;
> +
> + if (unlikely(total < sector_size))
> + return -EIO;
> + n_sectors = total / sector_size;
> + total = n_sectors * sector_size;
> +
> + /*
> + * Walk the bio_vec iterators to count how many SG entries we need
> + * for exactly @total bytes. bi_size of the iterators is at least
> + * @total by construction above.
> + */
> + iter_in = ctx->iter_in;
> + iter_in.bi_size = total;
> + __bio_for_each_segment(bv, ctx->bio_in, iter_in, iter_in)
> + n_sg_in++;
> +
> + iter_out = ctx->iter_out;
> + iter_out.bi_size = total;
> + __bio_for_each_segment(bv, ctx->bio_out, iter_out, iter_out)
> + n_sg_out++;
> +
> + sg_in = kmalloc_array(n_sg_in, sizeof(*sg_in), GFP_NOIO);
> + sg_out = (ctx->bio_in == ctx->bio_out) ? sg_in :
> + kmalloc_array(n_sg_out, sizeof(*sg_out), GFP_NOIO);
> + if (!sg_in || !sg_out) {
> + kfree(sg_in);
> + if (sg_out != sg_in)
> + kfree(sg_out);
> + return -ENOMEM;
> + }
> +
> + sg_init_table(sg_in, n_sg_in);
> + {
> + unsigned int i = 0;
> +
> + iter_in = ctx->iter_in;
> + iter_in.bi_size = total;
> + __bio_for_each_segment(bv, ctx->bio_in, iter_in, iter_in)
> + sg_set_page(&sg_in[i++], bv.bv_page, bv.bv_len,
> + bv.bv_offset);
> + }
> +
> + if (sg_out != sg_in) {
> + unsigned int i = 0;
> +
> + sg_init_table(sg_out, n_sg_out);
> + iter_out = ctx->iter_out;
> + iter_out.bi_size = total;
> + __bio_for_each_segment(bv, ctx->bio_out, iter_out, iter_out)
> + sg_set_page(&sg_out[i++], bv.bv_page, bv.bv_len,
> + bv.bv_offset);
> + }
> +
> + /*
> + * Compute the IV for the first data unit. The cipher will derive
> + * IVs for subsequent data units by treating this one as a 128-bit
> + * little-endian counter and adding the data-unit index, which
> + * matches the layout produced by plain and plain64.
> + */
> + dmreq->iv_sector = ctx->cc_sector;
> + if (test_bit(CRYPT_IV_LARGE_SECTORS, &cc->cipher_flags))
> + dmreq->iv_sector >>= cc->sector_shift;
> + dmreq->ctx = ctx;
> +
> + iv = iv_of_dmreq(cc, dmreq);
> + org_iv = org_iv_of_dmreq(cc, dmreq);
> + r = cc->iv_gen_ops->generator(cc, org_iv, dmreq);
> + if (r < 0)
> + goto out_free_sg;
> + memcpy(iv, org_iv, cc->iv_size);
> +
> + /* Stash the SG arrays for cleanup on completion / free. */
> + dmreq->sg_in_ext = sg_in;
> + dmreq->sg_out_ext = (sg_out == sg_in) ? NULL : sg_out;
> +
> + skcipher_request_set_crypt(req, sg_in, sg_out, total, iv);
> +
> + if (bio_data_dir(ctx->bio_in) == WRITE)
> + r = crypto_skcipher_encrypt(req);
> + else
> + r = crypto_skcipher_decrypt(req);
> +
> + *out_processed = total;
> + return r;
> +
> +out_free_sg:
> + kfree(sg_in);
> + if (sg_out != sg_in)
> + kfree(sg_out);
> + dmreq->sg_in_ext = NULL;
> + dmreq->sg_out_ext = NULL;
> + return r;
> +}
> +
> static void kcryptd_async_done(void *async_req, int error);
>
> static int crypt_alloc_req_skcipher(struct crypt_config *cc,
> struct convert_context *ctx)
> {
> unsigned int key_index = ctx->cc_sector & (cc->tfms_count - 1);
> + struct dm_crypt_request *dmreq;
>
> if (!ctx->r.req) {
> ctx->r.req = mempool_alloc(&cc->req_pool, in_interrupt() ? GFP_ATOMIC : GFP_NOIO);
> @@ -1516,6 +1652,18 @@ static int crypt_alloc_req_skcipher(struct crypt_config *cc,
>
> skcipher_request_set_tfm(ctx->r.req, cc->cipher_tfm.tfms[key_index]);
>
> + /*
> + * Initialise the heap-allocated scatterlist pointers so that
> + * crypt_free_req_skcipher() does not read uninitialised memory
> + * for paths that don't take the multi-data-unit branch. The
> + * dmreq trailer lives in the per-bio data area which is not
> + * zeroed by the dm core, and the request is reused from the
> + * mempool across many bios.
> + */
> + dmreq = dmreq_of_req(cc, ctx->r.req);
> + dmreq->sg_in_ext = NULL;
> + dmreq->sg_out_ext = NULL;
> +
> /*
> * Use REQ_MAY_BACKLOG so a cipher driver internally backlogs
> * requests if driver request queue is full.
> @@ -1562,6 +1710,12 @@ static void crypt_free_req_skcipher(struct crypt_config *cc,
> struct skcipher_request *req, struct bio *base_bio)
> {
> struct dm_crypt_io *io = dm_per_bio_data(base_bio, cc->per_bio_data_size);
> + struct dm_crypt_request *dmreq = dmreq_of_req(cc, req);
> +
> + kfree(dmreq->sg_in_ext);
> + dmreq->sg_in_ext = NULL;
> + kfree(dmreq->sg_out_ext);
> + dmreq->sg_out_ext = NULL;
>
> if ((struct skcipher_request *)(io + 1) != req)
> mempool_free(req, &cc->req_pool);
> @@ -1590,7 +1744,9 @@ static void crypt_free_req(struct crypt_config *cc, void *req, struct bio *base_
> static blk_status_t crypt_convert(struct crypt_config *cc,
> struct convert_context *ctx, bool atomic, bool reset_pending)
> {
> - unsigned int sector_step = cc->sector_size >> SECTOR_SHIFT;
> + const unsigned int sector_step = cc->sector_size >> SECTOR_SHIFT;
> + const bool multi_du = test_bit(CRYPT_MULTI_DATA_UNIT, &cc->cipher_flags);
> + unsigned int processed;
> int r;
>
> /*
> @@ -1611,8 +1767,13 @@ static blk_status_t crypt_convert(struct crypt_config *cc,
>
> atomic_inc(&ctx->cc_pending);
>
> + processed = cc->sector_size;
> if (crypt_integrity_aead(cc))
> r = crypt_convert_block_aead(cc, ctx, ctx->r.req_aead, ctx->tag_offset);
> + else if (multi_du)
> + r = crypt_convert_block_skcipher_multi(cc, ctx,
> + ctx->r.req,
> + &processed);
> else
> r = crypt_convert_block_skcipher(cc, ctx, ctx->r.req, ctx->tag_offset);
>
> @@ -1634,8 +1795,19 @@ static blk_status_t crypt_convert(struct crypt_config *cc,
> * exit and continue processing in a workqueue
> */
> ctx->r.req = NULL;
> - ctx->tag_offset++;
> - ctx->cc_sector += sector_step;
> + if (!multi_du) {
> + ctx->tag_offset++;
> + ctx->cc_sector += sector_step;
> + } else {
> + bio_advance_iter(ctx->bio_in,
> + &ctx->iter_in,
> + processed);
> + bio_advance_iter(ctx->bio_out,
> + &ctx->iter_out,
> + processed);
> + ctx->cc_sector +=
> + processed >> SECTOR_SHIFT;
> + }
> return BLK_STS_DEV_RESOURCE;
> }
> } else {
> @@ -1649,19 +1821,42 @@ static blk_status_t crypt_convert(struct crypt_config *cc,
> */
> case -EINPROGRESS:
> ctx->r.req = NULL;
> - ctx->tag_offset++;
> - ctx->cc_sector += sector_step;
> + if (!multi_du) {
> + ctx->tag_offset++;
> + ctx->cc_sector += sector_step;
> + } else {
> + bio_advance_iter(ctx->bio_in, &ctx->iter_in,
> + processed);
> + bio_advance_iter(ctx->bio_out, &ctx->iter_out,
> + processed);
> + ctx->cc_sector += processed >> SECTOR_SHIFT;
> + }
> continue;
> /*
> * The request was already processed (synchronously).
> */
> case 0:
> atomic_dec(&ctx->cc_pending);
> - ctx->cc_sector += sector_step;
> - ctx->tag_offset++;
> + if (!multi_du) {
> + ctx->cc_sector += sector_step;
> + ctx->tag_offset++;
> + } else {
> + bio_advance_iter(ctx->bio_in, &ctx->iter_in,
> + processed);
> + bio_advance_iter(ctx->bio_out, &ctx->iter_out,
> + processed);
> + ctx->cc_sector += processed >> SECTOR_SHIFT;
> + }
> if (!atomic)
> cond_resched();
> continue;
> + /*
> + * Out of memory for the multi-DU SG arrays — bounce back
> + * to the caller for requeue rather than failing the bio.
> + */
> + case -ENOMEM:
> + atomic_dec(&ctx->cc_pending);
> + return BLK_STS_DEV_RESOURCE;
> /*
> * There was a data integrity error.
> */
> @@ -3142,6 +3337,45 @@ static int crypt_ctr_cipher(struct dm_target *ti, char *cipher_in, char *key)
> }
> }
>
> + /*
> + * Enable multi-data-unit batching when the cipher supports it and
> + * the IV layout is one we can derive per-DU from a single starting
> + * IV: plain or plain64 produce a sequential 64-bit little-endian
> + * counter, which matches the convention of
> + * crypto_skcipher_set_data_unit_size(). Restrict to the simple
> + * case (single tfm, no integrity, no per-sector post() callback)
> + * to keep the consumer path small; modes like essiv, lmk, tcw,
> + * eboiv, plain64be, random, null, benbi, and elephant are
> + * deliberately excluded because their generators or post-IV hooks
> + * cannot be re-derived by the cipher between data units.
> + */
> + if (!crypt_integrity_aead(cc) && cc->tfms_count == 1 &&
> + cc->iv_gen_ops &&
> + (cc->iv_gen_ops == &crypt_iv_plain_ops ||
> + cc->iv_gen_ops == &crypt_iv_plain64_ops) &&
> + !cc->iv_gen_ops->post &&
> + !cc->integrity_tag_size && !cc->integrity_iv_size &&
> + crypto_skcipher_supports_multi_data_unit(cc->cipher_tfm.tfms[0])) {
> + ret = crypto_skcipher_set_data_unit_size(cc->cipher_tfm.tfms[0],
> + cc->sector_size);
> + if (!ret) {
> + set_bit(CRYPT_MULTI_DATA_UNIT, &cc->cipher_flags);
> + DMINFO("Using multi-data-unit crypto offload (du=%u)",
> + cc->sector_size);
> + } else {
> + /*
> + * The driver advertised the capability via cra_flags
> + * but rejected the requested data unit size. This is
> + * a driver bug worth seeing in dmesg; fall back to
> + * the per-sector path so the device still activates.
> + */
> + DMWARN_LIMIT("multi-DU offload disabled: %s rejected du=%u (%d)",
> + crypto_skcipher_driver_name(cc->cipher_tfm.tfms[0]),
> + cc->sector_size, ret);
> + ret = 0;
> + }
> + }
> +
> /* wipe the kernel key payload copy */
> if (cc->key_string)
> memset(cc->key, 0, cc->key_size * sizeof(u8));
> --
> 2.47.3
>
^ permalink raw reply
* [PATCH crypto 1/1] crypto: pcrypt: restore callback for non-parallel fallback
From: Ren Wei @ 2026-05-25 11:45 UTC (permalink / raw)
To: linux-crypto
Cc: steffen.klassert, herbert, davem, yiyang13, yuantan098, yifanwucs,
tomapufckgml, zcliangcn, bird, ruijieli51, n05ec
In-Reply-To: <cover.1779697691.git.ruijieli51@gmail.com>
From: Ruijie Li <ruijieli51@gmail.com>
pcrypt installs pcrypt_aead_done() on the child AEAD request before
trying to submit it through padata. If padata_do_parallel() returns
-EBUSY, pcrypt falls back to calling the child AEAD directly.
That fallback must not keep the padata completion callback. Otherwise
an asynchronous completion runs pcrypt_aead_done() even though the
request was never enrolled in padata.
Restore the original request callback and callback data before calling
the child AEAD directly. This keeps the fallback path aligned with a
direct AEAD request while leaving the parallel path unchanged.
Fixes: 662f2f13e66d ("crypto: pcrypt - Call crypto layer directly when padata_do_parallel() return -EBUSY")
Cc: stable@kernel.org
Reported-by: Yuan Tan <yuantan098@gmail.com>
Reported-by: Yifan Wu <yifanwucs@gmail.com>
Reported-by: Juefei Pu <tomapufckgml@gmail.com>
Reported-by: Zhengchuan Liang <zcliangcn@gmail.com>
Reported-by: Xin Liu <bird@lzu.edu.cn>
Assisted-by: Codex:gpt-5.4
Signed-off-by: Ruijie Li <ruijieli51@gmail.com>
Signed-off-by: Ren Wei <n05ec@lzu.edu.cn>
---
crypto/pcrypt.c | 4 ++++
1 file changed, 4 insertions(+)
diff --git a/crypto/pcrypt.c b/crypto/pcrypt.c
index ed0feaba2383..9f372442981e 100644
--- a/crypto/pcrypt.c
+++ b/crypto/pcrypt.c
@@ -122,6 +122,8 @@ static int pcrypt_aead_encrypt(struct aead_request *req)
return -EINPROGRESS;
if (err == -EBUSY) {
/* try non-parallel mode */
+ aead_request_set_callback(creq, flags, req->base.complete,
+ req->base.data);
return crypto_aead_encrypt(creq);
}
@@ -173,6 +175,8 @@ static int pcrypt_aead_decrypt(struct aead_request *req)
return -EINPROGRESS;
if (err == -EBUSY) {
/* try non-parallel mode */
+ aead_request_set_callback(creq, flags, req->base.complete,
+ req->base.data);
return crypto_aead_decrypt(creq);
}
--
2.47.3
^ permalink raw reply related
* Re: [PATCH 1/2] crypto: Delete Qualcomm crypto engine driver
From: Kuldeep Singh @ 2026-05-25 11:10 UTC (permalink / raw)
To: Dmitry Baryshkov
Cc: Eric Biggers, Krzysztof Kozlowski, Demi Marie Obenour, Herbert Xu,
David S. Miller, Thara Gopinath, Rob Herring, Krzysztof Kozlowski,
Conor Dooley, Bjorn Andersson, Konrad Dybcio, Russell King,
linux-kernel, linux-crypto, linux-arm-msm, Ard Biesheuvel,
devicetree, linux-arm-kernel
In-Reply-To: <rb35vwvusd733s2zbgsitof3cpoyyfope4cpeh47xldw27ufix@7fm6bgr4wtkz>
On 25-05-2026 13:41, Dmitry Baryshkov wrote:
> On Mon, May 25, 2026 at 11:59:33AM +0530, Kuldeep Singh wrote:
>>> This driver is more than an order of magnitude slower than the CPU for
>>> both encryption and hashing. See:
>>>
>>> https://lore.kernel.org/r/20250704070322.20692-1-ebiggers@kernel.org/
>>> https://lore.kernel.org/r/20250615031807.GA81869@sol/
>>>
>>> There are many examples of it having bugs as well, for example see the
>>> second link above.
>>>
>>> That's why it had to be disabled via the cra_priority system. This
>>> driver was actively making Linux worse.
>>>
>>> This isn't particularly unique to drivers/crypto/, of course. This one
>>> we just have data on, so it's a bit clearer.
>>>
>>> I've yet to see any real reason to keep this driver.
>>
>> https://lore.kernel.org/all/c1697372-54ec-4f57-85d9-ad375ff1a44d@oss.qualcomm.com/
>> Kindly check my latest reply to the thread. There are numerous usecases
>> like DRM(Digital rights management) coming up and qce driver is required
>> for secure content.
>>
>>> Crypto drivers need to be held to a higher standard than other device
>>> drivers, as well. The onus is on those who want to keep a particular
>>> crypto driver to prove that it's worth keeping.
>> Sure, I'll be working on stabilizing self_tests infra for qce.
>> Kindly allow sometime to go over failures in crypto selftest and will
>> submit fix if applicable.
>
> Why are not selftests a part of your default config? How are you testing
> the driver?
We are validating the driver probe and kcapi userspace tests while
validating qce algos. Locally we update all 3 algo priority and then run
kcapi_convinience.sh test suite to ensure it picks qce algos.
Please note, previously were not aware need to run selftests suite too.
Got to know about this once issue was reported.
>
>> So far, i am observing 2 ciphers failing(xts-aes-qce and ctr-aes-qce )
>> with CONFIG_CRYPTO_SELFTESTS enabled.
>
> Please make sure to run the tests on older platforms too, not just the
> latest & greatest.
Yes, will check on all the platforms.
--
Regards
Kuldeep
^ permalink raw reply
* Re: [PATCH 3/3] arm64: dts: qcom: shikra: Add qcrypto node support
From: Dmitry Baryshkov @ 2026-05-25 10:43 UTC (permalink / raw)
To: Kuldeep Singh
Cc: Konrad Dybcio, Thara Gopinath, Herbert Xu, David S. Miller,
Rob Herring, Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson,
Konrad Dybcio, Vinod Koul, Frank Li, Andy Gross, linux-arm-msm,
linux-crypto, devicetree, linux-kernel, dmaengine
In-Reply-To: <8a1d6c78-fd16-4994-bae9-cf75b1e7e3c5@oss.qualcomm.com>
On Mon, May 25, 2026 at 03:39:17PM +0530, Kuldeep Singh wrote:
> >>> These two entries are logically the same (SID & ~mask) as the first two,
> >>> does it still work if you remove them?
> >>
> >> Yes, resulting sid is same for 84/94 and 86/92.
> >> Basically, the resulting sid could be same, it's an optimization which
> >> smmu is doing which can result in same SMR(Stream matching register)
> >> routing 2 different sid to same context bank.
> >> So, 2 sid can be used even though resulting sid remains same.
> >>
> >> Also, DT usually dictates what hw capabilities are supported and hence,
> >> captured all apps entries here to match the hardware description.
> >>
> >> I hope this answers your query.
> >
> > It doesn't. Can we drop them?
>
> Could you please explain more on what's missing?
Usually we don't have duplciate SIDs in DT. Why is it not the case for
this device?
--
With best wishes
Dmitry
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox