* [PATCH 01/33] crypto: xts - Split out __xts_verify_key() helper
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 13:20 ` Thomas Huth
2026-07-07 5:34 ` [PATCH 02/33] lib/crypto: aes: Add ECB support Eric Biggers
` (31 subsequent siblings)
32 siblings, 1 reply; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Make the AES-XTS key verification code callable by the crypto library by
splitting out a helper function that doesn't use crypto_skcipher.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
include/crypto/xts.h | 18 ++++++++++++++----
1 file changed, 14 insertions(+), 4 deletions(-)
diff --git a/include/crypto/xts.h b/include/crypto/xts.h
index 15b16c4853d8..16aef89f021f 100644
--- a/include/crypto/xts.h
+++ b/include/crypto/xts.h
@@ -7,9 +7,9 @@
#include <linux/fips.h>
#define XTS_BLOCK_SIZE 16
+#define XTS_FORBID_WEAK_KEYS (1 << 0)
-static inline int xts_verify_key(struct crypto_skcipher *tfm,
- const u8 *key, unsigned int keylen)
+static inline int __xts_verify_key(const u8 *key, size_t keylen, int flags)
{
/*
* key consists of keys of equal size concatenated, therefore
@@ -29,12 +29,22 @@ static inline int xts_verify_key(struct crypto_skcipher *tfm,
* Ensure that the AES and tweak key are not identical when
* in FIPS mode or the FORBID_WEAK_KEYS flag is set.
*/
- if ((fips_enabled || (crypto_skcipher_get_flags(tfm) &
- CRYPTO_TFM_REQ_FORBID_WEAK_KEYS)) &&
+ if ((fips_enabled || (flags & XTS_FORBID_WEAK_KEYS)) &&
!crypto_memneq(key, key + (keylen / 2), keylen / 2))
return -EINVAL;
return 0;
}
+static inline int xts_verify_key(struct crypto_skcipher *tfm, const u8 *key,
+ unsigned int keylen)
+{
+ int flags = (crypto_skcipher_get_flags(tfm) &
+ CRYPTO_TFM_REQ_FORBID_WEAK_KEYS) ?
+ XTS_FORBID_WEAK_KEYS :
+ 0;
+
+ return __xts_verify_key(key, keylen, flags);
+}
+
#endif /* _CRYPTO_XTS_H */
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 02/33] lib/crypto: aes: Add ECB support
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
2026-07-07 5:34 ` [PATCH 01/33] crypto: xts - Split out __xts_verify_key() helper Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 13:59 ` Thomas Huth
2026-07-07 5:34 ` [PATCH 03/33] lib/crypto: aes: Add CBC and CBC-CTS support Eric Biggers
` (30 subsequent siblings)
32 siblings, 1 reply; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Add support for AES-ECB to the crypto library.
This will be used to provide a streamlined implementation of the
"ecb(aes)" crypto_skcipher algorithm. fs/crypto/keysetup_v1.c will also
use aes_ecb_encrypt() directly.
As usual, the architecture-optimized AES-ECB code will be migrated into
the library as well (using the hooks provided in this commit),
eliminating lots of repetitive boilerplate code.
ECB is obsolete of course, but we need this for parity with the
traditional API and to support some odd users of ECB in the kernel.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Create a documentation file libcrypto-unauth-encryption.rst to hold the
documentation for this and other unauthenticated encryption modes.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
.../crypto/libcrypto-unauth-encryption.rst | 28 ++++++++++
Documentation/crypto/libcrypto.rst | 1 +
include/crypto/aes-ecb.h | 49 ++++++++++++++++
lib/crypto/Kconfig | 9 ++-
lib/crypto/aes.c | 56 +++++++++++++++++++
lib/crypto/tests/Kconfig | 1 +
6 files changed, 142 insertions(+), 2 deletions(-)
create mode 100644 Documentation/crypto/libcrypto-unauth-encryption.rst
create mode 100644 include/crypto/aes-ecb.h
diff --git a/Documentation/crypto/libcrypto-unauth-encryption.rst b/Documentation/crypto/libcrypto-unauth-encryption.rst
new file mode 100644
index 000000000000..891c15279749
--- /dev/null
+++ b/Documentation/crypto/libcrypto-unauth-encryption.rst
@@ -0,0 +1,28 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later
+
+Unauthenticated encryption
+==========================
+
+Support for unauthenticated encryption and decryption, including bare stream
+ciphers and other length-preserving algorithms such as block ciphers in XTS
+mode. The legitimate use cases for these algorithms are:
+
+- Support for legacy protocols that really should have chosen an authenticated
+ mode (or even another primitive entirely) but didn't.
+
+- Internal components of authenticated modes. For example, AES-CTR is used by
+ AES-GCM and AES-CCM internally.
+
+- Storage encryption that cannot accommodate ciphertext expansion. Usually
+ AES-XTS is used for this.
+
+- Stream ciphers for key derivation and random number generation.
+
+Besides the above, these shouldn't be used.
+
+AES-ECB
+-------
+
+Support for AES in the ECB mode of operation.
+
+.. kernel-doc:: include/crypto/aes-ecb.h
diff --git a/Documentation/crypto/libcrypto.rst b/Documentation/crypto/libcrypto.rst
index a1557d45b0e5..bbf5ca137910 100644
--- a/Documentation/crypto/libcrypto.rst
+++ b/Documentation/crypto/libcrypto.rst
@@ -161,5 +161,6 @@ API documentation
libcrypto-blockcipher
libcrypto-hash
libcrypto-signature
+ libcrypto-unauth-encryption
libcrypto-utils
sha3
diff --git a/include/crypto/aes-ecb.h b/include/crypto/aes-ecb.h
new file mode 100644
index 000000000000..bfc56bdb082c
--- /dev/null
+++ b/include/crypto/aes-ecb.h
@@ -0,0 +1,49 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * AES-ECB unauthenticated encryption and decryption
+ *
+ * Copyright 2026 Google LLC
+ */
+#ifndef _CRYPTO_AES_ECB_H
+#define _CRYPTO_AES_ECB_H
+
+#include <crypto/aes.h>
+
+/**
+ * aes_ecb_encrypt() - Encrypt data using AES-ECB
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source data
+ * @len: Number of bytes to encrypt. Must be a multiple of AES_BLOCK_SIZE.
+ * @key: The key
+ *
+ * ECB mode is insecure by itself. This function exists only for compatibility
+ * with legacy protocols and for internal use by other modes.
+ *
+ * This supports incremental encryption, but the length of each chunk must be a
+ * multiple of AES_BLOCK_SIZE.
+ *
+ * Context: Any context.
+ */
+void aes_ecb_encrypt(u8 *dst, const u8 *src, size_t len, aes_encrypt_arg key);
+
+/**
+ * aes_ecb_decrypt() - Decrypt data using AES-ECB
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source data
+ * @len: Number of bytes to decrypt. Must be a multiple of AES_BLOCK_SIZE.
+ * @key: The key
+ *
+ * ECB mode is insecure by itself. This function exists only for compatibility
+ * with legacy protocols and for internal use by other modes.
+ *
+ * This supports incremental decryption, but the length of each chunk must be a
+ * multiple of AES_BLOCK_SIZE.
+ *
+ * Context: Any context.
+ */
+void aes_ecb_decrypt(u8 *dst, const u8 *src, size_t len,
+ const struct aes_key *key);
+
+#endif /* _CRYPTO_AES_ECB_H */
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index 591c1c2a7fb3..26514c181a7f 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -8,8 +8,7 @@ config CRYPTO_LIB_UTILS
config CRYPTO_LIB_AES
tristate
- # Select dependencies of modes that are part of libaes.
- select CRYPTO_LIB_UTILS if CRYPTO_LIB_AES_CBC_MACS
+ select CRYPTO_LIB_UTILS
config CRYPTO_LIB_AES_ARCH
bool
@@ -36,6 +35,12 @@ config CRYPTO_LIB_AES_CBC_MACS
this if your module uses any of the functions from
<crypto/aes-cbc-macs.h>.
+config CRYPTO_LIB_AES_ECB
+ tristate
+ select CRYPTO_LIB_AES
+ help
+ The AES-ECB library functions.
+
config CRYPTO_LIB_AESGCM
tristate
select CRYPTO_LIB_AES
diff --git a/lib/crypto/aes.c b/lib/crypto/aes.c
index ca733f15b2a8..e2f1ebf81405 100644
--- a/lib/crypto/aes.c
+++ b/lib/crypto/aes.c
@@ -5,6 +5,7 @@
*/
#include <crypto/aes-cbc-macs.h>
+#include <crypto/aes-ecb.h>
#include <crypto/aes.h>
#include <crypto/utils.h>
#include <linux/cache.h>
@@ -737,6 +738,61 @@ static inline void aes_cmac_fips_test(void)
}
#endif /* !CONFIG_CRYPTO_LIB_AES_CBC_MACS */
+#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_ECB)
+/*
+ * Hooks for optimized AES-ECB implementations, overridable by the architecture.
+ * They are called with len > 0 && len % AES_BLOCK_SIZE == 0. Returning false
+ * causes the fallback implementation to be used instead.
+ */
+#ifndef aes_ecb_encrypt_arch
+static bool aes_ecb_encrypt_arch(u8 *dst, const u8 *src, size_t len,
+ const struct aes_enckey *key)
+{
+ return false;
+}
+#endif
+#ifndef aes_ecb_decrypt_arch
+static bool aes_ecb_decrypt_arch(u8 *dst, const u8 *src, size_t len,
+ const struct aes_key *key)
+{
+ return false;
+}
+#endif
+
+void aes_ecb_encrypt(u8 *dst, const u8 *src, size_t len, aes_encrypt_arg key)
+{
+ if (WARN_ON_ONCE(len % AES_BLOCK_SIZE))
+ len = round_down(len, AES_BLOCK_SIZE);
+
+ if (unlikely(len == 0))
+ return;
+
+ if (likely(aes_ecb_encrypt_arch(dst, src, len, key.enc_key)))
+ return;
+
+ for (size_t i = 0; i < len; i += AES_BLOCK_SIZE)
+ aes_encrypt(key, &dst[i], &src[i]);
+}
+EXPORT_SYMBOL_GPL(aes_ecb_encrypt);
+
+void aes_ecb_decrypt(u8 *dst, const u8 *src, size_t len,
+ const struct aes_key *key)
+{
+ if (WARN_ON_ONCE(len % AES_BLOCK_SIZE))
+ len = round_down(len, AES_BLOCK_SIZE);
+
+ if (unlikely(len == 0))
+ return;
+
+ if (likely(aes_ecb_decrypt_arch(dst, src, len, key)))
+ return;
+
+ for (size_t i = 0; i < len; i += AES_BLOCK_SIZE)
+ aes_decrypt(key, &dst[i], &src[i]);
+}
+EXPORT_SYMBOL_GPL(aes_ecb_decrypt);
+#endif /* CONFIG_CRYPTO_LIB_AES_ECB */
+
static int __init aes_mod_init(void)
{
#ifdef aes_mod_init_arch
diff --git a/lib/crypto/tests/Kconfig b/lib/crypto/tests/Kconfig
index 9409c1a935c3..a57e87dbb1b1 100644
--- a/lib/crypto/tests/Kconfig
+++ b/lib/crypto/tests/Kconfig
@@ -145,6 +145,7 @@ config CRYPTO_LIB_ENABLE_ALL_FOR_KUNIT
tristate "Enable all crypto library code for KUnit tests"
depends on KUNIT
select CRYPTO_LIB_AES_CBC_MACS
+ select CRYPTO_LIB_AES_ECB
select CRYPTO_LIB_BLAKE2B
select CRYPTO_LIB_CHACHA20POLY1305
select CRYPTO_LIB_CURVE25519
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [PATCH 02/33] lib/crypto: aes: Add ECB support
2026-07-07 5:34 ` [PATCH 02/33] lib/crypto: aes: Add ECB support Eric Biggers
@ 2026-07-07 13:59 ` Thomas Huth
2026-07-07 19:22 ` Eric Biggers
0 siblings, 1 reply; 45+ messages in thread
From: Thomas Huth @ 2026-07-07 13:59 UTC (permalink / raw)
To: Eric Biggers, linux-crypto; +Cc: linux-kernel
On 07/07/2026 07.34, Eric Biggers wrote:
> Add support for AES-ECB to the crypto library.
>
> This will be used to provide a streamlined implementation of the
> "ecb(aes)" crypto_skcipher algorithm. fs/crypto/keysetup_v1.c will also
> use aes_ecb_encrypt() directly.
>
> As usual, the architecture-optimized AES-ECB code will be migrated into
> the library as well (using the hooks provided in this commit),
> eliminating lots of repetitive boilerplate code.
>
> ECB is obsolete of course, but we need this for parity with the
> traditional API and to support some odd users of ECB in the kernel.
>
> Initial test coverage is provided by the crypto_skcipher support added
> in a later commit. I'm planning a KUnit test suite as well.
>
> Create a documentation file libcrypto-unauth-encryption.rst to hold the
> documentation for this and other unauthenticated encryption modes.
>
> Signed-off-by: Eric Biggers <ebiggers@kernel.org>
> ---
> .../crypto/libcrypto-unauth-encryption.rst | 28 ++++++++++
> Documentation/crypto/libcrypto.rst | 1 +
> include/crypto/aes-ecb.h | 49 ++++++++++++++++
> lib/crypto/Kconfig | 9 ++-
> lib/crypto/aes.c | 56 +++++++++++++++++++
> lib/crypto/tests/Kconfig | 1 +
> 6 files changed, 142 insertions(+), 2 deletions(-)
> create mode 100644 Documentation/crypto/libcrypto-unauth-encryption.rst
> create mode 100644 include/crypto/aes-ecb.h
>
> diff --git a/Documentation/crypto/libcrypto-unauth-encryption.rst b/Documentation/crypto/libcrypto-unauth-encryption.rst
> new file mode 100644
> index 000000000000..891c15279749
> --- /dev/null
> +++ b/Documentation/crypto/libcrypto-unauth-encryption.rst
> @@ -0,0 +1,28 @@
> +.. SPDX-License-Identifier: GPL-2.0-or-later
> +
> +Unauthenticated encryption
> +==========================
> +
> +Support for unauthenticated encryption and decryption, including bare stream
> +ciphers and other length-preserving algorithms such as block ciphers in XTS
> +mode.
This sentence no verb?
> +
> +- Support for legacy protocols that really should have chosen an authenticated
> + mode (or even another primitive entirely) but didn't.
> +
> +- Internal components of authenticated modes. For example, AES-CTR is used by
> + AES-GCM and AES-CCM internally.
> +
> +- Storage encryption that cannot accommodate ciphertext expansion. Usually
> + AES-XTS is used for this.
> +
> +- Stream ciphers for key derivation and random number generation.
> +
> +Besides the above, these shouldn't be used.
> +
> +AES-ECB
> +-------
> +
> +Support for AES in the ECB mode of operation.
> +
> +.. kernel-doc:: include/crypto/aes-ecb.h
> diff --git a/Documentation/crypto/libcrypto.rst b/Documentation/crypto/libcrypto.rst
> index a1557d45b0e5..bbf5ca137910 100644
> --- a/Documentation/crypto/libcrypto.rst
> +++ b/Documentation/crypto/libcrypto.rst
> @@ -161,5 +161,6 @@ API documentation
> libcrypto-blockcipher
> libcrypto-hash
> libcrypto-signature
> + libcrypto-unauth-encryption
> libcrypto-utils
> sha3
> diff --git a/include/crypto/aes-ecb.h b/include/crypto/aes-ecb.h
> new file mode 100644
> index 000000000000..bfc56bdb082c
> --- /dev/null
> +++ b/include/crypto/aes-ecb.h
> @@ -0,0 +1,49 @@
> +/* SPDX-License-Identifier: GPL-2.0 */
> +/*
> + * AES-ECB unauthenticated encryption and decryption
> + *
> + * Copyright 2026 Google LLC
> + */
> +#ifndef _CRYPTO_AES_ECB_H
> +#define _CRYPTO_AES_ECB_H
> +
> +#include <crypto/aes.h>
> +
> +/**
> + * aes_ecb_encrypt() - Encrypt data using AES-ECB
> + * @dst: The destination buffer. Can be in-place or out-of-place. For other
> + * overlaps the behavior is unspecified.
> + * @src: The source data
> + * @len: Number of bytes to encrypt. Must be a multiple of AES_BLOCK_SIZE.
> + * @key: The key
> + *
> + * ECB mode is insecure by itself. This function exists only for compatibility
> + * with legacy protocols and for internal use by other modes.
> + *
> + * This supports incremental encryption, but the length of each chunk must be a
> + * multiple of AES_BLOCK_SIZE.
> + *
> + * Context: Any context.
> + */
> +void aes_ecb_encrypt(u8 *dst, const u8 *src, size_t len, aes_encrypt_arg key);
Other similar functions like aes_encrypt() use the key as first argument ...
so maybe do the same here, too, for consistency?
> +/**
> + * aes_ecb_decrypt() - Decrypt data using AES-ECB
> + * @dst: The destination buffer. Can be in-place or out-of-place. For other
> + * overlaps the behavior is unspecified.
> + * @src: The source data
> + * @len: Number of bytes to decrypt. Must be a multiple of AES_BLOCK_SIZE.
> + * @key: The key
> + *
> + * ECB mode is insecure by itself. This function exists only for compatibility
> + * with legacy protocols and for internal use by other modes.
> + *
> + * This supports incremental decryption, but the length of each chunk must be a
> + * multiple of AES_BLOCK_SIZE.
> + *
> + * Context: Any context.
> + */
> +void aes_ecb_decrypt(u8 *dst, const u8 *src, size_t len,
> + const struct aes_key *key);
dito
Apart from that, patch looks fine to me.
Thomas
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 02/33] lib/crypto: aes: Add ECB support
2026-07-07 13:59 ` Thomas Huth
@ 2026-07-07 19:22 ` Eric Biggers
2026-07-08 5:29 ` Thomas Huth
0 siblings, 1 reply; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 19:22 UTC (permalink / raw)
To: Thomas Huth
Cc: linux-crypto, linux-kernel, Jason A. Donenfeld, Ard Biesheuvel
On Tue, Jul 07, 2026 at 03:59:24PM +0200, Thomas Huth wrote:
> > +Unauthenticated encryption
> > +==========================
> > +
> > +Support for unauthenticated encryption and decryption, including bare stream
> > +ciphers and other length-preserving algorithms such as block ciphers in XTS
> > +mode.
>
> This sentence no verb?
It's a noun phrase that introduces what the section contains, before
transitioning into full sentences.
I used this in all existing Documentation/crypto/libcrypto-*.rst. It's
also fairly common in the help text for kconfig symbols (across the
kernel, not just the kconfig help text I've written).
I guess it's a bad practice. But if we go with something else here,
like "These functions provide support for ...", I should update the
existing libcrypto-*.rst too.
> > +void aes_ecb_encrypt(u8 *dst, const u8 *src, size_t len, aes_encrypt_arg key);
>
> Other similar functions like aes_encrypt() use the key as first argument ...
> so maybe do the same here, too, for consistency?
For single-block AES, there's indeed already aes_encrypt(key, dst, src)
and aes_decrypt(key, dst, src). But for actual AEAD encryption there's
already:
chacha20poly1305_encrypt(dst, src, src_len, ad, ad_len, nonce, key)
chacha20poly1305_decrypt(dst, src, src_len, ad, ad_len, nonce, key)
xchacha20poly1305_encrypt(dst, src, src_len, ad, ad_len, nonce, key)
xchacha20poly1305_decrypt(dst, src, src_len, ad, ad_len, nonce, key)
chacha20poly1305_encrypt_sg_inplace(src, src_len, ad, ad_len, nonce, key)
chacha20poly1305_decrypt_sg_inplace(src, src_len, ad, ad_len, nonce, key)
Those follow the convention described by Jason here:
https://lore.kernel.org/linux-crypto/aPT3dImhaI6Dpqs7@zx2c4.com/
This series prioritizes consistency with those (and other functions
taking [dst, src, len]] such as memcpy() and crypto_xor()), adding:
aes_ecb_encrypt(dst, src, len, key);
aes_ecb_decrypt(dst, src, len, key);
aes_cbc_encrypt(dst, src, len, iv, key);
aes_cbc_decrypt(dst, src, len, iv, key);
aes_cbc_cts_encrypt(dst, src, len, iv, key);
aes_cbc_cts_decrypt(dst, src, len, iv, key);
aes_ctr(dst, src, len, ctr, key);
aes_xctr(dst, src, len, ctr, iv, key);
aes_xts_encrypt(dst, src, len, tweak, key, cont);
aes_xts_decrypt(dst, src, len, tweak, key, cont);
aes_gcm_encrypt(dst, authtag, src, data_len, ad, ad_len, nonce, key)
aes_gcm_decrypt(dst, src, authtag, data_len, ad, ad_len, nonce, key)
aes_ccm_encrypt(dst, authtag, src, data_len, ad, ad_len, nonce, nonce_len, key)
aes_ccm_decrypt(dst, src, authtag, data_len, ad, ad_len, nonce, nonce_len, key)
(Side note: looking at it again, the last four maybe should all use
[dst, src, data_len, authtag]. In this series, the authtag is instead
grouped with the src or dst to which it's usually concatenated.)
If key is put at the beginning instead, it then raises the question of
why should it be different from nonce/iv/ctr and (ad, ad_len). So would
it really be:
aes_gcm_encrypt(key, dst, authtag, src, data_len, ad, ad_len, nonce)
... or would it actually be something like:
aes_gcm_encrypt(key, nonce, ad, ad_len, dst, authtag, src, data_len)
It's conventional to put "the object being operated on" at the
beginning, which could be argued to apply to the key. But alternatively
the key could just be considered another input. crypto_skcipher was
"object-like"; however, with the library the key is a simple struct, or
even just a byte array in the case of ChaCha20Poly1305.
There's no single right answer here. But we should consider the full
picture including the chacha20poly1305 functions. The (dst, src, len,
auxiliary stuff) order also helps for things like the AES_CRYPT_SG macro
in crypto/aes.c, as the "auxiliary stuff" is together and at the end.
Note that if we decide we like this order, we could reorder the
arguments of aes_encrypt() and aes_decrypt() to match.
Maybe let's see what other people prefer?
- Eric
^ permalink raw reply [flat|nested] 45+ messages in thread* Re: [PATCH 02/33] lib/crypto: aes: Add ECB support
2026-07-07 19:22 ` Eric Biggers
@ 2026-07-08 5:29 ` Thomas Huth
0 siblings, 0 replies; 45+ messages in thread
From: Thomas Huth @ 2026-07-08 5:29 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, linux-kernel, Jason A. Donenfeld, Ard Biesheuvel
On 07/07/2026 21.22, Eric Biggers wrote:
> On Tue, Jul 07, 2026 at 03:59:24PM +0200, Thomas Huth wrote:
>>> +Unauthenticated encryption
>>> +==========================
>>> +
>>> +Support for unauthenticated encryption and decryption, including bare stream
>>> +ciphers and other length-preserving algorithms such as block ciphers in XTS
>>> +mode.
>>
>> This sentence no verb?
>
> It's a noun phrase that introduces what the section contains, before
> transitioning into full sentences.
>
> I used this in all existing Documentation/crypto/libcrypto-*.rst. It's
> also fairly common in the help text for kconfig symbols (across the
> kernel, not just the kconfig help text I've written).
Fair point for the help texts (which should be concise), but (as a
non-native speaker) I did not expect it in doc file. Anyway, if it's already
like this in other files, then it's certainly fine here, too.
>>> +void aes_ecb_encrypt(u8 *dst, const u8 *src, size_t len, aes_encrypt_arg key);
>>
>> Other similar functions like aes_encrypt() use the key as first argument ...
>> so maybe do the same here, too, for consistency?
>
> For single-block AES, there's indeed already aes_encrypt(key, dst, src)
> and aes_decrypt(key, dst, src). But for actual AEAD encryption there's
> already:
>
> chacha20poly1305_encrypt(dst, src, src_len, ad, ad_len, nonce, key)
> chacha20poly1305_decrypt(dst, src, src_len, ad, ad_len, nonce, key)
> xchacha20poly1305_encrypt(dst, src, src_len, ad, ad_len, nonce, key)
> xchacha20poly1305_decrypt(dst, src, src_len, ad, ad_len, nonce, key)
> chacha20poly1305_encrypt_sg_inplace(src, src_len, ad, ad_len, nonce, key)
> chacha20poly1305_decrypt_sg_inplace(src, src_len, ad, ad_len, nonce, key)
>
> Those follow the convention described by Jason here:
> https://lore.kernel.org/linux-crypto/aPT3dImhaI6Dpqs7@zx2c4.com/
Ok, makes sense now, thanks for the explanation! Feel free to add my:
Reviewed-by: Thomas Huth <thuth@redhat.com>
to this patch if you like.
> Note that if we decide we like this order, we could reorder the
> arguments of aes_encrypt() and aes_decrypt() to match.
Yes, maybe a good idea for more consistency in the code...
Thomas
^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH 03/33] lib/crypto: aes: Add CBC and CBC-CTS support
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
2026-07-07 5:34 ` [PATCH 01/33] crypto: xts - Split out __xts_verify_key() helper Eric Biggers
2026-07-07 5:34 ` [PATCH 02/33] lib/crypto: aes: Add ECB support Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 04/33] lib/crypto: aes: Add CTR and XCTR support Eric Biggers
` (29 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Add support for AES-CBC and AES-CBC-CTS to the crypto library.
These will be used to provide streamlined implementations of the
"cbc(aes)" and "cts(cbc(aes))" crypto_skcipher algorithms. Most users
of these crypto_skcipher algorithms will also be able to switch to the
library, which as usual will be simpler and faster, e.g.:
- block/blk-crypto-fallback.c (for AES-128-CBC-ESSIV)
- fs/crypto/crypto.c (for AES-128-CBC-ESSIV)
- fs/crypto/fname.c (for AES-256-CTS and AES-128-CBC)
- kernel/bpf/crypto.c
- net/ceph/crypto.c
- security/keys/encrypted-keys/encrypted.c
As usual, the architecture-optimized AES-CBC and AES-CBC-CTS code will
be migrated into the library as well (using the hooks provided in this
commit), eliminating lots of repetitive boilerplate code.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
.../crypto/libcrypto-unauth-encryption.rst | 7 +
include/crypto/aes-cbc.h | 77 ++++++++
lib/crypto/Kconfig | 6 +
lib/crypto/aes.c | 174 ++++++++++++++++++
lib/crypto/tests/Kconfig | 1 +
5 files changed, 265 insertions(+)
create mode 100644 include/crypto/aes-cbc.h
diff --git a/Documentation/crypto/libcrypto-unauth-encryption.rst b/Documentation/crypto/libcrypto-unauth-encryption.rst
index 891c15279749..fb8106034089 100644
--- a/Documentation/crypto/libcrypto-unauth-encryption.rst
+++ b/Documentation/crypto/libcrypto-unauth-encryption.rst
@@ -20,6 +20,13 @@ mode. The legitimate use cases for these algorithms are:
Besides the above, these shouldn't be used.
+AES-CBC
+-------
+
+Support for AES in the CBC and CBC-CTS modes of operation.
+
+.. kernel-doc:: include/crypto/aes-cbc.h
+
AES-ECB
-------
diff --git a/include/crypto/aes-cbc.h b/include/crypto/aes-cbc.h
new file mode 100644
index 000000000000..3e8d1a9ce8a9
--- /dev/null
+++ b/include/crypto/aes-cbc.h
@@ -0,0 +1,77 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * AES-CBC and AES-CBC-CTS unauthenticated encryption and decryption
+ *
+ * Copyright 2026 Google LLC
+ */
+#ifndef _CRYPTO_AES_CBC_H
+#define _CRYPTO_AES_CBC_H
+
+#include <crypto/aes.h>
+
+/**
+ * aes_cbc_encrypt() - Encrypt data using AES-CBC
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source data
+ * @len: Number of bytes to encrypt. Must be a multiple of AES_BLOCK_SIZE.
+ * @iv: The initialization vector. It is updated with the next value, i.e. the
+ * last ciphertext block (or left unchanged if @len == 0).
+ * @key: The key
+ *
+ * This supports incremental encryption. The length of each chunk must be a
+ * multiple of AES_BLOCK_SIZE, and the updated @iv must be passed in each time.
+ *
+ * Context: Any context.
+ */
+void aes_cbc_encrypt(u8 *dst, const u8 *src, size_t len,
+ u8 iv[at_least AES_BLOCK_SIZE], aes_encrypt_arg key);
+
+/**
+ * aes_cbc_decrypt() - Decrypt data using AES-CBC
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source data
+ * @len: Number of bytes to decrypt. Must be a multiple of AES_BLOCK_SIZE.
+ * @iv: The initialization vector. It is updated with the next value, i.e. the
+ * last ciphertext block (or left unchanged if @len == 0).
+ * @key: The key
+ *
+ * This supports incremental decryption. The length of each chunk must be a
+ * multiple of AES_BLOCK_SIZE, and the updated @iv must be passed in each time.
+ *
+ * Context: Any context.
+ */
+void aes_cbc_decrypt(u8 *dst, const u8 *src, size_t len,
+ u8 iv[at_least AES_BLOCK_SIZE], const struct aes_key *key);
+
+/**
+ * aes_cbc_cts_encrypt() - Encrypt data using AES-CBC-CTS (CS3 variant)
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source data
+ * @len: Number of bytes to encrypt, at least AES_BLOCK_SIZE
+ * @iv: The initialization vector, clobbered by this function
+ * @key: The key
+ *
+ * Context: Any context.
+ */
+void aes_cbc_cts_encrypt(u8 *dst, const u8 *src, size_t len,
+ u8 iv[at_least AES_BLOCK_SIZE], aes_encrypt_arg key);
+
+/**
+ * aes_cbc_cts_decrypt() - Decrypt data using AES-CBC-CTS (CS3 variant)
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source data
+ * @len: Number of bytes to decrypt, at least AES_BLOCK_SIZE
+ * @iv: The initialization vector, clobbered by this function
+ * @key: The key
+ *
+ * Context: Any context.
+ */
+void aes_cbc_cts_decrypt(u8 *dst, const u8 *src, size_t len,
+ u8 iv[at_least AES_BLOCK_SIZE],
+ const struct aes_key *key);
+
+#endif /* _CRYPTO_AES_CBC_H */
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index 26514c181a7f..c64cc3e12b57 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -27,6 +27,12 @@ config CRYPTO_LIB_AESCFB
select CRYPTO_LIB_AES
select CRYPTO_LIB_UTILS
+config CRYPTO_LIB_AES_CBC
+ tristate
+ select CRYPTO_LIB_AES
+ help
+ The AES-CBC and AES-CBC-CTS library functions.
+
config CRYPTO_LIB_AES_CBC_MACS
tristate
select CRYPTO_LIB_AES
diff --git a/lib/crypto/aes.c b/lib/crypto/aes.c
index e2f1ebf81405..3635fbe946f3 100644
--- a/lib/crypto/aes.c
+++ b/lib/crypto/aes.c
@@ -5,6 +5,7 @@
*/
#include <crypto/aes-cbc-macs.h>
+#include <crypto/aes-cbc.h>
#include <crypto/aes-ecb.h>
#include <crypto/aes.h>
#include <crypto/utils.h>
@@ -793,6 +794,179 @@ void aes_ecb_decrypt(u8 *dst, const u8 *src, size_t len,
EXPORT_SYMBOL_GPL(aes_ecb_decrypt);
#endif /* CONFIG_CRYPTO_LIB_AES_ECB */
+#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_CBC)
+/*
+ * Hooks for optimized AES-CBC implementations, overridable by the architecture.
+ * They are called with len > 0 && len % AES_BLOCK_SIZE == 0. Returning false
+ * causes the fallback implementation to be used instead.
+ */
+#ifndef aes_cbc_encrypt_arch
+static bool aes_cbc_encrypt_arch(u8 *dst, const u8 *src, size_t len,
+ u8 iv[AES_BLOCK_SIZE],
+ const struct aes_enckey *key)
+{
+ return false;
+}
+#endif
+#ifndef aes_cbc_decrypt_arch
+static bool aes_cbc_decrypt_arch(u8 *dst, const u8 *src, size_t len,
+ u8 iv[AES_BLOCK_SIZE],
+ const struct aes_key *key)
+{
+ return false;
+}
+#endif
+
+void aes_cbc_encrypt(u8 *dst, const u8 *src, size_t len, u8 iv[AES_BLOCK_SIZE],
+ aes_encrypt_arg key)
+{
+ const u8 *prev = iv;
+
+ if (WARN_ON_ONCE(len % AES_BLOCK_SIZE))
+ len = round_down(len, AES_BLOCK_SIZE);
+
+ if (unlikely(len == 0))
+ return;
+
+ if (likely(aes_cbc_encrypt_arch(dst, src, len, iv, key.enc_key)))
+ return;
+
+ do {
+ crypto_xor_cpy(dst, src, prev, AES_BLOCK_SIZE);
+ aes_encrypt(key, dst, dst);
+ prev = dst;
+ dst += AES_BLOCK_SIZE;
+ src += AES_BLOCK_SIZE;
+ len -= AES_BLOCK_SIZE;
+ } while (len);
+ memcpy(iv, prev, AES_BLOCK_SIZE);
+}
+EXPORT_SYMBOL_GPL(aes_cbc_encrypt);
+
+void aes_cbc_decrypt(u8 *dst, const u8 *src, size_t len, u8 iv[AES_BLOCK_SIZE],
+ const struct aes_key *key)
+{
+ u8 next_iv[AES_BLOCK_SIZE];
+
+ if (WARN_ON_ONCE(len % AES_BLOCK_SIZE))
+ len = round_down(len, AES_BLOCK_SIZE);
+
+ if (unlikely(len == 0))
+ return;
+
+ if (likely(aes_cbc_decrypt_arch(dst, src, len, iv, key)))
+ return;
+
+ len -= AES_BLOCK_SIZE;
+ dst += len;
+ src += len;
+ memcpy(next_iv, src, AES_BLOCK_SIZE);
+ for (;;) {
+ aes_decrypt(key, dst, src);
+ if (len == 0)
+ break;
+ src -= AES_BLOCK_SIZE;
+ crypto_xor(dst, src, AES_BLOCK_SIZE);
+ dst -= AES_BLOCK_SIZE;
+ len -= AES_BLOCK_SIZE;
+ }
+ crypto_xor(dst, iv, AES_BLOCK_SIZE);
+ memcpy(iv, next_iv, AES_BLOCK_SIZE);
+}
+EXPORT_SYMBOL_GPL(aes_cbc_decrypt);
+
+/*
+ * Hooks for optimized AES-CBC-CTS implementations, overridable by the
+ * architecture. They are called with len > AES_BLOCK_SIZE. Returning false
+ * causes the fallback implementation to be used instead. The fallback
+ * implementation still uses the arch-optimized AES-CBC code if available, but
+ * direct implementation of AES-CBC-CTS is helpful on short messages.
+ */
+#ifndef aes_cbc_cts_encrypt_arch
+static bool aes_cbc_cts_encrypt_arch(u8 *dst, const u8 *src, size_t len,
+ u8 iv[AES_BLOCK_SIZE],
+ const struct aes_enckey *key)
+{
+ return false;
+}
+#endif
+#ifndef aes_cbc_cts_decrypt_arch
+static bool aes_cbc_cts_decrypt_arch(u8 *dst, const u8 *src, size_t len,
+ u8 iv[AES_BLOCK_SIZE],
+ const struct aes_key *key)
+{
+ return false;
+}
+#endif
+
+void aes_cbc_cts_encrypt(u8 *dst, const u8 *src, size_t len,
+ u8 iv[AES_BLOCK_SIZE], aes_encrypt_arg key)
+{
+ /* Offset to P[n] and C[n] (last plaintext and ciphertext block) */
+ size_t pn_offset = round_down(len - 1, AES_BLOCK_SIZE);
+ /* Length of P[n] and C[n], 1 <= pn_len <= AES_BLOCK_SIZE */
+ size_t pn_len = len - pn_offset;
+ u8 tmp[AES_BLOCK_SIZE] __aligned(__alignof__(long));
+ u8 *pad;
+
+ if (WARN_ON_ONCE(len < AES_BLOCK_SIZE))
+ return;
+
+ if (len == AES_BLOCK_SIZE) {
+ aes_cbc_encrypt(dst, src, len, iv, key);
+ return;
+ }
+ if (likely(aes_cbc_cts_encrypt_arch(dst, src, len, iv, key.enc_key)))
+ return;
+
+ /* CBC-encrypt all blocks except the last. */
+ aes_cbc_encrypt(dst, src, pn_offset, iv, key);
+
+ /* Compute C[n] and C[n - 1], considering that src may equal dst. */
+ pad = &dst[pn_offset - AES_BLOCK_SIZE];
+ memcpy(tmp, pad, AES_BLOCK_SIZE);
+ crypto_xor(tmp, &src[pn_offset], pn_len);
+ memcpy(&dst[pn_offset], pad, pn_len); /* C[n] */
+ aes_encrypt(key, pad, tmp); /* C[n - 1] */
+
+ memzero_explicit(tmp, sizeof(tmp));
+}
+EXPORT_SYMBOL_GPL(aes_cbc_cts_encrypt);
+
+void aes_cbc_cts_decrypt(u8 *dst, const u8 *src, size_t len,
+ u8 iv[AES_BLOCK_SIZE], const struct aes_key *key)
+{
+ /* Offset to P[n] and C[n] (last plaintext and ciphertext block) */
+ size_t pn_offset = round_down(len - 1, AES_BLOCK_SIZE);
+ /* Length of P[n] and C[n], 1 <= pn_len <= AES_BLOCK_SIZE */
+ size_t pn_len = len - pn_offset;
+ u8 *pad;
+
+ if (WARN_ON_ONCE(len < AES_BLOCK_SIZE))
+ return;
+
+ if (len == AES_BLOCK_SIZE) {
+ aes_cbc_decrypt(dst, src, len, iv, key);
+ return;
+ }
+ if (likely(aes_cbc_cts_decrypt_arch(dst, src, len, iv, key)))
+ return;
+
+ /* Compute P[0]..P[n - 2]. */
+ aes_cbc_decrypt(dst, src, pn_offset - AES_BLOCK_SIZE, iv, key);
+
+ /* Compute P[n] and P[n - 1], considering that src may equal dst. */
+ pad = &dst[pn_offset - AES_BLOCK_SIZE];
+ aes_decrypt(key, pad, &src[pn_offset - AES_BLOCK_SIZE]);
+ crypto_xor_cpy(&dst[pn_offset], &src[pn_offset], pad,
+ pn_len); /* P[n] */
+ crypto_xor(pad, &dst[pn_offset], pn_len);
+ aes_decrypt(key, pad, pad);
+ crypto_xor(pad, iv, AES_BLOCK_SIZE); /* P[n - 1] */
+}
+EXPORT_SYMBOL_GPL(aes_cbc_cts_decrypt);
+#endif /* CONFIG_CRYPTO_LIB_AES_CBC */
+
static int __init aes_mod_init(void)
{
#ifdef aes_mod_init_arch
diff --git a/lib/crypto/tests/Kconfig b/lib/crypto/tests/Kconfig
index a57e87dbb1b1..e78086f3c954 100644
--- a/lib/crypto/tests/Kconfig
+++ b/lib/crypto/tests/Kconfig
@@ -144,6 +144,7 @@ config CRYPTO_LIB_SM3_KUNIT_TEST
config CRYPTO_LIB_ENABLE_ALL_FOR_KUNIT
tristate "Enable all crypto library code for KUnit tests"
depends on KUNIT
+ select CRYPTO_LIB_AES_CBC
select CRYPTO_LIB_AES_CBC_MACS
select CRYPTO_LIB_AES_ECB
select CRYPTO_LIB_BLAKE2B
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 04/33] lib/crypto: aes: Add CTR and XCTR support
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (2 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 03/33] lib/crypto: aes: Add CBC and CBC-CTS support Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 05/33] lib/crypto: aes: Add XTS support Eric Biggers
` (28 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Add support for AES-CTR and AES-XCTR to the crypto library.
These will be used to provide streamlined implementations of the
"ctr(aes)" and "xctr(aes)" crypto_skcipher algorithms. Most users of
"ctr(aes)" will also be able to switch to the library, which as usual
will be simpler and faster, e.g.:
- net/mac80211/fils_aead.c
- net/mac802154/llsec.c
As usual, the architecture-optimized AES-CTR and AES-XCTR code will be
migrated into the library as well (using the hooks provided in this
commit), eliminating lots of repetitive boilerplate code.
This is also a prerequisite for supporting AES-GCM, AES-CCM, and
AES-HCTR2 in the crypto library.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
.../crypto/libcrypto-unauth-encryption.rst | 7 ++
include/crypto/aes-ctr.h | 56 +++++++++++
lib/crypto/Kconfig | 6 ++
lib/crypto/aes.c | 93 +++++++++++++++++++
lib/crypto/tests/Kconfig | 1 +
5 files changed, 163 insertions(+)
create mode 100644 include/crypto/aes-ctr.h
diff --git a/Documentation/crypto/libcrypto-unauth-encryption.rst b/Documentation/crypto/libcrypto-unauth-encryption.rst
index fb8106034089..6aca01d715da 100644
--- a/Documentation/crypto/libcrypto-unauth-encryption.rst
+++ b/Documentation/crypto/libcrypto-unauth-encryption.rst
@@ -27,6 +27,13 @@ Support for AES in the CBC and CBC-CTS modes of operation.
.. kernel-doc:: include/crypto/aes-cbc.h
+AES-CTR and AES-XCTR
+--------------------
+
+Support for AES in the CTR and XCTR modes of operation.
+
+.. kernel-doc:: include/crypto/aes-ctr.h
+
AES-ECB
-------
diff --git a/include/crypto/aes-ctr.h b/include/crypto/aes-ctr.h
new file mode 100644
index 000000000000..fa2b7d303e55
--- /dev/null
+++ b/include/crypto/aes-ctr.h
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * AES-CTR and AES-XCTR stream ciphers
+ *
+ * Copyright 2026 Google LLC
+ */
+#ifndef _CRYPTO_AES_CTR_H
+#define _CRYPTO_AES_CTR_H
+
+#include <crypto/aes.h>
+
+/**
+ * aes_ctr() - AES-CTR en/decryption
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source data
+ * @len: Number of bytes to en/decrypt
+ * @ctr: The counter. It will be incremented by ceil(@len / AES_BLOCK_SIZE).
+ * @key: The key
+ *
+ * This implements AES in counter mode with a 128-bit big endian counter.
+ *
+ * This supports incremental en/decryption. The length of each non-final chunk
+ * must be a multiple of AES_BLOCK_SIZE, and the updated @ctr must be passed in
+ * each time.
+ *
+ * Context: Any context.
+ */
+void aes_ctr(u8 *dst, const u8 *src, size_t len,
+ u8 ctr[at_least AES_BLOCK_SIZE], aes_encrypt_arg key);
+
+/**
+ * aes_xctr() - AES-XCTR en/decryption
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source data
+ * @len: Number of bytes to en/decrypt
+ * @ctr: The block counter. For the first call, set it to 1. It will be
+ * incremented by ceil(@len / AES_BLOCK_SIZE).
+ * @iv: The initialization vector
+ * @key: The key
+ *
+ * This implements AES in XOR Counter mode, as specified in the paper
+ * "Length-preserving encryption with HCTR2"
+ * (https://eprint.iacr.org/2021/1441.pdf).
+ *
+ * This supports incremental en/decryption. The length of each non-final chunk
+ * must be a multiple of AES_BLOCK_SIZE, and the updated @ctr must be passed in
+ * each time.
+ *
+ * Context: Any context.
+ */
+void aes_xctr(u8 *dst, const u8 *src, size_t len, u64 *ctr,
+ const u8 iv[at_least AES_BLOCK_SIZE], aes_encrypt_arg key);
+
+#endif /* _CRYPTO_AES_CTR_H */
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index c64cc3e12b57..96febc3df6d6 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -47,6 +47,12 @@ config CRYPTO_LIB_AES_ECB
help
The AES-ECB library functions.
+config CRYPTO_LIB_AES_CTR
+ tristate
+ select CRYPTO_LIB_AES
+ help
+ The AES-CTR and AES-XCTR library functions.
+
config CRYPTO_LIB_AESGCM
tristate
select CRYPTO_LIB_AES
diff --git a/lib/crypto/aes.c b/lib/crypto/aes.c
index 3635fbe946f3..9da274a72221 100644
--- a/lib/crypto/aes.c
+++ b/lib/crypto/aes.c
@@ -6,6 +6,7 @@
#include <crypto/aes-cbc-macs.h>
#include <crypto/aes-cbc.h>
+#include <crypto/aes-ctr.h>
#include <crypto/aes-ecb.h>
#include <crypto/aes.h>
#include <crypto/utils.h>
@@ -967,6 +968,98 @@ void aes_cbc_cts_decrypt(u8 *dst, const u8 *src, size_t len,
EXPORT_SYMBOL_GPL(aes_cbc_cts_decrypt);
#endif /* CONFIG_CRYPTO_LIB_AES_CBC */
+#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_CTR)
+/*
+ * Hooks for optimized AES-CTR and AES-XCTR implementations, overridable by the
+ * architecture. They are called with any len >= 0. Returning false causes the
+ * fallback implementation to be used instead.
+ */
+#ifndef aes_ctr_arch
+static bool aes_ctr_arch(u8 *dst, const u8 *src, size_t len,
+ u8 ctr[AES_BLOCK_SIZE], const struct aes_enckey *key)
+{
+ return false;
+}
+#endif
+#ifndef aes_xctr_arch
+static bool aes_xctr_arch(u8 *dst, const u8 *src, size_t len, u64 *ctr,
+ const u8 iv[AES_BLOCK_SIZE],
+ const struct aes_enckey *key)
+{
+ return false;
+}
+#endif
+
+static __always_inline void inc_be128_ctr(u8 ctr[AES_BLOCK_SIZE])
+{
+ /* Casts to u8 are needed because of the implicit integer promotion. */
+ if (((u8)++ctr[AES_BLOCK_SIZE - 1]) != 0)
+ return;
+ for (int i = AES_BLOCK_SIZE - 2; i >= 0; i--) {
+ if ((u8)++ctr[i] != 0)
+ break;
+ }
+}
+
+void aes_ctr(u8 *dst, const u8 *src, size_t len, u8 ctr[AES_BLOCK_SIZE],
+ aes_encrypt_arg key)
+{
+ u8 keystream[AES_BLOCK_SIZE] __aligned(__alignof__(long));
+
+ if (likely(aes_ctr_arch(dst, src, len, ctr, key.enc_key)))
+ return;
+
+ /* Handle the full blocks. */
+ for (; len >= AES_BLOCK_SIZE; len -= AES_BLOCK_SIZE) {
+ aes_encrypt(key, keystream, ctr);
+ crypto_xor_cpy(dst, src, keystream, AES_BLOCK_SIZE);
+ inc_be128_ctr(ctr);
+ dst += AES_BLOCK_SIZE;
+ src += AES_BLOCK_SIZE;
+ }
+ /* Handle any partial block at the end. */
+ if (len) {
+ aes_encrypt(key, keystream, ctr);
+ crypto_xor_cpy(dst, src, keystream, len);
+ /* Counter is incremented even with just a partial block. */
+ inc_be128_ctr(ctr);
+ }
+ memzero_explicit(keystream, sizeof(keystream));
+}
+EXPORT_SYMBOL_GPL(aes_ctr);
+
+void aes_xctr(u8 *dst, const u8 *src, size_t len, u64 *ctr,
+ const u8 iv[AES_BLOCK_SIZE], aes_encrypt_arg key)
+{
+ const __le64 iv0 = get_unaligned((const __le64 *)&iv[0]);
+ __le64 aes_input[2];
+ u8 keystream[AES_BLOCK_SIZE] __aligned(__alignof__(long));
+
+ if (likely(aes_xctr_arch(dst, src, len, ctr, iv, key.enc_key)))
+ return;
+
+ aes_input[1] = get_unaligned((const __le64 *)&iv[8]);
+ /* Handle the full blocks. */
+ for (; len >= AES_BLOCK_SIZE; len -= AES_BLOCK_SIZE) {
+ aes_input[0] = iv0 ^ cpu_to_le64((*ctr)++);
+ aes_encrypt(key, keystream, (const u8 *)aes_input);
+ crypto_xor_cpy(dst, src, keystream, AES_BLOCK_SIZE);
+ dst += AES_BLOCK_SIZE;
+ src += AES_BLOCK_SIZE;
+ }
+ /* Handle any partial block at the end. */
+ if (len) {
+ /* Counter is incremented even with just a partial block. */
+ aes_input[0] = iv0 ^ cpu_to_le64((*ctr)++);
+ aes_encrypt(key, keystream, (const u8 *)aes_input);
+ crypto_xor_cpy(dst, src, keystream, len);
+ }
+ memzero_explicit(keystream, sizeof(keystream));
+ memzero_explicit(aes_input, sizeof(aes_input));
+}
+EXPORT_SYMBOL_GPL(aes_xctr);
+#endif /* CONFIG_CRYPTO_LIB_AES_CTR */
+
static int __init aes_mod_init(void)
{
#ifdef aes_mod_init_arch
diff --git a/lib/crypto/tests/Kconfig b/lib/crypto/tests/Kconfig
index e78086f3c954..9284d0134d77 100644
--- a/lib/crypto/tests/Kconfig
+++ b/lib/crypto/tests/Kconfig
@@ -146,6 +146,7 @@ config CRYPTO_LIB_ENABLE_ALL_FOR_KUNIT
depends on KUNIT
select CRYPTO_LIB_AES_CBC
select CRYPTO_LIB_AES_CBC_MACS
+ select CRYPTO_LIB_AES_CTR
select CRYPTO_LIB_AES_ECB
select CRYPTO_LIB_BLAKE2B
select CRYPTO_LIB_CHACHA20POLY1305
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 05/33] lib/crypto: aes: Add XTS support
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (3 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 04/33] lib/crypto: aes: Add CTR and XCTR support Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 06/33] lib/crypto: aes: Add GCM support Eric Biggers
` (27 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Add support for AES-XTS to the crypto library.
This will be used to provide a streamlined implementation of the
"xts(aes)" crypto_skcipher algorithm. I'm also planning to use this
directly in fscrypt and blk-crypto-fallback.
As usual, the architecture-optimized AES-XTS code will be migrated into
the library as well (using the hooks provided in this commit),
eliminating lots of repetitive boilerplate code. Compared to direct
implementation of "xts(aes)", I've also eliminated the requirement for
architectures to implement ciphertext stealing, as the library just
handles it portably instead. That will simplify things considerably.
Initial test coverage is provided by the crypto_skcipher support added
in a later commit. I'm planning a KUnit test suite as well.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
.../crypto/libcrypto-unauth-encryption.rst | 7 +
include/crypto/aes-xts.h | 87 +++++++
lib/crypto/Kconfig | 6 +
lib/crypto/aes.c | 224 ++++++++++++++++++
lib/crypto/tests/Kconfig | 1 +
5 files changed, 325 insertions(+)
create mode 100644 include/crypto/aes-xts.h
diff --git a/Documentation/crypto/libcrypto-unauth-encryption.rst b/Documentation/crypto/libcrypto-unauth-encryption.rst
index 6aca01d715da..15deba7e53e8 100644
--- a/Documentation/crypto/libcrypto-unauth-encryption.rst
+++ b/Documentation/crypto/libcrypto-unauth-encryption.rst
@@ -40,3 +40,10 @@ AES-ECB
Support for AES in the ECB mode of operation.
.. kernel-doc:: include/crypto/aes-ecb.h
+
+AES-XTS
+-------
+
+Support for AES in the XTS mode of operation.
+
+.. kernel-doc:: include/crypto/aes-xts.h
diff --git a/include/crypto/aes-xts.h b/include/crypto/aes-xts.h
new file mode 100644
index 000000000000..226ac38d54ac
--- /dev/null
+++ b/include/crypto/aes-xts.h
@@ -0,0 +1,87 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * AES-XTS unauthenticated encryption and decryption
+ *
+ * Copyright 2026 Google LLC
+ */
+#ifndef _CRYPTO_AES_XTS_H
+#define _CRYPTO_AES_XTS_H
+
+#include <crypto/aes.h>
+#include <crypto/xts.h>
+
+/**
+ * struct aes_xts_key - A key prepared for AES-XTS encryption and decryption
+ *
+ * Note that (depending on the architecture) this typically is around 768 bytes,
+ * which makes it a bit too large to allocate on the stack in most cases.
+ */
+struct aes_xts_key {
+ /* private: */
+ struct aes_key main_key;
+ struct aes_enckey tweak_key;
+};
+
+/**
+ * aes_xts_preparekey() - Prepare an AES-XTS key
+ * @key: (output) The key structure to initialize
+ * @in_key: The raw AES-XTS key
+ * @key_len: Length of the raw key in bytes
+ * @flags: Optional flag XTS_FORBID_WEAK_KEYS to forbid keys whose two halves
+ * are the same.
+ *
+ * Context: Any context.
+ * Return:
+ * * 0 on success
+ * * -EINVAL if the key is rejected because its length isn't 32, 64, or (when
+ * FIPS mode isn't enabled) 48; or because its two halves are the same and
+ * either XTS_FORBID_WEAK_KEYS is given or FIPS mode is enabled.
+ */
+int __must_check aes_xts_preparekey(struct aes_xts_key *key, const u8 *in_key,
+ size_t key_len, int flags);
+
+/**
+ * aes_xts_encrypt() - Encrypt data using AES-XTS
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source data
+ * @len: Number of bytes to encrypt. Must be >= AES_BLOCK_SIZE. On non-last
+ * calls it also must be a multiple of AES_BLOCK_SIZE.
+ * @tweak: The tweak. It is updated with the next value, unless @len isn't a
+ * multiple of AES_BLOCK_SIZE in which case the value is unspecified.
+ * @key: The key
+ * @cont: %true to continue same message (skip tweak encryption)
+ *
+ * This supports both one-shot and incremental encryption. For incremental
+ * encryption, all non-last calls require @len aligned to AES_BLOCK_SIZE, and
+ * all non-first calls require @cont = %true.
+ *
+ * Context: Any context.
+ */
+void aes_xts_encrypt(u8 *dst, const u8 *src, size_t len,
+ u8 tweak[at_least AES_BLOCK_SIZE],
+ const struct aes_xts_key *key, bool cont);
+
+/**
+ * aes_xts_decrypt() - Decrypt data using AES-XTS
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source data
+ * @len: Number of bytes to decrypt. Must be >= AES_BLOCK_SIZE. On non-last
+ * calls it also must be a multiple of AES_BLOCK_SIZE.
+ * @tweak: The tweak. It is updated with the next value, unless @len isn't a
+ * multiple of AES_BLOCK_SIZE in which case the value is unspecified.
+ * @key: The key
+ * @cont: %true to continue same message (skip tweak encryption)
+ *
+ * This supports both one-shot and incremental decryption. For incremental
+ * decryption, all non-last calls require @len aligned to AES_BLOCK_SIZE, and
+ * all non-first calls require @cont = %true.
+ *
+ * Context: Any context.
+ */
+void aes_xts_decrypt(u8 *dst, const u8 *src, size_t len,
+ u8 tweak[at_least AES_BLOCK_SIZE],
+ const struct aes_xts_key *key, bool cont);
+
+#endif /* _CRYPTO_AES_XTS_H */
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index 96febc3df6d6..9af44cf743a7 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -53,6 +53,12 @@ config CRYPTO_LIB_AES_CTR
help
The AES-CTR and AES-XCTR library functions.
+config CRYPTO_LIB_AES_XTS
+ tristate
+ select CRYPTO_LIB_AES
+ help
+ The AES-XTS library functions.
+
config CRYPTO_LIB_AESGCM
tristate
select CRYPTO_LIB_AES
diff --git a/lib/crypto/aes.c b/lib/crypto/aes.c
index 9da274a72221..630702a4228c 100644
--- a/lib/crypto/aes.c
+++ b/lib/crypto/aes.c
@@ -8,7 +8,9 @@
#include <crypto/aes-cbc.h>
#include <crypto/aes-ctr.h>
#include <crypto/aes-ecb.h>
+#include <crypto/aes-xts.h>
#include <crypto/aes.h>
+#include <crypto/gf128mul.h>
#include <crypto/utils.h>
#include <linux/cache.h>
#include <linux/crypto.h>
@@ -1060,6 +1062,228 @@ void aes_xctr(u8 *dst, const u8 *src, size_t len, u64 *ctr,
EXPORT_SYMBOL_GPL(aes_xctr);
#endif /* CONFIG_CRYPTO_LIB_AES_CTR */
+#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_XTS)
+int aes_xts_preparekey(struct aes_xts_key *key, const u8 *in_key,
+ size_t key_len, int flags)
+{
+ int err;
+
+ err = __xts_verify_key(in_key, key_len, flags);
+ if (err)
+ return err;
+ /* First half of XTS key is the main key */
+ err = aes_preparekey(&key->main_key, in_key, key_len / 2);
+ if (err)
+ return err;
+ /* Second half of XTS key is the tweak key */
+ return aes_prepareenckey(&key->tweak_key, &in_key[key_len / 2],
+ key_len / 2);
+}
+EXPORT_SYMBOL_GPL(aes_xts_preparekey);
+
+/*
+ * Hooks for optimized AES-XTS implementations, overridable by the architecture.
+ * They are called with len > 0 && len % AES_BLOCK_SIZE == 0. In other words,
+ * they aren't expected to handle ciphertext stealing or empty inputs.
+ * Returning false causes the fallback implementation to be used instead.
+ *
+ * (Currently, all users of AES-XTS in the kernel seem to en/decrypt whole
+ * numbers of blocks anyway, with len >= 512. So there's no need to heavily
+ * optimize ciphertext stealing for short messages.)
+ */
+#ifndef aes_xts_encrypt_arch
+static bool aes_xts_encrypt_arch(u8 *dst, const u8 *src, size_t len,
+ u8 tweak[AES_BLOCK_SIZE],
+ const struct aes_xts_key *key, bool cont)
+{
+ return false;
+}
+#endif
+#ifndef aes_xts_decrypt_arch
+static bool aes_xts_decrypt_arch(u8 *dst, const u8 *src, size_t len,
+ u8 tweak[AES_BLOCK_SIZE],
+ const struct aes_xts_key *key, bool cont)
+{
+ return false;
+}
+#endif
+
+static noinline void aes_xts_crypt_nocts_blockbyblock(
+ u8 *dst, const u8 *src, size_t len, u8 tweak[AES_BLOCK_SIZE],
+ const struct aes_xts_key *key, bool cont, bool enc)
+{
+ le128 t;
+
+ if (cont)
+ memcpy(&t, tweak, sizeof(t));
+ else
+ aes_encrypt(&key->tweak_key, (u8 *)&t, tweak);
+ do {
+ crypto_xor_cpy(dst, src, (const u8 *)&t, AES_BLOCK_SIZE);
+ if (enc)
+ aes_encrypt(&key->main_key, dst, dst);
+ else
+ aes_decrypt(&key->main_key, dst, dst);
+ crypto_xor(dst, (const u8 *)&t, AES_BLOCK_SIZE);
+ gf128mul_x_ble(&t, &t);
+ dst += AES_BLOCK_SIZE;
+ src += AES_BLOCK_SIZE;
+ len -= AES_BLOCK_SIZE;
+ } while (len);
+ memcpy(tweak, &t, sizeof(t));
+ memzero_explicit(&t, sizeof(t));
+}
+
+/* Requires len > 0 && len % AES_BLOCK_SIZE == 0 */
+static __always_inline void aes_xts_encrypt_nocts(u8 *dst, const u8 *src,
+ size_t len,
+ u8 tweak[AES_BLOCK_SIZE],
+ const struct aes_xts_key *key,
+ bool cont)
+{
+ if (likely(aes_xts_encrypt_arch(dst, src, len, tweak, key, cont)))
+ return;
+
+ /*
+ * For the fallback, just go block-by-block. It could be implemented on
+ * top of AES-ECB, which could be significantly faster than this if the
+ * arch has optimized AES-ECB code but not AES-XTS. However, AES-XTS
+ * performance is important enough that it needs to be (and has been)
+ * implemented directly by every non-obsolete arch anyway.
+ */
+ aes_xts_crypt_nocts_blockbyblock(dst, src, len, tweak, key, cont,
+ /* enc= */ true);
+}
+
+/* Requires len > 0 && len % AES_BLOCK_SIZE == 0 */
+static __always_inline void aes_xts_decrypt_nocts(u8 *dst, const u8 *src,
+ size_t len,
+ u8 tweak[AES_BLOCK_SIZE],
+ const struct aes_xts_key *key,
+ bool cont)
+{
+ if (likely(aes_xts_decrypt_arch(dst, src, len, tweak, key, cont)))
+ return;
+
+ /* Just go block-by-block. See comment in aes_xts_encrypt_nocts(). */
+ aes_xts_crypt_nocts_blockbyblock(dst, src, len, tweak, key, cont,
+ /* enc= */ false);
+}
+
+static noinline void aes_xts_encrypt_cts(u8 *dst, const u8 *src, size_t len,
+ u8 tweak[AES_BLOCK_SIZE],
+ const struct aes_xts_key *key,
+ bool cont)
+{
+ size_t partial_len = len % AES_BLOCK_SIZE; /* Length of partial block */
+ size_t nocts_len = round_down(len, AES_BLOCK_SIZE);
+ u8 tmp_block[AES_BLOCK_SIZE] __aligned(__alignof__(long));
+
+ /* Encrypt all full blocks. */
+ aes_xts_encrypt_nocts(dst, src, nocts_len, tweak, key, cont);
+ dst += nocts_len - AES_BLOCK_SIZE;
+ src += nocts_len - AES_BLOCK_SIZE;
+
+ /*
+ * Swap the partial block with the first 'partial_len' bytes of the
+ * encrypted last full block. Note that a temporary buffer is needed to
+ * support in-place encryption.
+ */
+ memcpy(tmp_block, src + AES_BLOCK_SIZE, partial_len);
+ memcpy(dst + AES_BLOCK_SIZE, dst, partial_len);
+ memcpy(dst, tmp_block, partial_len);
+
+ /* Encrypt the last full block again. */
+ crypto_xor(dst, tweak, AES_BLOCK_SIZE);
+ aes_encrypt(&key->main_key, dst, dst);
+ crypto_xor(dst, tweak, AES_BLOCK_SIZE);
+ memzero_explicit(tmp_block, sizeof(tmp_block));
+}
+
+static noinline void aes_xts_decrypt_cts(u8 *dst, const u8 *src, size_t len,
+ u8 tweak[AES_BLOCK_SIZE],
+ const struct aes_xts_key *key,
+ bool cont)
+{
+ size_t partial_len = len % AES_BLOCK_SIZE; /* Length of partial block */
+ size_t nocts_len = round_down(len, AES_BLOCK_SIZE) - AES_BLOCK_SIZE;
+ union {
+ u8 block[AES_BLOCK_SIZE];
+ le128 tweak;
+ } tmp __aligned(__alignof__(long));
+
+ /*
+ * Decrypt all blocks except the last full block and the partial block.
+ * The last full block has to be handled specially because decryption
+ * ciphertext stealing uses the last two tweaks in reverse order.
+ *
+ * nocts_len == 0 is possible here, which aes_xts_decrypt_nocts()
+ * doesn't handle (so that the length doesn't get checked redundantly in
+ * the fast path). So handle that case specially as well.
+ */
+ if (nocts_len)
+ aes_xts_decrypt_nocts(dst, src, nocts_len, tweak, key, cont);
+ else if (!cont)
+ aes_encrypt(&key->tweak_key, tweak, tweak);
+ dst += nocts_len;
+ src += nocts_len;
+
+ /* Copy the tweak, advance it again, then decrypt last full block. */
+ memcpy(&tmp.tweak, tweak, AES_BLOCK_SIZE);
+ gf128mul_x_ble(&tmp.tweak, &tmp.tweak);
+ crypto_xor_cpy(dst, src, tmp.block, AES_BLOCK_SIZE);
+ aes_decrypt(&key->main_key, dst, dst);
+ crypto_xor(dst, tmp.block, AES_BLOCK_SIZE);
+
+ /*
+ * Swap the partial block with the first 'partial_len' bytes of the
+ * decrypted last full block. Note that a temporary buffer is needed to
+ * support in-place decryption.
+ */
+ memcpy(tmp.block, src + AES_BLOCK_SIZE, partial_len);
+ memcpy(dst + AES_BLOCK_SIZE, dst, partial_len);
+ memcpy(dst, tmp.block, partial_len);
+
+ /* Decrypt the last full block again. */
+ crypto_xor(dst, tweak, AES_BLOCK_SIZE);
+ aes_decrypt(&key->main_key, dst, dst);
+ crypto_xor(dst, tweak, AES_BLOCK_SIZE);
+ memzero_explicit(&tmp, sizeof(tmp));
+}
+
+void aes_xts_encrypt(u8 *dst, const u8 *src, size_t len,
+ u8 tweak[AES_BLOCK_SIZE], const struct aes_xts_key *key,
+ bool cont)
+{
+ if (WARN_ON_ONCE(len < AES_BLOCK_SIZE))
+ return;
+
+ if (unlikely(len % AES_BLOCK_SIZE)) {
+ aes_xts_encrypt_cts(dst, src, len, tweak, key, cont);
+ return;
+ }
+
+ aes_xts_encrypt_nocts(dst, src, len, tweak, key, cont);
+}
+EXPORT_SYMBOL_GPL(aes_xts_encrypt);
+
+void aes_xts_decrypt(u8 *dst, const u8 *src, size_t len,
+ u8 tweak[AES_BLOCK_SIZE], const struct aes_xts_key *key,
+ bool cont)
+{
+ if (WARN_ON_ONCE(len < AES_BLOCK_SIZE))
+ return;
+
+ if (unlikely(len % AES_BLOCK_SIZE)) {
+ aes_xts_decrypt_cts(dst, src, len, tweak, key, cont);
+ return;
+ }
+
+ aes_xts_decrypt_nocts(dst, src, len, tweak, key, cont);
+}
+EXPORT_SYMBOL_GPL(aes_xts_decrypt);
+#endif /* CONFIG_CRYPTO_LIB_AES_XTS */
+
static int __init aes_mod_init(void)
{
#ifdef aes_mod_init_arch
diff --git a/lib/crypto/tests/Kconfig b/lib/crypto/tests/Kconfig
index 9284d0134d77..b559e7c79e76 100644
--- a/lib/crypto/tests/Kconfig
+++ b/lib/crypto/tests/Kconfig
@@ -148,6 +148,7 @@ config CRYPTO_LIB_ENABLE_ALL_FOR_KUNIT
select CRYPTO_LIB_AES_CBC_MACS
select CRYPTO_LIB_AES_CTR
select CRYPTO_LIB_AES_ECB
+ select CRYPTO_LIB_AES_XTS
select CRYPTO_LIB_BLAKE2B
select CRYPTO_LIB_CHACHA20POLY1305
select CRYPTO_LIB_CURVE25519
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 06/33] lib/crypto: aes: Add GCM support
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (4 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 05/33] lib/crypto: aes: Add XTS support Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 07/33] lib/crypto: aes: Add CCM support Eric Biggers
` (26 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Add support for AES-GCM to the crypto library.
This will be used to provide streamlined implementations of the
"gcm(aes)" and "rfc4106(gcm(aes))" crypto_aead algorithms. Most users
of these will also be able to switch to the library, which as usual will
be faster and simpler, e.g.:
- drivers/net/macsec.c
- fs/smb/client/
- fs/smb/server/
- net/ceph/messenger_v2.c
- net/mac80211/ (for both GMAC and GCMP)
- net/tipc/crypto.c
- security/keys/trusted-keys/trusted_dcp.c
(I've already written proof-of-concept patches for all the above, and
they helped inform the API design.)
As usual, the architecture-optimized AES-GCM code will be migrated into
the library as well (using the hooks provided in this commit as well as
the GHASH ones), eliminating lots of repetitive boilerplate code.
Incremental en/decryption is supported. Incremental operation is a bit
controversial in AEAD APIs because users have to be careful not to
consume any decrypted data that hasn't been authenticated yet. But I do
think it's the right choice here. It's not fundamentally different from
the existing incremental MAC APIs, and it's the only approach that's
general enough to work well for all users in the kernel:
- An array of virtually-addressed buffers (like that used by
BoringSSL's EVP_AEAD_CTX_sealv() and EVP_AEAD_CTX_openv()) doesn't
work in the kernel in general, since in some cases the data for a
single AES-GCM message is contained in a large number of highmem
pages that each need to be mapped into memory individually. That
can be done efficiently only by using CPU-local mappings, but there
is a limited number of those.
Ceph messenger v2 is a great example, as it can send or receive up
to 32 MiB in a single AES-GCM message. And it needs the
en/decrypted data to go into a (potentially large) number of bvecs
provided by a custom iterator, as well as into four
virtually-addressed buffers, two of which can be large buffers in
the vmalloc region.
Even just allocating an array big enough to store all the pointers
can be problematic in the kernel. There are cases in which
decryption runs in GFP_NOIO context or even in softirq context,
where memory allocations are not as reliable as they normally are.
- Meanwhile, 'struct scatterlist' (the choice of crypto_aead) has
turned out to be really inconvenient for anyone who *does* just have
virtually-addressed buffers. This is especially true if they can be
in the vmalloc region, including the stack, as in that case the
conversion to a scatterlist has to be done page-by-page.
And even for users who have all of their data in bare 'struct page',
none of them actually use 'struct scatterlist' as their native data
structure anyway. They actually use skbs, bvecs, or other formats.
- iov_iter is attractive, but ultimately not general enough either
(considering the Ceph case for example), but also too general in
some ways (like having support for userspace addresses). Additional
iter types like ITER_SKB would help a bit, but bloating iov_iter
with more types would reduce performance elsewhere in the kernel.
Initial test coverage is provided by the crypto_aead support added in a
later commit. I'm planning a KUnit test suite as well.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
.../crypto/libcrypto-auth-encryption.rst | 13 +
Documentation/crypto/libcrypto.rst | 1 +
include/crypto/aes-gcm.h | 249 ++++++++++++++++
include/crypto/gcm.h | 2 +-
lib/crypto/Kconfig | 8 +
lib/crypto/aes.c | 269 ++++++++++++++++++
lib/crypto/tests/Kconfig | 1 +
7 files changed, 542 insertions(+), 1 deletion(-)
create mode 100644 Documentation/crypto/libcrypto-auth-encryption.rst
create mode 100644 include/crypto/aes-gcm.h
diff --git a/Documentation/crypto/libcrypto-auth-encryption.rst b/Documentation/crypto/libcrypto-auth-encryption.rst
new file mode 100644
index 000000000000..17412b7bd7bb
--- /dev/null
+++ b/Documentation/crypto/libcrypto-auth-encryption.rst
@@ -0,0 +1,13 @@
+.. SPDX-License-Identifier: GPL-2.0-or-later
+
+Authenticated encryption
+========================
+
+Support for authenticated encryption and decryption.
+
+AES-GCM
+-------
+
+Support for AES in the GCM mode of operation.
+
+.. kernel-doc:: include/crypto/aes-gcm.h
diff --git a/Documentation/crypto/libcrypto.rst b/Documentation/crypto/libcrypto.rst
index bbf5ca137910..b8a57e62f0fa 100644
--- a/Documentation/crypto/libcrypto.rst
+++ b/Documentation/crypto/libcrypto.rst
@@ -158,6 +158,7 @@ API documentation
.. toctree::
:maxdepth: 2
+ libcrypto-auth-encryption
libcrypto-blockcipher
libcrypto-hash
libcrypto-signature
diff --git a/include/crypto/aes-gcm.h b/include/crypto/aes-gcm.h
new file mode 100644
index 000000000000..c3549470ce65
--- /dev/null
+++ b/include/crypto/aes-gcm.h
@@ -0,0 +1,249 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * AES-GCM authenticated encryption and decryption
+ *
+ * Copyright 2026 Google LLC
+ */
+#ifndef _CRYPTO_AES_GCM_H
+#define _CRYPTO_AES_GCM_H
+
+#include <crypto/aes.h>
+#include <crypto/gcm.h>
+#include <crypto/gf128hash.h>
+
+/**
+ * struct aes_gcm_key - Prepared key for AES-GCM
+ */
+struct aes_gcm_key {
+ /* private: */
+ struct aes_enckey aes;
+ struct ghash_key ghash;
+ size_t authtag_len; /* Length of authentication tags in bytes */
+};
+
+/**
+ * struct aes_gcm_ctx - Context for incrementally en/decrypting a message
+ */
+struct aes_gcm_ctx {
+ /* private: */
+ /*
+ * Pointer to the key, which is assumed to live at least as long as this
+ * struct.
+ */
+ const struct aes_gcm_key *key;
+ /* The current GHASH context */
+ struct ghash_ctx ghash;
+ /*
+ * The current counter. This can be viewed as either a 128-bit big
+ * endian counter, or as a 96-bit nonce followed by a 32-bit big endian
+ * counter; it doesn't matter, since the last 32-bit word starts at 1,
+ * and AES-GCM is undefined for messages that would overflow that part.
+ * In practice this means that code optimized for AES-GCM can just
+ * increment the last 32-bit word (wrapping at 2^32), but when needed it
+ * can still call AES-CTR code that does a 128-bit increment.
+ *
+ * 'long' alignment is for crypto_xor() to work more efficiently.
+ */
+ union {
+ u8 ctr[AES_BLOCK_SIZE];
+ __be32 ctr32[AES_BLOCK_SIZE / 4];
+ } __aligned(__alignof__(long));
+ /* Buffered keystream for partial block updates */
+ u8 keystream[AES_BLOCK_SIZE] __aligned(__alignof__(long));
+ /* Encrypted counter of 1. This gets XOR'ed with the tag at the end. */
+ u8 j0_enc[AES_BLOCK_SIZE] __aligned(__alignof__(long));
+ /* Number of associated data bytes processed so far */
+ u64 ad_len;
+ /* Number of en/decrypted bytes processed so far */
+ u64 data_len;
+};
+
+/**
+ * aes_gcm_preparekey() - Prepare an AES-GCM key
+ * @key: (output) The key structure to initialize
+ * @in_key: The raw AES-GCM key
+ * @key_len: Length of the raw key in bytes: 16, 24, or 32
+ * @authtag_len: Length of the authentication tag in bytes:
+ * 4, 8, 12, 13, 14, 15, or 16. 16 is recommended.
+ *
+ * Users should use memzero_explicit() to zeroize the key at the end of its
+ * lifetime. (But if this function fails, zeroization is unnecessary.)
+ *
+ * Context: Any context.
+ * Return:
+ * * 0 on success
+ * * -EINVAL if either of the lengths is invalid
+ */
+int __must_check aes_gcm_preparekey(struct aes_gcm_key *key, const u8 *in_key,
+ size_t key_len, size_t authtag_len);
+
+/**
+ * aes_gcm_encrypt() - Encrypt a message with AES-GCM
+ * @dst: The destination ciphertext data. Can be in-place or out-of-place.
+ * For other overlaps the behavior is unspecified.
+ * @authtag: The output authentication tag. Length is the authtag_len that was
+ * passed to aes_gcm_preparekey(). Usually protocols using AES-GCM
+ * put the tag at the end of the ciphertext, in which case this should
+ * be set to @dst + @data_len and @dst must have room for the tag.
+ * @src: The source plaintext data
+ * @data_len: Length of plaintext in bytes (and ciphertext excluding the tag)
+ * @ad: The associated data
+ * @ad_len: Length of associated data in bytes
+ * @nonce: The 96-bit nonce
+ * @key: The key
+ *
+ * For AES-GMAC (i.e., AES-GCM without any data en/decrypted), use dst=NULL,
+ * src=NULL, and data_len=0.
+ *
+ * Context: Any context.
+ */
+void aes_gcm_encrypt(u8 *dst, u8 *authtag, const u8 *src, size_t data_len,
+ const u8 *ad, size_t ad_len, const u8 nonce[at_least 12],
+ const struct aes_gcm_key *key);
+
+/**
+ * aes_gcm_decrypt() - Decrypt a message with AES-GCM
+ * @dst: The destination plaintext data. Can be in-place or out-of-place.
+ * For other overlaps the behavior is unspecified.
+ * @src: The source ciphertext data
+ * @authtag: The stored authentication tag. Length is the authtag_len that was
+ * passed to aes_gcm_preparekey(). Usually protocols using AES-GCM
+ * put the tag at the end of the ciphertext, in which case this should
+ * be set to @src + @data_len and @src must have room for the tag.
+ * @data_len: Length of plaintext in bytes (and ciphertext excluding the tag)
+ * @ad: The associated data
+ * @ad_len: Length of associated data in bytes
+ * @nonce: The 96-bit nonce
+ * @key: The key
+ *
+ * For AES-GMAC (i.e., AES-GCM without any data en/decrypted), use dst=NULL,
+ * src=NULL, and data_len=0.
+ *
+ * Context: Any context.
+ * Return:
+ * * 0 on success. This is the only case where any decrypted data can be used.
+ * * -EBADMSG if the message is inauthentic
+ */
+int __must_check aes_gcm_decrypt(u8 *dst, const u8 *src, const u8 *authtag,
+ size_t data_len, const u8 *ad, size_t ad_len,
+ const u8 nonce[at_least 12],
+ const struct aes_gcm_key *key);
+
+/**
+ * aes_gcm_init() - Initialize context for incremental AES-GCM encryption or
+ * decryption, or AES-GMAC computation
+ * @ctx: The context to initialize
+ * @nonce: The 96-bit nonce
+ * @key: The key. Note that a pointer to the key is saved in the context, so
+ * the key must live at least as long as the context.
+ *
+ * IMPORTANT: Callers that are decrypting or computing a GMAC value for
+ * verification MUST NOT assume that any decrypted or associated data is
+ * authentic until the authentication tag has been verified. This incremental
+ * API is provided solely to support callers that can't efficiently use the
+ * one-shot functions due to using a nonlinear data layout.
+ *
+ * For incremental AES-GCM encryption, use:
+ *
+ * 1. aes_gcm_init()
+ * 2. aes_gcm_auth_update() (any number of times)
+ * 3. aes_gcm_encrypt_update() (any number of times)
+ * 4. aes_gcm_encrypt_final()
+ *
+ * For incremental AES-GCM decryption, use:
+ *
+ * 1. aes_gcm_init()
+ * 2. aes_gcm_auth_update() (any number of times)
+ * 3. aes_gcm_decrypt_update() (any number of times)
+ * 4. aes_gcm_decrypt_final()
+ *
+ * AES-GMAC is just AES-GCM with zero bytes en/decrypted. For incremental
+ * AES-GMAC computation, use:
+ *
+ * 1. aes_gcm_init()
+ * 2. aes_gcm_auth_update() (any number of times)
+ * 3. aes_gcm_encrypt_final() to return the computed tag to the caller, or
+ * aes_gcm_decrypt_final() to directly verify the computed tag.
+ *
+ * Context: Any context.
+ */
+void aes_gcm_init(struct aes_gcm_ctx *ctx, const u8 nonce[at_least 12],
+ const struct aes_gcm_key *key);
+
+/**
+ * aes_gcm_auth_update() - Incrementally update AES-GCM associated data
+ * @ctx: An AES-GCM context
+ * @ad: The associated data
+ * @len: Length of the associated data in bytes
+ *
+ * IMPORTANT: Callers that are decrypting or computing a GMAC value for
+ * verification MUST NOT assume that any decrypted or associated data is
+ * authentic until the authentication tag has been verified.
+ *
+ * Context: Any context.
+ */
+void aes_gcm_auth_update(struct aes_gcm_ctx *ctx, const u8 *ad, size_t len);
+
+/**
+ * aes_gcm_encrypt_update() - Incrementally encrypt data with AES-GCM
+ * @ctx: An AES-GCM context
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source plaintext data (not including auth tag)
+ * @len: Number of bytes to encrypt
+ *
+ * This can be called only after all associated data has been processed.
+ *
+ * Context: Any context.
+ */
+void aes_gcm_encrypt_update(struct aes_gcm_ctx *ctx, u8 *dst, const u8 *src,
+ size_t len);
+
+/**
+ * aes_gcm_decrypt_update() - Incrementally decrypt data with AES-GCM
+ * @ctx: An AES-GCM context
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source ciphertext data
+ * @len: Number of bytes to decrypt
+ *
+ * This can be called only after all associated data has been processed.
+ *
+ * IMPORTANT: Callers that are decrypting or computing a GMAC value for
+ * verification MUST NOT assume that any decrypted or associated data is
+ * authentic until the authentication tag has been verified.
+ *
+ * Context: Any context.
+ */
+void aes_gcm_decrypt_update(struct aes_gcm_ctx *ctx, u8 *dst, const u8 *src,
+ size_t len);
+
+/**
+ * aes_gcm_encrypt_final() - Finish encrypting a message with AES-GCM
+ * @ctx: An AES-GCM context
+ * @authtag: The output authentication tag. Length is the authtag_len that was
+ * passed to aes_gcm_preparekey().
+ *
+ * This also zeroizes @ctx, so the caller doesn't need to do it.
+ *
+ * Context: Any context.
+ */
+void aes_gcm_encrypt_final(struct aes_gcm_ctx *ctx, u8 *authtag);
+
+/**
+ * aes_gcm_decrypt_final() - Finish decrypting a message with AES-GCM
+ * @ctx: An AES-GCM context
+ * @authtag: The stored authentication tag. Length is the authtag_len that was
+ * passed to aes_gcm_preparekey().
+ *
+ * This also zeroizes @ctx, so the caller doesn't need to do it.
+ *
+ * Context: Any context.
+ * Return:
+ * * 0 on success
+ * * -EBADMSG if the message is inauthentic
+ */
+int __must_check aes_gcm_decrypt_final(struct aes_gcm_ctx *ctx,
+ const u8 *authtag);
+
+#endif /* _CRYPTO_AES_GCM_H */
diff --git a/include/crypto/gcm.h b/include/crypto/gcm.h
index 1d5f39ff1dc4..eb91d5603b74 100644
--- a/include/crypto/gcm.h
+++ b/include/crypto/gcm.h
@@ -13,7 +13,7 @@
/*
* validate authentication tag for GCM
*/
-static inline int crypto_gcm_check_authsize(unsigned int authsize)
+static inline int crypto_gcm_check_authsize(size_t authsize)
{
switch (authsize) {
case 4:
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index 9af44cf743a7..5d313c78b9fa 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -53,6 +53,14 @@ config CRYPTO_LIB_AES_CTR
help
The AES-CTR and AES-XCTR library functions.
+config CRYPTO_LIB_AES_GCM
+ tristate
+ select CRYPTO_LIB_AES
+ select CRYPTO_LIB_AES_CTR
+ select CRYPTO_LIB_GF128HASH
+ help
+ The AES-GCM library functions.
+
config CRYPTO_LIB_AES_XTS
tristate
select CRYPTO_LIB_AES
diff --git a/lib/crypto/aes.c b/lib/crypto/aes.c
index 630702a4228c..50f82437a5d8 100644
--- a/lib/crypto/aes.c
+++ b/lib/crypto/aes.c
@@ -8,6 +8,7 @@
#include <crypto/aes-cbc.h>
#include <crypto/aes-ctr.h>
#include <crypto/aes-ecb.h>
+#include <crypto/aes-gcm.h>
#include <crypto/aes-xts.h>
#include <crypto/aes.h>
#include <crypto/gf128mul.h>
@@ -1284,6 +1285,274 @@ void aes_xts_decrypt(u8 *dst, const u8 *src, size_t len,
EXPORT_SYMBOL_GPL(aes_xts_decrypt);
#endif /* CONFIG_CRYPTO_LIB_AES_XTS */
+#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_GCM)
+/*
+ * Hooks for optimized AES-GCM implementations, overridable by the architecture.
+ * They are called with len > 0 && len % AES_BLOCK_SIZE == 0. I.e. they aren't
+ * expected to handle empty inputs or partial blocks, as those cases are handled
+ * by non-arch-specific code instead.
+ *
+ * The GHASH accumulator is provided in POLYVAL format. The counter is provided
+ * in big endian format, and it's read-only, as the caller handles updating it.
+ *
+ * Returning false causes the fallback implementation to be used instead.
+ *
+ * These hooks are used only for en/decrypted data. For the associated data the
+ * GHASH functions are called instead, so those should be implemented too.
+ */
+#ifndef aes_gcm_encrypt_update_arch
+static bool aes_gcm_encrypt_update_arch(u8 *dst, const u8 *src, size_t len,
+ struct polyval_elem *ghash_acc,
+ const __be32 ctr32[4],
+ const struct aes_enckey *aes_key,
+ const struct ghash_key *ghash_key)
+{
+ return false;
+}
+#endif
+#ifndef aes_gcm_decrypt_update_arch
+static bool aes_gcm_decrypt_update_arch(u8 *dst, const u8 *src, size_t len,
+ struct polyval_elem *ghash_acc,
+ const __be32 ctr32[4],
+ const struct aes_enckey *aes_key,
+ const struct ghash_key *ghash_key)
+{
+ return false;
+}
+#endif
+
+int aes_gcm_preparekey(struct aes_gcm_key *key, const u8 *in_key,
+ size_t key_len, size_t authtag_len)
+{
+ u8 h[AES_BLOCK_SIZE] = { 0 };
+ int err;
+
+ err = crypto_gcm_check_authsize(authtag_len);
+ if (unlikely(err))
+ return err;
+
+ err = aes_prepareenckey(&key->aes, in_key, key_len);
+ if (unlikely(err))
+ return err;
+
+ aes_encrypt(&key->aes, h, h);
+ ghash_preparekey(&key->ghash, h);
+
+ key->authtag_len = authtag_len;
+
+ memzero_explicit(h, sizeof(h));
+ return 0;
+}
+EXPORT_SYMBOL_GPL(aes_gcm_preparekey);
+
+void aes_gcm_init(struct aes_gcm_ctx *ctx, const u8 nonce[12],
+ const struct aes_gcm_key *key)
+{
+ ctx->key = key;
+ ctx->ad_len = 0;
+ ctx->data_len = 0;
+ ghash_init(&ctx->ghash, &key->ghash);
+ memset(ctx->keystream, 0, sizeof(ctx->keystream));
+
+ memcpy(ctx->ctr32, nonce, 12);
+ ctx->ctr32[3] = cpu_to_be32(1);
+
+ aes_encrypt(&key->aes, ctx->j0_enc, ctx->ctr);
+ ctx->ctr32[3] = cpu_to_be32(2);
+}
+EXPORT_SYMBOL_GPL(aes_gcm_init);
+
+void aes_gcm_auth_update(struct aes_gcm_ctx *ctx, const u8 *ad, size_t len)
+{
+ WARN_ON_ONCE(ctx->data_len != 0);
+ if (len) {
+ ghash_update(&ctx->ghash, ad, len);
+ ctx->ad_len += len;
+ }
+}
+EXPORT_SYMBOL_GPL(aes_gcm_auth_update);
+
+static const u8 gcm_zeroes[AES_BLOCK_SIZE];
+
+static __always_inline void ghash_pad(struct ghash_ctx *ghash, u64 len)
+{
+ if (len % AES_BLOCK_SIZE)
+ ghash_update(ghash, gcm_zeroes, -len % AES_BLOCK_SIZE);
+}
+
+static __always_inline void aes_gcm_crypt_update(struct aes_gcm_ctx *ctx,
+ u8 *dst, const u8 *src,
+ size_t len, bool enc)
+{
+ size_t partial_len, n;
+
+ if (unlikely(len == 0))
+ return;
+
+ partial_len = ctx->data_len % AES_BLOCK_SIZE;
+ if (ctx->data_len == 0)
+ ghash_pad(&ctx->ghash, ctx->ad_len);
+ ctx->data_len += len;
+
+ if (unlikely(partial_len != 0)) {
+ /*
+ * The previous call ended on a non-block-aligned data_len, so
+ * continue using a previously-generated keystream block.
+ */
+ n = min(len, AES_BLOCK_SIZE - partial_len);
+ if (enc) {
+ crypto_xor_cpy(dst, src, &ctx->keystream[partial_len],
+ n);
+ ghash_update(&ctx->ghash, dst, n);
+ } else {
+ ghash_update(&ctx->ghash, src, n);
+ crypto_xor_cpy(dst, src, &ctx->keystream[partial_len],
+ n);
+ }
+ dst += n;
+ src += n;
+ len -= n;
+ }
+
+ if (len >= AES_BLOCK_SIZE) {
+ n = round_down(len, AES_BLOCK_SIZE);
+ if (enc) {
+ if (likely(aes_gcm_encrypt_update_arch(
+ dst, src, n, &ctx->ghash.acc, ctx->ctr32,
+ &ctx->key->aes, &ctx->key->ghash))) {
+ be32_add_cpu(&ctx->ctr32[3],
+ n / AES_BLOCK_SIZE);
+ } else {
+ aes_ctr(dst, src, n, ctx->ctr, &ctx->key->aes);
+ ghash_update(&ctx->ghash, dst, n);
+ }
+ } else {
+ if (likely(aes_gcm_decrypt_update_arch(
+ dst, src, n, &ctx->ghash.acc, ctx->ctr32,
+ &ctx->key->aes, &ctx->key->ghash))) {
+ be32_add_cpu(&ctx->ctr32[3],
+ n / AES_BLOCK_SIZE);
+ } else {
+ ghash_update(&ctx->ghash, src, n);
+ aes_ctr(dst, src, n, ctx->ctr, &ctx->key->aes);
+ }
+ }
+ dst += n;
+ src += n;
+ len -= n;
+ }
+
+ if (len != 0) {
+ /*
+ * Ending on a non-block aligned data_len. Generate the next
+ * keystream block, use the needed portion of it, and leave it
+ * cached in ctx->keystream in case this isn't the final call.
+ */
+ aes_encrypt(&ctx->key->aes, ctx->keystream, ctx->ctr);
+ be32_add_cpu(&ctx->ctr32[3], 1);
+ if (enc) {
+ crypto_xor_cpy(dst, src, ctx->keystream, len);
+ ghash_update(&ctx->ghash, dst, len);
+ } else {
+ ghash_update(&ctx->ghash, src, len);
+ crypto_xor_cpy(dst, src, ctx->keystream, len);
+ }
+ }
+}
+
+void aes_gcm_encrypt_update(struct aes_gcm_ctx *ctx, u8 *dst, const u8 *src,
+ size_t len)
+{
+ aes_gcm_crypt_update(ctx, dst, src, len, /* enc= */ true);
+}
+EXPORT_SYMBOL_GPL(aes_gcm_encrypt_update);
+
+void aes_gcm_decrypt_update(struct aes_gcm_ctx *ctx, u8 *dst, const u8 *src,
+ size_t len)
+{
+ aes_gcm_crypt_update(ctx, dst, src, len, /* enc= */ false);
+}
+EXPORT_SYMBOL_GPL(aes_gcm_decrypt_update);
+
+void aes_gcm_encrypt_final(struct aes_gcm_ctx *ctx, u8 *authtag)
+{
+ __be64 tail[2];
+
+ WARN_ON_ONCE(ctx->data_len > (1ULL << 36) - 32);
+
+ ghash_pad(&ctx->ghash,
+ ctx->data_len == 0 ? ctx->ad_len : ctx->data_len);
+
+ tail[0] = cpu_to_be64(ctx->ad_len * 8);
+ tail[1] = cpu_to_be64(ctx->data_len * 8);
+ ghash_update(&ctx->ghash, (const u8 *)tail, 16);
+ ghash_final(&ctx->ghash, ctx->ctr); /* Use ctr as temp buffer */
+
+ crypto_xor_cpy(authtag, ctx->ctr, ctx->j0_enc, ctx->key->authtag_len);
+ memzero_explicit(ctx, sizeof(*ctx));
+}
+EXPORT_SYMBOL_GPL(aes_gcm_encrypt_final);
+
+int aes_gcm_decrypt_final(struct aes_gcm_ctx *ctx, const u8 *authtag)
+{
+ __be64 tail[2];
+ int err;
+
+ WARN_ON_ONCE(ctx->data_len > (1ULL << 36) - 32);
+
+ ghash_pad(&ctx->ghash,
+ ctx->data_len == 0 ? ctx->ad_len : ctx->data_len);
+
+ tail[0] = cpu_to_be64(ctx->ad_len * 8);
+ tail[1] = cpu_to_be64(ctx->data_len * 8);
+ ghash_update(&ctx->ghash, (const u8 *)tail, 16);
+ ghash_final(&ctx->ghash, ctx->ctr); /* Use ctr as temp buffer */
+ crypto_xor(ctx->ctr, ctx->j0_enc, ctx->key->authtag_len);
+ err = crypto_memneq(ctx->ctr, authtag, ctx->key->authtag_len) ?
+ -EBADMSG :
+ 0;
+ memzero_explicit(ctx, sizeof(*ctx));
+ return err;
+}
+EXPORT_SYMBOL_GPL(aes_gcm_decrypt_final);
+
+void aes_gcm_encrypt(u8 *dst, u8 *authtag, const u8 *src, size_t data_len,
+ const u8 *ad, size_t ad_len, const u8 nonce[12],
+ const struct aes_gcm_key *key)
+{
+ struct aes_gcm_ctx ctx;
+
+ aes_gcm_init(&ctx, nonce, key);
+ aes_gcm_auth_update(&ctx, ad, ad_len);
+ aes_gcm_encrypt_update(&ctx, dst, src, data_len);
+ aes_gcm_encrypt_final(&ctx, authtag);
+}
+EXPORT_SYMBOL_GPL(aes_gcm_encrypt);
+
+int aes_gcm_decrypt(u8 *dst, const u8 *src, const u8 *authtag, size_t data_len,
+ const u8 *ad, size_t ad_len, const u8 nonce[12],
+ const struct aes_gcm_key *key)
+{
+ struct aes_gcm_ctx ctx;
+ int err;
+
+ aes_gcm_init(&ctx, nonce, key);
+ aes_gcm_auth_update(&ctx, ad, ad_len);
+ aes_gcm_decrypt_update(&ctx, dst, src, data_len);
+ err = aes_gcm_decrypt_final(&ctx, authtag);
+ if (unlikely(err)) {
+ /*
+ * Clear the inauthentic decrypted data so that callers won't
+ * receive it even if they fail to correctly handle errors.
+ */
+ memset(dst, 0, data_len);
+ }
+ return err;
+}
+EXPORT_SYMBOL_GPL(aes_gcm_decrypt);
+
+#endif /* CONFIG_CRYPTO_LIB_AES_GCM */
+
static int __init aes_mod_init(void)
{
#ifdef aes_mod_init_arch
diff --git a/lib/crypto/tests/Kconfig b/lib/crypto/tests/Kconfig
index b559e7c79e76..51183ffabbef 100644
--- a/lib/crypto/tests/Kconfig
+++ b/lib/crypto/tests/Kconfig
@@ -148,6 +148,7 @@ config CRYPTO_LIB_ENABLE_ALL_FOR_KUNIT
select CRYPTO_LIB_AES_CBC_MACS
select CRYPTO_LIB_AES_CTR
select CRYPTO_LIB_AES_ECB
+ select CRYPTO_LIB_AES_GCM
select CRYPTO_LIB_AES_XTS
select CRYPTO_LIB_BLAKE2B
select CRYPTO_LIB_CHACHA20POLY1305
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 07/33] lib/crypto: aes: Add CCM support
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (5 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 06/33] lib/crypto: aes: Add GCM support Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 08/33] crypto: aes - Add ECB support using library Eric Biggers
` (25 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Add support for AES-CCM to the crypto library.
This will be used to provide a streamlined implementation of the
"ccm(aes)" crypto_aead algorithm. Most users of "ccm(aes)" will also be
able to switch to the library, which as usual will be faster and
simpler, e.g.:
- fs/smb/client/
- fs/smb/server/
- net/mac80211/
- net/mac802154/
(I've already written proof-of-concept patches for all the above, and
they helped inform the API design.)
As in the AES-GCM API, incremental operation is supported. It has to be
used carefully, especially when decrypting, but it makes the API general
enough to work well for all users.
The AES-CCM library code calls aes_cbcmac_blocks() directly, bypassing
the higher-level aes_cbcmac_init(), aes_cbcmac_update(), and
aes_cbcmac_final(). The latter set of functions is useful only for
AES-CCM, so they don't make sense to keep around and will be removed
once the "ccm(aes)" crypto_aead starts using the AES-CCM library.
Initial test coverage is provided by the crypto_aead support added in a
later commit. I'm planning a KUnit test suite as well.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
.../crypto/libcrypto-auth-encryption.rst | 7 +
include/crypto/aes-ccm.h | 244 +++++++++++++++
lib/crypto/Kconfig | 8 +
lib/crypto/aes.c | 287 ++++++++++++++++++
lib/crypto/tests/Kconfig | 1 +
5 files changed, 547 insertions(+)
create mode 100644 include/crypto/aes-ccm.h
diff --git a/Documentation/crypto/libcrypto-auth-encryption.rst b/Documentation/crypto/libcrypto-auth-encryption.rst
index 17412b7bd7bb..f3fa5b5305eb 100644
--- a/Documentation/crypto/libcrypto-auth-encryption.rst
+++ b/Documentation/crypto/libcrypto-auth-encryption.rst
@@ -5,6 +5,13 @@ Authenticated encryption
Support for authenticated encryption and decryption.
+AES-CCM
+-------
+
+Support for AES in the CCM mode of operation.
+
+.. kernel-doc:: include/crypto/aes-ccm.h
+
AES-GCM
-------
diff --git a/include/crypto/aes-ccm.h b/include/crypto/aes-ccm.h
new file mode 100644
index 000000000000..9c179944a2ed
--- /dev/null
+++ b/include/crypto/aes-ccm.h
@@ -0,0 +1,244 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * AES-CCM authenticated encryption and decryption
+ *
+ * Copyright 2026 Google LLC
+ */
+#ifndef _CRYPTO_AES_CCM_H
+#define _CRYPTO_AES_CCM_H
+
+#include <crypto/aes.h>
+
+/**
+ * struct aes_ccm_key - Prepared key for AES-CCM
+ */
+struct aes_ccm_key {
+ /* private: */
+ struct aes_enckey aes;
+ size_t authtag_len; /* Length of authentication tags in bytes */
+};
+
+/**
+ * struct aes_ccm_ctx - Context for incrementally en/decrypting a message
+ */
+struct aes_ccm_ctx {
+ /* private: */
+ /*
+ * Pointer to the key, which is assumed to live at least as long as this
+ * struct.
+ */
+ const struct aes_ccm_key *key;
+ /*
+ * The current CBC-MAC chaining value. When not on a block boundary,
+ * the partial block has been XOR'ed into this. The number of partial
+ * bytes is 'partial_len'.
+ */
+ u8 mac[AES_BLOCK_SIZE] __aligned(__alignof__(__be64));
+ /* The current counter, a 128-bit big endian value */
+ u8 ctr[AES_BLOCK_SIZE] __aligned(__alignof__(__be64));
+ /* Buffered keystream for partial block updates */
+ u8 keystream[AES_BLOCK_SIZE] __aligned(__alignof__(long));
+ /* Encrypted counter of 0. This gets XOR'ed with the tag at the end */
+ u8 s0[AES_BLOCK_SIZE] __aligned(__alignof__(long));
+ /* Current partial block length, 0 <= partial_len < AES_BLOCK_SIZE */
+ u32 partial_len;
+ /* True if associated data padding has been done */
+ bool ad_padded;
+ /* Number of associated data bytes remaining to be provided */
+ s64 ad_remaining;
+ /* Number of en/decrypted data bytes remaining to be provided */
+ s64 data_remaining;
+};
+
+/**
+ * aes_ccm_preparekey() - Prepare an AES-CCM key
+ * @key: (output) The key structure to initialize
+ * @in_key: The raw AES-CCM key
+ * @key_len: Length of the raw key in bytes: 16, 24, or 32
+ * @authtag_len: Length of the authentication tag in bytes:
+ * 4, 6, 8, 10, 12, 14, or 16 (16 is recommended)
+ *
+ * Users should use memzero_explicit() to zeroize the key at the end of its
+ * lifetime. (But if this function fails, zeroization is unnecessary.)
+ *
+ * Context: Any context.
+ * Return:
+ * * 0 on success
+ * * -EINVAL if either of the lengths is invalid
+ */
+int __must_check aes_ccm_preparekey(struct aes_ccm_key *key, const u8 *in_key,
+ size_t key_len, size_t authtag_len);
+
+/**
+ * aes_ccm_encrypt() - Encrypt a message with AES-CCM
+ * @dst: The destination ciphertext data. Can be in-place or out-of-place.
+ * For other overlaps the behavior is unspecified.
+ * @authtag: The output authentication tag. Length is the authtag_len that was
+ * passed to aes_ccm_preparekey(). Usually protocols using AES-CCM
+ * put the tag at the end of the ciphertext, in which case this should
+ * be set to @dst + @data_len and @dst must have room for the tag.
+ * @src: The source plaintext data
+ * @data_len: Length of plaintext in bytes (and ciphertext excluding the tag)
+ * @ad: The associated data
+ * @ad_len: Length of associated data in bytes
+ * @nonce: The nonce
+ * @nonce_len: Length of the nonce in bytes: between 7 and 13 inclusive
+ * @key: The key
+ *
+ * Context: Any context.
+ * Return:
+ * * 0 on success
+ * * -EINVAL if nonce_len is invalid
+ * * -EOVERFLOW if data_len is too large for the selected nonce_len
+ * or if ad_len exceeds U32_MAX
+ */
+int __must_check aes_ccm_encrypt(u8 *dst, u8 *authtag, const u8 *src,
+ size_t data_len, const u8 *ad, size_t ad_len,
+ const u8 *nonce, size_t nonce_len,
+ const struct aes_ccm_key *key);
+
+/**
+ * aes_ccm_decrypt() - Decrypt a message with AES-CCM
+ * @dst: The destination plaintext data. Can be in-place or out-of-place.
+ * For other overlaps the behavior is unspecified.
+ * @src: The source ciphertext data
+ * @authtag: The stored authentication tag. Length is the authtag_len that was
+ * passed to aes_ccm_preparekey(). Usually protocols using AES-CCM
+ * put the tag at the end of the ciphertext, in which case this should
+ * be set to @src + @data_len and @src must have room for the tag.
+ * @data_len: Length of plaintext in bytes (and ciphertext excluding the tag)
+ * @ad: The associated data
+ * @ad_len: Length of associated data in bytes
+ * @nonce: The nonce
+ * @nonce_len: Length of the nonce in bytes: between 7 and 13 inclusive
+ * @key: The key
+ *
+ * Context: Any context.
+ * Return:
+ * * 0 on success. This is the only case where any decrypted data can be used.
+ * * -EBADMSG if the message is inauthentic
+ * * -EINVAL if nonce_len is invalid
+ * * -EOVERFLOW if data_len is too large for the selected nonce_len
+ * or if ad_len exceeds U32_MAX
+ */
+int __must_check aes_ccm_decrypt(u8 *dst, const u8 *src, const u8 *authtag,
+ size_t data_len, const u8 *ad, size_t ad_len,
+ const u8 *nonce, size_t nonce_len,
+ const struct aes_ccm_key *key);
+
+/**
+ * aes_ccm_init() - Initialize context for incremental AES-CCM encryption or
+ * decryption
+ * @ctx: The context to initialize
+ * @nonce: The nonce
+ * @nonce_len: Length of the nonce in bytes: between 7 and 13 inclusive
+ * @ad_len: Length of the associated data that will be provided in bytes
+ * @data_len: Length of the plaintext that will be provided in bytes
+ * @key: The key. Note that a pointer to the key is saved in the context, so
+ * the key must live at least as long as the context.
+ *
+ * Unlike AES-GCM, AES-CCM requires the total lengths of the associated data and
+ * plaintext to be known during initialization. Callers MUST ensure that these
+ * lengths are correct.
+ *
+ * IMPORTANT: Callers that are decrypting MUST NOT assume that any decrypted or
+ * associated data is authentic until the authentication tag has been verified.
+ * This incremental API is provided solely to support callers that can't
+ * efficiently use the one-shot functions due to using a nonlinear data layout.
+ *
+ * For incremental AES-CCM encryption, use:
+ *
+ * 1. aes_ccm_init()
+ * 2. aes_ccm_auth_update() (any number of times)
+ * 3. aes_ccm_encrypt_update() (any number of times)
+ * 4. aes_ccm_encrypt_final()
+ *
+ * For incremental AES-CCM decryption, use:
+ *
+ * 1. aes_ccm_init()
+ * 2. aes_ccm_auth_update() (any number of times)
+ * 3. aes_ccm_decrypt_update() (any number of times)
+ * 4. aes_ccm_decrypt_final()
+ *
+ * Context: Any context.
+ * Return:
+ * * 0 on success
+ * * -EINVAL if nonce_len is invalid
+ * * -EOVERFLOW if data_len is too large for the selected nonce_len
+ * or if ad_len exceeds U32_MAX
+ */
+int __must_check aes_ccm_init(struct aes_ccm_ctx *ctx, const u8 *nonce,
+ size_t nonce_len, u64 ad_len, u64 data_len,
+ const struct aes_ccm_key *key);
+
+/**
+ * aes_ccm_auth_update() - Incrementally update AES-CCM associated data
+ * @ctx: An AES-CCM context
+ * @ad: The associated data
+ * @len: Length of the associated data in bytes
+ *
+ * IMPORTANT: Callers MUST NOT assume that any decrypted or associated data is
+ * authentic until the authentication tag has been verified.
+ *
+ * Context: Any context.
+ */
+void aes_ccm_auth_update(struct aes_ccm_ctx *ctx, const u8 *ad, size_t len);
+
+/**
+ * aes_ccm_encrypt_update() - Incrementally encrypt data with AES-CCM
+ * @ctx: An AES-CCM context
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source plaintext data
+ * @len: Number of bytes to encrypt
+ *
+ * Context: Any context.
+ */
+void aes_ccm_encrypt_update(struct aes_ccm_ctx *ctx, u8 *dst, const u8 *src,
+ size_t len);
+
+/**
+ * aes_ccm_decrypt_update() - Incrementally decrypt data with AES-CCM
+ * @ctx: An AES-CCM context
+ * @dst: The destination buffer. Can be in-place or out-of-place. For other
+ * overlaps the behavior is unspecified.
+ * @src: The source ciphertext data (not including auth tag)
+ * @len: Number of bytes to decrypt
+ *
+ * IMPORTANT: Callers MUST NOT assume that any decrypted or associated data is
+ * authentic until the authentication tag has been verified.
+ *
+ * Context: Any context.
+ */
+void aes_ccm_decrypt_update(struct aes_ccm_ctx *ctx, u8 *dst, const u8 *src,
+ size_t len);
+
+/**
+ * aes_ccm_encrypt_final() - Finish encrypting a message with AES-CCM
+ * @ctx: An AES-CCM context
+ * @authtag: The output authentication tag. Length is the authtag_len that was
+ * passed to aes_ccm_preparekey().
+ *
+ * This also zeroizes @ctx, so the caller doesn't need to do it.
+ *
+ * Context: Any context.
+ */
+void aes_ccm_encrypt_final(struct aes_ccm_ctx *ctx, u8 *authtag);
+
+/**
+ * aes_ccm_decrypt_final() - Finish decrypting a message with AES-CCM
+ * @ctx: An AES-CCM context
+ * @authtag: The stored authentication tag. Length is the authtag_len that was
+ * passed to aes_ccm_preparekey().
+ *
+ * This also zeroizes @ctx, so the caller doesn't need to do it.
+ *
+ * Context: Any context.
+ * Return:
+ * * 0 on success. This is the only case where any decrypted data can be used.
+ * * -EBADMSG if the message is inauthentic
+ */
+int __must_check aes_ccm_decrypt_final(struct aes_ccm_ctx *ctx,
+ const u8 *authtag);
+
+#endif /* _CRYPTO_AES_CCM_H */
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index 5d313c78b9fa..4066427017bc 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -47,6 +47,14 @@ config CRYPTO_LIB_AES_ECB
help
The AES-ECB library functions.
+config CRYPTO_LIB_AES_CCM
+ tristate
+ select CRYPTO_LIB_AES
+ select CRYPTO_LIB_AES_CBC_MACS
+ select CRYPTO_LIB_AES_CTR
+ help
+ The AES-CCM library functions.
+
config CRYPTO_LIB_AES_CTR
tristate
select CRYPTO_LIB_AES
diff --git a/lib/crypto/aes.c b/lib/crypto/aes.c
index 50f82437a5d8..1a1b32e41ac1 100644
--- a/lib/crypto/aes.c
+++ b/lib/crypto/aes.c
@@ -6,6 +6,7 @@
#include <crypto/aes-cbc-macs.h>
#include <crypto/aes-cbc.h>
+#include <crypto/aes-ccm.h>
#include <crypto/aes-ctr.h>
#include <crypto/aes-ecb.h>
#include <crypto/aes-gcm.h>
@@ -1553,6 +1554,292 @@ EXPORT_SYMBOL_GPL(aes_gcm_decrypt);
#endif /* CONFIG_CRYPTO_LIB_AES_GCM */
+#if IS_ENABLED(CONFIG_CRYPTO_LIB_AES_CCM)
+int aes_ccm_preparekey(struct aes_ccm_key *key, const u8 *in_key,
+ size_t key_len, size_t authtag_len)
+{
+ int err;
+
+ if (authtag_len < 4 || authtag_len > 16 || authtag_len % 2)
+ return -EINVAL;
+
+ err = aes_prepareenckey(&key->aes, in_key, key_len);
+ if (err)
+ return err;
+
+ key->authtag_len = authtag_len;
+ return 0;
+}
+EXPORT_SYMBOL_GPL(aes_ccm_preparekey);
+
+int aes_ccm_init(struct aes_ccm_ctx *ctx, const u8 *nonce, size_t nonce_len,
+ u64 ad_len, u64 data_len, const struct aes_ccm_key *key)
+{
+ /*
+ * This is the value L defined in the CCM specification. It determines
+ * the maximum allowed message length, and it is itself determined by
+ * the nonce length. They are inversely related, i.e. the longer the
+ * nonce the smaller the maximum message length is.
+ */
+ unsigned int l = 15 - nonce_len;
+
+ if (unlikely(nonce_len < 7 || nonce_len > 13))
+ return -EINVAL;
+ /* Thus 2 <= l <= 8. */
+
+ if (unlikely(l < 8 && data_len >= (1ULL << (8 * l))))
+ return -EOVERFLOW;
+
+ if (unlikely(ad_len > U32_MAX))
+ return -EOVERFLOW;
+
+ ctx->key = key;
+ ctx->ad_remaining = ad_len;
+ ctx->data_remaining = data_len;
+ ctx->ad_padded = false;
+
+ /*
+ * Initialize the zero-th counter block to:
+ *
+ * L - 1 || nonce || 0
+ *
+ * ... and the zero-th CBC-MAC block to:
+ *
+ * Flags || nonce || data_len
+ */
+ *(__be64 *)&ctx->ctr[8] = 0;
+ *(__be64 *)&ctx->mac[8] = cpu_to_be64(data_len);
+ ctx->ctr[0] = l - 1;
+ ctx->mac[0] = (ad_len ? 0x40 : 0) |
+ (((key->authtag_len - 2) / 2) << 3) | (l - 1);
+ memcpy(&ctx->ctr[1], nonce, nonce_len); /* Overlapping store */
+ memcpy(&ctx->mac[1], nonce, nonce_len); /* Overlapping store */
+
+ /*
+ * Generate S_0 by encrypting the counter (this is used to encrypt the
+ * auth tag later), and encrypt the zero-th CBC-MAC block.
+ */
+ aes_encrypt(&key->aes, ctx->s0, ctx->ctr);
+ aes_encrypt(&key->aes, ctx->mac, ctx->mac);
+
+ /* Increment the counter from 0 to 1. */
+ ctx->ctr[15] = 1;
+
+ if (ad_len) {
+ /*
+ * Update CBC-MAC with the associated data length, represented
+ * using either 2 or 6 bytes depending on the length.
+ */
+ if (likely(ad_len < 0xff00)) {
+ *(__be16 *)&ctx->mac[0] ^= cpu_to_be16(ad_len);
+ ctx->partial_len = 2;
+ } else {
+ *(__be16 *)&ctx->mac[0] ^= cpu_to_be16(0xfffe);
+ *(__be32 *)&ctx->mac[2] ^= cpu_to_be32(ad_len);
+ ctx->partial_len = 6;
+ }
+ } else {
+ ctx->partial_len = 0;
+ }
+ return 0;
+}
+EXPORT_SYMBOL_GPL(aes_ccm_init);
+
+void aes_ccm_auth_update(struct aes_ccm_ctx *ctx, const u8 *ad, size_t len)
+{
+ size_t partial_len = ctx->partial_len;
+ bool enc_before = false;
+ size_t nblocks;
+
+ ctx->ad_remaining -= len;
+
+ if (partial_len) {
+ size_t n = min(len, AES_BLOCK_SIZE - partial_len);
+
+ crypto_xor(&ctx->mac[partial_len], ad, n);
+ ad += n;
+ len -= n;
+ partial_len += n;
+ if (partial_len < AES_BLOCK_SIZE) {
+ ctx->partial_len = partial_len;
+ return;
+ }
+ enc_before = true;
+ }
+
+ nblocks = len / AES_BLOCK_SIZE;
+ len %= AES_BLOCK_SIZE;
+ if (nblocks == 0) {
+ if (enc_before)
+ aes_encrypt(&ctx->key->aes, ctx->mac, ctx->mac);
+ } else {
+ aes_cbcmac_blocks(ctx->mac, &ctx->key->aes, ad, nblocks,
+ enc_before, /* enc_after= */ true);
+ ad += nblocks * AES_BLOCK_SIZE;
+ }
+ crypto_xor(ctx->mac, ad, len);
+ ctx->partial_len = len;
+}
+EXPORT_SYMBOL_GPL(aes_ccm_auth_update);
+
+static __always_inline void aes_ccm_crypt_update(struct aes_ccm_ctx *ctx,
+ u8 *dst, const u8 *src,
+ size_t len, bool enc)
+{
+ size_t partial_len = ctx->partial_len;
+ size_t n;
+
+ if (unlikely(len == 0))
+ return;
+
+ WARN_ON_ONCE(ctx->ad_remaining != 0);
+ ctx->data_remaining -= len;
+
+ if (!ctx->ad_padded) {
+ ctx->ad_padded = true;
+ if (partial_len)
+ aes_encrypt(&ctx->key->aes, ctx->mac, ctx->mac);
+ } else if (partial_len) {
+ /*
+ * The previous call ended on a non-block-aligned data_len, so
+ * continue using a previously-generated keystream block.
+ */
+ n = min(len, AES_BLOCK_SIZE - partial_len);
+ if (enc)
+ crypto_xor(&ctx->mac[partial_len], src, n);
+ crypto_xor_cpy(dst, src, &ctx->keystream[partial_len], n);
+ if (!enc)
+ crypto_xor(&ctx->mac[partial_len], dst, n);
+ dst += n;
+ src += n;
+ len -= n;
+ partial_len += n;
+ if (partial_len < AES_BLOCK_SIZE) {
+ ctx->partial_len = partial_len;
+ return;
+ }
+ aes_encrypt(&ctx->key->aes, ctx->mac, ctx->mac);
+ }
+
+ if (len >= AES_BLOCK_SIZE) {
+ n = round_down(len, AES_BLOCK_SIZE);
+ if (enc)
+ aes_cbcmac_blocks(ctx->mac, &ctx->key->aes, src,
+ len / AES_BLOCK_SIZE,
+ /* enc_before= */ false,
+ /* enc_after= */ true);
+ aes_ctr(dst, src, n, ctx->ctr, &ctx->key->aes);
+ if (!enc)
+ aes_cbcmac_blocks(ctx->mac, &ctx->key->aes, dst,
+ len / AES_BLOCK_SIZE,
+ /* enc_before= */ false,
+ /* enc_after= */ true);
+ dst += n;
+ src += n;
+ len -= n;
+ }
+
+ if (len) {
+ /*
+ * Ending on a non-block aligned data_len. Generate the next
+ * keystream block, use the needed portion of it, and leave it
+ * cached in ctx->keystream in case this isn't the final call.
+ */
+ aes_encrypt(&ctx->key->aes, ctx->keystream, ctx->ctr);
+ inc_be128_ctr(ctx->ctr);
+ if (enc)
+ crypto_xor(ctx->mac, src, len);
+ crypto_xor_cpy(dst, src, ctx->keystream, len);
+ if (!enc)
+ crypto_xor(ctx->mac, dst, len);
+ }
+ ctx->partial_len = len;
+}
+
+void aes_ccm_encrypt_update(struct aes_ccm_ctx *ctx, u8 *dst, const u8 *src,
+ size_t len)
+{
+ aes_ccm_crypt_update(ctx, dst, src, len, /* enc= */ true);
+}
+EXPORT_SYMBOL_GPL(aes_ccm_encrypt_update);
+
+void aes_ccm_decrypt_update(struct aes_ccm_ctx *ctx, u8 *dst, const u8 *src,
+ size_t len)
+{
+ aes_ccm_crypt_update(ctx, dst, src, len, /* enc= */ false);
+}
+EXPORT_SYMBOL_GPL(aes_ccm_decrypt_update);
+
+void aes_ccm_encrypt_final(struct aes_ccm_ctx *ctx, u8 *authtag)
+{
+ WARN_ON_ONCE(ctx->ad_remaining != 0);
+ WARN_ON_ONCE(ctx->data_remaining != 0);
+ if (ctx->partial_len)
+ aes_encrypt(&ctx->key->aes, ctx->mac, ctx->mac);
+ crypto_xor_cpy(authtag, ctx->mac, ctx->s0, ctx->key->authtag_len);
+ memzero_explicit(ctx, sizeof(*ctx));
+}
+EXPORT_SYMBOL_GPL(aes_ccm_encrypt_final);
+
+int aes_ccm_decrypt_final(struct aes_ccm_ctx *ctx, const u8 *authtag)
+{
+ int err;
+
+ WARN_ON_ONCE(ctx->ad_remaining != 0);
+ WARN_ON_ONCE(ctx->data_remaining != 0);
+ if (ctx->partial_len)
+ aes_encrypt(&ctx->key->aes, ctx->mac, ctx->mac);
+ crypto_xor(ctx->mac, ctx->s0, ctx->key->authtag_len);
+ err = crypto_memneq(ctx->mac, authtag, ctx->key->authtag_len) ?
+ -EBADMSG :
+ 0;
+ memzero_explicit(ctx, sizeof(*ctx));
+ return err;
+}
+EXPORT_SYMBOL_GPL(aes_ccm_decrypt_final);
+
+int aes_ccm_encrypt(u8 *dst, u8 *authtag, const u8 *src, size_t data_len,
+ const u8 *ad, size_t ad_len, const u8 *nonce,
+ size_t nonce_len, const struct aes_ccm_key *key)
+{
+ struct aes_ccm_ctx ctx;
+ int err;
+
+ err = aes_ccm_init(&ctx, nonce, nonce_len, ad_len, data_len, key);
+ if (unlikely(err))
+ return err;
+ aes_ccm_auth_update(&ctx, ad, ad_len);
+ aes_ccm_encrypt_update(&ctx, dst, src, data_len);
+ aes_ccm_encrypt_final(&ctx, authtag);
+ return 0;
+}
+EXPORT_SYMBOL_GPL(aes_ccm_encrypt);
+
+int aes_ccm_decrypt(u8 *dst, const u8 *src, const u8 *authtag, size_t data_len,
+ const u8 *ad, size_t ad_len, const u8 *nonce,
+ size_t nonce_len, const struct aes_ccm_key *key)
+{
+ struct aes_ccm_ctx ctx;
+ int err;
+
+ err = aes_ccm_init(&ctx, nonce, nonce_len, ad_len, data_len, key);
+ if (unlikely(err))
+ return err;
+ aes_ccm_auth_update(&ctx, ad, ad_len);
+ aes_ccm_decrypt_update(&ctx, dst, src, data_len);
+ err = aes_ccm_decrypt_final(&ctx, authtag);
+ if (unlikely(err)) {
+ /*
+ * Clear the inauthentic decrypted data so that callers won't
+ * receive it even if they fail to correctly handle errors.
+ */
+ memset(dst, 0, data_len);
+ }
+ return err;
+}
+EXPORT_SYMBOL_GPL(aes_ccm_decrypt);
+#endif /* CONFIG_CRYPTO_LIB_AES_CCM */
+
static int __init aes_mod_init(void)
{
#ifdef aes_mod_init_arch
diff --git a/lib/crypto/tests/Kconfig b/lib/crypto/tests/Kconfig
index 51183ffabbef..bc084dde424f 100644
--- a/lib/crypto/tests/Kconfig
+++ b/lib/crypto/tests/Kconfig
@@ -146,6 +146,7 @@ config CRYPTO_LIB_ENABLE_ALL_FOR_KUNIT
depends on KUNIT
select CRYPTO_LIB_AES_CBC
select CRYPTO_LIB_AES_CBC_MACS
+ select CRYPTO_LIB_AES_CCM
select CRYPTO_LIB_AES_CTR
select CRYPTO_LIB_AES_ECB
select CRYPTO_LIB_AES_GCM
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 08/33] crypto: aes - Add ECB support using library
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (6 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 07/33] lib/crypto: aes: Add CCM support Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 09/33] crypto: aes - Add CBC and CBC-CTS " Eric Biggers
` (24 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Implement the "ecb(aes)" crypto_skcipher algorithm using the
corresponding library functions.
Among other benefits, this allows the architecture-optimized AES-ECB
code to be migrated into the library while still leaving it accessible
via crypto_skcipher, eliminating lots of boilerplate code.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of this algorithm haven't yet
been migrated into the library. It will be boosted once that happens.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
crypto/Kconfig | 5 ++
crypto/aes.c | 165 ++++++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 169 insertions(+), 1 deletion(-)
diff --git a/crypto/Kconfig b/crypto/Kconfig
index f1e372195273..093508d13b8c 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -359,7 +359,12 @@ config CRYPTO_AES
select CRYPTO_ALGAPI
select CRYPTO_LIB_AES
select CRYPTO_LIB_AES_CBC_MACS if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
+ select CRYPTO_LIB_AES_ECB if CRYPTO_ECB
select CRYPTO_HASH if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
+ # CRYPTO_SKCIPHER should be selected only if a mode that needs it is
+ # enabled, but that doesn't work due to a recursive dependency caused by
+ # CRYPTO_SKCIPHER selecting CRYPTO_ECB. So just always select it.
+ select CRYPTO_SKCIPHER
help
AES cipher algorithms (Rijndael)(FIPS-197, ISO/IEC 18033-3)
diff --git a/crypto/aes.c b/crypto/aes.c
index 6bf23eb0503f..162715a82be3 100644
--- a/crypto/aes.c
+++ b/crypto/aes.c
@@ -6,12 +6,16 @@
*/
#include <crypto/aes-cbc-macs.h>
+#include <crypto/aes-ecb.h>
#include <crypto/aes.h>
#include <crypto/algapi.h>
#include <crypto/internal/hash.h>
+#include <crypto/internal/skcipher.h>
+#include <crypto/scatterwalk.h>
#include <linux/module.h>
static_assert(__alignof__(struct aes_key) <= CRYPTO_MINALIGN);
+static_assert(__alignof__(struct aes_enckey) <= CRYPTO_MINALIGN);
static int crypto_aes_setkey(struct crypto_tfm *tfm, const u8 *in_key,
unsigned int key_len)
@@ -85,7 +89,6 @@ static int __maybe_unused crypto_aes_cmac_digest(struct shash_desc *desc,
return 0;
}
-static_assert(__alignof__(struct aes_enckey) <= CRYPTO_MINALIGN);
#define AES_CBCMAC_KEY(tfm) ((struct aes_enckey *)crypto_shash_ctx(tfm))
#define AES_CBCMAC_CTX(desc) ((struct aes_cbcmac_ctx *)shash_desc_ctx(desc))
@@ -200,6 +203,149 @@ static struct shash_alg mac_algs[] = {
#endif
};
+static __maybe_unused int
+crypto_aes_skcipher_setkey(struct crypto_skcipher *tfm, const u8 *in_key,
+ unsigned int key_len)
+{
+ struct aes_key *key = crypto_skcipher_ctx(tfm);
+
+ return aes_preparekey(key, in_key, key_len);
+}
+
+static __maybe_unused int
+crypto_aes_skcipher_setenckey(struct crypto_skcipher *tfm, const u8 *in_key,
+ unsigned int key_len)
+{
+ struct aes_enckey *key = crypto_skcipher_ctx(tfm);
+
+ return aes_prepareenckey(key, in_key, key_len);
+}
+
+/*
+ * Call crypt_func() (a function that operates on simple virtual addresses) zero
+ * or more times to en/decrypt 'cryptlen' bytes of data from the source
+ * scatterlist 'src' and write it into the destination scatterlist 'dst',
+ * starting at 'start_pos' bytes into both.
+ *
+ * This always calls crypt_func() with a length that's a multiple of
+ * AES_BLOCK_SIZE, except the last call which includes any remainder. This is
+ * implemented by using an on-stack bounce buffer when necessary. The current
+ * implementation also tries to prefer passing at least 4 blocks, so e.g.
+ * scatterlist entries [16,16,16,16] result in a single 64-byte call.
+ *
+ * The scatterlists must describe either entirely different memory
+ * (out-of-place) or entirely the same memory (in-place). In the latter case,
+ * crypt_func() is always called with the source and dest pointers the same.
+ */
+#define AES_CRYPT_SG(crypt_func, dst, src, cryptlen, start_pos, ...) \
+ ({ \
+ unsigned int remaining = (cryptlen); \
+ \
+ if ((remaining) != 0) { \
+ struct scatter_walk src_walk, dst_walk; \
+ u8 tmp[4 * AES_BLOCK_SIZE] __aligned( \
+ __alignof__(long)); \
+ \
+ scatterwalk_start_at_pos(&src_walk, (src), \
+ (start_pos)); \
+ scatterwalk_start_at_pos(&dst_walk, (dst), \
+ (start_pos)); \
+ do { \
+ unsigned int src_avail = scatterwalk_clamp( \
+ &src_walk, (remaining)); \
+ unsigned int dst_avail = scatterwalk_clamp( \
+ &dst_walk, (remaining)); \
+ unsigned int n = min(src_avail, dst_avail); \
+ const u8 *src_virt; \
+ u8 *dst_virt; \
+ \
+ if (n < (remaining)) { \
+ if (n < sizeof(tmp)) { \
+ n = min((remaining), \
+ sizeof(tmp)); \
+ memcpy_from_scatterwalk( \
+ tmp, &src_walk, n); \
+ crypt_func(tmp, tmp, n, \
+ ##__VA_ARGS__); \
+ memcpy_to_scatterwalk( \
+ &dst_walk, tmp, n); \
+ (remaining) -= n; \
+ continue; \
+ } \
+ n = round_down(n, AES_BLOCK_SIZE); \
+ } \
+ \
+ scatterwalk_map(&dst_walk); \
+ dst_virt = dst_walk.addr; \
+ if (IS_ENABLED(CONFIG_HIGHMEM) && \
+ offset_in_page(src_walk.offset) == \
+ offset_in_page(dst_walk.offset) && \
+ sg_page(src_walk.sg) + (src_walk.offset / \
+ PAGE_SIZE) == \
+ sg_page(dst_walk.sg) + \
+ (dst_walk.offset / \
+ PAGE_SIZE)) { \
+ src_virt = dst_virt; \
+ } else { \
+ scatterwalk_map(&src_walk); \
+ src_virt = src_walk.addr; \
+ } \
+ crypt_func(dst_virt, src_virt, n, \
+ ##__VA_ARGS__); \
+ if (src_virt != dst_virt) \
+ scatterwalk_unmap(&src_walk); \
+ scatterwalk_advance(&src_walk, n); \
+ scatterwalk_done_dst(&dst_walk, n); \
+ (remaining) -= n; \
+ } while (remaining); \
+ memzero_explicit(tmp, sizeof(tmp)); \
+ } \
+ })
+
+/* AES-ECB */
+
+static __maybe_unused int crypto_aes_ecb_encrypt(struct skcipher_request *req)
+{
+ const struct aes_key *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+
+ if (unlikely(req->cryptlen % AES_BLOCK_SIZE))
+ return -EINVAL;
+ AES_CRYPT_SG(aes_ecb_encrypt, req->dst, req->src, req->cryptlen, 0,
+ key);
+ return 0;
+}
+
+static __maybe_unused int crypto_aes_ecb_decrypt(struct skcipher_request *req)
+{
+ const struct aes_key *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+
+ if (unlikely(req->cryptlen % AES_BLOCK_SIZE))
+ return -EINVAL;
+ AES_CRYPT_SG(aes_ecb_decrypt, req->dst, req->src, req->cryptlen, 0,
+ key);
+ return 0;
+}
+
+static struct skcipher_alg skcipher_algs[] = {
+#if IS_ENABLED(CONFIG_CRYPTO_ECB)
+ {
+ .base.cra_name = "ecb(aes)",
+ .base.cra_driver_name = "ecb-aes-lib",
+ .base.cra_priority = 110,
+ .base.cra_blocksize = AES_BLOCK_SIZE,
+ .base.cra_ctxsize = sizeof(struct aes_key),
+ .base.cra_module = THIS_MODULE,
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .setkey = crypto_aes_skcipher_setkey,
+ .encrypt = crypto_aes_ecb_encrypt,
+ .decrypt = crypto_aes_ecb_decrypt,
+ },
+#endif
+};
+
static int __init crypto_aes_mod_init(void)
{
int err = crypto_register_alg(&alg);
@@ -212,8 +358,18 @@ static int __init crypto_aes_mod_init(void)
if (err)
goto err_unregister_alg;
} /* Else, CONFIG_CRYPTO_HASH might not be enabled. */
+
+ if (ARRAY_SIZE(skcipher_algs) > 0) {
+ err = crypto_register_skciphers(skcipher_algs,
+ ARRAY_SIZE(skcipher_algs));
+ if (err)
+ goto err_unregister_macs;
+ }
return 0;
+err_unregister_macs:
+ if (ARRAY_SIZE(mac_algs) > 0)
+ crypto_unregister_shashes(mac_algs, ARRAY_SIZE(mac_algs));
err_unregister_alg:
crypto_unregister_alg(&alg);
return err;
@@ -222,6 +378,9 @@ module_init(crypto_aes_mod_init);
static void __exit crypto_aes_mod_exit(void)
{
+ if (ARRAY_SIZE(skcipher_algs) > 0)
+ crypto_unregister_skciphers(skcipher_algs,
+ ARRAY_SIZE(skcipher_algs));
if (ARRAY_SIZE(mac_algs) > 0)
crypto_unregister_shashes(mac_algs, ARRAY_SIZE(mac_algs));
crypto_unregister_alg(&alg);
@@ -245,3 +404,7 @@ MODULE_ALIAS_CRYPTO("xcbc-aes-lib");
MODULE_ALIAS_CRYPTO("cbcmac(aes)");
MODULE_ALIAS_CRYPTO("cbcmac-aes-lib");
#endif
+#if IS_ENABLED(CONFIG_CRYPTO_ECB)
+MODULE_ALIAS_CRYPTO("ecb(aes)");
+MODULE_ALIAS_CRYPTO("ecb-aes-lib");
+#endif
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 09/33] crypto: aes - Add CBC and CBC-CTS support using library
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (7 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 08/33] crypto: aes - Add ECB support using library Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 10/33] crypto: aes - Add CTR and XCTR " Eric Biggers
` (23 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Implement the "cbc(aes)" and "cts(cbc(aes))" crypto_skcipher algorithms
using the corresponding library functions.
Among other benefits, this allows the architecture-optimized AES-CBC and
AES-CBC-CTS code to be migrated into the library while still leaving it
accessible via crypto_skcipher, eliminating lots of boilerplate code.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of these algorithms haven't yet
been migrated into the library. It will be boosted once that happens.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
crypto/Kconfig | 1 +
crypto/aes.c | 163 +++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 164 insertions(+)
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 093508d13b8c..3d30d79878c2 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -358,6 +358,7 @@ config CRYPTO_AES
tristate "AES (Advanced Encryption Standard)"
select CRYPTO_ALGAPI
select CRYPTO_LIB_AES
+ select CRYPTO_LIB_AES_CBC if CRYPTO_CBC || CRYPTO_CTS
select CRYPTO_LIB_AES_CBC_MACS if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
select CRYPTO_LIB_AES_ECB if CRYPTO_ECB
select CRYPTO_HASH if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
diff --git a/crypto/aes.c b/crypto/aes.c
index 162715a82be3..5999f8117ce7 100644
--- a/crypto/aes.c
+++ b/crypto/aes.c
@@ -6,6 +6,7 @@
*/
#include <crypto/aes-cbc-macs.h>
+#include <crypto/aes-cbc.h>
#include <crypto/aes-ecb.h>
#include <crypto/aes.h>
#include <crypto/algapi.h>
@@ -328,6 +329,128 @@ static __maybe_unused int crypto_aes_ecb_decrypt(struct skcipher_request *req)
return 0;
}
+/* AES-CBC */
+
+static void crypto_aes_cbc_encrypt_sg(struct scatterlist *dst,
+ struct scatterlist *src,
+ unsigned int cryptlen,
+ u8 iv[AES_BLOCK_SIZE],
+ const struct aes_key *key)
+{
+ AES_CRYPT_SG(aes_cbc_encrypt, dst, src, cryptlen, 0, iv, key);
+}
+
+static void crypto_aes_cbc_decrypt_sg(struct scatterlist *dst,
+ struct scatterlist *src,
+ unsigned int cryptlen,
+ u8 iv[AES_BLOCK_SIZE],
+ const struct aes_key *key)
+{
+ AES_CRYPT_SG(aes_cbc_decrypt, dst, src, cryptlen, 0, iv, key);
+}
+
+static __maybe_unused int crypto_aes_cbc_encrypt(struct skcipher_request *req)
+{
+ const struct aes_key *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+
+ if (unlikely(req->cryptlen % AES_BLOCK_SIZE))
+ return -EINVAL;
+ crypto_aes_cbc_encrypt_sg(req->dst, req->src, req->cryptlen, req->iv,
+ key);
+ return 0;
+}
+
+static __maybe_unused int crypto_aes_cbc_decrypt(struct skcipher_request *req)
+{
+ const struct aes_key *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+
+ if (unlikely(req->cryptlen % AES_BLOCK_SIZE))
+ return -EINVAL;
+ crypto_aes_cbc_decrypt_sg(req->dst, req->src, req->cryptlen, req->iv,
+ key);
+ return 0;
+}
+
+/* AES-CBC-CTS */
+
+static noinline int
+crypto_aes_cbc_cts_crypt_nonlinear(struct skcipher_request *req, bool enc)
+{
+ const struct aes_key *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+ unsigned int main_len = req->cryptlen;
+ unsigned int tail_len =
+ ((main_len - 1) % AES_BLOCK_SIZE) + 1 + AES_BLOCK_SIZE;
+ u8 tmp[2 * AES_BLOCK_SIZE] __aligned(__alignof__(long));
+
+ if (main_len == AES_BLOCK_SIZE) {
+ /* Single block is a special case that just does CBC. */
+ if (enc)
+ crypto_aes_cbc_encrypt_sg(req->dst, req->src, main_len,
+ req->iv, key);
+ else
+ crypto_aes_cbc_decrypt_sg(req->dst, req->src, main_len,
+ req->iv, key);
+ return 0;
+ }
+ /* Just do the last two blocks separately. */
+ main_len -= tail_len;
+ if (enc)
+ crypto_aes_cbc_encrypt_sg(req->dst, req->src, main_len, req->iv,
+ key);
+ else
+ crypto_aes_cbc_decrypt_sg(req->dst, req->src, main_len, req->iv,
+ key);
+ memcpy_from_sglist(tmp, req->src, main_len, tail_len);
+ if (enc)
+ aes_cbc_cts_encrypt(tmp, tmp, tail_len, req->iv, key);
+ else
+ aes_cbc_cts_decrypt(tmp, tmp, tail_len, req->iv, key);
+ memcpy_to_sglist(req->dst, main_len, tmp, tail_len);
+ memzero_explicit(tmp, sizeof(tmp));
+ return 0;
+}
+
+static __maybe_unused int
+crypto_aes_cbc_cts_encrypt(struct skcipher_request *req)
+{
+ const struct aes_key *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+
+ if (unlikely(req->cryptlen < AES_BLOCK_SIZE))
+ return -EINVAL;
+ if (!IS_ENABLED(CONFIG_HIGHMEM) &&
+ likely(req->src->length >= req->cryptlen &&
+ req->dst->length >= req->cryptlen)) {
+ /* Fast path */
+ aes_cbc_cts_encrypt(sg_virt(req->dst), sg_virt(req->src),
+ req->cryptlen, req->iv, key);
+ return 0;
+ }
+ return crypto_aes_cbc_cts_crypt_nonlinear(req, /* enc= */ true);
+}
+
+static __maybe_unused int
+crypto_aes_cbc_cts_decrypt(struct skcipher_request *req)
+{
+ const struct aes_key *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+
+ if (unlikely(req->cryptlen < AES_BLOCK_SIZE))
+ return -EINVAL;
+ if (!IS_ENABLED(CONFIG_HIGHMEM) &&
+ likely(req->src->length >= req->cryptlen &&
+ req->dst->length >= req->cryptlen)) {
+ /* Fast path */
+ aes_cbc_cts_decrypt(sg_virt(req->dst), sg_virt(req->src),
+ req->cryptlen, req->iv, key);
+ return 0;
+ }
+ return crypto_aes_cbc_cts_crypt_nonlinear(req, /* enc= */ false);
+}
+
static struct skcipher_alg skcipher_algs[] = {
#if IS_ENABLED(CONFIG_CRYPTO_ECB)
{
@@ -344,6 +467,38 @@ static struct skcipher_alg skcipher_algs[] = {
.decrypt = crypto_aes_ecb_decrypt,
},
#endif
+#if IS_ENABLED(CONFIG_CRYPTO_CBC)
+ {
+ .base.cra_name = "cbc(aes)",
+ .base.cra_driver_name = "cbc-aes-lib",
+ .base.cra_priority = 110,
+ .base.cra_blocksize = AES_BLOCK_SIZE,
+ .base.cra_ctxsize = sizeof(struct aes_key),
+ .base.cra_module = THIS_MODULE,
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = crypto_aes_skcipher_setkey,
+ .encrypt = crypto_aes_cbc_encrypt,
+ .decrypt = crypto_aes_cbc_decrypt,
+ },
+#endif
+#if IS_ENABLED(CONFIG_CRYPTO_CTS)
+ {
+ .base.cra_name = "cts(cbc(aes))",
+ .base.cra_driver_name = "cts-cbc-aes-lib",
+ .base.cra_priority = 110,
+ .base.cra_blocksize = AES_BLOCK_SIZE,
+ .base.cra_ctxsize = sizeof(struct aes_key),
+ .base.cra_module = THIS_MODULE,
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = crypto_aes_skcipher_setkey,
+ .encrypt = crypto_aes_cbc_cts_encrypt,
+ .decrypt = crypto_aes_cbc_cts_decrypt,
+ },
+#endif
};
static int __init crypto_aes_mod_init(void)
@@ -408,3 +563,11 @@ MODULE_ALIAS_CRYPTO("cbcmac-aes-lib");
MODULE_ALIAS_CRYPTO("ecb(aes)");
MODULE_ALIAS_CRYPTO("ecb-aes-lib");
#endif
+#if IS_ENABLED(CONFIG_CRYPTO_CBC)
+MODULE_ALIAS_CRYPTO("cbc(aes)");
+MODULE_ALIAS_CRYPTO("cbc-aes-lib");
+#endif
+#if IS_ENABLED(CONFIG_CRYPTO_CTS)
+MODULE_ALIAS_CRYPTO("cts(cbc(aes))");
+MODULE_ALIAS_CRYPTO("cts-cbc-aes-lib");
+#endif
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 10/33] crypto: aes - Add CTR and XCTR support using library
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (8 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 09/33] crypto: aes - Add CBC and CBC-CTS " Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 11/33] crypto: aes - Add XTS " Eric Biggers
` (22 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Implement the "ctr(aes)" and "xctr(aes)" crypto_skcipher algorithms
using the corresponding library functions.
Among other benefits, this allows the architecture-optimized AES-CTR and
AES-XCTR code to be migrated into the library while still leaving it
accessible via crypto_skcipher, eliminating lots of boilerplate code.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of these algorithms haven't yet
been migrated into the library. It will be boosted once that happens.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
crypto/Kconfig | 1 +
crypto/aes.c | 68 ++++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 69 insertions(+)
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 3d30d79878c2..dd6785c68620 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -360,6 +360,7 @@ config CRYPTO_AES
select CRYPTO_LIB_AES
select CRYPTO_LIB_AES_CBC if CRYPTO_CBC || CRYPTO_CTS
select CRYPTO_LIB_AES_CBC_MACS if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
+ select CRYPTO_LIB_AES_CTR if CRYPTO_CTR || CRYPTO_XCTR
select CRYPTO_LIB_AES_ECB if CRYPTO_ECB
select CRYPTO_HASH if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
# CRYPTO_SKCIPHER should be selected only if a mode that needs it is
diff --git a/crypto/aes.c b/crypto/aes.c
index 5999f8117ce7..0919818bed03 100644
--- a/crypto/aes.c
+++ b/crypto/aes.c
@@ -7,6 +7,7 @@
#include <crypto/aes-cbc-macs.h>
#include <crypto/aes-cbc.h>
+#include <crypto/aes-ctr.h>
#include <crypto/aes-ecb.h>
#include <crypto/aes.h>
#include <crypto/algapi.h>
@@ -451,6 +452,31 @@ crypto_aes_cbc_cts_decrypt(struct skcipher_request *req)
return crypto_aes_cbc_cts_crypt_nonlinear(req, /* enc= */ false);
}
+/* AES-CTR */
+
+static __maybe_unused int crypto_aes_ctr_crypt(struct skcipher_request *req)
+{
+ const struct aes_enckey *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+
+ AES_CRYPT_SG(aes_ctr, req->dst, req->src, req->cryptlen, 0, req->iv,
+ key);
+ return 0;
+}
+
+/* AES-XCTR */
+
+static __maybe_unused int crypto_aes_xctr_crypt(struct skcipher_request *req)
+{
+ const struct aes_enckey *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+ u64 ctr = 1;
+
+ AES_CRYPT_SG(aes_xctr, req->dst, req->src, req->cryptlen, 0, &ctr,
+ req->iv, key);
+ return 0;
+}
+
static struct skcipher_alg skcipher_algs[] = {
#if IS_ENABLED(CONFIG_CRYPTO_ECB)
{
@@ -499,6 +525,40 @@ static struct skcipher_alg skcipher_algs[] = {
.decrypt = crypto_aes_cbc_cts_decrypt,
},
#endif
+#if IS_ENABLED(CONFIG_CRYPTO_CTR)
+ {
+ .base.cra_name = "ctr(aes)",
+ .base.cra_driver_name = "ctr-aes-lib",
+ .base.cra_priority = 110,
+ .base.cra_blocksize = 1,
+ .base.cra_ctxsize = sizeof(struct aes_enckey),
+ .base.cra_module = THIS_MODULE,
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .chunksize = AES_BLOCK_SIZE,
+ .setkey = crypto_aes_skcipher_setenckey,
+ .encrypt = crypto_aes_ctr_crypt,
+ .decrypt = crypto_aes_ctr_crypt,
+ },
+#endif
+#if IS_ENABLED(CONFIG_CRYPTO_XCTR)
+ {
+ .base.cra_name = "xctr(aes)",
+ .base.cra_driver_name = "xctr-aes-lib",
+ .base.cra_priority = 110,
+ .base.cra_blocksize = 1,
+ .base.cra_ctxsize = sizeof(struct aes_enckey),
+ .base.cra_module = THIS_MODULE,
+ .min_keysize = AES_MIN_KEY_SIZE,
+ .max_keysize = AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .chunksize = AES_BLOCK_SIZE,
+ .setkey = crypto_aes_skcipher_setenckey,
+ .encrypt = crypto_aes_xctr_crypt,
+ .decrypt = crypto_aes_xctr_crypt,
+ },
+#endif
};
static int __init crypto_aes_mod_init(void)
@@ -571,3 +631,11 @@ MODULE_ALIAS_CRYPTO("cbc-aes-lib");
MODULE_ALIAS_CRYPTO("cts(cbc(aes))");
MODULE_ALIAS_CRYPTO("cts-cbc-aes-lib");
#endif
+#if IS_ENABLED(CONFIG_CRYPTO_CTR)
+MODULE_ALIAS_CRYPTO("ctr(aes)");
+MODULE_ALIAS_CRYPTO("ctr-aes-lib");
+#endif
+#if IS_ENABLED(CONFIG_CRYPTO_XCTR)
+MODULE_ALIAS_CRYPTO("xctr(aes)");
+MODULE_ALIAS_CRYPTO("xctr-aes-lib");
+#endif
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 11/33] crypto: aes - Add XTS support using library
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (9 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 10/33] crypto: aes - Add CTR and XCTR " Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 12/33] crypto: aes - Add GCM " Eric Biggers
` (21 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Implement the "xts(aes)" crypto_skcipher algorithm using the
corresponding library functions.
Among other benefits, this allows the architecture-optimized AES-XTS
code to be migrated into the library while still leaving it accessible
via crypto_skcipher, eliminating lots of boilerplate code.
Fast paths similar to what x86_64 uses (to eliminate the scatterlist
walking overhead) are included. So we'll get that optimization for all
architectures.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of this algorithm haven't yet
been migrated into the library. It will be boosted once that happens.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
crypto/Kconfig | 1 +
crypto/aes.c | 120 +++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 121 insertions(+)
diff --git a/crypto/Kconfig b/crypto/Kconfig
index dd6785c68620..5e573051c903 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -362,6 +362,7 @@ config CRYPTO_AES
select CRYPTO_LIB_AES_CBC_MACS if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
select CRYPTO_LIB_AES_CTR if CRYPTO_CTR || CRYPTO_XCTR
select CRYPTO_LIB_AES_ECB if CRYPTO_ECB
+ select CRYPTO_LIB_AES_XTS if CRYPTO_XTS
select CRYPTO_HASH if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
# CRYPTO_SKCIPHER should be selected only if a mode that needs it is
# enabled, but that doesn't work due to a recursive dependency caused by
diff --git a/crypto/aes.c b/crypto/aes.c
index 0919818bed03..a59eb57c86de 100644
--- a/crypto/aes.c
+++ b/crypto/aes.c
@@ -9,6 +9,7 @@
#include <crypto/aes-cbc.h>
#include <crypto/aes-ctr.h>
#include <crypto/aes-ecb.h>
+#include <crypto/aes-xts.h>
#include <crypto/aes.h>
#include <crypto/algapi.h>
#include <crypto/internal/hash.h>
@@ -477,6 +478,105 @@ static __maybe_unused int crypto_aes_xctr_crypt(struct skcipher_request *req)
return 0;
}
+/* AES-XTS */
+
+static __maybe_unused int crypto_aes_xts_setkey(struct crypto_skcipher *tfm,
+ const u8 *in_key,
+ unsigned int key_len)
+{
+ struct aes_xts_key *key = crypto_skcipher_ctx(tfm);
+ int flags = (crypto_skcipher_get_flags(tfm) &
+ CRYPTO_TFM_REQ_FORBID_WEAK_KEYS) ?
+ XTS_FORBID_WEAK_KEYS :
+ 0;
+
+ return aes_xts_preparekey(key, in_key, key_len, flags);
+}
+
+static void aes_xts_crypt_wrapper(u8 *dst, const u8 *src, size_t len,
+ u8 iv[AES_BLOCK_SIZE],
+ const struct aes_xts_key *key, bool enc,
+ bool *cont)
+{
+ if (enc)
+ aes_xts_encrypt(dst, src, len, iv, key, *cont);
+ else
+ aes_xts_decrypt(dst, src, len, iv, key, *cont);
+ *cont = true;
+}
+
+/*
+ * This handles AES-XTS en/decryption requests that use a nonlinear scattelist
+ * layout or where HIGHMEM is enabled.
+ */
+static noinline int crypto_aes_xts_crypt_nonlinear(struct skcipher_request *req,
+ bool enc)
+{
+ const struct aes_xts_key *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+ u8 tmp[2 * AES_BLOCK_SIZE] __aligned(__alignof__(long));
+ unsigned int main_len = req->cryptlen;
+ unsigned int tail_len = main_len % AES_BLOCK_SIZE;
+ bool cont = false;
+
+ if (unlikely(tail_len)) {
+ /*
+ * Ciphertext stealing is needed.
+ * Just do the last two blocks separately.
+ */
+ tail_len += AES_BLOCK_SIZE;
+ main_len -= tail_len;
+ }
+
+ AES_CRYPT_SG(aes_xts_crypt_wrapper, req->dst, req->src, main_len, 0,
+ req->iv, key, enc, &cont);
+
+ if (unlikely(tail_len)) {
+ memcpy_from_sglist(tmp, req->src, main_len, tail_len);
+ aes_xts_crypt_wrapper(tmp, tmp, tail_len, req->iv, key, enc,
+ &cont);
+ memcpy_to_sglist(req->dst, main_len, tmp, tail_len);
+ memzero_explicit(tmp, sizeof(tmp));
+ }
+ return 0;
+}
+
+static __maybe_unused int crypto_aes_xts_encrypt(struct skcipher_request *req)
+{
+ const struct aes_xts_key *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+
+ if (unlikely(req->cryptlen < AES_BLOCK_SIZE))
+ return -EINVAL;
+ if (!IS_ENABLED(CONFIG_HIGHMEM) &&
+ likely(req->src->length >= req->cryptlen &&
+ req->dst->length >= req->cryptlen)) {
+ /* Fast path */
+ aes_xts_encrypt(sg_virt(req->dst), sg_virt(req->src),
+ req->cryptlen, req->iv, key, /* cont= */ false);
+ return 0;
+ }
+ return crypto_aes_xts_crypt_nonlinear(req, /* enc= */ true);
+}
+
+static __maybe_unused int crypto_aes_xts_decrypt(struct skcipher_request *req)
+{
+ const struct aes_xts_key *key =
+ crypto_skcipher_ctx(crypto_skcipher_reqtfm(req));
+
+ if (unlikely(req->cryptlen < AES_BLOCK_SIZE))
+ return -EINVAL;
+ if (!IS_ENABLED(CONFIG_HIGHMEM) &&
+ likely(req->src->length >= req->cryptlen &&
+ req->dst->length >= req->cryptlen)) {
+ /* Fast path */
+ aes_xts_decrypt(sg_virt(req->dst), sg_virt(req->src),
+ req->cryptlen, req->iv, key, /* cont= */ false);
+ return 0;
+ }
+ return crypto_aes_xts_crypt_nonlinear(req, /* enc= */ false);
+}
+
static struct skcipher_alg skcipher_algs[] = {
#if IS_ENABLED(CONFIG_CRYPTO_ECB)
{
@@ -559,6 +659,22 @@ static struct skcipher_alg skcipher_algs[] = {
.decrypt = crypto_aes_xctr_crypt,
},
#endif
+#if IS_ENABLED(CONFIG_CRYPTO_XTS)
+ {
+ .base.cra_name = "xts(aes)",
+ .base.cra_driver_name = "xts-aes-lib",
+ .base.cra_priority = 110,
+ .base.cra_blocksize = AES_BLOCK_SIZE,
+ .base.cra_ctxsize = sizeof(struct aes_xts_key),
+ .base.cra_module = THIS_MODULE,
+ .min_keysize = 2 * AES_MIN_KEY_SIZE,
+ .max_keysize = 2 * AES_MAX_KEY_SIZE,
+ .ivsize = AES_BLOCK_SIZE,
+ .setkey = crypto_aes_xts_setkey,
+ .encrypt = crypto_aes_xts_encrypt,
+ .decrypt = crypto_aes_xts_decrypt,
+ },
+#endif
};
static int __init crypto_aes_mod_init(void)
@@ -639,3 +755,7 @@ MODULE_ALIAS_CRYPTO("ctr-aes-lib");
MODULE_ALIAS_CRYPTO("xctr(aes)");
MODULE_ALIAS_CRYPTO("xctr-aes-lib");
#endif
+#if IS_ENABLED(CONFIG_CRYPTO_XTS)
+MODULE_ALIAS_CRYPTO("xts(aes)");
+MODULE_ALIAS_CRYPTO("xts-aes-lib");
+#endif
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 12/33] crypto: aes - Add GCM support using library
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (10 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 11/33] crypto: aes - Add XTS " Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 13/33] crypto: aes - Add CCM " Eric Biggers
` (20 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Implement the "gcm(aes)" and "rfc4106(gcm(aes))" crypto_aead algorithms
using the corresponding library functions.
Among other benefits, this allows the architecture-optimized AES-GCM
code to be migrated into the library while still leaving it accessible
via crypto_aead, eliminating lots of boilerplate code.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of these algorithms haven't yet
been migrated into the library. It will be boosted once that happens.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
crypto/Kconfig | 2 +
crypto/aes.c | 237 +++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 239 insertions(+)
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 5e573051c903..74dfe969216d 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -362,7 +362,9 @@ config CRYPTO_AES
select CRYPTO_LIB_AES_CBC_MACS if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
select CRYPTO_LIB_AES_CTR if CRYPTO_CTR || CRYPTO_XCTR
select CRYPTO_LIB_AES_ECB if CRYPTO_ECB
+ select CRYPTO_LIB_AES_GCM if CRYPTO_GCM
select CRYPTO_LIB_AES_XTS if CRYPTO_XTS
+ select CRYPTO_AEAD if CRYPTO_GCM
select CRYPTO_HASH if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
# CRYPTO_SKCIPHER should be selected only if a mode that needs it is
# enabled, but that doesn't work due to a recursive dependency caused by
diff --git a/crypto/aes.c b/crypto/aes.c
index a59eb57c86de..621ceed3d587 100644
--- a/crypto/aes.c
+++ b/crypto/aes.c
@@ -9,9 +9,11 @@
#include <crypto/aes-cbc.h>
#include <crypto/aes-ctr.h>
#include <crypto/aes-ecb.h>
+#include <crypto/aes-gcm.h>
#include <crypto/aes-xts.h>
#include <crypto/aes.h>
#include <crypto/algapi.h>
+#include <crypto/internal/aead.h>
#include <crypto/internal/hash.h>
#include <crypto/internal/skcipher.h>
#include <crypto/scatterwalk.h>
@@ -305,6 +307,29 @@ crypto_aes_skcipher_setenckey(struct crypto_skcipher *tfm, const u8 *in_key,
} \
})
+/*
+ * Call ad_func() as needed to process the associated data in the first
+ * 'assoclen' bytes of the scatterlist 'src'.
+ */
+#define AES_PROCESS_ASSOC_DATA(ad_func, src, assoclen, ctx) \
+ ({ \
+ unsigned int remaining = (assoclen); \
+ \
+ if (remaining != 0) { \
+ struct scatter_walk walk; \
+ \
+ scatterwalk_start(&walk, (src)); \
+ do { \
+ unsigned int n = \
+ scatterwalk_next(&walk, remaining); \
+ \
+ ad_func((ctx), walk.addr, n); \
+ scatterwalk_done_src(&walk, n); \
+ remaining -= n; \
+ } while (remaining); \
+ } \
+ })
+
/* AES-ECB */
static __maybe_unused int crypto_aes_ecb_encrypt(struct skcipher_request *req)
@@ -677,6 +702,200 @@ static struct skcipher_alg skcipher_algs[] = {
#endif
};
+/* AES-GCM */
+
+static __maybe_unused int crypto_aes_gcm_setkey(struct crypto_aead *tfm,
+ const u8 *in_key,
+ unsigned int key_len)
+{
+ struct aes_gcm_key *key = crypto_aead_ctx(tfm);
+
+ return aes_gcm_preparekey(key, in_key, key_len,
+ crypto_aead_authsize(tfm));
+}
+
+static __maybe_unused int crypto_aes_gcm_setauthsize(struct crypto_aead *tfm,
+ unsigned int authsize)
+{
+ struct aes_gcm_key *key = crypto_aead_ctx(tfm);
+
+ if (crypto_gcm_check_authsize(authsize) != 0)
+ return -EINVAL;
+ /* Synchronize the tag length to the struct aes_gcm_key. */
+ key->authtag_len = authsize;
+ return 0;
+}
+
+static void aes_gcm_encrypt_update_helper(u8 *dst, const u8 *src,
+ unsigned int len,
+ struct aes_gcm_ctx *ctx)
+{
+ aes_gcm_encrypt_update(ctx, dst, src, len);
+}
+
+static void aes_gcm_decrypt_update_helper(u8 *dst, const u8 *src,
+ unsigned int len,
+ struct aes_gcm_ctx *ctx)
+{
+ aes_gcm_decrypt_update(ctx, dst, src, len);
+}
+
+static int crypto_aes_gcm_encrypt_common(struct aead_request *req,
+ const struct aes_gcm_key *key,
+ u8 iv[12], unsigned int assoclen)
+{
+ struct aes_gcm_ctx ctx;
+ u8 authtag[16];
+
+ aes_gcm_init(&ctx, iv, key);
+ AES_PROCESS_ASSOC_DATA(aes_gcm_auth_update, req->src, assoclen, &ctx);
+ AES_CRYPT_SG(aes_gcm_encrypt_update_helper, req->dst, req->src,
+ req->cryptlen, req->assoclen, &ctx);
+ aes_gcm_encrypt_final(&ctx, authtag);
+ memcpy_to_sglist(req->dst, req->assoclen + req->cryptlen, authtag,
+ key->authtag_len);
+ memzero_explicit(authtag, sizeof(authtag));
+ return 0;
+}
+
+static int crypto_aes_gcm_decrypt_common(struct aead_request *req,
+ const struct aes_gcm_key *key,
+ u8 iv[12], unsigned int assoclen)
+{
+ struct aes_gcm_ctx ctx;
+ unsigned int data_len;
+ u8 authtag[16];
+ int err;
+
+ /* crypto_aead_decrypt() already checked cryptlen >= authsize. */
+ data_len = req->cryptlen - key->authtag_len;
+
+ aes_gcm_init(&ctx, iv, key);
+ AES_PROCESS_ASSOC_DATA(aes_gcm_auth_update, req->src, assoclen, &ctx);
+ AES_CRYPT_SG(aes_gcm_decrypt_update_helper, req->dst, req->src,
+ data_len, req->assoclen, &ctx);
+ memcpy_from_sglist(authtag, req->src, req->assoclen + data_len,
+ key->authtag_len);
+ err = aes_gcm_decrypt_final(&ctx, authtag);
+ memzero_explicit(authtag, sizeof(authtag));
+ return err;
+}
+
+static __maybe_unused int crypto_aes_gcm_encrypt(struct aead_request *req)
+{
+ struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ const struct aes_gcm_key *key = crypto_aead_ctx(tfm);
+
+ return crypto_aes_gcm_encrypt_common(req, key, req->iv, req->assoclen);
+}
+
+static __maybe_unused int crypto_aes_gcm_decrypt(struct aead_request *req)
+{
+ struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ const struct aes_gcm_key *key = crypto_aead_ctx(tfm);
+
+ return crypto_aes_gcm_decrypt_common(req, key, req->iv, req->assoclen);
+}
+
+struct aes_rfc4106_key {
+ struct aes_gcm_key gcm;
+ u8 nonce[4];
+};
+
+static __maybe_unused int crypto_aes_rfc4106_setkey(struct crypto_aead *tfm,
+ const u8 *in_key,
+ unsigned int key_len)
+{
+ struct aes_rfc4106_key *key = crypto_aead_ctx(tfm);
+
+ if (key_len < 4)
+ return -EINVAL;
+
+ key_len -= 4;
+ memcpy(key->nonce, in_key + key_len, 4);
+
+ return aes_gcm_preparekey(&key->gcm, in_key, key_len,
+ crypto_aead_authsize(tfm));
+}
+
+static __maybe_unused int
+crypto_aes_rfc4106_setauthsize(struct crypto_aead *tfm, unsigned int authsize)
+{
+ struct aes_rfc4106_key *key = crypto_aead_ctx(tfm);
+
+ if (crypto_rfc4106_check_authsize(authsize) != 0)
+ return -EINVAL;
+
+ /* Synchronize the tag length to the struct aes_gcm_key. */
+ key->gcm.authtag_len = authsize;
+ return 0;
+}
+
+static __maybe_unused int crypto_aes_rfc4106_encrypt(struct aead_request *req)
+{
+ struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ const struct aes_rfc4106_key *key = crypto_aead_ctx(tfm);
+ u8 iv[12];
+
+ if (crypto_ipsec_check_assoclen(req->assoclen) != 0)
+ return -EINVAL;
+ memcpy(iv, key->nonce, 4);
+ memcpy(&iv[4], req->iv, 8);
+
+ return crypto_aes_gcm_encrypt_common(req, &key->gcm, iv,
+ req->assoclen - 8);
+}
+
+static __maybe_unused int crypto_aes_rfc4106_decrypt(struct aead_request *req)
+{
+ struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ const struct aes_rfc4106_key *key = crypto_aead_ctx(tfm);
+ u8 iv[12];
+
+ if (crypto_ipsec_check_assoclen(req->assoclen) != 0)
+ return -EINVAL;
+ memcpy(iv, key->nonce, 4);
+ memcpy(&iv[4], req->iv, 8);
+
+ return crypto_aes_gcm_decrypt_common(req, &key->gcm, iv,
+ req->assoclen - 8);
+}
+
+static struct aead_alg aead_algs[] = {
+#if IS_ENABLED(CONFIG_CRYPTO_GCM)
+ {
+ .base.cra_name = "gcm(aes)",
+ .base.cra_driver_name = "gcm-aes-lib",
+ .base.cra_priority = 110,
+ .base.cra_blocksize = 1,
+ .base.cra_ctxsize = sizeof(struct aes_gcm_key),
+ .base.cra_module = THIS_MODULE,
+ .setkey = crypto_aes_gcm_setkey,
+ .setauthsize = crypto_aes_gcm_setauthsize,
+ .encrypt = crypto_aes_gcm_encrypt,
+ .decrypt = crypto_aes_gcm_decrypt,
+ .ivsize = GCM_AES_IV_SIZE,
+ .maxauthsize = AES_BLOCK_SIZE,
+ .chunksize = AES_BLOCK_SIZE,
+ },
+ {
+ .base.cra_name = "rfc4106(gcm(aes))",
+ .base.cra_driver_name = "rfc4106-gcm-aes-lib",
+ .base.cra_priority = 110,
+ .base.cra_blocksize = 1,
+ .base.cra_ctxsize = sizeof(struct aes_rfc4106_key),
+ .base.cra_module = THIS_MODULE,
+ .setkey = crypto_aes_rfc4106_setkey,
+ .setauthsize = crypto_aes_rfc4106_setauthsize,
+ .encrypt = crypto_aes_rfc4106_encrypt,
+ .decrypt = crypto_aes_rfc4106_decrypt,
+ .ivsize = GCM_RFC4106_IV_SIZE,
+ .maxauthsize = AES_BLOCK_SIZE,
+ .chunksize = AES_BLOCK_SIZE,
+ },
+#endif /* CONFIG_CRYPTO_GCM */
+};
+
static int __init crypto_aes_mod_init(void)
{
int err = crypto_register_alg(&alg);
@@ -696,8 +915,18 @@ static int __init crypto_aes_mod_init(void)
if (err)
goto err_unregister_macs;
}
+
+ if (ARRAY_SIZE(aead_algs) > 0) {
+ err = crypto_register_aeads(aead_algs, ARRAY_SIZE(aead_algs));
+ if (err)
+ goto err_unregister_skciphers;
+ } /* Else, CONFIG_CRYPTO_AEAD might not be enabled. */
return 0;
+err_unregister_skciphers:
+ if (ARRAY_SIZE(skcipher_algs) > 0)
+ crypto_unregister_skciphers(skcipher_algs,
+ ARRAY_SIZE(skcipher_algs));
err_unregister_macs:
if (ARRAY_SIZE(mac_algs) > 0)
crypto_unregister_shashes(mac_algs, ARRAY_SIZE(mac_algs));
@@ -709,6 +938,8 @@ module_init(crypto_aes_mod_init);
static void __exit crypto_aes_mod_exit(void)
{
+ if (ARRAY_SIZE(aead_algs) > 0)
+ crypto_unregister_aeads(aead_algs, ARRAY_SIZE(aead_algs));
if (ARRAY_SIZE(skcipher_algs) > 0)
crypto_unregister_skciphers(skcipher_algs,
ARRAY_SIZE(skcipher_algs));
@@ -759,3 +990,9 @@ MODULE_ALIAS_CRYPTO("xctr-aes-lib");
MODULE_ALIAS_CRYPTO("xts(aes)");
MODULE_ALIAS_CRYPTO("xts-aes-lib");
#endif
+#if IS_ENABLED(CONFIG_CRYPTO_GCM)
+MODULE_ALIAS_CRYPTO("gcm(aes)");
+MODULE_ALIAS_CRYPTO("gcm-aes-lib");
+MODULE_ALIAS_CRYPTO("rfc4106(gcm(aes))");
+MODULE_ALIAS_CRYPTO("rfc4106-gcm-aes-lib");
+#endif
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 13/33] crypto: aes - Add CCM support using library
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (11 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 12/33] crypto: aes - Add GCM " Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 14/33] x86/sev: Use new AES-GCM library Eric Biggers
` (19 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Implement the "ccm(aes)" crypto_aead algorithm using the corresponding
library functions.
Among other benefits, this allows the architecture-optimized AES-CCM
code to be migrated into the library while still leaving it accessible
via crypto_aead, eliminating lots of boilerplate code.
For now the cra_priority is set to just 110, since the
architecture-optimized implementations of this algorithm haven't yet
been migrated into the library. It will be boosted once that happens.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
crypto/Kconfig | 3 +-
crypto/aes.c | 138 +++++++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 140 insertions(+), 1 deletion(-)
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 74dfe969216d..e6b894dc784a 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -360,11 +360,12 @@ config CRYPTO_AES
select CRYPTO_LIB_AES
select CRYPTO_LIB_AES_CBC if CRYPTO_CBC || CRYPTO_CTS
select CRYPTO_LIB_AES_CBC_MACS if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
+ select CRYPTO_LIB_AES_CCM if CRYPTO_CCM
select CRYPTO_LIB_AES_CTR if CRYPTO_CTR || CRYPTO_XCTR
select CRYPTO_LIB_AES_ECB if CRYPTO_ECB
select CRYPTO_LIB_AES_GCM if CRYPTO_GCM
select CRYPTO_LIB_AES_XTS if CRYPTO_XTS
- select CRYPTO_AEAD if CRYPTO_GCM
+ select CRYPTO_AEAD if CRYPTO_GCM || CRYPTO_CCM
select CRYPTO_HASH if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
# CRYPTO_SKCIPHER should be selected only if a mode that needs it is
# enabled, but that doesn't work due to a recursive dependency caused by
diff --git a/crypto/aes.c b/crypto/aes.c
index 621ceed3d587..ac5190292b3c 100644
--- a/crypto/aes.c
+++ b/crypto/aes.c
@@ -7,6 +7,7 @@
#include <crypto/aes-cbc-macs.h>
#include <crypto/aes-cbc.h>
+#include <crypto/aes-ccm.h>
#include <crypto/aes-ctr.h>
#include <crypto/aes-ecb.h>
#include <crypto/aes-gcm.h>
@@ -861,6 +862,122 @@ static __maybe_unused int crypto_aes_rfc4106_decrypt(struct aead_request *req)
req->assoclen - 8);
}
+/* AES-CCM */
+
+static __maybe_unused int crypto_aes_ccm_setkey(struct crypto_aead *tfm,
+ const u8 *in_key,
+ unsigned int key_len)
+{
+ struct aes_ccm_key *key = crypto_aead_ctx(tfm);
+
+ return aes_ccm_preparekey(key, in_key, key_len,
+ crypto_aead_authsize(tfm));
+}
+
+static __maybe_unused int crypto_aes_ccm_setauthsize(struct crypto_aead *tfm,
+ unsigned int authsize)
+{
+ struct aes_ccm_key *key = crypto_aead_ctx(tfm);
+
+ if (authsize < 4 || authsize > 16 || authsize % 2)
+ return -EINVAL;
+ /* Synchronize the tag length to the struct aes_ccm_key. */
+ key->authtag_len = authsize;
+ return 0;
+}
+
+static void aes_ccm_encrypt_update_helper(u8 *dst, const u8 *src,
+ unsigned int len,
+ struct aes_ccm_ctx *ctx)
+{
+ aes_ccm_encrypt_update(ctx, dst, src, len);
+}
+
+static void aes_ccm_decrypt_update_helper(u8 *dst, const u8 *src,
+ unsigned int len,
+ struct aes_ccm_ctx *ctx)
+{
+ aes_ccm_decrypt_update(ctx, dst, src, len);
+}
+
+/*
+ * CCM accepts a variable-length nonce between 7 and 13 bytes inclusively, while
+ * crypto_aead assumes a fixed-length nonce. This is worked around by storing
+ * '14 - nonce_len' in the first byte.
+ */
+static inline bool crypto_aes_ccm_extract_nonce(struct aead_request *req,
+ const u8 **nonce_ret,
+ size_t *nonce_len_ret)
+{
+ int nonce_len = 14 - req->iv[0];
+
+ if (unlikely(nonce_len < 7 || nonce_len > 13))
+ return false;
+ *nonce_ret = &req->iv[1];
+ *nonce_len_ret = nonce_len;
+ return true;
+}
+
+static __maybe_unused int crypto_aes_ccm_encrypt(struct aead_request *req)
+{
+ struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ const struct aes_ccm_key *key = crypto_aead_ctx(tfm);
+ struct aes_ccm_ctx ctx;
+ const u8 *nonce;
+ size_t nonce_len;
+ u8 authtag[16];
+ int err;
+
+ if (!crypto_aes_ccm_extract_nonce(req, &nonce, &nonce_len))
+ return -EINVAL;
+
+ err = aes_ccm_init(&ctx, nonce, nonce_len, req->assoclen, req->cryptlen,
+ key);
+ if (err)
+ return err;
+ AES_PROCESS_ASSOC_DATA(aes_ccm_auth_update, req->src, req->assoclen,
+ &ctx);
+ AES_CRYPT_SG(aes_ccm_encrypt_update_helper, req->dst, req->src,
+ req->cryptlen, req->assoclen, &ctx);
+ aes_ccm_encrypt_final(&ctx, authtag);
+ memcpy_to_sglist(req->dst, req->assoclen + req->cryptlen, authtag,
+ key->authtag_len);
+ memzero_explicit(authtag, sizeof(authtag));
+ return 0;
+}
+
+static __maybe_unused int crypto_aes_ccm_decrypt(struct aead_request *req)
+{
+ struct crypto_aead *tfm = crypto_aead_reqtfm(req);
+ const struct aes_ccm_key *key = crypto_aead_ctx(tfm);
+ struct aes_ccm_ctx ctx;
+ const u8 *nonce;
+ size_t nonce_len;
+ unsigned int data_len;
+ u8 authtag[16];
+ int err;
+
+ if (!crypto_aes_ccm_extract_nonce(req, &nonce, &nonce_len))
+ return -EINVAL;
+
+ /* crypto_aead_decrypt() already checked cryptlen >= authsize. */
+ data_len = req->cryptlen - key->authtag_len;
+
+ err = aes_ccm_init(&ctx, nonce, nonce_len, req->assoclen, data_len,
+ key);
+ if (err)
+ return err;
+ AES_PROCESS_ASSOC_DATA(aes_ccm_auth_update, req->src, req->assoclen,
+ &ctx);
+ AES_CRYPT_SG(aes_ccm_decrypt_update_helper, req->dst, req->src,
+ data_len, req->assoclen, &ctx);
+ memcpy_from_sglist(authtag, req->src, req->assoclen + data_len,
+ key->authtag_len);
+ err = aes_ccm_decrypt_final(&ctx, authtag);
+ memzero_explicit(authtag, sizeof(authtag));
+ return err;
+}
+
static struct aead_alg aead_algs[] = {
#if IS_ENABLED(CONFIG_CRYPTO_GCM)
{
@@ -894,6 +1011,23 @@ static struct aead_alg aead_algs[] = {
.chunksize = AES_BLOCK_SIZE,
},
#endif /* CONFIG_CRYPTO_GCM */
+#if IS_ENABLED(CONFIG_CRYPTO_CCM)
+ {
+ .base.cra_name = "ccm(aes)",
+ .base.cra_driver_name = "ccm-aes-lib",
+ .base.cra_priority = 110,
+ .base.cra_blocksize = 1,
+ .base.cra_ctxsize = sizeof(struct aes_ccm_key),
+ .base.cra_module = THIS_MODULE,
+ .setkey = crypto_aes_ccm_setkey,
+ .setauthsize = crypto_aes_ccm_setauthsize,
+ .encrypt = crypto_aes_ccm_encrypt,
+ .decrypt = crypto_aes_ccm_decrypt,
+ .ivsize = 16,
+ .maxauthsize = 16,
+ .chunksize = AES_BLOCK_SIZE,
+ },
+#endif /* CONFIG_CRYPTO_CCM */
};
static int __init crypto_aes_mod_init(void)
@@ -996,3 +1130,7 @@ MODULE_ALIAS_CRYPTO("gcm-aes-lib");
MODULE_ALIAS_CRYPTO("rfc4106(gcm(aes))");
MODULE_ALIAS_CRYPTO("rfc4106-gcm-aes-lib");
#endif
+#if IS_ENABLED(CONFIG_CRYPTO_CCM)
+MODULE_ALIAS_CRYPTO("ccm(aes)");
+MODULE_ALIAS_CRYPTO("ccm-aes-lib");
+#endif
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 14/33] x86/sev: Use new AES-GCM library
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (12 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 13/33] crypto: aes - Add CCM " Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 15/33] lib/crypto: aesgcm: Remove old " Eric Biggers
` (18 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
The old AES-GCM library code is being replaced as part of an overhaul
that is adding support for all the common AES encryption modes with
consistent conventions. The SEV code is the only client of the old
AES-GCM code, so update it to the new API.
Besides some slight adjustments to the calling convention, the only
notable change is replacing the direct accesses to the auth tag length
field (which should be considered private until/unless someone truly
needs it) with the AUTHTAG_LEN constant.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
arch/x86/Kconfig | 2 +-
arch/x86/coco/sev/core.c | 44 ++++++++++++-------------
arch/x86/include/asm/sev.h | 2 +-
drivers/virt/coco/sev-guest/sev-guest.c | 7 ++--
4 files changed, 26 insertions(+), 29 deletions(-)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index bdad90f210e4..0b89641cac16 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -1495,7 +1495,7 @@ config AMD_MEM_ENCRYPT
select ARCH_HAS_CC_PLATFORM
select X86_MEM_ENCRYPT
select UNACCEPTED_MEMORY
- select CRYPTO_LIB_AESGCM
+ select CRYPTO_LIB_AES_GCM
help
Say yes to enable support for the encryption of system memory.
This requires an AMD processor that supports Secure Memory
diff --git a/arch/x86/coco/sev/core.c b/arch/x86/coco/sev/core.c
index ecd77d3217f3..085bbee6baf4 100644
--- a/arch/x86/coco/sev/core.c
+++ b/arch/x86/coco/sev/core.c
@@ -25,7 +25,7 @@
#include <linux/psp-sev.h>
#include <linux/dmi.h>
#include <uapi/linux/sev-guest.h>
-#include <crypto/gcm.h>
+#include <crypto/aes-gcm.h>
#include <asm/init.h>
#include <asm/cpu_entry_area.h>
@@ -1535,21 +1535,21 @@ static u8 *get_vmpck(int id, struct snp_secrets_page *secrets, u32 **seqno)
return key;
}
-static struct aesgcm_ctx *snp_init_crypto(u8 *key, size_t keylen)
+static struct aes_gcm_key *snp_init_crypto(const u8 *key, size_t keylen)
{
- struct aesgcm_ctx *ctx;
+ struct aes_gcm_key *gcm_key;
- ctx = kzalloc_obj(*ctx);
- if (!ctx)
+ gcm_key = kzalloc_obj(*gcm_key);
+ if (!gcm_key)
return NULL;
- if (aesgcm_expandkey(ctx, key, keylen, AUTHTAG_LEN)) {
- pr_err("Crypto context initialization failed\n");
- kfree(ctx);
+ if (aes_gcm_preparekey(gcm_key, key, keylen, AUTHTAG_LEN)) {
+ pr_err("AES-GCM key preparation failed\n");
+ kfree(gcm_key);
return NULL;
}
- return ctx;
+ return gcm_key;
}
int snp_msg_init(struct snp_msg_desc *mdesc, int vmpck_id)
@@ -1572,8 +1572,8 @@ int snp_msg_init(struct snp_msg_desc *mdesc, int vmpck_id)
mdesc->vmpck_id = vmpck_id;
- mdesc->ctx = snp_init_crypto(mdesc->vmpck, VMPCK_KEY_LEN);
- if (!mdesc->ctx)
+ mdesc->gcm_key = snp_init_crypto(mdesc->vmpck, VMPCK_KEY_LEN);
+ if (!mdesc->gcm_key)
return -ENOMEM;
return 0;
@@ -1624,7 +1624,7 @@ void snp_msg_free(struct snp_msg_desc *mdesc)
if (!mdesc)
return;
- kfree(mdesc->ctx);
+ kfree(mdesc->gcm_key);
free_shared_pages(mdesc->response, sizeof(struct snp_guest_msg));
free_shared_pages(mdesc->request, sizeof(struct snp_guest_msg));
iounmap((__force void __iomem *)mdesc->secrets);
@@ -1709,7 +1709,7 @@ static int verify_and_dec_payload(struct snp_msg_desc *mdesc, struct snp_guest_r
struct snp_guest_msg *req_msg = &mdesc->secret_request;
struct snp_guest_msg_hdr *req_msg_hdr = &req_msg->hdr;
struct snp_guest_msg_hdr *resp_msg_hdr = &resp_msg->hdr;
- struct aesgcm_ctx *ctx = mdesc->ctx;
+ struct aes_gcm_key *gcm_key = mdesc->gcm_key;
u8 iv[GCM_AES_IV_SIZE] = {};
pr_debug("response [seqno %lld type %d version %d sz %d]\n",
@@ -1732,23 +1732,21 @@ static int verify_and_dec_payload(struct snp_msg_desc *mdesc, struct snp_guest_r
* If the message size is greater than our buffer length then return
* an error.
*/
- if (unlikely((resp_msg_hdr->msg_sz + ctx->authsize) > req->resp_sz))
+ if (unlikely(resp_msg_hdr->msg_sz + AUTHTAG_LEN > req->resp_sz))
return -EBADMSG;
/* Decrypt the payload */
memcpy(iv, &resp_msg_hdr->msg_seqno, min(sizeof(iv), sizeof(resp_msg_hdr->msg_seqno)));
- if (!aesgcm_decrypt(ctx, req->resp_buf, resp_msg->payload, resp_msg_hdr->msg_sz,
- &resp_msg_hdr->algo, AAD_LEN, iv, resp_msg_hdr->authtag))
- return -EBADMSG;
-
- return 0;
+ return aes_gcm_decrypt(req->resp_buf, resp_msg->payload,
+ resp_msg_hdr->authtag, resp_msg_hdr->msg_sz,
+ &resp_msg_hdr->algo, AAD_LEN, iv, gcm_key);
}
static int enc_payload(struct snp_msg_desc *mdesc, u64 seqno, struct snp_guest_req *req)
{
struct snp_guest_msg *msg = &mdesc->secret_request;
struct snp_guest_msg_hdr *hdr = &msg->hdr;
- struct aesgcm_ctx *ctx = mdesc->ctx;
+ struct aes_gcm_key *gcm_key = mdesc->gcm_key;
u8 iv[GCM_AES_IV_SIZE] = {};
memset(msg, 0, sizeof(*msg));
@@ -1769,12 +1767,12 @@ static int enc_payload(struct snp_msg_desc *mdesc, u64 seqno, struct snp_guest_r
pr_debug("request [seqno %lld type %d version %d sz %d]\n",
hdr->msg_seqno, hdr->msg_type, hdr->msg_version, hdr->msg_sz);
- if (WARN_ON((req->req_sz + ctx->authsize) > sizeof(msg->payload)))
+ if (WARN_ON(req->req_sz + AUTHTAG_LEN > sizeof(msg->payload)))
return -EBADMSG;
memcpy(iv, &hdr->msg_seqno, min(sizeof(iv), sizeof(hdr->msg_seqno)));
- aesgcm_encrypt(ctx, msg->payload, req->req_buf, req->req_sz, &hdr->algo,
- AAD_LEN, iv, hdr->authtag);
+ aes_gcm_encrypt(msg->payload, hdr->authtag, req->req_buf, req->req_sz,
+ &hdr->algo, AAD_LEN, iv, gcm_key);
return 0;
}
diff --git a/arch/x86/include/asm/sev.h b/arch/x86/include/asm/sev.h
index 594cfa19cbd4..9e7a077c445d 100644
--- a/arch/x86/include/asm/sev.h
+++ b/arch/x86/include/asm/sev.h
@@ -314,7 +314,7 @@ struct snp_msg_desc {
struct snp_secrets_page *secrets;
- struct aesgcm_ctx *ctx;
+ struct aes_gcm_key *gcm_key;
u32 *os_area_msg_seqno;
u8 *vmpck;
diff --git a/drivers/virt/coco/sev-guest/sev-guest.c b/drivers/virt/coco/sev-guest/sev-guest.c
index d186ae55cf63..935537a41469 100644
--- a/drivers/virt/coco/sev-guest/sev-guest.c
+++ b/drivers/virt/coco/sev-guest/sev-guest.c
@@ -17,7 +17,6 @@
#include <linux/set_memory.h>
#include <linux/fs.h>
#include <linux/tsm.h>
-#include <crypto/gcm.h>
#include <linux/psp-sev.h>
#include <linux/sockptr.h>
#include <linux/cleanup.h>
@@ -87,7 +86,7 @@ static int get_report(struct snp_guest_dev *snp_dev, struct snp_guest_request_io
* response payload. Make sure that it has enough space to cover the
* authtag.
*/
- resp_len = sizeof(report_resp->data) + mdesc->ctx->authsize;
+ resp_len = sizeof(report_resp->data) + AUTHTAG_LEN;
report_resp = kzalloc(resp_len, GFP_KERNEL_ACCOUNT);
if (!report_resp)
return -ENOMEM;
@@ -130,7 +129,7 @@ static int get_derived_key(struct snp_guest_dev *snp_dev, struct snp_guest_reque
* response payload. Make sure that it has enough space to cover the
* authtag.
*/
- resp_len = sizeof(derived_key_resp->data) + mdesc->ctx->authsize;
+ resp_len = sizeof(derived_key_resp->data) + AUTHTAG_LEN;
derived_key_resp = kzalloc(resp_len, GFP_KERNEL_ACCOUNT);
if (!derived_key_resp)
return -ENOMEM;
@@ -230,7 +229,7 @@ static int get_ext_report(struct snp_guest_dev *snp_dev, struct snp_guest_reques
* response payload. Make sure that it has enough space to cover the
* authtag.
*/
- resp_len = sizeof(report_resp->data) + mdesc->ctx->authsize;
+ resp_len = sizeof(report_resp->data) + AUTHTAG_LEN;
report_resp = kzalloc(resp_len, GFP_KERNEL_ACCOUNT);
if (!report_resp) {
ret = -ENOMEM;
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 15/33] lib/crypto: aesgcm: Remove old AES-GCM library
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (13 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 14/33] x86/sev: Use new AES-GCM library Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 16/33] crypto: aes - Remove AES-CBC-MAC support Eric Biggers
` (17 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
The old AES-GCM library code is no longer used, so remove it.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
include/crypto/gcm.h | 21 --
lib/crypto/Kconfig | 6 -
lib/crypto/Makefile | 3 -
lib/crypto/aesgcm.c | 721 -------------------------------------------
4 files changed, 751 deletions(-)
delete mode 100644 lib/crypto/aesgcm.c
diff --git a/include/crypto/gcm.h b/include/crypto/gcm.h
index eb91d5603b74..1f4457608e4c 100644
--- a/include/crypto/gcm.h
+++ b/include/crypto/gcm.h
@@ -3,9 +3,6 @@
#include <linux/errno.h>
-#include <crypto/aes.h>
-#include <crypto/gf128hash.h>
-
#define GCM_AES_IV_SIZE 12
#define GCM_RFC4106_IV_SIZE 8
#define GCM_RFC4543_IV_SIZE 8
@@ -64,22 +61,4 @@ static inline int crypto_ipsec_check_assoclen(unsigned int assoclen)
return 0;
}
-struct aesgcm_ctx {
- struct ghash_key ghash_key;
- struct aes_enckey aes_key;
- unsigned int authsize;
-};
-
-int aesgcm_expandkey(struct aesgcm_ctx *ctx, const u8 *key,
- unsigned int keysize, unsigned int authsize);
-
-void aesgcm_encrypt(const struct aesgcm_ctx *ctx, u8 *dst, const u8 *src,
- int crypt_len, const u8 *assoc, int assoc_len,
- const u8 iv[GCM_AES_IV_SIZE], u8 *authtag);
-
-bool __must_check aesgcm_decrypt(const struct aesgcm_ctx *ctx, u8 *dst,
- const u8 *src, int crypt_len, const u8 *assoc,
- int assoc_len, const u8 iv[GCM_AES_IV_SIZE],
- const u8 *authtag);
-
#endif
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index 4066427017bc..f21b45854cc6 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -75,12 +75,6 @@ config CRYPTO_LIB_AES_XTS
help
The AES-XTS library functions.
-config CRYPTO_LIB_AESGCM
- tristate
- select CRYPTO_LIB_AES
- select CRYPTO_LIB_GF128HASH
- select CRYPTO_LIB_UTILS
-
config CRYPTO_LIB_ARC4
tristate
diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile
index f1e9bf89785f..ca068df1f71f 100644
--- a/lib/crypto/Makefile
+++ b/lib/crypto/Makefile
@@ -63,9 +63,6 @@ clean-files += powerpc/aesp8-ppc.S
obj-$(CONFIG_CRYPTO_LIB_AESCFB) += libaescfb.o
libaescfb-y := aescfb.o
-obj-$(CONFIG_CRYPTO_LIB_AESGCM) += libaesgcm.o
-libaesgcm-y := aesgcm.o
-
obj-$(CONFIG_CRYPTO_LIB_ARC4) += libarc4.o
libarc4-y := arc4.o
diff --git a/lib/crypto/aesgcm.c b/lib/crypto/aesgcm.c
deleted file mode 100644
index 1da31e1f747d..000000000000
--- a/lib/crypto/aesgcm.c
+++ /dev/null
@@ -1,721 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0
-/*
- * Minimal library implementation of GCM
- *
- * Copyright 2022 Google LLC
- */
-
-#include <crypto/gcm.h>
-#include <crypto/utils.h>
-#include <linux/export.h>
-#include <linux/module.h>
-
-/**
- * aesgcm_expandkey - Expands the AES and GHASH keys for the AES-GCM key
- * schedule
- *
- * @ctx: The data structure that will hold the AES-GCM key schedule
- * @key: The AES encryption input key
- * @keysize: The length in bytes of the input key
- * @authsize: The size in bytes of the GCM authentication tag
- *
- * Returns: 0 on success, or -EINVAL if @keysize or @authsize contain values
- * that are not permitted by the GCM specification.
- */
-int aesgcm_expandkey(struct aesgcm_ctx *ctx, const u8 *key,
- unsigned int keysize, unsigned int authsize)
-{
- u8 h[AES_BLOCK_SIZE] = {};
- int ret;
-
- ret = crypto_gcm_check_authsize(authsize) ?:
- aes_prepareenckey(&ctx->aes_key, key, keysize);
- if (ret)
- return ret;
-
- ctx->authsize = authsize;
- aes_encrypt(&ctx->aes_key, h, h);
- ghash_preparekey(&ctx->ghash_key, h);
- memzero_explicit(h, sizeof(h));
- return 0;
-}
-EXPORT_SYMBOL(aesgcm_expandkey);
-
-/**
- * aesgcm_mac - Generates the authentication tag using AES-GCM algorithm.
- * @ctx: The data structure that will hold the AES-GCM key schedule
- * @src: The input source data.
- * @src_len: Length of the source data.
- * @assoc: Points to the associated data.
- * @assoc_len: Length of the associated data values.
- * @ctr: Points to the counter value.
- * @authtag: The output buffer for the authentication tag.
- *
- * It takes in the AES-GCM context, source data, associated data, counter value,
- * and an output buffer for the authentication tag.
- */
-static void aesgcm_mac(const struct aesgcm_ctx *ctx, const u8 *src, int src_len,
- const u8 *assoc, int assoc_len, __be32 *ctr, u8 *authtag)
-{
- static const u8 zeroes[GHASH_BLOCK_SIZE];
- __be64 tail[2] = {
- cpu_to_be64((u64)assoc_len * 8),
- cpu_to_be64((u64)src_len * 8),
- };
- struct ghash_ctx ghash;
- u8 ghash_out[AES_BLOCK_SIZE];
- u8 enc_ctr[AES_BLOCK_SIZE];
-
- ghash_init(&ghash, &ctx->ghash_key);
-
- ghash_update(&ghash, assoc, assoc_len);
- ghash_update(&ghash, zeroes, -assoc_len & (GHASH_BLOCK_SIZE - 1));
-
- ghash_update(&ghash, src, src_len);
- ghash_update(&ghash, zeroes, -src_len & (GHASH_BLOCK_SIZE - 1));
-
- ghash_update(&ghash, (const u8 *)&tail, sizeof(tail));
-
- ghash_final(&ghash, ghash_out);
-
- ctr[3] = cpu_to_be32(1);
- aes_encrypt(&ctx->aes_key, enc_ctr, (const u8 *)ctr);
- crypto_xor_cpy(authtag, ghash_out, enc_ctr, ctx->authsize);
-
- memzero_explicit(ghash_out, sizeof(ghash_out));
- memzero_explicit(enc_ctr, sizeof(enc_ctr));
-}
-
-static void aesgcm_crypt(const struct aesgcm_ctx *ctx, u8 *dst, const u8 *src,
- int len, __be32 *ctr)
-{
- u8 buf[AES_BLOCK_SIZE];
- unsigned int n = 2;
-
- while (len > 0) {
- /*
- * The counter increment below must not result in overflow or
- * carry into the next 32-bit word, as this could result in
- * inadvertent IV reuse, which must be avoided at all cost for
- * stream ciphers such as AES-CTR. Given the range of 'int
- * len', this cannot happen, so no explicit test is necessary.
- */
- ctr[3] = cpu_to_be32(n++);
- aes_encrypt(&ctx->aes_key, buf, (const u8 *)ctr);
- crypto_xor_cpy(dst, src, buf, min(len, AES_BLOCK_SIZE));
-
- dst += AES_BLOCK_SIZE;
- src += AES_BLOCK_SIZE;
- len -= AES_BLOCK_SIZE;
- }
- memzero_explicit(buf, sizeof(buf));
-}
-
-/**
- * aesgcm_encrypt - Perform AES-GCM encryption on a block of data
- *
- * @ctx: The AES-GCM key schedule
- * @dst: Pointer to the ciphertext output buffer
- * @src: Pointer the plaintext (may equal @dst for encryption in place)
- * @crypt_len: The size in bytes of the plaintext and ciphertext.
- * @assoc: Pointer to the associated data,
- * @assoc_len: The size in bytes of the associated data
- * @iv: The initialization vector (IV) to use for this block of data
- * (must be 12 bytes in size as per the GCM spec recommendation)
- * @authtag: The address of the buffer in memory where the authentication
- * tag should be stored. The buffer is assumed to have space for
- * @ctx->authsize bytes.
- */
-void aesgcm_encrypt(const struct aesgcm_ctx *ctx, u8 *dst, const u8 *src,
- int crypt_len, const u8 *assoc, int assoc_len,
- const u8 iv[GCM_AES_IV_SIZE], u8 *authtag)
-{
- __be32 ctr[4];
-
- memcpy(ctr, iv, GCM_AES_IV_SIZE);
-
- aesgcm_crypt(ctx, dst, src, crypt_len, ctr);
- aesgcm_mac(ctx, dst, crypt_len, assoc, assoc_len, ctr, authtag);
-}
-EXPORT_SYMBOL(aesgcm_encrypt);
-
-/**
- * aesgcm_decrypt - Perform AES-GCM decryption on a block of data
- *
- * @ctx: The AES-GCM key schedule
- * @dst: Pointer to the plaintext output buffer
- * @src: Pointer the ciphertext (may equal @dst for decryption in place)
- * @crypt_len: The size in bytes of the plaintext and ciphertext.
- * @assoc: Pointer to the associated data,
- * @assoc_len: The size in bytes of the associated data
- * @iv: The initialization vector (IV) to use for this block of data
- * (must be 12 bytes in size as per the GCM spec recommendation)
- * @authtag: The address of the buffer in memory where the authentication
- * tag is stored.
- *
- * Returns: true on success, or false if the ciphertext failed authentication.
- * On failure, no plaintext will be returned.
- */
-bool __must_check aesgcm_decrypt(const struct aesgcm_ctx *ctx, u8 *dst,
- const u8 *src, int crypt_len, const u8 *assoc,
- int assoc_len, const u8 iv[GCM_AES_IV_SIZE],
- const u8 *authtag)
-{
- u8 tagbuf[AES_BLOCK_SIZE];
- __be32 ctr[4];
-
- memcpy(ctr, iv, GCM_AES_IV_SIZE);
-
- aesgcm_mac(ctx, src, crypt_len, assoc, assoc_len, ctr, tagbuf);
- if (crypto_memneq(authtag, tagbuf, ctx->authsize)) {
- memzero_explicit(tagbuf, sizeof(tagbuf));
- return false;
- }
- aesgcm_crypt(ctx, dst, src, crypt_len, ctr);
- return true;
-}
-EXPORT_SYMBOL(aesgcm_decrypt);
-
-MODULE_DESCRIPTION("Generic AES-GCM library");
-MODULE_AUTHOR("Ard Biesheuvel <ardb@kernel.org>");
-MODULE_LICENSE("GPL");
-
-#ifdef CONFIG_CRYPTO_SELFTESTS
-
-/*
- * Test code below. Vectors taken from crypto/testmgr.h
- */
-
-static const u8 __initconst ctext0[16] __nonstring =
- "\x58\xe2\xfc\xce\xfa\x7e\x30\x61"
- "\x36\x7f\x1d\x57\xa4\xe7\x45\x5a";
-
-static const u8 __initconst ptext1[16];
-
-static const u8 __initconst ctext1[32] __nonstring =
- "\x03\x88\xda\xce\x60\xb6\xa3\x92"
- "\xf3\x28\xc2\xb9\x71\xb2\xfe\x78"
- "\xab\x6e\x47\xd4\x2c\xec\x13\xbd"
- "\xf5\x3a\x67\xb2\x12\x57\xbd\xdf";
-
-static const u8 __initconst ptext2[64] __nonstring =
- "\xd9\x31\x32\x25\xf8\x84\x06\xe5"
- "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a"
- "\x86\xa7\xa9\x53\x15\x34\xf7\xda"
- "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72"
- "\x1c\x3c\x0c\x95\x95\x68\x09\x53"
- "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25"
- "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57"
- "\xba\x63\x7b\x39\x1a\xaf\xd2\x55";
-
-static const u8 __initconst ctext2[80] __nonstring =
- "\x42\x83\x1e\xc2\x21\x77\x74\x24"
- "\x4b\x72\x21\xb7\x84\xd0\xd4\x9c"
- "\xe3\xaa\x21\x2f\x2c\x02\xa4\xe0"
- "\x35\xc1\x7e\x23\x29\xac\xa1\x2e"
- "\x21\xd5\x14\xb2\x54\x66\x93\x1c"
- "\x7d\x8f\x6a\x5a\xac\x84\xaa\x05"
- "\x1b\xa3\x0b\x39\x6a\x0a\xac\x97"
- "\x3d\x58\xe0\x91\x47\x3f\x59\x85"
- "\x4d\x5c\x2a\xf3\x27\xcd\x64\xa6"
- "\x2c\xf3\x5a\xbd\x2b\xa6\xfa\xb4";
-
-static const u8 __initconst ptext3[60] __nonstring =
- "\xd9\x31\x32\x25\xf8\x84\x06\xe5"
- "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a"
- "\x86\xa7\xa9\x53\x15\x34\xf7\xda"
- "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72"
- "\x1c\x3c\x0c\x95\x95\x68\x09\x53"
- "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25"
- "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57"
- "\xba\x63\x7b\x39";
-
-static const u8 __initconst ctext3[76] __nonstring =
- "\x42\x83\x1e\xc2\x21\x77\x74\x24"
- "\x4b\x72\x21\xb7\x84\xd0\xd4\x9c"
- "\xe3\xaa\x21\x2f\x2c\x02\xa4\xe0"
- "\x35\xc1\x7e\x23\x29\xac\xa1\x2e"
- "\x21\xd5\x14\xb2\x54\x66\x93\x1c"
- "\x7d\x8f\x6a\x5a\xac\x84\xaa\x05"
- "\x1b\xa3\x0b\x39\x6a\x0a\xac\x97"
- "\x3d\x58\xe0\x91"
- "\x5b\xc9\x4f\xbc\x32\x21\xa5\xdb"
- "\x94\xfa\xe9\x5a\xe7\x12\x1a\x47";
-
-static const u8 __initconst ctext4[16] __nonstring =
- "\xcd\x33\xb2\x8a\xc7\x73\xf7\x4b"
- "\xa0\x0e\xd1\xf3\x12\x57\x24\x35";
-
-static const u8 __initconst ctext5[32] __nonstring =
- "\x98\xe7\x24\x7c\x07\xf0\xfe\x41"
- "\x1c\x26\x7e\x43\x84\xb0\xf6\x00"
- "\x2f\xf5\x8d\x80\x03\x39\x27\xab"
- "\x8e\xf4\xd4\x58\x75\x14\xf0\xfb";
-
-static const u8 __initconst ptext6[64] __nonstring =
- "\xd9\x31\x32\x25\xf8\x84\x06\xe5"
- "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a"
- "\x86\xa7\xa9\x53\x15\x34\xf7\xda"
- "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72"
- "\x1c\x3c\x0c\x95\x95\x68\x09\x53"
- "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25"
- "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57"
- "\xba\x63\x7b\x39\x1a\xaf\xd2\x55";
-
-static const u8 __initconst ctext6[80] __nonstring =
- "\x39\x80\xca\x0b\x3c\x00\xe8\x41"
- "\xeb\x06\xfa\xc4\x87\x2a\x27\x57"
- "\x85\x9e\x1c\xea\xa6\xef\xd9\x84"
- "\x62\x85\x93\xb4\x0c\xa1\xe1\x9c"
- "\x7d\x77\x3d\x00\xc1\x44\xc5\x25"
- "\xac\x61\x9d\x18\xc8\x4a\x3f\x47"
- "\x18\xe2\x44\x8b\x2f\xe3\x24\xd9"
- "\xcc\xda\x27\x10\xac\xad\xe2\x56"
- "\x99\x24\xa7\xc8\x58\x73\x36\xbf"
- "\xb1\x18\x02\x4d\xb8\x67\x4a\x14";
-
-static const u8 __initconst ctext7[16] __nonstring =
- "\x53\x0f\x8a\xfb\xc7\x45\x36\xb9"
- "\xa9\x63\xb4\xf1\xc4\xcb\x73\x8b";
-
-static const u8 __initconst ctext8[32] __nonstring =
- "\xce\xa7\x40\x3d\x4d\x60\x6b\x6e"
- "\x07\x4e\xc5\xd3\xba\xf3\x9d\x18"
- "\xd0\xd1\xc8\xa7\x99\x99\x6b\xf0"
- "\x26\x5b\x98\xb5\xd4\x8a\xb9\x19";
-
-static const u8 __initconst ptext9[64] __nonstring =
- "\xd9\x31\x32\x25\xf8\x84\x06\xe5"
- "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a"
- "\x86\xa7\xa9\x53\x15\x34\xf7\xda"
- "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72"
- "\x1c\x3c\x0c\x95\x95\x68\x09\x53"
- "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25"
- "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57"
- "\xba\x63\x7b\x39\x1a\xaf\xd2\x55";
-
-static const u8 __initconst ctext9[80] __nonstring =
- "\x52\x2d\xc1\xf0\x99\x56\x7d\x07"
- "\xf4\x7f\x37\xa3\x2a\x84\x42\x7d"
- "\x64\x3a\x8c\xdc\xbf\xe5\xc0\xc9"
- "\x75\x98\xa2\xbd\x25\x55\xd1\xaa"
- "\x8c\xb0\x8e\x48\x59\x0d\xbb\x3d"
- "\xa7\xb0\x8b\x10\x56\x82\x88\x38"
- "\xc5\xf6\x1e\x63\x93\xba\x7a\x0a"
- "\xbc\xc9\xf6\x62\x89\x80\x15\xad"
- "\xb0\x94\xda\xc5\xd9\x34\x71\xbd"
- "\xec\x1a\x50\x22\x70\xe3\xcc\x6c";
-
-static const u8 __initconst ptext10[60] __nonstring =
- "\xd9\x31\x32\x25\xf8\x84\x06\xe5"
- "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a"
- "\x86\xa7\xa9\x53\x15\x34\xf7\xda"
- "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72"
- "\x1c\x3c\x0c\x95\x95\x68\x09\x53"
- "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25"
- "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57"
- "\xba\x63\x7b\x39";
-
-static const u8 __initconst ctext10[76] __nonstring =
- "\x52\x2d\xc1\xf0\x99\x56\x7d\x07"
- "\xf4\x7f\x37\xa3\x2a\x84\x42\x7d"
- "\x64\x3a\x8c\xdc\xbf\xe5\xc0\xc9"
- "\x75\x98\xa2\xbd\x25\x55\xd1\xaa"
- "\x8c\xb0\x8e\x48\x59\x0d\xbb\x3d"
- "\xa7\xb0\x8b\x10\x56\x82\x88\x38"
- "\xc5\xf6\x1e\x63\x93\xba\x7a\x0a"
- "\xbc\xc9\xf6\x62"
- "\x76\xfc\x6e\xce\x0f\x4e\x17\x68"
- "\xcd\xdf\x88\x53\xbb\x2d\x55\x1b";
-
-static const u8 __initconst ptext11[60] __nonstring =
- "\xd9\x31\x32\x25\xf8\x84\x06\xe5"
- "\xa5\x59\x09\xc5\xaf\xf5\x26\x9a"
- "\x86\xa7\xa9\x53\x15\x34\xf7\xda"
- "\x2e\x4c\x30\x3d\x8a\x31\x8a\x72"
- "\x1c\x3c\x0c\x95\x95\x68\x09\x53"
- "\x2f\xcf\x0e\x24\x49\xa6\xb5\x25"
- "\xb1\x6a\xed\xf5\xaa\x0d\xe6\x57"
- "\xba\x63\x7b\x39";
-
-static const u8 __initconst ctext11[76] __nonstring =
- "\x39\x80\xca\x0b\x3c\x00\xe8\x41"
- "\xeb\x06\xfa\xc4\x87\x2a\x27\x57"
- "\x85\x9e\x1c\xea\xa6\xef\xd9\x84"
- "\x62\x85\x93\xb4\x0c\xa1\xe1\x9c"
- "\x7d\x77\x3d\x00\xc1\x44\xc5\x25"
- "\xac\x61\x9d\x18\xc8\x4a\x3f\x47"
- "\x18\xe2\x44\x8b\x2f\xe3\x24\xd9"
- "\xcc\xda\x27\x10"
- "\x25\x19\x49\x8e\x80\xf1\x47\x8f"
- "\x37\xba\x55\xbd\x6d\x27\x61\x8c";
-
-static const u8 __initconst ptext12[719] __nonstring =
- "\x42\xc1\xcc\x08\x48\x6f\x41\x3f"
- "\x2f\x11\x66\x8b\x2a\x16\xf0\xe0"
- "\x58\x83\xf0\xc3\x70\x14\xc0\x5b"
- "\x3f\xec\x1d\x25\x3c\x51\xd2\x03"
- "\xcf\x59\x74\x1f\xb2\x85\xb4\x07"
- "\xc6\x6a\x63\x39\x8a\x5b\xde\xcb"
- "\xaf\x08\x44\xbd\x6f\x91\x15\xe1"
- "\xf5\x7a\x6e\x18\xbd\xdd\x61\x50"
- "\x59\xa9\x97\xab\xbb\x0e\x74\x5c"
- "\x00\xa4\x43\x54\x04\x54\x9b\x3b"
- "\x77\xec\xfd\x5c\xa6\xe8\x7b\x08"
- "\xae\xe6\x10\x3f\x32\x65\xd1\xfc"
- "\xa4\x1d\x2c\x31\xfb\x33\x7a\xb3"
- "\x35\x23\xf4\x20\x41\xd4\xad\x82"
- "\x8b\xa4\xad\x96\x1c\x20\x53\xbe"
- "\x0e\xa6\xf4\xdc\x78\x49\x3e\x72"
- "\xb1\xa9\xb5\x83\xcb\x08\x54\xb7"
- "\xad\x49\x3a\xae\x98\xce\xa6\x66"
- "\x10\x30\x90\x8c\x55\x83\xd7\x7c"
- "\x8b\xe6\x53\xde\xd2\x6e\x18\x21"
- "\x01\x52\xd1\x9f\x9d\xbb\x9c\x73"
- "\x57\xcc\x89\x09\x75\x9b\x78\x70"
- "\xed\x26\x97\x4d\xb4\xe4\x0c\xa5"
- "\xfa\x70\x04\x70\xc6\x96\x1c\x7d"
- "\x54\x41\x77\xa8\xe3\xb0\x7e\x96"
- "\x82\xd9\xec\xa2\x87\x68\x55\xf9"
- "\x8f\x9e\x73\x43\x47\x6a\x08\x36"
- "\x93\x67\xa8\x2d\xde\xac\x41\xa9"
- "\x5c\x4d\x73\x97\x0f\x70\x68\xfa"
- "\x56\x4d\x00\xc2\x3b\x1f\xc8\xb9"
- "\x78\x1f\x51\x07\xe3\x9a\x13\x4e"
- "\xed\x2b\x2e\xa3\xf7\x44\xb2\xe7"
- "\xab\x19\x37\xd9\xba\x76\x5e\xd2"
- "\xf2\x53\x15\x17\x4c\x6b\x16\x9f"
- "\x02\x66\x49\xca\x7c\x91\x05\xf2"
- "\x45\x36\x1e\xf5\x77\xad\x1f\x46"
- "\xa8\x13\xfb\x63\xb6\x08\x99\x63"
- "\x82\xa2\xed\xb3\xac\xdf\x43\x19"
- "\x45\xea\x78\x73\xd9\xb7\x39\x11"
- "\xa3\x13\x7c\xf8\x3f\xf7\xad\x81"
- "\x48\x2f\xa9\x5c\x5f\xa0\xf0\x79"
- "\xa4\x47\x7d\x80\x20\x26\xfd\x63"
- "\x0a\xc7\x7e\x6d\x75\x47\xff\x76"
- "\x66\x2e\x8a\x6c\x81\x35\xaf\x0b"
- "\x2e\x6a\x49\x60\xc1\x10\xe1\xe1"
- "\x54\x03\xa4\x09\x0c\x37\x7a\x15"
- "\x23\x27\x5b\x8b\x4b\xa5\x64\x97"
- "\xae\x4a\x50\x73\x1f\x66\x1c\x5c"
- "\x03\x25\x3c\x8d\x48\x58\x71\x34"
- "\x0e\xec\x4e\x55\x1a\x03\x6a\xe5"
- "\xb6\x19\x2b\x84\x2a\x20\xd1\xea"
- "\x80\x6f\x96\x0e\x05\x62\xc7\x78"
- "\x87\x79\x60\x38\x46\xb4\x25\x57"
- "\x6e\x16\x63\xf8\xad\x6e\xd7\x42"
- "\x69\xe1\x88\xef\x6e\xd5\xb4\x9a"
- "\x3c\x78\x6c\x3b\xe5\xa0\x1d\x22"
- "\x86\x5c\x74\x3a\xeb\x24\x26\xc7"
- "\x09\xfc\x91\x96\x47\x87\x4f\x1a"
- "\xd6\x6b\x2c\x18\x47\xc0\xb8\x24"
- "\xa8\x5a\x4a\x9e\xcb\x03\xe7\x2a"
- "\x09\xe6\x4d\x9c\x6d\x86\x60\xf5"
- "\x2f\x48\x69\x37\x9f\xf2\xd2\xcb"
- "\x0e\x5a\xdd\x6e\x8a\xfb\x6a\xfe"
- "\x0b\x63\xde\x87\x42\x79\x8a\x68"
- "\x51\x28\x9b\x7a\xeb\xaf\xb8\x2f"
- "\x9d\xd1\xc7\x45\x90\x08\xc9\x83"
- "\xe9\x83\x84\xcb\x28\x69\x09\x69"
- "\xce\x99\x46\x00\x54\xcb\xd8\x38"
- "\xf9\x53\x4a\xbf\x31\xce\x57\x15"
- "\x33\xfa\x96\x04\x33\x42\xe3\xc0"
- "\xb7\x54\x4a\x65\x7a\x7c\x02\xe6"
- "\x19\x95\xd0\x0e\x82\x07\x63\xf9"
- "\xe1\x2b\x2a\xfc\x55\x92\x52\xc9"
- "\xb5\x9f\x23\x28\x60\xe7\x20\x51"
- "\x10\xd3\xed\x6d\x9b\xab\xb8\xe2"
- "\x5d\x9a\x34\xb3\xbe\x9c\x64\xcb"
- "\x78\xc6\x91\x22\x40\x91\x80\xbe"
- "\xd7\x78\x5c\x0e\x0a\xdc\x08\xe9"
- "\x67\x10\xa4\x83\x98\x79\x23\xe7"
- "\x92\xda\xa9\x22\x16\xb1\xe7\x78"
- "\xa3\x1c\x6c\x8f\x35\x7c\x4d\x37"
- "\x2f\x6e\x0b\x50\x5c\x34\xb9\xf9"
- "\xe6\x3d\x91\x0d\x32\x95\xaa\x3d"
- "\x48\x11\x06\xbb\x2d\xf2\x63\x88"
- "\x3f\x73\x09\xe2\x45\x56\x31\x51"
- "\xfa\x5e\x4e\x62\xf7\x90\xf9\xa9"
- "\x7d\x7b\x1b\xb1\xc8\x26\x6e\x66"
- "\xf6\x90\x9a\x7f\xf2\x57\xcc\x23"
- "\x59\xfa\xfa\xaa\x44\x04\x01\xa7"
- "\xa4\x78\xdb\x74\x3d\x8b\xb5";
-
-static const u8 __initconst ctext12[735] __nonstring =
- "\x84\x0b\xdb\xd5\xb7\xa8\xfe\x20"
- "\xbb\xb1\x12\x7f\x41\xea\xb3\xc0"
- "\xa2\xb4\x37\x19\x11\x58\xb6\x0b"
- "\x4c\x1d\x38\x05\x54\xd1\x16\x73"
- "\x8e\x1c\x20\x90\xa2\x9a\xb7\x74"
- "\x47\xe6\xd8\xfc\x18\x3a\xb4\xea"
- "\xd5\x16\x5a\x2c\x53\x01\x46\xb3"
- "\x18\x33\x74\x6c\x50\xf2\xe8\xc0"
- "\x73\xda\x60\x22\xeb\xe3\xe5\x9b"
- "\x20\x93\x6c\x4b\x37\x99\xb8\x23"
- "\x3b\x4e\xac\xe8\x5b\xe8\x0f\xb7"
- "\xc3\x8f\xfb\x4a\x37\xd9\x39\x95"
- "\x34\xf1\xdb\x8f\x71\xd9\xc7\x0b"
- "\x02\xf1\x63\xfc\x9b\xfc\xc5\xab"
- "\xb9\x14\x13\x21\xdf\xce\xaa\x88"
- "\x44\x30\x1e\xce\x26\x01\x92\xf8"
- "\x9f\x00\x4b\x0c\x4b\xf7\x5f\xe0"
- "\x89\xca\x94\x66\x11\x21\x97\xca"
- "\x3e\x83\x74\x2d\xdb\x4d\x11\xeb"
- "\x97\xc2\x14\xff\x9e\x1e\xa0\x6b"
- "\x08\xb4\x31\x2b\x85\xc6\x85\x6c"
- "\x90\xec\x39\xc0\xec\xb3\xb5\x4e"
- "\xf3\x9c\xe7\x83\x3a\x77\x0a\xf4"
- "\x56\xfe\xce\x18\x33\x6d\x0b\x2d"
- "\x33\xda\xc8\x05\x5c\xb4\x09\x2a"
- "\xde\x6b\x52\x98\x01\xef\x36\x3d"
- "\xbd\xf9\x8f\xa8\x3e\xaa\xcd\xd1"
- "\x01\x2d\x42\x49\xc3\xb6\x84\xbb"
- "\x48\x96\xe0\x90\x93\x6c\x48\x64"
- "\xd4\xfa\x7f\x93\x2c\xa6\x21\xc8"
- "\x7a\x23\x7b\xaa\x20\x56\x12\xae"
- "\x16\x9d\x94\x0f\x54\xa1\xec\xca"
- "\x51\x4e\xf2\x39\xf4\xf8\x5f\x04"
- "\x5a\x0d\xbf\xf5\x83\xa1\x15\xe1"
- "\xf5\x3c\xd8\x62\xa3\xed\x47\x89"
- "\x85\x4c\xe5\xdb\xac\x9e\x17\x1d"
- "\x0c\x09\xe3\x3e\x39\x5b\x4d\x74"
- "\x0e\xf5\x34\xee\x70\x11\x4c\xfd"
- "\xdb\x34\xb1\xb5\x10\x3f\x73\xb7"
- "\xf5\xfa\xed\xb0\x1f\xa5\xcd\x3c"
- "\x8d\x35\x83\xd4\x11\x44\x6e\x6c"
- "\x5b\xe0\x0e\x69\xa5\x39\xe5\xbb"
- "\xa9\x57\x24\x37\xe6\x1f\xdd\xcf"
- "\x16\x2a\x13\xf9\x6a\x2d\x90\xa0"
- "\x03\x60\x7a\xed\x69\xd5\x00\x8b"
- "\x7e\x4f\xcb\xb9\xfa\x91\xb9\x37"
- "\xc1\x26\xce\x90\x97\x22\x64\x64"
- "\xc1\x72\x43\x1b\xf6\xac\xc1\x54"
- "\x8a\x10\x9c\xdd\x8d\xd5\x8e\xb2"
- "\xe4\x85\xda\xe0\x20\x5f\xf4\xb4"
- "\x15\xb5\xa0\x8d\x12\x74\x49\x23"
- "\x3a\xdf\x4a\xd3\xf0\x3b\x89\xeb"
- "\xf8\xcc\x62\x7b\xfb\x93\x07\x41"
- "\x61\x26\x94\x58\x70\xa6\x3c\xe4"
- "\xff\x58\xc4\x13\x3d\xcb\x36\x6b"
- "\x32\xe5\xb2\x6d\x03\x74\x6f\x76"
- "\x93\x77\xde\x48\xc4\xfa\x30\x4a"
- "\xda\x49\x80\x77\x0f\x1c\xbe\x11"
- "\xc8\x48\xb1\xe5\xbb\xf2\x8a\xe1"
- "\x96\x2f\x9f\xd1\x8e\x8a\x5c\xe2"
- "\xf7\xd7\xd8\x54\xf3\x3f\xc4\x91"
- "\xb8\xfb\x86\xdc\x46\x24\x91\x60"
- "\x6c\x2f\xc9\x41\x37\x51\x49\x54"
- "\x09\x81\x21\xf3\x03\x9f\x2b\xe3"
- "\x1f\x39\x63\xaf\xf4\xd7\x53\x60"
- "\xa7\xc7\x54\xf9\xee\xb1\xb1\x7d"
- "\x75\x54\x65\x93\xfe\xb1\x68\x6b"
- "\x57\x02\xf9\xbb\x0e\xf9\xf8\xbf"
- "\x01\x12\x27\xb4\xfe\xe4\x79\x7a"
- "\x40\x5b\x51\x4b\xdf\x38\xec\xb1"
- "\x6a\x56\xff\x35\x4d\x42\x33\xaa"
- "\x6f\x1b\xe4\xdc\xe0\xdb\x85\x35"
- "\x62\x10\xd4\xec\xeb\xc5\x7e\x45"
- "\x1c\x6f\x17\xca\x3b\x8e\x2d\x66"
- "\x4f\x4b\x36\x56\xcd\x1b\x59\xaa"
- "\xd2\x9b\x17\xb9\x58\xdf\x7b\x64"
- "\x8a\xff\x3b\x9c\xa6\xb5\x48\x9e"
- "\xaa\xe2\x5d\x09\x71\x32\x5f\xb6"
- "\x29\xbe\xe7\xc7\x52\x7e\x91\x82"
- "\x6b\x6d\x33\xe1\x34\x06\x36\x21"
- "\x5e\xbe\x1e\x2f\x3e\xc1\xfb\xea"
- "\x49\x2c\xb5\xca\xf7\xb0\x37\xea"
- "\x1f\xed\x10\x04\xd9\x48\x0d\x1a"
- "\x1c\xfb\xe7\x84\x0e\x83\x53\x74"
- "\xc7\x65\xe2\x5c\xe5\xba\x73\x4c"
- "\x0e\xe1\xb5\x11\x45\x61\x43\x46"
- "\xaa\x25\x8f\xbd\x85\x08\xfa\x4c"
- "\x15\xc1\xc0\xd8\xf5\xdc\x16\xbb"
- "\x7b\x1d\xe3\x87\x57\xa7\x2a\x1d"
- "\x38\x58\x9e\x8a\x43\xdc\x57"
- "\xd1\x81\x7d\x2b\xe9\xff\x99\x3a"
- "\x4b\x24\x52\x58\x55\xe1\x49\x14";
-
-static struct {
- const u8 *ptext;
- const u8 *ctext;
-
- u8 key[AES_MAX_KEY_SIZE] __nonstring;
- u8 iv[GCM_AES_IV_SIZE] __nonstring;
- u8 assoc[20] __nonstring;
-
- int klen;
- int clen;
- int plen;
- int alen;
-} const aesgcm_tv[] __initconst = {
- { /* From McGrew & Viega - http://citeseer.ist.psu.edu/656989.html */
- .klen = 16,
- .ctext = ctext0,
- .clen = sizeof(ctext0),
- }, {
- .klen = 16,
- .ptext = ptext1,
- .plen = sizeof(ptext1),
- .ctext = ctext1,
- .clen = sizeof(ctext1),
- }, {
- .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c"
- "\x6d\x6a\x8f\x94\x67\x30\x83\x08",
- .klen = 16,
- .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad"
- "\xde\xca\xf8\x88",
- .ptext = ptext2,
- .plen = sizeof(ptext2),
- .ctext = ctext2,
- .clen = sizeof(ctext2),
- }, {
- .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c"
- "\x6d\x6a\x8f\x94\x67\x30\x83\x08",
- .klen = 16,
- .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad"
- "\xde\xca\xf8\x88",
- .ptext = ptext3,
- .plen = sizeof(ptext3),
- .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef"
- "\xfe\xed\xfa\xce\xde\xad\xbe\xef"
- "\xab\xad\xda\xd2",
- .alen = 20,
- .ctext = ctext3,
- .clen = sizeof(ctext3),
- }, {
- .klen = 24,
- .ctext = ctext4,
- .clen = sizeof(ctext4),
- }, {
- .klen = 24,
- .ptext = ptext1,
- .plen = sizeof(ptext1),
- .ctext = ctext5,
- .clen = sizeof(ctext5),
- }, {
- .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c"
- "\x6d\x6a\x8f\x94\x67\x30\x83\x08"
- "\xfe\xff\xe9\x92\x86\x65\x73\x1c",
- .klen = 24,
- .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad"
- "\xde\xca\xf8\x88",
- .ptext = ptext6,
- .plen = sizeof(ptext6),
- .ctext = ctext6,
- .clen = sizeof(ctext6),
- }, {
- .klen = 32,
- .ctext = ctext7,
- .clen = sizeof(ctext7),
- }, {
- .klen = 32,
- .ptext = ptext1,
- .plen = sizeof(ptext1),
- .ctext = ctext8,
- .clen = sizeof(ctext8),
- }, {
- .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c"
- "\x6d\x6a\x8f\x94\x67\x30\x83\x08"
- "\xfe\xff\xe9\x92\x86\x65\x73\x1c"
- "\x6d\x6a\x8f\x94\x67\x30\x83\x08",
- .klen = 32,
- .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad"
- "\xde\xca\xf8\x88",
- .ptext = ptext9,
- .plen = sizeof(ptext9),
- .ctext = ctext9,
- .clen = sizeof(ctext9),
- }, {
- .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c"
- "\x6d\x6a\x8f\x94\x67\x30\x83\x08"
- "\xfe\xff\xe9\x92\x86\x65\x73\x1c"
- "\x6d\x6a\x8f\x94\x67\x30\x83\x08",
- .klen = 32,
- .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad"
- "\xde\xca\xf8\x88",
- .ptext = ptext10,
- .plen = sizeof(ptext10),
- .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef"
- "\xfe\xed\xfa\xce\xde\xad\xbe\xef"
- "\xab\xad\xda\xd2",
- .alen = 20,
- .ctext = ctext10,
- .clen = sizeof(ctext10),
- }, {
- .key = "\xfe\xff\xe9\x92\x86\x65\x73\x1c"
- "\x6d\x6a\x8f\x94\x67\x30\x83\x08"
- "\xfe\xff\xe9\x92\x86\x65\x73\x1c",
- .klen = 24,
- .iv = "\xca\xfe\xba\xbe\xfa\xce\xdb\xad"
- "\xde\xca\xf8\x88",
- .ptext = ptext11,
- .plen = sizeof(ptext11),
- .assoc = "\xfe\xed\xfa\xce\xde\xad\xbe\xef"
- "\xfe\xed\xfa\xce\xde\xad\xbe\xef"
- "\xab\xad\xda\xd2",
- .alen = 20,
- .ctext = ctext11,
- .clen = sizeof(ctext11),
- }, {
- .key = "\x62\x35\xf8\x95\xfc\xa5\xeb\xf6"
- "\x0e\x92\x12\x04\xd3\xa1\x3f\x2e"
- "\x8b\x32\xcf\xe7\x44\xed\x13\x59"
- "\x04\x38\x77\xb0\xb9\xad\xb4\x38",
- .klen = 32,
- .iv = "\x00\xff\xff\xff\xff\x00\x00\xff"
- "\xff\xff\x00\xff",
- .ptext = ptext12,
- .plen = sizeof(ptext12),
- .ctext = ctext12,
- .clen = sizeof(ctext12),
- }
-};
-
-static int __init libaesgcm_init(void)
-{
- for (int i = 0; i < ARRAY_SIZE(aesgcm_tv); i++) {
- u8 tagbuf[AES_BLOCK_SIZE];
- int plen = aesgcm_tv[i].plen;
- struct aesgcm_ctx ctx;
- static u8 buf[sizeof(ptext12)];
-
- if (aesgcm_expandkey(&ctx, aesgcm_tv[i].key, aesgcm_tv[i].klen,
- aesgcm_tv[i].clen - plen)) {
- pr_err("aesgcm_expandkey() failed on vector %d\n", i);
- return -ENODEV;
- }
-
- if (!aesgcm_decrypt(&ctx, buf, aesgcm_tv[i].ctext, plen,
- aesgcm_tv[i].assoc, aesgcm_tv[i].alen,
- aesgcm_tv[i].iv, aesgcm_tv[i].ctext + plen)
- || memcmp(buf, aesgcm_tv[i].ptext, plen)) {
- pr_err("aesgcm_decrypt() #1 failed on vector %d\n", i);
- return -ENODEV;
- }
-
- /* encrypt in place */
- aesgcm_encrypt(&ctx, buf, buf, plen, aesgcm_tv[i].assoc,
- aesgcm_tv[i].alen, aesgcm_tv[i].iv, tagbuf);
- if (memcmp(buf, aesgcm_tv[i].ctext, plen)) {
- pr_err("aesgcm_encrypt() failed on vector %d\n", i);
- return -ENODEV;
- }
-
- /* decrypt in place */
- if (!aesgcm_decrypt(&ctx, buf, buf, plen, aesgcm_tv[i].assoc,
- aesgcm_tv[i].alen, aesgcm_tv[i].iv, tagbuf)
- || memcmp(buf, aesgcm_tv[i].ptext, plen)) {
- pr_err("aesgcm_decrypt() #2 failed on vector %d\n", i);
- return -ENODEV;
- }
- }
- return 0;
-}
-module_init(libaesgcm_init);
-
-static void __exit libaesgcm_exit(void)
-{
-}
-module_exit(libaesgcm_exit);
-#endif
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 16/33] crypto: aes - Remove AES-CBC-MAC support
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (14 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 15/33] lib/crypto: aesgcm: Remove old " Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 17/33] lib/crypto: aes: Remove aes_cbcmac_* functions Eric Biggers
` (16 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
"cbcmac(aes)" is used only by the "ccm" template when instantiated as
"ccm(aes)". Since native "ccm(aes)" support has been added to
crypto/aes.c, the "cbcmac(aes)" crypto_shash is no longer needed.
Note that "cmac(aes)" and "xcbc(aes)" remain supported.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
crypto/Kconfig | 4 +-
crypto/aes.c | 62 ----------------------------
crypto/testmgr.c | 6 +--
drivers/crypto/starfive/jh7110-aes.c | 2 +-
4 files changed, 6 insertions(+), 68 deletions(-)
diff --git a/crypto/Kconfig b/crypto/Kconfig
index e6b894dc784a..83adbd54d725 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -359,14 +359,14 @@ config CRYPTO_AES
select CRYPTO_ALGAPI
select CRYPTO_LIB_AES
select CRYPTO_LIB_AES_CBC if CRYPTO_CBC || CRYPTO_CTS
- select CRYPTO_LIB_AES_CBC_MACS if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
+ select CRYPTO_LIB_AES_CBC_MACS if CRYPTO_CMAC || CRYPTO_XCBC
select CRYPTO_LIB_AES_CCM if CRYPTO_CCM
select CRYPTO_LIB_AES_CTR if CRYPTO_CTR || CRYPTO_XCTR
select CRYPTO_LIB_AES_ECB if CRYPTO_ECB
select CRYPTO_LIB_AES_GCM if CRYPTO_GCM
select CRYPTO_LIB_AES_XTS if CRYPTO_XTS
select CRYPTO_AEAD if CRYPTO_GCM || CRYPTO_CCM
- select CRYPTO_HASH if CRYPTO_CMAC || CRYPTO_XCBC || CRYPTO_CCM
+ select CRYPTO_HASH if CRYPTO_CMAC || CRYPTO_XCBC
# CRYPTO_SKCIPHER should be selected only if a mode that needs it is
# enabled, but that doesn't work due to a recursive dependency caused by
# CRYPTO_SKCIPHER selecting CRYPTO_ECB. So just always select it.
diff --git a/crypto/aes.c b/crypto/aes.c
index ac5190292b3c..5f9c8eab67b4 100644
--- a/crypto/aes.c
+++ b/crypto/aes.c
@@ -95,47 +95,6 @@ static int __maybe_unused crypto_aes_cmac_digest(struct shash_desc *desc,
return 0;
}
-#define AES_CBCMAC_KEY(tfm) ((struct aes_enckey *)crypto_shash_ctx(tfm))
-#define AES_CBCMAC_CTX(desc) ((struct aes_cbcmac_ctx *)shash_desc_ctx(desc))
-
-static int __maybe_unused crypto_aes_cbcmac_setkey(struct crypto_shash *tfm,
- const u8 *in_key,
- unsigned int key_len)
-{
- return aes_prepareenckey(AES_CBCMAC_KEY(tfm), in_key, key_len);
-}
-
-static int __maybe_unused crypto_aes_cbcmac_init(struct shash_desc *desc)
-{
- aes_cbcmac_init(AES_CBCMAC_CTX(desc), AES_CBCMAC_KEY(desc->tfm));
- return 0;
-}
-
-static int __maybe_unused crypto_aes_cbcmac_update(struct shash_desc *desc,
- const u8 *data,
- unsigned int len)
-{
- aes_cbcmac_update(AES_CBCMAC_CTX(desc), data, len);
- return 0;
-}
-
-static int __maybe_unused crypto_aes_cbcmac_final(struct shash_desc *desc,
- u8 *out)
-{
- aes_cbcmac_final(AES_CBCMAC_CTX(desc), out);
- return 0;
-}
-
-static int __maybe_unused crypto_aes_cbcmac_digest(struct shash_desc *desc,
- const u8 *data,
- unsigned int len, u8 *out)
-{
- aes_cbcmac_init(AES_CBCMAC_CTX(desc), AES_CBCMAC_KEY(desc->tfm));
- aes_cbcmac_update(AES_CBCMAC_CTX(desc), data, len);
- aes_cbcmac_final(AES_CBCMAC_CTX(desc), out);
- return 0;
-}
-
static struct crypto_alg alg = {
.cra_name = "aes",
.cra_driver_name = "aes-lib",
@@ -190,23 +149,6 @@ static struct shash_alg mac_algs[] = {
.descsize = sizeof(struct aes_cmac_ctx),
},
#endif
-#if IS_ENABLED(CONFIG_CRYPTO_CCM)
- {
- .base.cra_name = "cbcmac(aes)",
- .base.cra_driver_name = "cbcmac-aes-lib",
- .base.cra_priority = 300,
- .base.cra_blocksize = AES_BLOCK_SIZE,
- .base.cra_ctxsize = sizeof(struct aes_enckey),
- .base.cra_module = THIS_MODULE,
- .digestsize = AES_BLOCK_SIZE,
- .setkey = crypto_aes_cbcmac_setkey,
- .init = crypto_aes_cbcmac_init,
- .update = crypto_aes_cbcmac_update,
- .final = crypto_aes_cbcmac_final,
- .digest = crypto_aes_cbcmac_digest,
- .descsize = sizeof(struct aes_cbcmac_ctx),
- },
-#endif
};
static __maybe_unused int
@@ -1096,10 +1038,6 @@ MODULE_ALIAS_CRYPTO("cmac-aes-lib");
MODULE_ALIAS_CRYPTO("xcbc(aes)");
MODULE_ALIAS_CRYPTO("xcbc-aes-lib");
#endif
-#if IS_ENABLED(CONFIG_CRYPTO_CCM)
-MODULE_ALIAS_CRYPTO("cbcmac(aes)");
-MODULE_ALIAS_CRYPTO("cbcmac-aes-lib");
-#endif
#if IS_ENABLED(CONFIG_CRYPTO_ECB)
MODULE_ALIAS_CRYPTO("ecb(aes)");
MODULE_ALIAS_CRYPTO("ecb-aes-lib");
diff --git a/crypto/testmgr.c b/crypto/testmgr.c
index 4958211fbfa9..b25c15173c76 100644
--- a/crypto/testmgr.c
+++ b/crypto/testmgr.c
@@ -4421,7 +4421,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}, {
#endif
.alg = "cbcmac(aes)",
- .generic_driver = "cbcmac-aes-lib",
+ .generic_driver = "cbcmac(aes-lib)",
.test = alg_test_hash,
.suite = {
.hash = __VECS(aes_cbcmac_tv_template)
@@ -4434,7 +4434,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}, {
.alg = "ccm(aes)",
- .generic_driver = "ccm_base(ctr(aes-lib),cbcmac-aes-lib)",
+ .generic_driver = "ccm_base(ctr(aes-lib),cbcmac(aes-lib))",
.test = alg_test_aead,
.fips_allowed = 1,
.suite = {
@@ -5239,7 +5239,7 @@ static const struct alg_test_desc alg_test_descs[] = {
}
}, {
.alg = "rfc4309(ccm(aes))",
- .generic_driver = "rfc4309(ccm_base(ctr(aes-lib),cbcmac-aes-lib))",
+ .generic_driver = "rfc4309(ccm_base(ctr(aes-lib),cbcmac(aes-lib)))",
.test = alg_test_aead,
.fips_allowed = 1,
.suite = {
diff --git a/drivers/crypto/starfive/jh7110-aes.c b/drivers/crypto/starfive/jh7110-aes.c
index a0713aa21250..c591a850f093 100644
--- a/drivers/crypto/starfive/jh7110-aes.c
+++ b/drivers/crypto/starfive/jh7110-aes.c
@@ -1003,7 +1003,7 @@ static int starfive_aes_ctr_init_tfm(struct crypto_skcipher *tfm)
static int starfive_aes_ccm_init_tfm(struct crypto_aead *tfm)
{
- return starfive_aes_aead_init_tfm(tfm, "ccm_base(ctr(aes-lib),cbcmac-aes-lib)");
+ return starfive_aes_aead_init_tfm(tfm, "ccm_base(ctr(aes-lib),cbcmac(aes-lib))");
}
static int starfive_aes_gcm_init_tfm(struct crypto_aead *tfm)
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 17/33] lib/crypto: aes: Remove aes_cbcmac_* functions
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (15 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 16/33] crypto: aes - Remove AES-CBC-MAC support Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 18/33] fscrypt: Use aes_ecb_encrypt() for v1 key derivation Eric Biggers
` (15 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
The aes_cbcmac_* functions are no longer used, except by their KUnit
test, since their functionality was folded directly into the AES-CCM
library code. Remove them.
Note that the aes_cmac_* and aes_xcbcmac_* functions remain unchanged.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
include/crypto/aes-cbc-macs.h | 22 +--------
lib/crypto/aes.c | 43 -----------------
lib/crypto/tests/aes_cbc_macs_kunit.c | 68 +--------------------------
3 files changed, 2 insertions(+), 131 deletions(-)
diff --git a/include/crypto/aes-cbc-macs.h b/include/crypto/aes-cbc-macs.h
index e61df108b926..95bbb8603420 100644
--- a/include/crypto/aes-cbc-macs.h
+++ b/include/crypto/aes-cbc-macs.h
@@ -1,6 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
- * Support for AES-CMAC, AES-XCBC-MAC, and AES-CBC-MAC
+ * Support for AES-CMAC and AES-XCBC-MAC
*
* Copyright 2026 Google LLC
*/
@@ -131,24 +131,4 @@ static inline void aes_cmac(const struct aes_cmac_key *key, const u8 *data,
aes_cmac_final(&ctx, out);
}
-/*
- * AES-CBC-MAC support. This is provided only for use by the implementation of
- * AES-CCM. It should have no other users. Warning: unlike AES-CMAC and
- * AES-XCBC-MAC, AES-CBC-MAC isn't a secure MAC for variable-length messages.
- */
-struct aes_cbcmac_ctx {
- const struct aes_enckey *key;
- size_t partial_len;
- u8 h[AES_BLOCK_SIZE];
-};
-static inline void aes_cbcmac_init(struct aes_cbcmac_ctx *ctx,
- const struct aes_enckey *key)
-{
- *ctx = (struct aes_cbcmac_ctx){ .key = key };
-}
-void aes_cbcmac_update(struct aes_cbcmac_ctx *ctx, const u8 *data,
- size_t data_len);
-void aes_cbcmac_final(struct aes_cbcmac_ctx *ctx,
- u8 out[at_least AES_BLOCK_SIZE]);
-
#endif /* _CRYPTO_AES_CBC_MACS_H */
diff --git a/lib/crypto/aes.c b/lib/crypto/aes.c
index 1a1b32e41ac1..aeefd55eacaa 100644
--- a/lib/crypto/aes.c
+++ b/lib/crypto/aes.c
@@ -674,49 +674,6 @@ void aes_cmac_final(struct aes_cmac_ctx *ctx, u8 out[AES_BLOCK_SIZE])
}
EXPORT_SYMBOL_GPL(aes_cmac_final);
-void aes_cbcmac_update(struct aes_cbcmac_ctx *ctx, const u8 *data,
- size_t data_len)
-{
- bool enc_before = false;
- size_t nblocks;
-
- if (ctx->partial_len) {
- size_t l = min(data_len, AES_BLOCK_SIZE - ctx->partial_len);
-
- crypto_xor(&ctx->h[ctx->partial_len], data, l);
- data += l;
- data_len -= l;
- ctx->partial_len += l;
- if (ctx->partial_len < AES_BLOCK_SIZE)
- return;
- enc_before = true;
- }
-
- nblocks = data_len / AES_BLOCK_SIZE;
- data_len %= AES_BLOCK_SIZE;
- if (nblocks == 0) {
- if (enc_before)
- aes_encrypt(ctx->key, ctx->h, ctx->h);
- } else {
- aes_cbcmac_blocks(ctx->h, ctx->key, data, nblocks, enc_before,
- /* enc_after= */ true);
- data += nblocks * AES_BLOCK_SIZE;
- }
- crypto_xor(ctx->h, data, data_len);
- ctx->partial_len = data_len;
-}
-EXPORT_SYMBOL_NS_GPL(aes_cbcmac_update, "CRYPTO_INTERNAL");
-
-void aes_cbcmac_final(struct aes_cbcmac_ctx *ctx, u8 out[AES_BLOCK_SIZE])
-{
- if (ctx->partial_len)
- aes_encrypt(ctx->key, out, ctx->h);
- else
- memcpy(out, ctx->h, AES_BLOCK_SIZE);
- memzero_explicit(ctx, sizeof(*ctx));
-}
-EXPORT_SYMBOL_NS_GPL(aes_cbcmac_final, "CRYPTO_INTERNAL");
-
/*
* FIPS cryptographic algorithm self-test for AES-CMAC. As per the FIPS 140-3
* Implementation Guidance, a cryptographic algorithm self-test for at least one
diff --git a/lib/crypto/tests/aes_cbc_macs_kunit.c b/lib/crypto/tests/aes_cbc_macs_kunit.c
index ae3745212f03..97a4594895c2 100644
--- a/lib/crypto/tests/aes_cbc_macs_kunit.c
+++ b/lib/crypto/tests/aes_cbc_macs_kunit.c
@@ -141,75 +141,10 @@ static void test_aes_xcbcmac_rfc3566(struct kunit *test)
KUNIT_ASSERT_MEMEQ(test, actual_mac, expected_mac, AES_BLOCK_SIZE);
}
-static void test_aes_cbcmac_rfc3610(struct kunit *test)
-{
- /*
- * The following AES-CBC-MAC test vector is extracted from RFC 3610
- * Packet Vector #11. It required some rearrangement to get the actual
- * input to AES-CBC-MAC from the values given.
- */
- static const u8 raw_key[AES_KEYSIZE_128] = {
- 0xc0, 0xc1, 0xc2, 0xc3, 0xc4, 0xc5, 0xc6, 0xc7,
- 0xc8, 0xc9, 0xca, 0xcb, 0xcc, 0xcd, 0xce, 0xcf,
- };
- const size_t unpadded_data_len = 52;
- static const u8 data[64] = {
- /* clang-format off */
- /* CCM header */
- 0x61, 0x00, 0x00, 0x00, 0x0d, 0x0c, 0x0b, 0x0a,
- 0xa0, 0xa1, 0xa2, 0xa3, 0xa4, 0xa5, 0x00, 0x14,
- /* CCM additional authentication blocks */
- 0x00, 0x0c, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05,
- 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x00, 0x00,
- /* CCM message blocks */
- 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13,
- 0x14, 0x15, 0x16, 0x17, 0x18, 0x19, 0x1a, 0x1b,
- 0x1c, 0x1d, 0x1e, 0x1f, 0x00, 0x00, 0x00, 0x00,
- 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
- /* clang-format on */
- };
- static const u8 expected_mac[AES_BLOCK_SIZE] = {
- 0x6b, 0x5e, 0x24, 0x34, 0x12, 0xcc, 0xc2, 0xad,
- 0x6f, 0x1b, 0x11, 0xc3, 0xa1, 0xa9, 0xd8, 0xbc,
- };
- struct aes_enckey key;
- struct aes_cbcmac_ctx ctx;
- u8 actual_mac[AES_BLOCK_SIZE];
- int err;
-
- err = aes_prepareenckey(&key, raw_key, sizeof(raw_key));
- KUNIT_ASSERT_EQ(test, err, 0);
-
- /*
- * Trailing zeroes should not affect the CBC-MAC value, up to the next
- * AES block boundary.
- */
- for (size_t data_len = unpadded_data_len; data_len <= sizeof(data);
- data_len++) {
- aes_cbcmac_init(&ctx, &key);
- aes_cbcmac_update(&ctx, data, data_len);
- aes_cbcmac_final(&ctx, actual_mac);
- KUNIT_ASSERT_MEMEQ(test, actual_mac, expected_mac,
- AES_BLOCK_SIZE);
-
- /* Incremental computations should produce the same result. */
- for (size_t part1_len = 0; part1_len <= data_len; part1_len++) {
- aes_cbcmac_init(&ctx, &key);
- aes_cbcmac_update(&ctx, data, part1_len);
- aes_cbcmac_update(&ctx, &data[part1_len],
- data_len - part1_len);
- aes_cbcmac_final(&ctx, actual_mac);
- KUNIT_ASSERT_MEMEQ(test, actual_mac, expected_mac,
- AES_BLOCK_SIZE);
- }
- }
-}
-
static struct kunit_case aes_cbc_macs_test_cases[] = {
HASH_KUNIT_CASES,
KUNIT_CASE(test_aes_cmac_rfc4493),
KUNIT_CASE(test_aes_xcbcmac_rfc3566),
- KUNIT_CASE(test_aes_cbcmac_rfc3610),
KUNIT_CASE(benchmark_hash),
{},
};
@@ -222,7 +157,6 @@ static struct kunit_suite aes_cbc_macs_test_suite = {
};
kunit_test_suite(aes_cbc_macs_test_suite);
-MODULE_DESCRIPTION(
- "KUnit tests and benchmark for AES-CMAC, AES-XCBC-MAC, and AES-CBC-MAC");
+MODULE_DESCRIPTION("KUnit tests and benchmark for AES-CMAC and AES-XCBC-MAC");
MODULE_IMPORT_NS("CRYPTO_INTERNAL");
MODULE_LICENSE("GPL");
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 18/33] fscrypt: Use aes_ecb_encrypt() for v1 key derivation
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (16 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 17/33] lib/crypto: aes: Remove aes_cbcmac_* functions Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 19/33] KEYS: encrypted: Use AES-CBC library instead of crypto_skcipher Eric Biggers
` (14 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there's a library function aes_ecb_encrypt(), use it instead of
a loop of aes_encrypt() calls.
Note that the use of AES-ECB here is purely for the key derivation
function used by the deprecated v1 encryption policies. v2 encryption
policies use HKDF-SHA512 instead. Nevertheless, the original version
has to continue to be supported for backwards compatibility.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
fs/crypto/Kconfig | 2 +-
fs/crypto/keysetup_v1.c | 5 ++---
2 files changed, 3 insertions(+), 4 deletions(-)
diff --git a/fs/crypto/Kconfig b/fs/crypto/Kconfig
index 983d8ad1f417..7662d8c45f07 100644
--- a/fs/crypto/Kconfig
+++ b/fs/crypto/Kconfig
@@ -3,7 +3,7 @@ config FS_ENCRYPTION
bool "FS Encryption (Per-file encryption)"
select CRYPTO
select CRYPTO_SKCIPHER
- select CRYPTO_LIB_AES
+ select CRYPTO_LIB_AES_ECB # for deprecated v1 key derivation function
select CRYPTO_LIB_SHA256
select CRYPTO_LIB_SHA512
select KEYS
diff --git a/fs/crypto/keysetup_v1.c b/fs/crypto/keysetup_v1.c
index e6e527c73f16..70385d82fa73 100644
--- a/fs/crypto/keysetup_v1.c
+++ b/fs/crypto/keysetup_v1.c
@@ -20,7 +20,7 @@
* managed alongside the master keys in the filesystem-level keyring)
*/
-#include <crypto/aes.h>
+#include <crypto/aes-ecb.h>
#include <crypto/utils.h>
#include <keys/user-type.h>
#include <linux/hashtable.h>
@@ -240,8 +240,7 @@ static int setup_v1_file_key_derived(struct fscrypt_inode_info *ci,
static_assert(FSCRYPT_FILE_NONCE_SIZE == AES_KEYSIZE_128);
aes_prepareenckey(&aes, ci->ci_nonce, FSCRYPT_FILE_NONCE_SIZE);
- for (unsigned int i = 0; i < derived_keysize; i += AES_BLOCK_SIZE)
- aes_encrypt(&aes, &derived_key[i], &raw_master_key[i]);
+ aes_ecb_encrypt(derived_key, raw_master_key, derived_keysize, &aes);
err = fscrypt_set_per_file_enc_key(ci, derived_key);
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 19/33] KEYS: encrypted: Use AES-CBC library instead of crypto_skcipher
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (17 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 18/33] fscrypt: Use aes_ecb_encrypt() for v1 key derivation Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 20/33] KEYS: trusted: dcp: Use AES-GCM library instead of crypto_aead Eric Biggers
` (13 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there's a library API for AES-CBC, use it instead of a
"cbc(aes)" crypto_skcipher. This significantly simplifies the code.
Note that this use of AES-CBC remains questionable (an AEAD should be
used instead), but it remains for backwards compatibility.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
security/keys/Kconfig | 4 +-
security/keys/encrypted-keys/encrypted.c | 194 +++++++----------------
2 files changed, 62 insertions(+), 136 deletions(-)
diff --git a/security/keys/Kconfig b/security/keys/Kconfig
index f4510d8cb485..d482f632791a 100644
--- a/security/keys/Kconfig
+++ b/security/keys/Kconfig
@@ -83,9 +83,7 @@ endif
config ENCRYPTED_KEYS
tristate "ENCRYPTED KEYS"
- select CRYPTO
- select CRYPTO_AES
- select CRYPTO_CBC
+ select CRYPTO_LIB_AES_CBC
select CRYPTO_LIB_SHA256
help
This option provides support for create/encrypting/decrypting keys
diff --git a/security/keys/encrypted-keys/encrypted.c b/security/keys/encrypted-keys/encrypted.c
index 59cb77b237b3..3dae2c29496b 100644
--- a/security/keys/encrypted-keys/encrypted.c
+++ b/security/keys/encrypted-keys/encrypted.c
@@ -25,11 +25,9 @@
#include <linux/key-type.h>
#include <linux/random.h>
#include <linux/rcupdate.h>
-#include <linux/scatterlist.h>
#include <linux/ctype.h>
-#include <crypto/aes.h>
+#include <crypto/aes-cbc.h>
#include <crypto/sha2.h>
-#include <crypto/skcipher.h>
#include <crypto/utils.h>
#include "encrypted.h"
@@ -37,17 +35,15 @@
static const char KEY_TRUSTED_PREFIX[] = "trusted:";
static const char KEY_USER_PREFIX[] = "user:";
-static const char blkcipher_alg[] = "cbc(aes)";
static const char key_format_default[] = "default";
static const char key_format_ecryptfs[] = "ecryptfs";
static const char key_format_enc32[] = "enc32";
-static unsigned int ivsize;
-static int blksize;
#define KEY_TRUSTED_PREFIX_LEN (sizeof (KEY_TRUSTED_PREFIX) - 1)
#define KEY_USER_PREFIX_LEN (sizeof (KEY_USER_PREFIX) - 1)
#define KEY_ECRYPTFS_DESC_LEN 16
#define HASH_SIZE SHA256_DIGEST_SIZE
+#define IV_SIZE AES_BLOCK_SIZE /* AES-CBC initialization vector size in bytes */
#define MAX_DATA_SIZE 4096
#define MIN_DATA_SIZE 20
#define KEY_ENC32_PAYLOAD_LEN 32
@@ -79,22 +75,6 @@ module_param(user_decrypted_data, bool, 0);
MODULE_PARM_DESC(user_decrypted_data,
"Allow instantiation of encrypted keys using provided decrypted data");
-static int aes_get_sizes(void)
-{
- struct crypto_skcipher *tfm;
-
- tfm = crypto_alloc_skcipher(blkcipher_alg, 0, CRYPTO_ALG_ASYNC);
- if (IS_ERR(tfm)) {
- pr_err("encrypted_key: failed to alloc_cipher (%ld)\n",
- PTR_ERR(tfm));
- return PTR_ERR(tfm);
- }
- ivsize = crypto_skcipher_ivsize(tfm);
- blksize = crypto_skcipher_blocksize(tfm);
- crypto_free_skcipher(tfm);
- return 0;
-}
-
/*
* valid_ecryptfs_desc - verify the description of a new/loaded encrypted key
*
@@ -354,39 +334,6 @@ static int get_derived_key(u8 *derived_key, enum derived_key_type key_type,
return 0;
}
-static struct skcipher_request *init_skcipher_req(const u8 *key,
- unsigned int key_len)
-{
- struct skcipher_request *req;
- struct crypto_skcipher *tfm;
- int ret;
-
- tfm = crypto_alloc_skcipher(blkcipher_alg, 0, CRYPTO_ALG_ASYNC);
- if (IS_ERR(tfm)) {
- pr_err("encrypted_key: failed to load %s transform (%ld)\n",
- blkcipher_alg, PTR_ERR(tfm));
- return ERR_CAST(tfm);
- }
-
- ret = crypto_skcipher_setkey(tfm, key, key_len);
- if (ret < 0) {
- pr_err("encrypted_key: failed to setkey (%d)\n", ret);
- crypto_free_skcipher(tfm);
- return ERR_PTR(ret);
- }
-
- req = skcipher_request_alloc(tfm, GFP_KERNEL);
- if (!req) {
- pr_err("encrypted_key: failed to allocate request for %s\n",
- blkcipher_alg);
- crypto_free_skcipher(tfm);
- return ERR_PTR(-ENOMEM);
- }
-
- skcipher_request_set_callback(req, 0, NULL, NULL);
- return req;
-}
-
static struct key *request_master_key(struct encrypted_key_payload *epayload,
const u8 **master_key, size_t *master_keylen)
{
@@ -427,42 +374,35 @@ static int derived_key_encrypt(struct encrypted_key_payload *epayload,
const u8 *derived_key,
unsigned int derived_keylen)
{
- struct scatterlist sg_in[2];
- struct scatterlist sg_out[1];
- struct crypto_skcipher *tfm;
- struct skcipher_request *req;
+ struct aes_enckey key;
unsigned int encrypted_datalen;
- u8 iv[AES_BLOCK_SIZE];
+ u8 iv[IV_SIZE];
int ret;
- encrypted_datalen = roundup(epayload->decrypted_datalen, blksize);
+ ret = aes_prepareenckey(&key, derived_key, derived_keylen);
+ if (ret < 0) /* Should never fail here, since a valid length was used */
+ return ret;
- req = init_skcipher_req(derived_key, derived_keylen);
- ret = PTR_ERR(req);
- if (IS_ERR(req))
- goto out;
dump_decrypted_data(epayload);
- sg_init_table(sg_in, 2);
- sg_set_buf(&sg_in[0], epayload->decrypted_data,
- epayload->decrypted_datalen);
- sg_set_page(&sg_in[1], ZERO_PAGE(0), AES_BLOCK_SIZE, 0);
-
- sg_init_table(sg_out, 1);
- sg_set_buf(sg_out, epayload->encrypted_data, encrypted_datalen);
-
+ /*
+ * Pad the plaintext with zeros up to the next multiple of
+ * AES_BLOCK_SIZE, to make it have a valid length for AES-CBC.
+ * To do this without needing a temporary buffer, copy the plaintext
+ * into ->encrypted_data, pad it, and encrypt it in-place.
+ */
+ encrypted_datalen =
+ roundup(epayload->decrypted_datalen, AES_BLOCK_SIZE);
+ memset(epayload->encrypted_data, 0, encrypted_datalen);
+ memcpy(epayload->encrypted_data, epayload->decrypted_data,
+ epayload->decrypted_datalen);
memcpy(iv, epayload->iv, sizeof(iv));
- skcipher_request_set_crypt(req, sg_in, sg_out, encrypted_datalen, iv);
- ret = crypto_skcipher_encrypt(req);
- tfm = crypto_skcipher_reqtfm(req);
- skcipher_request_free(req);
- crypto_free_skcipher(tfm);
- if (ret < 0)
- pr_err("encrypted_key: failed to encrypt (%d)\n", ret);
- else
- dump_encrypted_data(epayload, encrypted_datalen);
-out:
- return ret;
+ aes_cbc_encrypt(epayload->encrypted_data, epayload->encrypted_data,
+ encrypted_datalen, iv, &key);
+ dump_encrypted_data(epayload, encrypted_datalen);
+
+ memzero_explicit(&key, sizeof(key));
+ return 0;
}
static int datablob_hmac_append(struct encrypted_key_payload *epayload,
@@ -528,45 +468,37 @@ static int derived_key_decrypt(struct encrypted_key_payload *epayload,
const u8 *derived_key,
unsigned int derived_keylen)
{
- struct scatterlist sg_in[1];
- struct scatterlist sg_out[2];
- struct crypto_skcipher *tfm;
- struct skcipher_request *req;
+ struct aes_key key;
unsigned int encrypted_datalen;
- u8 iv[AES_BLOCK_SIZE];
- u8 *pad;
+ u8 iv[IV_SIZE];
+ u8 *tmp;
int ret;
- /* Throwaway buffer to hold the unused zero padding at the end */
- pad = kmalloc(AES_BLOCK_SIZE, GFP_KERNEL);
- if (!pad)
- return -ENOMEM;
+ ret = aes_preparekey(&key, derived_key, derived_keylen);
+ if (ret < 0) /* Should never fail here, since a valid length was used */
+ return ret;
- encrypted_datalen = roundup(epayload->decrypted_datalen, blksize);
- req = init_skcipher_req(derived_key, derived_keylen);
- ret = PTR_ERR(req);
- if (IS_ERR(req))
+ /*
+ * The plaintext was padded before encryption, so decrypt it into a
+ * temporary buffer that has space for the padding.
+ */
+ encrypted_datalen =
+ roundup(epayload->decrypted_datalen, AES_BLOCK_SIZE);
+ tmp = kmalloc(encrypted_datalen, GFP_KERNEL);
+ if (!tmp) {
+ ret = -ENOMEM;
goto out;
+ }
dump_encrypted_data(epayload, encrypted_datalen);
-
- sg_init_table(sg_in, 1);
- sg_init_table(sg_out, 2);
- sg_set_buf(sg_in, epayload->encrypted_data, encrypted_datalen);
- sg_set_buf(&sg_out[0], epayload->decrypted_data,
- epayload->decrypted_datalen);
- sg_set_buf(&sg_out[1], pad, AES_BLOCK_SIZE);
-
memcpy(iv, epayload->iv, sizeof(iv));
- skcipher_request_set_crypt(req, sg_in, sg_out, encrypted_datalen, iv);
- ret = crypto_skcipher_decrypt(req);
- tfm = crypto_skcipher_reqtfm(req);
- skcipher_request_free(req);
- crypto_free_skcipher(tfm);
- if (ret < 0)
- goto out;
+ aes_cbc_decrypt(tmp, epayload->encrypted_data, encrypted_datalen, iv,
+ &key);
+ memcpy(epayload->decrypted_data, tmp, epayload->decrypted_datalen);
dump_decrypted_data(epayload);
+ ret = 0;
out:
- kfree(pad);
+ kfree_sensitive(tmp);
+ memzero_explicit(&key, sizeof(key));
return ret;
}
@@ -630,10 +562,10 @@ static struct encrypted_key_payload *encrypted_key_alloc(struct key *key,
}
}
- encrypted_datalen = roundup(decrypted_datalen, blksize);
+ encrypted_datalen = roundup(decrypted_datalen, AES_BLOCK_SIZE);
- datablob_len = format_len + 1 + strlen(master_desc) + 1
- + strlen(datalen) + 1 + ivsize + 1 + encrypted_datalen;
+ datablob_len = format_len + 1 + strlen(master_desc) + 1 +
+ strlen(datalen) + 1 + IV_SIZE + 1 + encrypted_datalen;
ret = key_payload_reserve(key, payload_datalen + datablob_len
+ HASH_SIZE + 1);
@@ -664,13 +596,14 @@ static int encrypted_key_decrypt(struct encrypted_key_payload *epayload,
size_t asciilen;
int ret;
- encrypted_datalen = roundup(epayload->decrypted_datalen, blksize);
- asciilen = (ivsize + 1 + encrypted_datalen + HASH_SIZE) * 2;
+ encrypted_datalen =
+ roundup(epayload->decrypted_datalen, AES_BLOCK_SIZE);
+ asciilen = (IV_SIZE + 1 + encrypted_datalen + HASH_SIZE) * 2;
if (strlen(hex_encoded_iv) != asciilen)
return -EINVAL;
- hex_encoded_data = hex_encoded_iv + (2 * ivsize) + 2;
- ret = hex2bin(epayload->iv, hex_encoded_iv, ivsize);
+ hex_encoded_data = hex_encoded_iv + (2 * IV_SIZE) + 2;
+ ret = hex2bin(epayload->iv, hex_encoded_iv, IV_SIZE);
if (ret < 0)
return -EINVAL;
ret = hex2bin(epayload->encrypted_data, hex_encoded_data,
@@ -719,7 +652,7 @@ static void __ekey_init(struct encrypted_key_payload *epayload,
epayload->master_desc = epayload->format + format_len + 1;
epayload->datalen = epayload->master_desc + strlen(master_desc) + 1;
epayload->iv = epayload->datalen + strlen(datalen) + 1;
- epayload->encrypted_data = epayload->iv + ivsize + 1;
+ epayload->encrypted_data = epayload->iv + IV_SIZE + 1;
epayload->decrypted_data = epayload->payload_data;
if (!format)
@@ -763,11 +696,11 @@ static int encrypted_init(struct encrypted_key_payload *epayload,
if (hex_encoded_iv) {
ret = encrypted_key_decrypt(epayload, format, hex_encoded_iv);
} else if (decrypted_data) {
- get_random_bytes(epayload->iv, ivsize);
+ get_random_bytes(epayload->iv, IV_SIZE);
ret = hex2bin(epayload->decrypted_data, decrypted_data,
epayload->decrypted_datalen);
} else {
- get_random_bytes(epayload->iv, ivsize);
+ get_random_bytes(epayload->iv, IV_SIZE);
get_random_bytes(epayload->decrypted_data, epayload->decrypted_datalen);
}
return ret;
@@ -884,7 +817,7 @@ static int encrypted_update(struct key *key, struct key_preparsed_payload *prep)
__ekey_init(new_epayload, epayload->format, new_master_desc,
epayload->datalen);
- memcpy(new_epayload->iv, epayload->iv, ivsize);
+ memcpy(new_epayload->iv, epayload->iv, IV_SIZE);
memcpy(new_epayload->payload_data, epayload->payload_data,
epayload->payload_datalen);
@@ -918,9 +851,9 @@ static long encrypted_read(const struct key *key, char *buffer,
epayload = dereference_key_locked(key);
/* returns the hex encoded iv, encrypted-data, and hmac as ascii */
- asciiblob_len = epayload->datablob_len + ivsize + 1
- + roundup(epayload->decrypted_datalen, blksize)
- + (HASH_SIZE * 2);
+ asciiblob_len = epayload->datablob_len + IV_SIZE + 1 +
+ roundup(epayload->decrypted_datalen, AES_BLOCK_SIZE) +
+ (HASH_SIZE * 2);
if (!buffer || buflen < asciiblob_len)
return asciiblob_len;
@@ -982,11 +915,6 @@ EXPORT_SYMBOL_GPL(key_type_encrypted);
static int __init init_encrypted(void)
{
- int ret;
-
- ret = aes_get_sizes();
- if (ret < 0)
- return ret;
return register_key_type(&key_type_encrypted);
}
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 20/33] KEYS: trusted: dcp: Use AES-GCM library instead of crypto_aead
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (18 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 19/33] KEYS: encrypted: Use AES-CBC library instead of crypto_skcipher Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 21/33] libceph: Use AES-CBC library in ceph_aes_crypt() Eric Biggers
` (12 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there's a library API for AES-GCM, use it instead of a
"gcm(aes)" crypto_aead. This significantly simplifies the code.
Note, I've left unchanged two pre-existing issues with this code:
- trusted_key_unseal() doesn't validate that the decrypted key actually
fits in the buffer to which it's written.
- dcp_blob_fmt::nonce is unnecessarily long, at 16 bytes rather than the
12 bytes that are actually used by the AES-GCM operations. Probably
unfixable since it's part of the blob format.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
security/keys/trusted-keys/Kconfig | 1 +
security/keys/trusted-keys/trusted_dcp.c | 69 +++++-------------------
2 files changed, 15 insertions(+), 55 deletions(-)
diff --git a/security/keys/trusted-keys/Kconfig b/security/keys/trusted-keys/Kconfig
index e5a4a53aeab2..361882191ff9 100644
--- a/security/keys/trusted-keys/Kconfig
+++ b/security/keys/trusted-keys/Kconfig
@@ -63,6 +63,7 @@ config TRUSTED_KEYS_DCP
bool "DCP-based trusted keys"
depends on CRYPTO_DEV_MXS_DCP >= TRUSTED_KEYS
default y
+ select CRYPTO_LIB_AES_GCM
select HAVE_TRUSTED_KEYS_DEBUG
select HAVE_TRUSTED_KEYS
help
diff --git a/security/keys/trusted-keys/trusted_dcp.c b/security/keys/trusted-keys/trusted_dcp.c
index 7b6eb655df0c..f9b2e0ad162d 100644
--- a/security/keys/trusted-keys/trusted_dcp.c
+++ b/security/keys/trusted-keys/trusted_dcp.c
@@ -3,10 +3,8 @@
* Copyright (C) 2021 sigma star gmbh
*/
-#include <crypto/aead.h>
#include <crypto/aes.h>
-#include <crypto/algapi.h>
-#include <crypto/gcm.h>
+#include <crypto/aes-gcm.h>
#include <crypto/skcipher.h>
#include <keys/trusted-type.h>
#include <linux/key-type.h>
@@ -126,64 +124,25 @@ static int do_dcp_crypto(u8 *in, u8 *out, bool do_encrypt)
return res;
}
-static int do_aead_crypto(u8 *in, u8 *out, size_t len, u8 *key, u8 *nonce,
+static int do_aead_crypto(u8 *in, u8 *out, size_t len,
+ const u8 key[AES_KEYSIZE_128], const u8 nonce[12],
bool do_encrypt)
{
- struct aead_request *aead_req = NULL;
- struct scatterlist src_sg, dst_sg;
- struct crypto_aead *aead;
+ struct aes_gcm_key gcm;
int ret;
- DECLARE_CRYPTO_WAIT(wait);
-
- aead = crypto_alloc_aead("gcm(aes)", 0, CRYPTO_ALG_ASYNC);
- if (IS_ERR(aead)) {
- ret = PTR_ERR(aead);
- goto out;
- }
- ret = crypto_aead_setauthsize(aead, DCP_BLOB_AUTHLEN);
- if (ret < 0) {
- pr_err("Can't set crypto auth tag len: %d\n", ret);
- goto free_aead;
- }
-
- aead_req = aead_request_alloc(aead, GFP_KERNEL);
- if (!aead_req) {
- ret = -ENOMEM;
- goto free_aead;
- }
+ ret = aes_gcm_preparekey(&gcm, key, AES_KEYSIZE_128, DCP_BLOB_AUTHLEN);
+ if (ret) /* Should never fail here, since valid lengths were used. */
+ return ret;
- sg_init_one(&src_sg, in, len);
if (do_encrypt) {
- /*
- * If we encrypt our buffer has extra space for the auth tag.
- */
- sg_init_one(&dst_sg, out, len + DCP_BLOB_AUTHLEN);
+ aes_gcm_encrypt(out, out + len, in, len, NULL, 0, nonce, &gcm);
+ ret = 0;
} else {
- sg_init_one(&dst_sg, out, len);
+ ret = aes_gcm_decrypt(out, in, in + len, len, NULL, 0, nonce,
+ &gcm);
}
-
- aead_request_set_crypt(aead_req, &src_sg, &dst_sg, len, nonce);
- aead_request_set_callback(aead_req, CRYPTO_TFM_REQ_MAY_SLEEP,
- crypto_req_done, &wait);
- aead_request_set_ad(aead_req, 0);
-
- if (crypto_aead_setkey(aead, key, AES_KEYSIZE_128)) {
- pr_err("Can't set crypto AEAD key\n");
- ret = -EINVAL;
- goto free_req;
- }
-
- if (do_encrypt)
- ret = crypto_wait_req(crypto_aead_encrypt(aead_req), &wait);
- else
- ret = crypto_wait_req(crypto_aead_decrypt(aead_req), &wait);
-
-free_req:
- aead_request_free(aead_req);
-free_aead:
- crypto_free_aead(aead);
-out:
+ memzero_explicit(&gcm, sizeof(gcm));
return ret;
}
@@ -273,8 +232,8 @@ static int trusted_dcp_unseal(struct trusted_key_payload *p, char *datablob)
goto out;
}
- ret = do_aead_crypto(b->payload, p->key, p->key_len + DCP_BLOB_AUTHLEN,
- plain_blob_key, b->nonce, false);
+ ret = do_aead_crypto(b->payload, p->key, p->key_len, plain_blob_key,
+ b->nonce, false);
if (ret) {
pr_err("Unwrap of DCP payload failed: %i\n", ret);
goto out;
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 21/33] libceph: Use AES-CBC library in ceph_aes_crypt()
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (19 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 20/33] KEYS: trusted: dcp: Use AES-GCM library instead of crypto_aead Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 22/33] libceph: Reimplement messenger v2 encryption using AES-GCM library Eric Biggers
` (11 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there's a library API for AES-CBC, use it instead of a
"cbc(aes)" crypto_skcipher. This significantly simplifies the code.
Notably, the new ceph_aes_crypt() just operates on the actual buffer
directly. It no longer needs to be converted into a scatterlist.
This conversion also eliminates the broken code that intended to
allocate crypto_skcipher objects from a GFP_NOIO context. That isn't
actually supported: crypto_alloc_skcipher() takes crypto_alg_sem which
isn't GFP_NOIO safe, and it can also load kernel modules.
Note that ceph_aes_crypt() verifies only a single padding byte rather
than all of them. That's probably a bug. But I've left it unchanged,
as fixing it is outside the scope of this commit.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
net/ceph/Kconfig | 3 +--
net/ceph/crypto.c | 66 ++++++++++-------------------------------------
net/ceph/crypto.h | 3 ++-
3 files changed, 16 insertions(+), 56 deletions(-)
diff --git a/net/ceph/Kconfig b/net/ceph/Kconfig
index 7e2528cde4b9..74aa2817253b 100644
--- a/net/ceph/Kconfig
+++ b/net/ceph/Kconfig
@@ -3,10 +3,9 @@ config CEPH_LIB
tristate "Ceph core library"
depends on INET
select CRC32
- select CRYPTO_AES
- select CRYPTO_CBC
select CRYPTO_GCM
select CRYPTO_KRB5
+ select CRYPTO_LIB_AES_CBC
select CRYPTO_LIB_SHA256
select CRYPTO
select KEYS
diff --git a/net/ceph/crypto.c b/net/ceph/crypto.c
index de7496791c91..83bbef71bfaf 100644
--- a/net/ceph/crypto.c
+++ b/net/ceph/crypto.c
@@ -6,9 +6,8 @@
#include <linux/scatterlist.h>
#include <linux/sched.h>
#include <linux/slab.h>
-#include <crypto/aes.h>
+#include <crypto/aes-cbc.h>
#include <crypto/krb5.h>
-#include <crypto/skcipher.h>
#include <linux/key-type.h>
#include <linux/sched/mm.h>
@@ -17,27 +16,6 @@
#include <linux/ceph/decode.h>
#include "crypto.h"
-static int set_aes_tfm(struct ceph_crypto_key *key)
-{
- unsigned int noio_flag;
- int ret;
-
- noio_flag = memalloc_noio_save();
- key->aes_tfm = crypto_alloc_sync_skcipher("cbc(aes)", 0, 0);
- memalloc_noio_restore(noio_flag);
- if (IS_ERR(key->aes_tfm)) {
- ret = PTR_ERR(key->aes_tfm);
- key->aes_tfm = NULL;
- return ret;
- }
-
- ret = crypto_sync_skcipher_setkey(key->aes_tfm, key->key, key->len);
- if (ret)
- return ret;
-
- return 0;
-}
-
static int set_krb5_tfms(struct ceph_crypto_key *key, const u32 *key_usages,
int key_usage_cnt)
{
@@ -82,7 +60,7 @@ int ceph_crypto_key_prepare(struct ceph_crypto_key *key,
case CEPH_CRYPTO_NONE:
return 0; /* nothing to do */
case CEPH_CRYPTO_AES:
- return set_aes_tfm(key);
+ return aes_preparekey(&key->aes_key, key->key, key->len);
case CEPH_CRYPTO_AES256KRB5:
hmac_sha256_preparekey(&key->hmac_key, key->key, key->len);
return set_krb5_tfms(key, key_usages, key_usage_cnt);
@@ -174,10 +152,7 @@ void ceph_crypto_key_destroy(struct ceph_crypto_key *key)
key->key = NULL;
if (key->type == CEPH_CRYPTO_AES) {
- if (key->aes_tfm) {
- crypto_free_sync_skcipher(key->aes_tfm);
- key->aes_tfm = NULL;
- }
+ memzero_explicit(&key->aes_key, sizeof(key->aes_key));
} else if (key->type == CEPH_CRYPTO_AES256KRB5) {
memzero_explicit(&key->hmac_key, sizeof(key->hmac_key));
for (i = 0; i < ARRAY_SIZE(key->krb5_tfms); i++) {
@@ -264,26 +239,20 @@ static void teardown_sgtable(struct sg_table *sgt)
static int ceph_aes_crypt(const struct ceph_crypto_key *key, bool encrypt,
void *buf, int buf_len, int in_len, int *pout_len)
{
- SYNC_SKCIPHER_REQUEST_ON_STACK(req, key->aes_tfm);
- struct sg_table sgt;
- struct scatterlist prealloc_sg;
char iv[AES_BLOCK_SIZE] __aligned(8);
int pad_byte = AES_BLOCK_SIZE - (in_len & (AES_BLOCK_SIZE - 1));
int crypt_len = encrypt ? in_len + pad_byte : in_len;
- int ret;
WARN_ON(crypt_len > buf_len);
+ if (crypt_len <= 0 || crypt_len % AES_BLOCK_SIZE != 0) {
+ pr_err("%s: got bad crypt_len %d for %scrypt\n", __func__,
+ crypt_len, encrypt ? "en" : "de");
+ return -EINVAL;
+ }
if (encrypt)
memset(buf + in_len, pad_byte, pad_byte);
- ret = setup_sgtable(&sgt, &prealloc_sg, buf, crypt_len);
- if (ret)
- return ret;
memcpy(iv, aes_iv, AES_BLOCK_SIZE);
- skcipher_request_set_sync_tfm(req, key->aes_tfm);
- skcipher_request_set_callback(req, 0, NULL, NULL);
- skcipher_request_set_crypt(req, sgt.sgl, sgt.sgl, crypt_len, iv);
-
/*
print_hex_dump(KERN_ERR, "key: ", DUMP_PREFIX_NONE, 16, 1,
key->key, key->len, 1);
@@ -291,15 +260,10 @@ static int ceph_aes_crypt(const struct ceph_crypto_key *key, bool encrypt,
buf, crypt_len, 1);
*/
if (encrypt)
- ret = crypto_skcipher_encrypt(req);
+ aes_cbc_encrypt(buf, buf, crypt_len, iv, &key->aes_key);
else
- ret = crypto_skcipher_decrypt(req);
- skcipher_request_zero(req);
- if (ret) {
- pr_err("%s %scrypt failed: %d\n", __func__,
- encrypt ? "en" : "de", ret);
- goto out_sgt;
- }
+ aes_cbc_decrypt(buf, buf, crypt_len, iv, &key->aes_key);
+
/*
print_hex_dump(KERN_ERR, "out: ", DUMP_PREFIX_NONE, 16, 1,
buf, crypt_len, 1);
@@ -315,14 +279,10 @@ static int ceph_aes_crypt(const struct ceph_crypto_key *key, bool encrypt,
} else {
pr_err("%s got bad padding %d on in_len %d\n",
__func__, pad_byte, in_len);
- ret = -EPERM;
- goto out_sgt;
+ return -EPERM;
}
}
-
-out_sgt:
- teardown_sgtable(&sgt);
- return ret;
+ return 0;
}
static int ceph_krb5_encrypt(const struct ceph_crypto_key *key, int usage_slot,
diff --git a/net/ceph/crypto.h b/net/ceph/crypto.h
index 3a2ade15abbc..d4d97565ebfb 100644
--- a/net/ceph/crypto.h
+++ b/net/ceph/crypto.h
@@ -2,6 +2,7 @@
#ifndef _FS_CEPH_CRYPTO_H
#define _FS_CEPH_CRYPTO_H
+#include <crypto/aes.h>
#include <crypto/sha2.h>
#include <linux/ceph/types.h>
#include <linux/ceph/buffer.h>
@@ -19,7 +20,7 @@ struct ceph_crypto_key {
void *key;
union {
- struct crypto_sync_skcipher *aes_tfm;
+ struct aes_key aes_key;
struct {
struct hmac_sha256_key hmac_key;
const struct krb5_enctype *krb5_type;
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 22/33] libceph: Reimplement messenger v2 encryption using AES-GCM library
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (20 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 21/33] libceph: Use AES-CBC library in ceph_aes_crypt() Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 23/33] wifi: mac80211: Use AES-CTR library in fils_aead.c Eric Biggers
` (10 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Significantly rework the implementation of Ceph messenger v2 to use the
new AES-GCM library instead of a "gcm(aes)" crypto_aead.
This especially takes advantage of the library's support for virtual
addresses and incremental en/decryption. crypto_aead only supports
scatterlists and has no support for incremental operations, which
created significant difficulties for the Ceph messenger v2 code.
- For decryption (receive), the new code receives and decrypts the data
directly into its destination buffers, or into a bounce page when the
existing rxbounce mount option is enabled. This unifies the code
significantly with the non-secure case.
Previously, the entire received message tail (potentially tens of
megabytes) was buffered in a list of bounce pages. Then two
scatterlists were generated: one for all the different destination
buffers and one for the bounce pages. Besides being very complex, the
memory use was likely problematic as well. All that is removed.
- For encryption (send), similar changes are made. It differs slightly
in that a bounce page is still needed unconditionally. But the single
page just keeps getting reused, which is much more efficient.
As a bonus, this conversion also eliminates the broken code that
intended to allocate crypto_aead objects from a GFP_NOIO context. That
isn't actually supported: crypto_alloc_aead() takes crypto_alg_sem which
isn't GFP_NOIO safe, and it can also load kernel modules.
Note: incremental GCM decryption is sometimes perceived as dangerous, as
it invites users to consume data before the auth tag is verified. It
does seem to be safe here though, as success isn't reported up the stack
until the very end. (Also, the old code didn't verify the tag until the
data had already been written into the destination buffers either.)
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
include/linux/ceph/messenger.h | 22 +-
net/ceph/Kconfig | 2 +-
net/ceph/messenger_v2.c | 963 ++++++++++++---------------------
3 files changed, 357 insertions(+), 630 deletions(-)
diff --git a/include/linux/ceph/messenger.h b/include/linux/ceph/messenger.h
index 6aa4c6478c9f..eda9729023ef 100644
--- a/include/linux/ceph/messenger.h
+++ b/include/linux/ceph/messenger.h
@@ -2,9 +2,9 @@
#ifndef __FS_CEPH_MESSENGER_H
#define __FS_CEPH_MESSENGER_H
+#include <crypto/aes-gcm.h>
#include <crypto/sha2.h>
#include <linux/bvec.h>
-#include <linux/crypto.h>
#include <linux/kref.h>
#include <linux/mutex.h>
#include <linux/net.h>
@@ -401,8 +401,7 @@ struct ceph_connection_v2_info {
struct iov_iter out_iter;
struct kvec out_kvecs[8]; /* sendmsg */
- struct bio_vec out_bvec; /* sendpage (out_cursor, out_zero),
- sendmsg (out_enc_pages) */
+ struct bio_vec out_bvec; /* sendpage (out_cursor, out_zero) */
int out_kvec_cnt;
int out_state; /* OUT_S_* */
@@ -415,20 +414,15 @@ struct ceph_connection_v2_info {
struct hmac_sha256_key hmac_key; /* post-auth signature */
bool hmac_key_set;
- struct crypto_aead *gcm_tfm; /* on-wire encryption */
- struct aead_request *gcm_req;
- struct crypto_wait gcm_wait;
+ struct aes_gcm_key gcm_key; /* on-wire encryption */
+ bool gcm_key_set;
struct ceph_gcm_nonce in_gcm_nonce;
struct ceph_gcm_nonce out_gcm_nonce;
- struct page **in_enc_pages;
- int in_enc_page_cnt;
- int in_enc_resid;
- int in_enc_i;
- struct page **out_enc_pages;
- int out_enc_page_cnt;
- int out_enc_resid;
- int out_enc_i;
+ struct aes_gcm_ctx in_gcm_ctx;
+ struct aes_gcm_ctx out_gcm_ctx;
+ u8 *out_ciphertext;
+ u8 *out_front_middle_ciphertext;
int con_mode; /* CEPH_CON_MODE_* */
diff --git a/net/ceph/Kconfig b/net/ceph/Kconfig
index 74aa2817253b..de9170c44ebb 100644
--- a/net/ceph/Kconfig
+++ b/net/ceph/Kconfig
@@ -3,9 +3,9 @@ config CEPH_LIB
tristate "Ceph core library"
depends on INET
select CRC32
- select CRYPTO_GCM
select CRYPTO_KRB5
select CRYPTO_LIB_AES_CBC
+ select CRYPTO_LIB_AES_GCM
select CRYPTO_LIB_SHA256
select CRYPTO
select KEYS
diff --git a/net/ceph/messenger_v2.c b/net/ceph/messenger_v2.c
index 05f6eea299fc..7d7a99b6c008 100644
--- a/net/ceph/messenger_v2.c
+++ b/net/ceph/messenger_v2.c
@@ -7,15 +7,13 @@
#include <linux/ceph/ceph_debug.h>
-#include <crypto/aead.h>
+#include <crypto/aes-gcm.h>
#include <crypto/sha2.h>
#include <crypto/utils.h>
#include <linux/bvec.h>
#include <linux/crc32c.h>
#include <linux/net.h>
-#include <linux/scatterlist.h>
#include <linux/socket.h>
-#include <linux/sched/mm.h>
#include <net/sock.h>
#include <net/tcp.h>
@@ -54,9 +52,9 @@
#define IN_S_HANDLE_PREAMBLE 1
#define IN_S_HANDLE_CONTROL 2
#define IN_S_HANDLE_CONTROL_REMAINDER 3
-#define IN_S_PREPARE_READ_DATA 4
-#define IN_S_PREPARE_READ_DATA_CONT 5
-#define IN_S_PREPARE_READ_ENC_PAGE 6
+#define IN_S_DECRYPT_FRONT_AND_MIDDLE 4
+#define IN_S_PREPARE_READ_DATA 5
+#define IN_S_PREPARE_READ_DATA_CONT 6
#define IN_S_PREPARE_SPARSE_DATA 7
#define IN_S_PREPARE_SPARSE_DATA_CONT 8
#define IN_S_HANDLE_EPILOGUE 9
@@ -64,18 +62,18 @@
#define OUT_S_QUEUE_DATA 1
#define OUT_S_QUEUE_DATA_CONT 2
-#define OUT_S_QUEUE_ENC_PAGE 3
-#define OUT_S_QUEUE_ZEROS 4
-#define OUT_S_FINISH_MESSAGE 5
-#define OUT_S_GET_NEXT 6
+#define OUT_S_QUEUE_ENC_DATA 3
+#define OUT_S_QUEUE_ENC_EPILOGUE 4
+#define OUT_S_QUEUE_ZEROS 5
+#define OUT_S_FINISH_MESSAGE 6
+#define OUT_S_GET_NEXT 7
#define CTRL_BODY(p) ((void *)(p) + CEPH_PREAMBLE_LEN)
-#define FRONT_PAD(p) ((void *)(p) + CEPH_EPILOGUE_SECURE_LEN)
-#define MIDDLE_PAD(p) (FRONT_PAD(p) + CEPH_GCM_BLOCK_LEN)
-#define DATA_PAD(p) (MIDDLE_PAD(p) + CEPH_GCM_BLOCK_LEN)
#define CEPH_MSG_FLAGS (MSG_DONTWAIT | MSG_NOSIGNAL)
+#define OUT_CIPHERTEXT_BUFFER_SIZE PAGE_SIZE
+
static int do_recvmsg(struct socket *sock, struct iov_iter *it)
{
struct msghdr msg = { .msg_flags = CEPH_MSG_FLAGS };
@@ -424,10 +422,12 @@ static int __tail_onwire_len(int front_len, int middle_len, int data_len,
padded_len(data_len) + CEPH_EPILOGUE_SECURE_LEN;
}
-static int tail_onwire_len(const struct ceph_msg *msg, bool secure)
+static int data_padding_and_epilogue_onwire_len(struct ceph_connection *con,
+ int data_len)
{
- return __tail_onwire_len(front_len(msg), middle_len(msg),
- data_len(msg), secure);
+ if (con_secure(con))
+ return padding_len(data_len) + CEPH_EPILOGUE_SECURE_LEN;
+ return CEPH_EPILOGUE_PLAIN_LEN;
}
/* head_onwire_len(sizeof(struct ceph_msg_header2), false) */
@@ -702,12 +702,11 @@ static int setup_crypto(struct ceph_connection *con,
const u8 *session_key, int session_key_len,
const u8 *con_secret, int con_secret_len)
{
- unsigned int noio_flag;
int ret;
dout("%s con %p con_mode %d session_key_len %d con_secret_len %d\n",
__func__, con, con->v2.con_mode, session_key_len, con_secret_len);
- WARN_ON(con->v2.hmac_key_set || con->v2.gcm_tfm || con->v2.gcm_req);
+ WARN_ON(con->v2.hmac_key_set || con->v2.gcm_key_set);
if (con->v2.con_mode != CEPH_CON_MODE_CRC &&
con->v2.con_mode != CEPH_CON_MODE_SECURE) {
@@ -734,46 +733,26 @@ static int setup_crypto(struct ceph_connection *con,
return -EINVAL;
}
- noio_flag = memalloc_noio_save();
- con->v2.gcm_tfm = crypto_alloc_aead("gcm(aes)", 0, 0);
- memalloc_noio_restore(noio_flag);
- if (IS_ERR(con->v2.gcm_tfm)) {
- ret = PTR_ERR(con->v2.gcm_tfm);
- con->v2.gcm_tfm = NULL;
- pr_err("failed to allocate gcm tfm context: %d\n", ret);
- return ret;
- }
-
- WARN_ON((unsigned long)con_secret &
- crypto_aead_alignmask(con->v2.gcm_tfm));
- ret = crypto_aead_setkey(con->v2.gcm_tfm, con_secret, CEPH_GCM_KEY_LEN);
+ ret = aes_gcm_preparekey(&con->v2.gcm_key, con_secret, CEPH_GCM_KEY_LEN,
+ CEPH_GCM_TAG_LEN);
if (ret) {
- pr_err("failed to set gcm key: %d\n", ret);
+ /* This should never happen, since valid lengths were used. */
+ pr_err("failed to prepare gcm key: %d\n", ret);
return ret;
}
- WARN_ON(crypto_aead_ivsize(con->v2.gcm_tfm) != CEPH_GCM_IV_LEN);
- ret = crypto_aead_setauthsize(con->v2.gcm_tfm, CEPH_GCM_TAG_LEN);
- if (ret) {
- pr_err("failed to set gcm tag size: %d\n", ret);
- return ret;
- }
-
- con->v2.gcm_req = aead_request_alloc(con->v2.gcm_tfm, GFP_NOIO);
- if (!con->v2.gcm_req) {
- pr_err("failed to allocate gcm request\n");
- return -ENOMEM;
- }
-
- crypto_init_wait(&con->v2.gcm_wait);
- aead_request_set_callback(con->v2.gcm_req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- crypto_req_done, &con->v2.gcm_wait);
-
memcpy(&con->v2.in_gcm_nonce, con_secret + CEPH_GCM_KEY_LEN,
CEPH_GCM_IV_LEN);
memcpy(&con->v2.out_gcm_nonce,
con_secret + CEPH_GCM_KEY_LEN + CEPH_GCM_IV_LEN,
CEPH_GCM_IV_LEN);
+ if (!con->v2.out_ciphertext) {
+ con->v2.out_ciphertext =
+ kmalloc(OUT_CIPHERTEXT_BUFFER_SIZE, GFP_NOIO);
+ if (!con->v2.out_ciphertext)
+ return -ENOMEM;
+ }
+ con->v2.gcm_key_set = true;
return 0; /* auth_x, secure mode */
}
@@ -807,27 +786,6 @@ static void gcm_inc_nonce(struct ceph_gcm_nonce *nonce)
nonce->counter = cpu_to_le64(counter + 1);
}
-static int gcm_crypt(struct ceph_connection *con, bool encrypt,
- struct scatterlist *src, struct scatterlist *dst,
- int src_len)
-{
- struct ceph_gcm_nonce *nonce;
- int ret;
-
- nonce = encrypt ? &con->v2.out_gcm_nonce : &con->v2.in_gcm_nonce;
-
- aead_request_set_ad(con->v2.gcm_req, 0); /* no AAD */
- aead_request_set_crypt(con->v2.gcm_req, src, dst, src_len, (u8 *)nonce);
- ret = crypto_wait_req(encrypt ? crypto_aead_encrypt(con->v2.gcm_req) :
- crypto_aead_decrypt(con->v2.gcm_req),
- &con->v2.gcm_wait);
- if (ret)
- return ret;
-
- gcm_inc_nonce(nonce);
- return 0;
-}
-
static void get_bvec_at(struct ceph_msg_data_cursor *cursor,
struct bio_vec *bv)
{
@@ -845,309 +803,55 @@ static void get_bvec_at(struct ceph_msg_data_cursor *cursor,
bvec_set_page(bv, page, len, off);
}
-static int calc_sg_cnt(void *buf, int buf_len)
-{
- int sg_cnt;
-
- if (!buf_len)
- return 0;
-
- sg_cnt = need_padding(buf_len) ? 1 : 0;
- if (is_vmalloc_addr(buf)) {
- WARN_ON(offset_in_page(buf));
- sg_cnt += PAGE_ALIGN(buf_len) >> PAGE_SHIFT;
- } else {
- sg_cnt++;
- }
-
- return sg_cnt;
-}
-
-static int calc_sg_cnt_cursor(struct ceph_msg_data_cursor *cursor)
+static void ceph_aes_gcm_encrypt(u8 *data, u8 *authtag, size_t data_len,
+ struct ceph_connection *con)
{
- int data_len = cursor->total_resid;
- struct bio_vec bv;
- int sg_cnt;
-
- if (!data_len)
- return 0;
-
- sg_cnt = need_padding(data_len) ? 1 : 0;
- do {
- get_bvec_at(cursor, &bv);
- sg_cnt++;
-
- ceph_msg_data_advance(cursor, bv.bv_len);
- } while (cursor->total_resid);
-
- return sg_cnt;
+ aes_gcm_encrypt(data, authtag, data, data_len, NULL, 0,
+ (const u8 *)&con->v2.out_gcm_nonce, &con->v2.gcm_key);
+ gcm_inc_nonce(&con->v2.out_gcm_nonce);
}
-static void init_sgs(struct scatterlist **sg, void *buf, int buf_len, u8 *pad)
+static int ceph_aes_gcm_decrypt(u8 *data, const u8 *authtag, size_t data_len,
+ struct ceph_connection *con)
{
- void *end = buf + buf_len;
- struct page *page;
- int len;
- void *p;
-
- if (!buf_len)
- return;
-
- if (is_vmalloc_addr(buf)) {
- p = buf;
- do {
- page = vmalloc_to_page(p);
- len = min_t(int, end - p, PAGE_SIZE);
- WARN_ON(!page || !len || offset_in_page(p));
- sg_set_page(*sg, page, len, 0);
- *sg = sg_next(*sg);
- p += len;
- } while (p != end);
- } else {
- sg_set_buf(*sg, buf, buf_len);
- *sg = sg_next(*sg);
- }
-
- if (need_padding(buf_len)) {
- sg_set_buf(*sg, pad, padding_len(buf_len));
- *sg = sg_next(*sg);
- }
+ int err = aes_gcm_decrypt(data, data, authtag, data_len, NULL, 0,
+ (const u8 *)&con->v2.in_gcm_nonce,
+ &con->v2.gcm_key);
+ gcm_inc_nonce(&con->v2.in_gcm_nonce);
+ return err;
}
-static void init_sgs_cursor(struct scatterlist **sg,
- struct ceph_msg_data_cursor *cursor, u8 *pad)
+static void ceph_aes_gcm_encrypt_init(struct ceph_connection *con)
{
- int data_len = cursor->total_resid;
- struct bio_vec bv;
-
- if (!data_len)
- return;
-
- do {
- get_bvec_at(cursor, &bv);
- sg_set_page(*sg, bv.bv_page, bv.bv_len, bv.bv_offset);
- *sg = sg_next(*sg);
-
- ceph_msg_data_advance(cursor, bv.bv_len);
- } while (cursor->total_resid);
-
- if (need_padding(data_len)) {
- sg_set_buf(*sg, pad, padding_len(data_len));
- *sg = sg_next(*sg);
- }
+ aes_gcm_init(&con->v2.out_gcm_ctx, (const u8 *)&con->v2.out_gcm_nonce,
+ &con->v2.gcm_key);
+ gcm_inc_nonce(&con->v2.out_gcm_nonce);
}
-/**
- * init_sgs_pages: set up scatterlist on an array of page pointers
- * @sg: scatterlist to populate
- * @pages: pointer to page array
- * @dpos: position in the array to start (bytes)
- * @dlen: len to add to sg (bytes)
- * @pad: pointer to pad destination (if any)
- *
- * Populate the scatterlist from the page array, starting at an arbitrary
- * byte in the array and running for a specified length.
- */
-static void init_sgs_pages(struct scatterlist **sg, struct page **pages,
- int dpos, int dlen, u8 *pad)
+static void ceph_aes_gcm_decrypt_init(struct ceph_connection *con)
{
- int idx = dpos >> PAGE_SHIFT;
- int off = offset_in_page(dpos);
- int resid = dlen;
-
- do {
- int len = min(resid, (int)PAGE_SIZE - off);
-
- sg_set_page(*sg, pages[idx], len, off);
- *sg = sg_next(*sg);
- off = 0;
- ++idx;
- resid -= len;
- } while (resid);
-
- if (need_padding(dlen)) {
- sg_set_buf(*sg, pad, padding_len(dlen));
- *sg = sg_next(*sg);
- }
-}
-
-static int setup_message_sgs(struct sg_table *sgt, struct ceph_msg *msg,
- u8 *front_pad, u8 *middle_pad, u8 *data_pad,
- void *epilogue, struct page **pages, int dpos,
- bool add_tag)
-{
- struct ceph_msg_data_cursor cursor;
- struct scatterlist *cur_sg;
- int dlen = data_len(msg);
- int sg_cnt;
- int ret;
-
- if (!front_len(msg) && !middle_len(msg) && !data_len(msg))
- return 0;
-
- sg_cnt = 1; /* epilogue + [auth tag] */
- if (front_len(msg))
- sg_cnt += calc_sg_cnt(msg->front.iov_base,
- front_len(msg));
- if (middle_len(msg))
- sg_cnt += calc_sg_cnt(msg->middle->vec.iov_base,
- middle_len(msg));
- if (dlen) {
- if (pages) {
- sg_cnt += calc_pages_for(dpos, dlen);
- if (need_padding(dlen))
- sg_cnt++;
- } else {
- ceph_msg_data_cursor_init(&cursor, msg, dlen);
- sg_cnt += calc_sg_cnt_cursor(&cursor);
- }
- }
-
- ret = sg_alloc_table(sgt, sg_cnt, GFP_NOIO);
- if (ret)
- return ret;
-
- cur_sg = sgt->sgl;
- if (front_len(msg))
- init_sgs(&cur_sg, msg->front.iov_base, front_len(msg),
- front_pad);
- if (middle_len(msg))
- init_sgs(&cur_sg, msg->middle->vec.iov_base, middle_len(msg),
- middle_pad);
- if (dlen) {
- if (pages) {
- init_sgs_pages(&cur_sg, pages, dpos, dlen, data_pad);
- } else {
- ceph_msg_data_cursor_init(&cursor, msg, dlen);
- init_sgs_cursor(&cur_sg, &cursor, data_pad);
- }
- }
-
- WARN_ON(!sg_is_last(cur_sg));
- sg_set_buf(cur_sg, epilogue,
- CEPH_GCM_BLOCK_LEN + (add_tag ? CEPH_GCM_TAG_LEN : 0));
- return 0;
-}
-
-static int decrypt_preamble(struct ceph_connection *con)
-{
- struct scatterlist sg;
-
- sg_init_one(&sg, con->v2.in_buf, CEPH_PREAMBLE_SECURE_LEN);
- return gcm_crypt(con, false, &sg, &sg, CEPH_PREAMBLE_SECURE_LEN);
+ aes_gcm_init(&con->v2.in_gcm_ctx, (const u8 *)&con->v2.in_gcm_nonce,
+ &con->v2.gcm_key);
+ gcm_inc_nonce(&con->v2.in_gcm_nonce);
}
static int decrypt_control_remainder(struct ceph_connection *con)
{
int ctrl_len = con->v2.in_desc.fd_lens[0];
int rem_len = ctrl_len - CEPH_PREAMBLE_INLINE_LEN;
- int pt_len = padding_len(rem_len) + CEPH_GCM_TAG_LEN;
- struct scatterlist sgs[2];
+ int pad_len = padding_len(rem_len);
WARN_ON(con->v2.in_kvecs[0].iov_len != rem_len);
- WARN_ON(con->v2.in_kvecs[1].iov_len != pt_len);
+ WARN_ON(con->v2.in_kvecs[1].iov_len != pad_len + CEPH_GCM_TAG_LEN);
- sg_init_table(sgs, 2);
- sg_set_buf(&sgs[0], con->v2.in_kvecs[0].iov_base, rem_len);
- sg_set_buf(&sgs[1], con->v2.in_buf, pt_len);
-
- return gcm_crypt(con, false, sgs, sgs,
- padded_len(rem_len) + CEPH_GCM_TAG_LEN);
-}
-
-/* Process sparse read data that lives in a buffer */
-static int process_v2_sparse_read(struct ceph_connection *con,
- struct page **pages, int spos)
-{
- struct ceph_msg_data_cursor cursor;
- int ret;
-
- ceph_msg_data_cursor_init(&cursor, con->in_msg,
- con->in_msg->sparse_read_total);
-
- for (;;) {
- char *buf = NULL;
-
- ret = con->ops->sparse_read(con, &cursor, &buf);
- if (ret <= 0)
- return ret;
-
- dout("%s: sparse_read return %x buf %p\n", __func__, ret, buf);
-
- do {
- int idx = spos >> PAGE_SHIFT;
- int soff = offset_in_page(spos);
- struct page *spage = con->v2.in_enc_pages[idx];
- int len = min_t(int, ret, PAGE_SIZE - soff);
-
- if (buf) {
- memcpy_from_page(buf, spage, soff, len);
- buf += len;
- } else {
- struct bio_vec bv;
-
- get_bvec_at(&cursor, &bv);
- len = min_t(int, len, bv.bv_len);
- memcpy_page(bv.bv_page, bv.bv_offset,
- spage, soff, len);
- ceph_msg_data_advance(&cursor, len);
- }
- spos += len;
- ret -= len;
- } while (ret);
- }
-}
-
-static int decrypt_tail(struct ceph_connection *con)
-{
- struct sg_table enc_sgt = {};
- struct sg_table sgt = {};
- struct page **pages = NULL;
- bool sparse = !!con->in_msg->sparse_read_total;
- int dpos = 0;
- int tail_len;
- int ret;
-
- tail_len = tail_onwire_len(con->in_msg, true);
- ret = sg_alloc_table_from_pages(&enc_sgt, con->v2.in_enc_pages,
- con->v2.in_enc_page_cnt, 0, tail_len,
- GFP_NOIO);
- if (ret)
- goto out;
-
- if (sparse) {
- dpos = padded_len(front_len(con->in_msg) + padded_len(middle_len(con->in_msg)));
- pages = con->v2.in_enc_pages;
- }
-
- ret = setup_message_sgs(&sgt, con->in_msg, FRONT_PAD(con->v2.in_buf),
- MIDDLE_PAD(con->v2.in_buf), DATA_PAD(con->v2.in_buf),
- con->v2.in_buf, pages, dpos, true);
- if (ret)
- goto out;
-
- dout("%s con %p msg %p enc_page_cnt %d sg_cnt %d\n", __func__, con,
- con->in_msg, con->v2.in_enc_page_cnt, sgt.orig_nents);
- ret = gcm_crypt(con, false, enc_sgt.sgl, sgt.sgl, tail_len);
- if (ret)
- goto out;
-
- if (sparse && data_len(con->in_msg)) {
- ret = process_v2_sparse_read(con, con->v2.in_enc_pages, dpos);
- if (ret)
- goto out;
- }
-
- WARN_ON(!con->v2.in_enc_page_cnt);
- ceph_release_page_vector(con->v2.in_enc_pages,
- con->v2.in_enc_page_cnt);
- con->v2.in_enc_pages = NULL;
- con->v2.in_enc_page_cnt = 0;
-
-out:
- sg_free_table(&sgt);
- sg_free_table(&enc_sgt);
- return ret;
+ ceph_aes_gcm_decrypt_init(con);
+ aes_gcm_decrypt_update(&con->v2.in_gcm_ctx,
+ con->v2.in_kvecs[0].iov_base,
+ con->v2.in_kvecs[0].iov_base, rem_len);
+ aes_gcm_decrypt_update(&con->v2.in_gcm_ctx, con->v2.in_buf,
+ con->v2.in_buf, pad_len);
+ return aes_gcm_decrypt_final(&con->v2.in_gcm_ctx,
+ con->v2.in_buf + pad_len);
}
static int prepare_banner(struct ceph_connection *con)
@@ -1219,25 +923,18 @@ static void prepare_head_plain(struct ceph_connection *con, void *base,
}
}
-static int prepare_head_secure_small(struct ceph_connection *con,
- void *base, int ctrl_len)
+static void prepare_head_secure_small(struct ceph_connection *con, void *base,
+ int ctrl_len)
{
- struct scatterlist sg;
- int ret;
-
/* inline buffer padding? */
if (ctrl_len < CEPH_PREAMBLE_INLINE_LEN)
memset(CTRL_BODY(base) + ctrl_len, 0,
CEPH_PREAMBLE_INLINE_LEN - ctrl_len);
- sg_init_one(&sg, base, CEPH_PREAMBLE_SECURE_LEN);
- ret = gcm_crypt(con, true, &sg, &sg,
- CEPH_PREAMBLE_SECURE_LEN - CEPH_GCM_TAG_LEN);
- if (ret)
- return ret;
-
+ ceph_aes_gcm_encrypt(base,
+ base + CEPH_PREAMBLE_SECURE_LEN - CEPH_GCM_TAG_LEN,
+ CEPH_PREAMBLE_SECURE_LEN - CEPH_GCM_TAG_LEN, con);
add_out_kvec(con, base, CEPH_PREAMBLE_SECURE_LEN);
- return 0;
}
/*
@@ -1260,36 +957,25 @@ static int prepare_head_secure_small(struct ceph_connection *con,
*
* Preamble should already be encoded at the start of base.
*/
-static int prepare_head_secure_big(struct ceph_connection *con,
- void *base, int ctrl_len)
+static void prepare_head_secure_big(struct ceph_connection *con, void *base,
+ int ctrl_len)
{
int rem_len = ctrl_len - CEPH_PREAMBLE_INLINE_LEN;
void *rem = CTRL_BODY(base) + CEPH_PREAMBLE_INLINE_LEN;
void *rem_tag = rem + padded_len(rem_len);
void *pmbl_tag = rem_tag + CEPH_GCM_TAG_LEN;
- struct scatterlist sgs[2];
- int ret;
- sg_init_table(sgs, 2);
- sg_set_buf(&sgs[0], base, rem - base);
- sg_set_buf(&sgs[1], pmbl_tag, CEPH_GCM_TAG_LEN);
- ret = gcm_crypt(con, true, sgs, sgs, rem - base);
- if (ret)
- return ret;
+ ceph_aes_gcm_encrypt(base, pmbl_tag, rem - base, con);
/* control remainder padding? */
if (need_padding(rem_len))
memset(rem + rem_len, 0, padding_len(rem_len));
- sg_init_one(&sgs[0], rem, pmbl_tag - rem);
- ret = gcm_crypt(con, true, sgs, sgs, rem_tag - rem);
- if (ret)
- return ret;
+ ceph_aes_gcm_encrypt(rem, rem_tag, rem_tag - rem, con);
add_out_kvec(con, base, rem - base);
add_out_kvec(con, pmbl_tag, CEPH_GCM_TAG_LEN);
add_out_kvec(con, rem, pmbl_tag - rem);
- return 0;
}
static int __prepare_control(struct ceph_connection *con, int tag,
@@ -1298,7 +984,6 @@ static int __prepare_control(struct ceph_connection *con, int tag,
{
int total_len = ctrl_len + extdata_len;
struct ceph_frame_desc desc;
- int ret;
dout("%s con %p tag %d len %d (%d+%d)\n", __func__, con, tag,
total_len, ctrl_len, extdata_len);
@@ -1316,12 +1001,10 @@ static int __prepare_control(struct ceph_connection *con, int tag,
if (ctrl_len <= CEPH_PREAMBLE_INLINE_LEN)
/* fully inlined, inline buffer may need padding */
- ret = prepare_head_secure_small(con, base, ctrl_len);
+ prepare_head_secure_small(con, base, ctrl_len);
else
/* partially inlined, inline buffer is full */
- ret = prepare_head_secure_big(con, base, ctrl_len);
- if (ret)
- return ret;
+ prepare_head_secure_big(con, base, ctrl_len);
} else {
prepare_head_plain(con, base, ctrl_len, extdata, extdata_len,
to_be_signed);
@@ -1623,31 +1306,17 @@ static void prepare_message_plain(struct ceph_connection *con,
}
}
-/*
- * Unfortunately the kernel crypto API doesn't support streaming
- * (piecewise) operation for AEAD algorithms, so we can't get away
- * with a fixed size buffer and a couple sgs. Instead, we have to
- * allocate pages for the entire tail of the message (currently up
- * to ~32M) and two sgs arrays (up to ~256K each)...
- */
static int prepare_message_secure(struct ceph_connection *con,
struct ceph_msg *msg)
{
- void *zerop = page_address(ceph_zero_page);
- struct sg_table enc_sgt = {};
- struct sg_table sgt = {};
- struct page **enc_pages;
- int enc_page_cnt;
- int tail_len;
- int ret;
+ int front = front_len(msg);
+ int middle = middle_len(msg);
+ int data = data_len(msg);
- ret = prepare_head_secure_small(con, con->v2.out_buf,
- sizeof(struct ceph_msg_header2));
- if (ret)
- return ret;
+ prepare_head_secure_small(con, con->v2.out_buf,
+ sizeof(struct ceph_msg_header2));
- tail_len = tail_onwire_len(msg, true);
- if (!tail_len) {
+ if (!front && !middle && !data) {
/*
* Empty message: once the head is written,
* we are done -- there is no epilogue.
@@ -1657,42 +1326,43 @@ static int prepare_message_secure(struct ceph_connection *con,
}
encode_epilogue_secure(con, false);
- ret = setup_message_sgs(&sgt, msg, zerop, zerop, zerop,
- &con->v2.out_epil, NULL, 0, false);
- if (ret)
- goto out;
- enc_page_cnt = calc_pages_for(0, tail_len);
- enc_pages = ceph_alloc_page_vector(enc_page_cnt, GFP_NOIO);
- if (IS_ERR(enc_pages)) {
- ret = PTR_ERR(enc_pages);
- goto out;
- }
+ ceph_aes_gcm_encrypt_init(con);
- WARN_ON(con->v2.out_enc_pages || con->v2.out_enc_page_cnt);
- con->v2.out_enc_pages = enc_pages;
- con->v2.out_enc_page_cnt = enc_page_cnt;
- con->v2.out_enc_resid = tail_len;
- con->v2.out_enc_i = 0;
+ if (front || middle) {
+ u8 *bounce = con->v2.out_ciphertext;
+ int front_pad = padding_len(front);
+ int middle_pad = padding_len(middle);
+ int front_middle_len = padded_len(front) + padded_len(middle);
- ret = sg_alloc_table_from_pages(&enc_sgt, enc_pages, enc_page_cnt,
- 0, tail_len, GFP_NOIO);
- if (ret)
- goto out;
+ if (front_middle_len > OUT_CIPHERTEXT_BUFFER_SIZE) {
+ bounce = kvmalloc(front_middle_len, GFP_NOIO);
+ if (!bounce)
+ return -ENOMEM;
+ con->v2.out_front_middle_ciphertext = bounce;
+ }
- ret = gcm_crypt(con, true, sgt.sgl, enc_sgt.sgl,
- tail_len - CEPH_GCM_TAG_LEN);
- if (ret)
- goto out;
+ /* Copy front and middle to bounce, and zero-pad them. */
+ memcpy(bounce, msg->front.iov_base, front);
+ memset(&bounce[front], 0, front_pad);
+ memcpy(&bounce[front + front_pad], msg->middle->vec.iov_base,
+ middle);
+ memset(&bounce[front + front_pad + middle], 0, middle_pad);
- dout("%s con %p msg %p sg_cnt %d enc_page_cnt %d\n", __func__, con,
- msg, sgt.orig_nents, enc_page_cnt);
- con->v2.out_state = OUT_S_QUEUE_ENC_PAGE;
+ /* Encrypt and send the padded front and middle. */
+ aes_gcm_encrypt_update(&con->v2.out_gcm_ctx, bounce, bounce,
+ front_middle_len);
+ add_out_kvec(con, bounce, front_middle_len);
+ }
-out:
- sg_free_table(&sgt);
- sg_free_table(&enc_sgt);
- return ret;
+ if (data) {
+ ceph_msg_data_cursor_init(&con->v2.out_cursor, msg,
+ data_len(msg));
+ con->v2.out_state = OUT_S_QUEUE_ENC_DATA;
+ } else {
+ con->v2.out_state = OUT_S_QUEUE_ENC_EPILOGUE;
+ }
+ return 0;
}
static int prepare_message(struct ceph_connection *con, struct ceph_msg *msg)
@@ -1833,7 +1503,8 @@ static int prepare_read_data(struct ceph_connection *con)
{
struct bio_vec bv;
- con->in_data_crc = -1;
+ if (!con_secure(con))
+ con->in_data_crc = -1;
ceph_msg_data_cursor_init(&con->v2.in_cursor, con->in_msg,
data_len(con->in_msg));
@@ -1855,28 +1526,65 @@ static int prepare_read_data(struct ceph_connection *con)
return 0;
}
-static void prepare_read_data_cont(struct ceph_connection *con)
+static void finalize_one_data_read_bvec(struct ceph_connection *con)
{
- struct bio_vec bv;
+ int len = con->v2.in_bvec.bv_len;
if (ceph_test_opt(from_msgr(con->msgr), RXBOUNCE)) {
- con->in_data_crc = crc32c(con->in_data_crc,
- page_address(con->bounce_page),
- con->v2.in_bvec.bv_len);
+ struct bio_vec bv;
+ void *bounce_buf = page_address(con->bounce_page);
+ void *dst_addr;
get_bvec_at(&con->v2.in_cursor, &bv);
- memcpy_to_page(bv.bv_page, bv.bv_offset,
- page_address(con->bounce_page),
- con->v2.in_bvec.bv_len);
+ dst_addr = bvec_kmap_local(&bv);
+
+ if (con_secure(con)) {
+ aes_gcm_decrypt_update(&con->v2.in_gcm_ctx, dst_addr,
+ bounce_buf, len);
+ } else {
+ con->in_data_crc =
+ crc32c(con->in_data_crc, bounce_buf, len);
+ memcpy(dst_addr, bounce_buf, len);
+ }
+ flush_dcache_page(bv.bv_page);
+ kunmap_local(dst_addr);
} else {
- con->in_data_crc = ceph_crc32c_page(con->in_data_crc,
- con->v2.in_bvec.bv_page,
- con->v2.in_bvec.bv_offset,
- con->v2.in_bvec.bv_len);
+ void *dst_addr = bvec_kmap_local(&con->v2.in_bvec);
+
+ if (con_secure(con)) {
+ aes_gcm_decrypt_update(&con->v2.in_gcm_ctx, dst_addr,
+ dst_addr, len);
+ flush_dcache_page(con->v2.in_bvec.bv_page);
+ } else {
+ con->in_data_crc =
+ crc32c(con->in_data_crc, dst_addr, len);
+ }
+ kunmap_local(dst_addr);
}
+ ceph_msg_data_advance(&con->v2.in_cursor, len);
+}
+
+static void prepare_read_epilogue(struct ceph_connection *con)
+{
+ struct ceph_msg *msg = con->in_msg;
+ int len;
+
+ static_assert(sizeof(con->v2.in_buf) >= CEPH_EPILOGUE_PLAIN_LEN);
+ static_assert(sizeof(con->v2.in_buf) >=
+ CEPH_GCM_BLOCK_LEN + CEPH_EPILOGUE_SECURE_LEN);
+ len = data_padding_and_epilogue_onwire_len(con, data_len(msg));
+ reset_in_kvecs(con);
+ add_in_kvec(con, con->v2.in_buf, len);
+ con->v2.in_state = IN_S_HANDLE_EPILOGUE;
+}
+
+static void prepare_read_data_cont(struct ceph_connection *con)
+{
+ finalize_one_data_read_bvec(con);
- ceph_msg_data_advance(&con->v2.in_cursor, con->v2.in_bvec.bv_len);
if (con->v2.in_cursor.total_resid) {
+ struct bio_vec bv;
+
get_bvec_at(&con->v2.in_cursor, &bv);
if (ceph_test_opt(from_msgr(con->msgr), RXBOUNCE)) {
bv.bv_page = con->bounce_page;
@@ -1890,9 +1598,7 @@ static void prepare_read_data_cont(struct ceph_connection *con)
/*
* We've read all data. Prepare to read epilogue.
*/
- reset_in_kvecs(con);
- add_in_kvec(con, con->v2.in_buf, CEPH_EPILOGUE_PLAIN_LEN);
- con->v2.in_state = IN_S_HANDLE_EPILOGUE;
+ prepare_read_epilogue(con);
}
static int prepare_sparse_read_cont(struct ceph_connection *con)
@@ -1905,22 +1611,7 @@ static int prepare_sparse_read_cont(struct ceph_connection *con)
WARN_ON(con->v2.in_state != IN_S_PREPARE_SPARSE_DATA_CONT);
if (iov_iter_is_bvec(&con->v2.in_iter)) {
- if (ceph_test_opt(from_msgr(con->msgr), RXBOUNCE)) {
- con->in_data_crc = crc32c(con->in_data_crc,
- page_address(con->bounce_page),
- con->v2.in_bvec.bv_len);
- get_bvec_at(cursor, &bv);
- memcpy_to_page(bv.bv_page, bv.bv_offset,
- page_address(con->bounce_page),
- con->v2.in_bvec.bv_len);
- } else {
- con->in_data_crc = ceph_crc32c_page(con->in_data_crc,
- con->v2.in_bvec.bv_page,
- con->v2.in_bvec.bv_offset,
- con->v2.in_bvec.bv_len);
- }
-
- ceph_msg_data_advance(cursor, con->v2.in_bvec.bv_len);
+ finalize_one_data_read_bvec(con);
cursor->sr_resid -= con->v2.in_bvec.bv_len;
dout("%s: advance by 0x%x sr_resid 0x%x\n", __func__,
con->v2.in_bvec.bv_len, cursor->sr_resid);
@@ -1938,12 +1629,20 @@ static int prepare_sparse_read_cont(struct ceph_connection *con)
return 0;
}
} else if (iov_iter_is_kvec(&con->v2.in_iter)) {
- /* On first call, we have no kvec so don't compute crc */
+ /* On first call, we have no kvec */
if (con->v2.in_kvec_cnt) {
WARN_ON_ONCE(con->v2.in_kvec_cnt > 1);
- con->in_data_crc = crc32c(con->in_data_crc,
- con->v2.in_kvecs[0].iov_base,
- con->v2.in_kvecs[0].iov_len);
+ if (con_secure(con))
+ aes_gcm_decrypt_update(
+ &con->v2.in_gcm_ctx,
+ con->v2.in_kvecs[0].iov_base,
+ con->v2.in_kvecs[0].iov_base,
+ con->v2.in_kvecs[0].iov_len);
+ else
+ con->in_data_crc =
+ crc32c(con->in_data_crc,
+ con->v2.in_kvecs[0].iov_base,
+ con->v2.in_kvecs[0].iov_len);
}
} else {
return -EIO;
@@ -1955,9 +1654,7 @@ static int prepare_sparse_read_cont(struct ceph_connection *con)
if (ret < 0)
return ret;
- reset_in_kvecs(con);
- add_in_kvec(con, con->v2.in_buf, CEPH_EPILOGUE_PLAIN_LEN);
- con->v2.in_state = IN_S_HANDLE_EPILOGUE;
+ prepare_read_epilogue(con);
return 0;
}
@@ -2046,55 +1743,72 @@ static int prepare_read_tail_plain(struct ceph_connection *con)
return 0;
}
-static void prepare_read_enc_page(struct ceph_connection *con)
+static int decrypt_front_and_middle(struct ceph_connection *con)
{
- struct bio_vec bv;
-
- dout("%s con %p i %d resid %d\n", __func__, con, con->v2.in_enc_i,
- con->v2.in_enc_resid);
- WARN_ON(!con->v2.in_enc_resid);
-
- bvec_set_page(&bv, con->v2.in_enc_pages[con->v2.in_enc_i],
- min(con->v2.in_enc_resid, (int)PAGE_SIZE), 0);
-
- set_in_bvec(con, &bv);
- con->v2.in_enc_i++;
- con->v2.in_enc_resid -= bv.bv_len;
-
- if (con->v2.in_enc_resid) {
- con->v2.in_state = IN_S_PREPARE_READ_ENC_PAGE;
- return;
+ struct ceph_msg *msg = con->in_msg;
+ int front = front_len(msg);
+ int middle = middle_len(msg);
+
+ if (front) {
+ aes_gcm_decrypt_update(&con->v2.in_gcm_ctx, msg->front.iov_base,
+ msg->front.iov_base, front);
+ if (padding_len(front))
+ aes_gcm_decrypt_update(&con->v2.in_gcm_ctx,
+ con->v2.in_buf, con->v2.in_buf,
+ padding_len(front));
+ }
+ if (middle) {
+ aes_gcm_decrypt_update(&con->v2.in_gcm_ctx,
+ msg->middle->vec.iov_base,
+ msg->middle->vec.iov_base, middle);
+ if (padding_len(middle))
+ aes_gcm_decrypt_update(
+ &con->v2.in_gcm_ctx,
+ &con->v2.in_buf[CEPH_GCM_BLOCK_LEN],
+ &con->v2.in_buf[CEPH_GCM_BLOCK_LEN],
+ padding_len(middle));
}
- /*
- * We are set to read the last piece of ciphertext (ending
- * with epilogue) + auth tag.
- */
- WARN_ON(con->v2.in_enc_i != con->v2.in_enc_page_cnt);
- con->v2.in_state = IN_S_HANDLE_EPILOGUE;
+ if (data_len(msg)) {
+ if (msg->sparse_read_total)
+ return prepare_sparse_read_data(con);
+ else
+ return prepare_read_data(con);
+ }
+ prepare_read_epilogue(con);
+ return 0;
}
static int prepare_read_tail_secure(struct ceph_connection *con)
{
- struct page **enc_pages;
- int enc_page_cnt;
- int tail_len;
+ struct ceph_msg *msg = con->in_msg;
+ int front = front_len(msg);
+ int middle = middle_len(msg);
+ int data = data_len(msg);
- tail_len = tail_onwire_len(con->in_msg, true);
- WARN_ON(!tail_len);
+ ceph_aes_gcm_decrypt_init(con);
- enc_page_cnt = calc_pages_for(0, tail_len);
- enc_pages = ceph_alloc_page_vector(enc_page_cnt, GFP_NOIO);
- if (IS_ERR(enc_pages))
- return PTR_ERR(enc_pages);
+ if (!front && !middle) {
+ WARN_ON(!data);
+ return prepare_read_data(con);
+ }
- WARN_ON(con->v2.in_enc_pages || con->v2.in_enc_page_cnt);
- con->v2.in_enc_pages = enc_pages;
- con->v2.in_enc_page_cnt = enc_page_cnt;
- con->v2.in_enc_resid = tail_len;
- con->v2.in_enc_i = 0;
+ reset_in_kvecs(con);
+ if (front) {
+ add_in_kvec(con, msg->front.iov_base, front);
+ WARN_ON(msg->front.iov_len != front);
+ if (padding_len(front))
+ add_in_kvec(con, con->v2.in_buf, padding_len(front));
+ }
+ if (middle) {
+ add_in_kvec(con, msg->middle->vec.iov_base, middle);
+ WARN_ON(msg->middle->vec.iov_len != middle);
+ if (padding_len(middle))
+ add_in_kvec(con, &con->v2.in_buf[CEPH_GCM_BLOCK_LEN],
+ padding_len(middle));
+ }
- prepare_read_enc_page(con);
+ con->v2.in_state = IN_S_DECRYPT_FRONT_AND_MIDDLE;
return 0;
}
@@ -2117,6 +1831,13 @@ static void prepare_skip_message(struct ceph_connection *con)
if (!tail_len) {
__finish_skip(con);
} else {
+ if (con_secure(con)) {
+ /*
+ * In this case the tail decryption was never started,
+ * so the nonce needs to incremented explicitly.
+ */
+ gcm_inc_nonce(&con->v2.in_gcm_nonce);
+ }
set_in_skip(con, tail_len);
con->v2.in_state = IN_S_FINISH_SKIP;
}
@@ -2350,16 +2071,10 @@ static int process_auth_reply_more(struct ceph_connection *con,
return -EINVAL;
}
-/*
- * Align con_secret to avoid GFP_ATOMIC allocation inside
- * crypto_aead_setkey() called from setup_crypto(). __aligned(16)
- * isn't guaranteed to work for stack objects, so do it by hand.
- */
static int process_auth_done(struct ceph_connection *con, void *p, void *end)
{
u8 session_key[CEPH_MAX_KEY_LEN];
- u8 con_secret_buf[CEPH_MAX_CON_SECRET_LEN + 16];
- u8 *con_secret = PTR_ALIGN(&con_secret_buf[0], 16);
+ u8 con_secret[CEPH_MAX_CON_SECRET_LEN];
int session_key_len, con_secret_len;
int payload_len;
u64 global_id;
@@ -2413,7 +2128,7 @@ static int process_auth_done(struct ceph_connection *con, void *p, void *end)
out:
memzero_explicit(session_key, sizeof(session_key));
- memzero_explicit(con_secret_buf, sizeof(con_secret_buf));
+ memzero_explicit(con_secret, sizeof(con_secret));
return ret;
bad:
@@ -2940,15 +2655,14 @@ static int __handle_control(struct ceph_connection *con, void *p)
static int handle_preamble(struct ceph_connection *con)
{
struct ceph_frame_desc *desc = &con->v2.in_desc;
+ int tag_offset = CEPH_PREAMBLE_SECURE_LEN - CEPH_GCM_TAG_LEN;
int ret;
- if (con_secure(con)) {
- ret = decrypt_preamble(con);
- if (ret) {
- if (ret == -EBADMSG)
- con->error_msg = "integrity error, bad preamble auth tag";
- return ret;
- }
+ if (con_secure(con) &&
+ ceph_aes_gcm_decrypt(con->v2.in_buf, &con->v2.in_buf[tag_offset],
+ tag_offset, con) != 0) {
+ con->error_msg = "integrity error, bad preamble auth tag";
+ return -EBADMSG;
}
ret = decode_preamble(con->v2.in_buf, desc);
@@ -3018,19 +2732,28 @@ static int handle_control_remainder(struct ceph_connection *con)
static int handle_epilogue(struct ceph_connection *con)
{
+ struct ceph_msg *msg = con->in_msg;
u32 front_crc, middle_crc, data_crc;
int ret;
if (con_secure(con)) {
- ret = decrypt_tail(con);
- if (ret) {
- if (ret == -EBADMSG)
- con->error_msg = "integrity error, bad epilogue auth tag";
- return ret;
+ int pad = padding_len(data_len(msg));
+ int tag_offset =
+ pad + CEPH_EPILOGUE_SECURE_LEN - CEPH_GCM_TAG_LEN;
+
+ /* Decrypt the data padding and epilogue */
+ aes_gcm_decrypt_update(&con->v2.in_gcm_ctx, con->v2.in_buf,
+ con->v2.in_buf, tag_offset);
+
+ /* Verify the tail's auth tag */
+ if (aes_gcm_decrypt_final(&con->v2.in_gcm_ctx,
+ &con->v2.in_buf[tag_offset]) != 0) {
+ con->error_msg = "integrity error, bad tail auth tag";
+ return -EBADMSG;
}
/* just late_status */
- ret = decode_epilogue(con->v2.in_buf, NULL, NULL, NULL);
+ ret = decode_epilogue(&con->v2.in_buf[pad], NULL, NULL, NULL);
if (ret) {
con->error_msg = "protocol error, bad epilogue";
return ret;
@@ -3058,9 +2781,6 @@ static void finish_skip(struct ceph_connection *con)
{
dout("%s con %p\n", __func__, con);
- if (con_secure(con))
- gcm_inc_nonce(&con->v2.in_gcm_nonce);
-
__finish_skip(con);
}
@@ -3089,6 +2809,9 @@ static int populate_in_iter(struct ceph_connection *con)
case IN_S_HANDLE_CONTROL_REMAINDER:
ret = handle_control_remainder(con);
break;
+ case IN_S_DECRYPT_FRONT_AND_MIDDLE:
+ ret = decrypt_front_and_middle(con);
+ break;
case IN_S_PREPARE_READ_DATA:
ret = prepare_read_data(con);
break;
@@ -3096,10 +2819,6 @@ static int populate_in_iter(struct ceph_connection *con)
prepare_read_data_cont(con);
ret = 0;
break;
- case IN_S_PREPARE_READ_ENC_PAGE:
- prepare_read_enc_page(con);
- ret = 0;
- break;
case IN_S_PREPARE_SPARSE_DATA:
ret = prepare_sparse_read_data(con);
break;
@@ -3203,31 +2922,54 @@ static void queue_data_cont(struct ceph_connection *con, struct ceph_msg *msg)
con->v2.out_state = OUT_S_FINISH_MESSAGE;
}
-static void queue_enc_page(struct ceph_connection *con)
+/*
+ * Continue encrypting and sending the data. The original data can't be
+ * modified, so use a bounce buffer. Also copy the original data into the
+ * bounce buffer before encryption, so that the authentication tag is computed
+ * over a stable copy.
+ */
+static void queue_enc_data(struct ceph_connection *con)
{
+ struct ceph_msg_data_cursor *cursor = &con->v2.out_cursor;
struct bio_vec bv;
+ u8 *bounce = con->v2.out_ciphertext;
+ int len = min(cursor->total_resid, OUT_CIPHERTEXT_BUFFER_SIZE);
+ int i = 0;
+ int n;
- dout("%s con %p i %d resid %d\n", __func__, con, con->v2.out_enc_i,
- con->v2.out_enc_resid);
- WARN_ON(!con->v2.out_enc_resid);
+ dout("%s con %p resid %zu\n", __func__, con, cursor->total_resid);
+
+ do {
+ get_bvec_at(cursor, &bv);
+ n = min(len - i, (int)bv.bv_len);
+ memcpy_from_page(&bounce[i], bv.bv_page, bv.bv_offset, n);
+ ceph_msg_data_advance(cursor, n);
+ i += n;
+ } while (i < len);
+ aes_gcm_encrypt_update(&con->v2.out_gcm_ctx, bounce, bounce, len);
+ reset_out_kvecs(con);
+ add_out_kvec(con, bounce, len);
- bvec_set_page(&bv, con->v2.out_enc_pages[con->v2.out_enc_i],
- min(con->v2.out_enc_resid, (int)PAGE_SIZE), 0);
+ if (cursor->total_resid == 0)
+ con->v2.out_state = OUT_S_QUEUE_ENC_EPILOGUE;
+}
- set_out_bvec(con, &bv, false);
- con->v2.out_enc_i++;
- con->v2.out_enc_resid -= bv.bv_len;
+/* Enqueue the data padding (0-15 encrypted zeroes) and secure epilogue. */
+static void queue_enc_epilogue(struct ceph_connection *con)
+{
+ int pad = padding_len(data_len(con->out_msg));
+ int tag_offset = pad + CEPH_EPILOGUE_SECURE_LEN - CEPH_GCM_TAG_LEN;
+ u8 *bounce = con->v2.out_ciphertext;
- if (con->v2.out_enc_resid) {
- WARN_ON(con->v2.out_state != OUT_S_QUEUE_ENC_PAGE);
- return;
- }
+ memset(bounce, 0, pad);
+ memcpy(&bounce[pad], (u8 *)&con->v2.out_epil,
+ CEPH_EPILOGUE_SECURE_LEN - CEPH_GCM_TAG_LEN);
+ aes_gcm_encrypt_update(&con->v2.out_gcm_ctx, bounce, bounce,
+ tag_offset);
+ aes_gcm_encrypt_final(&con->v2.out_gcm_ctx, &bounce[tag_offset]);
- /*
- * We've queued the last piece of ciphertext (ending with
- * epilogue) + auth tag. Once it's written, we are done.
- */
- WARN_ON(con->v2.out_enc_i != con->v2.out_enc_page_cnt);
+ reset_out_kvecs(con);
+ add_out_kvec(con, bounce, tag_offset + CEPH_GCM_TAG_LEN);
con->v2.out_state = OUT_S_FINISH_MESSAGE;
}
@@ -3256,14 +2998,11 @@ static void finish_message(struct ceph_connection *con)
{
dout("%s con %p msg %p\n", __func__, con, con->out_msg);
- /* we end up here both plain and secure modes */
- if (con->v2.out_enc_pages) {
- WARN_ON(!con->v2.out_enc_page_cnt);
- ceph_release_page_vector(con->v2.out_enc_pages,
- con->v2.out_enc_page_cnt);
- con->v2.out_enc_pages = NULL;
- con->v2.out_enc_page_cnt = 0;
+ if (con->v2.out_front_middle_ciphertext) {
+ kvfree(con->v2.out_front_middle_ciphertext);
+ con->v2.out_front_middle_ciphertext = NULL;
}
+
/* message may have been revoked */
if (con->out_msg) {
ceph_msg_put(con->out_msg);
@@ -3297,8 +3036,11 @@ static int populate_out_iter(struct ceph_connection *con)
WARN_ON(!con->out_msg);
queue_data_cont(con, con->out_msg);
goto populated;
- case OUT_S_QUEUE_ENC_PAGE:
- queue_enc_page(con);
+ case OUT_S_QUEUE_ENC_DATA:
+ queue_enc_data(con);
+ goto populated;
+ case OUT_S_QUEUE_ENC_EPILOGUE:
+ queue_enc_epilogue(con);
goto populated;
case OUT_S_QUEUE_ZEROS:
WARN_ON(con->out_msg); /* revoked */
@@ -3621,7 +3363,8 @@ void ceph_con_v2_revoke(struct ceph_connection *con, struct ceph_msg *msg)
WARN_ON(con->v2.out_zero);
if (con_secure(con)) {
- WARN_ON(con->v2.out_state != OUT_S_QUEUE_ENC_PAGE &&
+ WARN_ON(con->v2.out_state != OUT_S_QUEUE_ENC_DATA &&
+ con->v2.out_state != OUT_S_QUEUE_ENC_EPILOGUE &&
con->v2.out_state != OUT_S_FINISH_MESSAGE);
dout("%s con %p secure - noop\n", __func__, con);
return;
@@ -3643,6 +3386,25 @@ void ceph_con_v2_revoke(struct ceph_connection *con, struct ceph_msg *msg)
}
}
+static void revoke_at_decrypt_front_and_middle(struct ceph_connection *con)
+{
+ struct ceph_msg *msg = con->in_msg;
+ int resid, remaining;
+
+ WARN_ON(!con_secure(con));
+ WARN_ON(!iov_iter_is_kvec(&con->v2.in_iter));
+ resid = iov_iter_count(&con->v2.in_iter);
+ WARN_ON(!resid);
+
+ remaining = data_len(msg) +
+ data_padding_and_epilogue_onwire_len(con, data_len(msg));
+ dout("%s con %p resid %d remaining %d\n", __func__, con, resid,
+ remaining);
+ con->v2.in_iter.count -= resid;
+ set_in_skip(con, resid + remaining);
+ con->v2.in_state = IN_S_FINISH_SKIP;
+}
+
static void revoke_at_prepare_read_data(struct ceph_connection *con)
{
int remaining;
@@ -3667,7 +3429,6 @@ static void revoke_at_prepare_read_data_cont(struct ceph_connection *con)
int recved, resid; /* current piece of data */
int remaining;
- WARN_ON(con_secure(con));
WARN_ON(!data_len(con->in_msg));
WARN_ON(!iov_iter_is_bvec(&con->v2.in_iter));
resid = iov_iter_count(&con->v2.in_iter);
@@ -3679,7 +3440,8 @@ static void revoke_at_prepare_read_data_cont(struct ceph_connection *con)
ceph_msg_data_advance(&con->v2.in_cursor, recved);
WARN_ON(resid > con->v2.in_cursor.total_resid);
- remaining = CEPH_EPILOGUE_PLAIN_LEN;
+ remaining = data_padding_and_epilogue_onwire_len(con,
+ data_len(con->in_msg));
dout("%s con %p total_resid %zu remaining %d\n", __func__, con,
con->v2.in_cursor.total_resid, remaining);
con->v2.in_iter.count -= resid;
@@ -3687,34 +3449,19 @@ static void revoke_at_prepare_read_data_cont(struct ceph_connection *con)
con->v2.in_state = IN_S_FINISH_SKIP;
}
-static void revoke_at_prepare_read_enc_page(struct ceph_connection *con)
-{
- int resid; /* current enc page (not necessarily data) */
-
- WARN_ON(!con_secure(con));
- WARN_ON(!iov_iter_is_bvec(&con->v2.in_iter));
- resid = iov_iter_count(&con->v2.in_iter);
- WARN_ON(!resid || resid > con->v2.in_bvec.bv_len);
-
- dout("%s con %p resid %d enc_resid %d\n", __func__, con, resid,
- con->v2.in_enc_resid);
- con->v2.in_iter.count -= resid;
- set_in_skip(con, resid + con->v2.in_enc_resid);
- con->v2.in_state = IN_S_FINISH_SKIP;
-}
-
static void revoke_at_prepare_sparse_data(struct ceph_connection *con)
{
int resid; /* current piece of data */
int remaining;
- WARN_ON(con_secure(con));
WARN_ON(!data_len(con->in_msg));
WARN_ON(!iov_iter_is_bvec(&con->v2.in_iter));
resid = iov_iter_count(&con->v2.in_iter);
dout("%s con %p resid %d\n", __func__, con, resid);
- remaining = CEPH_EPILOGUE_PLAIN_LEN + con->v2.data_len_remain;
+ remaining = con->v2.data_len_remain +
+ data_padding_and_epilogue_onwire_len(con,
+ data_len(con->in_msg));
con->v2.in_iter.count -= resid;
set_in_skip(con, resid + remaining);
con->v2.in_state = IN_S_FINISH_SKIP;
@@ -3736,6 +3483,9 @@ static void revoke_at_handle_epilogue(struct ceph_connection *con)
void ceph_con_v2_revoke_incoming(struct ceph_connection *con)
{
switch (con->v2.in_state) {
+ case IN_S_DECRYPT_FRONT_AND_MIDDLE:
+ revoke_at_decrypt_front_and_middle(con);
+ break;
case IN_S_PREPARE_SPARSE_DATA:
case IN_S_PREPARE_READ_DATA:
revoke_at_prepare_read_data(con);
@@ -3743,9 +3493,6 @@ void ceph_con_v2_revoke_incoming(struct ceph_connection *con)
case IN_S_PREPARE_READ_DATA_CONT:
revoke_at_prepare_read_data_cont(con);
break;
- case IN_S_PREPARE_READ_ENC_PAGE:
- revoke_at_prepare_read_enc_page(con);
- break;
case IN_S_PREPARE_SPARSE_DATA_CONT:
revoke_at_prepare_sparse_data(con);
break;
@@ -3782,33 +3529,19 @@ void ceph_con_v2_reset_protocol(struct ceph_connection *con)
clear_out_sign_kvecs(con);
free_conn_bufs(con);
- if (con->v2.in_enc_pages) {
- WARN_ON(!con->v2.in_enc_page_cnt);
- ceph_release_page_vector(con->v2.in_enc_pages,
- con->v2.in_enc_page_cnt);
- con->v2.in_enc_pages = NULL;
- con->v2.in_enc_page_cnt = 0;
- }
- if (con->v2.out_enc_pages) {
- WARN_ON(!con->v2.out_enc_page_cnt);
- ceph_release_page_vector(con->v2.out_enc_pages,
- con->v2.out_enc_page_cnt);
- con->v2.out_enc_pages = NULL;
- con->v2.out_enc_page_cnt = 0;
- }
+ kfree(con->v2.out_ciphertext);
+ con->v2.out_ciphertext = NULL;
+ kvfree(con->v2.out_front_middle_ciphertext);
+ con->v2.out_front_middle_ciphertext = NULL;
con->v2.con_mode = CEPH_CON_MODE_UNKNOWN;
memzero_explicit(&con->v2.in_gcm_nonce, CEPH_GCM_IV_LEN);
memzero_explicit(&con->v2.out_gcm_nonce, CEPH_GCM_IV_LEN);
+ memzero_explicit(&con->v2.in_gcm_ctx, sizeof(con->v2.in_gcm_ctx));
+ memzero_explicit(&con->v2.out_gcm_ctx, sizeof(con->v2.out_gcm_ctx));
memzero_explicit(&con->v2.hmac_key, sizeof(con->v2.hmac_key));
con->v2.hmac_key_set = false;
- if (con->v2.gcm_req) {
- aead_request_free(con->v2.gcm_req);
- con->v2.gcm_req = NULL;
- }
- if (con->v2.gcm_tfm) {
- crypto_free_aead(con->v2.gcm_tfm);
- con->v2.gcm_tfm = NULL;
- }
+ memzero_explicit(&con->v2.gcm_key, sizeof(con->v2.gcm_key));
+ con->v2.gcm_key_set = false;
}
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 23/33] wifi: mac80211: Use AES-CTR library in fils_aead.c
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (21 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 22/33] libceph: Reimplement messenger v2 encryption using AES-GCM library Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 24/33] wifi: mac80211: Use AES-GCM library for GMAC suite Eric Biggers
` (9 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there's a library API for AES-CTR, use it instead of a
"ctr(aes)" crypto_skcipher in aes_siv_encrypt() and aes_siv_decrypt().
This significantly simplifies the code.
Further simplify both aes_siv_encrypt() and aes_siv_decrypt() by
memmove()ing the source data to the destination buffer (which is offset
by 16 bytes from the source), then doing AES-CTR in-place. This
eliminates the need for the temporary buffer in aes_siv_encrypt(), or to
rely on an unsupported overlapped operation in aes_siv_decrypt(). The
latter was technically a bug, though presumably it worked accidentally.
Finally, also fix the auth tag verification to use crypto_memneq().
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
net/mac80211/Kconfig | 1 +
net/mac80211/fils_aead.c | 90 ++++++++++++----------------------------
2 files changed, 27 insertions(+), 64 deletions(-)
diff --git a/net/mac80211/Kconfig b/net/mac80211/Kconfig
index d6bc295e23a1..8fe97e63ff39 100644
--- a/net/mac80211/Kconfig
+++ b/net/mac80211/Kconfig
@@ -4,6 +4,7 @@ config MAC80211
depends on CFG80211
select CRYPTO
select CRYPTO_LIB_AES_CBC_MACS
+ select CRYPTO_LIB_AES_CTR
select CRYPTO_LIB_ARC4
select CRYPTO_AES
select CRYPTO_CCM
diff --git a/net/mac80211/fils_aead.c b/net/mac80211/fils_aead.c
index d2f4a17eab99..1084c3d0a0e7 100644
--- a/net/mac80211/fils_aead.c
+++ b/net/mac80211/fils_aead.c
@@ -5,7 +5,7 @@
*/
#include <crypto/aes-cbc-macs.h>
-#include <crypto/skcipher.h>
+#include <crypto/aes-ctr.h>
#include <crypto/utils.h>
#include "ieee80211_i.h"
@@ -73,11 +73,8 @@ static int aes_siv_encrypt(const u8 *key, size_t key_len,
size_t len[], u8 *out)
{
u8 v[AES_BLOCK_SIZE];
- struct crypto_skcipher *tfm2;
- struct skcipher_request *req;
+ struct aes_enckey aes_key;
int res;
- struct scatterlist src[1], dst[1];
- u8 *tmp;
key_len /= 2; /* S2V key || CTR key */
@@ -90,12 +87,12 @@ static int aes_siv_encrypt(const u8 *key, size_t key_len,
if (res)
return res;
- /* Use a temporary buffer of the plaintext to handle need for
- * overwriting this during AES-CTR.
+ /*
+ * Move the plaintext to prepare it for in-place encryption. This
+ * avoids needing to copy it into a temporary buffer to prevent it from
+ * being clobbered by the IV copy or AES-CTR itself when out == plain.
*/
- tmp = kmemdup(plain, plain_len, GFP_KERNEL);
- if (!tmp)
- return -ENOMEM;
+ memmove(out + AES_BLOCK_SIZE, plain, plain_len);
/* IV for CTR before encrypted data */
memcpy(out, v, AES_BLOCK_SIZE);
@@ -106,33 +103,14 @@ static int aes_siv_encrypt(const u8 *key, size_t key_len,
v[8] &= 0x7f;
v[12] &= 0x7f;
- /* CTR */
-
- tfm2 = crypto_alloc_skcipher("ctr(aes)", 0, CRYPTO_ALG_ASYNC);
- if (IS_ERR(tfm2)) {
- kfree(tmp);
- return PTR_ERR(tfm2);
- }
- /* K2 for CTR */
- res = crypto_skcipher_setkey(tfm2, key + key_len, key_len);
+ /* CTR with K2 */
+ res = aes_prepareenckey(&aes_key, key + key_len, key_len);
if (res)
- goto fail;
-
- req = skcipher_request_alloc(tfm2, GFP_KERNEL);
- if (!req) {
- res = -ENOMEM;
- goto fail;
- }
-
- sg_init_one(src, tmp, plain_len);
- sg_init_one(dst, out + AES_BLOCK_SIZE, plain_len);
- skcipher_request_set_crypt(req, src, dst, plain_len, v);
- res = crypto_skcipher_encrypt(req);
- skcipher_request_free(req);
-fail:
- kfree(tmp);
- crypto_free_skcipher(tfm2);
- return res;
+ return res;
+ aes_ctr(out + AES_BLOCK_SIZE, out + AES_BLOCK_SIZE, plain_len, v,
+ &aes_key);
+ memzero_explicit(&aes_key, sizeof(aes_key));
+ return 0;
}
/* Note: addr[] and len[] needs to have one extra slot at the end. */
@@ -141,9 +119,7 @@ static int aes_siv_decrypt(const u8 *key, size_t key_len,
size_t num_elem, const u8 *addr[], size_t len[],
u8 *out)
{
- struct crypto_skcipher *tfm2;
- struct skcipher_request *req;
- struct scatterlist src[1], dst[1];
+ struct aes_enckey aes_key;
size_t crypt_len;
int res;
u8 frame_iv[AES_BLOCK_SIZE], iv[AES_BLOCK_SIZE];
@@ -164,38 +140,24 @@ static int aes_siv_decrypt(const u8 *key, size_t key_len,
iv[8] &= 0x7f;
iv[12] &= 0x7f;
- /* CTR */
-
- tfm2 = crypto_alloc_skcipher("ctr(aes)", 0, CRYPTO_ALG_ASYNC);
- if (IS_ERR(tfm2))
- return PTR_ERR(tfm2);
- /* K2 for CTR */
- res = crypto_skcipher_setkey(tfm2, key + key_len, key_len);
- if (res) {
- crypto_free_skcipher(tfm2);
- return res;
- }
-
- req = skcipher_request_alloc(tfm2, GFP_KERNEL);
- if (!req) {
- crypto_free_skcipher(tfm2);
- return -ENOMEM;
- }
-
- sg_init_one(src, iv_crypt + AES_BLOCK_SIZE, crypt_len);
- sg_init_one(dst, out, crypt_len);
- skcipher_request_set_crypt(req, src, dst, crypt_len, iv);
- res = crypto_skcipher_decrypt(req);
- skcipher_request_free(req);
- crypto_free_skcipher(tfm2);
+ /* CTR with K2 */
+ res = aes_prepareenckey(&aes_key, key + key_len, key_len);
if (res)
return res;
+ /*
+ * aes_ctr(out, iv_crypt + AES_BLOCK_SIZE, ...) is an unsupported
+ * overlapped operation when out == iv_crypt, so memmove() the data to
+ * 'out' first to enable standard in-place operation.
+ */
+ memmove(out, iv_crypt + AES_BLOCK_SIZE, crypt_len);
+ aes_ctr(out, out, crypt_len, iv, &aes_key);
+ memzero_explicit(&aes_key, sizeof(aes_key));
/* S2V */
res = aes_s2v(key /* K1 */, key_len, num_elem, addr, len, check);
if (res)
return res;
- if (memcmp(check, frame_iv, AES_BLOCK_SIZE) != 0)
+ if (crypto_memneq(check, frame_iv, AES_BLOCK_SIZE))
return -EINVAL;
return 0;
}
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 24/33] wifi: mac80211: Use AES-GCM library for GMAC suite
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (22 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 23/33] wifi: mac80211: Use AES-CTR library in fils_aead.c Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 25/33] wifi: mac80211: Use crypto libraries for GCMP and CCMP suites Eric Biggers
` (8 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there's a library API for AES-GCM (of which AES-GMAC is a
special case), for implementing the GMAC cipher suite use it instead of
a "gcm(aes)" crypto_aead. This significantly simplifies the code and
eliminates per-skb heap allocations.
As a bonus, ieee80211_crypto_aes_gmac_decrypt() no longer needs to
allocate 'mic' on the heap, since it no longer needs to be representable
as a scatterlist for crypto_aead.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
net/mac80211/Kconfig | 1 +
net/mac80211/aes_gmac.c | 85 ++++++++---------------------------------
net/mac80211/aes_gmac.h | 10 ++---
net/mac80211/key.c | 11 ++----
net/mac80211/key.h | 3 +-
net/mac80211/wpa.c | 12 ++----
6 files changed, 29 insertions(+), 93 deletions(-)
diff --git a/net/mac80211/Kconfig b/net/mac80211/Kconfig
index 8fe97e63ff39..32808c5de0fb 100644
--- a/net/mac80211/Kconfig
+++ b/net/mac80211/Kconfig
@@ -5,6 +5,7 @@ config MAC80211
select CRYPTO
select CRYPTO_LIB_AES_CBC_MACS
select CRYPTO_LIB_AES_CTR
+ select CRYPTO_LIB_AES_GCM
select CRYPTO_LIB_ARC4
select CRYPTO_AES
select CRYPTO_CCM
diff --git a/net/mac80211/aes_gmac.c b/net/mac80211/aes_gmac.c
index 811a83d8d525..722d8983bb5c 100644
--- a/net/mac80211/aes_gmac.c
+++ b/net/mac80211/aes_gmac.c
@@ -6,89 +6,34 @@
#include <linux/kernel.h>
#include <linux/types.h>
-#include <linux/err.h>
-#include <crypto/aead.h>
-#include <crypto/aes.h>
+#include <crypto/aes-gcm.h>
#include <net/mac80211.h>
-#include "key.h"
#include "aes_gmac.h"
-int ieee80211_aes_gmac(struct crypto_aead *tfm, const u8 *aad, u8 *nonce,
- const u8 *data, size_t data_len, u8 *mic)
+int ieee80211_aes_gmac(const struct aes_gcm_key *key, const u8 *aad,
+ const u8 *nonce, const u8 *data, size_t data_len,
+ u8 *mic)
{
- struct scatterlist sg[5];
- u8 *zero, *__aad, iv[AES_BLOCK_SIZE];
- struct aead_request *aead_req;
- int reqsize = sizeof(*aead_req) + crypto_aead_reqsize(tfm);
+ static const u8 zero[IEEE80211_GMAC_MIC_LEN];
+ struct aes_gcm_ctx ctx;
const __le16 *fc;
- int ret;
if (data_len < IEEE80211_GMAC_MIC_LEN)
return -EINVAL;
- aead_req = kzalloc(reqsize + IEEE80211_GMAC_MIC_LEN + GMAC_AAD_LEN,
- GFP_ATOMIC);
- if (!aead_req)
- return -ENOMEM;
-
- zero = (u8 *)aead_req + reqsize;
- __aad = zero + IEEE80211_GMAC_MIC_LEN;
- memcpy(__aad, aad, GMAC_AAD_LEN);
+ aes_gcm_init(&ctx, nonce, key);
+ aes_gcm_auth_update(&ctx, aad, GMAC_AAD_LEN);
fc = (const __le16 *)aad;
if (ieee80211_is_beacon(*fc)) {
/* mask Timestamp field to zero */
- sg_init_table(sg, 5);
- sg_set_buf(&sg[0], __aad, GMAC_AAD_LEN);
- sg_set_buf(&sg[1], zero, 8);
- sg_set_buf(&sg[2], data + 8,
- data_len - 8 - IEEE80211_GMAC_MIC_LEN);
- sg_set_buf(&sg[3], zero, IEEE80211_GMAC_MIC_LEN);
- sg_set_buf(&sg[4], mic, IEEE80211_GMAC_MIC_LEN);
- } else {
- sg_init_table(sg, 4);
- sg_set_buf(&sg[0], __aad, GMAC_AAD_LEN);
- sg_set_buf(&sg[1], data, data_len - IEEE80211_GMAC_MIC_LEN);
- sg_set_buf(&sg[2], zero, IEEE80211_GMAC_MIC_LEN);
- sg_set_buf(&sg[3], mic, IEEE80211_GMAC_MIC_LEN);
+ aes_gcm_auth_update(&ctx, zero, 8);
+ data += 8;
+ data_len -= 8;
}
-
- memcpy(iv, nonce, GMAC_NONCE_LEN);
- memset(iv + GMAC_NONCE_LEN, 0, sizeof(iv) - GMAC_NONCE_LEN);
- iv[AES_BLOCK_SIZE - 1] = 0x01;
-
- aead_request_set_tfm(aead_req, tfm);
- aead_request_set_crypt(aead_req, sg, sg, 0, iv);
- aead_request_set_ad(aead_req, GMAC_AAD_LEN + data_len);
-
- ret = crypto_aead_encrypt(aead_req);
- kfree_sensitive(aead_req);
-
- return ret;
-}
-
-struct crypto_aead *ieee80211_aes_gmac_key_setup(const u8 key[],
- size_t key_len)
-{
- struct crypto_aead *tfm;
- int err;
-
- tfm = crypto_alloc_aead("gcm(aes)", 0, CRYPTO_ALG_ASYNC);
- if (IS_ERR(tfm))
- return tfm;
-
- err = crypto_aead_setkey(tfm, key, key_len);
- if (!err)
- err = crypto_aead_setauthsize(tfm, IEEE80211_GMAC_MIC_LEN);
- if (!err)
- return tfm;
-
- crypto_free_aead(tfm);
- return ERR_PTR(err);
-}
-
-void ieee80211_aes_gmac_key_free(struct crypto_aead *tfm)
-{
- crypto_free_aead(tfm);
+ aes_gcm_auth_update(&ctx, data, data_len - IEEE80211_GMAC_MIC_LEN);
+ aes_gcm_auth_update(&ctx, zero, IEEE80211_GMAC_MIC_LEN);
+ aes_gcm_encrypt_final(&ctx, mic);
+ return 0;
}
diff --git a/net/mac80211/aes_gmac.h b/net/mac80211/aes_gmac.h
index 206136b60bca..f31bdfbecf3c 100644
--- a/net/mac80211/aes_gmac.h
+++ b/net/mac80211/aes_gmac.h
@@ -6,15 +6,13 @@
#ifndef AES_GMAC_H
#define AES_GMAC_H
-#include <linux/crypto.h>
+#include <crypto/aes-gcm.h>
#define GMAC_AAD_LEN 20
#define GMAC_NONCE_LEN 12
-struct crypto_aead *ieee80211_aes_gmac_key_setup(const u8 key[],
- size_t key_len);
-int ieee80211_aes_gmac(struct crypto_aead *tfm, const u8 *aad, u8 *nonce,
- const u8 *data, size_t data_len, u8 *mic);
-void ieee80211_aes_gmac_key_free(struct crypto_aead *tfm);
+int ieee80211_aes_gmac(const struct aes_gcm_key *key, const u8 *aad,
+ const u8 *nonce, const u8 *data, size_t data_len,
+ u8 *mic);
#endif /* AES_GMAC_H */
diff --git a/net/mac80211/key.c b/net/mac80211/key.c
index f45e792abede..48404097e4f1 100644
--- a/net/mac80211/key.c
+++ b/net/mac80211/key.c
@@ -711,10 +711,9 @@ ieee80211_key_alloc(u32 cipher, int idx, size_t key_len,
/* Initialize AES key state here as an optimization so that
* it does not need to be initialized for every packet.
*/
- key->u.aes_gmac.tfm =
- ieee80211_aes_gmac_key_setup(key_data, key_len);
- if (IS_ERR(key->u.aes_gmac.tfm)) {
- err = PTR_ERR(key->u.aes_gmac.tfm);
+ err = aes_gcm_preparekey(&key->u.aes_gmac.key, key_data,
+ key_len, IEEE80211_GMAC_MIC_LEN);
+ if (err) {
kfree(key);
return ERR_PTR(err);
}
@@ -752,10 +751,6 @@ static void ieee80211_key_free_common(struct ieee80211_key *key)
case WLAN_CIPHER_SUITE_CCMP_256:
ieee80211_aes_key_free(key->u.ccmp.tfm);
break;
- case WLAN_CIPHER_SUITE_BIP_GMAC_128:
- case WLAN_CIPHER_SUITE_BIP_GMAC_256:
- ieee80211_aes_gmac_key_free(key->u.aes_gmac.tfm);
- break;
case WLAN_CIPHER_SUITE_GCMP:
case WLAN_CIPHER_SUITE_GCMP_256:
ieee80211_aes_gcm_key_free(key->u.gcmp.tfm);
diff --git a/net/mac80211/key.h b/net/mac80211/key.h
index 826e4e9387c5..d2dd2a76fa25 100644
--- a/net/mac80211/key.h
+++ b/net/mac80211/key.h
@@ -13,6 +13,7 @@
#include <linux/crypto.h>
#include <linux/rcupdate.h>
#include <crypto/aes-cbc-macs.h>
+#include <crypto/aes-gcm.h>
#include <crypto/arc4.h>
#include <net/mac80211.h>
@@ -100,7 +101,7 @@ struct ieee80211_key {
} aes_cmac;
struct {
u8 rx_pn[IEEE80211_GMAC_PN_LEN];
- struct crypto_aead *tfm;
+ struct aes_gcm_key key;
u32 replays; /* dot11RSNAStatsCMACReplays */
u32 icverrors; /* dot11RSNAStatsCMACICVErrors */
} aes_gmac;
diff --git a/net/mac80211/wpa.c b/net/mac80211/wpa.c
index be3a2e95303c..eb4a98537395 100644
--- a/net/mac80211/wpa.c
+++ b/net/mac80211/wpa.c
@@ -1007,7 +1007,7 @@ ieee80211_crypto_aes_gmac_encrypt(struct ieee80211_tx_data *tx)
bip_ipn_swap(nonce + ETH_ALEN, mmie->sequence_number);
/* MIC = AES-GMAC(IGTK, AAD || Management Frame Body || MMIE, 128) */
- if (ieee80211_aes_gmac(key->u.aes_gmac.tfm, aad, nonce,
+ if (ieee80211_aes_gmac(&key->u.aes_gmac.key, aad, nonce,
skb->data + 24, skb->len - 24, mmie->mic) < 0)
return TX_DROP;
@@ -1021,7 +1021,8 @@ ieee80211_crypto_aes_gmac_decrypt(struct ieee80211_rx_data *rx)
struct ieee80211_rx_status *status = IEEE80211_SKB_RXCB(skb);
struct ieee80211_key *key = rx->key;
struct ieee80211_mmie_16 *mmie;
- u8 aad[GMAC_AAD_LEN], *mic, ipn[6], nonce[GMAC_NONCE_LEN];
+ u8 aad[GMAC_AAD_LEN], ipn[6], nonce[GMAC_NONCE_LEN];
+ u8 mic[IEEE80211_GMAC_MIC_LEN];
struct ieee80211_hdr *hdr = (struct ieee80211_hdr *)skb->data;
if (!ieee80211_is_mgmt(hdr->frame_control))
@@ -1052,18 +1053,13 @@ ieee80211_crypto_aes_gmac_decrypt(struct ieee80211_rx_data *rx)
memcpy(nonce, hdr->addr2, ETH_ALEN);
memcpy(nonce + ETH_ALEN, ipn, 6);
- mic = kmalloc(IEEE80211_GMAC_MIC_LEN, GFP_ATOMIC);
- if (!mic)
- return RX_DROP_U_OOM;
- if (ieee80211_aes_gmac(key->u.aes_gmac.tfm, aad, nonce,
+ if (ieee80211_aes_gmac(&key->u.aes_gmac.key, aad, nonce,
skb->data + 24, skb->len - 24,
mic) < 0 ||
crypto_memneq(mic, mmie->mic, sizeof(mmie->mic))) {
key->u.aes_gmac.icverrors++;
- kfree(mic);
return RX_DROP_U_MIC_FAIL;
}
- kfree(mic);
}
memcpy(key->u.aes_gmac.rx_pn, ipn, 6);
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 25/33] wifi: mac80211: Use crypto libraries for GCMP and CCMP suites
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (23 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 24/33] wifi: mac80211: Use AES-GCM library for GMAC suite Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 26/33] macsec: Use AES-GCM library instead of crypto_aead Eric Biggers
` (7 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there are library APIs for AES-GCM and AES-CCM, for the GCMP
and CCMP cipher suites use them instead of "gcm(aes)" and "ccm(aes)"
crypto_aeads. This significantly simplifies the code and eliminates
per-skb heap allocations.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
net/mac80211/Kconfig | 5 +-
net/mac80211/Makefile | 1 -
net/mac80211/aead_api.c | 113 ----------------------------------------
net/mac80211/aead_api.h | 23 --------
net/mac80211/aes_ccm.h | 39 +++++---------
net/mac80211/aes_gcm.h | 38 +++++---------
net/mac80211/key.c | 31 ++++-------
net/mac80211/key.h | 5 +-
net/mac80211/wpa.c | 11 ++--
9 files changed, 45 insertions(+), 221 deletions(-)
delete mode 100644 net/mac80211/aead_api.c
delete mode 100644 net/mac80211/aead_api.h
diff --git a/net/mac80211/Kconfig b/net/mac80211/Kconfig
index 32808c5de0fb..5658f089d915 100644
--- a/net/mac80211/Kconfig
+++ b/net/mac80211/Kconfig
@@ -2,14 +2,11 @@
config MAC80211
tristate "Generic IEEE 802.11 Networking Stack (mac80211)"
depends on CFG80211
- select CRYPTO
select CRYPTO_LIB_AES_CBC_MACS
+ select CRYPTO_LIB_AES_CCM
select CRYPTO_LIB_AES_CTR
select CRYPTO_LIB_AES_GCM
select CRYPTO_LIB_ARC4
- select CRYPTO_AES
- select CRYPTO_CCM
- select CRYPTO_GCM
select CRC32
help
This option enables the hardware independent IEEE 802.11
diff --git a/net/mac80211/Makefile b/net/mac80211/Makefile
index 36f1e3e2222b..2ae7a57565cd 100644
--- a/net/mac80211/Makefile
+++ b/net/mac80211/Makefile
@@ -7,7 +7,6 @@ mac80211-y := \
driver-ops.o \
sta_info.o \
wep.o \
- aead_api.o \
wpa.o \
scan.o offchannel.o \
ht.o agg-tx.o agg-rx.o \
diff --git a/net/mac80211/aead_api.c b/net/mac80211/aead_api.c
deleted file mode 100644
index b00d6f5b33f4..000000000000
--- a/net/mac80211/aead_api.c
+++ /dev/null
@@ -1,113 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Copyright 2003-2004, Instant802 Networks, Inc.
- * Copyright 2005-2006, Devicescape Software, Inc.
- * Copyright 2014-2015, Qualcomm Atheros, Inc.
- *
- * Rewrite: Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
- */
-
-#include <linux/kernel.h>
-#include <linux/types.h>
-#include <linux/err.h>
-#include <linux/scatterlist.h>
-#include <crypto/aead.h>
-
-#include "aead_api.h"
-
-int aead_encrypt(struct crypto_aead *tfm, u8 *b_0, u8 *aad, size_t aad_len,
- u8 *data, size_t data_len, u8 *mic)
-{
- size_t mic_len = crypto_aead_authsize(tfm);
- struct scatterlist sg[3];
- struct aead_request *aead_req;
- int reqsize = sizeof(*aead_req) + crypto_aead_reqsize(tfm);
- u8 *__aad;
- int ret;
-
- aead_req = kzalloc(reqsize + aad_len, GFP_ATOMIC);
- if (!aead_req)
- return -ENOMEM;
-
- __aad = (u8 *)aead_req + reqsize;
- memcpy(__aad, aad, aad_len);
-
- sg_init_table(sg, 3);
- sg_set_buf(&sg[0], __aad, aad_len);
- sg_set_buf(&sg[1], data, data_len);
- sg_set_buf(&sg[2], mic, mic_len);
-
- aead_request_set_tfm(aead_req, tfm);
- aead_request_set_crypt(aead_req, sg, sg, data_len, b_0);
- aead_request_set_ad(aead_req, sg[0].length);
-
- ret = crypto_aead_encrypt(aead_req);
- kfree_sensitive(aead_req);
-
- return ret;
-}
-
-int aead_decrypt(struct crypto_aead *tfm, u8 *b_0, u8 *aad, size_t aad_len,
- u8 *data, size_t data_len, u8 *mic)
-{
- size_t mic_len = crypto_aead_authsize(tfm);
- struct scatterlist sg[3];
- struct aead_request *aead_req;
- int reqsize = sizeof(*aead_req) + crypto_aead_reqsize(tfm);
- u8 *__aad;
- int err;
-
- if (data_len == 0)
- return -EINVAL;
-
- aead_req = kzalloc(reqsize + aad_len, GFP_ATOMIC);
- if (!aead_req)
- return -ENOMEM;
-
- __aad = (u8 *)aead_req + reqsize;
- memcpy(__aad, aad, aad_len);
-
- sg_init_table(sg, 3);
- sg_set_buf(&sg[0], __aad, aad_len);
- sg_set_buf(&sg[1], data, data_len);
- sg_set_buf(&sg[2], mic, mic_len);
-
- aead_request_set_tfm(aead_req, tfm);
- aead_request_set_crypt(aead_req, sg, sg, data_len + mic_len, b_0);
- aead_request_set_ad(aead_req, sg[0].length);
-
- err = crypto_aead_decrypt(aead_req);
- kfree_sensitive(aead_req);
-
- return err;
-}
-
-struct crypto_aead *
-aead_key_setup_encrypt(const char *alg, const u8 key[],
- size_t key_len, size_t mic_len)
-{
- struct crypto_aead *tfm;
- int err;
-
- tfm = crypto_alloc_aead(alg, 0, CRYPTO_ALG_ASYNC);
- if (IS_ERR(tfm))
- return tfm;
-
- err = crypto_aead_setkey(tfm, key, key_len);
- if (err)
- goto free_aead;
- err = crypto_aead_setauthsize(tfm, mic_len);
- if (err)
- goto free_aead;
-
- return tfm;
-
-free_aead:
- crypto_free_aead(tfm);
- return ERR_PTR(err);
-}
-
-void aead_key_free(struct crypto_aead *tfm)
-{
- crypto_free_aead(tfm);
-}
diff --git a/net/mac80211/aead_api.h b/net/mac80211/aead_api.h
deleted file mode 100644
index 7d463b80926a..000000000000
--- a/net/mac80211/aead_api.h
+++ /dev/null
@@ -1,23 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-
-#ifndef _AEAD_API_H
-#define _AEAD_API_H
-
-#include <crypto/aead.h>
-#include <linux/crypto.h>
-
-struct crypto_aead *
-aead_key_setup_encrypt(const char *alg, const u8 key[],
- size_t key_len, size_t mic_len);
-
-int aead_encrypt(struct crypto_aead *tfm, u8 *b_0, u8 *aad,
- size_t aad_len, u8 *data,
- size_t data_len, u8 *mic);
-
-int aead_decrypt(struct crypto_aead *tfm, u8 *b_0, u8 *aad,
- size_t aad_len, u8 *data,
- size_t data_len, u8 *mic);
-
-void aead_key_free(struct crypto_aead *tfm);
-
-#endif /* _AEAD_API_H */
diff --git a/net/mac80211/aes_ccm.h b/net/mac80211/aes_ccm.h
index 96256193cf49..f8d5e9cbf0dc 100644
--- a/net/mac80211/aes_ccm.h
+++ b/net/mac80211/aes_ccm.h
@@ -7,39 +7,26 @@
#ifndef AES_CCM_H
#define AES_CCM_H
-#include "aead_api.h"
+#include <asm/byteorder.h>
+#include <crypto/aes-ccm.h>
#define CCM_AAD_LEN 32
-static inline struct crypto_aead *
-ieee80211_aes_key_setup_encrypt(const u8 key[], size_t key_len, size_t mic_len)
+static inline int ieee80211_aes_ccm_encrypt(const struct aes_ccm_key *key,
+ const u8 *b_0, const u8 *aad,
+ u8 *data, size_t data_len, u8 *mic)
{
- return aead_key_setup_encrypt("ccm(aes)", key, key_len, mic_len);
+ return aes_ccm_encrypt(data, mic, data, data_len, aad + 2,
+ be16_to_cpup((__be16 *)aad), b_0 + 1, 13, key);
}
-static inline int
-ieee80211_aes_ccm_encrypt(struct crypto_aead *tfm,
- u8 *b_0, u8 *aad, u8 *data,
- size_t data_len, u8 *mic)
+static inline int ieee80211_aes_ccm_decrypt(const struct aes_ccm_key *key,
+ const u8 *b_0, const u8 *aad,
+ u8 *data, size_t data_len,
+ const u8 *mic)
{
- return aead_encrypt(tfm, b_0, aad + 2,
- be16_to_cpup((__be16 *)aad),
- data, data_len, mic);
-}
-
-static inline int
-ieee80211_aes_ccm_decrypt(struct crypto_aead *tfm,
- u8 *b_0, u8 *aad, u8 *data,
- size_t data_len, u8 *mic)
-{
- return aead_decrypt(tfm, b_0, aad + 2,
- be16_to_cpup((__be16 *)aad),
- data, data_len, mic);
-}
-
-static inline void ieee80211_aes_key_free(struct crypto_aead *tfm)
-{
- return aead_key_free(tfm);
+ return aes_ccm_decrypt(data, data, mic, data_len, aad + 2,
+ be16_to_cpup((__be16 *)aad), b_0 + 1, 13, key);
}
#endif /* AES_CCM_H */
diff --git a/net/mac80211/aes_gcm.h b/net/mac80211/aes_gcm.h
index b14093b2f7a9..ba13617aeb75 100644
--- a/net/mac80211/aes_gcm.h
+++ b/net/mac80211/aes_gcm.h
@@ -6,38 +6,26 @@
#ifndef AES_GCM_H
#define AES_GCM_H
-#include "aead_api.h"
+#include <asm/byteorder.h>
+#include <crypto/aes-gcm.h>
#define GCM_AAD_LEN 32
-static inline int ieee80211_aes_gcm_encrypt(struct crypto_aead *tfm,
- u8 *j_0, u8 *aad, u8 *data,
- size_t data_len, u8 *mic)
+static inline void ieee80211_aes_gcm_encrypt(const struct aes_gcm_key *key,
+ const u8 *j_0, const u8 *aad,
+ u8 *data, size_t data_len, u8 *mic)
{
- return aead_encrypt(tfm, j_0, aad + 2,
- be16_to_cpup((__be16 *)aad),
- data, data_len, mic);
+ aes_gcm_encrypt(data, mic, data, data_len, aad + 2,
+ be16_to_cpup((__be16 *)aad), j_0, key);
}
-static inline int ieee80211_aes_gcm_decrypt(struct crypto_aead *tfm,
- u8 *j_0, u8 *aad, u8 *data,
- size_t data_len, u8 *mic)
+static inline int ieee80211_aes_gcm_decrypt(const struct aes_gcm_key *key,
+ const u8 *j_0, const u8 *aad,
+ u8 *data, size_t data_len,
+ const u8 *mic)
{
- return aead_decrypt(tfm, j_0, aad + 2,
- be16_to_cpup((__be16 *)aad),
- data, data_len, mic);
-}
-
-static inline struct crypto_aead *
-ieee80211_aes_gcm_key_setup_encrypt(const u8 key[], size_t key_len)
-{
- return aead_key_setup_encrypt("gcm(aes)", key,
- key_len, IEEE80211_GCMP_MIC_LEN);
-}
-
-static inline void ieee80211_aes_gcm_key_free(struct crypto_aead *tfm)
-{
- return aead_key_free(tfm);
+ return aes_gcm_decrypt(data, data, mic, data_len, aad + 2,
+ be16_to_cpup((__be16 *)aad), j_0, key);
}
#endif /* AES_GCM_H */
diff --git a/net/mac80211/key.c b/net/mac80211/key.c
index 48404097e4f1..8c8a82d91bf2 100644
--- a/net/mac80211/key.c
+++ b/net/mac80211/key.c
@@ -652,10 +652,9 @@ ieee80211_key_alloc(u32 cipher, int idx, size_t key_len,
* Initialize AES key state here as an optimization so that
* it does not need to be initialized for every packet.
*/
- key->u.ccmp.tfm = ieee80211_aes_key_setup_encrypt(
- key_data, key_len, IEEE80211_CCMP_MIC_LEN);
- if (IS_ERR(key->u.ccmp.tfm)) {
- err = PTR_ERR(key->u.ccmp.tfm);
+ err = aes_ccm_preparekey(&key->u.ccmp.key, key_data, key_len,
+ IEEE80211_CCMP_MIC_LEN);
+ if (err) {
kfree(key);
return ERR_PTR(err);
}
@@ -670,10 +669,9 @@ ieee80211_key_alloc(u32 cipher, int idx, size_t key_len,
/* Initialize AES key state here as an optimization so that
* it does not need to be initialized for every packet.
*/
- key->u.ccmp.tfm = ieee80211_aes_key_setup_encrypt(
- key_data, key_len, IEEE80211_CCMP_256_MIC_LEN);
- if (IS_ERR(key->u.ccmp.tfm)) {
- err = PTR_ERR(key->u.ccmp.tfm);
+ err = aes_ccm_preparekey(&key->u.ccmp.key, key_data, key_len,
+ IEEE80211_CCMP_256_MIC_LEN);
+ if (err) {
kfree(key);
return ERR_PTR(err);
}
@@ -729,10 +727,9 @@ ieee80211_key_alloc(u32 cipher, int idx, size_t key_len,
/* Initialize AES key state here as an optimization so that
* it does not need to be initialized for every packet.
*/
- key->u.gcmp.tfm = ieee80211_aes_gcm_key_setup_encrypt(key_data,
- key_len);
- if (IS_ERR(key->u.gcmp.tfm)) {
- err = PTR_ERR(key->u.gcmp.tfm);
+ err = aes_gcm_preparekey(&key->u.gcmp.key, key_data, key_len,
+ IEEE80211_GCMP_MIC_LEN);
+ if (err) {
kfree(key);
return ERR_PTR(err);
}
@@ -746,16 +743,6 @@ ieee80211_key_alloc(u32 cipher, int idx, size_t key_len,
static void ieee80211_key_free_common(struct ieee80211_key *key)
{
- switch (key->conf.cipher) {
- case WLAN_CIPHER_SUITE_CCMP:
- case WLAN_CIPHER_SUITE_CCMP_256:
- ieee80211_aes_key_free(key->u.ccmp.tfm);
- break;
- case WLAN_CIPHER_SUITE_GCMP:
- case WLAN_CIPHER_SUITE_GCMP_256:
- ieee80211_aes_gcm_key_free(key->u.gcmp.tfm);
- break;
- }
kfree_sensitive(key);
}
diff --git a/net/mac80211/key.h b/net/mac80211/key.h
index d2dd2a76fa25..a43db120f61c 100644
--- a/net/mac80211/key.h
+++ b/net/mac80211/key.h
@@ -13,6 +13,7 @@
#include <linux/crypto.h>
#include <linux/rcupdate.h>
#include <crypto/aes-cbc-macs.h>
+#include <crypto/aes-ccm.h>
#include <crypto/aes-gcm.h>
#include <crypto/arc4.h>
#include <net/mac80211.h>
@@ -90,7 +91,7 @@ struct ieee80211_key {
* Management frames.
*/
u8 rx_pn[IEEE80211_NUM_TIDS + 1][IEEE80211_CCMP_PN_LEN];
- struct crypto_aead *tfm;
+ struct aes_ccm_key key;
u32 replays; /* dot11RSNAStatsCCMPReplays */
} ccmp;
struct {
@@ -112,7 +113,7 @@ struct ieee80211_key {
* Management frames.
*/
u8 rx_pn[IEEE80211_NUM_TIDS + 1][IEEE80211_GCMP_PN_LEN];
- struct crypto_aead *tfm;
+ struct aes_gcm_key key;
u32 replays; /* dot11RSNAStatsGCMPReplays */
} gcmp;
struct {
diff --git a/net/mac80211/wpa.c b/net/mac80211/wpa.c
index eb4a98537395..db57d383606f 100644
--- a/net/mac80211/wpa.c
+++ b/net/mac80211/wpa.c
@@ -492,7 +492,7 @@ static int ccmp_encrypt_skb(struct ieee80211_tx_data *tx, struct sk_buff *skb,
ccmp_special_blocks(skb, pn, b_0, aad,
key->conf.flags & IEEE80211_KEY_FLAG_SPP_AMSDU,
false);
- return ieee80211_aes_ccm_encrypt(key->u.ccmp.tfm, b_0, aad, pos, len,
+ return ieee80211_aes_ccm_encrypt(&key->u.ccmp.key, b_0, aad, pos, len,
skb_put(skb, mic_len));
}
@@ -587,7 +587,7 @@ ieee80211_crypto_ccmp_decrypt(struct ieee80211_rx_data *rx,
aad_nonce_computed);
if (ieee80211_aes_ccm_decrypt(
- key->u.ccmp.tfm, b_0, aad,
+ &key->u.ccmp.key, b_0, aad,
skb->data + hdrlen + IEEE80211_CCMP_HDR_LEN,
data_len,
skb->data + skb->len - mic_len))
@@ -709,8 +709,9 @@ static int gcmp_encrypt_skb(struct ieee80211_tx_data *tx, struct sk_buff *skb)
gcmp_special_blocks(skb, pn, j_0, aad,
key->conf.flags & IEEE80211_KEY_FLAG_SPP_AMSDU,
false);
- return ieee80211_aes_gcm_encrypt(key->u.gcmp.tfm, j_0, aad, pos, len,
- skb_put(skb, IEEE80211_GCMP_MIC_LEN));
+ ieee80211_aes_gcm_encrypt(&key->u.gcmp.key, j_0, aad, pos, len,
+ skb_put(skb, IEEE80211_GCMP_MIC_LEN));
+ return 0;
}
ieee80211_tx_result
@@ -798,7 +799,7 @@ ieee80211_crypto_gcmp_decrypt(struct ieee80211_rx_data *rx)
aad_nonce_computed);
if (ieee80211_aes_gcm_decrypt(
- key->u.gcmp.tfm, j_0, aad,
+ &key->u.gcmp.key, j_0, aad,
skb->data + hdrlen + IEEE80211_GCMP_HDR_LEN,
data_len,
skb->data + skb->len -
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 26/33] macsec: Use AES-GCM library instead of crypto_aead
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (24 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 25/33] wifi: mac80211: Use crypto libraries for GCMP and CCMP suites Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 27/33] wifi: ipw2x00: Use AES-CCM library Eric Biggers
` (6 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there's a library API for AES-GCM, use it instead of a
"gcm(aes)" crypto_aead. This significantly simplifies the code.
Notably, per-skb dynamic memory allocations and conversions to
scatterlists are eliminated. We now just iterate through the skb's data
and process it directly.
Asynchronous en/decryption operations are no longer supported, as that
execution model has been found to not be beneficial.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
drivers/net/Kconfig | 4 +-
drivers/net/macsec.c | 327 ++++++++++++++-----------------------------
include/net/macsec.h | 5 +-
3 files changed, 106 insertions(+), 230 deletions(-)
diff --git a/drivers/net/Kconfig b/drivers/net/Kconfig
index ff79c466712d..e7322d5a1164 100644
--- a/drivers/net/Kconfig
+++ b/drivers/net/Kconfig
@@ -317,9 +317,7 @@ config AMT
config MACSEC
tristate "IEEE 802.1AE MAC-level encryption (MACsec)"
- select CRYPTO
- select CRYPTO_AES
- select CRYPTO_GCM
+ select CRYPTO_LIB_AES_GCM
select GRO_CELLS
help
MACsec is an encryption standard for Ethernet.
diff --git a/drivers/net/macsec.c b/drivers/net/macsec.c
index fb009120a924..ce32dae77a60 100644
--- a/drivers/net/macsec.c
+++ b/drivers/net/macsec.c
@@ -9,7 +9,7 @@
#include <linux/skbuff.h>
#include <linux/socket.h>
#include <linux/module.h>
-#include <crypto/aead.h>
+#include <crypto/aes-gcm.h>
#include <linux/etherdevice.h>
#include <linux/netdevice.h>
#include <linux/rtnetlink.h>
@@ -134,7 +134,6 @@ static struct macsec_rxh_data *macsec_data_rtnl(const struct net_device *dev)
}
struct macsec_cb {
- struct aead_request *req;
union {
struct macsec_tx_sa *tx_sa;
struct macsec_rx_sa *rx_sa;
@@ -181,9 +180,8 @@ static void free_rxsa_work(struct work_struct *work)
struct macsec_rx_sa *sa =
container_of(to_rcu_work(work), struct macsec_rx_sa, destroy_work);
- crypto_free_aead(sa->key.tfm);
free_percpu(sa->stats);
- kfree(sa);
+ kfree_sensitive(sa);
}
static void macsec_rxsa_put(struct macsec_rx_sa *sa)
@@ -210,9 +208,8 @@ static void free_txsa_work(struct work_struct *work)
struct macsec_tx_sa *sa =
container_of(to_rcu_work(work), struct macsec_tx_sa, destroy_work);
- crypto_free_aead(sa->key.tfm);
free_percpu(sa->stats);
- kfree(sa);
+ kfree_sensitive(sa);
}
static void macsec_txsa_put(struct macsec_tx_sa *sa)
@@ -541,70 +538,72 @@ static void count_tx(struct net_device *dev, int ret, int len)
dev_sw_netstats_tx_add(dev, 1, len);
}
-static void macsec_encrypt_done(void *data, int err)
+static void macsec_crypt_chunk(struct aes_gcm_ctx *ctx, u8 *data, size_t avail,
+ size_t *assoc_len, size_t *crypt_len,
+ bool encrypt)
{
- struct sk_buff *skb = data;
- struct net_device *dev = skb->dev;
- struct macsec_dev *macsec = macsec_priv(dev);
- struct macsec_tx_sa *sa = macsec_skb_cb(skb)->tx_sa;
- int len, ret;
+ size_t n;
- aead_request_free(macsec_skb_cb(skb)->req);
-
- rcu_read_lock_bh();
- macsec_count_tx(skb, &macsec->secy.tx_sc, macsec_skb_cb(skb)->tx_sa);
- /* packet is encrypted/protected so tx_bytes must be calculated */
- len = macsec_msdu_len(skb) + 2 * ETH_ALEN;
- macsec_encrypt_finish(skb, dev);
- ret = dev_queue_xmit(skb);
- count_tx(dev, ret, len);
- rcu_read_unlock_bh();
+ if (*assoc_len && avail) {
+ /* Associated data */
+ n = min(avail, *assoc_len);
+ aes_gcm_auth_update(ctx, data, n);
+ data += n;
+ avail -= n;
+ *assoc_len -= n;
+ }
- macsec_txsa_put(sa);
- dev_put(dev);
+ if (*crypt_len && avail) {
+ /* En/decrypted data */
+ n = min(avail, *crypt_len);
+ if (encrypt)
+ aes_gcm_encrypt_update(ctx, data, data, n);
+ else
+ aes_gcm_decrypt_update(ctx, data, data, n);
+ *crypt_len -= n;
+ }
}
-static struct aead_request *macsec_alloc_req(struct crypto_aead *tfm,
- unsigned char **iv,
- struct scatterlist **sg,
- int num_frags)
+/**
+ * macsec_crypt_skb() - Encrypt or decrypt an skb in-place using AES-GCM
+ * @ctx: An AES-GCM context
+ * @skb: The socket buffer to process
+ * @assoc_len: Length of associated data
+ * @crypt_len: Length of payload data to encrypt or decrypt
+ * @encrypt: True for encryption, false for decryption
+ *
+ * Updates the given AES-GCM context with @assoc_len bytes of associated data
+ * from the skb, then encrypts or decrypts @crypt_len bytes in-place. Context
+ * initialization and finalization are handled by the caller.
+ *
+ * The data (both associated and en/decrypted) is taken from the linear head and
+ * frag_list. It is assumed that the @skb was processed by skb_cow_data(),
+ * meaning it has no page fragments (nr_frags == 0) and is writable.
+ */
+static void macsec_crypt_skb(struct aes_gcm_ctx *ctx, struct sk_buff *skb,
+ size_t assoc_len, size_t crypt_len, bool encrypt)
{
- size_t size, iv_offset, sg_offset;
- struct aead_request *req;
- void *tmp;
+ struct sk_buff *frag_iter;
- size = sizeof(struct aead_request) + crypto_aead_reqsize(tfm);
- iv_offset = size;
- size += GCM_AES_IV_LEN;
-
- size = ALIGN(size, __alignof__(struct scatterlist));
- sg_offset = size;
- size += sizeof(struct scatterlist) * num_frags;
-
- tmp = kmalloc(size, GFP_ATOMIC);
- if (!tmp)
- return NULL;
-
- *iv = (unsigned char *)(tmp + iv_offset);
- *sg = (struct scatterlist *)(tmp + sg_offset);
- req = tmp;
-
- aead_request_set_tfm(req, tfm);
-
- return req;
+ WARN_ON_ONCE(skb_shinfo(skb)->nr_frags);
+ macsec_crypt_chunk(ctx, skb->data, skb_headlen(skb), &assoc_len,
+ &crypt_len, encrypt);
+ skb_walk_frags(skb, frag_iter)
+ macsec_crypt_chunk(ctx, frag_iter->data, frag_iter->len,
+ &assoc_len, &crypt_len, encrypt);
}
static struct sk_buff *macsec_encrypt(struct sk_buff *skb,
struct net_device *dev)
{
int ret;
- struct scatterlist *sg;
struct sk_buff *trailer;
- unsigned char *iv;
+ u8 iv[GCM_AES_IV_LEN];
struct ethhdr *eth;
struct macsec_eth_header *hh;
- size_t unprotected_len;
- struct aead_request *req;
+ size_t unprotected_len, assoc_len, crypt_len;
+ struct aes_gcm_ctx ctx;
+ u8 icv[MACSEC_DEFAULT_ICV_LEN];
struct macsec_secy *secy;
struct macsec_tx_sc *tx_sc;
struct macsec_tx_sa *tx_sa;
@@ -681,56 +680,32 @@ static struct sk_buff *macsec_encrypt(struct sk_buff *skb,
return ERR_PTR(ret);
}
- req = macsec_alloc_req(tx_sa->key.tfm, &iv, &sg, ret);
- if (!req) {
- macsec_txsa_put(tx_sa);
- kfree_skb(skb);
- return ERR_PTR(-ENOMEM);
- }
-
if (secy->xpn)
macsec_fill_iv_xpn(iv, tx_sa->ssci, pn.full64, tx_sa->key.salt);
else
macsec_fill_iv(iv, secy->sci, pn.lower);
- sg_init_table(sg, ret);
- ret = skb_to_sgvec(skb, sg, 0, skb->len);
- if (unlikely(ret < 0)) {
- aead_request_free(req);
- macsec_txsa_put(tx_sa);
- kfree_skb(skb);
- return ERR_PTR(ret);
- }
-
if (tx_sc->encrypt) {
- int len = skb->len - macsec_hdr_len(sci_present) -
- secy->icv_len;
- aead_request_set_crypt(req, sg, sg, len, iv);
- aead_request_set_ad(req, macsec_hdr_len(sci_present));
+ assoc_len = macsec_hdr_len(sci_present);
+ crypt_len = skb->len - assoc_len - secy->icv_len;
} else {
- aead_request_set_crypt(req, sg, sg, 0, iv);
- aead_request_set_ad(req, skb->len - secy->icv_len);
+ assoc_len = skb->len - secy->icv_len;
+ crypt_len = 0;
}
- macsec_skb_cb(skb)->req = req;
- macsec_skb_cb(skb)->tx_sa = tx_sa;
- macsec_skb_cb(skb)->has_sci = sci_present;
- aead_request_set_callback(req, 0, macsec_encrypt_done, skb);
+ aes_gcm_init(&ctx, iv, &tx_sa->key.gcm_key);
+ macsec_crypt_skb(&ctx, skb, assoc_len, crypt_len, true);
+ aes_gcm_encrypt_final(&ctx, icv);
- dev_hold(skb->dev);
- ret = crypto_aead_encrypt(req);
- if (ret == -EINPROGRESS) {
- return ERR_PTR(ret);
- } else if (ret != 0) {
- dev_put(skb->dev);
- kfree_skb(skb);
- aead_request_free(req);
+ ret = skb_store_bits(skb, skb->len - secy->icv_len, icv, secy->icv_len);
+ if (unlikely(ret < 0)) {
macsec_txsa_put(tx_sa);
- return ERR_PTR(-EINVAL);
+ kfree_skb(skb);
+ return ERR_PTR(ret);
}
- dev_put(skb->dev);
- aead_request_free(req);
+ macsec_skb_cb(skb)->tx_sa = tx_sa;
+ macsec_skb_cb(skb)->has_sci = sci_present;
macsec_txsa_put(tx_sa);
return skb;
@@ -844,45 +819,6 @@ static void count_rx(struct net_device *dev, int len)
dev_sw_netstats_rx_add(dev, len);
}
-static void macsec_decrypt_done(void *data, int err)
-{
- struct sk_buff *skb = data;
- struct net_device *dev = skb->dev;
- struct macsec_dev *macsec = macsec_priv(dev);
- struct macsec_rx_sa *rx_sa = macsec_skb_cb(skb)->rx_sa;
- struct macsec_rx_sc *rx_sc = rx_sa->sc;
- int len;
- u32 pn;
-
- aead_request_free(macsec_skb_cb(skb)->req);
-
- if (!err)
- macsec_skb_cb(skb)->valid = true;
-
- rcu_read_lock_bh();
- pn = ntohl(macsec_ethhdr(skb)->packet_number);
- if (!macsec_post_decrypt(skb, &macsec->secy, pn)) {
- rcu_read_unlock_bh();
- kfree_skb(skb);
- goto out;
- }
-
- macsec_finalize_skb(skb, macsec->secy.icv_len,
- macsec_extra_len(macsec_skb_cb(skb)->has_sci));
- len = skb->len;
- macsec_reset_skb(skb, macsec->secy.netdev);
-
- if (gro_cells_receive(&macsec->gro_cells, skb) == NET_RX_SUCCESS)
- count_rx(dev, len);
-
- rcu_read_unlock_bh();
-
-out:
- macsec_rxsa_put(rx_sa);
- macsec_rxsc_put(rx_sc);
- dev_put(dev);
-}
-
static struct sk_buff *macsec_decrypt(struct sk_buff *skb,
struct net_device *dev,
struct macsec_rx_sa *rx_sa,
@@ -890,12 +826,13 @@ static struct sk_buff *macsec_decrypt(struct sk_buff *skb,
struct macsec_secy *secy)
{
int ret;
- struct scatterlist *sg;
struct sk_buff *trailer;
- unsigned char *iv;
- struct aead_request *req;
+ u8 iv[GCM_AES_IV_LEN];
+ size_t assoc_len, crypt_len;
+ struct aes_gcm_ctx ctx;
struct macsec_eth_header *hdr;
u32 hdr_pn;
+ u8 icv[MACSEC_DEFAULT_ICV_LEN];
u16 icv_len = secy->icv_len;
macsec_skb_cb(skb)->valid = false;
@@ -908,11 +845,6 @@ static struct sk_buff *macsec_decrypt(struct sk_buff *skb,
kfree_skb(skb);
return ERR_PTR(ret);
}
- req = macsec_alloc_req(rx_sa->key.tfm, &iv, &sg, ret);
- if (!req) {
- kfree_skb(skb);
- return ERR_PTR(-ENOMEM);
- }
hdr = (struct macsec_eth_header *)skb->data;
hdr_pn = ntohl(hdr->packet_number);
@@ -931,55 +863,33 @@ static struct sk_buff *macsec_decrypt(struct sk_buff *skb,
macsec_fill_iv(iv, sci, hdr_pn);
}
- sg_init_table(sg, ret);
- ret = skb_to_sgvec(skb, sg, 0, skb->len);
- if (unlikely(ret < 0)) {
- aead_request_free(req);
- kfree_skb(skb);
- return ERR_PTR(ret);
- }
-
if (hdr->tci_an & MACSEC_TCI_E) {
/* confidentiality: ethernet + macsec header
* authenticated, encrypted payload
*/
- int len = skb->len - macsec_hdr_len(macsec_skb_cb(skb)->has_sci);
-
- aead_request_set_crypt(req, sg, sg, len, iv);
- aead_request_set_ad(req, macsec_hdr_len(macsec_skb_cb(skb)->has_sci));
- skb = skb_unshare(skb, GFP_ATOMIC);
- if (!skb) {
- aead_request_free(req);
- return ERR_PTR(-ENOMEM);
- }
+ assoc_len = macsec_hdr_len(macsec_skb_cb(skb)->has_sci);
+ crypt_len = skb->len - assoc_len - icv_len;
} else {
/* integrity only: all headers + data authenticated */
- aead_request_set_crypt(req, sg, sg, icv_len, iv);
- aead_request_set_ad(req, skb->len - icv_len);
+ assoc_len = skb->len - icv_len;
+ crypt_len = 0;
}
- macsec_skb_cb(skb)->req = req;
- skb->dev = dev;
- aead_request_set_callback(req, 0, macsec_decrypt_done, skb);
+ aes_gcm_init(&ctx, iv, &rx_sa->key.gcm_key);
+ macsec_crypt_skb(&ctx, skb, assoc_len, crypt_len, false);
- dev_hold(dev);
- ret = crypto_aead_decrypt(req);
- if (ret == -EINPROGRESS) {
+ ret = skb_copy_bits(skb, skb->len - icv_len, icv, icv_len);
+ if (unlikely(ret < 0)) {
+ memzero_explicit(&ctx, sizeof(ctx));
+ kfree_skb(skb);
return ERR_PTR(ret);
- } else if (ret != 0) {
- /* decryption/authentication failed
- * 10.6 if validateFrames is disabled, deliver anyway
- */
- if (ret != -EBADMSG) {
- kfree_skb(skb);
- skb = ERR_PTR(ret);
- }
- } else {
- macsec_skb_cb(skb)->valid = true;
}
- dev_put(dev);
- aead_request_free(req);
+ if (aes_gcm_decrypt_final(&ctx, icv) == 0)
+ macsec_skb_cb(skb)->valid = true;
+ /* else, decryption/authentication failed
+ * 10.6 if validateFrames is disabled, deliver anyway
+ */
return skb;
}
@@ -1276,11 +1186,8 @@ static rx_handler_result_t macsec_handle_frame(struct sk_buff **pskb)
skb = macsec_decrypt(skb, dev, rx_sa, sci, secy);
if (IS_ERR(skb)) {
- /* the decrypt callback needs the reference */
- if (PTR_ERR(skb) != -EINPROGRESS) {
- macsec_rxsa_put(rx_sa);
- macsec_rxsc_put(rx_sc);
- }
+ macsec_rxsa_put(rx_sa);
+ macsec_rxsc_put(rx_sc);
rcu_read_unlock();
*pskb = NULL;
return RX_HANDLER_CONSUMED;
@@ -1370,41 +1277,19 @@ static rx_handler_result_t macsec_handle_frame(struct sk_buff **pskb)
return RX_HANDLER_PASS;
}
-static struct crypto_aead *macsec_alloc_tfm(char *key, int key_len, int icv_len)
-{
- struct crypto_aead *tfm;
- int ret;
-
- tfm = crypto_alloc_aead("gcm(aes)", 0, 0);
-
- if (IS_ERR(tfm))
- return tfm;
-
- ret = crypto_aead_setkey(tfm, key, key_len);
- if (ret < 0)
- goto fail;
-
- ret = crypto_aead_setauthsize(tfm, icv_len);
- if (ret < 0)
- goto fail;
-
- return tfm;
-fail:
- crypto_free_aead(tfm);
- return ERR_PTR(ret);
-}
-
static int init_rx_sa(struct macsec_rx_sa *rx_sa, char *sak, int key_len,
int icv_len)
{
+ int err;
+
rx_sa->stats = alloc_percpu(struct macsec_rx_sa_stats);
if (!rx_sa->stats)
return -ENOMEM;
- rx_sa->key.tfm = macsec_alloc_tfm(sak, key_len, icv_len);
- if (IS_ERR(rx_sa->key.tfm)) {
+ err = aes_gcm_preparekey(&rx_sa->key.gcm_key, sak, key_len, icv_len);
+ if (err) {
free_percpu(rx_sa->stats);
- return PTR_ERR(rx_sa->key.tfm);
+ return err;
}
rx_sa->ssci = MACSEC_UNDEF_SSCI;
@@ -1498,14 +1383,16 @@ static struct macsec_rx_sc *create_rx_sc(struct net_device *dev, sci_t sci,
static int init_tx_sa(struct macsec_tx_sa *tx_sa, char *sak, int key_len,
int icv_len)
{
+ int err;
+
tx_sa->stats = alloc_percpu(struct macsec_tx_sa_stats);
if (!tx_sa->stats)
return -ENOMEM;
- tx_sa->key.tfm = macsec_alloc_tfm(sak, key_len, icv_len);
- if (IS_ERR(tx_sa->key.tfm)) {
+ err = aes_gcm_preparekey(&tx_sa->key.gcm_key, sak, key_len, icv_len);
+ if (err) {
free_percpu(tx_sa->stats);
- return PTR_ERR(tx_sa->key.tfm);
+ return err;
}
tx_sa->ssci = MACSEC_UNDEF_SSCI;
@@ -1813,7 +1700,7 @@ static int macsec_add_rxsa(struct sk_buff *skb, struct genl_info *info)
err = init_rx_sa(rx_sa, nla_data(tb_sa[MACSEC_SA_ATTR_KEY]),
secy->key_len, secy->icv_len);
if (err < 0) {
- kfree(rx_sa);
+ kfree_sensitive(rx_sa);
rtnl_unlock();
return err;
}
@@ -2021,7 +1908,7 @@ static int macsec_add_txsa(struct sk_buff *skb, struct genl_info *info)
err = init_tx_sa(tx_sa, nla_data(tb_sa[MACSEC_SA_ATTR_KEY]),
secy->key_len, secy->icv_len);
if (err < 0) {
- kfree(tx_sa);
+ kfree_sensitive(tx_sa);
rtnl_unlock();
return err;
}
@@ -3508,8 +3395,7 @@ static netdev_tx_t macsec_start_xmit(struct sk_buff *skb,
len = skb->len;
skb = macsec_encrypt(skb, dev);
if (IS_ERR(skb)) {
- if (PTR_ERR(skb) != -EINPROGRESS)
- DEV_STATS_INC(dev, tx_dropped);
+ DEV_STATS_INC(dev, tx_dropped);
return NETDEV_TX_OK;
}
@@ -4316,17 +4202,8 @@ static int macsec_validate_attr(struct nlattr *tb[], struct nlattr *data[],
if (data[IFLA_MACSEC_ICV_LEN]) {
icv_len = nla_get_u8(data[IFLA_MACSEC_ICV_LEN]);
- if (icv_len != MACSEC_DEFAULT_ICV_LEN) {
- char dummy_key[DEFAULT_SAK_LEN] = { 0 };
- struct crypto_aead *dummy_tfm;
-
- dummy_tfm = macsec_alloc_tfm(dummy_key,
- DEFAULT_SAK_LEN,
- icv_len);
- if (IS_ERR(dummy_tfm))
- return PTR_ERR(dummy_tfm);
- crypto_free_aead(dummy_tfm);
- }
+ if (crypto_gcm_check_authsize(icv_len) != 0)
+ return -EINVAL;
}
es = nla_get_u8_default(data[IFLA_MACSEC_ES], false);
diff --git a/include/net/macsec.h b/include/net/macsec.h
index d962093ee923..a315a33b3ac3 100644
--- a/include/net/macsec.h
+++ b/include/net/macsec.h
@@ -7,6 +7,7 @@
#ifndef _NET_MACSEC_H_
#define _NET_MACSEC_H_
+#include <crypto/aes-gcm.h>
#include <linux/u64_stats_sync.h>
#include <linux/if_vlan.h>
#include <linux/workqueue.h>
@@ -63,12 +64,12 @@ typedef union pn {
/**
* struct macsec_key - SA key
* @id: user-provided key identifier
- * @tfm: crypto struct, key storage
+ * @gcm_key: the AES-GCM key
* @salt: salt used to generate IV in XPN cipher suites
*/
struct macsec_key {
u8 id[MACSEC_KEYID_LEN];
- struct crypto_aead *tfm;
+ struct aes_gcm_key gcm_key;
salt_t salt;
};
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 27/33] wifi: ipw2x00: Use AES-CCM library
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (25 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 26/33] macsec: Use AES-GCM library instead of crypto_aead Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 28/33] mac802154: Use AES-CCM and AES-CTR libraries Eric Biggers
` (5 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there's a library API for AES-CCM, use it instead of a
"ccm(aes)" crypto_aead.
This significantly simplifies the code:
- Dynamic memory allocations go away, including ones during Tx and Rx.
- The additional authenticated data is now just constructed on the
stack, as it no longer needs to be representable in a scatterlist.
- The AES-CCM library supports variable-length nonces via the
self-explanatory nonce_len parameter, so for the needed 13-byte nonce
just use that instead of the odd workaround used by "ccm(aes)" where
the nonce was prefixed with a byte containing '14 - nonce_len'.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
drivers/net/wireless/intel/ipw2x00/Kconfig | 2 +-
.../intel/ipw2x00/libipw_crypto_ccmp.c | 117 +++++-------------
2 files changed, 29 insertions(+), 90 deletions(-)
diff --git a/drivers/net/wireless/intel/ipw2x00/Kconfig b/drivers/net/wireless/intel/ipw2x00/Kconfig
index b508f14542d5..a1073d34f634 100644
--- a/drivers/net/wireless/intel/ipw2x00/Kconfig
+++ b/drivers/net/wireless/intel/ipw2x00/Kconfig
@@ -153,8 +153,8 @@ config LIBIPW
tristate
depends on PCI && CFG80211
select WIRELESS_EXT
- select CRYPTO
select CRYPTO_LIB_ARC4
+ select CRYPTO_LIB_AES_CCM
select CRC32
help
This option enables the hardware independent IEEE 802.11
diff --git a/drivers/net/wireless/intel/ipw2x00/libipw_crypto_ccmp.c b/drivers/net/wireless/intel/ipw2x00/libipw_crypto_ccmp.c
index 631a4dd86cab..aa6dce7acba1 100644
--- a/drivers/net/wireless/intel/ipw2x00/libipw_crypto_ccmp.c
+++ b/drivers/net/wireless/intel/ipw2x00/libipw_crypto_ccmp.c
@@ -19,15 +19,14 @@
#include <asm/string.h>
#include <linux/wireless.h>
#include <linux/ieee80211.h>
-#include <linux/crypto.h>
-#include <crypto/aead.h>
+#include <crypto/aes-ccm.h>
#include "libipw.h"
-#define AES_BLOCK_LEN 16
#define CCMP_HDR_LEN 8
#define CCMP_MIC_LEN 8
#define CCMP_TK_LEN 16
#define CCMP_PN_LEN 6
+#define CCMP_NONCE_LEN 13
struct libipw_ccmp_data {
u8 key[CCMP_TK_LEN];
@@ -42,11 +41,7 @@ struct libipw_ccmp_data {
int key_idx;
- struct crypto_aead *tfm;
-
- /* scratch buffers for virt_to_page() (crypto API) */
- u8 tx_aad[2 * AES_BLOCK_LEN];
- u8 rx_aad[2 * AES_BLOCK_LEN];
+ struct aes_ccm_key ccm_key;
};
static void *libipw_ccmp_init(int key_idx)
@@ -55,37 +50,20 @@ static void *libipw_ccmp_init(int key_idx)
priv = kzalloc_obj(*priv, GFP_ATOMIC);
if (priv == NULL)
- goto fail;
+ return NULL;
priv->key_idx = key_idx;
- priv->tfm = crypto_alloc_aead("ccm(aes)", 0, CRYPTO_ALG_ASYNC);
- if (IS_ERR(priv->tfm)) {
- priv->tfm = NULL;
- goto fail;
- }
-
return priv;
-
- fail:
- if (priv) {
- if (priv->tfm)
- crypto_free_aead(priv->tfm);
- kfree(priv);
- }
-
- return NULL;
}
static void libipw_ccmp_deinit(void *priv)
{
- struct libipw_ccmp_data *_priv = priv;
- if (_priv && _priv->tfm)
- crypto_free_aead(_priv->tfm);
- kfree(priv);
+ kfree_sensitive(priv);
}
-static int ccmp_init_iv_and_aad(const struct ieee80211_hdr *hdr,
- const u8 *pn, u8 *iv, u8 *aad)
+static int ccmp_init_nonce_and_aad(const struct ieee80211_hdr *hdr,
+ const u8 *pn, u8 nonce[CCMP_NONCE_LEN],
+ u8 *aad)
{
u8 *pos, qc = 0;
size_t aad_len;
@@ -105,19 +83,11 @@ static int ccmp_init_iv_and_aad(const struct ieee80211_hdr *hdr,
aad_len += 2;
}
- /* In CCM, the initial vectors (IV) used for CTR mode encryption and CBC
- * mode authentication are not allowed to collide, yet both are derived
- * from the same vector. We only set L := 1 here to indicate that the
- * data size can be represented in (L+1) bytes. The CCM layer will take
- * care of storing the data length in the top (L+1) bytes and setting
- * and clearing the other bits as is required to derive the two IVs.
- */
- iv[0] = 0x1;
-
/* Nonce: QC | A2 | PN */
- iv[1] = qc;
- memcpy(iv + 2, hdr->addr2, ETH_ALEN);
- memcpy(iv + 8, pn, CCMP_PN_LEN);
+ nonce[0] = qc;
+ memcpy(nonce + 1, hdr->addr2, ETH_ALEN);
+ memcpy(nonce + 7, pn, CCMP_PN_LEN);
+ static_assert(7 + CCMP_PN_LEN == CCMP_NONCE_LEN);
/* AAD:
* FC with bits 4..6 and 11..13 masked to zero; 14 is always one
@@ -184,12 +154,9 @@ static int libipw_ccmp_encrypt(struct sk_buff *skb, int hdr_len, void *priv)
{
struct libipw_ccmp_data *key = priv;
struct ieee80211_hdr *hdr;
- struct aead_request *req;
- struct scatterlist sg[2];
- u8 *aad = key->tx_aad;
- u8 iv[AES_BLOCK_LEN];
+ u8 aad[2 * AES_BLOCK_SIZE];
+ u8 nonce[CCMP_NONCE_LEN];
int len, data_len, aad_len;
- int ret;
if (skb_tailroom(skb) < CCMP_MIC_LEN || skb->len < hdr_len)
return -1;
@@ -199,28 +166,16 @@ static int libipw_ccmp_encrypt(struct sk_buff *skb, int hdr_len, void *priv)
if (len < 0)
return -1;
- req = aead_request_alloc(key->tfm, GFP_ATOMIC);
- if (!req)
- return -ENOMEM;
-
hdr = (struct ieee80211_hdr *)skb->data;
- aad_len = ccmp_init_iv_and_aad(hdr, key->tx_pn, iv, aad);
+ aad_len = ccmp_init_nonce_and_aad(hdr, key->tx_pn, nonce, aad);
skb_put(skb, CCMP_MIC_LEN);
- sg_init_table(sg, 2);
- sg_set_buf(&sg[0], aad, aad_len);
- sg_set_buf(&sg[1], skb->data + hdr_len + CCMP_HDR_LEN,
- data_len + CCMP_MIC_LEN);
-
- aead_request_set_callback(req, 0, NULL, NULL);
- aead_request_set_ad(req, aad_len);
- aead_request_set_crypt(req, sg, sg, data_len, iv);
-
- ret = crypto_aead_encrypt(req);
- aead_request_free(req);
-
- return ret;
+ return aes_ccm_encrypt(skb->data + hdr_len + CCMP_HDR_LEN,
+ skb->data + hdr_len + CCMP_HDR_LEN + data_len,
+ skb->data + hdr_len + CCMP_HDR_LEN, data_len,
+ aad, aad_len, nonce, CCMP_NONCE_LEN,
+ &key->ccm_key);
}
/*
@@ -249,10 +204,8 @@ static int libipw_ccmp_decrypt(struct sk_buff *skb, int hdr_len, void *priv)
struct libipw_ccmp_data *key = priv;
u8 keyidx, *pos;
struct ieee80211_hdr *hdr;
- struct aead_request *req;
- struct scatterlist sg[2];
- u8 *aad = key->rx_aad;
- u8 iv[AES_BLOCK_LEN];
+ u8 aad[2 * AES_BLOCK_SIZE];
+ u8 nonce[CCMP_NONCE_LEN];
u8 pn[6];
int aad_len, ret;
size_t data_len = skb->len - hdr_len - CCMP_HDR_LEN;
@@ -303,23 +256,11 @@ static int libipw_ccmp_decrypt(struct sk_buff *skb, int hdr_len, void *priv)
return -4;
}
- req = aead_request_alloc(key->tfm, GFP_ATOMIC);
- if (!req)
- return -ENOMEM;
-
- aad_len = ccmp_init_iv_and_aad(hdr, pn, iv, aad);
-
- sg_init_table(sg, 2);
- sg_set_buf(&sg[0], aad, aad_len);
- sg_set_buf(&sg[1], pos, data_len);
-
- aead_request_set_callback(req, 0, NULL, NULL);
- aead_request_set_ad(req, aad_len);
- aead_request_set_crypt(req, sg, sg, data_len, iv);
-
- ret = crypto_aead_decrypt(req);
- aead_request_free(req);
+ aad_len = ccmp_init_nonce_and_aad(hdr, pn, nonce, aad);
+ ret = aes_ccm_decrypt(pos, pos, pos + data_len - CCMP_MIC_LEN,
+ data_len - CCMP_MIC_LEN, aad, aad_len, nonce,
+ CCMP_NONCE_LEN, &key->ccm_key);
if (ret) {
net_dbg_ratelimited("CCMP: decrypt failed: STA=%pM (%d)\n",
hdr->addr2, ret);
@@ -341,12 +282,10 @@ static int libipw_ccmp_set_key(void *key, int len, u8 * seq, void *priv)
{
struct libipw_ccmp_data *data = priv;
int keyidx;
- struct crypto_aead *tfm = data->tfm;
keyidx = data->key_idx;
memset(data, 0, sizeof(*data));
data->key_idx = keyidx;
- data->tfm = tfm;
if (len == CCMP_TK_LEN) {
memcpy(data->key, key, CCMP_TK_LEN);
data->key_set = 1;
@@ -358,8 +297,8 @@ static int libipw_ccmp_set_key(void *key, int len, u8 * seq, void *priv)
data->rx_pn[4] = seq[1];
data->rx_pn[5] = seq[0];
}
- if (crypto_aead_setauthsize(data->tfm, CCMP_MIC_LEN) ||
- crypto_aead_setkey(data->tfm, data->key, CCMP_TK_LEN))
+ if (aes_ccm_preparekey(&data->ccm_key, data->key, CCMP_TK_LEN,
+ CCMP_MIC_LEN))
return -1;
} else if (len == 0)
data->key_set = 0;
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 28/33] mac802154: Use AES-CCM and AES-CTR libraries
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (26 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 27/33] wifi: ipw2x00: Use AES-CCM library Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 5:34 ` [PATCH 29/33] bpf: crypto: Use AES-CBC and AES-ECB libraries Eric Biggers
` (4 subsequent siblings)
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there are library APIs for AES-CCM and AES-CTR, use these
instead of "ccm(aes)" crypto_aeads and a "ctr(aes)" crypto_skcipher.
This significantly simplifies the code, especially considering that the
existing code already supported only linear skbs anyway.
Note that as before, 'struct mac802154_llsec_key' contains four copies
of the same key prepared in different ways: one for AES-CTR, and one for
each AES-CCM authentication tag length 4, 8, and 16 bytes. That
continues to be wasteful; however, the crypto APIs are very much working
as intended in tying the key to the algorithm and authentication tag
length. It shows this driver is doing something wrong by reusing the
same key for multiple purposes. At some point this driver will need to
be fixed to allow keys with only one algorithm and tag length at a time,
but that's outside the scope of this cleanup.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
net/mac802154/Kconfig | 6 +-
net/mac802154/llsec.c | 158 +++++++++++++-----------------------------
net/mac802154/llsec.h | 8 ++-
3 files changed, 55 insertions(+), 117 deletions(-)
diff --git a/net/mac802154/Kconfig b/net/mac802154/Kconfig
index 901167b1e6f5..5195556670fa 100644
--- a/net/mac802154/Kconfig
+++ b/net/mac802154/Kconfig
@@ -3,11 +3,7 @@ config MAC802154
tristate "Generic IEEE 802.15.4 Soft Networking Stack (mac802154)"
depends on IEEE802154
select CRC_CCITT
- select CRYPTO
- select CRYPTO_AUTHENC
- select CRYPTO_CCM
- select CRYPTO_CTR
- select CRYPTO_AES
+ select CRYPTO_LIB_AES_CCM
help
This option enables the hardware independent IEEE 802.15.4
networking stack for SoftMAC devices (the ones implementing
diff --git a/net/mac802154/llsec.c b/net/mac802154/llsec.c
index 5e7cc11fab3a..367e98c48864 100644
--- a/net/mac802154/llsec.c
+++ b/net/mac802154/llsec.c
@@ -6,14 +6,12 @@
* Phoebe Buckheister <phoebe.buckheister@itwm.fraunhofer.de>
*/
-#include <linux/err.h>
#include <linux/bug.h>
-#include <linux/completion.h>
#include <linux/ieee802154.h>
#include <linux/rculist.h>
-#include <crypto/aead.h>
-#include <crypto/skcipher.h>
+#include <crypto/aes-ccm.h>
+#include <crypto/aes-ctr.h>
#include "ieee802154_i.h"
#include "llsec.h"
@@ -110,11 +108,13 @@ int mac802154_llsec_set_params(struct mac802154_llsec *sec,
return 0;
}
+static const int authsizes[3] = { 4, 8, 16 };
+
static struct mac802154_llsec_key*
llsec_key_alloc(const struct ieee802154_llsec_key *template)
{
- const int authsizes[3] = { 4, 8, 16 };
struct mac802154_llsec_key *key;
+ int err;
int i;
key = kzalloc_obj(*key);
@@ -124,37 +124,21 @@ llsec_key_alloc(const struct ieee802154_llsec_key *template)
kref_init(&key->ref);
key->key = *template;
- BUILD_BUG_ON(ARRAY_SIZE(authsizes) != ARRAY_SIZE(key->tfm));
-
- for (i = 0; i < ARRAY_SIZE(key->tfm); i++) {
- key->tfm[i] = crypto_alloc_aead("ccm(aes)", 0,
- CRYPTO_ALG_ASYNC);
- if (IS_ERR(key->tfm[i]))
- goto err_tfm;
- if (crypto_aead_setkey(key->tfm[i], template->key,
- IEEE802154_LLSEC_KEY_SIZE))
- goto err_tfm;
- if (crypto_aead_setauthsize(key->tfm[i], authsizes[i]))
- goto err_tfm;
+ BUILD_BUG_ON(ARRAY_SIZE(authsizes) != ARRAY_SIZE(key->ccm_keys));
+ for (i = 0; i < ARRAY_SIZE(key->ccm_keys); i++) {
+ err = aes_ccm_preparekey(&key->ccm_keys[i], template->key,
+ IEEE802154_LLSEC_KEY_SIZE,
+ authsizes[i]);
+ if (err)
+ goto err_free;
}
-
- key->tfm0 = crypto_alloc_sync_skcipher("ctr(aes)", 0, 0);
- if (IS_ERR(key->tfm0))
- goto err_tfm;
-
- if (crypto_sync_skcipher_setkey(key->tfm0, template->key,
- IEEE802154_LLSEC_KEY_SIZE))
- goto err_tfm0;
-
+ err = aes_prepareenckey(&key->ctr_key, template->key,
+ IEEE802154_LLSEC_KEY_SIZE);
+ if (err)
+ goto err_free;
return key;
-err_tfm0:
- crypto_free_sync_skcipher(key->tfm0);
-err_tfm:
- for (i = 0; i < ARRAY_SIZE(key->tfm); i++)
- if (!IS_ERR_OR_NULL(key->tfm[i]))
- crypto_free_aead(key->tfm[i]);
-
+err_free:
kfree_sensitive(key);
return NULL;
}
@@ -162,14 +146,8 @@ llsec_key_alloc(const struct ieee802154_llsec_key *template)
static void llsec_key_release(struct kref *ref)
{
struct mac802154_llsec_key *key;
- int i;
key = container_of(ref, struct mac802154_llsec_key, ref);
-
- for (i = 0; i < ARRAY_SIZE(key->tfm); i++)
- crypto_free_aead(key->tfm[i]);
-
- crypto_free_sync_skcipher(key->tfm0);
kfree_sensitive(key);
}
@@ -621,34 +599,24 @@ llsec_do_encrypt_unauth(struct sk_buff *skb, const struct mac802154_llsec *sec,
struct mac802154_llsec_key *key)
{
u8 iv[16];
- struct scatterlist src;
- SYNC_SKCIPHER_REQUEST_ON_STACK(req, key->tfm0);
- int err, datalen;
+ int datalen;
unsigned char *data;
llsec_geniv(iv, sec->params.hwaddr, &hdr->sec);
/* Compute data payload offset and data length */
data = skb_mac_header(skb) + skb->mac_len;
datalen = skb_tail_pointer(skb) - data;
- sg_init_one(&src, data, datalen);
- skcipher_request_set_sync_tfm(req, key->tfm0);
- skcipher_request_set_callback(req, 0, NULL, NULL);
- skcipher_request_set_crypt(req, &src, &src, datalen, iv);
- err = crypto_skcipher_encrypt(req);
- skcipher_request_zero(req);
- return err;
+ aes_ctr(data, data, datalen, iv, &key->ctr_key);
+ return 0;
}
-static struct crypto_aead*
-llsec_tfm_by_len(struct mac802154_llsec_key *key, int authlen)
+static const struct aes_ccm_key *
+llsec_ccm_key_by_authlen(const struct mac802154_llsec_key *key, int authlen)
{
- int i;
-
- for (i = 0; i < ARRAY_SIZE(key->tfm); i++)
- if (crypto_aead_authsize(key->tfm[i]) == authlen)
- return key->tfm[i];
-
+ for (int i = 0; i < ARRAY_SIZE(authsizes); i++)
+ if (authsizes[i] == authlen)
+ return &key->ccm_keys[i];
BUG();
}
@@ -658,41 +626,29 @@ llsec_do_encrypt_auth(struct sk_buff *skb, const struct mac802154_llsec *sec,
struct mac802154_llsec_key *key)
{
u8 iv[16];
- unsigned char *data;
- int authlen, assoclen, datalen, rc;
- struct scatterlist sg;
- struct aead_request *req;
+ u8 *authtag, *assoc, *data;
+ int authlen, assoclen, datalen;
authlen = ieee802154_sechdr_authtag_len(&hdr->sec);
llsec_geniv(iv, sec->params.hwaddr, &hdr->sec);
- req = aead_request_alloc(llsec_tfm_by_len(key, authlen), GFP_ATOMIC);
- if (!req)
- return -ENOMEM;
-
+ assoc = skb_mac_header(skb);
assoclen = skb->mac_len;
-
- data = skb_mac_header(skb) + skb->mac_len;
+ data = assoc + assoclen;
datalen = skb_tail_pointer(skb) - data;
+ authtag = &data[datalen];
skb_put(skb, authlen);
- sg_init_one(&sg, skb_mac_header(skb), assoclen + datalen + authlen);
-
if (!(hdr->sec.level & IEEE802154_SCF_SECLEVEL_ENC)) {
assoclen += datalen;
datalen = 0;
}
- aead_request_set_callback(req, 0, NULL, NULL);
- aead_request_set_crypt(req, &sg, &sg, datalen, iv);
- aead_request_set_ad(req, assoclen);
-
- rc = crypto_aead_encrypt(req);
-
- kfree_sensitive(req);
-
- return rc;
+ return aes_ccm_encrypt(data, authtag, data, datalen, assoc, assoclen,
+ /* This just takes the nonce directly. */
+ &iv[1], 13,
+ llsec_ccm_key_by_authlen(key, authlen));
}
static int llsec_do_encrypt(struct sk_buff *skb,
@@ -849,23 +805,13 @@ llsec_do_decrypt_unauth(struct sk_buff *skb, const struct mac802154_llsec *sec,
u8 iv[16];
unsigned char *data;
int datalen;
- struct scatterlist src;
- SYNC_SKCIPHER_REQUEST_ON_STACK(req, key->tfm0);
- int err;
llsec_geniv(iv, dev_addr, &hdr->sec);
data = skb_mac_header(skb) + skb->mac_len;
datalen = skb_tail_pointer(skb) - data;
- sg_init_one(&src, data, datalen);
-
- skcipher_request_set_sync_tfm(req, key->tfm0);
- skcipher_request_set_callback(req, 0, NULL, NULL);
- skcipher_request_set_crypt(req, &src, &src, datalen, iv);
-
- err = crypto_skcipher_decrypt(req);
- skcipher_request_zero(req);
- return err;
+ aes_ctr(data, data, datalen, iv, &key->ctr_key);
+ return 0;
}
static int
@@ -874,37 +820,31 @@ llsec_do_decrypt_auth(struct sk_buff *skb, const struct mac802154_llsec *sec,
struct mac802154_llsec_key *key, __le64 dev_addr)
{
u8 iv[16];
- unsigned char *data;
+ u8 *authtag, *assoc, *data;
int authlen, datalen, assoclen, rc;
- struct scatterlist sg;
- struct aead_request *req;
authlen = ieee802154_sechdr_authtag_len(&hdr->sec);
llsec_geniv(iv, dev_addr, &hdr->sec);
- req = aead_request_alloc(llsec_tfm_by_len(key, authlen), GFP_ATOMIC);
- if (!req)
- return -ENOMEM;
-
+ assoc = skb_mac_header(skb);
assoclen = skb->mac_len;
-
- data = skb_mac_header(skb) + skb->mac_len;
+ data = assoc + assoclen;
datalen = skb_tail_pointer(skb) - data;
- sg_init_one(&sg, skb_mac_header(skb), assoclen + datalen);
+ if (datalen < authlen)
+ return -EBADMSG;
+ datalen -= authlen;
+ authtag = &data[datalen];
if (!(hdr->sec.level & IEEE802154_SCF_SECLEVEL_ENC)) {
- assoclen += datalen - authlen;
- datalen = authlen;
+ assoclen += datalen;
+ datalen = 0;
}
- aead_request_set_callback(req, 0, NULL, NULL);
- aead_request_set_crypt(req, &sg, &sg, datalen, iv);
- aead_request_set_ad(req, assoclen);
-
- rc = crypto_aead_decrypt(req);
-
- kfree_sensitive(req);
+ rc = aes_ccm_decrypt(data, data, authtag, datalen, assoc, assoclen,
+ /* This just takes the nonce directly. */
+ &iv[1], 13,
+ llsec_ccm_key_by_authlen(key, authlen));
skb_trim(skb, skb->len - authlen);
return rc;
diff --git a/net/mac802154/llsec.h b/net/mac802154/llsec.h
index ddeb79228204..2b9815dbe38f 100644
--- a/net/mac802154/llsec.h
+++ b/net/mac802154/llsec.h
@@ -9,6 +9,8 @@
#ifndef MAC802154_LLSEC_H
#define MAC802154_LLSEC_H
+#include <crypto/aes-ccm.h>
+#include <crypto/aes-ctr.h>
#include <linux/slab.h>
#include <linux/hashtable.h>
#include <linux/kref.h>
@@ -19,9 +21,9 @@
struct mac802154_llsec_key {
struct ieee802154_llsec_key key;
- /* one tfm for each authsize (4/8/16) */
- struct crypto_aead *tfm[3];
- struct crypto_sync_skcipher *tfm0;
+ /* one for each authsize (4/8/16) */
+ struct aes_ccm_key ccm_keys[3];
+ struct aes_enckey ctr_key;
struct kref ref;
};
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 29/33] bpf: crypto: Use AES-CBC and AES-ECB libraries
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (27 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 28/33] mac802154: Use AES-CCM and AES-CTR libraries Eric Biggers
@ 2026-07-07 5:34 ` Eric Biggers
2026-07-07 15:01 ` Vadim Fedorenko
2026-07-07 5:35 ` [PATCH 30/33] bpf: crypto: Add AES-GCM support Eric Biggers
` (3 subsequent siblings)
32 siblings, 1 reply; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:34 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
BPF crypto was implemented using the lskcipher API, which doesn't seem
to be going anywhere. It supports only "arc4", "cbc(aes)", "ecb(aes)",
and only with unoptimized implementations.
Library APIs also have been found to be a much better approach, for a
variety of reasons, including reduced overhead, greater flexibility, and
having to be explicit about the crypto algorithms that are supported.
We can safely ignore the theoretical "arc4" support in BPF crypto as
unused, which leaves "cbc(aes)" and "ecb(aes)". Why these algorithms
were chosen, it's unclear. Regardless, I'll assume that "cbc(aes)" and
"ecb(aes)" need to continue to be supported for backwards compatibility.
There are library APIs for these now, which are much easier to use and
more efficient. Reimplement BPF crypto on top of them, greatly
simplifying the code. As part of this, the bpf_crypto_type abstraction
layer is removed, as it's not useful.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
MAINTAINERS | 2 -
crypto/Makefile | 3 -
crypto/bpf_crypto_skcipher.c | 83 -----------
include/linux/bpf_crypto.h | 24 ----
kernel/bpf/Kconfig | 12 ++
kernel/bpf/Makefile | 4 +-
kernel/bpf/crypto.c | 270 ++++++++++++++---------------------
7 files changed, 124 insertions(+), 274 deletions(-)
delete mode 100644 crypto/bpf_crypto_skcipher.c
delete mode 100644 include/linux/bpf_crypto.h
diff --git a/MAINTAINERS b/MAINTAINERS
index 4a8b0fd665ce..6585b1a26563 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -4894,8 +4894,6 @@ BPF [CRYPTO]
M: Vadim Fedorenko <vadim.fedorenko@linux.dev>
L: bpf@vger.kernel.org
S: Maintained
-F: crypto/bpf_crypto_skcipher.c
-F: include/linux/bpf_crypto.h
F: kernel/bpf/crypto.c
BPF [DOCUMENTATION] (Related to Standardization)
diff --git a/crypto/Makefile b/crypto/Makefile
index 8386d55a9755..33bb5ad595e8 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -22,9 +22,6 @@ crypto_skcipher-y += lskcipher.o
crypto_skcipher-y += skcipher.o
obj-$(CONFIG_CRYPTO_SKCIPHER2) += crypto_skcipher.o
-ifeq ($(CONFIG_BPF_SYSCALL),y)
-obj-$(CONFIG_CRYPTO_SKCIPHER2) += bpf_crypto_skcipher.o
-endif
obj-$(CONFIG_CRYPTO_SEQIV) += seqiv.o
obj-$(CONFIG_CRYPTO_ECHAINIV) += echainiv.o
diff --git a/crypto/bpf_crypto_skcipher.c b/crypto/bpf_crypto_skcipher.c
deleted file mode 100644
index a88798d3e8c8..000000000000
--- a/crypto/bpf_crypto_skcipher.c
+++ /dev/null
@@ -1,83 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/* Copyright (c) 2024 Meta, Inc */
-#include <linux/types.h>
-#include <linux/module.h>
-#include <linux/bpf_crypto.h>
-#include <crypto/skcipher.h>
-
-static void *bpf_crypto_lskcipher_alloc_tfm(const char *algo)
-{
- return crypto_alloc_lskcipher(algo, 0, 0);
-}
-
-static void bpf_crypto_lskcipher_free_tfm(void *tfm)
-{
- crypto_free_lskcipher(tfm);
-}
-
-static int bpf_crypto_lskcipher_has_algo(const char *algo)
-{
- return crypto_has_skcipher(algo, CRYPTO_ALG_TYPE_LSKCIPHER, CRYPTO_ALG_TYPE_MASK);
-}
-
-static int bpf_crypto_lskcipher_setkey(void *tfm, const u8 *key, unsigned int keylen)
-{
- return crypto_lskcipher_setkey(tfm, key, keylen);
-}
-
-static u32 bpf_crypto_lskcipher_get_flags(void *tfm)
-{
- return crypto_lskcipher_get_flags(tfm);
-}
-
-static unsigned int bpf_crypto_lskcipher_ivsize(void *tfm)
-{
- return crypto_lskcipher_ivsize(tfm);
-}
-
-static unsigned int bpf_crypto_lskcipher_statesize(void *tfm)
-{
- return crypto_lskcipher_statesize(tfm);
-}
-
-static int bpf_crypto_lskcipher_encrypt(void *tfm, const u8 *src, u8 *dst,
- unsigned int len, u8 *siv)
-{
- return crypto_lskcipher_encrypt(tfm, src, dst, len, siv);
-}
-
-static int bpf_crypto_lskcipher_decrypt(void *tfm, const u8 *src, u8 *dst,
- unsigned int len, u8 *siv)
-{
- return crypto_lskcipher_decrypt(tfm, src, dst, len, siv);
-}
-
-static const struct bpf_crypto_type bpf_crypto_lskcipher_type = {
- .alloc_tfm = bpf_crypto_lskcipher_alloc_tfm,
- .free_tfm = bpf_crypto_lskcipher_free_tfm,
- .has_algo = bpf_crypto_lskcipher_has_algo,
- .setkey = bpf_crypto_lskcipher_setkey,
- .encrypt = bpf_crypto_lskcipher_encrypt,
- .decrypt = bpf_crypto_lskcipher_decrypt,
- .ivsize = bpf_crypto_lskcipher_ivsize,
- .statesize = bpf_crypto_lskcipher_statesize,
- .get_flags = bpf_crypto_lskcipher_get_flags,
- .owner = THIS_MODULE,
- .name = "skcipher",
-};
-
-static int __init bpf_crypto_skcipher_init(void)
-{
- return bpf_crypto_register_type(&bpf_crypto_lskcipher_type);
-}
-
-static void __exit bpf_crypto_skcipher_exit(void)
-{
- int err = bpf_crypto_unregister_type(&bpf_crypto_lskcipher_type);
- WARN_ON_ONCE(err);
-}
-
-module_init(bpf_crypto_skcipher_init);
-module_exit(bpf_crypto_skcipher_exit);
-MODULE_LICENSE("GPL");
-MODULE_DESCRIPTION("Symmetric key cipher support for BPF");
diff --git a/include/linux/bpf_crypto.h b/include/linux/bpf_crypto.h
deleted file mode 100644
index a41e71d4e2d9..000000000000
--- a/include/linux/bpf_crypto.h
+++ /dev/null
@@ -1,24 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/* Copyright (c) 2024 Meta Platforms, Inc. and affiliates. */
-#ifndef _BPF_CRYPTO_H
-#define _BPF_CRYPTO_H
-
-struct bpf_crypto_type {
- void *(*alloc_tfm)(const char *algo);
- void (*free_tfm)(void *tfm);
- int (*has_algo)(const char *algo);
- int (*setkey)(void *tfm, const u8 *key, unsigned int keylen);
- int (*setauthsize)(void *tfm, unsigned int authsize);
- int (*encrypt)(void *tfm, const u8 *src, u8 *dst, unsigned int len, u8 *iv);
- int (*decrypt)(void *tfm, const u8 *src, u8 *dst, unsigned int len, u8 *iv);
- unsigned int (*ivsize)(void *tfm);
- unsigned int (*statesize)(void *tfm);
- u32 (*get_flags)(void *tfm);
- struct module *owner;
- char name[14];
-};
-
-int bpf_crypto_register_type(const struct bpf_crypto_type *type);
-int bpf_crypto_unregister_type(const struct bpf_crypto_type *type);
-
-#endif /* _BPF_CRYPTO_H */
diff --git a/kernel/bpf/Kconfig b/kernel/bpf/Kconfig
index eb3de35734f0..c4f1086b2daf 100644
--- a/kernel/bpf/Kconfig
+++ b/kernel/bpf/Kconfig
@@ -87,6 +87,18 @@ config BPF_UNPRIV_DEFAULT_OFF
If you are unsure how to answer this question, answer Y.
+config BPF_CRYPTO
+ def_bool y
+ depends on BPF_SYSCALL
+ select CRYPTO_LIB_AES_CBC
+ select CRYPTO_LIB_AES_ECB
+ help
+ Provide the kfuncs needed for BPF programs to encrypt and decrypt
+ data. The supported algorithms are:
+
+ - AES-CBC
+ - AES-ECB
+
source "kernel/bpf/preload/Kconfig"
config BPF_LSM
diff --git a/kernel/bpf/Makefile b/kernel/bpf/Makefile
index 4dc41bf5780c..42899bfcbd82 100644
--- a/kernel/bpf/Makefile
+++ b/kernel/bpf/Makefile
@@ -55,9 +55,7 @@ obj-$(CONFIG_BPF_SYSCALL) += cpumask.o
# semantics within pahole are revisited accordingly.
obj-${CONFIG_BPF_LSM} += bpf_lsm_proto.o bpf_lsm.o
endif
-ifneq ($(CONFIG_CRYPTO),)
-obj-$(CONFIG_BPF_SYSCALL) += crypto.o
-endif
+obj-$(CONFIG_BPF_CRYPTO) += crypto.o
obj-$(CONFIG_BPF_PRELOAD) += preload/
obj-$(CONFIG_BPF_SYSCALL) += relo_core.o
diff --git a/kernel/bpf/crypto.c b/kernel/bpf/crypto.c
index 51f89cecefb4..17e0d2fd422a 100644
--- a/kernel/bpf/crypto.c
+++ b/kernel/bpf/crypto.c
@@ -1,19 +1,13 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright (c) 2024 Meta, Inc */
#include <linux/bpf.h>
-#include <linux/bpf_crypto.h>
#include <linux/bpf_mem_alloc.h>
#include <linux/btf.h>
#include <linux/btf_ids.h>
#include <linux/filter.h>
-#include <linux/scatterlist.h>
#include <linux/skbuff.h>
-#include <crypto/skcipher.h>
-
-struct bpf_crypto_type_list {
- const struct bpf_crypto_type *type;
- struct list_head list;
-};
+#include <crypto/aes-cbc.h>
+#include <crypto/aes-ecb.h>
/* BPF crypto initialization parameters struct */
/**
@@ -36,94 +30,53 @@ struct bpf_crypto_params {
u32 authsize;
};
-static LIST_HEAD(bpf_crypto_types);
-static DECLARE_RWSEM(bpf_crypto_types_sem);
+enum bpf_crypto_algo_id {
+ BPF_ALGO_AES_CBC,
+ BPF_ALGO_AES_ECB,
+};
+
+static const struct {
+ const char *type_name;
+ const char *algo_name;
+ enum bpf_crypto_algo_id algo;
+} bpf_crypto_algos[] = {
+ { "skcipher", "cbc(aes)", BPF_ALGO_AES_CBC },
+ { "skcipher", "ecb(aes)", BPF_ALGO_AES_ECB },
+};
+
+static bool bpf_crypto_find_algo(const struct bpf_crypto_params *params,
+ enum bpf_crypto_algo_id *id_ret)
+{
+ for (size_t i = 0; i < ARRAY_SIZE(bpf_crypto_algos); i++) {
+ if (strncmp(bpf_crypto_algos[i].type_name, params->type,
+ sizeof(params->type)) == 0 &&
+ strncmp(bpf_crypto_algos[i].algo_name, params->algo,
+ sizeof(params->algo)) == 0) {
+ *id_ret = bpf_crypto_algos[i].algo;
+ return true;
+ }
+ }
+ return false;
+}
/**
* struct bpf_crypto_ctx - refcounted BPF crypto context structure
- * @type: The pointer to bpf crypto type
- * @tfm: The pointer to instance of crypto API struct.
- * @siv_len: Size of IV and state storage for cipher
+ * @algo: The crypto algorithm ID
+ * @key: The crypto key
* @rcu: The RCU head used to free the crypto context with RCU safety.
* @usage: Object reference counter. When the refcount goes to 0, the
* memory is released back to the BPF allocator, which provides
* RCU safety.
*/
struct bpf_crypto_ctx {
- const struct bpf_crypto_type *type;
- void *tfm;
- u32 siv_len;
+ enum bpf_crypto_algo_id algo;
+ union {
+ struct aes_key aes;
+ } key;
struct rcu_head rcu;
refcount_t usage;
};
-int bpf_crypto_register_type(const struct bpf_crypto_type *type)
-{
- struct bpf_crypto_type_list *node;
- int err = -EBUSY;
-
- down_write(&bpf_crypto_types_sem);
- list_for_each_entry(node, &bpf_crypto_types, list) {
- if (!strcmp(node->type->name, type->name))
- goto unlock;
- }
-
- node = kmalloc_obj(*node);
- err = -ENOMEM;
- if (!node)
- goto unlock;
-
- node->type = type;
- list_add(&node->list, &bpf_crypto_types);
- err = 0;
-
-unlock:
- up_write(&bpf_crypto_types_sem);
-
- return err;
-}
-EXPORT_SYMBOL_GPL(bpf_crypto_register_type);
-
-int bpf_crypto_unregister_type(const struct bpf_crypto_type *type)
-{
- struct bpf_crypto_type_list *node;
- int err = -ENOENT;
-
- down_write(&bpf_crypto_types_sem);
- list_for_each_entry(node, &bpf_crypto_types, list) {
- if (strcmp(node->type->name, type->name))
- continue;
-
- list_del(&node->list);
- kfree(node);
- err = 0;
- break;
- }
- up_write(&bpf_crypto_types_sem);
-
- return err;
-}
-EXPORT_SYMBOL_GPL(bpf_crypto_unregister_type);
-
-static const struct bpf_crypto_type *bpf_crypto_get_type(const char *name)
-{
- const struct bpf_crypto_type *type = ERR_PTR(-ENOENT);
- struct bpf_crypto_type_list *node;
-
- down_read(&bpf_crypto_types_sem);
- list_for_each_entry(node, &bpf_crypto_types, list) {
- if (strcmp(node->type->name, name))
- continue;
-
- if (try_module_get(node->type->owner))
- type = node->type;
- break;
- }
- up_read(&bpf_crypto_types_sem);
-
- return type;
-}
-
__bpf_kfunc_start_defs();
/**
@@ -146,7 +99,6 @@ __bpf_kfunc struct bpf_crypto_ctx *
bpf_crypto_ctx_create(const struct bpf_crypto_params *params, u32 params__sz,
int *err)
{
- const struct bpf_crypto_type *type;
struct bpf_crypto_ctx *ctx;
if (!params || params->reserved[0] || params->reserved[1] ||
@@ -155,69 +107,39 @@ bpf_crypto_ctx_create(const struct bpf_crypto_params *params, u32 params__sz,
return NULL;
}
- type = bpf_crypto_get_type(params->type);
- if (IS_ERR(type)) {
- *err = PTR_ERR(type);
- return NULL;
- }
-
- if (!type->has_algo(params->algo)) {
- *err = -EOPNOTSUPP;
- goto err_module_put;
- }
-
- if (!!params->authsize ^ !!type->setauthsize) {
- *err = -EOPNOTSUPP;
- goto err_module_put;
- }
-
- if (!params->key_len || params->key_len > sizeof(params->key)) {
- *err = -EINVAL;
- goto err_module_put;
- }
-
ctx = kzalloc_obj(*ctx);
if (!ctx) {
*err = -ENOMEM;
- goto err_module_put;
+ return NULL;
}
- ctx->type = type;
- ctx->tfm = type->alloc_tfm(params->algo);
- if (IS_ERR(ctx->tfm)) {
- *err = PTR_ERR(ctx->tfm);
- goto err_free_ctx;
+ if (!bpf_crypto_find_algo(params, &ctx->algo)) {
+ *err = -ENOENT;
+ goto out;
}
- if (params->authsize) {
- *err = type->setauthsize(ctx->tfm, params->authsize);
- if (*err)
- goto err_free_tfm;
+ switch (ctx->algo) {
+ case BPF_ALGO_AES_CBC:
+ case BPF_ALGO_AES_ECB:
+ if (params->authsize)
+ *err = -EOPNOTSUPP;
+ else
+ *err = aes_preparekey(&ctx->key.aes, params->key,
+ params->key_len);
+ break;
+ default:
+ WARN_ON(1);
+ *err = -ENOENT;
}
- *err = type->setkey(ctx->tfm, params->key, params->key_len);
- if (*err)
- goto err_free_tfm;
-
- if (type->get_flags(ctx->tfm) & CRYPTO_TFM_NEED_KEY) {
- *err = -EINVAL;
- goto err_free_tfm;
+out:
+ if (*err) {
+ kfree_sensitive(ctx);
+ return NULL;
}
- ctx->siv_len = type->ivsize(ctx->tfm) + type->statesize(ctx->tfm);
-
refcount_set(&ctx->usage, 1);
-
return ctx;
-
-err_free_tfm:
- type->free_tfm(ctx->tfm);
-err_free_ctx:
- kfree(ctx);
-err_module_put:
- module_put(type->owner);
-
- return NULL;
}
static void crypto_free_cb(struct rcu_head *head)
@@ -225,9 +147,7 @@ static void crypto_free_cb(struct rcu_head *head)
struct bpf_crypto_ctx *ctx;
ctx = container_of(head, struct bpf_crypto_ctx, rcu);
- ctx->type->free_tfm(ctx->tfm);
- module_put(ctx->type->owner);
- kfree(ctx);
+ kfree_sensitive(ctx);
}
/**
@@ -267,27 +187,53 @@ __bpf_kfunc void bpf_crypto_ctx_release_dtor(void *ctx)
}
CFI_NOSEAL(bpf_crypto_ctx_release_dtor);
+static int bpf_aes_cbc_crypt(u8 *dst, u32 dst_len, const u8 *src, u32 src_len,
+ u8 *iv, u32 iv_len,
+ const struct bpf_crypto_ctx *ctx, bool decrypt)
+{
+ if (iv_len != AES_BLOCK_SIZE)
+ return -EINVAL;
+ if (src_len % AES_BLOCK_SIZE || dst_len < src_len)
+ return -EINVAL;
+ if (decrypt)
+ aes_cbc_decrypt(dst, src, src_len, iv, &ctx->key.aes);
+ else
+ aes_cbc_encrypt(dst, src, src_len, iv, &ctx->key.aes);
+ return 0;
+}
+
+static int bpf_aes_ecb_crypt(u8 *dst, u32 dst_len, const u8 *src, u32 src_len,
+ u8 *iv, u32 iv_len,
+ const struct bpf_crypto_ctx *ctx, bool decrypt)
+{
+ if (iv_len != 0)
+ return -EINVAL;
+ if (src_len % AES_BLOCK_SIZE || dst_len < src_len)
+ return -EINVAL;
+ if (decrypt)
+ aes_ecb_decrypt(dst, src, src_len, &ctx->key.aes);
+ else
+ aes_ecb_encrypt(dst, src, src_len, &ctx->key.aes);
+ return 0;
+}
+
static int bpf_crypto_crypt(const struct bpf_crypto_ctx *ctx,
const struct bpf_dynptr_kern *src,
const struct bpf_dynptr_kern *dst,
- const struct bpf_dynptr_kern *siv,
+ const struct bpf_dynptr_kern *iv,
bool decrypt)
{
- u32 src_len, dst_len, siv_len;
+ u32 src_len, dst_len, iv_len;
const u8 *psrc;
u8 *pdst, *piv;
- int err;
if (__bpf_dynptr_is_rdonly(dst))
return -EINVAL;
- siv_len = siv ? __bpf_dynptr_size(siv) : 0;
+ iv_len = iv ? __bpf_dynptr_size(iv) : 0;
src_len = __bpf_dynptr_size(src);
dst_len = __bpf_dynptr_size(dst);
- if (!src_len || !dst_len || src_len > dst_len)
- return -EINVAL;
-
- if (siv_len != ctx->siv_len)
+ if (!src_len || !dst_len)
return -EINVAL;
psrc = __bpf_dynptr_data(src, src_len);
@@ -297,14 +243,20 @@ static int bpf_crypto_crypt(const struct bpf_crypto_ctx *ctx,
if (!pdst)
return -EINVAL;
- piv = siv_len ? __bpf_dynptr_data_rw(siv, siv_len) : NULL;
- if (siv_len && !piv)
+ piv = iv_len ? __bpf_dynptr_data_rw(iv, iv_len) : NULL;
+ if (iv_len && !piv)
return -EINVAL;
- err = decrypt ? ctx->type->decrypt(ctx->tfm, psrc, pdst, src_len, piv)
- : ctx->type->encrypt(ctx->tfm, psrc, pdst, src_len, piv);
-
- return err;
+ switch (ctx->algo) {
+ case BPF_ALGO_AES_CBC:
+ return bpf_aes_cbc_crypt(pdst, dst_len, psrc, src_len, piv,
+ iv_len, ctx, decrypt);
+ case BPF_ALGO_AES_ECB:
+ return bpf_aes_ecb_crypt(pdst, dst_len, psrc, src_len, piv,
+ iv_len, ctx, decrypt);
+ default:
+ return -EINVAL;
+ }
}
/**
@@ -312,20 +264,20 @@ static int bpf_crypto_crypt(const struct bpf_crypto_ctx *ctx,
* @ctx: The crypto context being used. The ctx must be a trusted pointer.
* @src: bpf_dynptr to the encrypted data. Must be a trusted pointer.
* @dst: bpf_dynptr to the buffer where to store the result. Must be a trusted pointer.
- * @siv__nullable: bpf_dynptr to IV data and state data to be used by decryptor. May be NULL.
+ * @iv__nullable: bpf_dynptr to the initialization vector. May be NULL.
*
* Decrypts provided buffer using IV data and the crypto context. Crypto context must be configured.
*/
__bpf_kfunc int bpf_crypto_decrypt(struct bpf_crypto_ctx *ctx,
const struct bpf_dynptr *src,
const struct bpf_dynptr *dst,
- const struct bpf_dynptr *siv__nullable)
+ const struct bpf_dynptr *iv__nullable)
{
const struct bpf_dynptr_kern *src_kern = (struct bpf_dynptr_kern *)src;
const struct bpf_dynptr_kern *dst_kern = (struct bpf_dynptr_kern *)dst;
- const struct bpf_dynptr_kern *siv_kern = (struct bpf_dynptr_kern *)siv__nullable;
+ const struct bpf_dynptr_kern *iv_kern = (struct bpf_dynptr_kern *)iv__nullable;
- return bpf_crypto_crypt(ctx, src_kern, dst_kern, siv_kern, true);
+ return bpf_crypto_crypt(ctx, src_kern, dst_kern, iv_kern, true);
}
/**
@@ -333,20 +285,20 @@ __bpf_kfunc int bpf_crypto_decrypt(struct bpf_crypto_ctx *ctx,
* @ctx: The crypto context being used. The ctx must be a trusted pointer.
* @src: bpf_dynptr to the plain data. Must be a trusted pointer.
* @dst: bpf_dynptr to the buffer where to store the result. Must be a trusted pointer.
- * @siv__nullable: bpf_dynptr to IV data and state data to be used by decryptor. May be NULL.
+ * @iv__nullable: bpf_dynptr to the initialization vector. May be NULL.
*
* Encrypts provided buffer using IV data and the crypto context. Crypto context must be configured.
*/
__bpf_kfunc int bpf_crypto_encrypt(struct bpf_crypto_ctx *ctx,
const struct bpf_dynptr *src,
const struct bpf_dynptr *dst,
- const struct bpf_dynptr *siv__nullable)
+ const struct bpf_dynptr *iv__nullable)
{
const struct bpf_dynptr_kern *src_kern = (struct bpf_dynptr_kern *)src;
const struct bpf_dynptr_kern *dst_kern = (struct bpf_dynptr_kern *)dst;
- const struct bpf_dynptr_kern *siv_kern = (struct bpf_dynptr_kern *)siv__nullable;
+ const struct bpf_dynptr_kern *iv_kern = (struct bpf_dynptr_kern *)iv__nullable;
- return bpf_crypto_crypt(ctx, src_kern, dst_kern, siv_kern, false);
+ return bpf_crypto_crypt(ctx, src_kern, dst_kern, iv_kern, false);
}
__bpf_kfunc_end_defs();
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [PATCH 29/33] bpf: crypto: Use AES-CBC and AES-ECB libraries
2026-07-07 5:34 ` [PATCH 29/33] bpf: crypto: Use AES-CBC and AES-ECB libraries Eric Biggers
@ 2026-07-07 15:01 ` Vadim Fedorenko
2026-07-07 18:20 ` Eric Biggers
0 siblings, 1 reply; 45+ messages in thread
From: Vadim Fedorenko @ 2026-07-07 15:01 UTC (permalink / raw)
To: Eric Biggers, linux-crypto; +Cc: linux-kernel, bpf
cc +bpf
On 07/07/2026 06:34, Eric Biggers wrote:
> BPF crypto was implemented using the lskcipher API, which doesn't seem
> to be going anywhere. It supports only "arc4", "cbc(aes)", "ecb(aes)",
> and only with unoptimized implementations.
>
> Library APIs also have been found to be a much better approach, for a
> variety of reasons, including reduced overhead, greater flexibility, and
> having to be explicit about the crypto algorithms that are supported.
>
> We can safely ignore the theoretical "arc4" support in BPF crypto as
> unused, which leaves "cbc(aes)" and "ecb(aes)". Why these algorithms
> were chosen, it's unclear. Regardless, I'll assume that "cbc(aes)" and
> "ecb(aes)" need to continue to be supported for backwards compatibility.
That was done for single use case of decrypting small blocks in TC
layer with "cbc(aes)", with assumption of extending it later.
This change looks great, but it would be great to CC bpf folks just to
be aware of the refactoring.
>
> There are library APIs for these now, which are much easier to use and
> more efficient. Reimplement BPF crypto on top of them, greatly
> simplifying the code. As part of this, the bpf_crypto_type abstraction
> layer is removed, as it's not useful.
>
> Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Reviewed-by: Vadim Fedorenko <vadim.fedorenko@linux.dev>
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 29/33] bpf: crypto: Use AES-CBC and AES-ECB libraries
2026-07-07 15:01 ` Vadim Fedorenko
@ 2026-07-07 18:20 ` Eric Biggers
2026-07-07 22:50 ` Vadim Fedorenko
0 siblings, 1 reply; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 18:20 UTC (permalink / raw)
To: Vadim Fedorenko; +Cc: linux-crypto, linux-kernel, bpf
On Tue, Jul 07, 2026 at 04:01:13PM +0100, Vadim Fedorenko wrote:
> cc +bpf
>
> On 07/07/2026 06:34, Eric Biggers wrote:
> > BPF crypto was implemented using the lskcipher API, which doesn't seem
> > to be going anywhere. It supports only "arc4", "cbc(aes)", "ecb(aes)",
> > and only with unoptimized implementations.
> >
> > Library APIs also have been found to be a much better approach, for a
> > variety of reasons, including reduced overhead, greater flexibility, and
> > having to be explicit about the crypto algorithms that are supported.
> >
> > We can safely ignore the theoretical "arc4" support in BPF crypto as
> > unused, which leaves "cbc(aes)" and "ecb(aes)". Why these algorithms
> > were chosen, it's unclear. Regardless, I'll assume that "cbc(aes)" and
> > "ecb(aes)" need to continue to be supported for backwards compatibility.
>
> That was done for single use case of decrypting small blocks in TC
> layer with "cbc(aes)", with assumption of extending it later.
What protocol is using AES-CBC? And is the kernel encrypting or
decrypting the data elsewhere, or it is just routing an encrypted packet
and only the BPF program decrypts it?
Does this mean the AES-ECB support is unnecessary and can be dropped?
> This change looks great, but it would be great to CC bpf folks just to
> be aware of the refactoring.
Sure. I'm only looking to apply patches 1-13 for now; the rest (bpf,
fscrypt, keyrings, libceph, mac80211, macsec, mac802154, smb, ksmbd,
tipc) are proof of concept, showing how the library APIs can be used in
a wide range of kernel subsystems. I didn't want to spam the entire
series to 20 mailing lists. I'll resend them individually later.
Thanks!
- Eric
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 29/33] bpf: crypto: Use AES-CBC and AES-ECB libraries
2026-07-07 18:20 ` Eric Biggers
@ 2026-07-07 22:50 ` Vadim Fedorenko
2026-07-07 23:16 ` Eric Biggers
0 siblings, 1 reply; 45+ messages in thread
From: Vadim Fedorenko @ 2026-07-07 22:50 UTC (permalink / raw)
To: Eric Biggers; +Cc: linux-crypto, linux-kernel, bpf
On 07/07/2026 19:20, Eric Biggers wrote:
> On Tue, Jul 07, 2026 at 04:01:13PM +0100, Vadim Fedorenko wrote:
>> cc +bpf
>>
>> On 07/07/2026 06:34, Eric Biggers wrote:
>>> BPF crypto was implemented using the lskcipher API, which doesn't seem
>>> to be going anywhere. It supports only "arc4", "cbc(aes)", "ecb(aes)",
>>> and only with unoptimized implementations.
>>>
>>> Library APIs also have been found to be a much better approach, for a
>>> variety of reasons, including reduced overhead, greater flexibility, and
>>> having to be explicit about the crypto algorithms that are supported.
>>>
>>> We can safely ignore the theoretical "arc4" support in BPF crypto as
>>> unused, which leaves "cbc(aes)" and "ecb(aes)". Why these algorithms
>>> were chosen, it's unclear. Regardless, I'll assume that "cbc(aes)" and
>>> "ecb(aes)" need to continue to be supported for backwards compatibility.
>>
>> That was done for single use case of decrypting small blocks in TC
>> layer with "cbc(aes)", with assumption of extending it later.
>
> What protocol is using AES-CBC? And is the kernel encrypting or
> decrypting the data elsewhere, or it is just routing an encrypted packet
> and only the BPF program decrypts it?
That's a "home-made" part of UDP encapsulation, the kernel routes it
further based on the info collected by BPF decrypt program.
> Does this mean the AES-ECB support is unnecessary and can be dropped?
Let's keep it if it doesn't hurt.
>> This change looks great, but it would be great to CC bpf folks just to
>> be aware of the refactoring.
>
> Sure. I'm only looking to apply patches 1-13 for now; the rest (bpf,
> fscrypt, keyrings, libceph, mac80211, macsec, mac802154, smb, ksmbd,
> tipc) are proof of concept, showing how the library APIs can be used in
> a wide range of kernel subsystems. I didn't want to spam the entire
> series to 20 mailing lists. I'll resend them individually later.
Ok, cool. But you can keep my Rb anyways - I checked the code and run
tests.
>
> Thanks!
>
> - Eric
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 29/33] bpf: crypto: Use AES-CBC and AES-ECB libraries
2026-07-07 22:50 ` Vadim Fedorenko
@ 2026-07-07 23:16 ` Eric Biggers
2026-07-08 11:47 ` Vadim Fedorenko
0 siblings, 1 reply; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 23:16 UTC (permalink / raw)
To: Vadim Fedorenko; +Cc: linux-crypto, linux-kernel, bpf
On Tue, Jul 07, 2026 at 11:50:33PM +0100, Vadim Fedorenko wrote:
> > Does this mean the AES-ECB support is unnecessary and can be dropped?
>
> Let's keep it if it doesn't hurt.
It kind of does. It is "bad" crypto that would have to continue to be
maintained, and someone might start using it accidentally.
- Eric
^ permalink raw reply [flat|nested] 45+ messages in thread
* Re: [PATCH 29/33] bpf: crypto: Use AES-CBC and AES-ECB libraries
2026-07-07 23:16 ` Eric Biggers
@ 2026-07-08 11:47 ` Vadim Fedorenko
0 siblings, 0 replies; 45+ messages in thread
From: Vadim Fedorenko @ 2026-07-08 11:47 UTC (permalink / raw)
To: Eric Biggers; +Cc: linux-crypto, linux-kernel, bpf
On 08/07/2026 00:16, Eric Biggers wrote:
> On Tue, Jul 07, 2026 at 11:50:33PM +0100, Vadim Fedorenko wrote:
>>> Does this mean the AES-ECB support is unnecessary and can be dropped?
>>
>> Let's keep it if it doesn't hurt.
>
> It kind of does. It is "bad" crypto that would have to continue to be
> maintained, and someone might start using it accidentally.
AES-ECB was introduced as a cipher to use in QUIC-LB draft,
draft-ietf-quic-load-balancers-21
I know it is expired draft, but it may be worth keeping this cipher
as QUIC WG github is still active
^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH 30/33] bpf: crypto: Add AES-GCM support
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (28 preceding siblings ...)
2026-07-07 5:34 ` [PATCH 29/33] bpf: crypto: Use AES-CBC and AES-ECB libraries Eric Biggers
@ 2026-07-07 5:35 ` Eric Biggers
2026-07-07 15:02 ` Vadim Fedorenko
2026-07-07 5:35 ` [PATCH 31/33] smb: client: Use AES-GCM and AES-CCM libraries Eric Biggers
` (2 subsequent siblings)
32 siblings, 1 reply; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:35 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Add AES-GCM support as requested at
https://lore.kernel.org/r/CAHAB8Wy1APeCcm7_OfrNYeZFcMXfZ5rUSeDX7-c7WO_rGg2Zig@mail.gmail.com/
With the library this is straightforward to do.
The associated data is assumed to be empty. If control over that is
needed, support would need to be added for it as well.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
kernel/bpf/Kconfig | 2 ++
kernel/bpf/crypto.c | 32 ++++++++++++++++++++++++++++++++
2 files changed, 34 insertions(+)
diff --git a/kernel/bpf/Kconfig b/kernel/bpf/Kconfig
index c4f1086b2daf..d14dd1788bd4 100644
--- a/kernel/bpf/Kconfig
+++ b/kernel/bpf/Kconfig
@@ -92,12 +92,14 @@ config BPF_CRYPTO
depends on BPF_SYSCALL
select CRYPTO_LIB_AES_CBC
select CRYPTO_LIB_AES_ECB
+ select CRYPTO_LIB_AES_GCM
help
Provide the kfuncs needed for BPF programs to encrypt and decrypt
data. The supported algorithms are:
- AES-CBC
- AES-ECB
+ - AES-GCM
source "kernel/bpf/preload/Kconfig"
diff --git a/kernel/bpf/crypto.c b/kernel/bpf/crypto.c
index 17e0d2fd422a..934d049ee91c 100644
--- a/kernel/bpf/crypto.c
+++ b/kernel/bpf/crypto.c
@@ -8,6 +8,7 @@
#include <linux/skbuff.h>
#include <crypto/aes-cbc.h>
#include <crypto/aes-ecb.h>
+#include <crypto/aes-gcm.h>
/* BPF crypto initialization parameters struct */
/**
@@ -33,6 +34,7 @@ struct bpf_crypto_params {
enum bpf_crypto_algo_id {
BPF_ALGO_AES_CBC,
BPF_ALGO_AES_ECB,
+ BPF_ALGO_AES_GCM,
};
static const struct {
@@ -42,6 +44,7 @@ static const struct {
} bpf_crypto_algos[] = {
{ "skcipher", "cbc(aes)", BPF_ALGO_AES_CBC },
{ "skcipher", "ecb(aes)", BPF_ALGO_AES_ECB },
+ { "aead", "gcm(aes)", BPF_ALGO_AES_GCM },
};
static bool bpf_crypto_find_algo(const struct bpf_crypto_params *params,
@@ -72,6 +75,7 @@ struct bpf_crypto_ctx {
enum bpf_crypto_algo_id algo;
union {
struct aes_key aes;
+ struct aes_gcm_key aes_gcm;
} key;
struct rcu_head rcu;
refcount_t usage;
@@ -127,6 +131,10 @@ bpf_crypto_ctx_create(const struct bpf_crypto_params *params, u32 params__sz,
*err = aes_preparekey(&ctx->key.aes, params->key,
params->key_len);
break;
+ case BPF_ALGO_AES_GCM:
+ *err = aes_gcm_preparekey(&ctx->key.aes_gcm, params->key,
+ params->key_len, params->authsize);
+ break;
default:
WARN_ON(1);
*err = -ENOENT;
@@ -217,6 +225,27 @@ static int bpf_aes_ecb_crypt(u8 *dst, u32 dst_len, const u8 *src, u32 src_len,
return 0;
}
+static int bpf_aes_gcm_crypt(u8 *dst, u32 dst_len, const u8 *src, u32 src_len,
+ u8 *iv, u32 iv_len,
+ const struct bpf_crypto_ctx *ctx, bool decrypt)
+{
+ const struct aes_gcm_key *key = &ctx->key.aes_gcm;
+ u32 authtag_len = key->authtag_len;
+
+ if (iv_len != GCM_AES_IV_SIZE)
+ return -EINVAL;
+ if (decrypt) {
+ if (src_len < authtag_len || dst_len < src_len - authtag_len)
+ return -EINVAL;
+ return aes_gcm_decrypt(dst, src, src + src_len - authtag_len,
+ src_len - authtag_len, NULL, 0, iv, key);
+ }
+ if (dst_len < authtag_len || dst_len - authtag_len < src_len)
+ return -EINVAL;
+ aes_gcm_encrypt(dst, dst + src_len, src, src_len, NULL, 0, iv, key);
+ return 0;
+}
+
static int bpf_crypto_crypt(const struct bpf_crypto_ctx *ctx,
const struct bpf_dynptr_kern *src,
const struct bpf_dynptr_kern *dst,
@@ -254,6 +283,9 @@ static int bpf_crypto_crypt(const struct bpf_crypto_ctx *ctx,
case BPF_ALGO_AES_ECB:
return bpf_aes_ecb_crypt(pdst, dst_len, psrc, src_len, piv,
iv_len, ctx, decrypt);
+ case BPF_ALGO_AES_GCM:
+ return bpf_aes_gcm_crypt(pdst, dst_len, psrc, src_len, piv,
+ iv_len, ctx, decrypt);
default:
return -EINVAL;
}
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 31/33] smb: client: Use AES-GCM and AES-CCM libraries
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (29 preceding siblings ...)
2026-07-07 5:35 ` [PATCH 30/33] bpf: crypto: Add AES-GCM support Eric Biggers
@ 2026-07-07 5:35 ` Eric Biggers
2026-07-07 5:35 ` [PATCH 32/33] ksmbd: " Eric Biggers
2026-07-07 5:35 ` [PATCH 33/33] net: tipc: Use AES-GCM library instead of crypto_aead Eric Biggers
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:35 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there are library APIs for AES-GCM and AES-CCM, use them
instead of "gcm(aes)" and "ccm(aes)" crypto_aeads. This significantly
simplifies the code, especially since the need to build scatterlists
goes away.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
fs/smb/client/Kconfig | 7 +-
fs/smb/client/cifsencrypt.c | 14 --
fs/smb/client/cifsfs.c | 4 -
fs/smb/client/cifsglob.h | 8 -
fs/smb/client/cifsproto.h | 89 -----------
fs/smb/client/connect.c | 14 +-
fs/smb/client/smb2ops.c | 275 ++++++++++++++++------------------
fs/smb/client/smb2pdu.c | 2 -
fs/smb/client/smb2proto.h | 1 -
fs/smb/client/smb2transport.c | 39 -----
10 files changed, 135 insertions(+), 318 deletions(-)
diff --git a/fs/smb/client/Kconfig b/fs/smb/client/Kconfig
index 2b7db5fb0fd9..5d3e4024b6b5 100644
--- a/fs/smb/client/Kconfig
+++ b/fs/smb/client/Kconfig
@@ -4,12 +4,9 @@ config CIFS
depends on INET
select NLS
select NLS_UCS2_UTILS
- select CRYPTO
- select CRYPTO_AEAD2
- select CRYPTO_CCM
- select CRYPTO_GCM
- select CRYPTO_AES
select CRYPTO_LIB_AES_CBC_MACS
+ select CRYPTO_LIB_AES_CCM
+ select CRYPTO_LIB_AES_GCM
select CRYPTO_LIB_ARC4
select CRYPTO_LIB_MD5
select CRYPTO_LIB_SHA256
diff --git a/fs/smb/client/cifsencrypt.c b/fs/smb/client/cifsencrypt.c
index 34804e9842a8..5db6d26e933d 100644
--- a/fs/smb/client/cifsencrypt.c
+++ b/fs/smb/client/cifsencrypt.c
@@ -21,7 +21,6 @@
#include <linux/highmem.h>
#include <linux/fips.h>
#include <linux/iov_iter.h>
-#include <crypto/aead.h>
#include <crypto/aes-cbc-macs.h>
#include <crypto/arc4.h>
#include <crypto/md5.h>
@@ -499,16 +498,3 @@ calc_seckey(struct cifs_ses *ses)
kfree_sensitive(ctx_arc4);
return 0;
}
-
-void
-cifs_crypto_secmech_release(struct TCP_Server_Info *server)
-{
- if (server->secmech.enc) {
- crypto_free_aead(server->secmech.enc);
- server->secmech.enc = NULL;
- }
- if (server->secmech.dec) {
- crypto_free_aead(server->secmech.dec);
- server->secmech.dec = NULL;
- }
-}
diff --git a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c
index ea4fc0fa68ca..609fd0fb2285 100644
--- a/fs/smb/client/cifsfs.c
+++ b/fs/smb/client/cifsfs.c
@@ -2198,9 +2198,5 @@ MODULE_DESCRIPTION
"also older servers complying with the SNIA CIFS Specification)");
MODULE_VERSION(CIFS_VERSION);
MODULE_SOFTDEP("nls");
-MODULE_SOFTDEP("aes");
-MODULE_SOFTDEP("aead2");
-MODULE_SOFTDEP("ccm");
-MODULE_SOFTDEP("gcm");
module_init(init_cifs)
module_exit(exit_cifs)
diff --git a/fs/smb/client/cifsglob.h b/fs/smb/client/cifsglob.h
index 99f9e6dca62b..c14b55365770 100644
--- a/fs/smb/client/cifsglob.h
+++ b/fs/smb/client/cifsglob.h
@@ -13,7 +13,6 @@
#include <linux/in6.h>
#include <linux/inet.h>
#include <linux/slab.h>
-#include <linux/scatterlist.h>
#include <linux/mm.h>
#include <linux/mempool.h>
#include <linux/workqueue.h>
@@ -220,12 +219,6 @@ struct session_key {
char *response;
};
-/* encryption related structure/fields, not specific to a sec mech */
-struct cifs_secmech {
- struct crypto_aead *enc; /* smb3 encryption AEAD TFM (AES-CCM and AES-GCM) */
- struct crypto_aead *dec; /* smb3 decryption AEAD TFM (AES-CCM and AES-GCM) */
-};
-
/* per smb session structure/fields */
struct ntlmssp_auth {
bool sesskey_per_smbsess; /* whether session key is per smb session */
@@ -745,7 +738,6 @@ struct TCP_Server_Info {
unsigned long lstrp; /* when we got last response from this server */
unsigned long neg_start; /* when negotiate started (jiffies) */
unsigned long reconn_delay; /* when resched session and tcon reconnect */
- struct cifs_secmech secmech; /* crypto sec mech functs, descriptors */
#define CIFS_NEGFLAVOR_UNENCAP 1 /* wct == 17, but no ext_sec */
#define CIFS_NEGFLAVOR_EXTENDED 2 /* wct == 17, ext_sec bit set */
char negflavor; /* NEGOTIATE response flavor */
diff --git a/fs/smb/client/cifsproto.h b/fs/smb/client/cifsproto.h
index c4ababcb51a3..637fe7d503cb 100644
--- a/fs/smb/client/cifsproto.h
+++ b/fs/smb/client/cifsproto.h
@@ -308,7 +308,6 @@ struct cifs_tcon *tcon_info_alloc(bool dir_leases_enabled,
void tconInfoFree(struct cifs_tcon *tcon, enum smb3_tcon_ref_trace trace);
int setup_ntlmv2_rsp(struct cifs_ses *ses, const struct nls_table *nls_cp);
-void cifs_crypto_secmech_release(struct TCP_Server_Info *server);
int calc_seckey(struct cifs_ses *ses);
int generate_smb30signingkey(struct cifs_ses *ses,
struct TCP_Server_Info *server);
@@ -508,94 +507,6 @@ static inline int smb_EIO2(enum smb_eio_trace trace, unsigned long info, unsigne
return -EIO;
}
-static inline int cifs_get_num_sgs(const struct smb_rqst *rqst,
- int num_rqst,
- const u8 *sig)
-{
- unsigned int len, skip;
- unsigned int nents = 0;
- unsigned long addr;
- size_t data_size;
- int i, j;
-
- /*
- * The first rqst has a transform header where the first 20 bytes are
- * not part of the encrypted blob.
- */
- skip = 20;
-
- /* Assumes the first rqst has a transform header as the first iov.
- * I.e.
- * rqst[0].rq_iov[0] is transform header
- * rqst[0].rq_iov[1+] data to be encrypted/decrypted
- * rqst[1+].rq_iov[0+] data to be encrypted/decrypted
- */
- for (i = 0; i < num_rqst; i++) {
- data_size = iov_iter_count(&rqst[i].rq_iter);
-
- /* We really don't want a mixture of pinned and unpinned pages
- * in the sglist. It's hard to keep track of which is what.
- * Instead, we convert to a BVEC-type iterator higher up.
- */
- if (data_size &&
- WARN_ON_ONCE(user_backed_iter(&rqst[i].rq_iter)))
- return smb_EIO(smb_eio_trace_user_iter);
-
- /* We also don't want to have any extra refs or pins to clean
- * up in the sglist.
- */
- if (data_size &&
- WARN_ON_ONCE(iov_iter_extract_will_pin(&rqst[i].rq_iter)))
- return smb_EIO(smb_eio_trace_extract_will_pin);
-
- for (j = 0; j < rqst[i].rq_nvec; j++) {
- struct kvec *iov = &rqst[i].rq_iov[j];
-
- addr = (unsigned long)iov->iov_base + skip;
- if (is_vmalloc_or_module_addr((void *)addr)) {
- len = iov->iov_len - skip;
- nents += DIV_ROUND_UP(offset_in_page(addr) + len,
- PAGE_SIZE);
- } else {
- nents++;
- }
- skip = 0;
- }
- if (data_size)
- nents += iov_iter_npages(&rqst[i].rq_iter, INT_MAX);
- }
- nents += DIV_ROUND_UP(offset_in_page(sig) + SMB2_SIGNATURE_SIZE, PAGE_SIZE);
- return nents;
-}
-
-/* We can not use the normal sg_set_buf() as we will sometimes pass a
- * stack object as buf.
- */
-static inline void cifs_sg_set_buf(struct sg_table *sgtable,
- const void *buf,
- unsigned int buflen)
-{
- unsigned long addr = (unsigned long)buf;
- unsigned int off = offset_in_page(addr);
-
- addr &= PAGE_MASK;
- if (is_vmalloc_or_module_addr((void *)addr)) {
- do {
- unsigned int len = min_t(unsigned int, buflen, PAGE_SIZE - off);
-
- sg_set_page(&sgtable->sgl[sgtable->nents++],
- vmalloc_to_page((void *)addr), len, off);
-
- off = 0;
- addr += PAGE_SIZE;
- buflen -= len;
- } while (buflen);
- } else {
- sg_set_page(&sgtable->sgl[sgtable->nents++],
- virt_to_page((void *)addr), buflen, off);
- }
-}
-
static inline int cifs_get_writable_file(struct cifsInodeInfo *cifs_inode,
unsigned int find_flags,
struct cifsFileInfo **ret_file)
diff --git a/fs/smb/client/connect.c b/fs/smb/client/connect.c
index 85aec302c89e..77454380fe26 100644
--- a/fs/smb/client/connect.c
+++ b/fs/smb/client/connect.c
@@ -1730,8 +1730,6 @@ cifs_put_tcp_session(struct TCP_Server_Info *server, int from_reconnect)
server->tcpStatus = CifsExiting;
spin_unlock(&server->srv_lock);
- cifs_crypto_secmech_release(server);
-
kfree_sensitive(server->session_key.response);
server->session_key.response = NULL;
server->session_key.len = 0;
@@ -1848,7 +1846,7 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx,
#ifndef CONFIG_CIFS_SMB_DIRECT
cifs_dbg(VFS, "CONFIG_CIFS_SMB_DIRECT is not enabled\n");
rc = -ENOENT;
- goto out_err_crypto_release;
+ goto out_err_put_net;
#endif
tcp_ses->smbd_conn = smbd_get_connection(
tcp_ses, (struct sockaddr *)&ctx->dstaddr);
@@ -1858,13 +1856,13 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx,
goto smbd_connected;
} else {
rc = -ENOENT;
- goto out_err_crypto_release;
+ goto out_err_put_net;
}
}
rc = ip_connect(tcp_ses);
if (rc < 0) {
cifs_dbg(VFS, "Error connecting to socket. Aborting operation.\n");
- goto out_err_crypto_release;
+ goto out_err_put_net;
}
smbd_connected:
/*
@@ -1895,7 +1893,7 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx,
rc = PTR_ERR(tcp_ses->tsk);
cifs_dbg(VFS, "error %d create cifsd thread\n", rc);
module_put(THIS_MODULE);
- goto out_err_crypto_release;
+ goto out_err_put_net;
}
/* thread created, put it on the list */
spin_lock(&cifs_tcp_ses_lock);
@@ -1913,9 +1911,7 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx,
return tcp_ses;
-out_err_crypto_release:
- cifs_crypto_secmech_release(tcp_ses);
-
+out_err_put_net:
put_net(cifs_net_ns(tcp_ses));
out_err:
diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c
index 06e9322a762a..7c92410c5461 100644
--- a/fs/smb/client/smb2ops.c
+++ b/fs/smb/client/smb2ops.c
@@ -5,13 +5,14 @@
* Copyright (c) 2012, Jeff Layton <jlayton@redhat.com>
*/
+#include <crypto/aes-ccm.h>
+#include <crypto/aes-gcm.h>
+#include <linux/iov_iter.h>
#include <linux/pagemap.h>
#include <linux/vfs.h>
#include <linux/falloc.h>
-#include <linux/scatterlist.h>
#include <linux/uuid.h>
#include <linux/sort.h>
-#include <crypto/aead.h>
#include <linux/fiemap.h>
#include <linux/folio_queue.h>
#include <uapi/linux/magic.h>
@@ -4351,86 +4352,82 @@ fill_transform_hdr(struct smb2_transform_hdr *tr_hdr, unsigned int orig_len,
memcpy(&tr_hdr->SessionId, &shdr->SessionId, 8);
}
-static void *smb2_aead_req_alloc(struct crypto_aead *tfm, const struct smb_rqst *rqst,
- int num_rqst, const u8 *sig, u8 **iv,
- struct aead_request **req, struct sg_table *sgt,
- unsigned int *num_sgs)
+struct smb3_crypt_ctx {
+ union {
+ struct aes_gcm_ctx gcm;
+ struct aes_ccm_ctx ccm;
+ };
+ __le16 cipher_type;
+ int enc;
+};
+
+static size_t smb3_crypt_step(void *data, size_t progress, size_t len,
+ void *priv, void *priv2)
{
- unsigned int req_size = sizeof(**req) + crypto_aead_reqsize(tfm);
- unsigned int iv_size = crypto_aead_ivsize(tfm);
- unsigned int len;
- int ret;
- u8 *p;
-
- ret = cifs_get_num_sgs(rqst, num_rqst, sig);
- if (ret < 0)
- return ERR_PTR(ret);
- *num_sgs = ret;
-
- len = iv_size;
- len += crypto_aead_alignmask(tfm) & ~(crypto_tfm_ctx_alignment() - 1);
- len = ALIGN(len, crypto_tfm_ctx_alignment());
- len += req_size;
- len = ALIGN(len, __alignof__(struct scatterlist));
- len += array_size(*num_sgs, sizeof(struct scatterlist));
-
- p = kzalloc(len, GFP_NOFS);
- if (!p)
- return ERR_PTR(-ENOMEM);
-
- *iv = (u8 *)PTR_ALIGN(p, crypto_aead_alignmask(tfm) + 1);
- *req = (struct aead_request *)PTR_ALIGN(*iv + iv_size,
- crypto_tfm_ctx_alignment());
- sgt->sgl = (struct scatterlist *)PTR_ALIGN((u8 *)*req + req_size,
- __alignof__(struct scatterlist));
- return p;
+ struct smb3_crypt_ctx *ctx = priv;
+
+ switch (ctx->cipher_type) {
+ case SMB2_ENCRYPTION_AES128_GCM:
+ case SMB2_ENCRYPTION_AES256_GCM:
+ if (ctx->enc)
+ aes_gcm_encrypt_update(&ctx->gcm, data, data, len);
+ else
+ aes_gcm_decrypt_update(&ctx->gcm, data, data, len);
+ return 0;
+ case SMB2_ENCRYPTION_AES128_CCM:
+ case SMB2_ENCRYPTION_AES256_CCM:
+ if (ctx->enc)
+ aes_ccm_encrypt_update(&ctx->ccm, data, data, len);
+ else
+ aes_ccm_decrypt_update(&ctx->ccm, data, data, len);
+ return 0;
+ default:
+ WARN_ON_ONCE(1);
+ return len;
+ }
}
-static void *smb2_get_aead_req(struct crypto_aead *tfm, struct smb_rqst *rqst,
- int num_rqst, const u8 *sig, u8 **iv,
- struct aead_request **req, struct scatterlist **sgl)
+static int smb3_crypt_rqst(struct smb_rqst *rqst, struct smb3_crypt_ctx *ctx,
+ unsigned int start_vec, unsigned int *remaining)
{
- struct sg_table sgtable = {};
- unsigned int skip, num_sgs, i, j;
- ssize_t rc;
- void *p;
+ for (unsigned int i = start_vec; i < rqst->rq_nvec; i++) {
+ size_t len = min(rqst->rq_iov[i].iov_len, *remaining);
- p = smb2_aead_req_alloc(tfm, rqst, num_rqst, sig, iv, req, &sgtable, &num_sgs);
- if (IS_ERR(p))
- return ERR_CAST(p);
+ smb3_crypt_step(rqst->rq_iov[i].iov_base, 0, len, ctx, NULL);
+ *remaining -= len;
+ }
+ if (*remaining > 0 && iov_iter_count(&rqst->rq_iter) > 0) {
+ struct iov_iter tmp_iter = rqst->rq_iter;
+ size_t maxsize = min(iov_iter_count(&tmp_iter), *remaining);
+ size_t did;
- sg_init_marker(sgtable.sgl, num_sgs);
+ did = iterate_and_advance_kernel(&tmp_iter, maxsize, ctx, NULL,
+ smb3_crypt_step);
+ if (did != maxsize)
+ return -EIO;
+ *remaining -= did;
+ }
+ return 0;
+}
+static int smb3_crypt_rqsts(struct smb_rqst *rqst, int num_rqst,
+ unsigned int crypt_len, struct smb3_crypt_ctx *ctx)
+{
/*
- * The first rqst has a transform header where the
- * first 20 bytes are not part of the encrypted blob.
+ * Encrypt or decrypt the payload data.
+ * Skip the transform header rqst[0].rq_iov[0].
*/
- skip = 20;
+ for (int i = 0; i < num_rqst; i++) {
+ int rc = smb3_crypt_rqst(&rqst[i], ctx, i == 0 ? 1 : 0,
+ &crypt_len);
- for (i = 0; i < num_rqst; i++) {
- struct iov_iter *iter = &rqst[i].rq_iter;
- size_t count = iov_iter_count(iter);
-
- for (j = 0; j < rqst[i].rq_nvec; j++) {
- cifs_sg_set_buf(&sgtable,
- rqst[i].rq_iov[j].iov_base + skip,
- rqst[i].rq_iov[j].iov_len - skip);
-
- /* See the above comment on the 'skip' assignment */
- skip = 0;
- }
- sgtable.orig_nents = sgtable.nents;
-
- rc = extract_iter_to_sg(iter, count, &sgtable,
- num_sgs - sgtable.nents, 0);
- iov_iter_revert(iter, rc);
- sgtable.orig_nents = sgtable.nents;
+ if (rc)
+ return rc;
}
-
- cifs_sg_set_buf(&sgtable, sig, SMB2_SIGNATURE_SIZE);
- sg_mark_end(&sgtable.sgl[sgtable.nents - 1]);
- *sgl = sgtable.sgl;
- return p;
+ if (crypt_len != 0)
+ /* Processed less data than expected. */
+ return -EOVERFLOW;
+ return 0;
}
static int
@@ -4468,22 +4465,22 @@ smb2_get_enc_key(struct TCP_Server_Info *server, __u64 ses_id, int enc, u8 *key)
* On success return encrypted data in iov[1-N] and pages, leave iov[0]
* untouched.
*/
-static int
-crypt_message(struct TCP_Server_Info *server, int num_rqst,
- struct smb_rqst *rqst, int enc, struct crypto_aead *tfm)
+static int crypt_message(struct TCP_Server_Info *server, int num_rqst,
+ struct smb_rqst *rqst, int enc)
{
struct smb2_transform_hdr *tr_hdr =
(struct smb2_transform_hdr *)rqst[0].rq_iov[0].iov_base;
+ __le16 cipher_type = server->cipher_type;
+ const u8 *assoc_data = (const u8 *)tr_hdr + 20;
unsigned int assoc_data_len = sizeof(struct smb2_transform_hdr) - 20;
int rc = 0;
- struct scatterlist *sg;
- u8 sign[SMB2_SIGNATURE_SIZE] = {};
u8 key[SMB3_ENC_DEC_KEY_SIZE];
- struct aead_request *req;
- u8 *iv;
- DECLARE_CRYPTO_WAIT(wait);
+ union {
+ struct aes_gcm_key gcm;
+ struct aes_ccm_key ccm;
+ } crypt_key;
+ struct smb3_crypt_ctx ctx;
unsigned int crypt_len = le32_to_cpu(tr_hdr->OriginalMessageSize);
- void *creq;
rc = smb2_get_enc_key(server, le64_to_cpu(tr_hdr->SessionId), enc, key);
if (rc) {
@@ -4492,54 +4489,61 @@ crypt_message(struct TCP_Server_Info *server, int num_rqst,
return rc;
}
- if ((server->cipher_type == SMB2_ENCRYPTION_AES256_CCM) ||
- (server->cipher_type == SMB2_ENCRYPTION_AES256_GCM))
- rc = crypto_aead_setkey(tfm, key, SMB3_GCM256_CRYPTKEY_SIZE);
- else
- rc = crypto_aead_setkey(tfm, key, SMB3_GCM128_CRYPTKEY_SIZE);
-
- if (rc) {
- cifs_server_dbg(VFS, "%s: Failed to set aead key %d\n", __func__, rc);
- return rc;
- }
+ ctx.cipher_type = cipher_type;
+ ctx.enc = enc;
- rc = crypto_aead_setauthsize(tfm, SMB2_SIGNATURE_SIZE);
- if (rc) {
- cifs_server_dbg(VFS, "%s: Failed to set authsize %d\n", __func__, rc);
- return rc;
- }
+ if (cipher_type == SMB2_ENCRYPTION_AES128_GCM ||
+ cipher_type == SMB2_ENCRYPTION_AES256_GCM) {
+ size_t key_size = (cipher_type == SMB2_ENCRYPTION_AES128_GCM) ?
+ AES_KEYSIZE_128 :
+ AES_KEYSIZE_256;
- creq = smb2_get_aead_req(tfm, rqst, num_rqst, sign, &iv, &req, &sg);
- if (IS_ERR(creq))
- return PTR_ERR(creq);
+ rc = aes_gcm_preparekey(&crypt_key.gcm, key, key_size,
+ SMB2_SIGNATURE_SIZE);
+ if (rc)
+ goto out;
- if (!enc) {
- memcpy(sign, &tr_hdr->Signature, SMB2_SIGNATURE_SIZE);
- crypt_len += SMB2_SIGNATURE_SIZE;
- }
+ aes_gcm_init(&ctx.gcm, tr_hdr->Nonce, &crypt_key.gcm);
+ aes_gcm_auth_update(&ctx.gcm, assoc_data, assoc_data_len);
+ rc = smb3_crypt_rqsts(rqst, num_rqst, crypt_len, &ctx);
+ if (rc)
+ goto out;
+ if (enc)
+ aes_gcm_encrypt_final(&ctx.gcm, tr_hdr->Signature);
+ else
+ rc = aes_gcm_decrypt_final(&ctx.gcm, tr_hdr->Signature);
+ } else if (cipher_type == SMB2_ENCRYPTION_AES128_CCM ||
+ cipher_type == SMB2_ENCRYPTION_AES256_CCM) {
+ size_t key_size = (cipher_type == SMB2_ENCRYPTION_AES128_CCM) ?
+ AES_KEYSIZE_128 :
+ AES_KEYSIZE_256;
+
+ rc = aes_ccm_preparekey(&crypt_key.ccm, key, key_size,
+ SMB2_SIGNATURE_SIZE);
+ if (rc)
+ goto out;
- if ((server->cipher_type == SMB2_ENCRYPTION_AES128_GCM) ||
- (server->cipher_type == SMB2_ENCRYPTION_AES256_GCM))
- memcpy(iv, (char *)tr_hdr->Nonce, SMB3_AES_GCM_NONCE);
- else {
- iv[0] = 3;
- memcpy(iv + 1, (char *)tr_hdr->Nonce, SMB3_AES_CCM_NONCE);
+ rc = aes_ccm_init(&ctx.ccm, tr_hdr->Nonce, SMB3_AES_CCM_NONCE,
+ assoc_data_len, crypt_len, &crypt_key.ccm);
+ if (rc)
+ goto out;
+ aes_ccm_auth_update(&ctx.ccm, assoc_data, assoc_data_len);
+ rc = smb3_crypt_rqsts(rqst, num_rqst, crypt_len, &ctx);
+ if (rc)
+ goto out;
+ if (enc)
+ aes_ccm_encrypt_final(&ctx.ccm, tr_hdr->Signature);
+ else
+ rc = aes_ccm_decrypt_final(&ctx.ccm, tr_hdr->Signature);
+ } else {
+ WARN_ON_ONCE(1);
+ rc = -EINVAL;
}
- aead_request_set_tfm(req, tfm);
- aead_request_set_crypt(req, sg, sg, crypt_len, iv);
- aead_request_set_ad(req, assoc_data_len);
-
- aead_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- crypto_req_done, &wait);
-
- rc = crypto_wait_req(enc ? crypto_aead_encrypt(req)
- : crypto_aead_decrypt(req), &wait);
-
- if (!rc && enc)
- memcpy(&tr_hdr->Signature, sign, SMB2_SIGNATURE_SIZE);
-
- kfree_sensitive(creq);
+out:
+ memzero_explicit(&ctx, sizeof(ctx));
+ memzero_explicit(&crypt_key, sizeof(crypt_key));
+ memzero_explicit(key, sizeof(key));
return rc;
}
@@ -4623,7 +4627,7 @@ smb3_init_transform_rq(struct TCP_Server_Info *server, int num_rqst,
/* fill the 1st iov with a transform header */
fill_transform_hdr(tr_hdr, orig_len, old_rq, server->cipher_type);
- rc = crypt_message(server, num_rqst, new_rq, 1, server->secmech.enc);
+ rc = crypt_message(server, num_rqst, new_rq, 1);
cifs_dbg(FYI, "Encrypt message returned %d\n", rc);
if (rc)
goto err_free;
@@ -4648,7 +4652,6 @@ decrypt_raw_data(struct TCP_Server_Info *server, char *buf,
unsigned int buf_data_size, struct iov_iter *iter,
bool is_offloaded)
{
- struct crypto_aead *tfm;
struct smb_rqst rqst = {NULL};
struct kvec iov[2];
size_t iter_size = 0;
@@ -4666,31 +4669,9 @@ decrypt_raw_data(struct TCP_Server_Info *server, char *buf,
iter_size = iov_iter_count(iter);
}
- if (is_offloaded) {
- if ((server->cipher_type == SMB2_ENCRYPTION_AES128_GCM) ||
- (server->cipher_type == SMB2_ENCRYPTION_AES256_GCM))
- tfm = crypto_alloc_aead("gcm(aes)", 0, 0);
- else
- tfm = crypto_alloc_aead("ccm(aes)", 0, 0);
- if (IS_ERR(tfm)) {
- rc = PTR_ERR(tfm);
- cifs_server_dbg(VFS, "%s: Failed alloc decrypt TFM, rc=%d\n", __func__, rc);
-
- return rc;
- }
- } else {
- rc = smb3_crypto_aead_allocate(server);
- if (unlikely(rc))
- return rc;
- tfm = server->secmech.dec;
- }
-
- rc = crypt_message(server, 1, &rqst, 0, tfm);
+ rc = crypt_message(server, 1, &rqst, 0);
cifs_dbg(FYI, "Decrypt message returned %d\n", rc);
- if (is_offloaded)
- crypto_free_aead(tfm);
-
if (rc)
return rc;
diff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c
index 95c0efe9d43b..3eb8bd1e7a6b 100644
--- a/fs/smb/client/smb2pdu.c
+++ b/fs/smb/client/smb2pdu.c
@@ -1343,8 +1343,6 @@ SMB2_negotiate(const unsigned int xid,
cifs_server_dbg(VFS, "Missing expected negotiate contexts\n");
}
- if (server->cipher_type && !rc)
- rc = smb3_crypto_aead_allocate(server);
neg_exit:
free_rsp_buf(resp_buftype, rsp);
return rc;
diff --git a/fs/smb/client/smb2proto.h b/fs/smb/client/smb2proto.h
index 78a4e1c340f9..141d8d1336a5 100644
--- a/fs/smb/client/smb2proto.h
+++ b/fs/smb/client/smb2proto.h
@@ -113,7 +113,6 @@ int smb2_unlock_range(struct cifsFileInfo *cfile, struct file_lock *flock,
const unsigned int xid);
int smb2_push_mandatory_locks(struct cifsFileInfo *cfile);
void smb2_reconnect_server(struct work_struct *work);
-int smb3_crypto_aead_allocate(struct TCP_Server_Info *server);
unsigned long smb_rqst_len(struct TCP_Server_Info *server,
struct smb_rqst *rqst);
void smb2_set_next_command(struct cifs_tcon *tcon, struct smb_rqst *rqst);
diff --git a/fs/smb/client/smb2transport.c b/fs/smb/client/smb2transport.c
index 1143ee52470a..23884e311939 100644
--- a/fs/smb/client/smb2transport.c
+++ b/fs/smb/client/smb2transport.c
@@ -18,7 +18,6 @@
#include <asm/processor.h>
#include <linux/mempool.h>
#include <linux/highmem.h>
-#include <crypto/aead.h>
#include <crypto/aes-cbc-macs.h>
#include <crypto/sha2.h>
#include <crypto/utils.h>
@@ -776,41 +775,3 @@ smb2_setup_async_request(struct TCP_Server_Info *server, struct smb_rqst *rqst)
return mid;
}
-
-int
-smb3_crypto_aead_allocate(struct TCP_Server_Info *server)
-{
- struct crypto_aead *tfm;
-
- if (!server->secmech.enc) {
- if ((server->cipher_type == SMB2_ENCRYPTION_AES128_GCM) ||
- (server->cipher_type == SMB2_ENCRYPTION_AES256_GCM))
- tfm = crypto_alloc_aead("gcm(aes)", 0, 0);
- else
- tfm = crypto_alloc_aead("ccm(aes)", 0, 0);
- if (IS_ERR(tfm)) {
- cifs_server_dbg(VFS, "%s: Failed alloc encrypt aead\n",
- __func__);
- return PTR_ERR(tfm);
- }
- server->secmech.enc = tfm;
- }
-
- if (!server->secmech.dec) {
- if ((server->cipher_type == SMB2_ENCRYPTION_AES128_GCM) ||
- (server->cipher_type == SMB2_ENCRYPTION_AES256_GCM))
- tfm = crypto_alloc_aead("gcm(aes)", 0, 0);
- else
- tfm = crypto_alloc_aead("ccm(aes)", 0, 0);
- if (IS_ERR(tfm)) {
- crypto_free_aead(server->secmech.enc);
- server->secmech.enc = NULL;
- cifs_server_dbg(VFS, "%s: Failed to alloc decrypt aead\n",
- __func__);
- return PTR_ERR(tfm);
- }
- server->secmech.dec = tfm;
- }
-
- return 0;
-}
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* [PATCH 32/33] ksmbd: Use AES-GCM and AES-CCM libraries
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (30 preceding siblings ...)
2026-07-07 5:35 ` [PATCH 31/33] smb: client: Use AES-GCM and AES-CCM libraries Eric Biggers
@ 2026-07-07 5:35 ` Eric Biggers
2026-07-07 10:51 ` Namjae Jeon
2026-07-07 5:35 ` [PATCH 33/33] net: tipc: Use AES-GCM library instead of crypto_aead Eric Biggers
32 siblings, 1 reply; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:35 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there are library APIs for AES-GCM and AES-CCM, use them
instead of "gcm(aes)" and "ccm(aes)" crypto_aeads. This significantly
simplifies the code, especially since the pool of crypto_aead objects
and all the scatterlist building code go away.
Move the encryption and decryption code directly into smb3_decrypt_req()
and smb3_encrypt_resp() to take advantage of their respective data
layouts.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
fs/smb/server/Kconfig | 6 +-
fs/smb/server/Makefile | 2 +-
fs/smb/server/auth.c | 227 -------------------------------------
fs/smb/server/auth.h | 2 -
fs/smb/server/crypto_ctx.c | 176 ----------------------------
fs/smb/server/crypto_ctx.h | 32 ------
fs/smb/server/server.c | 14 +--
fs/smb/server/smb2pdu.c | 133 +++++++++++++++++++---
8 files changed, 124 insertions(+), 468 deletions(-)
delete mode 100644 fs/smb/server/crypto_ctx.c
delete mode 100644 fs/smb/server/crypto_ctx.h
diff --git a/fs/smb/server/Kconfig b/fs/smb/server/Kconfig
index 08d8b7a965a6..6b1a23d01acd 100644
--- a/fs/smb/server/Kconfig
+++ b/fs/smb/server/Kconfig
@@ -6,17 +6,15 @@ config SMB_SERVER
select NLS
select NLS_UTF8
select NLS_UCS2_UTILS
- select CRYPTO
select CRYPTO_LIB_AES_CBC_MACS
+ select CRYPTO_LIB_AES_CCM
+ select CRYPTO_LIB_AES_GCM
select CRYPTO_LIB_ARC4
select CRYPTO_LIB_DES
select CRYPTO_LIB_MD5
select CRYPTO_LIB_SHA256
select CRYPTO_LIB_SHA512
select CRYPTO_LIB_UTILS
- select CRYPTO_AEAD2
- select CRYPTO_CCM
- select CRYPTO_GCM
select ASN1
select OID_REGISTRY
select CRC32
diff --git a/fs/smb/server/Makefile b/fs/smb/server/Makefile
index a3e9306055e8..da1c972b9042 100644
--- a/fs/smb/server/Makefile
+++ b/fs/smb/server/Makefile
@@ -5,7 +5,7 @@
obj-$(CONFIG_SMB_SERVER) += ksmbd.o
ksmbd-y := unicode.o auth.o vfs.o vfs_cache.o server.o ndr.o \
- misc.o oplock.o connection.o ksmbd_work.o crypto_ctx.o \
+ misc.o oplock.o connection.o ksmbd_work.o \
mgmt/ksmbd_ida.o mgmt/user_config.o mgmt/share_config.o \
mgmt/tree_connect.o mgmt/user_session.o smb_common.o \
transport_tcp.o transport_ipc.o smbacl.o smb2pdu.o \
diff --git a/fs/smb/server/auth.c b/fs/smb/server/auth.c
index 86f521e849d5..07aaeb2fcd34 100644
--- a/fs/smb/server/auth.c
+++ b/fs/smb/server/auth.c
@@ -11,27 +11,23 @@
#include <linux/writeback.h>
#include <linux/uio.h>
#include <linux/xattr.h>
-#include <crypto/aead.h>
#include <crypto/aes-cbc-macs.h>
#include <crypto/md5.h>
#include <crypto/sha2.h>
#include <crypto/utils.h>
#include <linux/random.h>
-#include <linux/scatterlist.h>
#include "auth.h"
#include "glob.h"
#include <linux/fips.h>
#include <crypto/arc4.h>
-#include <crypto/des.h>
#include "server.h"
#include "smb_common.h"
#include "connection.h"
#include "mgmt/user_session.h"
#include "mgmt/user_config.h"
-#include "crypto_ctx.h"
#include "transport_ipc.h"
/*
@@ -700,226 +696,3 @@ int ksmbd_gen_preauth_integrity_hash(struct ksmbd_conn *conn, char *buf,
sha512_final(&sha_ctx, pi_hash);
return 0;
}
-
-static int ksmbd_get_encryption_key(struct ksmbd_work *work, __u64 ses_id,
- int enc, u8 *key)
-{
- struct ksmbd_session *sess;
- u8 *ses_enc_key;
-
- if (enc)
- sess = work->sess;
- else
- sess = ksmbd_session_lookup_all(work->conn, ses_id);
- if (!sess)
- return -EINVAL;
-
- ses_enc_key = enc ? sess->smb3encryptionkey :
- sess->smb3decryptionkey;
- memcpy(key, ses_enc_key, SMB3_ENC_DEC_KEY_SIZE);
- if (!enc)
- ksmbd_user_session_put(sess);
-
- return 0;
-}
-
-static inline void smb2_sg_set_buf(struct scatterlist *sg, const void *buf,
- unsigned int buflen)
-{
- void *addr;
-
- if (is_vmalloc_addr(buf))
- addr = vmalloc_to_page(buf);
- else
- addr = virt_to_page(buf);
- sg_set_page(sg, addr, buflen, offset_in_page(buf));
-}
-
-static struct scatterlist *ksmbd_init_sg(struct kvec *iov, unsigned int nvec,
- u8 *sign)
-{
- struct scatterlist *sg;
- unsigned int assoc_data_len = sizeof(struct smb2_transform_hdr) - 20;
- int i, *nr_entries, total_entries = 0, sg_idx = 0;
-
- if (!nvec)
- return NULL;
-
- nr_entries = kzalloc_objs(int, nvec, KSMBD_DEFAULT_GFP);
- if (!nr_entries)
- return NULL;
-
- for (i = 0; i < nvec - 1; i++) {
- unsigned long kaddr = (unsigned long)iov[i + 1].iov_base;
-
- if (is_vmalloc_addr(iov[i + 1].iov_base)) {
- nr_entries[i] = ((kaddr + iov[i + 1].iov_len +
- PAGE_SIZE - 1) >> PAGE_SHIFT) -
- (kaddr >> PAGE_SHIFT);
- } else {
- nr_entries[i]++;
- }
- total_entries += nr_entries[i];
- }
-
- /* Add two entries for transform header and signature */
- total_entries += 2;
-
- sg = kmalloc_objs(struct scatterlist, total_entries, KSMBD_DEFAULT_GFP);
- if (!sg) {
- kfree(nr_entries);
- return NULL;
- }
-
- sg_init_table(sg, total_entries);
- smb2_sg_set_buf(&sg[sg_idx++], iov[0].iov_base + 24, assoc_data_len);
- for (i = 0; i < nvec - 1; i++) {
- void *data = iov[i + 1].iov_base;
- int len = iov[i + 1].iov_len;
-
- if (is_vmalloc_addr(data)) {
- int j, offset = offset_in_page(data);
-
- for (j = 0; j < nr_entries[i]; j++) {
- unsigned int bytes = PAGE_SIZE - offset;
-
- if (!len)
- break;
-
- if (bytes > len)
- bytes = len;
-
- sg_set_page(&sg[sg_idx++],
- vmalloc_to_page(data), bytes,
- offset_in_page(data));
-
- data += bytes;
- len -= bytes;
- offset = 0;
- }
- } else {
- sg_set_page(&sg[sg_idx++], virt_to_page(data), len,
- offset_in_page(data));
- }
- }
- smb2_sg_set_buf(&sg[sg_idx], sign, SMB2_SIGNATURE_SIZE);
- kfree(nr_entries);
- return sg;
-}
-
-int ksmbd_crypt_message(struct ksmbd_work *work, struct kvec *iov,
- unsigned int nvec, int enc)
-{
- struct ksmbd_conn *conn = work->conn;
- struct smb2_transform_hdr *tr_hdr = smb_get_msg(iov[0].iov_base);
- unsigned int assoc_data_len = sizeof(struct smb2_transform_hdr) - 20;
- int rc;
- DECLARE_CRYPTO_WAIT(wait);
- struct scatterlist *sg;
- u8 sign[SMB2_SIGNATURE_SIZE] = {};
- u8 key[SMB3_ENC_DEC_KEY_SIZE];
- struct aead_request *req;
- char *iv;
- unsigned int iv_len;
- struct crypto_aead *tfm;
- unsigned int crypt_len = le32_to_cpu(tr_hdr->OriginalMessageSize);
- struct ksmbd_crypto_ctx *ctx;
-
- rc = ksmbd_get_encryption_key(work,
- le64_to_cpu(tr_hdr->SessionId),
- enc,
- key);
- if (rc) {
- pr_err("Could not get %scryption key\n", enc ? "en" : "de");
- return rc;
- }
-
- if (conn->cipher_type == SMB2_ENCRYPTION_AES128_GCM ||
- conn->cipher_type == SMB2_ENCRYPTION_AES256_GCM)
- ctx = ksmbd_crypto_ctx_find_gcm();
- else
- ctx = ksmbd_crypto_ctx_find_ccm();
- if (!ctx) {
- pr_err("crypto alloc failed\n");
- return -ENOMEM;
- }
-
- if (conn->cipher_type == SMB2_ENCRYPTION_AES128_GCM ||
- conn->cipher_type == SMB2_ENCRYPTION_AES256_GCM)
- tfm = CRYPTO_GCM(ctx);
- else
- tfm = CRYPTO_CCM(ctx);
-
- if (conn->cipher_type == SMB2_ENCRYPTION_AES256_CCM ||
- conn->cipher_type == SMB2_ENCRYPTION_AES256_GCM)
- rc = crypto_aead_setkey(tfm, key, SMB3_GCM256_CRYPTKEY_SIZE);
- else
- rc = crypto_aead_setkey(tfm, key, SMB3_GCM128_CRYPTKEY_SIZE);
- if (rc) {
- pr_err("Failed to set aead key %d\n", rc);
- goto free_ctx;
- }
-
- rc = crypto_aead_setauthsize(tfm, SMB2_SIGNATURE_SIZE);
- if (rc) {
- pr_err("Failed to set authsize %d\n", rc);
- goto free_ctx;
- }
-
- req = aead_request_alloc(tfm, KSMBD_DEFAULT_GFP);
- if (!req) {
- rc = -ENOMEM;
- goto free_ctx;
- }
-
- if (!enc) {
- memcpy(sign, &tr_hdr->Signature, SMB2_SIGNATURE_SIZE);
- crypt_len += SMB2_SIGNATURE_SIZE;
- }
-
- sg = ksmbd_init_sg(iov, nvec, sign);
- if (!sg) {
- pr_err("Failed to init sg\n");
- rc = -ENOMEM;
- goto free_req;
- }
-
- iv_len = crypto_aead_ivsize(tfm);
- iv = kzalloc(iv_len, KSMBD_DEFAULT_GFP);
- if (!iv) {
- rc = -ENOMEM;
- goto free_sg;
- }
-
- if (conn->cipher_type == SMB2_ENCRYPTION_AES128_GCM ||
- conn->cipher_type == SMB2_ENCRYPTION_AES256_GCM) {
- memcpy(iv, (char *)tr_hdr->Nonce, SMB3_AES_GCM_NONCE);
- } else {
- iv[0] = 3;
- memcpy(iv + 1, (char *)tr_hdr->Nonce, SMB3_AES_CCM_NONCE);
- }
-
- aead_request_set_crypt(req, sg, sg, crypt_len, iv);
- aead_request_set_ad(req, assoc_data_len);
- aead_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG |
- CRYPTO_TFM_REQ_MAY_SLEEP,
- crypto_req_done, &wait);
-
- rc = crypto_wait_req(enc ? crypto_aead_encrypt(req) :
- crypto_aead_decrypt(req), &wait);
- if (rc)
- goto free_iv;
-
- if (enc)
- memcpy(&tr_hdr->Signature, sign, SMB2_SIGNATURE_SIZE);
-
-free_iv:
- kfree(iv);
-free_sg:
- kfree(sg);
-free_req:
- aead_request_free(req);
-free_ctx:
- ksmbd_release_crypto_ctx(ctx);
- return rc;
-}
diff --git a/fs/smb/server/auth.h b/fs/smb/server/auth.h
index 5767aabc63c9..f438167e9cc2 100644
--- a/fs/smb/server/auth.h
+++ b/fs/smb/server/auth.h
@@ -36,8 +36,6 @@ struct ksmbd_conn;
struct ksmbd_work;
struct kvec;
-int ksmbd_crypt_message(struct ksmbd_work *work, struct kvec *iov,
- unsigned int nvec, int enc);
void ksmbd_copy_gss_neg_header(void *buf);
int ksmbd_auth_ntlmv2(struct ksmbd_conn *conn, struct ksmbd_session *sess,
struct ntlmv2_resp *ntlmv2, int blen, char *domain_name,
diff --git a/fs/smb/server/crypto_ctx.c b/fs/smb/server/crypto_ctx.c
deleted file mode 100644
index 2fe7d3300480..000000000000
--- a/fs/smb/server/crypto_ctx.c
+++ /dev/null
@@ -1,176 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Copyright (C) 2019 Samsung Electronics Co., Ltd.
- */
-
-#include <linux/kernel.h>
-#include <linux/string.h>
-#include <linux/err.h>
-#include <linux/slab.h>
-#include <linux/wait.h>
-#include <linux/sched.h>
-
-#include "glob.h"
-#include "crypto_ctx.h"
-
-struct crypto_ctx_list {
- spinlock_t ctx_lock;
- int avail_ctx;
- struct list_head idle_ctx;
- wait_queue_head_t ctx_wait;
-};
-
-static struct crypto_ctx_list ctx_list;
-
-static inline void free_aead(struct crypto_aead *aead)
-{
- if (aead)
- crypto_free_aead(aead);
-}
-
-static struct crypto_aead *alloc_aead(int id)
-{
- struct crypto_aead *tfm = NULL;
-
- switch (id) {
- case CRYPTO_AEAD_AES_GCM:
- tfm = crypto_alloc_aead("gcm(aes)", 0, 0);
- break;
- case CRYPTO_AEAD_AES_CCM:
- tfm = crypto_alloc_aead("ccm(aes)", 0, 0);
- break;
- default:
- pr_err("Does not support encrypt ahead(id : %d)\n", id);
- return NULL;
- }
-
- if (IS_ERR(tfm)) {
- pr_err("Failed to alloc encrypt aead : %ld\n", PTR_ERR(tfm));
- return NULL;
- }
-
- return tfm;
-}
-
-static void ctx_free(struct ksmbd_crypto_ctx *ctx)
-{
- int i;
-
- for (i = 0; i < CRYPTO_AEAD_MAX; i++)
- free_aead(ctx->ccmaes[i]);
- kfree(ctx);
-}
-
-static struct ksmbd_crypto_ctx *ksmbd_find_crypto_ctx(void)
-{
- struct ksmbd_crypto_ctx *ctx;
-
- while (1) {
- spin_lock(&ctx_list.ctx_lock);
- if (!list_empty(&ctx_list.idle_ctx)) {
- ctx = list_entry(ctx_list.idle_ctx.next,
- struct ksmbd_crypto_ctx,
- list);
- list_del(&ctx->list);
- spin_unlock(&ctx_list.ctx_lock);
- return ctx;
- }
-
- if (ctx_list.avail_ctx > num_online_cpus()) {
- spin_unlock(&ctx_list.ctx_lock);
- wait_event(ctx_list.ctx_wait,
- !list_empty(&ctx_list.idle_ctx));
- continue;
- }
-
- ctx_list.avail_ctx++;
- spin_unlock(&ctx_list.ctx_lock);
-
- ctx = kzalloc_obj(struct ksmbd_crypto_ctx, KSMBD_DEFAULT_GFP);
- if (!ctx) {
- spin_lock(&ctx_list.ctx_lock);
- ctx_list.avail_ctx--;
- spin_unlock(&ctx_list.ctx_lock);
- wait_event(ctx_list.ctx_wait,
- !list_empty(&ctx_list.idle_ctx));
- continue;
- }
- break;
- }
- return ctx;
-}
-
-void ksmbd_release_crypto_ctx(struct ksmbd_crypto_ctx *ctx)
-{
- if (!ctx)
- return;
-
- spin_lock(&ctx_list.ctx_lock);
- if (ctx_list.avail_ctx <= num_online_cpus()) {
- list_add(&ctx->list, &ctx_list.idle_ctx);
- spin_unlock(&ctx_list.ctx_lock);
- wake_up(&ctx_list.ctx_wait);
- return;
- }
-
- ctx_list.avail_ctx--;
- spin_unlock(&ctx_list.ctx_lock);
- ctx_free(ctx);
-}
-
-static struct ksmbd_crypto_ctx *____crypto_aead_ctx_find(int id)
-{
- struct ksmbd_crypto_ctx *ctx;
-
- if (id >= CRYPTO_AEAD_MAX)
- return NULL;
-
- ctx = ksmbd_find_crypto_ctx();
- if (ctx->ccmaes[id])
- return ctx;
-
- ctx->ccmaes[id] = alloc_aead(id);
- if (ctx->ccmaes[id])
- return ctx;
- ksmbd_release_crypto_ctx(ctx);
- return NULL;
-}
-
-struct ksmbd_crypto_ctx *ksmbd_crypto_ctx_find_gcm(void)
-{
- return ____crypto_aead_ctx_find(CRYPTO_AEAD_AES_GCM);
-}
-
-struct ksmbd_crypto_ctx *ksmbd_crypto_ctx_find_ccm(void)
-{
- return ____crypto_aead_ctx_find(CRYPTO_AEAD_AES_CCM);
-}
-
-void ksmbd_crypto_destroy(void)
-{
- struct ksmbd_crypto_ctx *ctx;
-
- while (!list_empty(&ctx_list.idle_ctx)) {
- ctx = list_entry(ctx_list.idle_ctx.next,
- struct ksmbd_crypto_ctx,
- list);
- list_del(&ctx->list);
- ctx_free(ctx);
- }
-}
-
-int ksmbd_crypto_create(void)
-{
- struct ksmbd_crypto_ctx *ctx;
-
- spin_lock_init(&ctx_list.ctx_lock);
- INIT_LIST_HEAD(&ctx_list.idle_ctx);
- init_waitqueue_head(&ctx_list.ctx_wait);
- ctx_list.avail_ctx = 1;
-
- ctx = kzalloc_obj(struct ksmbd_crypto_ctx, KSMBD_DEFAULT_GFP);
- if (!ctx)
- return -ENOMEM;
- list_add(&ctx->list, &ctx_list.idle_ctx);
- return 0;
-}
diff --git a/fs/smb/server/crypto_ctx.h b/fs/smb/server/crypto_ctx.h
deleted file mode 100644
index b22c6e086f03..000000000000
--- a/fs/smb/server/crypto_ctx.h
+++ /dev/null
@@ -1,32 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * Copyright (C) 2019 Samsung Electronics Co., Ltd.
- */
-
-#ifndef __CRYPTO_CTX_H__
-#define __CRYPTO_CTX_H__
-
-#include <crypto/aead.h>
-
-enum {
- CRYPTO_AEAD_AES_GCM = 16,
- CRYPTO_AEAD_AES_CCM,
- CRYPTO_AEAD_MAX,
-};
-
-struct ksmbd_crypto_ctx {
- struct list_head list;
-
- struct crypto_aead *ccmaes[CRYPTO_AEAD_MAX];
-};
-
-#define CRYPTO_GCM(c) ((c)->ccmaes[CRYPTO_AEAD_AES_GCM])
-#define CRYPTO_CCM(c) ((c)->ccmaes[CRYPTO_AEAD_AES_CCM])
-
-void ksmbd_release_crypto_ctx(struct ksmbd_crypto_ctx *ctx);
-struct ksmbd_crypto_ctx *ksmbd_crypto_ctx_find_gcm(void);
-struct ksmbd_crypto_ctx *ksmbd_crypto_ctx_find_ccm(void);
-void ksmbd_crypto_destroy(void);
-int ksmbd_crypto_create(void);
-
-#endif /* __CRYPTO_CTX_H__ */
diff --git a/fs/smb/server/server.c b/fs/smb/server/server.c
index 36a5ea4828ad..dece42e58d66 100644
--- a/fs/smb/server/server.c
+++ b/fs/smb/server/server.c
@@ -19,7 +19,6 @@
#include "connection.h"
#include "transport_ipc.h"
#include "mgmt/user_session.h"
-#include "crypto_ctx.h"
#include "auth.h"
#include "stats.h"
#include "compress.h"
@@ -564,7 +563,6 @@ static int ksmbd_server_shutdown(void)
ksmbd_workqueue_destroy();
ksmbd_ipc_release();
ksmbd_conn_transport_destroy();
- ksmbd_crypto_destroy();
ksmbd_free_global_file_table();
destroy_lease_table(NULL);
ksmbd_work_pool_destroy();
@@ -612,13 +610,9 @@ static int __init ksmbd_server_init(void)
if (ret)
goto err_destroy_file_table;
- ret = ksmbd_crypto_create();
- if (ret)
- goto err_release_inode_hash;
-
ret = ksmbd_workqueue_init();
if (ret)
- goto err_crypto_destroy;
+ goto err_release_inode_hash;
ret = ksmbd_conn_wq_init();
if (ret)
@@ -628,8 +622,6 @@ static int __init ksmbd_server_init(void)
err_workqueue_destroy:
ksmbd_workqueue_destroy();
-err_crypto_destroy:
- ksmbd_crypto_destroy();
err_release_inode_hash:
ksmbd_release_inode_hash();
err_destroy_file_table:
@@ -666,9 +658,5 @@ MODULE_AUTHOR("Namjae Jeon <linkinjeon@kernel.org>");
MODULE_DESCRIPTION("Linux kernel CIFS/SMB SERVER");
MODULE_LICENSE("GPL");
MODULE_SOFTDEP("pre: nls");
-MODULE_SOFTDEP("pre: aes");
-MODULE_SOFTDEP("pre: aead2");
-MODULE_SOFTDEP("pre: ccm");
-MODULE_SOFTDEP("pre: gcm");
module_init(ksmbd_server_init)
module_exit(ksmbd_server_exit)
diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c
index 097f51fc7ed6..6668d63157c7 100644
--- a/fs/smb/server/smb2pdu.c
+++ b/fs/smb/server/smb2pdu.c
@@ -4,6 +4,8 @@
* Copyright (C) 2018 Samsung Electronics Co., Ltd.
*/
+#include <crypto/aes-ccm.h>
+#include <crypto/aes-gcm.h>
#include <crypto/utils.h>
#include <linux/inetdevice.h>
#include <net/addrconf.h>
@@ -9731,9 +9733,28 @@ static void fill_transform_hdr(void *tr_buf, char *old_buf, __le16 cipher_type)
int smb3_encrypt_resp(struct ksmbd_work *work)
{
+ struct ksmbd_session *sess = work->sess;
+ __le16 cipher_type = work->conn->cipher_type;
struct kvec *iov = work->iov;
+ unsigned int nvec = work->iov_idx + 1;
int rc = -ENOMEM;
void *tr_buf;
+ struct smb2_transform_hdr *tr_hdr;
+ union {
+ struct {
+ struct aes_gcm_key key;
+ struct aes_gcm_ctx ctx;
+ } gcm;
+ struct {
+ struct aes_ccm_key key;
+ struct aes_ccm_ctx ctx;
+ } ccm;
+ } u;
+ u8 *assoc_data;
+ size_t orig_size, assoc_data_size;
+
+ if (!sess)
+ return -EINVAL;
tr_buf = kzalloc(sizeof(struct smb2_transform_hdr) + 4, KSMBD_DEFAULT_GFP);
if (!tr_buf)
@@ -9742,11 +9763,60 @@ int smb3_encrypt_resp(struct ksmbd_work *work)
/* fill transform header */
fill_transform_hdr(tr_buf, work->response_buf, work->conn->cipher_type);
+ work->tr_buf = tr_buf;
+
+ tr_hdr = smb_get_msg(tr_buf);
+ orig_size = le32_to_cpu(tr_hdr->OriginalMessageSize);
+ assoc_data = tr_buf + 24;
+ assoc_data_size = sizeof(struct smb2_transform_hdr) - 20;
+
iov[0].iov_base = tr_buf;
iov[0].iov_len = sizeof(struct smb2_transform_hdr) + 4;
- work->tr_buf = tr_buf;
- return ksmbd_crypt_message(work, iov, work->iov_idx + 1, 1);
+ if (cipher_type == SMB2_ENCRYPTION_AES128_GCM ||
+ cipher_type == SMB2_ENCRYPTION_AES256_GCM) {
+ size_t key_size = (cipher_type == SMB2_ENCRYPTION_AES128_GCM) ?
+ AES_KEYSIZE_128 :
+ AES_KEYSIZE_256;
+
+ rc = aes_gcm_preparekey(&u.gcm.key, sess->smb3encryptionkey,
+ key_size, SMB2_SIGNATURE_SIZE);
+ if (rc)
+ goto out;
+ aes_gcm_init(&u.gcm.ctx, tr_hdr->Nonce, &u.gcm.key);
+ aes_gcm_auth_update(&u.gcm.ctx, assoc_data, assoc_data_size);
+ for (unsigned int i = 1; i < nvec; i++)
+ aes_gcm_encrypt_update(&u.gcm.ctx, iov[i].iov_base,
+ iov[i].iov_base, iov[i].iov_len);
+ aes_gcm_encrypt_final(&u.gcm.ctx, tr_hdr->Signature);
+ rc = 0;
+ } else if (cipher_type == SMB2_ENCRYPTION_AES128_CCM ||
+ cipher_type == SMB2_ENCRYPTION_AES256_CCM) {
+ size_t key_size = (cipher_type == SMB2_ENCRYPTION_AES128_CCM) ?
+ AES_KEYSIZE_128 :
+ AES_KEYSIZE_256;
+
+ rc = aes_ccm_preparekey(&u.ccm.key, sess->smb3encryptionkey,
+ key_size, SMB2_SIGNATURE_SIZE);
+ if (rc)
+ goto out;
+ rc = aes_ccm_init(&u.ccm.ctx, tr_hdr->Nonce, SMB3_AES_CCM_NONCE,
+ assoc_data_size, orig_size, &u.ccm.key);
+ if (rc)
+ goto out;
+ aes_ccm_auth_update(&u.ccm.ctx, assoc_data, assoc_data_size);
+ for (unsigned int i = 1; i < nvec; i++)
+ aes_ccm_encrypt_update(&u.ccm.ctx, iov[i].iov_base,
+ iov[i].iov_base, iov[i].iov_len);
+ aes_ccm_encrypt_final(&u.ccm.ctx, tr_hdr->Signature);
+ rc = 0;
+ } else {
+ WARN_ON_ONCE(1);
+ rc = -EOPNOTSUPP;
+ }
+out:
+ memzero_explicit(&u, sizeof(u));
+ return rc;
}
bool smb3_is_transform_hdr(void *buf)
@@ -9759,11 +9829,17 @@ bool smb3_is_transform_hdr(void *buf)
int smb3_decrypt_req(struct ksmbd_work *work)
{
struct ksmbd_session *sess;
+ __le16 cipher_type = work->conn->cipher_type;
char *buf = work->request_buf;
unsigned int pdu_length = get_rfc1002_len(buf);
- struct kvec iov[2];
int buf_data_size = pdu_length - sizeof(struct smb2_transform_hdr);
struct smb2_transform_hdr *tr_hdr = smb_get_msg(buf);
+ union {
+ struct aes_gcm_key gcm;
+ struct aes_ccm_key ccm;
+ } key;
+ u8 *data, *assoc_data;
+ size_t orig_size, assoc_data_size;
int rc = 0;
if (pdu_length < sizeof(struct smb2_transform_hdr) ||
@@ -9773,10 +9849,14 @@ int smb3_decrypt_req(struct ksmbd_work *work)
return -ECONNABORTED;
}
- if (buf_data_size < le32_to_cpu(tr_hdr->OriginalMessageSize)) {
+ orig_size = le32_to_cpu(tr_hdr->OriginalMessageSize);
+ if (buf_data_size < orig_size) {
pr_err("Transform message is broken\n");
return -ECONNABORTED;
}
+ data = buf + sizeof(struct smb2_transform_hdr) + 4;
+ assoc_data = buf + 24;
+ assoc_data_size = sizeof(struct smb2_transform_hdr) - 20;
sess = ksmbd_session_lookup_all(work->conn, le64_to_cpu(tr_hdr->SessionId));
if (!sess) {
@@ -9784,19 +9864,46 @@ int smb3_decrypt_req(struct ksmbd_work *work)
le64_to_cpu(tr_hdr->SessionId));
return -ECONNABORTED;
}
- ksmbd_user_session_put(sess);
- iov[0].iov_base = buf;
- iov[0].iov_len = sizeof(struct smb2_transform_hdr) + 4;
- iov[1].iov_base = buf + sizeof(struct smb2_transform_hdr) + 4;
- iov[1].iov_len = buf_data_size;
- rc = ksmbd_crypt_message(work, iov, 2, 0);
+ if (cipher_type == SMB2_ENCRYPTION_AES128_GCM ||
+ cipher_type == SMB2_ENCRYPTION_AES256_GCM) {
+ size_t key_size = (cipher_type == SMB2_ENCRYPTION_AES128_GCM) ?
+ AES_KEYSIZE_128 :
+ AES_KEYSIZE_256;
+
+ rc = aes_gcm_preparekey(&key.gcm, sess->smb3decryptionkey,
+ key_size, SMB2_SIGNATURE_SIZE);
+ if (rc)
+ goto put_session;
+ rc = aes_gcm_decrypt(data, data, tr_hdr->Signature, orig_size,
+ assoc_data, assoc_data_size, tr_hdr->Nonce,
+ &key.gcm);
+ } else if (cipher_type == SMB2_ENCRYPTION_AES128_CCM ||
+ cipher_type == SMB2_ENCRYPTION_AES256_CCM) {
+ size_t key_size = (cipher_type == SMB2_ENCRYPTION_AES128_CCM) ?
+ AES_KEYSIZE_128 :
+ AES_KEYSIZE_256;
+
+ rc = aes_ccm_preparekey(&key.ccm, sess->smb3decryptionkey,
+ key_size, SMB2_SIGNATURE_SIZE);
+ if (rc)
+ goto put_session;
+ rc = aes_ccm_decrypt(data, data, tr_hdr->Signature, orig_size,
+ assoc_data, assoc_data_size, tr_hdr->Nonce,
+ SMB3_AES_CCM_NONCE, &key.ccm);
+ } else {
+ WARN_ON_ONCE(1);
+ rc = -EOPNOTSUPP;
+ }
+put_session:
+ ksmbd_user_session_put(sess);
if (rc)
- return rc;
+ goto out;
- memmove(buf + 4, iov[1].iov_base, buf_data_size);
+ memmove(buf + 4, data, buf_data_size);
*(__be32 *)buf = cpu_to_be32(buf_data_size);
-
+out:
+ memzero_explicit(&key, sizeof(key));
return rc;
}
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread* Re: [PATCH 32/33] ksmbd: Use AES-GCM and AES-CCM libraries
2026-07-07 5:35 ` [PATCH 32/33] ksmbd: " Eric Biggers
@ 2026-07-07 10:51 ` Namjae Jeon
0 siblings, 0 replies; 45+ messages in thread
From: Namjae Jeon @ 2026-07-07 10:51 UTC (permalink / raw)
To: Eric Biggers; +Cc: linux-crypto, linux-kernel
On Tue, Jul 7, 2026 at 2:44 PM Eric Biggers <ebiggers@kernel.org> wrote:
>
> Now that there are library APIs for AES-GCM and AES-CCM, use them
> instead of "gcm(aes)" and "ccm(aes)" crypto_aeads. This significantly
> simplifies the code, especially since the pool of crypto_aead objects
> and all the scatterlist building code go away.
>
> Move the encryption and decryption code directly into smb3_decrypt_req()
> and smb3_encrypt_resp() to take advantage of their respective data
> layouts.
>
> Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Acked-by: Namjae Jeon <linkinjeon@kernel.org>
Thanks!
^ permalink raw reply [flat|nested] 45+ messages in thread
* [PATCH 33/33] net: tipc: Use AES-GCM library instead of crypto_aead
2026-07-07 5:34 [PATCH 00/33] Library APIs for AES encryption modes Eric Biggers
` (31 preceding siblings ...)
2026-07-07 5:35 ` [PATCH 32/33] ksmbd: " Eric Biggers
@ 2026-07-07 5:35 ` Eric Biggers
32 siblings, 0 replies; 45+ messages in thread
From: Eric Biggers @ 2026-07-07 5:35 UTC (permalink / raw)
To: linux-crypto; +Cc: linux-kernel, Eric Biggers
Now that there's a library API for AES-GCM, use it instead of a
"gcm(aes)" crypto_aead. This significantly simplifies the code.
Notably, per-skb dynamic memory allocations and conversions to
scatterlists are eliminated. For encryption the skb is linear, so just
call the one-shot function aes_gcm_encrypt(). For decryption, just
iterate through the skb's data and decrypt it directly.
There was also an unnecessary copy of the key being made (in a
crypto_aead object) for every CPU. That is removed as well. The
library en/decryption functions treat struct aes_gcm_key as read-only,
so it can be used by an unlimited number of threads with no contention.
Asynchronous en/decryption operations are no longer supported, as that
execution model for symmetric crypto has been found to be harmful. It
generated endless bugs and didn't actually improve performance.
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
net/tipc/Kconfig | 4 +-
net/tipc/crypto.c | 467 ++++++++--------------------------------------
net/tipc/msg.h | 3 -
net/tipc/sysctl.c | 6 +
4 files changed, 85 insertions(+), 395 deletions(-)
diff --git a/net/tipc/Kconfig b/net/tipc/Kconfig
index 18f62135e47b..39fa54b04bb9 100644
--- a/net/tipc/Kconfig
+++ b/net/tipc/Kconfig
@@ -38,9 +38,7 @@ config TIPC_MEDIA_UDP
config TIPC_CRYPTO
bool "TIPC encryption support"
depends on TIPC
- select CRYPTO
- select CRYPTO_AES
- select CRYPTO_GCM
+ select CRYPTO_LIB_AES_GCM
default y
help
Saying Y here will enable support for TIPC encryption.
diff --git a/net/tipc/crypto.c b/net/tipc/crypto.c
index 16f1ed1f6b1b..e0040309a788 100644
--- a/net/tipc/crypto.c
+++ b/net/tipc/crypto.c
@@ -34,8 +34,7 @@
* POSSIBILITY OF SUCH DAMAGE.
*/
-#include <crypto/aead.h>
-#include <crypto/aes.h>
+#include <crypto/aes-gcm.h>
#include <crypto/rng.h>
#include "crypto.h"
#include "msg.h"
@@ -69,9 +68,9 @@ enum {
enum {
STAT_OK,
STAT_NOK,
- STAT_ASYNC,
- STAT_ASYNC_OK,
- STAT_ASYNC_NOK,
+ STAT_ASYNC, /* no longer used */
+ STAT_ASYNC_OK, /* no longer used */
+ STAT_ASYNC_NOK, /* no longer used */
STAT_BADKEYS, /* tx only */
STAT_BADMSGS = STAT_BADKEYS, /* rx only */
STAT_NOKEYS,
@@ -121,19 +120,9 @@ struct tipc_key {
};
};
-/**
- * struct tipc_tfm - TIPC TFM structure to form a list of TFMs
- * @tfm: cipher handle/key
- * @list: linked list of TFMs
- */
-struct tipc_tfm {
- struct crypto_aead *tfm;
- struct list_head list;
-};
-
/**
* struct tipc_aead - TIPC AEAD key structure
- * @tfm_entry: per-cpu pointer to one entry in TFM list
+ * @gcm_key: the AES-GCM key
* @crypto: TIPC crypto owns this key
* @cloned: reference to the source key in case cloning
* @users: the number of the key users (TX/RX)
@@ -149,7 +138,7 @@ struct tipc_tfm {
*/
struct tipc_aead {
#define TIPC_AEAD_HINT_LEN (5)
- struct tipc_tfm * __percpu *tfm_entry;
+ struct aes_gcm_key gcm_key;
struct tipc_crypto *crypto;
struct tipc_aead *cloned;
atomic_t users;
@@ -235,19 +224,6 @@ struct tipc_crypto {
} ____cacheline_aligned;
-/* struct tipc_crypto_tx_ctx - TX context for callbacks */
-struct tipc_crypto_tx_ctx {
- struct tipc_aead *aead;
- struct tipc_bearer *bearer;
- struct tipc_media_addr dst;
-};
-
-/* struct tipc_crypto_rx_ctx - RX context for callbacks */
-struct tipc_crypto_rx_ctx {
- struct tipc_aead *aead;
- struct tipc_bearer *bearer;
-};
-
static struct tipc_aead *tipc_aead_get(struct tipc_aead __rcu *aead);
static inline void tipc_aead_put(struct tipc_aead *aead);
static void tipc_aead_free(struct rcu_head *rp);
@@ -255,22 +231,15 @@ static int tipc_aead_users(struct tipc_aead __rcu *aead);
static void tipc_aead_users_inc(struct tipc_aead __rcu *aead, int lim);
static void tipc_aead_users_dec(struct tipc_aead __rcu *aead, int lim);
static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val);
-static struct crypto_aead *tipc_aead_tfm_next(struct tipc_aead *aead);
static int tipc_aead_init(struct tipc_aead **aead, struct tipc_aead_key *ukey,
u8 mode);
static int tipc_aead_clone(struct tipc_aead **dst, struct tipc_aead *src);
-static void *tipc_aead_mem_alloc(struct crypto_aead *tfm,
- unsigned int crypto_ctx_size,
- u8 **iv, struct aead_request **req,
- struct scatterlist **sg, int nsg);
static int tipc_aead_encrypt(struct tipc_aead *aead, struct sk_buff *skb,
struct tipc_bearer *b,
struct tipc_media_addr *dst,
struct tipc_node *__dnode);
-static void tipc_aead_encrypt_done(void *data, int err);
static int tipc_aead_decrypt(struct net *net, struct tipc_aead *aead,
struct sk_buff *skb, struct tipc_bearer *b);
-static void tipc_aead_decrypt_done(void *data, int err);
static inline int tipc_ehdr_size(struct tipc_ehdr *ehdr);
static int tipc_ehdr_build(struct net *net, struct tipc_aead *aead,
u8 tx_key, struct sk_buff *skb,
@@ -335,12 +304,6 @@ int tipc_aead_key_validate(struct tipc_aead_key *ukey, struct genl_info *info)
{
int keylen;
- /* Check if algorithm exists */
- if (unlikely(!crypto_has_alg(ukey->alg_name, 0, 0))) {
- GENL_SET_ERR_MSG(info, "unable to load the algorithm (module existed?)");
- return -ENODEV;
- }
-
/* Currently, we only support the "gcm(aes)" cipher algorithm */
if (strcmp(ukey->alg_name, "gcm(aes)")) {
GENL_SET_ERR_MSG(info, "not supported yet the algorithm");
@@ -391,30 +354,15 @@ static inline void tipc_aead_put(struct tipc_aead *aead)
}
/**
- * tipc_aead_free - Release AEAD key incl. all the TFMs in the list
+ * tipc_aead_free - Release AEAD key
* @rp: rcu head pointer
*/
static void tipc_aead_free(struct rcu_head *rp)
{
struct tipc_aead *aead = container_of(rp, struct tipc_aead, rcu);
- struct tipc_tfm *tfm_entry, *head, *tmp;
- if (aead->cloned) {
+ if (aead->cloned)
tipc_aead_put(aead->cloned);
- } else {
- head = *get_cpu_ptr(aead->tfm_entry);
- put_cpu_ptr(aead->tfm_entry);
- list_for_each_entry_safe(tfm_entry, tmp, &head->list, list) {
- crypto_free_aead(tfm_entry->tfm);
- list_del(&tfm_entry->list);
- kfree(tfm_entry);
- }
- /* Free the head */
- crypto_free_aead(head->tfm);
- list_del(&head->list);
- kfree(head);
- }
- free_percpu(aead->tfm_entry);
kfree_sensitive(aead->key);
kfree_sensitive(aead);
}
@@ -472,44 +420,21 @@ static void tipc_aead_users_set(struct tipc_aead __rcu *aead, int val)
rcu_read_unlock();
}
-/**
- * tipc_aead_tfm_next - Move TFM entry to the next one in list and return it
- * @aead: the AEAD key pointer
- */
-static struct crypto_aead *tipc_aead_tfm_next(struct tipc_aead *aead)
-{
- struct tipc_tfm **tfm_entry;
- struct crypto_aead *tfm;
-
- tfm_entry = get_cpu_ptr(aead->tfm_entry);
- *tfm_entry = list_next_entry(*tfm_entry, list);
- tfm = (*tfm_entry)->tfm;
- put_cpu_ptr(tfm_entry);
-
- return tfm;
-}
-
/**
* tipc_aead_init - Initiate TIPC AEAD
* @aead: returned new TIPC AEAD key handle pointer
* @ukey: pointer to user key data
* @mode: the key mode
*
- * Allocate a (list of) new cipher transformation (TFM) with the specific user
- * key data if valid. The number of the allocated TFMs can be set via the sysfs
- * "net/tipc/max_tfms" first.
- * Also, all the other AEAD data are also initialized.
+ * Allocate a new AEAD key container and prepare the AES-GCM key.
*
* Return: 0 if the initiation is successful, otherwise: < 0
*/
static int tipc_aead_init(struct tipc_aead **aead, struct tipc_aead_key *ukey,
u8 mode)
{
- struct tipc_tfm *tfm_entry, *head;
- struct crypto_aead *tfm;
struct tipc_aead *tmp;
- int keylen, err, cpu;
- int tfm_cnt = 0;
+ int keylen, err;
if (unlikely(*aead))
return -EEXIST;
@@ -522,59 +447,9 @@ static int tipc_aead_init(struct tipc_aead **aead, struct tipc_aead_key *ukey,
/* The key consists of two parts: [AES-KEY][SALT] */
keylen = ukey->keylen - TIPC_AES_GCM_SALT_SIZE;
- /* Allocate per-cpu TFM entry pointer */
- tmp->tfm_entry = alloc_percpu(struct tipc_tfm *);
- if (!tmp->tfm_entry) {
- kfree_sensitive(tmp);
- return -ENOMEM;
- }
-
- /* Make a list of TFMs with the user key data */
- do {
- tfm = crypto_alloc_aead(ukey->alg_name, 0, 0);
- if (IS_ERR(tfm)) {
- err = PTR_ERR(tfm);
- break;
- }
-
- if (unlikely(!tfm_cnt &&
- crypto_aead_ivsize(tfm) != TIPC_AES_GCM_IV_SIZE)) {
- crypto_free_aead(tfm);
- err = -ENOTSUPP;
- break;
- }
-
- err = crypto_aead_setauthsize(tfm, TIPC_AES_GCM_TAG_SIZE);
- err |= crypto_aead_setkey(tfm, ukey->key, keylen);
- if (unlikely(err)) {
- crypto_free_aead(tfm);
- break;
- }
-
- tfm_entry = kmalloc_obj(*tfm_entry);
- if (unlikely(!tfm_entry)) {
- crypto_free_aead(tfm);
- err = -ENOMEM;
- break;
- }
- INIT_LIST_HEAD(&tfm_entry->list);
- tfm_entry->tfm = tfm;
-
- /* First entry? */
- if (!tfm_cnt) {
- head = tfm_entry;
- for_each_possible_cpu(cpu) {
- *per_cpu_ptr(tmp->tfm_entry, cpu) = head;
- }
- } else {
- list_add_tail(&tfm_entry->list, &head->list);
- }
-
- } while (++tfm_cnt < sysctl_tipc_max_tfms);
-
- /* Not any TFM is allocated? */
- if (!tfm_cnt) {
- free_percpu(tmp->tfm_entry);
+ err = aes_gcm_preparekey(&tmp->gcm_key, ukey->key, keylen,
+ TIPC_AES_GCM_TAG_SIZE);
+ if (unlikely(err)) {
kfree_sensitive(tmp);
return err;
}
@@ -606,10 +481,8 @@ static int tipc_aead_init(struct tipc_aead **aead, struct tipc_aead_key *ukey,
* @dst: dest key for the cloning
* @src: source key to clone from
*
- * Make a "copy" of the source AEAD key data to the dest, the TFMs list is
- * common for the keys.
- * A reference to the source is hold in the "cloned" pointer for the later
- * freeing purposes.
+ * Make a "copy" of the source AEAD key data to the dest. A reference to the
+ * source is held in the "cloned" pointer for later freeing purposes.
*
* Note: this must be done in cluster-key mode only!
* Return: 0 in case of success, otherwise < 0
@@ -617,7 +490,6 @@ static int tipc_aead_init(struct tipc_aead **aead, struct tipc_aead_key *ukey,
static int tipc_aead_clone(struct tipc_aead **dst, struct tipc_aead *src)
{
struct tipc_aead *aead;
- int cpu;
if (!src)
return -ENOKEY;
@@ -632,16 +504,7 @@ static int tipc_aead_clone(struct tipc_aead **dst, struct tipc_aead *src)
if (unlikely(!aead))
return -ENOMEM;
- aead->tfm_entry = alloc_percpu_gfp(struct tipc_tfm *, GFP_ATOMIC);
- if (unlikely(!aead->tfm_entry)) {
- kfree_sensitive(aead);
- return -ENOMEM;
- }
-
- for_each_possible_cpu(cpu) {
- *per_cpu_ptr(aead->tfm_entry, cpu) =
- *per_cpu_ptr(src->tfm_entry, cpu);
- }
+ aead->gcm_key = src->gcm_key;
memcpy(aead->hint, src->hint, sizeof(src->hint));
aead->mode = src->mode;
@@ -658,53 +521,54 @@ static int tipc_aead_clone(struct tipc_aead **dst, struct tipc_aead *src)
return 0;
}
+static void tipc_decrypt_chunk(struct aes_gcm_ctx *ctx, u8 *data, size_t avail,
+ size_t *assoc_len, size_t *crypt_len)
+{
+ size_t n;
+
+ if (*assoc_len && avail) {
+ /* Associated data */
+ n = min(avail, *assoc_len);
+ aes_gcm_auth_update(ctx, data, n);
+ data += n;
+ avail -= n;
+ *assoc_len -= n;
+ }
+
+ if (*crypt_len && avail) {
+ /* En/decrypted data */
+ n = min(avail, *crypt_len);
+ aes_gcm_decrypt_update(ctx, data, data, n);
+ *crypt_len -= n;
+ }
+}
+
/**
- * tipc_aead_mem_alloc - Allocate memory for AEAD request operations
- * @tfm: cipher handle to be registered with the request
- * @crypto_ctx_size: size of crypto context for callback
- * @iv: returned pointer to IV data
- * @req: returned pointer to AEAD request data
- * @sg: returned pointer to SG lists
- * @nsg: number of SG lists to be allocated
+ * tipc_decrypt_skb() - Decrypt an skb in-place using AES-GCM
+ * @ctx: An AES-GCM context
+ * @skb: The socket buffer to process
+ * @assoc_len: Length of associated data
+ * @crypt_len: Length of payload data to encrypt or decrypt
*
- * Allocate memory to store the crypto context data, AEAD request, IV and SG
- * lists, the memory layout is as follows:
- * crypto_ctx || iv || aead_req || sg[]
+ * Updates the given AES-GCM context with @assoc_len bytes of associated data
+ * from the skb, then decrypts @crypt_len bytes in-place. Context
+ * initialization and finalization are handled by the caller.
*
- * Return: the pointer to the memory areas in case of success, otherwise NULL
+ * The data (both associated and en/decrypted) is taken from the linear head and
+ * frag_list. It is assumed that the @skb was processed by skb_cow_data(),
+ * meaning it has no page fragments (nr_frags == 0) and is writable.
*/
-static void *tipc_aead_mem_alloc(struct crypto_aead *tfm,
- unsigned int crypto_ctx_size,
- u8 **iv, struct aead_request **req,
- struct scatterlist **sg, int nsg)
+static void tipc_decrypt_skb(struct aes_gcm_ctx *ctx, struct sk_buff *skb,
+ size_t assoc_len, size_t crypt_len)
{
- unsigned int iv_size, req_size;
- unsigned int len;
- u8 *mem;
-
- iv_size = crypto_aead_ivsize(tfm);
- req_size = sizeof(**req) + crypto_aead_reqsize(tfm);
-
- len = crypto_ctx_size;
- len += iv_size;
- len += crypto_aead_alignmask(tfm) & ~(crypto_tfm_ctx_alignment() - 1);
- len = ALIGN(len, crypto_tfm_ctx_alignment());
- len += req_size;
- len = ALIGN(len, __alignof__(struct scatterlist));
- len += nsg * sizeof(**sg);
-
- mem = kmalloc(len, GFP_ATOMIC);
- if (!mem)
- return NULL;
-
- *iv = (u8 *)PTR_ALIGN(mem + crypto_ctx_size,
- crypto_aead_alignmask(tfm) + 1);
- *req = (struct aead_request *)PTR_ALIGN(*iv + iv_size,
- crypto_tfm_ctx_alignment());
- *sg = (struct scatterlist *)PTR_ALIGN((u8 *)*req + req_size,
- __alignof__(struct scatterlist));
-
- return (void *)mem;
+ struct sk_buff *frag_iter;
+
+ WARN_ON_ONCE(skb_shinfo(skb)->nr_frags);
+ tipc_decrypt_chunk(ctx, skb->data, skb_headlen(skb), &assoc_len,
+ &crypt_len);
+ skb_walk_frags(skb, frag_iter)
+ tipc_decrypt_chunk(ctx, frag_iter->data, frag_iter->len,
+ &assoc_len, &crypt_len);
}
/**
@@ -717,7 +581,6 @@ static void *tipc_aead_mem_alloc(struct crypto_aead *tfm,
*
* Return:
* * 0 : if the encryption has completed
- * * -EINPROGRESS/-EBUSY : if a callback will be performed
* * < 0 : the encryption has failed
*/
static int tipc_aead_encrypt(struct tipc_aead *aead, struct sk_buff *skb,
@@ -725,16 +588,11 @@ static int tipc_aead_encrypt(struct tipc_aead *aead, struct sk_buff *skb,
struct tipc_media_addr *dst,
struct tipc_node *__dnode)
{
- struct crypto_aead *tfm = tipc_aead_tfm_next(aead);
- struct tipc_crypto_tx_ctx *tx_ctx;
- struct aead_request *req;
struct sk_buff *trailer;
- struct scatterlist *sg;
struct tipc_ehdr *ehdr;
- int ehsz, len, tailen, nsg, rc;
- void *ctx;
+ int ehsz, len, tailen, nsg;
u32 salt;
- u8 *iv;
+ u8 iv[TIPC_AES_GCM_IV_SIZE];
/* Make sure message len at least 4-byte aligned */
len = ALIGN(skb->len, 4);
@@ -760,20 +618,6 @@ static int tipc_aead_encrypt(struct tipc_aead *aead, struct sk_buff *skb,
pskb_put(skb, trailer, tailen);
- /* Allocate memory for the AEAD operation */
- ctx = tipc_aead_mem_alloc(tfm, sizeof(*tx_ctx), &iv, &req, &sg, nsg);
- if (unlikely(!ctx))
- return -ENOMEM;
- TIPC_SKB_CB(skb)->crypto_ctx = ctx;
-
- /* Map skb to the sg lists */
- sg_init_table(sg, nsg);
- rc = skb_to_sgvec(skb, sg, 0, skb->len);
- if (unlikely(rc < 0)) {
- pr_err("TX: skb_to_sgvec() returned %d, nsg %d!\n", rc, nsg);
- goto exit;
- }
-
/* Prepare IV: [SALT (4 octets)][SEQNO (8 octets)]
* In case we're in cluster-key mode, SALT is varied by xor-ing with
* the source address (or w0 of id), otherwise with the dest address
@@ -788,78 +632,12 @@ static int tipc_aead_encrypt(struct tipc_aead *aead, struct sk_buff *skb,
memcpy(iv, &salt, 4);
memcpy(iv + 4, (u8 *)&ehdr->seqno, 8);
- /* Prepare request */
ehsz = tipc_ehdr_size(ehdr);
- aead_request_set_tfm(req, tfm);
- aead_request_set_ad(req, ehsz);
- aead_request_set_crypt(req, sg, sg, len - ehsz, iv);
-
- /* Set callback function & data */
- aead_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- tipc_aead_encrypt_done, skb);
- tx_ctx = (struct tipc_crypto_tx_ctx *)ctx;
- tx_ctx->aead = aead;
- tx_ctx->bearer = b;
- memcpy(&tx_ctx->dst, dst, sizeof(*dst));
-
- /* Hold bearer */
- if (unlikely(!tipc_bearer_hold(b))) {
- rc = -ENODEV;
- goto exit;
- }
-
- /* Get net to avoid freed tipc_crypto when delete namespace */
- if (!maybe_get_net(aead->crypto->net)) {
- tipc_bearer_put(b);
- rc = -ENODEV;
- goto exit;
- }
-
- /* Now, do encrypt */
- rc = crypto_aead_encrypt(req);
- if (rc == -EINPROGRESS || rc == -EBUSY)
- return rc;
-
- tipc_bearer_put(b);
- put_net(aead->crypto->net);
-
-exit:
- kfree(ctx);
- TIPC_SKB_CB(skb)->crypto_ctx = NULL;
- return rc;
-}
-static void tipc_aead_encrypt_done(void *data, int err)
-{
- struct sk_buff *skb = data;
- struct tipc_crypto_tx_ctx *tx_ctx = TIPC_SKB_CB(skb)->crypto_ctx;
- struct tipc_bearer *b = tx_ctx->bearer;
- struct tipc_aead *aead = tx_ctx->aead;
- struct tipc_crypto *tx = aead->crypto;
- struct net *net = tx->net;
-
- switch (err) {
- case 0:
- this_cpu_inc(tx->stats->stat[STAT_ASYNC_OK]);
- rcu_read_lock();
- if (likely(test_bit(0, &b->up)))
- b->media->send_msg(net, skb, b, &tx_ctx->dst);
- else
- kfree_skb(skb);
- rcu_read_unlock();
- break;
- case -EINPROGRESS:
- return;
- default:
- this_cpu_inc(tx->stats->stat[STAT_ASYNC_NOK]);
- kfree_skb(skb);
- break;
- }
-
- kfree(tx_ctx);
- tipc_bearer_put(b);
- tipc_aead_put(aead);
- put_net(net);
+ /* Encrypt the skb in-place. */
+ aes_gcm_encrypt(skb->data + ehsz, skb->data + len, skb->data + ehsz,
+ len - ehsz, skb->data, ehsz, iv, &aead->gcm_key);
+ return 0;
}
/**
@@ -871,22 +649,18 @@ static void tipc_aead_encrypt_done(void *data, int err)
*
* Return:
* * 0 : if the decryption has completed
- * * -EINPROGRESS/-EBUSY : if a callback will be performed
* * < 0 : the decryption has failed
*/
static int tipc_aead_decrypt(struct net *net, struct tipc_aead *aead,
struct sk_buff *skb, struct tipc_bearer *b)
{
- struct tipc_crypto_rx_ctx *rx_ctx;
- struct aead_request *req;
- struct crypto_aead *tfm;
+ struct aes_gcm_ctx ctx;
struct sk_buff *unused;
- struct scatterlist *sg;
struct tipc_ehdr *ehdr;
- int ehsz, nsg, rc;
- void *ctx;
+ int ehsz, nsg, rc, crypt_len;
u32 salt;
- u8 *iv;
+ u8 iv[TIPC_AES_GCM_IV_SIZE];
+ u8 authtag[TIPC_AES_GCM_TAG_SIZE];
if (unlikely(!aead))
return -ENOKEY;
@@ -897,21 +671,6 @@ static int tipc_aead_decrypt(struct net *net, struct tipc_aead *aead,
return nsg;
}
- /* Allocate memory for the AEAD operation */
- tfm = tipc_aead_tfm_next(aead);
- ctx = tipc_aead_mem_alloc(tfm, sizeof(*rx_ctx), &iv, &req, &sg, nsg);
- if (unlikely(!ctx))
- return -ENOMEM;
- TIPC_SKB_CB(skb)->crypto_ctx = ctx;
-
- /* Map skb to the sg lists */
- sg_init_table(sg, nsg);
- rc = skb_to_sgvec(skb, sg, 0, skb->len);
- if (unlikely(rc < 0)) {
- pr_err("RX: skb_to_sgvec() returned %d, nsg %d\n", rc, nsg);
- goto exit;
- }
-
/* Reconstruct IV: */
ehdr = (struct tipc_ehdr *)skb->data;
salt = aead->salt;
@@ -922,77 +681,19 @@ static int tipc_aead_decrypt(struct net *net, struct tipc_aead *aead,
memcpy(iv, &salt, 4);
memcpy(iv + 4, (u8 *)&ehdr->seqno, 8);
- /* Prepare request */
ehsz = tipc_ehdr_size(ehdr);
- aead_request_set_tfm(req, tfm);
- aead_request_set_ad(req, ehsz);
- aead_request_set_crypt(req, sg, sg, skb->len - ehsz, iv);
-
- /* Set callback function & data */
- aead_request_set_callback(req, CRYPTO_TFM_REQ_MAY_BACKLOG,
- tipc_aead_decrypt_done, skb);
- rx_ctx = (struct tipc_crypto_rx_ctx *)ctx;
- rx_ctx->aead = aead;
- rx_ctx->bearer = b;
-
- /* Hold bearer */
- if (unlikely(!tipc_bearer_hold(b))) {
- rc = -ENODEV;
- goto exit;
- }
+ crypt_len = skb->len - ehsz - aead->authsize;
+ if (unlikely(crypt_len < 0))
+ return -EBADMSG;
- /* Get net to avoid freed tipc_crypto when delete namespace */
- if (!maybe_get_net(net)) {
- tipc_bearer_put(b);
- rc = -ENODEV;
- goto exit;
- }
-
- /* Now, do decrypt */
- rc = crypto_aead_decrypt(req);
- if (rc == -EINPROGRESS || rc == -EBUSY)
+ rc = skb_copy_bits(skb, skb->len - aead->authsize, authtag,
+ aead->authsize);
+ if (unlikely(rc < 0))
return rc;
- tipc_bearer_put(b);
- put_net(net);
-
-exit:
- kfree(ctx);
- TIPC_SKB_CB(skb)->crypto_ctx = NULL;
- return rc;
-}
-
-static void tipc_aead_decrypt_done(void *data, int err)
-{
- struct sk_buff *skb = data;
- struct tipc_crypto_rx_ctx *rx_ctx = TIPC_SKB_CB(skb)->crypto_ctx;
- struct tipc_bearer *b = rx_ctx->bearer;
- struct tipc_aead *aead = rx_ctx->aead;
- struct tipc_crypto_stats __percpu *stats = aead->crypto->stats;
- struct net *net = aead->crypto->net;
-
- switch (err) {
- case 0:
- this_cpu_inc(stats->stat[STAT_ASYNC_OK]);
- break;
- case -EINPROGRESS:
- return;
- default:
- this_cpu_inc(stats->stat[STAT_ASYNC_NOK]);
- break;
- }
-
- kfree(rx_ctx);
- tipc_crypto_rcv_complete(net, aead, b, &skb, err);
- if (likely(skb)) {
- if (likely(test_bit(0, &b->up)))
- tipc_rcv(net, skb, b);
- else
- kfree_skb(skb);
- }
-
- tipc_bearer_put(b);
- put_net(net);
+ aes_gcm_init(&ctx, iv, &aead->gcm_key);
+ tipc_decrypt_skb(&ctx, skb, ehsz, crypt_len);
+ return aes_gcm_decrypt_final(&ctx, authtag);
}
static inline int tipc_ehdr_size(struct tipc_ehdr *ehdr)
@@ -1680,7 +1381,6 @@ static inline void tipc_crypto_clone_msg(struct net *net, struct sk_buff *_skb,
*
* Return:
* * 0 : the encryption has succeeded (or no encryption)
- * * -EINPROGRESS/-EBUSY : the encryption is ongoing, a callback will be made
* * -ENOKEK : the encryption has failed due to no key
* * -EKEYREVOKED : the encryption has failed due to key revoked
* * -ENOMEM : the encryption has failed due to no memory
@@ -1769,11 +1469,6 @@ int tipc_crypto_xmit(struct net *net, struct sk_buff **skb,
case 0:
this_cpu_inc(stats->stat[STAT_OK]);
break;
- case -EINPROGRESS:
- case -EBUSY:
- this_cpu_inc(stats->stat[STAT_ASYNC]);
- *skb = NULL;
- return rc;
default:
this_cpu_inc(stats->stat[STAT_NOK]);
if (rc == -ENOKEY)
@@ -1805,7 +1500,6 @@ int tipc_crypto_xmit(struct net *net, struct sk_buff **skb,
*
* Return:
* * 0 : the decryption has successfully completed
- * * -EINPROGRESS/-EBUSY : the decryption is ongoing, a callback will be made
* * -ENOKEY : the decryption has failed due to no key
* * -EBADMSG : the decryption has failed due to bad message
* * -ENOMEM : the decryption has failed due to no memory
@@ -1859,11 +1553,6 @@ int tipc_crypto_rcv(struct net *net, struct tipc_crypto *rx,
case 0:
this_cpu_inc(stats->stat[STAT_OK]);
break;
- case -EINPROGRESS:
- case -EBUSY:
- this_cpu_inc(stats->stat[STAT_ASYNC]);
- *skb = NULL;
- return rc;
default:
this_cpu_inc(stats->stat[STAT_NOK]);
if (rc == -ENOKEY) {
diff --git a/net/tipc/msg.h b/net/tipc/msg.h
index c5eec16213d7..c7f0105148bb 100644
--- a/net/tipc/msg.h
+++ b/net/tipc/msg.h
@@ -138,9 +138,6 @@ struct tipc_skb_cb {
u8 flags;
};
u8 reserved;
-#ifdef CONFIG_TIPC_CRYPTO
- void *crypto_ctx;
-#endif
} __packed;
#define TIPC_SKB_CB(__skb) ((struct tipc_skb_cb *)&((__skb)->cb[0]))
diff --git a/net/tipc/sysctl.c b/net/tipc/sysctl.c
index 30d2e06e3d8c..3e8d9def5674 100644
--- a/net/tipc/sysctl.c
+++ b/net/tipc/sysctl.c
@@ -67,6 +67,12 @@ static struct ctl_table tipc_table[] = {
},
#ifdef CONFIG_TIPC_CRYPTO
{
+ /*
+ * This sysctl no longer has any effect on the way that TIPC
+ * uses the crypto subsystem. However, a special value can
+ * still be written to this to trigger debug commands. See
+ * tipc_crypto_do_cmd().
+ */
.procname = "max_tfms",
.data = &sysctl_tipc_max_tfms,
.maxlen = sizeof(sysctl_tipc_max_tfms),
--
2.54.0
^ permalink raw reply related [flat|nested] 45+ messages in thread