* [PATCH 0/4] crypto: lib/chacha - improve type safety
@ 2025-05-05 18:18 Eric Biggers
2025-05-05 18:18 ` [PATCH 1/4] crypto: lib/chacha - strongly type the ChaCha state Eric Biggers
` (4 more replies)
0 siblings, 5 replies; 7+ messages in thread
From: Eric Biggers @ 2025-05-05 18:18 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arm-kernel, linux-mips, linuxppc-dev,
linux-riscv, linux-s390, linux-bcachefs, Jason A . Donenfeld ,
Theodore Ts'o
This series can also be retrieved from:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git chacha-prototypes-v1
This series improves type safety and clarity in the ChaCha functions by
strongly typing the state array and adding explicit array bounds for
other fixed-length arrays. No functional changes.
Eric Biggers (4):
crypto: lib/chacha - strongly type the ChaCha state
crypto: lib/chacha - use struct assignment to copy state
crypto: lib/chacha - add strongly-typed state zeroization
crypto: lib/chacha - add array bounds to function prototypes
arch/arm/lib/crypto/chacha-glue.c | 34 ++++----
arch/arm/lib/crypto/chacha-scalar-core.S | 5 +-
arch/arm64/lib/crypto/chacha-neon-glue.c | 27 ++++---
arch/mips/lib/crypto/chacha-glue.c | 6 +-
arch/powerpc/lib/crypto/chacha-p10-glue.c | 17 ++--
arch/powerpc/lib/crypto/chacha-p10le-8x.S | 6 +-
arch/riscv/lib/crypto/chacha-riscv64-glue.c | 9 ++-
arch/riscv/lib/crypto/chacha-riscv64-zvkb.S | 10 +--
arch/s390/lib/crypto/chacha-glue.c | 13 +--
arch/x86/lib/crypto/chacha_glue.c | 62 ++++++++------
crypto/chacha.c | 20 ++---
drivers/char/random.c | 41 +++++-----
fs/bcachefs/checksum.c | 18 ++---
include/crypto/chacha.h | 80 ++++++++++++-------
lib/crypto/chacha.c | 40 +++++-----
lib/crypto/chacha20poly1305-selftest.c | 8 +-
lib/crypto/chacha20poly1305.c | 53 ++++++------
lib/crypto/libchacha.c | 2 +-
.../crypto/chacha20-s390/test-cipher.c | 10 +--
19 files changed, 253 insertions(+), 208 deletions(-)
base-commit: 64745a9ca890ed60d78162ec511e1983e1946d73
--
2.49.0
^ permalink raw reply [flat|nested] 7+ messages in thread
* [PATCH 1/4] crypto: lib/chacha - strongly type the ChaCha state
2025-05-05 18:18 [PATCH 0/4] crypto: lib/chacha - improve type safety Eric Biggers
@ 2025-05-05 18:18 ` Eric Biggers
2025-05-06 16:09 ` Kent Overstreet
2025-05-05 18:18 ` [PATCH 2/4] crypto: lib/chacha - use struct assignment to copy state Eric Biggers
` (3 subsequent siblings)
4 siblings, 1 reply; 7+ messages in thread
From: Eric Biggers @ 2025-05-05 18:18 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arm-kernel, linux-mips, linuxppc-dev,
linux-riscv, linux-s390, linux-bcachefs, Jason A . Donenfeld ,
Theodore Ts'o
From: Eric Biggers <ebiggers@google.com>
The ChaCha state matrix is 16 32-bit words. Currently it is represented
in the code as a raw u32 array, or even just a pointer to u32. This
weak typing is error-prone. Instead, introduce struct chacha_state:
struct chacha_state {
u32 x[16];
};
Convert all ChaCha and HChaCha functions to use struct chacha_state.
No functional changes.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/arm/lib/crypto/chacha-glue.c | 30 +++++----
arch/arm/lib/crypto/chacha-scalar-core.S | 5 +-
arch/arm64/lib/crypto/chacha-neon-glue.c | 23 ++++---
arch/mips/lib/crypto/chacha-glue.c | 6 +-
arch/powerpc/lib/crypto/chacha-p10-glue.c | 15 +++--
arch/powerpc/lib/crypto/chacha-p10le-8x.S | 6 +-
arch/riscv/lib/crypto/chacha-riscv64-glue.c | 8 +--
arch/riscv/lib/crypto/chacha-riscv64-zvkb.S | 10 +--
arch/s390/lib/crypto/chacha-glue.c | 11 ++--
arch/x86/lib/crypto/chacha_glue.c | 58 ++++++++++-------
crypto/chacha.c | 16 ++---
drivers/char/random.c | 41 ++++++------
fs/bcachefs/checksum.c | 18 ++---
include/crypto/chacha.h | 65 +++++++++++--------
lib/crypto/chacha.c | 35 +++++-----
lib/crypto/chacha20poly1305-selftest.c | 8 +--
lib/crypto/chacha20poly1305.c | 51 ++++++++-------
lib/crypto/libchacha.c | 2 +-
.../crypto/chacha20-s390/test-cipher.c | 10 +--
19 files changed, 228 insertions(+), 190 deletions(-)
diff --git a/arch/arm/lib/crypto/chacha-glue.c b/arch/arm/lib/crypto/chacha-glue.c
index 1e28736834a0..0c2b4c62d484 100644
--- a/arch/arm/lib/crypto/chacha-glue.c
+++ b/arch/arm/lib/crypto/chacha-glue.c
@@ -15,28 +15,31 @@
#include <asm/cputype.h>
#include <asm/hwcap.h>
#include <asm/neon.h>
#include <asm/simd.h>
-asmlinkage void chacha_block_xor_neon(const u32 *state, u8 *dst, const u8 *src,
- int nrounds);
-asmlinkage void chacha_4block_xor_neon(const u32 *state, u8 *dst, const u8 *src,
+asmlinkage void chacha_block_xor_neon(const struct chacha_state *state,
+ u8 *dst, const u8 *src, int nrounds);
+asmlinkage void chacha_4block_xor_neon(const struct chacha_state *state,
+ u8 *dst, const u8 *src,
int nrounds, unsigned int nbytes);
-asmlinkage void hchacha_block_arm(const u32 *state, u32 *out, int nrounds);
-asmlinkage void hchacha_block_neon(const u32 *state, u32 *out, int nrounds);
+asmlinkage void hchacha_block_arm(const struct chacha_state *state,
+ u32 *out, int nrounds);
+asmlinkage void hchacha_block_neon(const struct chacha_state *state,
+ u32 *out, int nrounds);
asmlinkage void chacha_doarm(u8 *dst, const u8 *src, unsigned int bytes,
- const u32 *state, int nrounds);
+ const struct chacha_state *state, int nrounds);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(use_neon);
static inline bool neon_usable(void)
{
return static_branch_likely(&use_neon) && crypto_simd_usable();
}
-static void chacha_doneon(u32 *state, u8 *dst, const u8 *src,
+static void chacha_doneon(struct chacha_state *state, u8 *dst, const u8 *src,
unsigned int bytes, int nrounds)
{
u8 buf[CHACHA_BLOCK_SIZE];
while (bytes > CHACHA_BLOCK_SIZE) {
@@ -44,26 +47,27 @@ static void chacha_doneon(u32 *state, u8 *dst, const u8 *src,
chacha_4block_xor_neon(state, dst, src, nrounds, l);
bytes -= l;
src += l;
dst += l;
- state[12] += DIV_ROUND_UP(l, CHACHA_BLOCK_SIZE);
+ state->x[12] += DIV_ROUND_UP(l, CHACHA_BLOCK_SIZE);
}
if (bytes) {
const u8 *s = src;
u8 *d = dst;
if (bytes != CHACHA_BLOCK_SIZE)
s = d = memcpy(buf, src, bytes);
chacha_block_xor_neon(state, d, s, nrounds);
if (d != dst)
memcpy(dst, buf, bytes);
- state[12]++;
+ state->x[12]++;
}
}
-void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
+void hchacha_block_arch(const struct chacha_state *state, u32 *stream,
+ int nrounds)
{
if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON) || !neon_usable()) {
hchacha_block_arm(state, stream, nrounds);
} else {
kernel_neon_begin();
@@ -71,17 +75,17 @@ void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
kernel_neon_end();
}
}
EXPORT_SYMBOL(hchacha_block_arch);
-void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes,
- int nrounds)
+void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
+ unsigned int bytes, int nrounds)
{
if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON) || !neon_usable() ||
bytes <= CHACHA_BLOCK_SIZE) {
chacha_doarm(dst, src, bytes, state, nrounds);
- state[12] += DIV_ROUND_UP(bytes, CHACHA_BLOCK_SIZE);
+ state->x[12] += DIV_ROUND_UP(bytes, CHACHA_BLOCK_SIZE);
return;
}
do {
unsigned int todo = min_t(unsigned int, bytes, SZ_4K);
diff --git a/arch/arm/lib/crypto/chacha-scalar-core.S b/arch/arm/lib/crypto/chacha-scalar-core.S
index 083fe1ab96d0..d20b5de755cc 100644
--- a/arch/arm/lib/crypto/chacha-scalar-core.S
+++ b/arch/arm/lib/crypto/chacha-scalar-core.S
@@ -365,11 +365,11 @@
.Ldone\@:
.endm // _chacha
/*
* void chacha_doarm(u8 *dst, const u8 *src, unsigned int bytes,
- * const u32 *state, int nrounds);
+ * const struct chacha_state *state, int nrounds);
*/
ENTRY(chacha_doarm)
cmp r2, #0 // len == 0?
reteq lr
@@ -405,11 +405,12 @@ ENTRY(chacha_doarm)
1: _chacha 12
b 0b
ENDPROC(chacha_doarm)
/*
- * void hchacha_block_arm(const u32 state[16], u32 out[8], int nrounds);
+ * void hchacha_block_arm(const struct chacha_state *state,
+ * u32 out[8], int nrounds);
*/
ENTRY(hchacha_block_arm)
push {r1,r4-r11,lr}
cmp r2, #12 // ChaCha12 ?
diff --git a/arch/arm64/lib/crypto/chacha-neon-glue.c b/arch/arm64/lib/crypto/chacha-neon-glue.c
index 2b0de97a6daf..7b451b3c7240 100644
--- a/arch/arm64/lib/crypto/chacha-neon-glue.c
+++ b/arch/arm64/lib/crypto/chacha-neon-glue.c
@@ -26,19 +26,21 @@
#include <asm/hwcap.h>
#include <asm/neon.h>
#include <asm/simd.h>
-asmlinkage void chacha_block_xor_neon(u32 *state, u8 *dst, const u8 *src,
- int nrounds);
-asmlinkage void chacha_4block_xor_neon(u32 *state, u8 *dst, const u8 *src,
+asmlinkage void chacha_block_xor_neon(const struct chacha_state *state,
+ u8 *dst, const u8 *src, int nrounds);
+asmlinkage void chacha_4block_xor_neon(const struct chacha_state *state,
+ u8 *dst, const u8 *src,
int nrounds, int bytes);
-asmlinkage void hchacha_block_neon(const u32 *state, u32 *out, int nrounds);
+asmlinkage void hchacha_block_neon(const struct chacha_state *state,
+ u32 *out, int nrounds);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon);
-static void chacha_doneon(u32 *state, u8 *dst, const u8 *src,
+static void chacha_doneon(struct chacha_state *state, u8 *dst, const u8 *src,
int bytes, int nrounds)
{
while (bytes > 0) {
int l = min(bytes, CHACHA_BLOCK_SIZE * 5);
@@ -46,22 +48,23 @@ static void chacha_doneon(u32 *state, u8 *dst, const u8 *src,
u8 buf[CHACHA_BLOCK_SIZE];
memcpy(buf, src, l);
chacha_block_xor_neon(state, buf, buf, nrounds);
memcpy(dst, buf, l);
- state[12] += 1;
+ state->x[12] += 1;
break;
}
chacha_4block_xor_neon(state, dst, src, nrounds, l);
bytes -= l;
src += l;
dst += l;
- state[12] += DIV_ROUND_UP(l, CHACHA_BLOCK_SIZE);
+ state->x[12] += DIV_ROUND_UP(l, CHACHA_BLOCK_SIZE);
}
}
-void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
+void hchacha_block_arch(const struct chacha_state *state, u32 *stream,
+ int nrounds)
{
if (!static_branch_likely(&have_neon) || !crypto_simd_usable()) {
hchacha_block_generic(state, stream, nrounds);
} else {
kernel_neon_begin();
@@ -69,12 +72,12 @@ void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
kernel_neon_end();
}
}
EXPORT_SYMBOL(hchacha_block_arch);
-void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes,
- int nrounds)
+void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
+ unsigned int bytes, int nrounds)
{
if (!static_branch_likely(&have_neon) || bytes <= CHACHA_BLOCK_SIZE ||
!crypto_simd_usable())
return chacha_crypt_generic(state, dst, src, bytes, nrounds);
diff --git a/arch/mips/lib/crypto/chacha-glue.c b/arch/mips/lib/crypto/chacha-glue.c
index 334ecb29fb8f..75df4040cded 100644
--- a/arch/mips/lib/crypto/chacha-glue.c
+++ b/arch/mips/lib/crypto/chacha-glue.c
@@ -7,15 +7,17 @@
#include <crypto/chacha.h>
#include <linux/kernel.h>
#include <linux/module.h>
-asmlinkage void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src,
+asmlinkage void chacha_crypt_arch(struct chacha_state *state,
+ u8 *dst, const u8 *src,
unsigned int bytes, int nrounds);
EXPORT_SYMBOL(chacha_crypt_arch);
-asmlinkage void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds);
+asmlinkage void hchacha_block_arch(const struct chacha_state *state,
+ u32 *stream, int nrounds);
EXPORT_SYMBOL(hchacha_block_arch);
bool chacha_is_arch_optimized(void)
{
return true;
diff --git a/arch/powerpc/lib/crypto/chacha-p10-glue.c b/arch/powerpc/lib/crypto/chacha-p10-glue.c
index 51daeaf5d26e..a6e6a8da1b8b 100644
--- a/arch/powerpc/lib/crypto/chacha-p10-glue.c
+++ b/arch/powerpc/lib/crypto/chacha-p10-glue.c
@@ -12,12 +12,12 @@
#include <linux/cpufeature.h>
#include <linux/sizes.h>
#include <asm/simd.h>
#include <asm/switch_to.h>
-asmlinkage void chacha_p10le_8x(u32 *state, u8 *dst, const u8 *src,
- unsigned int len, int nrounds);
+asmlinkage void chacha_p10le_8x(const struct chacha_state *state, u8 *dst,
+ const u8 *src, unsigned int len, int nrounds);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_p10);
static void vsx_begin(void)
{
@@ -29,35 +29,36 @@ static void vsx_end(void)
{
disable_kernel_vsx();
preempt_enable();
}
-static void chacha_p10_do_8x(u32 *state, u8 *dst, const u8 *src,
+static void chacha_p10_do_8x(struct chacha_state *state, u8 *dst, const u8 *src,
unsigned int bytes, int nrounds)
{
unsigned int l = bytes & ~0x0FF;
if (l > 0) {
chacha_p10le_8x(state, dst, src, l, nrounds);
bytes -= l;
src += l;
dst += l;
- state[12] += l / CHACHA_BLOCK_SIZE;
+ state->x[12] += l / CHACHA_BLOCK_SIZE;
}
if (bytes > 0)
chacha_crypt_generic(state, dst, src, bytes, nrounds);
}
-void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
+void hchacha_block_arch(const struct chacha_state *state,
+ u32 *stream, int nrounds)
{
hchacha_block_generic(state, stream, nrounds);
}
EXPORT_SYMBOL(hchacha_block_arch);
-void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes,
- int nrounds)
+void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
+ unsigned int bytes, int nrounds)
{
if (!static_branch_likely(&have_p10) || bytes <= CHACHA_BLOCK_SIZE ||
!crypto_simd_usable())
return chacha_crypt_generic(state, dst, src, bytes, nrounds);
diff --git a/arch/powerpc/lib/crypto/chacha-p10le-8x.S b/arch/powerpc/lib/crypto/chacha-p10le-8x.S
index 17bedb66b822..b29562bd5d40 100644
--- a/arch/powerpc/lib/crypto/chacha-p10le-8x.S
+++ b/arch/powerpc/lib/crypto/chacha-p10le-8x.S
@@ -5,13 +5,10 @@
# Copyright 2023- IBM Corp. All rights reserved
#
#===================================================================================
# Written by Danny Tsen <dtsen@us.ibm.com>
#
-# chacha_p10le_8x(u32 *state, byte *dst, const byte *src,
-# size_t len, int nrounds);
-#
# do rounds, 8 quarter rounds
# 1. a += b; d ^= a; d <<<= 16;
# 2. c += d; b ^= c; b <<<= 12;
# 3. a += b; d ^= a; d <<<= 8;
# 4. c += d; b ^= c; b <<<= 7
@@ -573,11 +570,12 @@
stxvw4x \S+47, 31, 16
.endm
#
-# chacha20_p10le_8x(u32 *state, byte *dst, const byte *src, size_t len, int nrounds);
+# void chacha_p10le_8x(const struct chacha_state *state, u8 *dst, const u8 *src,
+# unsigned int len, int nrounds);
#
SYM_FUNC_START(chacha_p10le_8x)
.align 5
cmpdi 6, 0
ble Out_no_chacha
diff --git a/arch/riscv/lib/crypto/chacha-riscv64-glue.c b/arch/riscv/lib/crypto/chacha-riscv64-glue.c
index 1740e1ca3a94..57541621981e 100644
--- a/arch/riscv/lib/crypto/chacha-riscv64-glue.c
+++ b/arch/riscv/lib/crypto/chacha-riscv64-glue.c
@@ -13,21 +13,21 @@
#include <linux/linkage.h>
#include <linux/module.h>
static __ro_after_init DEFINE_STATIC_KEY_FALSE(use_zvkb);
-asmlinkage void chacha_zvkb(u32 state[16], const u8 *in, u8 *out,
+asmlinkage void chacha_zvkb(struct chacha_state *state, const u8 *in, u8 *out,
size_t nblocks, int nrounds);
-void hchacha_block_arch(const u32 *state, u32 *out, int nrounds)
+void hchacha_block_arch(const struct chacha_state *state, u32 *out, int nrounds)
{
hchacha_block_generic(state, out, nrounds);
}
EXPORT_SYMBOL(hchacha_block_arch);
-void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes,
- int nrounds)
+void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
+ unsigned int bytes, int nrounds)
{
u8 block_buffer[CHACHA_BLOCK_SIZE];
unsigned int full_blocks = bytes / CHACHA_BLOCK_SIZE;
unsigned int tail_bytes = bytes % CHACHA_BLOCK_SIZE;
diff --git a/arch/riscv/lib/crypto/chacha-riscv64-zvkb.S b/arch/riscv/lib/crypto/chacha-riscv64-zvkb.S
index ab4423b3880e..b777d0b4e379 100644
--- a/arch/riscv/lib/crypto/chacha-riscv64-zvkb.S
+++ b/arch/riscv/lib/crypto/chacha-riscv64-zvkb.S
@@ -130,19 +130,19 @@
vror.vi \b1, \b1, 32 - 7
vror.vi \b2, \b2, 32 - 7
vror.vi \b3, \b3, 32 - 7
.endm
-// void chacha_zvkb(u32 state[16], const u8 *in, u8 *out, size_t nblocks,
-// int nrounds);
+// void chacha_zvkb(struct chacha_state *state, const u8 *in, u8 *out,
+// size_t nblocks, int nrounds);
//
// |nblocks| is the number of 64-byte blocks to process, and must be nonzero.
//
// |state| gives the ChaCha state matrix, including the 32-bit counter in
-// state[12] following the RFC7539 convention; note that this differs from the
-// original Salsa20 paper which uses a 64-bit counter in state[12..13]. The
-// updated 32-bit counter is written back to state[12] before returning.
+// state->x[12] following the RFC7539 convention; note that this differs from
+// the original Salsa20 paper which uses a 64-bit counter in state->x[12..13].
+// The updated 32-bit counter is written back to state->x[12] before returning.
SYM_FUNC_START(chacha_zvkb)
addi sp, sp, -96
sd s0, 0(sp)
sd s1, 8(sp)
sd s2, 16(sp)
diff --git a/arch/s390/lib/crypto/chacha-glue.c b/arch/s390/lib/crypto/chacha-glue.c
index b3ffaa555385..0a9fd50c1bd8 100644
--- a/arch/s390/lib/crypto/chacha-glue.c
+++ b/arch/s390/lib/crypto/chacha-glue.c
@@ -14,18 +14,19 @@
#include <linux/module.h>
#include <linux/sizes.h>
#include <asm/fpu.h>
#include "chacha-s390.h"
-void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
+void hchacha_block_arch(const struct chacha_state *state,
+ u32 *stream, int nrounds)
{
/* TODO: implement hchacha_block_arch() in assembly */
hchacha_block_generic(state, stream, nrounds);
}
EXPORT_SYMBOL(hchacha_block_arch);
-void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src,
+void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
unsigned int bytes, int nrounds)
{
/* s390 chacha20 implementation has 20 rounds hard-coded,
* it cannot handle a block of data or less, but otherwise
* it can handle data of arbitrary size
@@ -34,15 +35,15 @@ void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src,
chacha_crypt_generic(state, dst, src, bytes, nrounds);
} else {
DECLARE_KERNEL_FPU_ONSTACK32(vxstate);
kernel_fpu_begin(&vxstate, KERNEL_VXR);
- chacha20_vx(dst, src, bytes, &state[4], &state[12]);
+ chacha20_vx(dst, src, bytes, &state->x[4], &state->x[12]);
kernel_fpu_end(&vxstate, KERNEL_VXR);
- state[12] += round_up(bytes, CHACHA_BLOCK_SIZE) /
- CHACHA_BLOCK_SIZE;
+ state->x[12] += round_up(bytes, CHACHA_BLOCK_SIZE) /
+ CHACHA_BLOCK_SIZE;
}
}
EXPORT_SYMBOL(chacha_crypt_arch);
bool chacha_is_arch_optimized(void)
diff --git a/arch/x86/lib/crypto/chacha_glue.c b/arch/x86/lib/crypto/chacha_glue.c
index 94fcefbc8827..6f00a56e3e9a 100644
--- a/arch/x86/lib/crypto/chacha_glue.c
+++ b/arch/x86/lib/crypto/chacha_glue.c
@@ -10,28 +10,37 @@
#include <linux/jump_label.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/sizes.h>
-asmlinkage void chacha_block_xor_ssse3(u32 *state, u8 *dst, const u8 *src,
+asmlinkage void chacha_block_xor_ssse3(const struct chacha_state *state,
+ u8 *dst, const u8 *src,
unsigned int len, int nrounds);
-asmlinkage void chacha_4block_xor_ssse3(u32 *state, u8 *dst, const u8 *src,
+asmlinkage void chacha_4block_xor_ssse3(const struct chacha_state *state,
+ u8 *dst, const u8 *src,
unsigned int len, int nrounds);
-asmlinkage void hchacha_block_ssse3(const u32 *state, u32 *out, int nrounds);
+asmlinkage void hchacha_block_ssse3(const struct chacha_state *state,
+ u32 *out, int nrounds);
-asmlinkage void chacha_2block_xor_avx2(u32 *state, u8 *dst, const u8 *src,
+asmlinkage void chacha_2block_xor_avx2(const struct chacha_state *state,
+ u8 *dst, const u8 *src,
unsigned int len, int nrounds);
-asmlinkage void chacha_4block_xor_avx2(u32 *state, u8 *dst, const u8 *src,
+asmlinkage void chacha_4block_xor_avx2(const struct chacha_state *state,
+ u8 *dst, const u8 *src,
unsigned int len, int nrounds);
-asmlinkage void chacha_8block_xor_avx2(u32 *state, u8 *dst, const u8 *src,
+asmlinkage void chacha_8block_xor_avx2(const struct chacha_state *state,
+ u8 *dst, const u8 *src,
unsigned int len, int nrounds);
-asmlinkage void chacha_2block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src,
+asmlinkage void chacha_2block_xor_avx512vl(const struct chacha_state *state,
+ u8 *dst, const u8 *src,
unsigned int len, int nrounds);
-asmlinkage void chacha_4block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src,
+asmlinkage void chacha_4block_xor_avx512vl(const struct chacha_state *state,
+ u8 *dst, const u8 *src,
unsigned int len, int nrounds);
-asmlinkage void chacha_8block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src,
+asmlinkage void chacha_8block_xor_avx512vl(const struct chacha_state *state,
+ u8 *dst, const u8 *src,
unsigned int len, int nrounds);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(chacha_use_simd);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(chacha_use_avx2);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(chacha_use_avx512vl);
@@ -40,86 +49,87 @@ static unsigned int chacha_advance(unsigned int len, unsigned int maxblocks)
{
len = min(len, maxblocks * CHACHA_BLOCK_SIZE);
return round_up(len, CHACHA_BLOCK_SIZE) / CHACHA_BLOCK_SIZE;
}
-static void chacha_dosimd(u32 *state, u8 *dst, const u8 *src,
+static void chacha_dosimd(struct chacha_state *state, u8 *dst, const u8 *src,
unsigned int bytes, int nrounds)
{
if (static_branch_likely(&chacha_use_avx512vl)) {
while (bytes >= CHACHA_BLOCK_SIZE * 8) {
chacha_8block_xor_avx512vl(state, dst, src, bytes,
nrounds);
bytes -= CHACHA_BLOCK_SIZE * 8;
src += CHACHA_BLOCK_SIZE * 8;
dst += CHACHA_BLOCK_SIZE * 8;
- state[12] += 8;
+ state->x[12] += 8;
}
if (bytes > CHACHA_BLOCK_SIZE * 4) {
chacha_8block_xor_avx512vl(state, dst, src, bytes,
nrounds);
- state[12] += chacha_advance(bytes, 8);
+ state->x[12] += chacha_advance(bytes, 8);
return;
}
if (bytes > CHACHA_BLOCK_SIZE * 2) {
chacha_4block_xor_avx512vl(state, dst, src, bytes,
nrounds);
- state[12] += chacha_advance(bytes, 4);
+ state->x[12] += chacha_advance(bytes, 4);
return;
}
if (bytes) {
chacha_2block_xor_avx512vl(state, dst, src, bytes,
nrounds);
- state[12] += chacha_advance(bytes, 2);
+ state->x[12] += chacha_advance(bytes, 2);
return;
}
}
if (static_branch_likely(&chacha_use_avx2)) {
while (bytes >= CHACHA_BLOCK_SIZE * 8) {
chacha_8block_xor_avx2(state, dst, src, bytes, nrounds);
bytes -= CHACHA_BLOCK_SIZE * 8;
src += CHACHA_BLOCK_SIZE * 8;
dst += CHACHA_BLOCK_SIZE * 8;
- state[12] += 8;
+ state->x[12] += 8;
}
if (bytes > CHACHA_BLOCK_SIZE * 4) {
chacha_8block_xor_avx2(state, dst, src, bytes, nrounds);
- state[12] += chacha_advance(bytes, 8);
+ state->x[12] += chacha_advance(bytes, 8);
return;
}
if (bytes > CHACHA_BLOCK_SIZE * 2) {
chacha_4block_xor_avx2(state, dst, src, bytes, nrounds);
- state[12] += chacha_advance(bytes, 4);
+ state->x[12] += chacha_advance(bytes, 4);
return;
}
if (bytes > CHACHA_BLOCK_SIZE) {
chacha_2block_xor_avx2(state, dst, src, bytes, nrounds);
- state[12] += chacha_advance(bytes, 2);
+ state->x[12] += chacha_advance(bytes, 2);
return;
}
}
while (bytes >= CHACHA_BLOCK_SIZE * 4) {
chacha_4block_xor_ssse3(state, dst, src, bytes, nrounds);
bytes -= CHACHA_BLOCK_SIZE * 4;
src += CHACHA_BLOCK_SIZE * 4;
dst += CHACHA_BLOCK_SIZE * 4;
- state[12] += 4;
+ state->x[12] += 4;
}
if (bytes > CHACHA_BLOCK_SIZE) {
chacha_4block_xor_ssse3(state, dst, src, bytes, nrounds);
- state[12] += chacha_advance(bytes, 4);
+ state->x[12] += chacha_advance(bytes, 4);
return;
}
if (bytes) {
chacha_block_xor_ssse3(state, dst, src, bytes, nrounds);
- state[12]++;
+ state->x[12]++;
}
}
-void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
+void hchacha_block_arch(const struct chacha_state *state,
+ u32 *stream, int nrounds)
{
if (!static_branch_likely(&chacha_use_simd)) {
hchacha_block_generic(state, stream, nrounds);
} else {
kernel_fpu_begin();
@@ -127,12 +137,12 @@ void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
kernel_fpu_end();
}
}
EXPORT_SYMBOL(hchacha_block_arch);
-void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes,
- int nrounds)
+void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
+ unsigned int bytes, int nrounds)
{
if (!static_branch_likely(&chacha_use_simd) ||
bytes <= CHACHA_BLOCK_SIZE)
return chacha_crypt_generic(state, dst, src, bytes, nrounds);
diff --git a/crypto/chacha.c b/crypto/chacha.c
index 28a8ad6197ab..73ce62a9ac22 100644
--- a/crypto/chacha.c
+++ b/crypto/chacha.c
@@ -48,28 +48,28 @@ static int chacha12_setkey(struct crypto_skcipher *tfm,
static int chacha_stream_xor(struct skcipher_request *req,
const struct chacha_ctx *ctx, const u8 *iv,
bool arch)
{
struct skcipher_walk walk;
- u32 state[16];
+ struct chacha_state state;
int err;
err = skcipher_walk_virt(&walk, req, false);
- chacha_init(state, ctx->key, iv);
+ chacha_init(&state, ctx->key, iv);
while (walk.nbytes > 0) {
unsigned int nbytes = walk.nbytes;
if (nbytes < walk.total)
nbytes = round_down(nbytes, CHACHA_BLOCK_SIZE);
if (arch)
- chacha_crypt(state, walk.dst.virt.addr,
+ chacha_crypt(&state, walk.dst.virt.addr,
walk.src.virt.addr, nbytes, ctx->nrounds);
else
- chacha_crypt_generic(state, walk.dst.virt.addr,
+ chacha_crypt_generic(&state, walk.dst.virt.addr,
walk.src.virt.addr, nbytes,
ctx->nrounds);
err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
}
@@ -95,19 +95,19 @@ static int crypto_chacha_crypt_arch(struct skcipher_request *req)
static int crypto_xchacha_crypt(struct skcipher_request *req, bool arch)
{
struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
const struct chacha_ctx *ctx = crypto_skcipher_ctx(tfm);
struct chacha_ctx subctx;
- u32 state[16];
+ struct chacha_state state;
u8 real_iv[16];
/* Compute the subkey given the original key and first 128 nonce bits */
- chacha_init(state, ctx->key, req->iv);
+ chacha_init(&state, ctx->key, req->iv);
if (arch)
- hchacha_block(state, subctx.key, ctx->nrounds);
+ hchacha_block(&state, subctx.key, ctx->nrounds);
else
- hchacha_block_generic(state, subctx.key, ctx->nrounds);
+ hchacha_block_generic(&state, subctx.key, ctx->nrounds);
subctx.nrounds = ctx->nrounds;
/* Build the real IV */
memcpy(&real_iv[0], req->iv + 24, 8); /* stream position */
memcpy(&real_iv[8], req->iv + 16, 8); /* remaining 64 nonce bits */
diff --git a/drivers/char/random.c b/drivers/char/random.c
index 38f2fab29c56..9f876ed2655b 100644
--- a/drivers/char/random.c
+++ b/drivers/char/random.c
@@ -307,24 +307,24 @@ static void crng_reseed(struct work_struct *work)
*
* The returned ChaCha state contains within it a copy of the old
* key value, at index 4, so the state should always be zeroed out
* immediately after using in order to maintain forward secrecy.
* If the state cannot be erased in a timely manner, then it is
- * safer to set the random_data parameter to &chacha_state[4] so
- * that this function overwrites it before returning.
+ * safer to set the random_data parameter to &chacha_state->x[4]
+ * so that this function overwrites it before returning.
*/
static void crng_fast_key_erasure(u8 key[CHACHA_KEY_SIZE],
- u32 chacha_state[CHACHA_STATE_WORDS],
+ struct chacha_state *chacha_state,
u8 *random_data, size_t random_data_len)
{
u8 first_block[CHACHA_BLOCK_SIZE];
BUG_ON(random_data_len > 32);
chacha_init_consts(chacha_state);
- memcpy(&chacha_state[4], key, CHACHA_KEY_SIZE);
- memset(&chacha_state[12], 0, sizeof(u32) * 4);
+ memcpy(&chacha_state->x[4], key, CHACHA_KEY_SIZE);
+ memset(&chacha_state->x[12], 0, sizeof(u32) * 4);
chacha20_block(chacha_state, first_block);
memcpy(key, first_block, CHACHA_KEY_SIZE);
memcpy(random_data, first_block + CHACHA_KEY_SIZE, random_data_len);
memzero_explicit(first_block, sizeof(first_block));
@@ -333,11 +333,11 @@ static void crng_fast_key_erasure(u8 key[CHACHA_KEY_SIZE],
/*
* This function returns a ChaCha state that you may use for generating
* random data. It also returns up to 32 bytes on its own of random data
* that may be used; random_data_len may not be greater than 32.
*/
-static void crng_make_state(u32 chacha_state[CHACHA_STATE_WORDS],
+static void crng_make_state(struct chacha_state *chacha_state,
u8 *random_data, size_t random_data_len)
{
unsigned long flags;
struct crng *crng;
@@ -393,38 +393,38 @@ static void crng_make_state(u32 chacha_state[CHACHA_STATE_WORDS],
local_unlock_irqrestore(&crngs.lock, flags);
}
static void _get_random_bytes(void *buf, size_t len)
{
- u32 chacha_state[CHACHA_STATE_WORDS];
+ struct chacha_state chacha_state;
u8 tmp[CHACHA_BLOCK_SIZE];
size_t first_block_len;
if (!len)
return;
first_block_len = min_t(size_t, 32, len);
- crng_make_state(chacha_state, buf, first_block_len);
+ crng_make_state(&chacha_state, buf, first_block_len);
len -= first_block_len;
buf += first_block_len;
while (len) {
if (len < CHACHA_BLOCK_SIZE) {
- chacha20_block(chacha_state, tmp);
+ chacha20_block(&chacha_state, tmp);
memcpy(buf, tmp, len);
memzero_explicit(tmp, sizeof(tmp));
break;
}
- chacha20_block(chacha_state, buf);
- if (unlikely(chacha_state[12] == 0))
- ++chacha_state[13];
+ chacha20_block(&chacha_state, buf);
+ if (unlikely(chacha_state.x[12] == 0))
+ ++chacha_state.x[13];
len -= CHACHA_BLOCK_SIZE;
buf += CHACHA_BLOCK_SIZE;
}
- memzero_explicit(chacha_state, sizeof(chacha_state));
+ memzero_explicit(&chacha_state, sizeof(chacha_state));
}
/*
* This returns random bytes in arbitrary quantities. The quality of the
* random bytes is good as /dev/urandom. In order to ensure that the
@@ -439,11 +439,11 @@ void get_random_bytes(void *buf, size_t len)
}
EXPORT_SYMBOL(get_random_bytes);
static ssize_t get_random_bytes_user(struct iov_iter *iter)
{
- u32 chacha_state[CHACHA_STATE_WORDS];
+ struct chacha_state chacha_state;
u8 block[CHACHA_BLOCK_SIZE];
size_t ret = 0, copied;
if (unlikely(!iov_iter_count(iter)))
return 0;
@@ -451,25 +451,26 @@ static ssize_t get_random_bytes_user(struct iov_iter *iter)
/*
* Immediately overwrite the ChaCha key at index 4 with random
* bytes, in case userspace causes copy_to_iter() below to sleep
* forever, so that we still retain forward secrecy in that case.
*/
- crng_make_state(chacha_state, (u8 *)&chacha_state[4], CHACHA_KEY_SIZE);
+ crng_make_state(&chacha_state, (u8 *)&chacha_state.x[4],
+ CHACHA_KEY_SIZE);
/*
* However, if we're doing a read of len <= 32, we don't need to
* use chacha_state after, so we can simply return those bytes to
* the user directly.
*/
if (iov_iter_count(iter) <= CHACHA_KEY_SIZE) {
- ret = copy_to_iter(&chacha_state[4], CHACHA_KEY_SIZE, iter);
+ ret = copy_to_iter(&chacha_state.x[4], CHACHA_KEY_SIZE, iter);
goto out_zero_chacha;
}
for (;;) {
- chacha20_block(chacha_state, block);
- if (unlikely(chacha_state[12] == 0))
- ++chacha_state[13];
+ chacha20_block(&chacha_state, block);
+ if (unlikely(chacha_state.x[12] == 0))
+ ++chacha_state.x[13];
copied = copy_to_iter(block, sizeof(block), iter);
ret += copied;
if (!iov_iter_count(iter) || copied != sizeof(block))
break;
@@ -482,11 +483,11 @@ static ssize_t get_random_bytes_user(struct iov_iter *iter)
}
}
memzero_explicit(block, sizeof(block));
out_zero_chacha:
- memzero_explicit(chacha_state, sizeof(chacha_state));
+ memzero_explicit(&chacha_state, sizeof(chacha_state));
return ret ? ret : -EFAULT;
}
/*
* Batched entropy returns random integers. The quality of the random
diff --git a/fs/bcachefs/checksum.c b/fs/bcachefs/checksum.c
index d0a34a097b80..312fda4bb1b5 100644
--- a/fs/bcachefs/checksum.c
+++ b/fs/bcachefs/checksum.c
@@ -89,11 +89,11 @@ static void bch2_checksum_update(struct bch2_checksum_state *state, const void *
default:
BUG();
}
}
-static void bch2_chacha20_init(u32 state[CHACHA_STATE_WORDS],
+static void bch2_chacha20_init(struct chacha_state *state,
const struct bch_key *key, struct nonce nonce)
{
u32 key_words[CHACHA_KEY_SIZE / sizeof(u32)];
BUILD_BUG_ON(sizeof(key_words) != sizeof(*key));
@@ -107,15 +107,15 @@ static void bch2_chacha20_init(u32 state[CHACHA_STATE_WORDS],
}
static void bch2_chacha20(const struct bch_key *key, struct nonce nonce,
void *data, size_t len)
{
- u32 state[CHACHA_STATE_WORDS];
+ struct chacha_state state;
- bch2_chacha20_init(state, key, nonce);
- chacha20_crypt(state, data, data, len);
- memzero_explicit(state, sizeof(state));
+ bch2_chacha20_init(&state, key, nonce);
+ chacha20_crypt(&state, data, data, len);
+ memzero_explicit(&state, sizeof(state));
}
static void bch2_poly1305_init(struct poly1305_desc_ctx *desc,
struct bch_fs *c, struct nonce nonce)
{
@@ -255,18 +255,18 @@ struct bch_csum bch2_checksum_bio(struct bch_fs *c, unsigned type,
int __bch2_encrypt_bio(struct bch_fs *c, unsigned type,
struct nonce nonce, struct bio *bio)
{
struct bio_vec bv;
struct bvec_iter iter;
- u32 chacha_state[CHACHA_STATE_WORDS];
+ struct chacha_state chacha_state;
int ret = 0;
if (bch2_fs_inconsistent_on(!c->chacha20_key_set,
c, "attempting to encrypt without encryption key"))
return -BCH_ERR_no_encryption_key;
- bch2_chacha20_init(chacha_state, &c->chacha20_key, nonce);
+ bch2_chacha20_init(&chacha_state, &c->chacha20_key, nonce);
bio_for_each_segment(bv, bio, iter) {
void *p;
/*
@@ -278,14 +278,14 @@ int __bch2_encrypt_bio(struct bch_fs *c, unsigned type,
ret = -EIO;
break;
}
p = bvec_kmap_local(&bv);
- chacha20_crypt(chacha_state, p, p, bv.bv_len);
+ chacha20_crypt(&chacha_state, p, p, bv.bv_len);
kunmap_local(p);
}
- memzero_explicit(chacha_state, sizeof(chacha_state));
+ memzero_explicit(&chacha_state, sizeof(chacha_state));
return ret;
}
struct bch_csum bch2_checksum_merge(unsigned type, struct bch_csum a,
struct bch_csum b, size_t b_len)
diff --git a/include/crypto/chacha.h b/include/crypto/chacha.h
index 58129e18cc31..64fb270f2bfc 100644
--- a/include/crypto/chacha.h
+++ b/include/crypto/chacha.h
@@ -28,20 +28,27 @@
#define CHACHA_STATE_WORDS (CHACHA_BLOCK_SIZE / sizeof(u32))
/* 192-bit nonce, then 64-bit stream position */
#define XCHACHA_IV_SIZE 32
-void chacha_block_generic(u32 *state, u8 *stream, int nrounds);
-static inline void chacha20_block(u32 *state, u8 *stream)
+struct chacha_state {
+ u32 x[CHACHA_STATE_WORDS];
+};
+
+void chacha_block_generic(struct chacha_state *state, u8 *stream, int nrounds);
+static inline void chacha20_block(struct chacha_state *state, u8 *stream)
{
chacha_block_generic(state, stream, 20);
}
-void hchacha_block_arch(const u32 *state, u32 *out, int nrounds);
-void hchacha_block_generic(const u32 *state, u32 *out, int nrounds);
+void hchacha_block_arch(const struct chacha_state *state, u32 *out,
+ int nrounds);
+void hchacha_block_generic(const struct chacha_state *state, u32 *out,
+ int nrounds);
-static inline void hchacha_block(const u32 *state, u32 *out, int nrounds)
+static inline void hchacha_block(const struct chacha_state *state, u32 *out,
+ int nrounds)
{
if (IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA))
hchacha_block_arch(state, out, nrounds);
else
hchacha_block_generic(state, out, nrounds);
@@ -52,51 +59,53 @@ enum chacha_constants { /* expand 32-byte k */
CHACHA_CONSTANT_ND_3 = 0x3320646eU,
CHACHA_CONSTANT_2_BY = 0x79622d32U,
CHACHA_CONSTANT_TE_K = 0x6b206574U
};
-static inline void chacha_init_consts(u32 *state)
+static inline void chacha_init_consts(struct chacha_state *state)
{
- state[0] = CHACHA_CONSTANT_EXPA;
- state[1] = CHACHA_CONSTANT_ND_3;
- state[2] = CHACHA_CONSTANT_2_BY;
- state[3] = CHACHA_CONSTANT_TE_K;
+ state->x[0] = CHACHA_CONSTANT_EXPA;
+ state->x[1] = CHACHA_CONSTANT_ND_3;
+ state->x[2] = CHACHA_CONSTANT_2_BY;
+ state->x[3] = CHACHA_CONSTANT_TE_K;
}
-static inline void chacha_init(u32 *state, const u32 *key, const u8 *iv)
+static inline void chacha_init(struct chacha_state *state,
+ const u32 *key, const u8 *iv)
{
chacha_init_consts(state);
- state[4] = key[0];
- state[5] = key[1];
- state[6] = key[2];
- state[7] = key[3];
- state[8] = key[4];
- state[9] = key[5];
- state[10] = key[6];
- state[11] = key[7];
- state[12] = get_unaligned_le32(iv + 0);
- state[13] = get_unaligned_le32(iv + 4);
- state[14] = get_unaligned_le32(iv + 8);
- state[15] = get_unaligned_le32(iv + 12);
+ state->x[4] = key[0];
+ state->x[5] = key[1];
+ state->x[6] = key[2];
+ state->x[7] = key[3];
+ state->x[8] = key[4];
+ state->x[9] = key[5];
+ state->x[10] = key[6];
+ state->x[11] = key[7];
+ state->x[12] = get_unaligned_le32(iv + 0);
+ state->x[13] = get_unaligned_le32(iv + 4);
+ state->x[14] = get_unaligned_le32(iv + 8);
+ state->x[15] = get_unaligned_le32(iv + 12);
}
-void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src,
+void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
unsigned int bytes, int nrounds);
-void chacha_crypt_generic(u32 *state, u8 *dst, const u8 *src,
+void chacha_crypt_generic(struct chacha_state *state, u8 *dst, const u8 *src,
unsigned int bytes, int nrounds);
-static inline void chacha_crypt(u32 *state, u8 *dst, const u8 *src,
+static inline void chacha_crypt(struct chacha_state *state,
+ u8 *dst, const u8 *src,
unsigned int bytes, int nrounds)
{
if (IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA))
chacha_crypt_arch(state, dst, src, bytes, nrounds);
else
chacha_crypt_generic(state, dst, src, bytes, nrounds);
}
-static inline void chacha20_crypt(u32 *state, u8 *dst, const u8 *src,
- unsigned int bytes)
+static inline void chacha20_crypt(struct chacha_state *state,
+ u8 *dst, const u8 *src, unsigned int bytes)
{
chacha_crypt(state, dst, src, bytes, 20);
}
#if IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA)
diff --git a/lib/crypto/chacha.c b/lib/crypto/chacha.c
index 3cdda3b5ee06..a7f5eb091839 100644
--- a/lib/crypto/chacha.c
+++ b/lib/crypto/chacha.c
@@ -11,12 +11,13 @@
#include <linux/bitops.h>
#include <linux/string.h>
#include <linux/unaligned.h>
#include <crypto/chacha.h>
-static void chacha_permute(u32 *x, int nrounds)
+static void chacha_permute(struct chacha_state *state, int nrounds)
{
+ u32 *x = state->x;
int i;
/* whitelist the allowed round counts */
WARN_ON_ONCE(nrounds != 20 && nrounds != 12);
@@ -63,52 +64,54 @@ static void chacha_permute(u32 *x, int nrounds)
}
}
/**
* chacha_block_generic - generate one keystream block and increment block counter
- * @state: input state matrix (16 32-bit words)
+ * @state: input state matrix
* @stream: output keystream block (64 bytes)
* @nrounds: number of rounds (20 or 12; 20 is recommended)
*
* This is the ChaCha core, a function from 64-byte strings to 64-byte strings.
* The caller has already converted the endianness of the input. This function
* also handles incrementing the block counter in the input matrix.
*/
-void chacha_block_generic(u32 *state, u8 *stream, int nrounds)
+void chacha_block_generic(struct chacha_state *state, u8 *stream, int nrounds)
{
- u32 x[16];
+ struct chacha_state permuted_state;
int i;
- memcpy(x, state, 64);
+ memcpy(permuted_state.x, state->x, 64);
- chacha_permute(x, nrounds);
+ chacha_permute(&permuted_state, nrounds);
- for (i = 0; i < ARRAY_SIZE(x); i++)
- put_unaligned_le32(x[i] + state[i], &stream[i * sizeof(u32)]);
+ for (i = 0; i < ARRAY_SIZE(state->x); i++)
+ put_unaligned_le32(permuted_state.x[i] + state->x[i],
+ &stream[i * sizeof(u32)]);
- state[12]++;
+ state->x[12]++;
}
EXPORT_SYMBOL(chacha_block_generic);
/**
* hchacha_block_generic - abbreviated ChaCha core, for XChaCha
- * @state: input state matrix (16 32-bit words)
+ * @state: input state matrix
* @stream: output (8 32-bit words)
* @nrounds: number of rounds (20 or 12; 20 is recommended)
*
* HChaCha is the ChaCha equivalent of HSalsa and is an intermediate step
* towards XChaCha (see https://cr.yp.to/snuffle/xsalsa-20081128.pdf). HChaCha
* skips the final addition of the initial state, and outputs only certain words
* of the state. It should not be used for streaming directly.
*/
-void hchacha_block_generic(const u32 *state, u32 *stream, int nrounds)
+void hchacha_block_generic(const struct chacha_state *state,
+ u32 *stream, int nrounds)
{
- u32 x[16];
+ struct chacha_state permuted_state;
- memcpy(x, state, 64);
+ memcpy(permuted_state.x, state->x, 64);
- chacha_permute(x, nrounds);
+ chacha_permute(&permuted_state, nrounds);
- memcpy(&stream[0], &x[0], 16);
- memcpy(&stream[4], &x[12], 16);
+ memcpy(&stream[0], &permuted_state.x[0], 16);
+ memcpy(&stream[4], &permuted_state.x[12], 16);
}
EXPORT_SYMBOL(hchacha_block_generic);
diff --git a/lib/crypto/chacha20poly1305-selftest.c b/lib/crypto/chacha20poly1305-selftest.c
index 2ea61c28be4f..e4c85bc5a6d7 100644
--- a/lib/crypto/chacha20poly1305-selftest.c
+++ b/lib/crypto/chacha20poly1305-selftest.c
@@ -8830,11 +8830,11 @@ chacha20poly1305_encrypt_bignonce(u8 *dst, const u8 *src, const size_t src_len,
const u8 nonce[12],
const u8 key[CHACHA20POLY1305_KEY_SIZE])
{
const u8 *pad0 = page_address(ZERO_PAGE(0));
struct poly1305_desc_ctx poly1305_state;
- u32 chacha20_state[CHACHA_STATE_WORDS];
+ struct chacha_state chacha20_state;
union {
u8 block0[POLY1305_KEY_SIZE];
__le64 lens[2];
} b = {{ 0 }};
u8 bottom_row[16] = { 0 };
@@ -8842,16 +8842,16 @@ chacha20poly1305_encrypt_bignonce(u8 *dst, const u8 *src, const size_t src_len,
int i;
memcpy(&bottom_row[4], nonce, 12);
for (i = 0; i < 8; ++i)
le_key[i] = get_unaligned_le32(key + sizeof(le_key[i]) * i);
- chacha_init(chacha20_state, le_key, bottom_row);
- chacha20_crypt(chacha20_state, b.block0, b.block0, sizeof(b.block0));
+ chacha_init(&chacha20_state, le_key, bottom_row);
+ chacha20_crypt(&chacha20_state, b.block0, b.block0, sizeof(b.block0));
poly1305_init(&poly1305_state, b.block0);
poly1305_update(&poly1305_state, ad, ad_len);
poly1305_update(&poly1305_state, pad0, (0x10 - ad_len) & 0xf);
- chacha20_crypt(chacha20_state, dst, src, src_len);
+ chacha20_crypt(&chacha20_state, dst, src, src_len);
poly1305_update(&poly1305_state, dst, src_len);
poly1305_update(&poly1305_state, pad0, (0x10 - src_len) & 0xf);
b.lens[0] = cpu_to_le64(ad_len);
b.lens[1] = cpu_to_le64(src_len);
poly1305_update(&poly1305_state, (u8 *)b.lens, sizeof(b.lens));
diff --git a/lib/crypto/chacha20poly1305.c b/lib/crypto/chacha20poly1305.c
index 9cfa886f1f89..ed81f0658956 100644
--- a/lib/crypto/chacha20poly1305.c
+++ b/lib/crypto/chacha20poly1305.c
@@ -30,11 +30,12 @@ static void chacha_load_key(u32 *k, const u8 *in)
k[5] = get_unaligned_le32(in + 20);
k[6] = get_unaligned_le32(in + 24);
k[7] = get_unaligned_le32(in + 28);
}
-static void xchacha_init(u32 *chacha_state, const u8 *key, const u8 *nonce)
+static void xchacha_init(struct chacha_state *chacha_state,
+ const u8 *key, const u8 *nonce)
{
u32 k[CHACHA_KEY_WORDS];
u8 iv[CHACHA_IV_SIZE];
memset(iv, 0, 8);
@@ -52,11 +53,12 @@ static void xchacha_init(u32 *chacha_state, const u8 *key, const u8 *nonce)
memzero_explicit(iv, sizeof(iv));
}
static void
__chacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
- const u8 *ad, const size_t ad_len, u32 *chacha_state)
+ const u8 *ad, const size_t ad_len,
+ struct chacha_state *chacha_state)
{
const u8 *pad0 = page_address(ZERO_PAGE(0));
struct poly1305_desc_ctx poly1305_state;
union {
u8 block0[POLY1305_KEY_SIZE];
@@ -80,30 +82,31 @@ __chacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
b.lens[1] = cpu_to_le64(src_len);
poly1305_update(&poly1305_state, (u8 *)b.lens, sizeof(b.lens));
poly1305_final(&poly1305_state, dst + src_len);
- memzero_explicit(chacha_state, CHACHA_STATE_WORDS * sizeof(u32));
+ memzero_explicit(chacha_state, sizeof(*chacha_state));
memzero_explicit(&b, sizeof(b));
}
void chacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
const u8 *ad, const size_t ad_len,
const u64 nonce,
const u8 key[CHACHA20POLY1305_KEY_SIZE])
{
- u32 chacha_state[CHACHA_STATE_WORDS];
+ struct chacha_state chacha_state;
u32 k[CHACHA_KEY_WORDS];
__le64 iv[2];
chacha_load_key(k, key);
iv[0] = 0;
iv[1] = cpu_to_le64(nonce);
- chacha_init(chacha_state, k, (u8 *)iv);
- __chacha20poly1305_encrypt(dst, src, src_len, ad, ad_len, chacha_state);
+ chacha_init(&chacha_state, k, (u8 *)iv);
+ __chacha20poly1305_encrypt(dst, src, src_len, ad, ad_len,
+ &chacha_state);
memzero_explicit(iv, sizeof(iv));
memzero_explicit(k, sizeof(k));
}
EXPORT_SYMBOL(chacha20poly1305_encrypt);
@@ -111,20 +114,22 @@ EXPORT_SYMBOL(chacha20poly1305_encrypt);
void xchacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
const u8 *ad, const size_t ad_len,
const u8 nonce[XCHACHA20POLY1305_NONCE_SIZE],
const u8 key[CHACHA20POLY1305_KEY_SIZE])
{
- u32 chacha_state[CHACHA_STATE_WORDS];
+ struct chacha_state chacha_state;
- xchacha_init(chacha_state, key, nonce);
- __chacha20poly1305_encrypt(dst, src, src_len, ad, ad_len, chacha_state);
+ xchacha_init(&chacha_state, key, nonce);
+ __chacha20poly1305_encrypt(dst, src, src_len, ad, ad_len,
+ &chacha_state);
}
EXPORT_SYMBOL(xchacha20poly1305_encrypt);
static bool
__chacha20poly1305_decrypt(u8 *dst, const u8 *src, const size_t src_len,
- const u8 *ad, const size_t ad_len, u32 *chacha_state)
+ const u8 *ad, const size_t ad_len,
+ struct chacha_state *chacha_state)
{
const u8 *pad0 = page_address(ZERO_PAGE(0));
struct poly1305_desc_ctx poly1305_state;
size_t dst_len;
int ret;
@@ -167,25 +172,25 @@ __chacha20poly1305_decrypt(u8 *dst, const u8 *src, const size_t src_len,
bool chacha20poly1305_decrypt(u8 *dst, const u8 *src, const size_t src_len,
const u8 *ad, const size_t ad_len,
const u64 nonce,
const u8 key[CHACHA20POLY1305_KEY_SIZE])
{
- u32 chacha_state[CHACHA_STATE_WORDS];
+ struct chacha_state chacha_state;
u32 k[CHACHA_KEY_WORDS];
__le64 iv[2];
bool ret;
chacha_load_key(k, key);
iv[0] = 0;
iv[1] = cpu_to_le64(nonce);
- chacha_init(chacha_state, k, (u8 *)iv);
+ chacha_init(&chacha_state, k, (u8 *)iv);
ret = __chacha20poly1305_decrypt(dst, src, src_len, ad, ad_len,
- chacha_state);
+ &chacha_state);
- memzero_explicit(chacha_state, sizeof(chacha_state));
+ memzero_explicit(&chacha_state, sizeof(chacha_state));
memzero_explicit(iv, sizeof(iv));
memzero_explicit(k, sizeof(k));
return ret;
}
EXPORT_SYMBOL(chacha20poly1305_decrypt);
@@ -193,15 +198,15 @@ EXPORT_SYMBOL(chacha20poly1305_decrypt);
bool xchacha20poly1305_decrypt(u8 *dst, const u8 *src, const size_t src_len,
const u8 *ad, const size_t ad_len,
const u8 nonce[XCHACHA20POLY1305_NONCE_SIZE],
const u8 key[CHACHA20POLY1305_KEY_SIZE])
{
- u32 chacha_state[CHACHA_STATE_WORDS];
+ struct chacha_state chacha_state;
- xchacha_init(chacha_state, key, nonce);
+ xchacha_init(&chacha_state, key, nonce);
return __chacha20poly1305_decrypt(dst, src, src_len, ad, ad_len,
- chacha_state);
+ &chacha_state);
}
EXPORT_SYMBOL(xchacha20poly1305_decrypt);
static
bool chacha20poly1305_crypt_sg_inplace(struct scatterlist *src,
@@ -211,11 +216,11 @@ bool chacha20poly1305_crypt_sg_inplace(struct scatterlist *src,
const u8 key[CHACHA20POLY1305_KEY_SIZE],
int encrypt)
{
const u8 *pad0 = page_address(ZERO_PAGE(0));
struct poly1305_desc_ctx poly1305_state;
- u32 chacha_state[CHACHA_STATE_WORDS];
+ struct chacha_state chacha_state;
struct sg_mapping_iter miter;
size_t partial = 0;
unsigned int flags;
bool ret = true;
int sl;
@@ -238,12 +243,12 @@ bool chacha20poly1305_crypt_sg_inplace(struct scatterlist *src,
chacha_load_key(b.k, key);
b.iv[0] = 0;
b.iv[1] = cpu_to_le64(nonce);
- chacha_init(chacha_state, b.k, (u8 *)b.iv);
- chacha20_crypt(chacha_state, b.block0, pad0, sizeof(b.block0));
+ chacha_init(&chacha_state, b.k, (u8 *)b.iv);
+ chacha20_crypt(&chacha_state, b.block0, pad0, sizeof(b.block0));
poly1305_init(&poly1305_state, b.block0);
if (unlikely(ad_len)) {
poly1305_update(&poly1305_state, ad, ad_len);
if (ad_len & 0xf)
@@ -274,17 +279,17 @@ bool chacha20poly1305_crypt_sg_inplace(struct scatterlist *src,
if (likely(length >= CHACHA_BLOCK_SIZE || length == sl)) {
size_t l = length;
if (unlikely(length < sl))
l &= ~(CHACHA_BLOCK_SIZE - 1);
- chacha20_crypt(chacha_state, addr, addr, l);
+ chacha20_crypt(&chacha_state, addr, addr, l);
addr += l;
length -= l;
}
if (unlikely(length > 0)) {
- chacha20_crypt(chacha_state, b.chacha_stream, pad0,
+ chacha20_crypt(&chacha_state, b.chacha_stream, pad0,
CHACHA_BLOCK_SIZE);
crypto_xor(addr, b.chacha_stream, length);
partial = length;
}
@@ -321,11 +326,11 @@ bool chacha20poly1305_crypt_sg_inplace(struct scatterlist *src,
sizeof(b.mac[1]), src_len, !encrypt);
ret = encrypt ||
!crypto_memneq(b.mac[0], b.mac[1], POLY1305_DIGEST_SIZE);
}
- memzero_explicit(chacha_state, sizeof(chacha_state));
+ memzero_explicit(&chacha_state, sizeof(chacha_state));
memzero_explicit(&b, sizeof(b));
return ret;
}
diff --git a/lib/crypto/libchacha.c b/lib/crypto/libchacha.c
index cc1be0496eb9..ebcca381e248 100644
--- a/lib/crypto/libchacha.c
+++ b/lib/crypto/libchacha.c
@@ -10,11 +10,11 @@
#include <linux/module.h>
#include <crypto/algapi.h> // for crypto_xor_cpy
#include <crypto/chacha.h>
-void chacha_crypt_generic(u32 *state, u8 *dst, const u8 *src,
+void chacha_crypt_generic(struct chacha_state *state, u8 *dst, const u8 *src,
unsigned int bytes, int nrounds)
{
/* aligned to potentially speed up crypto_xor() */
u8 stream[CHACHA_BLOCK_SIZE] __aligned(sizeof(long));
diff --git a/tools/testing/crypto/chacha20-s390/test-cipher.c b/tools/testing/crypto/chacha20-s390/test-cipher.c
index 35ea65c54ffa..827507844e8f 100644
--- a/tools/testing/crypto/chacha20-s390/test-cipher.c
+++ b/tools/testing/crypto/chacha20-s390/test-cipher.c
@@ -48,11 +48,11 @@ struct skcipher_def {
};
/* Perform cipher operations with the chacha lib */
static int test_lib_chacha(u8 *revert, u8 *cipher, u8 *plain)
{
- u32 chacha_state[CHACHA_STATE_WORDS];
+ struct chacha_state chacha_state;
u8 iv[16], key[32];
u64 start, end;
memset(key, 'X', sizeof(key));
memset(iv, 'I', sizeof(iv));
@@ -64,14 +64,14 @@ static int test_lib_chacha(u8 *revert, u8 *cipher, u8 *plain)
print_hex_dump(KERN_INFO, "iv: ", DUMP_PREFIX_OFFSET,
16, 1, iv, 16, 1);
}
/* Encrypt */
- chacha_init(chacha_state, (u32 *)key, iv);
+ chacha_init(&chacha_state, (u32 *)key, iv);
start = ktime_get_ns();
- chacha_crypt_arch(chacha_state, cipher, plain, data_size, 20);
+ chacha_crypt_arch(&chacha_state, cipher, plain, data_size, 20);
end = ktime_get_ns();
if (debug)
print_hex_dump(KERN_INFO, "encr:", DUMP_PREFIX_OFFSET,
@@ -79,14 +79,14 @@ static int test_lib_chacha(u8 *revert, u8 *cipher, u8 *plain)
(data_size > 64 ? 64 : data_size), 1);
pr_info("lib encryption took: %lld nsec", end - start);
/* Decrypt */
- chacha_init(chacha_state, (u32 *)key, iv);
+ chacha_init(&chacha_state, (u32 *)key, iv);
start = ktime_get_ns();
- chacha_crypt_arch(chacha_state, revert, cipher, data_size, 20);
+ chacha_crypt_arch(&chacha_state, revert, cipher, data_size, 20);
end = ktime_get_ns();
if (debug)
print_hex_dump(KERN_INFO, "decr:", DUMP_PREFIX_OFFSET,
16, 1, revert,
--
2.49.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 2/4] crypto: lib/chacha - use struct assignment to copy state
2025-05-05 18:18 [PATCH 0/4] crypto: lib/chacha - improve type safety Eric Biggers
2025-05-05 18:18 ` [PATCH 1/4] crypto: lib/chacha - strongly type the ChaCha state Eric Biggers
@ 2025-05-05 18:18 ` Eric Biggers
2025-05-05 18:18 ` [PATCH 3/4] crypto: lib/chacha - add strongly-typed state zeroization Eric Biggers
` (2 subsequent siblings)
4 siblings, 0 replies; 7+ messages in thread
From: Eric Biggers @ 2025-05-05 18:18 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arm-kernel, linux-mips, linuxppc-dev,
linux-riscv, linux-s390, linux-bcachefs, Jason A . Donenfeld ,
Theodore Ts'o
From: Eric Biggers <ebiggers@google.com>
Use struct assignment instead of memcpy() in lib/crypto/chacha.c where
appropriate. No functional change.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
lib/crypto/chacha.c | 8 ++------
1 file changed, 2 insertions(+), 6 deletions(-)
diff --git a/lib/crypto/chacha.c b/lib/crypto/chacha.c
index a7f5eb091839..ae50e441f9fb 100644
--- a/lib/crypto/chacha.c
+++ b/lib/crypto/chacha.c
@@ -74,15 +74,13 @@ static void chacha_permute(struct chacha_state *state, int nrounds)
* The caller has already converted the endianness of the input. This function
* also handles incrementing the block counter in the input matrix.
*/
void chacha_block_generic(struct chacha_state *state, u8 *stream, int nrounds)
{
- struct chacha_state permuted_state;
+ struct chacha_state permuted_state = *state;
int i;
- memcpy(permuted_state.x, state->x, 64);
-
chacha_permute(&permuted_state, nrounds);
for (i = 0; i < ARRAY_SIZE(state->x); i++)
put_unaligned_le32(permuted_state.x[i] + state->x[i],
&stream[i * sizeof(u32)]);
@@ -103,13 +101,11 @@ EXPORT_SYMBOL(chacha_block_generic);
* of the state. It should not be used for streaming directly.
*/
void hchacha_block_generic(const struct chacha_state *state,
u32 *stream, int nrounds)
{
- struct chacha_state permuted_state;
-
- memcpy(permuted_state.x, state->x, 64);
+ struct chacha_state permuted_state = *state;
chacha_permute(&permuted_state, nrounds);
memcpy(&stream[0], &permuted_state.x[0], 16);
memcpy(&stream[4], &permuted_state.x[12], 16);
--
2.49.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 3/4] crypto: lib/chacha - add strongly-typed state zeroization
2025-05-05 18:18 [PATCH 0/4] crypto: lib/chacha - improve type safety Eric Biggers
2025-05-05 18:18 ` [PATCH 1/4] crypto: lib/chacha - strongly type the ChaCha state Eric Biggers
2025-05-05 18:18 ` [PATCH 2/4] crypto: lib/chacha - use struct assignment to copy state Eric Biggers
@ 2025-05-05 18:18 ` Eric Biggers
2025-05-05 18:18 ` [PATCH 4/4] crypto: lib/chacha - add array bounds to function prototypes Eric Biggers
2025-05-12 5:45 ` [PATCH 0/4] crypto: lib/chacha - improve type safety Herbert Xu
4 siblings, 0 replies; 7+ messages in thread
From: Eric Biggers @ 2025-05-05 18:18 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arm-kernel, linux-mips, linuxppc-dev,
linux-riscv, linux-s390, linux-bcachefs, Jason A . Donenfeld ,
Theodore Ts'o
From: Eric Biggers <ebiggers@google.com>
Now that the ChaCha state matrix is strongly-typed, add a helper
function chacha_zeroize_state() which zeroizes it. Then convert all
applicable callers to use it instead of direct memzero_explicit. No
functional changes.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
drivers/char/random.c | 4 ++--
fs/bcachefs/checksum.c | 4 ++--
include/crypto/chacha.h | 6 ++++++
lib/crypto/chacha20poly1305.c | 6 +++---
4 files changed, 13 insertions(+), 7 deletions(-)
diff --git a/drivers/char/random.c b/drivers/char/random.c
index 9f876ed2655b..5f22a08101f6 100644
--- a/drivers/char/random.c
+++ b/drivers/char/random.c
@@ -420,11 +420,11 @@ static void _get_random_bytes(void *buf, size_t len)
++chacha_state.x[13];
len -= CHACHA_BLOCK_SIZE;
buf += CHACHA_BLOCK_SIZE;
}
- memzero_explicit(&chacha_state, sizeof(chacha_state));
+ chacha_zeroize_state(&chacha_state);
}
/*
* This returns random bytes in arbitrary quantities. The quality of the
* random bytes is good as /dev/urandom. In order to ensure that the
@@ -483,11 +483,11 @@ static ssize_t get_random_bytes_user(struct iov_iter *iter)
}
}
memzero_explicit(block, sizeof(block));
out_zero_chacha:
- memzero_explicit(&chacha_state, sizeof(chacha_state));
+ chacha_zeroize_state(&chacha_state);
return ret ? ret : -EFAULT;
}
/*
* Batched entropy returns random integers. The quality of the random
diff --git a/fs/bcachefs/checksum.c b/fs/bcachefs/checksum.c
index 312fda4bb1b5..a4df8eba75f3 100644
--- a/fs/bcachefs/checksum.c
+++ b/fs/bcachefs/checksum.c
@@ -111,11 +111,11 @@ static void bch2_chacha20(const struct bch_key *key, struct nonce nonce,
{
struct chacha_state state;
bch2_chacha20_init(&state, key, nonce);
chacha20_crypt(&state, data, data, len);
- memzero_explicit(&state, sizeof(state));
+ chacha_zeroize_state(&state);
}
static void bch2_poly1305_init(struct poly1305_desc_ctx *desc,
struct bch_fs *c, struct nonce nonce)
{
@@ -281,11 +281,11 @@ int __bch2_encrypt_bio(struct bch_fs *c, unsigned type,
p = bvec_kmap_local(&bv);
chacha20_crypt(&chacha_state, p, p, bv.bv_len);
kunmap_local(p);
}
- memzero_explicit(&chacha_state, sizeof(chacha_state));
+ chacha_zeroize_state(&chacha_state);
return ret;
}
struct bch_csum bch2_checksum_merge(unsigned type, struct bch_csum a,
struct bch_csum b, size_t b_len)
diff --git a/include/crypto/chacha.h b/include/crypto/chacha.h
index 64fb270f2bfc..7c2e6c68919b 100644
--- a/include/crypto/chacha.h
+++ b/include/crypto/chacha.h
@@ -14,10 +14,11 @@
#ifndef _CRYPTO_CHACHA_H
#define _CRYPTO_CHACHA_H
#include <linux/unaligned.h>
+#include <linux/string.h>
#include <linux/types.h>
/* 32-bit stream position, then 96-bit nonce (RFC7539 convention) */
#define CHACHA_IV_SIZE 16
@@ -106,10 +107,15 @@ static inline void chacha20_crypt(struct chacha_state *state,
u8 *dst, const u8 *src, unsigned int bytes)
{
chacha_crypt(state, dst, src, bytes, 20);
}
+static inline void chacha_zeroize_state(struct chacha_state *state)
+{
+ memzero_explicit(state, sizeof(*state));
+}
+
#if IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA)
bool chacha_is_arch_optimized(void);
#else
static inline bool chacha_is_arch_optimized(void)
{
diff --git a/lib/crypto/chacha20poly1305.c b/lib/crypto/chacha20poly1305.c
index ed81f0658956..2e7bbc1a67ea 100644
--- a/lib/crypto/chacha20poly1305.c
+++ b/lib/crypto/chacha20poly1305.c
@@ -82,11 +82,11 @@ __chacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
b.lens[1] = cpu_to_le64(src_len);
poly1305_update(&poly1305_state, (u8 *)b.lens, sizeof(b.lens));
poly1305_final(&poly1305_state, dst + src_len);
- memzero_explicit(chacha_state, sizeof(*chacha_state));
+ chacha_zeroize_state(chacha_state);
memzero_explicit(&b, sizeof(b));
}
void chacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
const u8 *ad, const size_t ad_len,
@@ -186,11 +186,11 @@ bool chacha20poly1305_decrypt(u8 *dst, const u8 *src, const size_t src_len,
chacha_init(&chacha_state, k, (u8 *)iv);
ret = __chacha20poly1305_decrypt(dst, src, src_len, ad, ad_len,
&chacha_state);
- memzero_explicit(&chacha_state, sizeof(chacha_state));
+ chacha_zeroize_state(&chacha_state);
memzero_explicit(iv, sizeof(iv));
memzero_explicit(k, sizeof(k));
return ret;
}
EXPORT_SYMBOL(chacha20poly1305_decrypt);
@@ -326,11 +326,11 @@ bool chacha20poly1305_crypt_sg_inplace(struct scatterlist *src,
sizeof(b.mac[1]), src_len, !encrypt);
ret = encrypt ||
!crypto_memneq(b.mac[0], b.mac[1], POLY1305_DIGEST_SIZE);
}
- memzero_explicit(&chacha_state, sizeof(chacha_state));
+ chacha_zeroize_state(&chacha_state);
memzero_explicit(&b, sizeof(b));
return ret;
}
--
2.49.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* [PATCH 4/4] crypto: lib/chacha - add array bounds to function prototypes
2025-05-05 18:18 [PATCH 0/4] crypto: lib/chacha - improve type safety Eric Biggers
` (2 preceding siblings ...)
2025-05-05 18:18 ` [PATCH 3/4] crypto: lib/chacha - add strongly-typed state zeroization Eric Biggers
@ 2025-05-05 18:18 ` Eric Biggers
2025-05-12 5:45 ` [PATCH 0/4] crypto: lib/chacha - improve type safety Herbert Xu
4 siblings, 0 replies; 7+ messages in thread
From: Eric Biggers @ 2025-05-05 18:18 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arm-kernel, linux-mips, linuxppc-dev,
linux-riscv, linux-s390, linux-bcachefs, Jason A . Donenfeld ,
Theodore Ts'o
From: Eric Biggers <ebiggers@google.com>
Add explicit array bounds to the function prototypes for the parameters
that didn't already get handled by the conversion to use chacha_state:
- chacha_block_*():
Change 'u8 *out' or 'u8 *stream' to u8 out[CHACHA_BLOCK_SIZE].
- hchacha_block_*():
Change 'u32 *out' or 'u32 *stream' to u32 out[HCHACHA_OUT_WORDS].
- chacha_init():
Change 'const u32 *key' to 'const u32 key[CHACHA_KEY_WORDS]'.
Change 'const u8 *iv' to 'const u8 iv[CHACHA_IV_SIZE]'.
No functional changes. This just makes it clear when fixed-size arrays
are expected.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/arm/lib/crypto/chacha-glue.c | 12 ++++-----
arch/arm/lib/crypto/chacha-scalar-core.S | 2 +-
arch/arm64/lib/crypto/chacha-neon-glue.c | 10 ++++----
arch/mips/lib/crypto/chacha-glue.c | 2 +-
arch/powerpc/lib/crypto/chacha-p10-glue.c | 4 +--
arch/riscv/lib/crypto/chacha-riscv64-glue.c | 3 ++-
arch/s390/lib/crypto/chacha-glue.c | 4 +--
arch/x86/lib/crypto/chacha_glue.c | 8 +++---
crypto/chacha.c | 4 +--
include/crypto/chacha.h | 27 ++++++++++++---------
lib/crypto/chacha.c | 15 ++++++------
lib/crypto/chacha20poly1305.c | 2 --
12 files changed, 49 insertions(+), 44 deletions(-)
diff --git a/arch/arm/lib/crypto/chacha-glue.c b/arch/arm/lib/crypto/chacha-glue.c
index 0c2b4c62d484..88ec96415283 100644
--- a/arch/arm/lib/crypto/chacha-glue.c
+++ b/arch/arm/lib/crypto/chacha-glue.c
@@ -21,13 +21,13 @@ asmlinkage void chacha_block_xor_neon(const struct chacha_state *state,
u8 *dst, const u8 *src, int nrounds);
asmlinkage void chacha_4block_xor_neon(const struct chacha_state *state,
u8 *dst, const u8 *src,
int nrounds, unsigned int nbytes);
asmlinkage void hchacha_block_arm(const struct chacha_state *state,
- u32 *out, int nrounds);
+ u32 out[HCHACHA_OUT_WORDS], int nrounds);
asmlinkage void hchacha_block_neon(const struct chacha_state *state,
- u32 *out, int nrounds);
+ u32 out[HCHACHA_OUT_WORDS], int nrounds);
asmlinkage void chacha_doarm(u8 *dst, const u8 *src, unsigned int bytes,
const struct chacha_state *state, int nrounds);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(use_neon);
@@ -62,18 +62,18 @@ static void chacha_doneon(struct chacha_state *state, u8 *dst, const u8 *src,
memcpy(dst, buf, bytes);
state->x[12]++;
}
}
-void hchacha_block_arch(const struct chacha_state *state, u32 *stream,
- int nrounds)
+void hchacha_block_arch(const struct chacha_state *state,
+ u32 out[HCHACHA_OUT_WORDS], int nrounds)
{
if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON) || !neon_usable()) {
- hchacha_block_arm(state, stream, nrounds);
+ hchacha_block_arm(state, out, nrounds);
} else {
kernel_neon_begin();
- hchacha_block_neon(state, stream, nrounds);
+ hchacha_block_neon(state, out, nrounds);
kernel_neon_end();
}
}
EXPORT_SYMBOL(hchacha_block_arch);
diff --git a/arch/arm/lib/crypto/chacha-scalar-core.S b/arch/arm/lib/crypto/chacha-scalar-core.S
index d20b5de755cc..4951df05c158 100644
--- a/arch/arm/lib/crypto/chacha-scalar-core.S
+++ b/arch/arm/lib/crypto/chacha-scalar-core.S
@@ -406,11 +406,11 @@ ENTRY(chacha_doarm)
b 0b
ENDPROC(chacha_doarm)
/*
* void hchacha_block_arm(const struct chacha_state *state,
- * u32 out[8], int nrounds);
+ * u32 out[HCHACHA_OUT_WORDS], int nrounds);
*/
ENTRY(hchacha_block_arm)
push {r1,r4-r11,lr}
cmp r2, #12 // ChaCha12 ?
diff --git a/arch/arm64/lib/crypto/chacha-neon-glue.c b/arch/arm64/lib/crypto/chacha-neon-glue.c
index 7b451b3c7240..d0188f974ca5 100644
--- a/arch/arm64/lib/crypto/chacha-neon-glue.c
+++ b/arch/arm64/lib/crypto/chacha-neon-glue.c
@@ -32,11 +32,11 @@ asmlinkage void chacha_block_xor_neon(const struct chacha_state *state,
u8 *dst, const u8 *src, int nrounds);
asmlinkage void chacha_4block_xor_neon(const struct chacha_state *state,
u8 *dst, const u8 *src,
int nrounds, int bytes);
asmlinkage void hchacha_block_neon(const struct chacha_state *state,
- u32 *out, int nrounds);
+ u32 out[HCHACHA_OUT_WORDS], int nrounds);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon);
static void chacha_doneon(struct chacha_state *state, u8 *dst, const u8 *src,
int bytes, int nrounds)
@@ -59,18 +59,18 @@ static void chacha_doneon(struct chacha_state *state, u8 *dst, const u8 *src,
dst += l;
state->x[12] += DIV_ROUND_UP(l, CHACHA_BLOCK_SIZE);
}
}
-void hchacha_block_arch(const struct chacha_state *state, u32 *stream,
- int nrounds)
+void hchacha_block_arch(const struct chacha_state *state,
+ u32 out[HCHACHA_OUT_WORDS], int nrounds)
{
if (!static_branch_likely(&have_neon) || !crypto_simd_usable()) {
- hchacha_block_generic(state, stream, nrounds);
+ hchacha_block_generic(state, out, nrounds);
} else {
kernel_neon_begin();
- hchacha_block_neon(state, stream, nrounds);
+ hchacha_block_neon(state, out, nrounds);
kernel_neon_end();
}
}
EXPORT_SYMBOL(hchacha_block_arch);
diff --git a/arch/mips/lib/crypto/chacha-glue.c b/arch/mips/lib/crypto/chacha-glue.c
index 75df4040cded..88c097594eb0 100644
--- a/arch/mips/lib/crypto/chacha-glue.c
+++ b/arch/mips/lib/crypto/chacha-glue.c
@@ -13,11 +13,11 @@ asmlinkage void chacha_crypt_arch(struct chacha_state *state,
u8 *dst, const u8 *src,
unsigned int bytes, int nrounds);
EXPORT_SYMBOL(chacha_crypt_arch);
asmlinkage void hchacha_block_arch(const struct chacha_state *state,
- u32 *stream, int nrounds);
+ u32 out[HCHACHA_OUT_WORDS], int nrounds);
EXPORT_SYMBOL(hchacha_block_arch);
bool chacha_is_arch_optimized(void)
{
return true;
diff --git a/arch/powerpc/lib/crypto/chacha-p10-glue.c b/arch/powerpc/lib/crypto/chacha-p10-glue.c
index a6e6a8da1b8b..fcd23c6f1590 100644
--- a/arch/powerpc/lib/crypto/chacha-p10-glue.c
+++ b/arch/powerpc/lib/crypto/chacha-p10-glue.c
@@ -47,13 +47,13 @@ static void chacha_p10_do_8x(struct chacha_state *state, u8 *dst, const u8 *src,
if (bytes > 0)
chacha_crypt_generic(state, dst, src, bytes, nrounds);
}
void hchacha_block_arch(const struct chacha_state *state,
- u32 *stream, int nrounds)
+ u32 out[HCHACHA_OUT_WORDS], int nrounds)
{
- hchacha_block_generic(state, stream, nrounds);
+ hchacha_block_generic(state, out, nrounds);
}
EXPORT_SYMBOL(hchacha_block_arch);
void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
unsigned int bytes, int nrounds)
diff --git a/arch/riscv/lib/crypto/chacha-riscv64-glue.c b/arch/riscv/lib/crypto/chacha-riscv64-glue.c
index 57541621981e..8c3f11d79be3 100644
--- a/arch/riscv/lib/crypto/chacha-riscv64-glue.c
+++ b/arch/riscv/lib/crypto/chacha-riscv64-glue.c
@@ -16,11 +16,12 @@
static __ro_after_init DEFINE_STATIC_KEY_FALSE(use_zvkb);
asmlinkage void chacha_zvkb(struct chacha_state *state, const u8 *in, u8 *out,
size_t nblocks, int nrounds);
-void hchacha_block_arch(const struct chacha_state *state, u32 *out, int nrounds)
+void hchacha_block_arch(const struct chacha_state *state,
+ u32 out[HCHACHA_OUT_WORDS], int nrounds)
{
hchacha_block_generic(state, out, nrounds);
}
EXPORT_SYMBOL(hchacha_block_arch);
diff --git a/arch/s390/lib/crypto/chacha-glue.c b/arch/s390/lib/crypto/chacha-glue.c
index 0a9fd50c1bd8..f95ba3483bbc 100644
--- a/arch/s390/lib/crypto/chacha-glue.c
+++ b/arch/s390/lib/crypto/chacha-glue.c
@@ -15,14 +15,14 @@
#include <linux/sizes.h>
#include <asm/fpu.h>
#include "chacha-s390.h"
void hchacha_block_arch(const struct chacha_state *state,
- u32 *stream, int nrounds)
+ u32 out[HCHACHA_OUT_WORDS], int nrounds)
{
/* TODO: implement hchacha_block_arch() in assembly */
- hchacha_block_generic(state, stream, nrounds);
+ hchacha_block_generic(state, out, nrounds);
}
EXPORT_SYMBOL(hchacha_block_arch);
void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
unsigned int bytes, int nrounds)
diff --git a/arch/x86/lib/crypto/chacha_glue.c b/arch/x86/lib/crypto/chacha_glue.c
index 6f00a56e3e9a..10b2c945f541 100644
--- a/arch/x86/lib/crypto/chacha_glue.c
+++ b/arch/x86/lib/crypto/chacha_glue.c
@@ -17,11 +17,11 @@ asmlinkage void chacha_block_xor_ssse3(const struct chacha_state *state,
unsigned int len, int nrounds);
asmlinkage void chacha_4block_xor_ssse3(const struct chacha_state *state,
u8 *dst, const u8 *src,
unsigned int len, int nrounds);
asmlinkage void hchacha_block_ssse3(const struct chacha_state *state,
- u32 *out, int nrounds);
+ u32 out[HCHACHA_OUT_WORDS], int nrounds);
asmlinkage void chacha_2block_xor_avx2(const struct chacha_state *state,
u8 *dst, const u8 *src,
unsigned int len, int nrounds);
asmlinkage void chacha_4block_xor_avx2(const struct chacha_state *state,
@@ -125,17 +125,17 @@ static void chacha_dosimd(struct chacha_state *state, u8 *dst, const u8 *src,
state->x[12]++;
}
}
void hchacha_block_arch(const struct chacha_state *state,
- u32 *stream, int nrounds)
+ u32 out[HCHACHA_OUT_WORDS], int nrounds)
{
if (!static_branch_likely(&chacha_use_simd)) {
- hchacha_block_generic(state, stream, nrounds);
+ hchacha_block_generic(state, out, nrounds);
} else {
kernel_fpu_begin();
- hchacha_block_ssse3(state, stream, nrounds);
+ hchacha_block_ssse3(state, out, nrounds);
kernel_fpu_end();
}
}
EXPORT_SYMBOL(hchacha_block_arch);
diff --git a/crypto/chacha.c b/crypto/chacha.c
index 73ce62a9ac22..c3a11f4e2d13 100644
--- a/crypto/chacha.c
+++ b/crypto/chacha.c
@@ -44,12 +44,12 @@ static int chacha12_setkey(struct crypto_skcipher *tfm,
{
return chacha_setkey(tfm, key, keysize, 12);
}
static int chacha_stream_xor(struct skcipher_request *req,
- const struct chacha_ctx *ctx, const u8 *iv,
- bool arch)
+ const struct chacha_ctx *ctx,
+ const u8 iv[CHACHA_IV_SIZE], bool arch)
{
struct skcipher_walk walk;
struct chacha_state state;
int err;
diff --git a/include/crypto/chacha.h b/include/crypto/chacha.h
index 7c2e6c68919b..91f6b4cf561c 100644
--- a/include/crypto/chacha.h
+++ b/include/crypto/chacha.h
@@ -24,32 +24,36 @@
#define CHACHA_KEY_SIZE 32
#define CHACHA_BLOCK_SIZE 64
#define CHACHAPOLY_IV_SIZE 12
-#define CHACHA_STATE_WORDS (CHACHA_BLOCK_SIZE / sizeof(u32))
+#define CHACHA_KEY_WORDS 8
+#define CHACHA_STATE_WORDS 16
+#define HCHACHA_OUT_WORDS 8
/* 192-bit nonce, then 64-bit stream position */
#define XCHACHA_IV_SIZE 32
struct chacha_state {
u32 x[CHACHA_STATE_WORDS];
};
-void chacha_block_generic(struct chacha_state *state, u8 *stream, int nrounds);
-static inline void chacha20_block(struct chacha_state *state, u8 *stream)
+void chacha_block_generic(struct chacha_state *state,
+ u8 out[CHACHA_BLOCK_SIZE], int nrounds);
+static inline void chacha20_block(struct chacha_state *state,
+ u8 out[CHACHA_BLOCK_SIZE])
{
- chacha_block_generic(state, stream, 20);
+ chacha_block_generic(state, out, 20);
}
-void hchacha_block_arch(const struct chacha_state *state, u32 *out,
- int nrounds);
-void hchacha_block_generic(const struct chacha_state *state, u32 *out,
- int nrounds);
+void hchacha_block_arch(const struct chacha_state *state,
+ u32 out[HCHACHA_OUT_WORDS], int nrounds);
+void hchacha_block_generic(const struct chacha_state *state,
+ u32 out[HCHACHA_OUT_WORDS], int nrounds);
-static inline void hchacha_block(const struct chacha_state *state, u32 *out,
- int nrounds)
+static inline void hchacha_block(const struct chacha_state *state,
+ u32 out[HCHACHA_OUT_WORDS], int nrounds)
{
if (IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA))
hchacha_block_arch(state, out, nrounds);
else
hchacha_block_generic(state, out, nrounds);
@@ -69,11 +73,12 @@ static inline void chacha_init_consts(struct chacha_state *state)
state->x[2] = CHACHA_CONSTANT_2_BY;
state->x[3] = CHACHA_CONSTANT_TE_K;
}
static inline void chacha_init(struct chacha_state *state,
- const u32 *key, const u8 *iv)
+ const u32 key[CHACHA_KEY_WORDS],
+ const u8 iv[CHACHA_IV_SIZE])
{
chacha_init_consts(state);
state->x[4] = key[0];
state->x[5] = key[1];
state->x[6] = key[2];
diff --git a/lib/crypto/chacha.c b/lib/crypto/chacha.c
index ae50e441f9fb..ced87dd31a97 100644
--- a/lib/crypto/chacha.c
+++ b/lib/crypto/chacha.c
@@ -65,49 +65,50 @@ static void chacha_permute(struct chacha_state *state, int nrounds)
}
/**
* chacha_block_generic - generate one keystream block and increment block counter
* @state: input state matrix
- * @stream: output keystream block (64 bytes)
+ * @out: output keystream block
* @nrounds: number of rounds (20 or 12; 20 is recommended)
*
* This is the ChaCha core, a function from 64-byte strings to 64-byte strings.
* The caller has already converted the endianness of the input. This function
* also handles incrementing the block counter in the input matrix.
*/
-void chacha_block_generic(struct chacha_state *state, u8 *stream, int nrounds)
+void chacha_block_generic(struct chacha_state *state,
+ u8 out[CHACHA_BLOCK_SIZE], int nrounds)
{
struct chacha_state permuted_state = *state;
int i;
chacha_permute(&permuted_state, nrounds);
for (i = 0; i < ARRAY_SIZE(state->x); i++)
put_unaligned_le32(permuted_state.x[i] + state->x[i],
- &stream[i * sizeof(u32)]);
+ &out[i * sizeof(u32)]);
state->x[12]++;
}
EXPORT_SYMBOL(chacha_block_generic);
/**
* hchacha_block_generic - abbreviated ChaCha core, for XChaCha
* @state: input state matrix
- * @stream: output (8 32-bit words)
+ * @out: the output words
* @nrounds: number of rounds (20 or 12; 20 is recommended)
*
* HChaCha is the ChaCha equivalent of HSalsa and is an intermediate step
* towards XChaCha (see https://cr.yp.to/snuffle/xsalsa-20081128.pdf). HChaCha
* skips the final addition of the initial state, and outputs only certain words
* of the state. It should not be used for streaming directly.
*/
void hchacha_block_generic(const struct chacha_state *state,
- u32 *stream, int nrounds)
+ u32 out[HCHACHA_OUT_WORDS], int nrounds)
{
struct chacha_state permuted_state = *state;
chacha_permute(&permuted_state, nrounds);
- memcpy(&stream[0], &permuted_state.x[0], 16);
- memcpy(&stream[4], &permuted_state.x[12], 16);
+ memcpy(&out[0], &permuted_state.x[0], 16);
+ memcpy(&out[4], &permuted_state.x[12], 16);
}
EXPORT_SYMBOL(hchacha_block_generic);
diff --git a/lib/crypto/chacha20poly1305.c b/lib/crypto/chacha20poly1305.c
index 2e7bbc1a67ea..fbd3690e2531 100644
--- a/lib/crypto/chacha20poly1305.c
+++ b/lib/crypto/chacha20poly1305.c
@@ -16,12 +16,10 @@
#include <linux/kernel.h>
#include <linux/init.h>
#include <linux/mm.h>
#include <linux/module.h>
-#define CHACHA_KEY_WORDS (CHACHA_KEY_SIZE / sizeof(u32))
-
static void chacha_load_key(u32 *k, const u8 *in)
{
k[0] = get_unaligned_le32(in);
k[1] = get_unaligned_le32(in + 4);
k[2] = get_unaligned_le32(in + 8);
--
2.49.0
^ permalink raw reply related [flat|nested] 7+ messages in thread
* Re: [PATCH 1/4] crypto: lib/chacha - strongly type the ChaCha state
2025-05-05 18:18 ` [PATCH 1/4] crypto: lib/chacha - strongly type the ChaCha state Eric Biggers
@ 2025-05-06 16:09 ` Kent Overstreet
0 siblings, 0 replies; 7+ messages in thread
From: Kent Overstreet @ 2025-05-06 16:09 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, linux-kernel, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, linux-bcachefs,
Jason A . Donenfeld , Theodore Ts'o
On Mon, May 05, 2025 at 11:18:21AM -0700, Eric Biggers wrote:
> From: Eric Biggers <ebiggers@google.com>
>
> The ChaCha state matrix is 16 32-bit words. Currently it is represented
> in the code as a raw u32 array, or even just a pointer to u32. This
> weak typing is error-prone. Instead, introduce struct chacha_state:
>
> struct chacha_state {
> u32 x[16];
> };
>
> Convert all ChaCha and HChaCha functions to use struct chacha_state.
> No functional changes.
>
> Signed-off-by: Eric Biggers <ebiggers@google.com>
Acked-by: Kent Overstreet <kent.overstreet@linux.dev>
> ---
> arch/arm/lib/crypto/chacha-glue.c | 30 +++++----
> arch/arm/lib/crypto/chacha-scalar-core.S | 5 +-
> arch/arm64/lib/crypto/chacha-neon-glue.c | 23 ++++---
> arch/mips/lib/crypto/chacha-glue.c | 6 +-
> arch/powerpc/lib/crypto/chacha-p10-glue.c | 15 +++--
> arch/powerpc/lib/crypto/chacha-p10le-8x.S | 6 +-
> arch/riscv/lib/crypto/chacha-riscv64-glue.c | 8 +--
> arch/riscv/lib/crypto/chacha-riscv64-zvkb.S | 10 +--
> arch/s390/lib/crypto/chacha-glue.c | 11 ++--
> arch/x86/lib/crypto/chacha_glue.c | 58 ++++++++++-------
> crypto/chacha.c | 16 ++---
> drivers/char/random.c | 41 ++++++------
> fs/bcachefs/checksum.c | 18 ++---
> include/crypto/chacha.h | 65 +++++++++++--------
> lib/crypto/chacha.c | 35 +++++-----
> lib/crypto/chacha20poly1305-selftest.c | 8 +--
> lib/crypto/chacha20poly1305.c | 51 ++++++++-------
> lib/crypto/libchacha.c | 2 +-
> .../crypto/chacha20-s390/test-cipher.c | 10 +--
> 19 files changed, 228 insertions(+), 190 deletions(-)
>
> diff --git a/arch/arm/lib/crypto/chacha-glue.c b/arch/arm/lib/crypto/chacha-glue.c
> index 1e28736834a0..0c2b4c62d484 100644
> --- a/arch/arm/lib/crypto/chacha-glue.c
> +++ b/arch/arm/lib/crypto/chacha-glue.c
> @@ -15,28 +15,31 @@
> #include <asm/cputype.h>
> #include <asm/hwcap.h>
> #include <asm/neon.h>
> #include <asm/simd.h>
>
> -asmlinkage void chacha_block_xor_neon(const u32 *state, u8 *dst, const u8 *src,
> - int nrounds);
> -asmlinkage void chacha_4block_xor_neon(const u32 *state, u8 *dst, const u8 *src,
> +asmlinkage void chacha_block_xor_neon(const struct chacha_state *state,
> + u8 *dst, const u8 *src, int nrounds);
> +asmlinkage void chacha_4block_xor_neon(const struct chacha_state *state,
> + u8 *dst, const u8 *src,
> int nrounds, unsigned int nbytes);
> -asmlinkage void hchacha_block_arm(const u32 *state, u32 *out, int nrounds);
> -asmlinkage void hchacha_block_neon(const u32 *state, u32 *out, int nrounds);
> +asmlinkage void hchacha_block_arm(const struct chacha_state *state,
> + u32 *out, int nrounds);
> +asmlinkage void hchacha_block_neon(const struct chacha_state *state,
> + u32 *out, int nrounds);
>
> asmlinkage void chacha_doarm(u8 *dst, const u8 *src, unsigned int bytes,
> - const u32 *state, int nrounds);
> + const struct chacha_state *state, int nrounds);
>
> static __ro_after_init DEFINE_STATIC_KEY_FALSE(use_neon);
>
> static inline bool neon_usable(void)
> {
> return static_branch_likely(&use_neon) && crypto_simd_usable();
> }
>
> -static void chacha_doneon(u32 *state, u8 *dst, const u8 *src,
> +static void chacha_doneon(struct chacha_state *state, u8 *dst, const u8 *src,
> unsigned int bytes, int nrounds)
> {
> u8 buf[CHACHA_BLOCK_SIZE];
>
> while (bytes > CHACHA_BLOCK_SIZE) {
> @@ -44,26 +47,27 @@ static void chacha_doneon(u32 *state, u8 *dst, const u8 *src,
>
> chacha_4block_xor_neon(state, dst, src, nrounds, l);
> bytes -= l;
> src += l;
> dst += l;
> - state[12] += DIV_ROUND_UP(l, CHACHA_BLOCK_SIZE);
> + state->x[12] += DIV_ROUND_UP(l, CHACHA_BLOCK_SIZE);
> }
> if (bytes) {
> const u8 *s = src;
> u8 *d = dst;
>
> if (bytes != CHACHA_BLOCK_SIZE)
> s = d = memcpy(buf, src, bytes);
> chacha_block_xor_neon(state, d, s, nrounds);
> if (d != dst)
> memcpy(dst, buf, bytes);
> - state[12]++;
> + state->x[12]++;
> }
> }
>
> -void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
> +void hchacha_block_arch(const struct chacha_state *state, u32 *stream,
> + int nrounds)
> {
> if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON) || !neon_usable()) {
> hchacha_block_arm(state, stream, nrounds);
> } else {
> kernel_neon_begin();
> @@ -71,17 +75,17 @@ void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
> kernel_neon_end();
> }
> }
> EXPORT_SYMBOL(hchacha_block_arch);
>
> -void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes,
> - int nrounds)
> +void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
> + unsigned int bytes, int nrounds)
> {
> if (!IS_ENABLED(CONFIG_KERNEL_MODE_NEON) || !neon_usable() ||
> bytes <= CHACHA_BLOCK_SIZE) {
> chacha_doarm(dst, src, bytes, state, nrounds);
> - state[12] += DIV_ROUND_UP(bytes, CHACHA_BLOCK_SIZE);
> + state->x[12] += DIV_ROUND_UP(bytes, CHACHA_BLOCK_SIZE);
> return;
> }
>
> do {
> unsigned int todo = min_t(unsigned int, bytes, SZ_4K);
> diff --git a/arch/arm/lib/crypto/chacha-scalar-core.S b/arch/arm/lib/crypto/chacha-scalar-core.S
> index 083fe1ab96d0..d20b5de755cc 100644
> --- a/arch/arm/lib/crypto/chacha-scalar-core.S
> +++ b/arch/arm/lib/crypto/chacha-scalar-core.S
> @@ -365,11 +365,11 @@
> .Ldone\@:
> .endm // _chacha
>
> /*
> * void chacha_doarm(u8 *dst, const u8 *src, unsigned int bytes,
> - * const u32 *state, int nrounds);
> + * const struct chacha_state *state, int nrounds);
> */
> ENTRY(chacha_doarm)
> cmp r2, #0 // len == 0?
> reteq lr
>
> @@ -405,11 +405,12 @@ ENTRY(chacha_doarm)
> 1: _chacha 12
> b 0b
> ENDPROC(chacha_doarm)
>
> /*
> - * void hchacha_block_arm(const u32 state[16], u32 out[8], int nrounds);
> + * void hchacha_block_arm(const struct chacha_state *state,
> + * u32 out[8], int nrounds);
> */
> ENTRY(hchacha_block_arm)
> push {r1,r4-r11,lr}
>
> cmp r2, #12 // ChaCha12 ?
> diff --git a/arch/arm64/lib/crypto/chacha-neon-glue.c b/arch/arm64/lib/crypto/chacha-neon-glue.c
> index 2b0de97a6daf..7b451b3c7240 100644
> --- a/arch/arm64/lib/crypto/chacha-neon-glue.c
> +++ b/arch/arm64/lib/crypto/chacha-neon-glue.c
> @@ -26,19 +26,21 @@
>
> #include <asm/hwcap.h>
> #include <asm/neon.h>
> #include <asm/simd.h>
>
> -asmlinkage void chacha_block_xor_neon(u32 *state, u8 *dst, const u8 *src,
> - int nrounds);
> -asmlinkage void chacha_4block_xor_neon(u32 *state, u8 *dst, const u8 *src,
> +asmlinkage void chacha_block_xor_neon(const struct chacha_state *state,
> + u8 *dst, const u8 *src, int nrounds);
> +asmlinkage void chacha_4block_xor_neon(const struct chacha_state *state,
> + u8 *dst, const u8 *src,
> int nrounds, int bytes);
> -asmlinkage void hchacha_block_neon(const u32 *state, u32 *out, int nrounds);
> +asmlinkage void hchacha_block_neon(const struct chacha_state *state,
> + u32 *out, int nrounds);
>
> static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon);
>
> -static void chacha_doneon(u32 *state, u8 *dst, const u8 *src,
> +static void chacha_doneon(struct chacha_state *state, u8 *dst, const u8 *src,
> int bytes, int nrounds)
> {
> while (bytes > 0) {
> int l = min(bytes, CHACHA_BLOCK_SIZE * 5);
>
> @@ -46,22 +48,23 @@ static void chacha_doneon(u32 *state, u8 *dst, const u8 *src,
> u8 buf[CHACHA_BLOCK_SIZE];
>
> memcpy(buf, src, l);
> chacha_block_xor_neon(state, buf, buf, nrounds);
> memcpy(dst, buf, l);
> - state[12] += 1;
> + state->x[12] += 1;
> break;
> }
> chacha_4block_xor_neon(state, dst, src, nrounds, l);
> bytes -= l;
> src += l;
> dst += l;
> - state[12] += DIV_ROUND_UP(l, CHACHA_BLOCK_SIZE);
> + state->x[12] += DIV_ROUND_UP(l, CHACHA_BLOCK_SIZE);
> }
> }
>
> -void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
> +void hchacha_block_arch(const struct chacha_state *state, u32 *stream,
> + int nrounds)
> {
> if (!static_branch_likely(&have_neon) || !crypto_simd_usable()) {
> hchacha_block_generic(state, stream, nrounds);
> } else {
> kernel_neon_begin();
> @@ -69,12 +72,12 @@ void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
> kernel_neon_end();
> }
> }
> EXPORT_SYMBOL(hchacha_block_arch);
>
> -void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes,
> - int nrounds)
> +void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
> + unsigned int bytes, int nrounds)
> {
> if (!static_branch_likely(&have_neon) || bytes <= CHACHA_BLOCK_SIZE ||
> !crypto_simd_usable())
> return chacha_crypt_generic(state, dst, src, bytes, nrounds);
>
> diff --git a/arch/mips/lib/crypto/chacha-glue.c b/arch/mips/lib/crypto/chacha-glue.c
> index 334ecb29fb8f..75df4040cded 100644
> --- a/arch/mips/lib/crypto/chacha-glue.c
> +++ b/arch/mips/lib/crypto/chacha-glue.c
> @@ -7,15 +7,17 @@
>
> #include <crypto/chacha.h>
> #include <linux/kernel.h>
> #include <linux/module.h>
>
> -asmlinkage void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src,
> +asmlinkage void chacha_crypt_arch(struct chacha_state *state,
> + u8 *dst, const u8 *src,
> unsigned int bytes, int nrounds);
> EXPORT_SYMBOL(chacha_crypt_arch);
>
> -asmlinkage void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds);
> +asmlinkage void hchacha_block_arch(const struct chacha_state *state,
> + u32 *stream, int nrounds);
> EXPORT_SYMBOL(hchacha_block_arch);
>
> bool chacha_is_arch_optimized(void)
> {
> return true;
> diff --git a/arch/powerpc/lib/crypto/chacha-p10-glue.c b/arch/powerpc/lib/crypto/chacha-p10-glue.c
> index 51daeaf5d26e..a6e6a8da1b8b 100644
> --- a/arch/powerpc/lib/crypto/chacha-p10-glue.c
> +++ b/arch/powerpc/lib/crypto/chacha-p10-glue.c
> @@ -12,12 +12,12 @@
> #include <linux/cpufeature.h>
> #include <linux/sizes.h>
> #include <asm/simd.h>
> #include <asm/switch_to.h>
>
> -asmlinkage void chacha_p10le_8x(u32 *state, u8 *dst, const u8 *src,
> - unsigned int len, int nrounds);
> +asmlinkage void chacha_p10le_8x(const struct chacha_state *state, u8 *dst,
> + const u8 *src, unsigned int len, int nrounds);
>
> static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_p10);
>
> static void vsx_begin(void)
> {
> @@ -29,35 +29,36 @@ static void vsx_end(void)
> {
> disable_kernel_vsx();
> preempt_enable();
> }
>
> -static void chacha_p10_do_8x(u32 *state, u8 *dst, const u8 *src,
> +static void chacha_p10_do_8x(struct chacha_state *state, u8 *dst, const u8 *src,
> unsigned int bytes, int nrounds)
> {
> unsigned int l = bytes & ~0x0FF;
>
> if (l > 0) {
> chacha_p10le_8x(state, dst, src, l, nrounds);
> bytes -= l;
> src += l;
> dst += l;
> - state[12] += l / CHACHA_BLOCK_SIZE;
> + state->x[12] += l / CHACHA_BLOCK_SIZE;
> }
>
> if (bytes > 0)
> chacha_crypt_generic(state, dst, src, bytes, nrounds);
> }
>
> -void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
> +void hchacha_block_arch(const struct chacha_state *state,
> + u32 *stream, int nrounds)
> {
> hchacha_block_generic(state, stream, nrounds);
> }
> EXPORT_SYMBOL(hchacha_block_arch);
>
> -void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes,
> - int nrounds)
> +void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
> + unsigned int bytes, int nrounds)
> {
> if (!static_branch_likely(&have_p10) || bytes <= CHACHA_BLOCK_SIZE ||
> !crypto_simd_usable())
> return chacha_crypt_generic(state, dst, src, bytes, nrounds);
>
> diff --git a/arch/powerpc/lib/crypto/chacha-p10le-8x.S b/arch/powerpc/lib/crypto/chacha-p10le-8x.S
> index 17bedb66b822..b29562bd5d40 100644
> --- a/arch/powerpc/lib/crypto/chacha-p10le-8x.S
> +++ b/arch/powerpc/lib/crypto/chacha-p10le-8x.S
> @@ -5,13 +5,10 @@
> # Copyright 2023- IBM Corp. All rights reserved
> #
> #===================================================================================
> # Written by Danny Tsen <dtsen@us.ibm.com>
> #
> -# chacha_p10le_8x(u32 *state, byte *dst, const byte *src,
> -# size_t len, int nrounds);
> -#
> # do rounds, 8 quarter rounds
> # 1. a += b; d ^= a; d <<<= 16;
> # 2. c += d; b ^= c; b <<<= 12;
> # 3. a += b; d ^= a; d <<<= 8;
> # 4. c += d; b ^= c; b <<<= 7
> @@ -573,11 +570,12 @@
> stxvw4x \S+47, 31, 16
>
> .endm
>
> #
> -# chacha20_p10le_8x(u32 *state, byte *dst, const byte *src, size_t len, int nrounds);
> +# void chacha_p10le_8x(const struct chacha_state *state, u8 *dst, const u8 *src,
> +# unsigned int len, int nrounds);
> #
> SYM_FUNC_START(chacha_p10le_8x)
> .align 5
> cmpdi 6, 0
> ble Out_no_chacha
> diff --git a/arch/riscv/lib/crypto/chacha-riscv64-glue.c b/arch/riscv/lib/crypto/chacha-riscv64-glue.c
> index 1740e1ca3a94..57541621981e 100644
> --- a/arch/riscv/lib/crypto/chacha-riscv64-glue.c
> +++ b/arch/riscv/lib/crypto/chacha-riscv64-glue.c
> @@ -13,21 +13,21 @@
> #include <linux/linkage.h>
> #include <linux/module.h>
>
> static __ro_after_init DEFINE_STATIC_KEY_FALSE(use_zvkb);
>
> -asmlinkage void chacha_zvkb(u32 state[16], const u8 *in, u8 *out,
> +asmlinkage void chacha_zvkb(struct chacha_state *state, const u8 *in, u8 *out,
> size_t nblocks, int nrounds);
>
> -void hchacha_block_arch(const u32 *state, u32 *out, int nrounds)
> +void hchacha_block_arch(const struct chacha_state *state, u32 *out, int nrounds)
> {
> hchacha_block_generic(state, out, nrounds);
> }
> EXPORT_SYMBOL(hchacha_block_arch);
>
> -void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes,
> - int nrounds)
> +void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
> + unsigned int bytes, int nrounds)
> {
> u8 block_buffer[CHACHA_BLOCK_SIZE];
> unsigned int full_blocks = bytes / CHACHA_BLOCK_SIZE;
> unsigned int tail_bytes = bytes % CHACHA_BLOCK_SIZE;
>
> diff --git a/arch/riscv/lib/crypto/chacha-riscv64-zvkb.S b/arch/riscv/lib/crypto/chacha-riscv64-zvkb.S
> index ab4423b3880e..b777d0b4e379 100644
> --- a/arch/riscv/lib/crypto/chacha-riscv64-zvkb.S
> +++ b/arch/riscv/lib/crypto/chacha-riscv64-zvkb.S
> @@ -130,19 +130,19 @@
> vror.vi \b1, \b1, 32 - 7
> vror.vi \b2, \b2, 32 - 7
> vror.vi \b3, \b3, 32 - 7
> .endm
>
> -// void chacha_zvkb(u32 state[16], const u8 *in, u8 *out, size_t nblocks,
> -// int nrounds);
> +// void chacha_zvkb(struct chacha_state *state, const u8 *in, u8 *out,
> +// size_t nblocks, int nrounds);
> //
> // |nblocks| is the number of 64-byte blocks to process, and must be nonzero.
> //
> // |state| gives the ChaCha state matrix, including the 32-bit counter in
> -// state[12] following the RFC7539 convention; note that this differs from the
> -// original Salsa20 paper which uses a 64-bit counter in state[12..13]. The
> -// updated 32-bit counter is written back to state[12] before returning.
> +// state->x[12] following the RFC7539 convention; note that this differs from
> +// the original Salsa20 paper which uses a 64-bit counter in state->x[12..13].
> +// The updated 32-bit counter is written back to state->x[12] before returning.
> SYM_FUNC_START(chacha_zvkb)
> addi sp, sp, -96
> sd s0, 0(sp)
> sd s1, 8(sp)
> sd s2, 16(sp)
> diff --git a/arch/s390/lib/crypto/chacha-glue.c b/arch/s390/lib/crypto/chacha-glue.c
> index b3ffaa555385..0a9fd50c1bd8 100644
> --- a/arch/s390/lib/crypto/chacha-glue.c
> +++ b/arch/s390/lib/crypto/chacha-glue.c
> @@ -14,18 +14,19 @@
> #include <linux/module.h>
> #include <linux/sizes.h>
> #include <asm/fpu.h>
> #include "chacha-s390.h"
>
> -void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
> +void hchacha_block_arch(const struct chacha_state *state,
> + u32 *stream, int nrounds)
> {
> /* TODO: implement hchacha_block_arch() in assembly */
> hchacha_block_generic(state, stream, nrounds);
> }
> EXPORT_SYMBOL(hchacha_block_arch);
>
> -void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src,
> +void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
> unsigned int bytes, int nrounds)
> {
> /* s390 chacha20 implementation has 20 rounds hard-coded,
> * it cannot handle a block of data or less, but otherwise
> * it can handle data of arbitrary size
> @@ -34,15 +35,15 @@ void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src,
> chacha_crypt_generic(state, dst, src, bytes, nrounds);
> } else {
> DECLARE_KERNEL_FPU_ONSTACK32(vxstate);
>
> kernel_fpu_begin(&vxstate, KERNEL_VXR);
> - chacha20_vx(dst, src, bytes, &state[4], &state[12]);
> + chacha20_vx(dst, src, bytes, &state->x[4], &state->x[12]);
> kernel_fpu_end(&vxstate, KERNEL_VXR);
>
> - state[12] += round_up(bytes, CHACHA_BLOCK_SIZE) /
> - CHACHA_BLOCK_SIZE;
> + state->x[12] += round_up(bytes, CHACHA_BLOCK_SIZE) /
> + CHACHA_BLOCK_SIZE;
> }
> }
> EXPORT_SYMBOL(chacha_crypt_arch);
>
> bool chacha_is_arch_optimized(void)
> diff --git a/arch/x86/lib/crypto/chacha_glue.c b/arch/x86/lib/crypto/chacha_glue.c
> index 94fcefbc8827..6f00a56e3e9a 100644
> --- a/arch/x86/lib/crypto/chacha_glue.c
> +++ b/arch/x86/lib/crypto/chacha_glue.c
> @@ -10,28 +10,37 @@
> #include <linux/jump_label.h>
> #include <linux/kernel.h>
> #include <linux/module.h>
> #include <linux/sizes.h>
>
> -asmlinkage void chacha_block_xor_ssse3(u32 *state, u8 *dst, const u8 *src,
> +asmlinkage void chacha_block_xor_ssse3(const struct chacha_state *state,
> + u8 *dst, const u8 *src,
> unsigned int len, int nrounds);
> -asmlinkage void chacha_4block_xor_ssse3(u32 *state, u8 *dst, const u8 *src,
> +asmlinkage void chacha_4block_xor_ssse3(const struct chacha_state *state,
> + u8 *dst, const u8 *src,
> unsigned int len, int nrounds);
> -asmlinkage void hchacha_block_ssse3(const u32 *state, u32 *out, int nrounds);
> +asmlinkage void hchacha_block_ssse3(const struct chacha_state *state,
> + u32 *out, int nrounds);
>
> -asmlinkage void chacha_2block_xor_avx2(u32 *state, u8 *dst, const u8 *src,
> +asmlinkage void chacha_2block_xor_avx2(const struct chacha_state *state,
> + u8 *dst, const u8 *src,
> unsigned int len, int nrounds);
> -asmlinkage void chacha_4block_xor_avx2(u32 *state, u8 *dst, const u8 *src,
> +asmlinkage void chacha_4block_xor_avx2(const struct chacha_state *state,
> + u8 *dst, const u8 *src,
> unsigned int len, int nrounds);
> -asmlinkage void chacha_8block_xor_avx2(u32 *state, u8 *dst, const u8 *src,
> +asmlinkage void chacha_8block_xor_avx2(const struct chacha_state *state,
> + u8 *dst, const u8 *src,
> unsigned int len, int nrounds);
>
> -asmlinkage void chacha_2block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src,
> +asmlinkage void chacha_2block_xor_avx512vl(const struct chacha_state *state,
> + u8 *dst, const u8 *src,
> unsigned int len, int nrounds);
> -asmlinkage void chacha_4block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src,
> +asmlinkage void chacha_4block_xor_avx512vl(const struct chacha_state *state,
> + u8 *dst, const u8 *src,
> unsigned int len, int nrounds);
> -asmlinkage void chacha_8block_xor_avx512vl(u32 *state, u8 *dst, const u8 *src,
> +asmlinkage void chacha_8block_xor_avx512vl(const struct chacha_state *state,
> + u8 *dst, const u8 *src,
> unsigned int len, int nrounds);
>
> static __ro_after_init DEFINE_STATIC_KEY_FALSE(chacha_use_simd);
> static __ro_after_init DEFINE_STATIC_KEY_FALSE(chacha_use_avx2);
> static __ro_after_init DEFINE_STATIC_KEY_FALSE(chacha_use_avx512vl);
> @@ -40,86 +49,87 @@ static unsigned int chacha_advance(unsigned int len, unsigned int maxblocks)
> {
> len = min(len, maxblocks * CHACHA_BLOCK_SIZE);
> return round_up(len, CHACHA_BLOCK_SIZE) / CHACHA_BLOCK_SIZE;
> }
>
> -static void chacha_dosimd(u32 *state, u8 *dst, const u8 *src,
> +static void chacha_dosimd(struct chacha_state *state, u8 *dst, const u8 *src,
> unsigned int bytes, int nrounds)
> {
> if (static_branch_likely(&chacha_use_avx512vl)) {
> while (bytes >= CHACHA_BLOCK_SIZE * 8) {
> chacha_8block_xor_avx512vl(state, dst, src, bytes,
> nrounds);
> bytes -= CHACHA_BLOCK_SIZE * 8;
> src += CHACHA_BLOCK_SIZE * 8;
> dst += CHACHA_BLOCK_SIZE * 8;
> - state[12] += 8;
> + state->x[12] += 8;
> }
> if (bytes > CHACHA_BLOCK_SIZE * 4) {
> chacha_8block_xor_avx512vl(state, dst, src, bytes,
> nrounds);
> - state[12] += chacha_advance(bytes, 8);
> + state->x[12] += chacha_advance(bytes, 8);
> return;
> }
> if (bytes > CHACHA_BLOCK_SIZE * 2) {
> chacha_4block_xor_avx512vl(state, dst, src, bytes,
> nrounds);
> - state[12] += chacha_advance(bytes, 4);
> + state->x[12] += chacha_advance(bytes, 4);
> return;
> }
> if (bytes) {
> chacha_2block_xor_avx512vl(state, dst, src, bytes,
> nrounds);
> - state[12] += chacha_advance(bytes, 2);
> + state->x[12] += chacha_advance(bytes, 2);
> return;
> }
> }
>
> if (static_branch_likely(&chacha_use_avx2)) {
> while (bytes >= CHACHA_BLOCK_SIZE * 8) {
> chacha_8block_xor_avx2(state, dst, src, bytes, nrounds);
> bytes -= CHACHA_BLOCK_SIZE * 8;
> src += CHACHA_BLOCK_SIZE * 8;
> dst += CHACHA_BLOCK_SIZE * 8;
> - state[12] += 8;
> + state->x[12] += 8;
> }
> if (bytes > CHACHA_BLOCK_SIZE * 4) {
> chacha_8block_xor_avx2(state, dst, src, bytes, nrounds);
> - state[12] += chacha_advance(bytes, 8);
> + state->x[12] += chacha_advance(bytes, 8);
> return;
> }
> if (bytes > CHACHA_BLOCK_SIZE * 2) {
> chacha_4block_xor_avx2(state, dst, src, bytes, nrounds);
> - state[12] += chacha_advance(bytes, 4);
> + state->x[12] += chacha_advance(bytes, 4);
> return;
> }
> if (bytes > CHACHA_BLOCK_SIZE) {
> chacha_2block_xor_avx2(state, dst, src, bytes, nrounds);
> - state[12] += chacha_advance(bytes, 2);
> + state->x[12] += chacha_advance(bytes, 2);
> return;
> }
> }
>
> while (bytes >= CHACHA_BLOCK_SIZE * 4) {
> chacha_4block_xor_ssse3(state, dst, src, bytes, nrounds);
> bytes -= CHACHA_BLOCK_SIZE * 4;
> src += CHACHA_BLOCK_SIZE * 4;
> dst += CHACHA_BLOCK_SIZE * 4;
> - state[12] += 4;
> + state->x[12] += 4;
> }
> if (bytes > CHACHA_BLOCK_SIZE) {
> chacha_4block_xor_ssse3(state, dst, src, bytes, nrounds);
> - state[12] += chacha_advance(bytes, 4);
> + state->x[12] += chacha_advance(bytes, 4);
> return;
> }
> if (bytes) {
> chacha_block_xor_ssse3(state, dst, src, bytes, nrounds);
> - state[12]++;
> + state->x[12]++;
> }
> }
>
> -void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
> +void hchacha_block_arch(const struct chacha_state *state,
> + u32 *stream, int nrounds)
> {
> if (!static_branch_likely(&chacha_use_simd)) {
> hchacha_block_generic(state, stream, nrounds);
> } else {
> kernel_fpu_begin();
> @@ -127,12 +137,12 @@ void hchacha_block_arch(const u32 *state, u32 *stream, int nrounds)
> kernel_fpu_end();
> }
> }
> EXPORT_SYMBOL(hchacha_block_arch);
>
> -void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src, unsigned int bytes,
> - int nrounds)
> +void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
> + unsigned int bytes, int nrounds)
> {
> if (!static_branch_likely(&chacha_use_simd) ||
> bytes <= CHACHA_BLOCK_SIZE)
> return chacha_crypt_generic(state, dst, src, bytes, nrounds);
>
> diff --git a/crypto/chacha.c b/crypto/chacha.c
> index 28a8ad6197ab..73ce62a9ac22 100644
> --- a/crypto/chacha.c
> +++ b/crypto/chacha.c
> @@ -48,28 +48,28 @@ static int chacha12_setkey(struct crypto_skcipher *tfm,
> static int chacha_stream_xor(struct skcipher_request *req,
> const struct chacha_ctx *ctx, const u8 *iv,
> bool arch)
> {
> struct skcipher_walk walk;
> - u32 state[16];
> + struct chacha_state state;
> int err;
>
> err = skcipher_walk_virt(&walk, req, false);
>
> - chacha_init(state, ctx->key, iv);
> + chacha_init(&state, ctx->key, iv);
>
> while (walk.nbytes > 0) {
> unsigned int nbytes = walk.nbytes;
>
> if (nbytes < walk.total)
> nbytes = round_down(nbytes, CHACHA_BLOCK_SIZE);
>
> if (arch)
> - chacha_crypt(state, walk.dst.virt.addr,
> + chacha_crypt(&state, walk.dst.virt.addr,
> walk.src.virt.addr, nbytes, ctx->nrounds);
> else
> - chacha_crypt_generic(state, walk.dst.virt.addr,
> + chacha_crypt_generic(&state, walk.dst.virt.addr,
> walk.src.virt.addr, nbytes,
> ctx->nrounds);
> err = skcipher_walk_done(&walk, walk.nbytes - nbytes);
> }
>
> @@ -95,19 +95,19 @@ static int crypto_chacha_crypt_arch(struct skcipher_request *req)
> static int crypto_xchacha_crypt(struct skcipher_request *req, bool arch)
> {
> struct crypto_skcipher *tfm = crypto_skcipher_reqtfm(req);
> const struct chacha_ctx *ctx = crypto_skcipher_ctx(tfm);
> struct chacha_ctx subctx;
> - u32 state[16];
> + struct chacha_state state;
> u8 real_iv[16];
>
> /* Compute the subkey given the original key and first 128 nonce bits */
> - chacha_init(state, ctx->key, req->iv);
> + chacha_init(&state, ctx->key, req->iv);
> if (arch)
> - hchacha_block(state, subctx.key, ctx->nrounds);
> + hchacha_block(&state, subctx.key, ctx->nrounds);
> else
> - hchacha_block_generic(state, subctx.key, ctx->nrounds);
> + hchacha_block_generic(&state, subctx.key, ctx->nrounds);
> subctx.nrounds = ctx->nrounds;
>
> /* Build the real IV */
> memcpy(&real_iv[0], req->iv + 24, 8); /* stream position */
> memcpy(&real_iv[8], req->iv + 16, 8); /* remaining 64 nonce bits */
> diff --git a/drivers/char/random.c b/drivers/char/random.c
> index 38f2fab29c56..9f876ed2655b 100644
> --- a/drivers/char/random.c
> +++ b/drivers/char/random.c
> @@ -307,24 +307,24 @@ static void crng_reseed(struct work_struct *work)
> *
> * The returned ChaCha state contains within it a copy of the old
> * key value, at index 4, so the state should always be zeroed out
> * immediately after using in order to maintain forward secrecy.
> * If the state cannot be erased in a timely manner, then it is
> - * safer to set the random_data parameter to &chacha_state[4] so
> - * that this function overwrites it before returning.
> + * safer to set the random_data parameter to &chacha_state->x[4]
> + * so that this function overwrites it before returning.
> */
> static void crng_fast_key_erasure(u8 key[CHACHA_KEY_SIZE],
> - u32 chacha_state[CHACHA_STATE_WORDS],
> + struct chacha_state *chacha_state,
> u8 *random_data, size_t random_data_len)
> {
> u8 first_block[CHACHA_BLOCK_SIZE];
>
> BUG_ON(random_data_len > 32);
>
> chacha_init_consts(chacha_state);
> - memcpy(&chacha_state[4], key, CHACHA_KEY_SIZE);
> - memset(&chacha_state[12], 0, sizeof(u32) * 4);
> + memcpy(&chacha_state->x[4], key, CHACHA_KEY_SIZE);
> + memset(&chacha_state->x[12], 0, sizeof(u32) * 4);
> chacha20_block(chacha_state, first_block);
>
> memcpy(key, first_block, CHACHA_KEY_SIZE);
> memcpy(random_data, first_block + CHACHA_KEY_SIZE, random_data_len);
> memzero_explicit(first_block, sizeof(first_block));
> @@ -333,11 +333,11 @@ static void crng_fast_key_erasure(u8 key[CHACHA_KEY_SIZE],
> /*
> * This function returns a ChaCha state that you may use for generating
> * random data. It also returns up to 32 bytes on its own of random data
> * that may be used; random_data_len may not be greater than 32.
> */
> -static void crng_make_state(u32 chacha_state[CHACHA_STATE_WORDS],
> +static void crng_make_state(struct chacha_state *chacha_state,
> u8 *random_data, size_t random_data_len)
> {
> unsigned long flags;
> struct crng *crng;
>
> @@ -393,38 +393,38 @@ static void crng_make_state(u32 chacha_state[CHACHA_STATE_WORDS],
> local_unlock_irqrestore(&crngs.lock, flags);
> }
>
> static void _get_random_bytes(void *buf, size_t len)
> {
> - u32 chacha_state[CHACHA_STATE_WORDS];
> + struct chacha_state chacha_state;
> u8 tmp[CHACHA_BLOCK_SIZE];
> size_t first_block_len;
>
> if (!len)
> return;
>
> first_block_len = min_t(size_t, 32, len);
> - crng_make_state(chacha_state, buf, first_block_len);
> + crng_make_state(&chacha_state, buf, first_block_len);
> len -= first_block_len;
> buf += first_block_len;
>
> while (len) {
> if (len < CHACHA_BLOCK_SIZE) {
> - chacha20_block(chacha_state, tmp);
> + chacha20_block(&chacha_state, tmp);
> memcpy(buf, tmp, len);
> memzero_explicit(tmp, sizeof(tmp));
> break;
> }
>
> - chacha20_block(chacha_state, buf);
> - if (unlikely(chacha_state[12] == 0))
> - ++chacha_state[13];
> + chacha20_block(&chacha_state, buf);
> + if (unlikely(chacha_state.x[12] == 0))
> + ++chacha_state.x[13];
> len -= CHACHA_BLOCK_SIZE;
> buf += CHACHA_BLOCK_SIZE;
> }
>
> - memzero_explicit(chacha_state, sizeof(chacha_state));
> + memzero_explicit(&chacha_state, sizeof(chacha_state));
> }
>
> /*
> * This returns random bytes in arbitrary quantities. The quality of the
> * random bytes is good as /dev/urandom. In order to ensure that the
> @@ -439,11 +439,11 @@ void get_random_bytes(void *buf, size_t len)
> }
> EXPORT_SYMBOL(get_random_bytes);
>
> static ssize_t get_random_bytes_user(struct iov_iter *iter)
> {
> - u32 chacha_state[CHACHA_STATE_WORDS];
> + struct chacha_state chacha_state;
> u8 block[CHACHA_BLOCK_SIZE];
> size_t ret = 0, copied;
>
> if (unlikely(!iov_iter_count(iter)))
> return 0;
> @@ -451,25 +451,26 @@ static ssize_t get_random_bytes_user(struct iov_iter *iter)
> /*
> * Immediately overwrite the ChaCha key at index 4 with random
> * bytes, in case userspace causes copy_to_iter() below to sleep
> * forever, so that we still retain forward secrecy in that case.
> */
> - crng_make_state(chacha_state, (u8 *)&chacha_state[4], CHACHA_KEY_SIZE);
> + crng_make_state(&chacha_state, (u8 *)&chacha_state.x[4],
> + CHACHA_KEY_SIZE);
> /*
> * However, if we're doing a read of len <= 32, we don't need to
> * use chacha_state after, so we can simply return those bytes to
> * the user directly.
> */
> if (iov_iter_count(iter) <= CHACHA_KEY_SIZE) {
> - ret = copy_to_iter(&chacha_state[4], CHACHA_KEY_SIZE, iter);
> + ret = copy_to_iter(&chacha_state.x[4], CHACHA_KEY_SIZE, iter);
> goto out_zero_chacha;
> }
>
> for (;;) {
> - chacha20_block(chacha_state, block);
> - if (unlikely(chacha_state[12] == 0))
> - ++chacha_state[13];
> + chacha20_block(&chacha_state, block);
> + if (unlikely(chacha_state.x[12] == 0))
> + ++chacha_state.x[13];
>
> copied = copy_to_iter(block, sizeof(block), iter);
> ret += copied;
> if (!iov_iter_count(iter) || copied != sizeof(block))
> break;
> @@ -482,11 +483,11 @@ static ssize_t get_random_bytes_user(struct iov_iter *iter)
> }
> }
>
> memzero_explicit(block, sizeof(block));
> out_zero_chacha:
> - memzero_explicit(chacha_state, sizeof(chacha_state));
> + memzero_explicit(&chacha_state, sizeof(chacha_state));
> return ret ? ret : -EFAULT;
> }
>
> /*
> * Batched entropy returns random integers. The quality of the random
> diff --git a/fs/bcachefs/checksum.c b/fs/bcachefs/checksum.c
> index d0a34a097b80..312fda4bb1b5 100644
> --- a/fs/bcachefs/checksum.c
> +++ b/fs/bcachefs/checksum.c
> @@ -89,11 +89,11 @@ static void bch2_checksum_update(struct bch2_checksum_state *state, const void *
> default:
> BUG();
> }
> }
>
> -static void bch2_chacha20_init(u32 state[CHACHA_STATE_WORDS],
> +static void bch2_chacha20_init(struct chacha_state *state,
> const struct bch_key *key, struct nonce nonce)
> {
> u32 key_words[CHACHA_KEY_SIZE / sizeof(u32)];
>
> BUILD_BUG_ON(sizeof(key_words) != sizeof(*key));
> @@ -107,15 +107,15 @@ static void bch2_chacha20_init(u32 state[CHACHA_STATE_WORDS],
> }
>
> static void bch2_chacha20(const struct bch_key *key, struct nonce nonce,
> void *data, size_t len)
> {
> - u32 state[CHACHA_STATE_WORDS];
> + struct chacha_state state;
>
> - bch2_chacha20_init(state, key, nonce);
> - chacha20_crypt(state, data, data, len);
> - memzero_explicit(state, sizeof(state));
> + bch2_chacha20_init(&state, key, nonce);
> + chacha20_crypt(&state, data, data, len);
> + memzero_explicit(&state, sizeof(state));
> }
>
> static void bch2_poly1305_init(struct poly1305_desc_ctx *desc,
> struct bch_fs *c, struct nonce nonce)
> {
> @@ -255,18 +255,18 @@ struct bch_csum bch2_checksum_bio(struct bch_fs *c, unsigned type,
> int __bch2_encrypt_bio(struct bch_fs *c, unsigned type,
> struct nonce nonce, struct bio *bio)
> {
> struct bio_vec bv;
> struct bvec_iter iter;
> - u32 chacha_state[CHACHA_STATE_WORDS];
> + struct chacha_state chacha_state;
> int ret = 0;
>
> if (bch2_fs_inconsistent_on(!c->chacha20_key_set,
> c, "attempting to encrypt without encryption key"))
> return -BCH_ERR_no_encryption_key;
>
> - bch2_chacha20_init(chacha_state, &c->chacha20_key, nonce);
> + bch2_chacha20_init(&chacha_state, &c->chacha20_key, nonce);
>
> bio_for_each_segment(bv, bio, iter) {
> void *p;
>
> /*
> @@ -278,14 +278,14 @@ int __bch2_encrypt_bio(struct bch_fs *c, unsigned type,
> ret = -EIO;
> break;
> }
>
> p = bvec_kmap_local(&bv);
> - chacha20_crypt(chacha_state, p, p, bv.bv_len);
> + chacha20_crypt(&chacha_state, p, p, bv.bv_len);
> kunmap_local(p);
> }
> - memzero_explicit(chacha_state, sizeof(chacha_state));
> + memzero_explicit(&chacha_state, sizeof(chacha_state));
> return ret;
> }
>
> struct bch_csum bch2_checksum_merge(unsigned type, struct bch_csum a,
> struct bch_csum b, size_t b_len)
> diff --git a/include/crypto/chacha.h b/include/crypto/chacha.h
> index 58129e18cc31..64fb270f2bfc 100644
> --- a/include/crypto/chacha.h
> +++ b/include/crypto/chacha.h
> @@ -28,20 +28,27 @@
> #define CHACHA_STATE_WORDS (CHACHA_BLOCK_SIZE / sizeof(u32))
>
> /* 192-bit nonce, then 64-bit stream position */
> #define XCHACHA_IV_SIZE 32
>
> -void chacha_block_generic(u32 *state, u8 *stream, int nrounds);
> -static inline void chacha20_block(u32 *state, u8 *stream)
> +struct chacha_state {
> + u32 x[CHACHA_STATE_WORDS];
> +};
> +
> +void chacha_block_generic(struct chacha_state *state, u8 *stream, int nrounds);
> +static inline void chacha20_block(struct chacha_state *state, u8 *stream)
> {
> chacha_block_generic(state, stream, 20);
> }
>
> -void hchacha_block_arch(const u32 *state, u32 *out, int nrounds);
> -void hchacha_block_generic(const u32 *state, u32 *out, int nrounds);
> +void hchacha_block_arch(const struct chacha_state *state, u32 *out,
> + int nrounds);
> +void hchacha_block_generic(const struct chacha_state *state, u32 *out,
> + int nrounds);
>
> -static inline void hchacha_block(const u32 *state, u32 *out, int nrounds)
> +static inline void hchacha_block(const struct chacha_state *state, u32 *out,
> + int nrounds)
> {
> if (IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA))
> hchacha_block_arch(state, out, nrounds);
> else
> hchacha_block_generic(state, out, nrounds);
> @@ -52,51 +59,53 @@ enum chacha_constants { /* expand 32-byte k */
> CHACHA_CONSTANT_ND_3 = 0x3320646eU,
> CHACHA_CONSTANT_2_BY = 0x79622d32U,
> CHACHA_CONSTANT_TE_K = 0x6b206574U
> };
>
> -static inline void chacha_init_consts(u32 *state)
> +static inline void chacha_init_consts(struct chacha_state *state)
> {
> - state[0] = CHACHA_CONSTANT_EXPA;
> - state[1] = CHACHA_CONSTANT_ND_3;
> - state[2] = CHACHA_CONSTANT_2_BY;
> - state[3] = CHACHA_CONSTANT_TE_K;
> + state->x[0] = CHACHA_CONSTANT_EXPA;
> + state->x[1] = CHACHA_CONSTANT_ND_3;
> + state->x[2] = CHACHA_CONSTANT_2_BY;
> + state->x[3] = CHACHA_CONSTANT_TE_K;
> }
>
> -static inline void chacha_init(u32 *state, const u32 *key, const u8 *iv)
> +static inline void chacha_init(struct chacha_state *state,
> + const u32 *key, const u8 *iv)
> {
> chacha_init_consts(state);
> - state[4] = key[0];
> - state[5] = key[1];
> - state[6] = key[2];
> - state[7] = key[3];
> - state[8] = key[4];
> - state[9] = key[5];
> - state[10] = key[6];
> - state[11] = key[7];
> - state[12] = get_unaligned_le32(iv + 0);
> - state[13] = get_unaligned_le32(iv + 4);
> - state[14] = get_unaligned_le32(iv + 8);
> - state[15] = get_unaligned_le32(iv + 12);
> + state->x[4] = key[0];
> + state->x[5] = key[1];
> + state->x[6] = key[2];
> + state->x[7] = key[3];
> + state->x[8] = key[4];
> + state->x[9] = key[5];
> + state->x[10] = key[6];
> + state->x[11] = key[7];
> + state->x[12] = get_unaligned_le32(iv + 0);
> + state->x[13] = get_unaligned_le32(iv + 4);
> + state->x[14] = get_unaligned_le32(iv + 8);
> + state->x[15] = get_unaligned_le32(iv + 12);
> }
>
> -void chacha_crypt_arch(u32 *state, u8 *dst, const u8 *src,
> +void chacha_crypt_arch(struct chacha_state *state, u8 *dst, const u8 *src,
> unsigned int bytes, int nrounds);
> -void chacha_crypt_generic(u32 *state, u8 *dst, const u8 *src,
> +void chacha_crypt_generic(struct chacha_state *state, u8 *dst, const u8 *src,
> unsigned int bytes, int nrounds);
>
> -static inline void chacha_crypt(u32 *state, u8 *dst, const u8 *src,
> +static inline void chacha_crypt(struct chacha_state *state,
> + u8 *dst, const u8 *src,
> unsigned int bytes, int nrounds)
> {
> if (IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA))
> chacha_crypt_arch(state, dst, src, bytes, nrounds);
> else
> chacha_crypt_generic(state, dst, src, bytes, nrounds);
> }
>
> -static inline void chacha20_crypt(u32 *state, u8 *dst, const u8 *src,
> - unsigned int bytes)
> +static inline void chacha20_crypt(struct chacha_state *state,
> + u8 *dst, const u8 *src, unsigned int bytes)
> {
> chacha_crypt(state, dst, src, bytes, 20);
> }
>
> #if IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_CHACHA)
> diff --git a/lib/crypto/chacha.c b/lib/crypto/chacha.c
> index 3cdda3b5ee06..a7f5eb091839 100644
> --- a/lib/crypto/chacha.c
> +++ b/lib/crypto/chacha.c
> @@ -11,12 +11,13 @@
> #include <linux/bitops.h>
> #include <linux/string.h>
> #include <linux/unaligned.h>
> #include <crypto/chacha.h>
>
> -static void chacha_permute(u32 *x, int nrounds)
> +static void chacha_permute(struct chacha_state *state, int nrounds)
> {
> + u32 *x = state->x;
> int i;
>
> /* whitelist the allowed round counts */
> WARN_ON_ONCE(nrounds != 20 && nrounds != 12);
>
> @@ -63,52 +64,54 @@ static void chacha_permute(u32 *x, int nrounds)
> }
> }
>
> /**
> * chacha_block_generic - generate one keystream block and increment block counter
> - * @state: input state matrix (16 32-bit words)
> + * @state: input state matrix
> * @stream: output keystream block (64 bytes)
> * @nrounds: number of rounds (20 or 12; 20 is recommended)
> *
> * This is the ChaCha core, a function from 64-byte strings to 64-byte strings.
> * The caller has already converted the endianness of the input. This function
> * also handles incrementing the block counter in the input matrix.
> */
> -void chacha_block_generic(u32 *state, u8 *stream, int nrounds)
> +void chacha_block_generic(struct chacha_state *state, u8 *stream, int nrounds)
> {
> - u32 x[16];
> + struct chacha_state permuted_state;
> int i;
>
> - memcpy(x, state, 64);
> + memcpy(permuted_state.x, state->x, 64);
>
> - chacha_permute(x, nrounds);
> + chacha_permute(&permuted_state, nrounds);
>
> - for (i = 0; i < ARRAY_SIZE(x); i++)
> - put_unaligned_le32(x[i] + state[i], &stream[i * sizeof(u32)]);
> + for (i = 0; i < ARRAY_SIZE(state->x); i++)
> + put_unaligned_le32(permuted_state.x[i] + state->x[i],
> + &stream[i * sizeof(u32)]);
>
> - state[12]++;
> + state->x[12]++;
> }
> EXPORT_SYMBOL(chacha_block_generic);
>
> /**
> * hchacha_block_generic - abbreviated ChaCha core, for XChaCha
> - * @state: input state matrix (16 32-bit words)
> + * @state: input state matrix
> * @stream: output (8 32-bit words)
> * @nrounds: number of rounds (20 or 12; 20 is recommended)
> *
> * HChaCha is the ChaCha equivalent of HSalsa and is an intermediate step
> * towards XChaCha (see https://cr.yp.to/snuffle/xsalsa-20081128.pdf). HChaCha
> * skips the final addition of the initial state, and outputs only certain words
> * of the state. It should not be used for streaming directly.
> */
> -void hchacha_block_generic(const u32 *state, u32 *stream, int nrounds)
> +void hchacha_block_generic(const struct chacha_state *state,
> + u32 *stream, int nrounds)
> {
> - u32 x[16];
> + struct chacha_state permuted_state;
>
> - memcpy(x, state, 64);
> + memcpy(permuted_state.x, state->x, 64);
>
> - chacha_permute(x, nrounds);
> + chacha_permute(&permuted_state, nrounds);
>
> - memcpy(&stream[0], &x[0], 16);
> - memcpy(&stream[4], &x[12], 16);
> + memcpy(&stream[0], &permuted_state.x[0], 16);
> + memcpy(&stream[4], &permuted_state.x[12], 16);
> }
> EXPORT_SYMBOL(hchacha_block_generic);
> diff --git a/lib/crypto/chacha20poly1305-selftest.c b/lib/crypto/chacha20poly1305-selftest.c
> index 2ea61c28be4f..e4c85bc5a6d7 100644
> --- a/lib/crypto/chacha20poly1305-selftest.c
> +++ b/lib/crypto/chacha20poly1305-selftest.c
> @@ -8830,11 +8830,11 @@ chacha20poly1305_encrypt_bignonce(u8 *dst, const u8 *src, const size_t src_len,
> const u8 nonce[12],
> const u8 key[CHACHA20POLY1305_KEY_SIZE])
> {
> const u8 *pad0 = page_address(ZERO_PAGE(0));
> struct poly1305_desc_ctx poly1305_state;
> - u32 chacha20_state[CHACHA_STATE_WORDS];
> + struct chacha_state chacha20_state;
> union {
> u8 block0[POLY1305_KEY_SIZE];
> __le64 lens[2];
> } b = {{ 0 }};
> u8 bottom_row[16] = { 0 };
> @@ -8842,16 +8842,16 @@ chacha20poly1305_encrypt_bignonce(u8 *dst, const u8 *src, const size_t src_len,
> int i;
>
> memcpy(&bottom_row[4], nonce, 12);
> for (i = 0; i < 8; ++i)
> le_key[i] = get_unaligned_le32(key + sizeof(le_key[i]) * i);
> - chacha_init(chacha20_state, le_key, bottom_row);
> - chacha20_crypt(chacha20_state, b.block0, b.block0, sizeof(b.block0));
> + chacha_init(&chacha20_state, le_key, bottom_row);
> + chacha20_crypt(&chacha20_state, b.block0, b.block0, sizeof(b.block0));
> poly1305_init(&poly1305_state, b.block0);
> poly1305_update(&poly1305_state, ad, ad_len);
> poly1305_update(&poly1305_state, pad0, (0x10 - ad_len) & 0xf);
> - chacha20_crypt(chacha20_state, dst, src, src_len);
> + chacha20_crypt(&chacha20_state, dst, src, src_len);
> poly1305_update(&poly1305_state, dst, src_len);
> poly1305_update(&poly1305_state, pad0, (0x10 - src_len) & 0xf);
> b.lens[0] = cpu_to_le64(ad_len);
> b.lens[1] = cpu_to_le64(src_len);
> poly1305_update(&poly1305_state, (u8 *)b.lens, sizeof(b.lens));
> diff --git a/lib/crypto/chacha20poly1305.c b/lib/crypto/chacha20poly1305.c
> index 9cfa886f1f89..ed81f0658956 100644
> --- a/lib/crypto/chacha20poly1305.c
> +++ b/lib/crypto/chacha20poly1305.c
> @@ -30,11 +30,12 @@ static void chacha_load_key(u32 *k, const u8 *in)
> k[5] = get_unaligned_le32(in + 20);
> k[6] = get_unaligned_le32(in + 24);
> k[7] = get_unaligned_le32(in + 28);
> }
>
> -static void xchacha_init(u32 *chacha_state, const u8 *key, const u8 *nonce)
> +static void xchacha_init(struct chacha_state *chacha_state,
> + const u8 *key, const u8 *nonce)
> {
> u32 k[CHACHA_KEY_WORDS];
> u8 iv[CHACHA_IV_SIZE];
>
> memset(iv, 0, 8);
> @@ -52,11 +53,12 @@ static void xchacha_init(u32 *chacha_state, const u8 *key, const u8 *nonce)
> memzero_explicit(iv, sizeof(iv));
> }
>
> static void
> __chacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
> - const u8 *ad, const size_t ad_len, u32 *chacha_state)
> + const u8 *ad, const size_t ad_len,
> + struct chacha_state *chacha_state)
> {
> const u8 *pad0 = page_address(ZERO_PAGE(0));
> struct poly1305_desc_ctx poly1305_state;
> union {
> u8 block0[POLY1305_KEY_SIZE];
> @@ -80,30 +82,31 @@ __chacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
> b.lens[1] = cpu_to_le64(src_len);
> poly1305_update(&poly1305_state, (u8 *)b.lens, sizeof(b.lens));
>
> poly1305_final(&poly1305_state, dst + src_len);
>
> - memzero_explicit(chacha_state, CHACHA_STATE_WORDS * sizeof(u32));
> + memzero_explicit(chacha_state, sizeof(*chacha_state));
> memzero_explicit(&b, sizeof(b));
> }
>
> void chacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
> const u8 *ad, const size_t ad_len,
> const u64 nonce,
> const u8 key[CHACHA20POLY1305_KEY_SIZE])
> {
> - u32 chacha_state[CHACHA_STATE_WORDS];
> + struct chacha_state chacha_state;
> u32 k[CHACHA_KEY_WORDS];
> __le64 iv[2];
>
> chacha_load_key(k, key);
>
> iv[0] = 0;
> iv[1] = cpu_to_le64(nonce);
>
> - chacha_init(chacha_state, k, (u8 *)iv);
> - __chacha20poly1305_encrypt(dst, src, src_len, ad, ad_len, chacha_state);
> + chacha_init(&chacha_state, k, (u8 *)iv);
> + __chacha20poly1305_encrypt(dst, src, src_len, ad, ad_len,
> + &chacha_state);
>
> memzero_explicit(iv, sizeof(iv));
> memzero_explicit(k, sizeof(k));
> }
> EXPORT_SYMBOL(chacha20poly1305_encrypt);
> @@ -111,20 +114,22 @@ EXPORT_SYMBOL(chacha20poly1305_encrypt);
> void xchacha20poly1305_encrypt(u8 *dst, const u8 *src, const size_t src_len,
> const u8 *ad, const size_t ad_len,
> const u8 nonce[XCHACHA20POLY1305_NONCE_SIZE],
> const u8 key[CHACHA20POLY1305_KEY_SIZE])
> {
> - u32 chacha_state[CHACHA_STATE_WORDS];
> + struct chacha_state chacha_state;
>
> - xchacha_init(chacha_state, key, nonce);
> - __chacha20poly1305_encrypt(dst, src, src_len, ad, ad_len, chacha_state);
> + xchacha_init(&chacha_state, key, nonce);
> + __chacha20poly1305_encrypt(dst, src, src_len, ad, ad_len,
> + &chacha_state);
> }
> EXPORT_SYMBOL(xchacha20poly1305_encrypt);
>
> static bool
> __chacha20poly1305_decrypt(u8 *dst, const u8 *src, const size_t src_len,
> - const u8 *ad, const size_t ad_len, u32 *chacha_state)
> + const u8 *ad, const size_t ad_len,
> + struct chacha_state *chacha_state)
> {
> const u8 *pad0 = page_address(ZERO_PAGE(0));
> struct poly1305_desc_ctx poly1305_state;
> size_t dst_len;
> int ret;
> @@ -167,25 +172,25 @@ __chacha20poly1305_decrypt(u8 *dst, const u8 *src, const size_t src_len,
> bool chacha20poly1305_decrypt(u8 *dst, const u8 *src, const size_t src_len,
> const u8 *ad, const size_t ad_len,
> const u64 nonce,
> const u8 key[CHACHA20POLY1305_KEY_SIZE])
> {
> - u32 chacha_state[CHACHA_STATE_WORDS];
> + struct chacha_state chacha_state;
> u32 k[CHACHA_KEY_WORDS];
> __le64 iv[2];
> bool ret;
>
> chacha_load_key(k, key);
>
> iv[0] = 0;
> iv[1] = cpu_to_le64(nonce);
>
> - chacha_init(chacha_state, k, (u8 *)iv);
> + chacha_init(&chacha_state, k, (u8 *)iv);
> ret = __chacha20poly1305_decrypt(dst, src, src_len, ad, ad_len,
> - chacha_state);
> + &chacha_state);
>
> - memzero_explicit(chacha_state, sizeof(chacha_state));
> + memzero_explicit(&chacha_state, sizeof(chacha_state));
> memzero_explicit(iv, sizeof(iv));
> memzero_explicit(k, sizeof(k));
> return ret;
> }
> EXPORT_SYMBOL(chacha20poly1305_decrypt);
> @@ -193,15 +198,15 @@ EXPORT_SYMBOL(chacha20poly1305_decrypt);
> bool xchacha20poly1305_decrypt(u8 *dst, const u8 *src, const size_t src_len,
> const u8 *ad, const size_t ad_len,
> const u8 nonce[XCHACHA20POLY1305_NONCE_SIZE],
> const u8 key[CHACHA20POLY1305_KEY_SIZE])
> {
> - u32 chacha_state[CHACHA_STATE_WORDS];
> + struct chacha_state chacha_state;
>
> - xchacha_init(chacha_state, key, nonce);
> + xchacha_init(&chacha_state, key, nonce);
> return __chacha20poly1305_decrypt(dst, src, src_len, ad, ad_len,
> - chacha_state);
> + &chacha_state);
> }
> EXPORT_SYMBOL(xchacha20poly1305_decrypt);
>
> static
> bool chacha20poly1305_crypt_sg_inplace(struct scatterlist *src,
> @@ -211,11 +216,11 @@ bool chacha20poly1305_crypt_sg_inplace(struct scatterlist *src,
> const u8 key[CHACHA20POLY1305_KEY_SIZE],
> int encrypt)
> {
> const u8 *pad0 = page_address(ZERO_PAGE(0));
> struct poly1305_desc_ctx poly1305_state;
> - u32 chacha_state[CHACHA_STATE_WORDS];
> + struct chacha_state chacha_state;
> struct sg_mapping_iter miter;
> size_t partial = 0;
> unsigned int flags;
> bool ret = true;
> int sl;
> @@ -238,12 +243,12 @@ bool chacha20poly1305_crypt_sg_inplace(struct scatterlist *src,
> chacha_load_key(b.k, key);
>
> b.iv[0] = 0;
> b.iv[1] = cpu_to_le64(nonce);
>
> - chacha_init(chacha_state, b.k, (u8 *)b.iv);
> - chacha20_crypt(chacha_state, b.block0, pad0, sizeof(b.block0));
> + chacha_init(&chacha_state, b.k, (u8 *)b.iv);
> + chacha20_crypt(&chacha_state, b.block0, pad0, sizeof(b.block0));
> poly1305_init(&poly1305_state, b.block0);
>
> if (unlikely(ad_len)) {
> poly1305_update(&poly1305_state, ad, ad_len);
> if (ad_len & 0xf)
> @@ -274,17 +279,17 @@ bool chacha20poly1305_crypt_sg_inplace(struct scatterlist *src,
> if (likely(length >= CHACHA_BLOCK_SIZE || length == sl)) {
> size_t l = length;
>
> if (unlikely(length < sl))
> l &= ~(CHACHA_BLOCK_SIZE - 1);
> - chacha20_crypt(chacha_state, addr, addr, l);
> + chacha20_crypt(&chacha_state, addr, addr, l);
> addr += l;
> length -= l;
> }
>
> if (unlikely(length > 0)) {
> - chacha20_crypt(chacha_state, b.chacha_stream, pad0,
> + chacha20_crypt(&chacha_state, b.chacha_stream, pad0,
> CHACHA_BLOCK_SIZE);
> crypto_xor(addr, b.chacha_stream, length);
> partial = length;
> }
>
> @@ -321,11 +326,11 @@ bool chacha20poly1305_crypt_sg_inplace(struct scatterlist *src,
> sizeof(b.mac[1]), src_len, !encrypt);
> ret = encrypt ||
> !crypto_memneq(b.mac[0], b.mac[1], POLY1305_DIGEST_SIZE);
> }
>
> - memzero_explicit(chacha_state, sizeof(chacha_state));
> + memzero_explicit(&chacha_state, sizeof(chacha_state));
> memzero_explicit(&b, sizeof(b));
>
> return ret;
> }
>
> diff --git a/lib/crypto/libchacha.c b/lib/crypto/libchacha.c
> index cc1be0496eb9..ebcca381e248 100644
> --- a/lib/crypto/libchacha.c
> +++ b/lib/crypto/libchacha.c
> @@ -10,11 +10,11 @@
> #include <linux/module.h>
>
> #include <crypto/algapi.h> // for crypto_xor_cpy
> #include <crypto/chacha.h>
>
> -void chacha_crypt_generic(u32 *state, u8 *dst, const u8 *src,
> +void chacha_crypt_generic(struct chacha_state *state, u8 *dst, const u8 *src,
> unsigned int bytes, int nrounds)
> {
> /* aligned to potentially speed up crypto_xor() */
> u8 stream[CHACHA_BLOCK_SIZE] __aligned(sizeof(long));
>
> diff --git a/tools/testing/crypto/chacha20-s390/test-cipher.c b/tools/testing/crypto/chacha20-s390/test-cipher.c
> index 35ea65c54ffa..827507844e8f 100644
> --- a/tools/testing/crypto/chacha20-s390/test-cipher.c
> +++ b/tools/testing/crypto/chacha20-s390/test-cipher.c
> @@ -48,11 +48,11 @@ struct skcipher_def {
> };
>
> /* Perform cipher operations with the chacha lib */
> static int test_lib_chacha(u8 *revert, u8 *cipher, u8 *plain)
> {
> - u32 chacha_state[CHACHA_STATE_WORDS];
> + struct chacha_state chacha_state;
> u8 iv[16], key[32];
> u64 start, end;
>
> memset(key, 'X', sizeof(key));
> memset(iv, 'I', sizeof(iv));
> @@ -64,14 +64,14 @@ static int test_lib_chacha(u8 *revert, u8 *cipher, u8 *plain)
> print_hex_dump(KERN_INFO, "iv: ", DUMP_PREFIX_OFFSET,
> 16, 1, iv, 16, 1);
> }
>
> /* Encrypt */
> - chacha_init(chacha_state, (u32 *)key, iv);
> + chacha_init(&chacha_state, (u32 *)key, iv);
>
> start = ktime_get_ns();
> - chacha_crypt_arch(chacha_state, cipher, plain, data_size, 20);
> + chacha_crypt_arch(&chacha_state, cipher, plain, data_size, 20);
> end = ktime_get_ns();
>
>
> if (debug)
> print_hex_dump(KERN_INFO, "encr:", DUMP_PREFIX_OFFSET,
> @@ -79,14 +79,14 @@ static int test_lib_chacha(u8 *revert, u8 *cipher, u8 *plain)
> (data_size > 64 ? 64 : data_size), 1);
>
> pr_info("lib encryption took: %lld nsec", end - start);
>
> /* Decrypt */
> - chacha_init(chacha_state, (u32 *)key, iv);
> + chacha_init(&chacha_state, (u32 *)key, iv);
>
> start = ktime_get_ns();
> - chacha_crypt_arch(chacha_state, revert, cipher, data_size, 20);
> + chacha_crypt_arch(&chacha_state, revert, cipher, data_size, 20);
> end = ktime_get_ns();
>
> if (debug)
> print_hex_dump(KERN_INFO, "decr:", DUMP_PREFIX_OFFSET,
> 16, 1, revert,
> --
> 2.49.0
>
^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: [PATCH 0/4] crypto: lib/chacha - improve type safety
2025-05-05 18:18 [PATCH 0/4] crypto: lib/chacha - improve type safety Eric Biggers
` (3 preceding siblings ...)
2025-05-05 18:18 ` [PATCH 4/4] crypto: lib/chacha - add array bounds to function prototypes Eric Biggers
@ 2025-05-12 5:45 ` Herbert Xu
4 siblings, 0 replies; 7+ messages in thread
From: Herbert Xu @ 2025-05-12 5:45 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, linux-kernel, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, linux-bcachefs, Jason,
tytso
Eric Biggers <ebiggers@kernel.org> wrote:
> This series can also be retrieved from:
>
> git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git chacha-prototypes-v1
>
> This series improves type safety and clarity in the ChaCha functions by
> strongly typing the state array and adding explicit array bounds for
> other fixed-length arrays. No functional changes.
>
> Eric Biggers (4):
> crypto: lib/chacha - strongly type the ChaCha state
> crypto: lib/chacha - use struct assignment to copy state
> crypto: lib/chacha - add strongly-typed state zeroization
> crypto: lib/chacha - add array bounds to function prototypes
>
> arch/arm/lib/crypto/chacha-glue.c | 34 ++++----
> arch/arm/lib/crypto/chacha-scalar-core.S | 5 +-
> arch/arm64/lib/crypto/chacha-neon-glue.c | 27 ++++---
> arch/mips/lib/crypto/chacha-glue.c | 6 +-
> arch/powerpc/lib/crypto/chacha-p10-glue.c | 17 ++--
> arch/powerpc/lib/crypto/chacha-p10le-8x.S | 6 +-
> arch/riscv/lib/crypto/chacha-riscv64-glue.c | 9 ++-
> arch/riscv/lib/crypto/chacha-riscv64-zvkb.S | 10 +--
> arch/s390/lib/crypto/chacha-glue.c | 13 +--
> arch/x86/lib/crypto/chacha_glue.c | 62 ++++++++------
> crypto/chacha.c | 20 ++---
> drivers/char/random.c | 41 +++++-----
> fs/bcachefs/checksum.c | 18 ++---
> include/crypto/chacha.h | 80 ++++++++++++-------
> lib/crypto/chacha.c | 40 +++++-----
> lib/crypto/chacha20poly1305-selftest.c | 8 +-
> lib/crypto/chacha20poly1305.c | 53 ++++++------
> lib/crypto/libchacha.c | 2 +-
> .../crypto/chacha20-s390/test-cipher.c | 10 +--
> 19 files changed, 253 insertions(+), 208 deletions(-)
>
>
> base-commit: 64745a9ca890ed60d78162ec511e1983e1946d73
All applied. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2025-05-12 5:46 UTC | newest]
Thread overview: 7+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-05-05 18:18 [PATCH 0/4] crypto: lib/chacha - improve type safety Eric Biggers
2025-05-05 18:18 ` [PATCH 1/4] crypto: lib/chacha - strongly type the ChaCha state Eric Biggers
2025-05-06 16:09 ` Kent Overstreet
2025-05-05 18:18 ` [PATCH 2/4] crypto: lib/chacha - use struct assignment to copy state Eric Biggers
2025-05-05 18:18 ` [PATCH 3/4] crypto: lib/chacha - add strongly-typed state zeroization Eric Biggers
2025-05-05 18:18 ` [PATCH 4/4] crypto: lib/chacha - add array bounds to function prototypes Eric Biggers
2025-05-12 5:45 ` [PATCH 0/4] crypto: lib/chacha - improve type safety Herbert Xu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).