* [PATCH v4 00/13] Architecture-optimized SHA-256 library API
@ 2025-04-28 17:00 Eric Biggers
2025-04-28 17:00 ` [PATCH v4 01/13] crypto: sha256 - support arch-optimized lib and expose through shash Eric Biggers
` (13 more replies)
0 siblings, 14 replies; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
This is based on cryptodev commit 2dfc7cd74a5e062a. It can also be
retrieved from:
git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git sha256-lib-v4
Following the example of several other algorithms (e.g. CRC32, ChaCha,
Poly1305, BLAKE2s), this series refactors the kernel's existing
architecture-optimized SHA-256 code to be available via the library API,
instead of just via the crypto_shash API as it was before. It also
reimplements the SHA-256 crypto_shash API on top of the library API.
This makes it possible to use the SHA-256 library in
performance-critical cases. The new design is also much simpler, with a
negative diffstat of almost 1200 lines. Finally, this also fixes the
longstanding issue where the arch-optimized SHA-256 was disabled by
default, so people often forgot to enable it.
For now the SHA-256 library is well-covered by the crypto_shash
self-tests, but I plan to add a test for the library directly later.
I've fully tested this series on arm, arm64, riscv, and x86. On mips,
powerpc, s390, and sparc I've only been able to partially test it, since
QEMU does not support the SHA-256 instructions on those platforms. If
anyone with access to a mips, powerpc, s390, or sparc system that has
SHA-256 instructions can verify that the crypto self-tests still pass,
Changed v1 => v4:
- Moved sha256_generic_blocks() into its own module to avoid a
circular module dependency.
- Added Ard's Reviewed-by tags.
- Rebased onto cryptodev.
Eric Biggers (13):
crypto: sha256 - support arch-optimized lib and expose through shash
crypto: arm/sha256 - implement library instead of shash
crypto: arm64/sha256 - remove obsolete chunking logic
crypto: arm64/sha256 - implement library instead of shash
crypto: mips/sha256 - implement library instead of shash
crypto: powerpc/sha256 - implement library instead of shash
crypto: riscv/sha256 - implement library instead of shash
crypto: s390/sha256 - implement library instead of shash
crypto: sparc - move opcodes.h into asm directory
crypto: sparc/sha256 - implement library instead of shash
crypto: x86/sha256 - implement library instead of shash
crypto: sha256 - remove sha256_base.h
crypto: lib/sha256 - improve function prototypes
arch/arm/configs/exynos_defconfig | 1 -
arch/arm/configs/milbeaut_m10v_defconfig | 1 -
arch/arm/configs/multi_v7_defconfig | 1 -
arch/arm/configs/omap2plus_defconfig | 1 -
arch/arm/configs/pxa_defconfig | 1 -
arch/arm/crypto/Kconfig | 21 -
arch/arm/crypto/Makefile | 8 +-
arch/arm/crypto/sha2-ce-glue.c | 87 ----
arch/arm/crypto/sha256_glue.c | 107 -----
arch/arm/crypto/sha256_glue.h | 9 -
arch/arm/crypto/sha256_neon_glue.c | 75 ---
arch/arm/lib/crypto/.gitignore | 1 +
arch/arm/lib/crypto/Kconfig | 6 +
arch/arm/lib/crypto/Makefile | 8 +-
arch/arm/{ => lib}/crypto/sha256-armv4.pl | 0
.../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 10 +-
arch/arm/lib/crypto/sha256.c | 64 +++
arch/arm64/configs/defconfig | 1 -
arch/arm64/crypto/Kconfig | 19 -
arch/arm64/crypto/Makefile | 13 +-
arch/arm64/crypto/sha2-ce-glue.c | 138 ------
arch/arm64/crypto/sha256-glue.c | 171 -------
arch/arm64/lib/crypto/.gitignore | 1 +
arch/arm64/lib/crypto/Kconfig | 5 +
arch/arm64/lib/crypto/Makefile | 9 +-
.../crypto/sha2-armv8.pl} | 0
.../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 36 +-
arch/arm64/lib/crypto/sha256.c | 75 +++
arch/mips/cavium-octeon/Kconfig | 6 +
.../mips/cavium-octeon/crypto/octeon-sha256.c | 135 ++----
arch/mips/configs/cavium_octeon_defconfig | 1 -
arch/mips/crypto/Kconfig | 10 -
arch/powerpc/crypto/Kconfig | 11 -
arch/powerpc/crypto/Makefile | 2 -
arch/powerpc/crypto/sha256-spe-glue.c | 128 ------
arch/powerpc/lib/crypto/Kconfig | 6 +
arch/powerpc/lib/crypto/Makefile | 3 +
.../powerpc/{ => lib}/crypto/sha256-spe-asm.S | 0
arch/powerpc/lib/crypto/sha256.c | 70 +++
arch/riscv/crypto/Kconfig | 11 -
arch/riscv/crypto/Makefile | 3 -
arch/riscv/crypto/sha256-riscv64-glue.c | 125 -----
arch/riscv/lib/crypto/Kconfig | 7 +
arch/riscv/lib/crypto/Makefile | 3 +
.../sha256-riscv64-zvknha_or_zvknhb-zvkb.S | 4 +-
arch/riscv/lib/crypto/sha256.c | 62 +++
arch/s390/configs/debug_defconfig | 1 -
arch/s390/configs/defconfig | 1 -
arch/s390/crypto/Kconfig | 10 -
arch/s390/crypto/Makefile | 1 -
arch/s390/crypto/sha256_s390.c | 144 ------
arch/s390/lib/crypto/Kconfig | 6 +
arch/s390/lib/crypto/Makefile | 2 +
arch/s390/lib/crypto/sha256.c | 47 ++
arch/sparc/crypto/Kconfig | 10 -
arch/sparc/crypto/Makefile | 2 -
arch/sparc/crypto/aes_asm.S | 3 +-
arch/sparc/crypto/aes_glue.c | 3 +-
arch/sparc/crypto/camellia_asm.S | 3 +-
arch/sparc/crypto/camellia_glue.c | 3 +-
arch/sparc/crypto/des_asm.S | 3 +-
arch/sparc/crypto/des_glue.c | 3 +-
arch/sparc/crypto/md5_asm.S | 3 +-
arch/sparc/crypto/md5_glue.c | 3 +-
arch/sparc/crypto/sha1_asm.S | 3 +-
arch/sparc/crypto/sha1_glue.c | 3 +-
arch/sparc/crypto/sha256_glue.c | 129 ------
arch/sparc/crypto/sha512_asm.S | 3 +-
arch/sparc/crypto/sha512_glue.c | 3 +-
arch/sparc/{crypto => include/asm}/opcodes.h | 6 +-
arch/sparc/lib/Makefile | 1 +
arch/sparc/lib/crc32c_asm.S | 3 +-
arch/sparc/lib/crypto/Kconfig | 8 +
arch/sparc/lib/crypto/Makefile | 4 +
arch/sparc/lib/crypto/sha256.c | 64 +++
arch/sparc/{ => lib}/crypto/sha256_asm.S | 5 +-
arch/x86/crypto/Kconfig | 14 -
arch/x86/crypto/Makefile | 3 -
arch/x86/crypto/sha256_ssse3_glue.c | 432 ------------------
arch/x86/lib/crypto/Kconfig | 7 +
arch/x86/lib/crypto/Makefile | 3 +
arch/x86/{ => lib}/crypto/sha256-avx-asm.S | 12 +-
arch/x86/{ => lib}/crypto/sha256-avx2-asm.S | 12 +-
.../crypto/sha256-ni-asm.S} | 36 +-
arch/x86/{ => lib}/crypto/sha256-ssse3-asm.S | 14 +-
arch/x86/lib/crypto/sha256.c | 74 +++
crypto/Kconfig | 1 +
crypto/Makefile | 3 +-
crypto/sha256.c | 201 ++++++++
crypto/sha256_generic.c | 102 -----
include/crypto/internal/sha2.h | 28 ++
include/crypto/sha2.h | 23 +-
include/crypto/sha256_base.h | 180 --------
lib/crypto/Kconfig | 22 +
lib/crypto/Makefile | 3 +
lib/crypto/sha256-generic.c | 137 ++++++
lib/crypto/sha256.c | 204 ++++-----
97 files changed, 1128 insertions(+), 2319 deletions(-)
delete mode 100644 arch/arm/crypto/sha2-ce-glue.c
delete mode 100644 arch/arm/crypto/sha256_glue.c
delete mode 100644 arch/arm/crypto/sha256_glue.h
delete mode 100644 arch/arm/crypto/sha256_neon_glue.c
rename arch/arm/{ => lib}/crypto/sha256-armv4.pl (100%)
rename arch/arm/{crypto/sha2-ce-core.S => lib/crypto/sha256-ce.S} (91%)
create mode 100644 arch/arm/lib/crypto/sha256.c
delete mode 100644 arch/arm64/crypto/sha2-ce-glue.c
delete mode 100644 arch/arm64/crypto/sha256-glue.c
rename arch/arm64/{crypto/sha512-armv8.pl => lib/crypto/sha2-armv8.pl} (100%)
rename arch/arm64/{crypto/sha2-ce-core.S => lib/crypto/sha256-ce.S} (80%)
create mode 100644 arch/arm64/lib/crypto/sha256.c
delete mode 100644 arch/powerpc/crypto/sha256-spe-glue.c
rename arch/powerpc/{ => lib}/crypto/sha256-spe-asm.S (100%)
create mode 100644 arch/powerpc/lib/crypto/sha256.c
delete mode 100644 arch/riscv/crypto/sha256-riscv64-glue.c
rename arch/riscv/{ => lib}/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S (98%)
create mode 100644 arch/riscv/lib/crypto/sha256.c
delete mode 100644 arch/s390/crypto/sha256_s390.c
create mode 100644 arch/s390/lib/crypto/sha256.c
delete mode 100644 arch/sparc/crypto/sha256_glue.c
rename arch/sparc/{crypto => include/asm}/opcodes.h (96%)
create mode 100644 arch/sparc/lib/crypto/Kconfig
create mode 100644 arch/sparc/lib/crypto/Makefile
create mode 100644 arch/sparc/lib/crypto/sha256.c
rename arch/sparc/{ => lib}/crypto/sha256_asm.S (95%)
delete mode 100644 arch/x86/crypto/sha256_ssse3_glue.c
rename arch/x86/{ => lib}/crypto/sha256-avx-asm.S (98%)
rename arch/x86/{ => lib}/crypto/sha256-avx2-asm.S (98%)
rename arch/x86/{crypto/sha256_ni_asm.S => lib/crypto/sha256-ni-asm.S} (85%)
rename arch/x86/{ => lib}/crypto/sha256-ssse3-asm.S (98%)
create mode 100644 arch/x86/lib/crypto/sha256.c
create mode 100644 crypto/sha256.c
delete mode 100644 crypto/sha256_generic.c
create mode 100644 include/crypto/internal/sha2.h
delete mode 100644 include/crypto/sha256_base.h
create mode 100644 lib/crypto/sha256-generic.c
base-commit: 2dfc7cd74a5e062a5405560447517e7aab1c7341
--
2.49.0
^ permalink raw reply [flat|nested] 27+ messages in thread
* [PATCH v4 01/13] crypto: sha256 - support arch-optimized lib and expose through shash
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-04-30 3:48 ` Herbert Xu
2025-04-28 17:00 ` [PATCH v4 02/13] crypto: arm/sha256 - implement library instead of shash Eric Biggers
` (12 subsequent siblings)
13 siblings, 1 reply; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
As has been done for various other algorithms, rework the design of the
SHA-256 library to support arch-optimized implementations, and make
crypto/sha256.c expose both generic and arch-optimized shash algorithms
that wrap the library functions.
This allows users of the SHA-256 library functions to take advantage of
the arch-optimized code, and this makes it much simpler to integrate
SHA-256 for each architecture.
Note that sha256_base.h is not used in the new design. It will be
removed once all the architecture-specific code has been updated.
Move the generic block function into its own module to avoid a circular
dependency from libsha256.ko => sha256-$ARCH.ko => libsha256.ko.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
crypto/Kconfig | 1 +
crypto/Makefile | 3 +-
crypto/sha256.c | 201 +++++++++++++++++++++++++++++++++
crypto/sha256_generic.c | 102 -----------------
include/crypto/internal/sha2.h | 28 +++++
include/crypto/sha2.h | 15 +--
include/crypto/sha256_base.h | 9 +-
lib/crypto/Kconfig | 19 ++++
lib/crypto/Makefile | 3 +
lib/crypto/sha256-generic.c | 137 ++++++++++++++++++++++
lib/crypto/sha256.c | 196 ++++++++++++++------------------
11 files changed, 487 insertions(+), 227 deletions(-)
create mode 100644 crypto/sha256.c
delete mode 100644 crypto/sha256_generic.c
create mode 100644 include/crypto/internal/sha2.h
create mode 100644 lib/crypto/sha256-generic.c
diff --git a/crypto/Kconfig b/crypto/Kconfig
index 9878286d1d683..daf46053d25a5 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -992,10 +992,11 @@ config CRYPTO_SHA1
config CRYPTO_SHA256
tristate "SHA-224 and SHA-256"
select CRYPTO_HASH
select CRYPTO_LIB_SHA256
+ select CRYPTO_LIB_SHA256_GENERIC
help
SHA-224 and SHA-256 secure hash algorithms (FIPS 180, ISO/IEC 10118-3)
This is required for IPsec AH (XFRM_AH) and IPsec ESP (XFRM_ESP).
Used by the btrfs filesystem, Ceph, NFS, and SMB.
diff --git a/crypto/Makefile b/crypto/Makefile
index 5d2f2a28d8a07..2a23926b9f4f5 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -74,11 +74,12 @@ obj-$(CONFIG_CRYPTO_XCBC) += xcbc.o
obj-$(CONFIG_CRYPTO_NULL2) += crypto_null.o
obj-$(CONFIG_CRYPTO_MD4) += md4.o
obj-$(CONFIG_CRYPTO_MD5) += md5.o
obj-$(CONFIG_CRYPTO_RMD160) += rmd160.o
obj-$(CONFIG_CRYPTO_SHA1) += sha1_generic.o
-obj-$(CONFIG_CRYPTO_SHA256) += sha256_generic.o
+obj-$(CONFIG_CRYPTO_SHA256) += sha256.o
+CFLAGS_sha256.o += -DARCH=$(ARCH)
obj-$(CONFIG_CRYPTO_SHA512) += sha512_generic.o
obj-$(CONFIG_CRYPTO_SHA3) += sha3_generic.o
obj-$(CONFIG_CRYPTO_SM3_GENERIC) += sm3_generic.o
obj-$(CONFIG_CRYPTO_STREEBOG) += streebog_generic.o
obj-$(CONFIG_CRYPTO_WP512) += wp512.o
diff --git a/crypto/sha256.c b/crypto/sha256.c
new file mode 100644
index 0000000000000..1c2edcf9453dc
--- /dev/null
+++ b/crypto/sha256.c
@@ -0,0 +1,201 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Crypto API wrapper for the SHA-256 and SHA-224 library functions
+ *
+ * Copyright (c) Jean-Luc Cooke <jlcooke@certainkey.com>
+ * Copyright (c) Andrew McDonald <andrew@mcdonald.org.uk>
+ * Copyright (c) 2002 James Morris <jmorris@intercode.com.au>
+ * SHA224 Support Copyright 2007 Intel Corporation <jonathan.lynch@intel.com>
+ */
+#include <crypto/internal/hash.h>
+#include <crypto/internal/sha2.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+const u8 sha224_zero_message_hash[SHA224_DIGEST_SIZE] = {
+ 0xd1, 0x4a, 0x02, 0x8c, 0x2a, 0x3a, 0x2b, 0xc9, 0x47,
+ 0x61, 0x02, 0xbb, 0x28, 0x82, 0x34, 0xc4, 0x15, 0xa2,
+ 0xb0, 0x1f, 0x82, 0x8e, 0xa6, 0x2a, 0xc5, 0xb3, 0xe4,
+ 0x2f
+};
+EXPORT_SYMBOL_GPL(sha224_zero_message_hash);
+
+const u8 sha256_zero_message_hash[SHA256_DIGEST_SIZE] = {
+ 0xe3, 0xb0, 0xc4, 0x42, 0x98, 0xfc, 0x1c, 0x14,
+ 0x9a, 0xfb, 0xf4, 0xc8, 0x99, 0x6f, 0xb9, 0x24,
+ 0x27, 0xae, 0x41, 0xe4, 0x64, 0x9b, 0x93, 0x4c,
+ 0xa4, 0x95, 0x99, 0x1b, 0x78, 0x52, 0xb8, 0x55
+};
+EXPORT_SYMBOL_GPL(sha256_zero_message_hash);
+
+static int crypto_sha256_init(struct shash_desc *desc)
+{
+ sha256_init(shash_desc_ctx(desc));
+ return 0;
+}
+
+static int crypto_sha256_update_generic(struct shash_desc *desc, const u8 *data,
+ unsigned int len)
+{
+ sha256_update_generic(shash_desc_ctx(desc), data, len);
+ return 0;
+}
+
+static int crypto_sha256_update_arch(struct shash_desc *desc, const u8 *data,
+ unsigned int len)
+{
+ sha256_update(shash_desc_ctx(desc), data, len);
+ return 0;
+}
+
+static int crypto_sha256_final_generic(struct shash_desc *desc, u8 *out)
+{
+ sha256_final_generic(shash_desc_ctx(desc), out);
+ return 0;
+}
+
+static int crypto_sha256_final_arch(struct shash_desc *desc, u8 *out)
+{
+ sha256_final(shash_desc_ctx(desc), out);
+ return 0;
+}
+
+static int crypto_sha256_finup_generic(struct shash_desc *desc, const u8 *data,
+ unsigned int len, u8 *out)
+{
+ struct sha256_state *sctx = shash_desc_ctx(desc);
+
+ sha256_update_generic(sctx, data, len);
+ sha256_final_generic(sctx, out);
+ return 0;
+}
+
+static int crypto_sha256_finup_arch(struct shash_desc *desc, const u8 *data,
+ unsigned int len, u8 *out)
+{
+ struct sha256_state *sctx = shash_desc_ctx(desc);
+
+ sha256_update(sctx, data, len);
+ sha256_final(sctx, out);
+ return 0;
+}
+
+static int crypto_sha256_digest_generic(struct shash_desc *desc, const u8 *data,
+ unsigned int len, u8 *out)
+{
+ struct sha256_state *sctx = shash_desc_ctx(desc);
+
+ sha256_init(sctx);
+ sha256_update_generic(sctx, data, len);
+ sha256_final_generic(sctx, out);
+ return 0;
+}
+
+static int crypto_sha256_digest_arch(struct shash_desc *desc, const u8 *data,
+ unsigned int len, u8 *out)
+{
+ sha256(data, len, out);
+ return 0;
+}
+
+static int crypto_sha224_init(struct shash_desc *desc)
+{
+ sha224_init(shash_desc_ctx(desc));
+ return 0;
+}
+
+static int crypto_sha224_final_generic(struct shash_desc *desc, u8 *out)
+{
+ sha224_final_generic(shash_desc_ctx(desc), out);
+ return 0;
+}
+
+static int crypto_sha224_final_arch(struct shash_desc *desc, u8 *out)
+{
+ sha224_final(shash_desc_ctx(desc), out);
+ return 0;
+}
+
+static struct shash_alg algs[] = {
+ {
+ .base.cra_name = "sha256",
+ .base.cra_driver_name = "sha256-generic",
+ .base.cra_priority = 100,
+ .base.cra_blocksize = SHA256_BLOCK_SIZE,
+ .base.cra_module = THIS_MODULE,
+ .digestsize = SHA256_DIGEST_SIZE,
+ .init = crypto_sha256_init,
+ .update = crypto_sha256_update_generic,
+ .final = crypto_sha256_final_generic,
+ .finup = crypto_sha256_finup_generic,
+ .digest = crypto_sha256_digest_generic,
+ .descsize = sizeof(struct sha256_state),
+ },
+ {
+ .base.cra_name = "sha224",
+ .base.cra_driver_name = "sha224-generic",
+ .base.cra_priority = 100,
+ .base.cra_blocksize = SHA224_BLOCK_SIZE,
+ .base.cra_module = THIS_MODULE,
+ .digestsize = SHA224_DIGEST_SIZE,
+ .init = crypto_sha224_init,
+ .update = crypto_sha256_update_generic,
+ .final = crypto_sha224_final_generic,
+ .descsize = sizeof(struct sha256_state),
+ },
+ {
+ .base.cra_name = "sha256",
+ .base.cra_driver_name = "sha256-" __stringify(ARCH),
+ .base.cra_priority = 300,
+ .base.cra_blocksize = SHA256_BLOCK_SIZE,
+ .base.cra_module = THIS_MODULE,
+ .digestsize = SHA256_DIGEST_SIZE,
+ .init = crypto_sha256_init,
+ .update = crypto_sha256_update_arch,
+ .final = crypto_sha256_final_arch,
+ .finup = crypto_sha256_finup_arch,
+ .digest = crypto_sha256_digest_arch,
+ .descsize = sizeof(struct sha256_state),
+ },
+ {
+ .base.cra_name = "sha224",
+ .base.cra_driver_name = "sha224-" __stringify(ARCH),
+ .base.cra_priority = 300,
+ .base.cra_blocksize = SHA224_BLOCK_SIZE,
+ .base.cra_module = THIS_MODULE,
+ .digestsize = SHA224_DIGEST_SIZE,
+ .init = crypto_sha224_init,
+ .update = crypto_sha256_update_arch,
+ .final = crypto_sha224_final_arch,
+ .descsize = sizeof(struct sha256_state),
+ },
+};
+
+static unsigned int num_algs;
+
+static int __init crypto_sha256_mod_init(void)
+{
+ /* register the arch flavours only if they differ from generic */
+ num_algs = ARRAY_SIZE(algs);
+ BUILD_BUG_ON(ARRAY_SIZE(algs) % 2 != 0);
+ if (!sha256_is_arch_optimized())
+ num_algs /= 2;
+ return crypto_register_shashes(algs, ARRAY_SIZE(algs));
+}
+subsys_initcall(crypto_sha256_mod_init);
+
+static void __exit crypto_sha256_mod_exit(void)
+{
+ crypto_unregister_shashes(algs, num_algs);
+}
+module_exit(crypto_sha256_mod_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("Crypto API wrapper for the SHA-256 and SHA-224 library functions");
+
+MODULE_ALIAS_CRYPTO("sha256");
+MODULE_ALIAS_CRYPTO("sha256-generic");
+MODULE_ALIAS_CRYPTO("sha256-" __stringify(ARCH));
+MODULE_ALIAS_CRYPTO("sha224");
+MODULE_ALIAS_CRYPTO("sha224-generic");
+MODULE_ALIAS_CRYPTO("sha224-" __stringify(ARCH));
diff --git a/crypto/sha256_generic.c b/crypto/sha256_generic.c
deleted file mode 100644
index 05084e5bbaec8..0000000000000
--- a/crypto/sha256_generic.c
+++ /dev/null
@@ -1,102 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Crypto API wrapper for the generic SHA256 code from lib/crypto/sha256.c
- *
- * Copyright (c) Jean-Luc Cooke <jlcooke@certainkey.com>
- * Copyright (c) Andrew McDonald <andrew@mcdonald.org.uk>
- * Copyright (c) 2002 James Morris <jmorris@intercode.com.au>
- * SHA224 Support Copyright 2007 Intel Corporation <jonathan.lynch@intel.com>
- */
-#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
-#include <crypto/sha256_base.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-
-const u8 sha224_zero_message_hash[SHA224_DIGEST_SIZE] = {
- 0xd1, 0x4a, 0x02, 0x8c, 0x2a, 0x3a, 0x2b, 0xc9, 0x47,
- 0x61, 0x02, 0xbb, 0x28, 0x82, 0x34, 0xc4, 0x15, 0xa2,
- 0xb0, 0x1f, 0x82, 0x8e, 0xa6, 0x2a, 0xc5, 0xb3, 0xe4,
- 0x2f
-};
-EXPORT_SYMBOL_GPL(sha224_zero_message_hash);
-
-const u8 sha256_zero_message_hash[SHA256_DIGEST_SIZE] = {
- 0xe3, 0xb0, 0xc4, 0x42, 0x98, 0xfc, 0x1c, 0x14,
- 0x9a, 0xfb, 0xf4, 0xc8, 0x99, 0x6f, 0xb9, 0x24,
- 0x27, 0xae, 0x41, 0xe4, 0x64, 0x9b, 0x93, 0x4c,
- 0xa4, 0x95, 0x99, 0x1b, 0x78, 0x52, 0xb8, 0x55
-};
-EXPORT_SYMBOL_GPL(sha256_zero_message_hash);
-
-static void sha256_block(struct crypto_sha256_state *sctx, const u8 *input,
- int blocks)
-{
- sha256_transform_blocks(sctx, input, blocks);
-}
-
-static int crypto_sha256_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- return sha256_base_do_update_blocks(desc, data, len, sha256_block);
-}
-
-static int crypto_sha256_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *hash)
-{
- sha256_base_do_finup(desc, data, len, sha256_block);
- return sha256_base_finish(desc, hash);
-}
-
-static struct shash_alg sha256_algs[2] = { {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = crypto_sha256_update,
- .finup = crypto_sha256_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha256",
- .cra_driver_name= "sha256-generic",
- .cra_priority = 100,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-}, {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = crypto_sha256_update,
- .finup = crypto_sha256_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha224",
- .cra_driver_name= "sha224-generic",
- .cra_priority = 100,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-} };
-
-static int __init sha256_generic_mod_init(void)
-{
- return crypto_register_shashes(sha256_algs, ARRAY_SIZE(sha256_algs));
-}
-
-static void __exit sha256_generic_mod_fini(void)
-{
- crypto_unregister_shashes(sha256_algs, ARRAY_SIZE(sha256_algs));
-}
-
-subsys_initcall(sha256_generic_mod_init);
-module_exit(sha256_generic_mod_fini);
-
-MODULE_LICENSE("GPL");
-MODULE_DESCRIPTION("SHA-224 and SHA-256 Secure Hash Algorithm");
-
-MODULE_ALIAS_CRYPTO("sha224");
-MODULE_ALIAS_CRYPTO("sha224-generic");
-MODULE_ALIAS_CRYPTO("sha256");
-MODULE_ALIAS_CRYPTO("sha256-generic");
diff --git a/include/crypto/internal/sha2.h b/include/crypto/internal/sha2.h
new file mode 100644
index 0000000000000..d641c67abcbc3
--- /dev/null
+++ b/include/crypto/internal/sha2.h
@@ -0,0 +1,28 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef _CRYPTO_INTERNAL_SHA2_H
+#define _CRYPTO_INTERNAL_SHA2_H
+
+#include <crypto/sha2.h>
+
+void sha256_update_generic(struct sha256_state *sctx,
+ const u8 *data, size_t len);
+void sha256_final_generic(struct sha256_state *sctx,
+ u8 out[SHA256_DIGEST_SIZE]);
+void sha224_final_generic(struct sha256_state *sctx,
+ u8 out[SHA224_DIGEST_SIZE]);
+
+#if IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_SHA256)
+bool sha256_is_arch_optimized(void);
+#else
+static inline bool sha256_is_arch_optimized(void)
+{
+ return false;
+}
+#endif
+void sha256_blocks_generic(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+void sha256_blocks_arch(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+
+#endif /* _CRYPTO_INTERNAL_SHA2_H */
diff --git a/include/crypto/sha2.h b/include/crypto/sha2.h
index abbd882f7849f..444484d1b1cfa 100644
--- a/include/crypto/sha2.h
+++ b/include/crypto/sha2.h
@@ -11,10 +11,11 @@
#define SHA224_DIGEST_SIZE 28
#define SHA224_BLOCK_SIZE 64
#define SHA256_DIGEST_SIZE 32
#define SHA256_BLOCK_SIZE 64
+#define SHA256_STATE_WORDS 8
#define SHA384_DIGEST_SIZE 48
#define SHA384_BLOCK_SIZE 128
#define SHA512_DIGEST_SIZE 64
@@ -64,36 +65,26 @@ extern const u8 sha256_zero_message_hash[SHA256_DIGEST_SIZE];
extern const u8 sha384_zero_message_hash[SHA384_DIGEST_SIZE];
extern const u8 sha512_zero_message_hash[SHA512_DIGEST_SIZE];
struct crypto_sha256_state {
- u32 state[SHA256_DIGEST_SIZE / 4];
+ u32 state[SHA256_STATE_WORDS];
u64 count;
};
struct sha256_state {
- u32 state[SHA256_DIGEST_SIZE / 4];
+ u32 state[SHA256_STATE_WORDS];
u64 count;
u8 buf[SHA256_BLOCK_SIZE];
};
struct sha512_state {
u64 state[SHA512_DIGEST_SIZE / 8];
u64 count[2];
u8 buf[SHA512_BLOCK_SIZE];
};
-/*
- * Stand-alone implementation of the SHA256 algorithm. It is designed to
- * have as little dependencies as possible so it can be used in the
- * kexec_file purgatory. In other cases you should generally use the
- * hash APIs from include/crypto/hash.h. Especially when hashing large
- * amounts of data as those APIs may be hw-accelerated.
- *
- * For details see lib/crypto/sha256.c
- */
-
static inline void sha256_init(struct sha256_state *sctx)
{
sctx->state[0] = SHA256_H0;
sctx->state[1] = SHA256_H1;
sctx->state[2] = SHA256_H2;
diff --git a/include/crypto/sha256_base.h b/include/crypto/sha256_base.h
index 08cd5e41d4fdb..6878fb9c26c04 100644
--- a/include/crypto/sha256_base.h
+++ b/include/crypto/sha256_base.h
@@ -7,11 +7,11 @@
#ifndef _CRYPTO_SHA256_BASE_H
#define _CRYPTO_SHA256_BASE_H
#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
+#include <crypto/internal/sha2.h>
#include <linux/math.h>
#include <linux/string.h>
#include <linux/types.h>
#include <linux/unaligned.h>
@@ -172,9 +172,12 @@ static inline int sha256_base_finish(struct shash_desc *desc, u8 *out)
struct crypto_sha256_state *sctx = shash_desc_ctx(desc);
return __sha256_base_finish(sctx->state, out, digest_size);
}
-void sha256_transform_blocks(struct crypto_sha256_state *sst,
- const u8 *input, int blocks);
+static inline void sha256_transform_blocks(struct crypto_sha256_state *sst,
+ const u8 *input, int blocks)
+{
+ sha256_blocks_generic(sst->state, input, blocks);
+}
#endif /* _CRYPTO_SHA256_BASE_H */
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index af2368799579f..7fe678047939b 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -137,10 +137,29 @@ config CRYPTO_LIB_CHACHA20POLY1305
config CRYPTO_LIB_SHA1
tristate
config CRYPTO_LIB_SHA256
tristate
+ help
+ Enable the SHA-256 library interface. This interface may be fulfilled
+ by either the generic implementation or an arch-specific one, if one
+ is available and enabled.
+
+config CRYPTO_ARCH_HAVE_LIB_SHA256
+ bool
+ help
+ Declares whether the architecture provides an arch-specific
+ accelerated implementation of the SHA-256 library interface.
+
+config CRYPTO_LIB_SHA256_GENERIC
+ tristate
+ default CRYPTO_LIB_SHA256 if !CRYPTO_ARCH_HAVE_LIB_SHA256
+ help
+ This symbol can be selected by arch implementations of the SHA-256
+ library interface that require the generic code as a fallback, e.g.,
+ for SIMD implementations. If no arch specific implementation is
+ enabled, this implementation serves the users of CRYPTO_LIB_SHA256.
config CRYPTO_LIB_SM3
tristate
if !KMSAN # avoid false positives from assembly
diff --git a/lib/crypto/Makefile b/lib/crypto/Makefile
index 4dd62bc5bee3d..71d3d05d666a2 100644
--- a/lib/crypto/Makefile
+++ b/lib/crypto/Makefile
@@ -49,10 +49,13 @@ obj-$(CONFIG_CRYPTO_LIB_SHA1) += libsha1.o
libsha1-y := sha1.o
obj-$(CONFIG_CRYPTO_LIB_SHA256) += libsha256.o
libsha256-y := sha256.o
+obj-$(CONFIG_CRYPTO_LIB_SHA256_GENERIC) += libsha256-generic.o
+libsha256-generic-y := sha256-generic.o
+
ifneq ($(CONFIG_CRYPTO_MANAGER_DISABLE_TESTS),y)
libblake2s-y += blake2s-selftest.o
libchacha20poly1305-y += chacha20poly1305-selftest.o
libcurve25519-y += curve25519-selftest.o
endif
diff --git a/lib/crypto/sha256-generic.c b/lib/crypto/sha256-generic.c
new file mode 100644
index 0000000000000..a16ad4f25ebb7
--- /dev/null
+++ b/lib/crypto/sha256-generic.c
@@ -0,0 +1,137 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * SHA-256, as specified in
+ * http://csrc.nist.gov/groups/STM/cavp/documents/shs/sha256-384-512.pdf
+ *
+ * SHA-256 code by Jean-Luc Cooke <jlcooke@certainkey.com>.
+ *
+ * Copyright (c) Jean-Luc Cooke <jlcooke@certainkey.com>
+ * Copyright (c) Andrew McDonald <andrew@mcdonald.org.uk>
+ * Copyright (c) 2002 James Morris <jmorris@intercode.com.au>
+ * Copyright (c) 2014 Red Hat Inc.
+ */
+
+#include <crypto/internal/sha2.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/string.h>
+#include <linux/unaligned.h>
+
+static const u32 SHA256_K[] = {
+ 0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5,
+ 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
+ 0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3,
+ 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
+ 0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc,
+ 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
+ 0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7,
+ 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
+ 0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13,
+ 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
+ 0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3,
+ 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
+ 0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5,
+ 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
+ 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208,
+ 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2,
+};
+
+static inline u32 Ch(u32 x, u32 y, u32 z)
+{
+ return z ^ (x & (y ^ z));
+}
+
+static inline u32 Maj(u32 x, u32 y, u32 z)
+{
+ return (x & y) | (z & (x | y));
+}
+
+#define e0(x) (ror32(x, 2) ^ ror32(x, 13) ^ ror32(x, 22))
+#define e1(x) (ror32(x, 6) ^ ror32(x, 11) ^ ror32(x, 25))
+#define s0(x) (ror32(x, 7) ^ ror32(x, 18) ^ (x >> 3))
+#define s1(x) (ror32(x, 17) ^ ror32(x, 19) ^ (x >> 10))
+
+static inline void LOAD_OP(int I, u32 *W, const u8 *input)
+{
+ W[I] = get_unaligned_be32((__u32 *)input + I);
+}
+
+static inline void BLEND_OP(int I, u32 *W)
+{
+ W[I] = s1(W[I-2]) + W[I-7] + s0(W[I-15]) + W[I-16];
+}
+
+#define SHA256_ROUND(i, a, b, c, d, e, f, g, h) do { \
+ u32 t1, t2; \
+ t1 = h + e1(e) + Ch(e, f, g) + SHA256_K[i] + W[i]; \
+ t2 = e0(a) + Maj(a, b, c); \
+ d += t1; \
+ h = t1 + t2; \
+} while (0)
+
+static void sha256_block_generic(u32 state[SHA256_STATE_WORDS],
+ const u8 *input, u32 W[64])
+{
+ u32 a, b, c, d, e, f, g, h;
+ int i;
+
+ /* load the input */
+ for (i = 0; i < 16; i += 8) {
+ LOAD_OP(i + 0, W, input);
+ LOAD_OP(i + 1, W, input);
+ LOAD_OP(i + 2, W, input);
+ LOAD_OP(i + 3, W, input);
+ LOAD_OP(i + 4, W, input);
+ LOAD_OP(i + 5, W, input);
+ LOAD_OP(i + 6, W, input);
+ LOAD_OP(i + 7, W, input);
+ }
+
+ /* now blend */
+ for (i = 16; i < 64; i += 8) {
+ BLEND_OP(i + 0, W);
+ BLEND_OP(i + 1, W);
+ BLEND_OP(i + 2, W);
+ BLEND_OP(i + 3, W);
+ BLEND_OP(i + 4, W);
+ BLEND_OP(i + 5, W);
+ BLEND_OP(i + 6, W);
+ BLEND_OP(i + 7, W);
+ }
+
+ /* load the state into our registers */
+ a = state[0]; b = state[1]; c = state[2]; d = state[3];
+ e = state[4]; f = state[5]; g = state[6]; h = state[7];
+
+ /* now iterate */
+ for (i = 0; i < 64; i += 8) {
+ SHA256_ROUND(i + 0, a, b, c, d, e, f, g, h);
+ SHA256_ROUND(i + 1, h, a, b, c, d, e, f, g);
+ SHA256_ROUND(i + 2, g, h, a, b, c, d, e, f);
+ SHA256_ROUND(i + 3, f, g, h, a, b, c, d, e);
+ SHA256_ROUND(i + 4, e, f, g, h, a, b, c, d);
+ SHA256_ROUND(i + 5, d, e, f, g, h, a, b, c);
+ SHA256_ROUND(i + 6, c, d, e, f, g, h, a, b);
+ SHA256_ROUND(i + 7, b, c, d, e, f, g, h, a);
+ }
+
+ state[0] += a; state[1] += b; state[2] += c; state[3] += d;
+ state[4] += e; state[5] += f; state[6] += g; state[7] += h;
+}
+
+void sha256_blocks_generic(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks)
+{
+ u32 W[64];
+
+ do {
+ sha256_block_generic(state, data, W);
+ data += SHA256_BLOCK_SIZE;
+ } while (--nblocks);
+
+ memzero_explicit(W, sizeof(W));
+}
+EXPORT_SYMBOL_GPL(sha256_blocks_generic);
+
+MODULE_DESCRIPTION("SHA-256 Algorithm (generic implementation)");
+MODULE_LICENSE("GPL");
diff --git a/lib/crypto/sha256.c b/lib/crypto/sha256.c
index a89bab377de1a..4b19cf977ef1b 100644
--- a/lib/crypto/sha256.c
+++ b/lib/crypto/sha256.c
@@ -9,152 +9,109 @@
* Copyright (c) Andrew McDonald <andrew@mcdonald.org.uk>
* Copyright (c) 2002 James Morris <jmorris@intercode.com.au>
* Copyright (c) 2014 Red Hat Inc.
*/
-#include <linux/unaligned.h>
-#include <crypto/sha256_base.h>
+#include <crypto/internal/sha2.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/string.h>
+#include <linux/unaligned.h>
-static const u32 SHA256_K[] = {
- 0x428a2f98, 0x71374491, 0xb5c0fbcf, 0xe9b5dba5,
- 0x3956c25b, 0x59f111f1, 0x923f82a4, 0xab1c5ed5,
- 0xd807aa98, 0x12835b01, 0x243185be, 0x550c7dc3,
- 0x72be5d74, 0x80deb1fe, 0x9bdc06a7, 0xc19bf174,
- 0xe49b69c1, 0xefbe4786, 0x0fc19dc6, 0x240ca1cc,
- 0x2de92c6f, 0x4a7484aa, 0x5cb0a9dc, 0x76f988da,
- 0x983e5152, 0xa831c66d, 0xb00327c8, 0xbf597fc7,
- 0xc6e00bf3, 0xd5a79147, 0x06ca6351, 0x14292967,
- 0x27b70a85, 0x2e1b2138, 0x4d2c6dfc, 0x53380d13,
- 0x650a7354, 0x766a0abb, 0x81c2c92e, 0x92722c85,
- 0xa2bfe8a1, 0xa81a664b, 0xc24b8b70, 0xc76c51a3,
- 0xd192e819, 0xd6990624, 0xf40e3585, 0x106aa070,
- 0x19a4c116, 0x1e376c08, 0x2748774c, 0x34b0bcb5,
- 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3,
- 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208,
- 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2,
-};
-
-static inline u32 Ch(u32 x, u32 y, u32 z)
-{
- return z ^ (x & (y ^ z));
-}
-
-static inline u32 Maj(u32 x, u32 y, u32 z)
-{
- return (x & y) | (z & (x | y));
-}
-
-#define e0(x) (ror32(x, 2) ^ ror32(x, 13) ^ ror32(x, 22))
-#define e1(x) (ror32(x, 6) ^ ror32(x, 11) ^ ror32(x, 25))
-#define s0(x) (ror32(x, 7) ^ ror32(x, 18) ^ (x >> 3))
-#define s1(x) (ror32(x, 17) ^ ror32(x, 19) ^ (x >> 10))
+/*
+ * If __DISABLE_EXPORTS is defined, then this file is being compiled for a
+ * pre-boot environment. In that case, ignore the kconfig options, pull the
+ * generic code into the same translation unit, and use that only.
+ */
+#ifdef __DISABLE_EXPORTS
+#include "sha256-generic.c"
+#endif
-static inline void LOAD_OP(int I, u32 *W, const u8 *input)
+static inline void sha256_blocks(u32 state[SHA256_STATE_WORDS], const u8 *data,
+ size_t nblocks, bool force_generic)
{
- W[I] = get_unaligned_be32((__u32 *)input + I);
+#if IS_ENABLED(CONFIG_CRYPTO_ARCH_HAVE_LIB_SHA256) && !defined(__DISABLE_EXPORTS)
+ if (!force_generic)
+ return sha256_blocks_arch(state, data, nblocks);
+#endif
+ sha256_blocks_generic(state, data, nblocks);
}
-static inline void BLEND_OP(int I, u32 *W)
+static inline void __sha256_update(struct sha256_state *sctx, const u8 *data,
+ size_t len, bool force_generic)
{
- W[I] = s1(W[I-2]) + W[I-7] + s0(W[I-15]) + W[I-16];
-}
-
-#define SHA256_ROUND(i, a, b, c, d, e, f, g, h) do { \
- u32 t1, t2; \
- t1 = h + e1(e) + Ch(e, f, g) + SHA256_K[i] + W[i]; \
- t2 = e0(a) + Maj(a, b, c); \
- d += t1; \
- h = t1 + t2; \
-} while (0)
+ size_t partial = sctx->count % SHA256_BLOCK_SIZE;
-static void sha256_transform(u32 *state, const u8 *input, u32 *W)
-{
- u32 a, b, c, d, e, f, g, h;
- int i;
-
- /* load the input */
- for (i = 0; i < 16; i += 8) {
- LOAD_OP(i + 0, W, input);
- LOAD_OP(i + 1, W, input);
- LOAD_OP(i + 2, W, input);
- LOAD_OP(i + 3, W, input);
- LOAD_OP(i + 4, W, input);
- LOAD_OP(i + 5, W, input);
- LOAD_OP(i + 6, W, input);
- LOAD_OP(i + 7, W, input);
- }
+ sctx->count += len;
- /* now blend */
- for (i = 16; i < 64; i += 8) {
- BLEND_OP(i + 0, W);
- BLEND_OP(i + 1, W);
- BLEND_OP(i + 2, W);
- BLEND_OP(i + 3, W);
- BLEND_OP(i + 4, W);
- BLEND_OP(i + 5, W);
- BLEND_OP(i + 6, W);
- BLEND_OP(i + 7, W);
- }
+ if (partial + len >= SHA256_BLOCK_SIZE) {
+ size_t nblocks;
- /* load the state into our registers */
- a = state[0]; b = state[1]; c = state[2]; d = state[3];
- e = state[4]; f = state[5]; g = state[6]; h = state[7];
-
- /* now iterate */
- for (i = 0; i < 64; i += 8) {
- SHA256_ROUND(i + 0, a, b, c, d, e, f, g, h);
- SHA256_ROUND(i + 1, h, a, b, c, d, e, f, g);
- SHA256_ROUND(i + 2, g, h, a, b, c, d, e, f);
- SHA256_ROUND(i + 3, f, g, h, a, b, c, d, e);
- SHA256_ROUND(i + 4, e, f, g, h, a, b, c, d);
- SHA256_ROUND(i + 5, d, e, f, g, h, a, b, c);
- SHA256_ROUND(i + 6, c, d, e, f, g, h, a, b);
- SHA256_ROUND(i + 7, b, c, d, e, f, g, h, a);
- }
+ if (partial) {
+ size_t l = SHA256_BLOCK_SIZE - partial;
- state[0] += a; state[1] += b; state[2] += c; state[3] += d;
- state[4] += e; state[5] += f; state[6] += g; state[7] += h;
-}
+ memcpy(&sctx->buf[partial], data, l);
+ data += l;
+ len -= l;
-void sha256_transform_blocks(struct crypto_sha256_state *sst,
- const u8 *input, int blocks)
-{
- u32 W[64];
+ sha256_blocks(sctx->state, sctx->buf, 1, force_generic);
+ }
- do {
- sha256_transform(sst->state, input, W);
- input += SHA256_BLOCK_SIZE;
- } while (--blocks);
+ nblocks = len / SHA256_BLOCK_SIZE;
+ len %= SHA256_BLOCK_SIZE;
- memzero_explicit(W, sizeof(W));
+ if (nblocks) {
+ sha256_blocks(sctx->state, data, nblocks,
+ force_generic);
+ data += nblocks * SHA256_BLOCK_SIZE;
+ }
+ partial = 0;
+ }
+ if (len)
+ memcpy(&sctx->buf[partial], data, len);
}
-EXPORT_SYMBOL_GPL(sha256_transform_blocks);
void sha256_update(struct sha256_state *sctx, const u8 *data, unsigned int len)
{
- lib_sha256_base_do_update(sctx, data, len, sha256_transform_blocks);
+ __sha256_update(sctx, data, len, false);
}
EXPORT_SYMBOL(sha256_update);
-static void __sha256_final(struct sha256_state *sctx, u8 *out, int digest_size)
+static inline void __sha256_final(struct sha256_state *sctx, u8 *out,
+ size_t digest_size, bool force_generic)
{
- lib_sha256_base_do_finalize(sctx, sha256_transform_blocks);
- lib_sha256_base_finish(sctx, out, digest_size);
+ const size_t bit_offset = SHA256_BLOCK_SIZE - sizeof(__be64);
+ __be64 *bits = (__be64 *)&sctx->buf[bit_offset];
+ size_t partial = sctx->count % SHA256_BLOCK_SIZE;
+ size_t i;
+
+ sctx->buf[partial++] = 0x80;
+ if (partial > bit_offset) {
+ memset(&sctx->buf[partial], 0, SHA256_BLOCK_SIZE - partial);
+ sha256_blocks(sctx->state, sctx->buf, 1, force_generic);
+ partial = 0;
+ }
+
+ memset(&sctx->buf[partial], 0, bit_offset - partial);
+ *bits = cpu_to_be64(sctx->count << 3);
+ sha256_blocks(sctx->state, sctx->buf, 1, force_generic);
+
+ for (i = 0; i < digest_size; i += 4)
+ put_unaligned_be32(sctx->state[i / 4], out + i);
+
+ memzero_explicit(sctx, sizeof(*sctx));
}
void sha256_final(struct sha256_state *sctx, u8 *out)
{
- __sha256_final(sctx, out, 32);
+ __sha256_final(sctx, out, SHA256_DIGEST_SIZE, false);
}
EXPORT_SYMBOL(sha256_final);
void sha224_final(struct sha256_state *sctx, u8 *out)
{
- __sha256_final(sctx, out, 28);
+ __sha256_final(sctx, out, SHA224_DIGEST_SIZE, false);
}
EXPORT_SYMBOL(sha224_final);
void sha256(const u8 *data, unsigned int len, u8 *out)
{
@@ -164,7 +121,28 @@ void sha256(const u8 *data, unsigned int len, u8 *out)
sha256_update(&sctx, data, len);
sha256_final(&sctx, out);
}
EXPORT_SYMBOL(sha256);
+#if IS_ENABLED(CONFIG_CRYPTO_SHA256) && !defined(__DISABLE_EXPORTS)
+void sha256_update_generic(struct sha256_state *sctx,
+ const u8 *data, size_t len)
+{
+ __sha256_update(sctx, data, len, true);
+}
+EXPORT_SYMBOL(sha256_update_generic);
+
+void sha256_final_generic(struct sha256_state *sctx, u8 out[SHA256_DIGEST_SIZE])
+{
+ __sha256_final(sctx, out, SHA256_DIGEST_SIZE, true);
+}
+EXPORT_SYMBOL(sha256_final_generic);
+
+void sha224_final_generic(struct sha256_state *sctx, u8 out[SHA224_DIGEST_SIZE])
+{
+ __sha256_final(sctx, out, SHA224_DIGEST_SIZE, true);
+}
+EXPORT_SYMBOL(sha224_final_generic);
+#endif
+
MODULE_DESCRIPTION("SHA-256 Algorithm");
MODULE_LICENSE("GPL");
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 02/13] crypto: arm/sha256 - implement library instead of shash
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
2025-04-28 17:00 ` [PATCH v4 01/13] crypto: sha256 - support arch-optimized lib and expose through shash Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-04-28 17:00 ` [PATCH v4 03/13] crypto: arm64/sha256 - remove obsolete chunking logic Eric Biggers
` (11 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Instead of providing crypto_shash algorithms for the arch-optimized
SHA-256 code, instead implement the SHA-256 library. This is much
simpler, it makes the SHA-256 library functions be arch-optimized, and
it fixes the longstanding issue where the arch-optimized SHA-256 was
disabled by default. SHA-256 still remains available through
crypto_shash, but individual architectures no longer need to handle it.
To merge the scalar, NEON, and CE code all into one module cleanly, add
!CPU_V7M as a direct dependency of the CE code. Previously, !CPU_V7M
was only a direct dependency of the scalar and NEON code. The result is
still the same because CPU_V7M implies !KERNEL_MODE_NEON, so !CPU_V7M
was already an indirect dependency of the CE code.
To match sha256_blocks_arch(), change the type of the nblocks parameter
of the assembly functions from int to size_t. The assembly functions
actually already treated it as size_t.
While renaming the assembly files, also fix the naming quirk where
"sha2" meant sha256. (SHA-512 is also part of SHA-2.)
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/arm/configs/exynos_defconfig | 1 -
arch/arm/configs/milbeaut_m10v_defconfig | 1 -
arch/arm/configs/multi_v7_defconfig | 1 -
arch/arm/configs/omap2plus_defconfig | 1 -
arch/arm/configs/pxa_defconfig | 1 -
arch/arm/crypto/Kconfig | 21 ----
arch/arm/crypto/Makefile | 8 +-
arch/arm/crypto/sha2-ce-glue.c | 87 --------------
arch/arm/crypto/sha256_glue.c | 107 ------------------
arch/arm/crypto/sha256_glue.h | 9 --
arch/arm/crypto/sha256_neon_glue.c | 75 ------------
arch/arm/lib/crypto/.gitignore | 1 +
arch/arm/lib/crypto/Kconfig | 6 +
arch/arm/lib/crypto/Makefile | 8 +-
arch/arm/{ => lib}/crypto/sha256-armv4.pl | 0
.../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 10 +-
arch/arm/lib/crypto/sha256.c | 64 +++++++++++
17 files changed, 84 insertions(+), 317 deletions(-)
delete mode 100644 arch/arm/crypto/sha2-ce-glue.c
delete mode 100644 arch/arm/crypto/sha256_glue.c
delete mode 100644 arch/arm/crypto/sha256_glue.h
delete mode 100644 arch/arm/crypto/sha256_neon_glue.c
rename arch/arm/{ => lib}/crypto/sha256-armv4.pl (100%)
rename arch/arm/{crypto/sha2-ce-core.S => lib/crypto/sha256-ce.S} (91%)
create mode 100644 arch/arm/lib/crypto/sha256.c
diff --git a/arch/arm/configs/exynos_defconfig b/arch/arm/configs/exynos_defconfig
index 7ad48fdda1dac..244dd5dec98bd 100644
--- a/arch/arm/configs/exynos_defconfig
+++ b/arch/arm/configs/exynos_defconfig
@@ -362,11 +362,10 @@ CONFIG_CRYPTO_LZ4=m
CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CRYPTO_USER_API_SKCIPHER=m
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_USER_API_AEAD=m
CONFIG_CRYPTO_SHA1_ARM_NEON=m
-CONFIG_CRYPTO_SHA256_ARM=m
CONFIG_CRYPTO_SHA512_ARM=m
CONFIG_CRYPTO_AES_ARM_BS=m
CONFIG_CRYPTO_CHACHA20_NEON=m
CONFIG_CRYPTO_DEV_EXYNOS_RNG=y
CONFIG_CRYPTO_DEV_S5P=y
diff --git a/arch/arm/configs/milbeaut_m10v_defconfig b/arch/arm/configs/milbeaut_m10v_defconfig
index acd16204f8d7f..fce33c1eb65bf 100644
--- a/arch/arm/configs/milbeaut_m10v_defconfig
+++ b/arch/arm/configs/milbeaut_m10v_defconfig
@@ -99,11 +99,10 @@ CONFIG_CRYPTO_MANAGER=y
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_SEQIV=m
CONFIG_CRYPTO_GHASH_ARM_CE=m
CONFIG_CRYPTO_SHA1_ARM_NEON=m
CONFIG_CRYPTO_SHA1_ARM_CE=m
-CONFIG_CRYPTO_SHA2_ARM_CE=m
CONFIG_CRYPTO_SHA512_ARM=m
CONFIG_CRYPTO_AES_ARM=m
CONFIG_CRYPTO_AES_ARM_BS=m
CONFIG_CRYPTO_AES_ARM_CE=m
CONFIG_CRYPTO_CHACHA20_NEON=m
diff --git a/arch/arm/configs/multi_v7_defconfig b/arch/arm/configs/multi_v7_defconfig
index ad037c175fdb0..96178acedad0b 100644
--- a/arch/arm/configs/multi_v7_defconfig
+++ b/arch/arm/configs/multi_v7_defconfig
@@ -1299,11 +1299,10 @@ CONFIG_CRYPTO_USER_API_SKCIPHER=m
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_USER_API_AEAD=m
CONFIG_CRYPTO_GHASH_ARM_CE=m
CONFIG_CRYPTO_SHA1_ARM_NEON=m
CONFIG_CRYPTO_SHA1_ARM_CE=m
-CONFIG_CRYPTO_SHA2_ARM_CE=m
CONFIG_CRYPTO_SHA512_ARM=m
CONFIG_CRYPTO_AES_ARM=m
CONFIG_CRYPTO_AES_ARM_BS=m
CONFIG_CRYPTO_AES_ARM_CE=m
CONFIG_CRYPTO_CHACHA20_NEON=m
diff --git a/arch/arm/configs/omap2plus_defconfig b/arch/arm/configs/omap2plus_defconfig
index 113d6dfe52435..57d9e4dba29e3 100644
--- a/arch/arm/configs/omap2plus_defconfig
+++ b/arch/arm/configs/omap2plus_defconfig
@@ -695,11 +695,10 @@ CONFIG_NLS_CODEPAGE_437=y
CONFIG_NLS_ISO8859_1=y
CONFIG_SECURITY=y
CONFIG_CRYPTO_MICHAEL_MIC=y
CONFIG_CRYPTO_GHASH_ARM_CE=m
CONFIG_CRYPTO_SHA1_ARM_NEON=m
-CONFIG_CRYPTO_SHA256_ARM=m
CONFIG_CRYPTO_SHA512_ARM=m
CONFIG_CRYPTO_AES_ARM=m
CONFIG_CRYPTO_AES_ARM_BS=m
CONFIG_CRYPTO_CHACHA20_NEON=m
CONFIG_CRYPTO_DEV_OMAP=m
diff --git a/arch/arm/configs/pxa_defconfig b/arch/arm/configs/pxa_defconfig
index de0ac8f521d76..fa631523616f8 100644
--- a/arch/arm/configs/pxa_defconfig
+++ b/arch/arm/configs/pxa_defconfig
@@ -658,11 +658,10 @@ CONFIG_CRYPTO_WP512=m
CONFIG_CRYPTO_ANUBIS=m
CONFIG_CRYPTO_XCBC=m
CONFIG_CRYPTO_DEFLATE=y
CONFIG_CRYPTO_LZO=y
CONFIG_CRYPTO_SHA1_ARM=m
-CONFIG_CRYPTO_SHA256_ARM=m
CONFIG_CRYPTO_SHA512_ARM=m
CONFIG_CRYPTO_AES_ARM=m
CONFIG_CRC_CCITT=y
CONFIG_CRC_T10DIF=m
CONFIG_FONTS=y
diff --git a/arch/arm/crypto/Kconfig b/arch/arm/crypto/Kconfig
index 1f889d6bab77d..7efb9a8596e4e 100644
--- a/arch/arm/crypto/Kconfig
+++ b/arch/arm/crypto/Kconfig
@@ -91,31 +91,10 @@ config CRYPTO_SHA1_ARM_CE
help
SHA-1 secure hash algorithm (FIPS 180)
Architecture: arm using ARMv8 Crypto Extensions
-config CRYPTO_SHA2_ARM_CE
- tristate "Hash functions: SHA-224 and SHA-256 (ARMv8 Crypto Extensions)"
- depends on KERNEL_MODE_NEON
- select CRYPTO_SHA256_ARM
- select CRYPTO_HASH
- help
- SHA-224 and SHA-256 secure hash algorithms (FIPS 180)
-
- Architecture: arm using
- - ARMv8 Crypto Extensions
-
-config CRYPTO_SHA256_ARM
- tristate "Hash functions: SHA-224 and SHA-256 (NEON)"
- select CRYPTO_HASH
- depends on !CPU_V7M
- help
- SHA-224 and SHA-256 secure hash algorithms (FIPS 180)
-
- Architecture: arm using
- - NEON (Advanced SIMD) extensions
-
config CRYPTO_SHA512_ARM
tristate "Hash functions: SHA-384 and SHA-512 (NEON)"
select CRYPTO_HASH
depends on !CPU_V7M
help
diff --git a/arch/arm/crypto/Makefile b/arch/arm/crypto/Makefile
index ecabe6603e080..8479137c6e800 100644
--- a/arch/arm/crypto/Makefile
+++ b/arch/arm/crypto/Makefile
@@ -5,32 +5,27 @@
obj-$(CONFIG_CRYPTO_AES_ARM) += aes-arm.o
obj-$(CONFIG_CRYPTO_AES_ARM_BS) += aes-arm-bs.o
obj-$(CONFIG_CRYPTO_SHA1_ARM) += sha1-arm.o
obj-$(CONFIG_CRYPTO_SHA1_ARM_NEON) += sha1-arm-neon.o
-obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o
obj-$(CONFIG_CRYPTO_SHA512_ARM) += sha512-arm.o
obj-$(CONFIG_CRYPTO_BLAKE2B_NEON) += blake2b-neon.o
obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) += nhpoly1305-neon.o
obj-$(CONFIG_CRYPTO_CURVE25519_NEON) += curve25519-neon.o
obj-$(CONFIG_CRYPTO_AES_ARM_CE) += aes-arm-ce.o
obj-$(CONFIG_CRYPTO_SHA1_ARM_CE) += sha1-arm-ce.o
-obj-$(CONFIG_CRYPTO_SHA2_ARM_CE) += sha2-arm-ce.o
obj-$(CONFIG_CRYPTO_GHASH_ARM_CE) += ghash-arm-ce.o
aes-arm-y := aes-cipher-core.o aes-cipher-glue.o
aes-arm-bs-y := aes-neonbs-core.o aes-neonbs-glue.o
sha1-arm-y := sha1-armv4-large.o sha1_glue.o
sha1-arm-neon-y := sha1-armv7-neon.o sha1_neon_glue.o
-sha256-arm-neon-$(CONFIG_KERNEL_MODE_NEON) := sha256_neon_glue.o
-sha256-arm-y := sha256-core.o sha256_glue.o $(sha256-arm-neon-y)
sha512-arm-neon-$(CONFIG_KERNEL_MODE_NEON) := sha512-neon-glue.o
sha512-arm-y := sha512-core.o sha512-glue.o $(sha512-arm-neon-y)
blake2b-neon-y := blake2b-neon-core.o blake2b-neon-glue.o
sha1-arm-ce-y := sha1-ce-core.o sha1-ce-glue.o
-sha2-arm-ce-y := sha2-ce-core.o sha2-ce-glue.o
aes-arm-ce-y := aes-ce-core.o aes-ce-glue.o
ghash-arm-ce-y := ghash-ce-core.o ghash-ce-glue.o
nhpoly1305-neon-y := nh-neon-core.o nhpoly1305-neon-glue.o
curve25519-neon-y := curve25519-core.o curve25519-glue.o
@@ -38,11 +33,10 @@ quiet_cmd_perl = PERL $@
cmd_perl = $(PERL) $(<) > $(@)
$(obj)/%-core.S: $(src)/%-armv4.pl
$(call cmd,perl)
-clean-files += sha256-core.S sha512-core.S
+clean-files += sha512-core.S
aflags-thumb2-$(CONFIG_THUMB2_KERNEL) := -U__thumb2__ -D__thumb2__=1
-AFLAGS_sha256-core.o += $(aflags-thumb2-y)
AFLAGS_sha512-core.o += $(aflags-thumb2-y)
diff --git a/arch/arm/crypto/sha2-ce-glue.c b/arch/arm/crypto/sha2-ce-glue.c
deleted file mode 100644
index 1e9d16f796787..0000000000000
--- a/arch/arm/crypto/sha2-ce-glue.c
+++ /dev/null
@@ -1,87 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * sha2-ce-glue.c - SHA-224/SHA-256 using ARMv8 Crypto Extensions
- *
- * Copyright (C) 2015 Linaro Ltd <ard.biesheuvel@linaro.org>
- */
-
-#include <asm/neon.h>
-#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
-#include <crypto/sha256_base.h>
-#include <linux/cpufeature.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-
-MODULE_DESCRIPTION("SHA-224/SHA-256 secure hash using ARMv8 Crypto Extensions");
-MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
-MODULE_LICENSE("GPL v2");
-
-asmlinkage void sha2_ce_transform(struct crypto_sha256_state *sst,
- u8 const *src, int blocks);
-
-static int sha2_ce_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- int remain;
-
- kernel_neon_begin();
- remain = sha256_base_do_update_blocks(desc, data, len,
- sha2_ce_transform);
- kernel_neon_end();
- return remain;
-}
-
-static int sha2_ce_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- kernel_neon_begin();
- sha256_base_do_finup(desc, data, len, sha2_ce_transform);
- kernel_neon_end();
- return sha256_base_finish(desc, out);
-}
-
-static struct shash_alg algs[] = { {
- .init = sha224_base_init,
- .update = sha2_ce_update,
- .finup = sha2_ce_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .digestsize = SHA224_DIGEST_SIZE,
- .base = {
- .cra_name = "sha224",
- .cra_driver_name = "sha224-ce",
- .cra_priority = 300,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-}, {
- .init = sha256_base_init,
- .update = sha2_ce_update,
- .finup = sha2_ce_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .digestsize = SHA256_DIGEST_SIZE,
- .base = {
- .cra_name = "sha256",
- .cra_driver_name = "sha256-ce",
- .cra_priority = 300,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-} };
-
-static int __init sha2_ce_mod_init(void)
-{
- return crypto_register_shashes(algs, ARRAY_SIZE(algs));
-}
-
-static void __exit sha2_ce_mod_fini(void)
-{
- crypto_unregister_shashes(algs, ARRAY_SIZE(algs));
-}
-
-module_cpu_feature_match(SHA2, sha2_ce_mod_init);
-module_exit(sha2_ce_mod_fini);
diff --git a/arch/arm/crypto/sha256_glue.c b/arch/arm/crypto/sha256_glue.c
deleted file mode 100644
index d04c4e6bae6d3..0000000000000
--- a/arch/arm/crypto/sha256_glue.c
+++ /dev/null
@@ -1,107 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Glue code for the SHA256 Secure Hash Algorithm assembly implementation
- * using optimized ARM assembler and NEON instructions.
- *
- * Copyright © 2015 Google Inc.
- *
- * This file is based on sha256_ssse3_glue.c:
- * Copyright (C) 2013 Intel Corporation
- * Author: Tim Chen <tim.c.chen@linux.intel.com>
- */
-
-#include <asm/neon.h>
-#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
-#include <crypto/sha256_base.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-
-#include "sha256_glue.h"
-
-asmlinkage void sha256_block_data_order(struct crypto_sha256_state *state,
- const u8 *data, int num_blks);
-
-static int crypto_sha256_arm_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- /* make sure casting to sha256_block_fn() is safe */
- BUILD_BUG_ON(offsetof(struct crypto_sha256_state, state) != 0);
-
- return sha256_base_do_update_blocks(desc, data, len,
- sha256_block_data_order);
-}
-
-static int crypto_sha256_arm_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- sha256_base_do_finup(desc, data, len, sha256_block_data_order);
- return sha256_base_finish(desc, out);
-}
-
-static struct shash_alg algs[] = { {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = crypto_sha256_arm_update,
- .finup = crypto_sha256_arm_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha256",
- .cra_driver_name = "sha256-asm",
- .cra_priority = 150,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-}, {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = crypto_sha256_arm_update,
- .finup = crypto_sha256_arm_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha224",
- .cra_driver_name = "sha224-asm",
- .cra_priority = 150,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-} };
-
-static int __init sha256_mod_init(void)
-{
- int res = crypto_register_shashes(algs, ARRAY_SIZE(algs));
-
- if (res < 0)
- return res;
-
- if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && cpu_has_neon()) {
- res = crypto_register_shashes(sha256_neon_algs,
- ARRAY_SIZE(sha256_neon_algs));
-
- if (res < 0)
- crypto_unregister_shashes(algs, ARRAY_SIZE(algs));
- }
-
- return res;
-}
-
-static void __exit sha256_mod_fini(void)
-{
- crypto_unregister_shashes(algs, ARRAY_SIZE(algs));
-
- if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && cpu_has_neon())
- crypto_unregister_shashes(sha256_neon_algs,
- ARRAY_SIZE(sha256_neon_algs));
-}
-
-module_init(sha256_mod_init);
-module_exit(sha256_mod_fini);
-
-MODULE_LICENSE("GPL");
-MODULE_DESCRIPTION("SHA256 Secure Hash Algorithm (ARM), including NEON");
-
-MODULE_ALIAS_CRYPTO("sha256");
diff --git a/arch/arm/crypto/sha256_glue.h b/arch/arm/crypto/sha256_glue.h
deleted file mode 100644
index 9881c9a115d1f..0000000000000
--- a/arch/arm/crypto/sha256_glue.h
+++ /dev/null
@@ -1,9 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _CRYPTO_SHA256_GLUE_H
-#define _CRYPTO_SHA256_GLUE_H
-
-#include <crypto/hash.h>
-
-extern struct shash_alg sha256_neon_algs[2];
-
-#endif /* _CRYPTO_SHA256_GLUE_H */
diff --git a/arch/arm/crypto/sha256_neon_glue.c b/arch/arm/crypto/sha256_neon_glue.c
deleted file mode 100644
index 76eb3cdc21c96..0000000000000
--- a/arch/arm/crypto/sha256_neon_glue.c
+++ /dev/null
@@ -1,75 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Glue code for the SHA256 Secure Hash Algorithm assembly implementation
- * using NEON instructions.
- *
- * Copyright © 2015 Google Inc.
- *
- * This file is based on sha512_neon_glue.c:
- * Copyright © 2014 Jussi Kivilinna <jussi.kivilinna@iki.fi>
- */
-
-#include <asm/neon.h>
-#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
-#include <crypto/sha256_base.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-
-#include "sha256_glue.h"
-
-asmlinkage void sha256_block_data_order_neon(
- struct crypto_sha256_state *digest, const u8 *data, int num_blks);
-
-static int crypto_sha256_neon_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- int remain;
-
- kernel_neon_begin();
- remain = sha256_base_do_update_blocks(desc, data, len,
- sha256_block_data_order_neon);
- kernel_neon_end();
- return remain;
-}
-
-static int crypto_sha256_neon_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- kernel_neon_begin();
- sha256_base_do_finup(desc, data, len, sha256_block_data_order_neon);
- kernel_neon_end();
- return sha256_base_finish(desc, out);
-}
-
-struct shash_alg sha256_neon_algs[] = { {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = crypto_sha256_neon_update,
- .finup = crypto_sha256_neon_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha256",
- .cra_driver_name = "sha256-neon",
- .cra_priority = 250,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-}, {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = crypto_sha256_neon_update,
- .finup = crypto_sha256_neon_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha224",
- .cra_driver_name = "sha224-neon",
- .cra_priority = 250,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-} };
diff --git a/arch/arm/lib/crypto/.gitignore b/arch/arm/lib/crypto/.gitignore
index 0d47d4f21c6de..12d74d8b03d0a 100644
--- a/arch/arm/lib/crypto/.gitignore
+++ b/arch/arm/lib/crypto/.gitignore
@@ -1,2 +1,3 @@
# SPDX-License-Identifier: GPL-2.0-only
poly1305-core.S
+sha256-core.S
diff --git a/arch/arm/lib/crypto/Kconfig b/arch/arm/lib/crypto/Kconfig
index e8444fd0aae30..9f3ff30f40328 100644
--- a/arch/arm/lib/crypto/Kconfig
+++ b/arch/arm/lib/crypto/Kconfig
@@ -20,5 +20,11 @@ config CRYPTO_CHACHA20_NEON
config CRYPTO_POLY1305_ARM
tristate
default CRYPTO_LIB_POLY1305
select CRYPTO_ARCH_HAVE_LIB_POLY1305
+
+config CRYPTO_SHA256_ARM
+ tristate
+ depends on !CPU_V7M
+ default CRYPTO_LIB_SHA256
+ select CRYPTO_ARCH_HAVE_LIB_SHA256
diff --git a/arch/arm/lib/crypto/Makefile b/arch/arm/lib/crypto/Makefile
index 4c042a4c77ed6..431f77c3ff6fd 100644
--- a/arch/arm/lib/crypto/Makefile
+++ b/arch/arm/lib/crypto/Makefile
@@ -8,19 +8,25 @@ chacha-neon-y := chacha-scalar-core.o chacha-glue.o
chacha-neon-$(CONFIG_KERNEL_MODE_NEON) += chacha-neon-core.o
obj-$(CONFIG_CRYPTO_POLY1305_ARM) += poly1305-arm.o
poly1305-arm-y := poly1305-core.o poly1305-glue.o
+obj-$(CONFIG_CRYPTO_SHA256_ARM) += sha256-arm.o
+sha256-arm-y := sha256.o sha256-core.o
+sha256-arm-$(CONFIG_KERNEL_MODE_NEON) += sha256-ce.o
+
quiet_cmd_perl = PERL $@
cmd_perl = $(PERL) $(<) > $(@)
$(obj)/%-core.S: $(src)/%-armv4.pl
$(call cmd,perl)
-clean-files += poly1305-core.S
+clean-files += poly1305-core.S sha256-core.S
aflags-thumb2-$(CONFIG_THUMB2_KERNEL) := -U__thumb2__ -D__thumb2__=1
# massage the perlasm code a bit so we only get the NEON routine if we need it
poly1305-aflags-$(CONFIG_CPU_V7) := -U__LINUX_ARM_ARCH__ -D__LINUX_ARM_ARCH__=5
poly1305-aflags-$(CONFIG_KERNEL_MODE_NEON) := -U__LINUX_ARM_ARCH__ -D__LINUX_ARM_ARCH__=7
AFLAGS_poly1305-core.o += $(poly1305-aflags-y) $(aflags-thumb2-y)
+
+AFLAGS_sha256-core.o += $(aflags-thumb2-y)
diff --git a/arch/arm/crypto/sha256-armv4.pl b/arch/arm/lib/crypto/sha256-armv4.pl
similarity index 100%
rename from arch/arm/crypto/sha256-armv4.pl
rename to arch/arm/lib/crypto/sha256-armv4.pl
diff --git a/arch/arm/crypto/sha2-ce-core.S b/arch/arm/lib/crypto/sha256-ce.S
similarity index 91%
rename from arch/arm/crypto/sha2-ce-core.S
rename to arch/arm/lib/crypto/sha256-ce.S
index b6369d2440a19..ac2c9b01b22d2 100644
--- a/arch/arm/crypto/sha2-ce-core.S
+++ b/arch/arm/lib/crypto/sha256-ce.S
@@ -1,8 +1,8 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
- * sha2-ce-core.S - SHA-224/256 secure hash using ARMv8 Crypto Extensions
+ * sha256-ce.S - SHA-224/256 secure hash using ARMv8 Crypto Extensions
*
* Copyright (C) 2015 Linaro Ltd.
* Author: Ard Biesheuvel <ard.biesheuvel@linaro.org>
*/
@@ -65,14 +65,14 @@
.word 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3
.word 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208
.word 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2
/*
- * void sha2_ce_transform(struct sha256_state *sst, u8 const *src,
- int blocks);
+ * void sha256_ce_transform(u32 state[SHA256_STATE_WORDS],
+ * const u8 *data, size_t nblocks);
*/
-ENTRY(sha2_ce_transform)
+ENTRY(sha256_ce_transform)
/* load state */
vld1.32 {dga-dgb}, [r0]
/* load input */
0: vld1.32 {q0-q1}, [r1]!
@@ -118,6 +118,6 @@ ENTRY(sha2_ce_transform)
bne 0b
/* store new state */
vst1.32 {dga-dgb}, [r0]
bx lr
-ENDPROC(sha2_ce_transform)
+ENDPROC(sha256_ce_transform)
diff --git a/arch/arm/lib/crypto/sha256.c b/arch/arm/lib/crypto/sha256.c
new file mode 100644
index 0000000000000..3a8dfc304807a
--- /dev/null
+++ b/arch/arm/lib/crypto/sha256.c
@@ -0,0 +1,64 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * SHA-256 optimized for ARM
+ *
+ * Copyright 2025 Google LLC
+ */
+#include <asm/neon.h>
+#include <crypto/internal/sha2.h>
+#include <crypto/internal/simd.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+asmlinkage void sha256_block_data_order(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+asmlinkage void sha256_block_data_order_neon(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+asmlinkage void sha256_ce_transform(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon);
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_ce);
+
+void sha256_blocks_arch(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks)
+{
+ if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) &&
+ static_branch_likely(&have_neon) && crypto_simd_usable()) {
+ kernel_neon_begin();
+ if (static_branch_likely(&have_ce))
+ sha256_ce_transform(state, data, nblocks);
+ else
+ sha256_block_data_order_neon(state, data, nblocks);
+ kernel_neon_end();
+ } else {
+ sha256_block_data_order(state, data, nblocks);
+ }
+}
+EXPORT_SYMBOL(sha256_blocks_arch);
+
+bool sha256_is_arch_optimized(void)
+{
+ /* We always can use at least the ARM scalar implementation. */
+ return true;
+}
+EXPORT_SYMBOL(sha256_is_arch_optimized);
+
+static int __init sha256_arm_mod_init(void)
+{
+ if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) && (elf_hwcap & HWCAP_NEON)) {
+ static_branch_enable(&have_neon);
+ if (elf_hwcap2 & HWCAP2_SHA2)
+ static_branch_enable(&have_ce);
+ }
+ return 0;
+}
+arch_initcall(sha256_arm_mod_init);
+
+static void __exit sha256_arm_mod_exit(void)
+{
+}
+module_exit(sha256_arm_mod_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("SHA-256 optimized for ARM");
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 03/13] crypto: arm64/sha256 - remove obsolete chunking logic
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
2025-04-28 17:00 ` [PATCH v4 01/13] crypto: sha256 - support arch-optimized lib and expose through shash Eric Biggers
2025-04-28 17:00 ` [PATCH v4 02/13] crypto: arm/sha256 - implement library instead of shash Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-04-28 17:00 ` [PATCH v4 04/13] crypto: arm64/sha256 - implement library instead of shash Eric Biggers
` (10 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Since kernel-mode NEON sections are now preemptible on arm64, there is
no longer any need to limit the length of them.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/arm64/crypto/sha256-glue.c | 19 ++-----------------
1 file changed, 2 insertions(+), 17 deletions(-)
diff --git a/arch/arm64/crypto/sha256-glue.c b/arch/arm64/crypto/sha256-glue.c
index 26f9fdfae87bf..d63ea82e1374e 100644
--- a/arch/arm64/crypto/sha256-glue.c
+++ b/arch/arm64/crypto/sha256-glue.c
@@ -84,27 +84,12 @@ static struct shash_alg algs[] = { {
} };
static int sha256_update_neon(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
- do {
- unsigned int chunk = len;
-
- /*
- * Don't hog the CPU for the entire time it takes to process all
- * input when running on a preemptible kernel, but process the
- * data block by block instead.
- */
- if (IS_ENABLED(CONFIG_PREEMPTION))
- chunk = SHA256_BLOCK_SIZE;
-
- chunk -= sha256_base_do_update_blocks(desc, data, chunk,
- sha256_neon_transform);
- data += chunk;
- len -= chunk;
- } while (len >= SHA256_BLOCK_SIZE);
- return len;
+ return sha256_base_do_update_blocks(desc, data, len,
+ sha256_neon_transform);
}
static int sha256_finup_neon(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 04/13] crypto: arm64/sha256 - implement library instead of shash
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
` (2 preceding siblings ...)
2025-04-28 17:00 ` [PATCH v4 03/13] crypto: arm64/sha256 - remove obsolete chunking logic Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-04-28 17:00 ` [PATCH v4 05/13] crypto: mips/sha256 " Eric Biggers
` (9 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Instead of providing crypto_shash algorithms for the arch-optimized
SHA-256 code, instead implement the SHA-256 library. This is much
simpler, it makes the SHA-256 library functions be arch-optimized, and
it fixes the longstanding issue where the arch-optimized SHA-256 was
disabled by default. SHA-256 still remains available through
crypto_shash, but individual architectures no longer need to handle it.
Remove support for SHA-256 finalization from the ARMv8 CE assembly code,
since the library does not yet support architecture-specific overrides
of the finalization. (Support for that has been omitted for now, for
simplicity and because usually it isn't performance-critical.)
To match sha256_blocks_arch(), change the type of the nblocks parameter
of the assembly functions from int or 'unsigned int' to size_t. Update
the ARMv8 CE assembly function accordingly. The scalar and NEON
assembly functions actually already treated it as size_t.
While renaming the assembly files, also fix the naming quirks where
"sha2" meant sha256, and "sha512" meant both sha256 and sha512.
Reviewed-by: Ard Biesheuvel <ardb@kernel.org>
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/arm64/configs/defconfig | 1 -
arch/arm64/crypto/Kconfig | 19 ---
arch/arm64/crypto/Makefile | 13 +-
arch/arm64/crypto/sha2-ce-glue.c | 138 ----------------
arch/arm64/crypto/sha256-glue.c | 156 ------------------
arch/arm64/lib/crypto/.gitignore | 1 +
arch/arm64/lib/crypto/Kconfig | 5 +
arch/arm64/lib/crypto/Makefile | 9 +-
.../crypto/sha2-armv8.pl} | 0
.../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 36 +---
arch/arm64/lib/crypto/sha256.c | 75 +++++++++
11 files changed, 98 insertions(+), 355 deletions(-)
delete mode 100644 arch/arm64/crypto/sha2-ce-glue.c
delete mode 100644 arch/arm64/crypto/sha256-glue.c
rename arch/arm64/{crypto/sha512-armv8.pl => lib/crypto/sha2-armv8.pl} (100%)
rename arch/arm64/{crypto/sha2-ce-core.S => lib/crypto/sha256-ce.S} (80%)
create mode 100644 arch/arm64/lib/crypto/sha256.c
diff --git a/arch/arm64/configs/defconfig b/arch/arm64/configs/defconfig
index 5bb8f09422a22..b0d4c7d173ea7 100644
--- a/arch/arm64/configs/defconfig
+++ b/arch/arm64/configs/defconfig
@@ -1735,11 +1735,10 @@ CONFIG_CRYPTO_MICHAEL_MIC=m
CONFIG_CRYPTO_ANSI_CPRNG=y
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_CHACHA20_NEON=m
CONFIG_CRYPTO_GHASH_ARM64_CE=y
CONFIG_CRYPTO_SHA1_ARM64_CE=y
-CONFIG_CRYPTO_SHA2_ARM64_CE=y
CONFIG_CRYPTO_SHA512_ARM64_CE=m
CONFIG_CRYPTO_SHA3_ARM64=m
CONFIG_CRYPTO_SM3_ARM64_CE=m
CONFIG_CRYPTO_AES_ARM64_CE_BLK=y
CONFIG_CRYPTO_AES_ARM64_BS=m
diff --git a/arch/arm64/crypto/Kconfig b/arch/arm64/crypto/Kconfig
index 55a7d87a67690..c44b0f202a1f5 100644
--- a/arch/arm64/crypto/Kconfig
+++ b/arch/arm64/crypto/Kconfig
@@ -34,29 +34,10 @@ config CRYPTO_SHA1_ARM64_CE
SHA-1 secure hash algorithm (FIPS 180)
Architecture: arm64 using:
- ARMv8 Crypto Extensions
-config CRYPTO_SHA256_ARM64
- tristate "Hash functions: SHA-224 and SHA-256"
- select CRYPTO_HASH
- help
- SHA-224 and SHA-256 secure hash algorithms (FIPS 180)
-
- Architecture: arm64
-
-config CRYPTO_SHA2_ARM64_CE
- tristate "Hash functions: SHA-224 and SHA-256 (ARMv8 Crypto Extensions)"
- depends on KERNEL_MODE_NEON
- select CRYPTO_HASH
- select CRYPTO_SHA256_ARM64
- help
- SHA-224 and SHA-256 secure hash algorithms (FIPS 180)
-
- Architecture: arm64 using:
- - ARMv8 Crypto Extensions
-
config CRYPTO_SHA512_ARM64
tristate "Hash functions: SHA-384 and SHA-512"
select CRYPTO_HASH
help
SHA-384 and SHA-512 secure hash algorithms (FIPS 180)
diff --git a/arch/arm64/crypto/Makefile b/arch/arm64/crypto/Makefile
index 089ae3ddde810..c231c980c5142 100644
--- a/arch/arm64/crypto/Makefile
+++ b/arch/arm64/crypto/Makefile
@@ -6,13 +6,10 @@
#
obj-$(CONFIG_CRYPTO_SHA1_ARM64_CE) += sha1-ce.o
sha1-ce-y := sha1-ce-glue.o sha1-ce-core.o
-obj-$(CONFIG_CRYPTO_SHA2_ARM64_CE) += sha2-ce.o
-sha2-ce-y := sha2-ce-glue.o sha2-ce-core.o
-
obj-$(CONFIG_CRYPTO_SHA512_ARM64_CE) += sha512-ce.o
sha512-ce-y := sha512-ce-glue.o sha512-ce-core.o
obj-$(CONFIG_CRYPTO_SHA3_ARM64) += sha3-ce.o
sha3-ce-y := sha3-ce-glue.o sha3-ce-core.o
@@ -54,13 +51,10 @@ obj-$(CONFIG_CRYPTO_AES_ARM64_CE_BLK) += aes-ce-blk.o
aes-ce-blk-y := aes-glue-ce.o aes-ce.o
obj-$(CONFIG_CRYPTO_AES_ARM64_NEON_BLK) += aes-neon-blk.o
aes-neon-blk-y := aes-glue-neon.o aes-neon.o
-obj-$(CONFIG_CRYPTO_SHA256_ARM64) += sha256-arm64.o
-sha256-arm64-y := sha256-glue.o sha256-core.o
-
obj-$(CONFIG_CRYPTO_SHA512_ARM64) += sha512-arm64.o
sha512-arm64-y := sha512-glue.o sha512-core.o
obj-$(CONFIG_CRYPTO_NHPOLY1305_NEON) += nhpoly1305-neon.o
nhpoly1305-neon-y := nh-neon-core.o nhpoly1305-neon-glue.o
@@ -72,12 +66,9 @@ obj-$(CONFIG_CRYPTO_AES_ARM64_BS) += aes-neon-bs.o
aes-neon-bs-y := aes-neonbs-core.o aes-neonbs-glue.o
quiet_cmd_perlasm = PERLASM $@
cmd_perlasm = $(PERL) $(<) void $(@)
-$(obj)/%-core.S: $(src)/%-armv8.pl
- $(call cmd,perlasm)
-
-$(obj)/sha256-core.S: $(src)/sha512-armv8.pl
+$(obj)/sha512-core.S: $(src)/../lib/crypto/sha2-armv8.pl
$(call cmd,perlasm)
-clean-files += sha256-core.S sha512-core.S
+clean-files += sha512-core.S
diff --git a/arch/arm64/crypto/sha2-ce-glue.c b/arch/arm64/crypto/sha2-ce-glue.c
deleted file mode 100644
index 912c215101eb1..0000000000000
--- a/arch/arm64/crypto/sha2-ce-glue.c
+++ /dev/null
@@ -1,138 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * sha2-ce-glue.c - SHA-224/SHA-256 using ARMv8 Crypto Extensions
- *
- * Copyright (C) 2014 - 2017 Linaro Ltd <ard.biesheuvel@linaro.org>
- */
-
-#include <asm/neon.h>
-#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
-#include <crypto/sha256_base.h>
-#include <linux/cpufeature.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/string.h>
-
-MODULE_DESCRIPTION("SHA-224/SHA-256 secure hash using ARMv8 Crypto Extensions");
-MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
-MODULE_LICENSE("GPL v2");
-MODULE_ALIAS_CRYPTO("sha224");
-MODULE_ALIAS_CRYPTO("sha256");
-
-struct sha256_ce_state {
- struct crypto_sha256_state sst;
- u32 finalize;
-};
-
-extern const u32 sha256_ce_offsetof_count;
-extern const u32 sha256_ce_offsetof_finalize;
-
-asmlinkage int __sha256_ce_transform(struct sha256_ce_state *sst, u8 const *src,
- int blocks);
-
-static void sha256_ce_transform(struct crypto_sha256_state *sst, u8 const *src,
- int blocks)
-{
- while (blocks) {
- int rem;
-
- kernel_neon_begin();
- rem = __sha256_ce_transform(container_of(sst,
- struct sha256_ce_state,
- sst), src, blocks);
- kernel_neon_end();
- src += (blocks - rem) * SHA256_BLOCK_SIZE;
- blocks = rem;
- }
-}
-
-const u32 sha256_ce_offsetof_count = offsetof(struct sha256_ce_state,
- sst.count);
-const u32 sha256_ce_offsetof_finalize = offsetof(struct sha256_ce_state,
- finalize);
-
-static int sha256_ce_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- struct sha256_ce_state *sctx = shash_desc_ctx(desc);
-
- sctx->finalize = 0;
- return sha256_base_do_update_blocks(desc, data, len,
- sha256_ce_transform);
-}
-
-static int sha256_ce_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- struct sha256_ce_state *sctx = shash_desc_ctx(desc);
- bool finalize = !(len % SHA256_BLOCK_SIZE) && len;
-
- /*
- * Allow the asm code to perform the finalization if there is no
- * partial data and the input is a round multiple of the block size.
- */
- sctx->finalize = finalize;
-
- if (finalize)
- sha256_base_do_update_blocks(desc, data, len,
- sha256_ce_transform);
- else
- sha256_base_do_finup(desc, data, len, sha256_ce_transform);
- return sha256_base_finish(desc, out);
-}
-
-static int sha256_ce_digest(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- sha256_base_init(desc);
- return sha256_ce_finup(desc, data, len, out);
-}
-
-static struct shash_alg algs[] = { {
- .init = sha224_base_init,
- .update = sha256_ce_update,
- .finup = sha256_ce_finup,
- .descsize = sizeof(struct sha256_ce_state),
- .statesize = sizeof(struct crypto_sha256_state),
- .digestsize = SHA224_DIGEST_SIZE,
- .base = {
- .cra_name = "sha224",
- .cra_driver_name = "sha224-ce",
- .cra_priority = 200,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-}, {
- .init = sha256_base_init,
- .update = sha256_ce_update,
- .finup = sha256_ce_finup,
- .digest = sha256_ce_digest,
- .descsize = sizeof(struct sha256_ce_state),
- .statesize = sizeof(struct crypto_sha256_state),
- .digestsize = SHA256_DIGEST_SIZE,
- .base = {
- .cra_name = "sha256",
- .cra_driver_name = "sha256-ce",
- .cra_priority = 200,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-} };
-
-static int __init sha2_ce_mod_init(void)
-{
- return crypto_register_shashes(algs, ARRAY_SIZE(algs));
-}
-
-static void __exit sha2_ce_mod_fini(void)
-{
- crypto_unregister_shashes(algs, ARRAY_SIZE(algs));
-}
-
-module_cpu_feature_match(SHA2, sha2_ce_mod_init);
-module_exit(sha2_ce_mod_fini);
diff --git a/arch/arm64/crypto/sha256-glue.c b/arch/arm64/crypto/sha256-glue.c
deleted file mode 100644
index d63ea82e1374e..0000000000000
--- a/arch/arm64/crypto/sha256-glue.c
+++ /dev/null
@@ -1,156 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Linux/arm64 port of the OpenSSL SHA256 implementation for AArch64
- *
- * Copyright (c) 2016 Linaro Ltd. <ard.biesheuvel@linaro.org>
- */
-
-#include <asm/neon.h>
-#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
-#include <crypto/sha256_base.h>
-#include <linux/cpufeature.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-
-MODULE_DESCRIPTION("SHA-224/SHA-256 secure hash for arm64");
-MODULE_AUTHOR("Andy Polyakov <appro@openssl.org>");
-MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
-MODULE_LICENSE("GPL v2");
-MODULE_ALIAS_CRYPTO("sha224");
-MODULE_ALIAS_CRYPTO("sha256");
-
-asmlinkage void sha256_block_data_order(u32 *digest, const void *data,
- unsigned int num_blks);
-EXPORT_SYMBOL(sha256_block_data_order);
-
-static void sha256_arm64_transform(struct crypto_sha256_state *sst,
- u8 const *src, int blocks)
-{
- sha256_block_data_order(sst->state, src, blocks);
-}
-
-asmlinkage void sha256_block_neon(u32 *digest, const void *data,
- unsigned int num_blks);
-
-static void sha256_neon_transform(struct crypto_sha256_state *sst,
- u8 const *src, int blocks)
-{
- kernel_neon_begin();
- sha256_block_neon(sst->state, src, blocks);
- kernel_neon_end();
-}
-
-static int crypto_sha256_arm64_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- return sha256_base_do_update_blocks(desc, data, len,
- sha256_arm64_transform);
-}
-
-static int crypto_sha256_arm64_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- sha256_base_do_finup(desc, data, len, sha256_arm64_transform);
- return sha256_base_finish(desc, out);
-}
-
-static struct shash_alg algs[] = { {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = crypto_sha256_arm64_update,
- .finup = crypto_sha256_arm64_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base.cra_name = "sha256",
- .base.cra_driver_name = "sha256-arm64",
- .base.cra_priority = 125,
- .base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .base.cra_blocksize = SHA256_BLOCK_SIZE,
- .base.cra_module = THIS_MODULE,
-}, {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = crypto_sha256_arm64_update,
- .finup = crypto_sha256_arm64_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base.cra_name = "sha224",
- .base.cra_driver_name = "sha224-arm64",
- .base.cra_priority = 125,
- .base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .base.cra_blocksize = SHA224_BLOCK_SIZE,
- .base.cra_module = THIS_MODULE,
-} };
-
-static int sha256_update_neon(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- return sha256_base_do_update_blocks(desc, data, len,
- sha256_neon_transform);
-}
-
-static int sha256_finup_neon(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- if (len >= SHA256_BLOCK_SIZE) {
- int remain = sha256_update_neon(desc, data, len);
-
- data += len - remain;
- len = remain;
- }
- sha256_base_do_finup(desc, data, len, sha256_neon_transform);
- return sha256_base_finish(desc, out);
-}
-
-static struct shash_alg neon_algs[] = { {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = sha256_update_neon,
- .finup = sha256_finup_neon,
- .descsize = sizeof(struct crypto_sha256_state),
- .base.cra_name = "sha256",
- .base.cra_driver_name = "sha256-arm64-neon",
- .base.cra_priority = 150,
- .base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .base.cra_blocksize = SHA256_BLOCK_SIZE,
- .base.cra_module = THIS_MODULE,
-}, {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = sha256_update_neon,
- .finup = sha256_finup_neon,
- .descsize = sizeof(struct crypto_sha256_state),
- .base.cra_name = "sha224",
- .base.cra_driver_name = "sha224-arm64-neon",
- .base.cra_priority = 150,
- .base.cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .base.cra_blocksize = SHA224_BLOCK_SIZE,
- .base.cra_module = THIS_MODULE,
-} };
-
-static int __init sha256_mod_init(void)
-{
- int ret = crypto_register_shashes(algs, ARRAY_SIZE(algs));
- if (ret)
- return ret;
-
- if (cpu_have_named_feature(ASIMD)) {
- ret = crypto_register_shashes(neon_algs, ARRAY_SIZE(neon_algs));
- if (ret)
- crypto_unregister_shashes(algs, ARRAY_SIZE(algs));
- }
- return ret;
-}
-
-static void __exit sha256_mod_fini(void)
-{
- if (cpu_have_named_feature(ASIMD))
- crypto_unregister_shashes(neon_algs, ARRAY_SIZE(neon_algs));
- crypto_unregister_shashes(algs, ARRAY_SIZE(algs));
-}
-
-module_init(sha256_mod_init);
-module_exit(sha256_mod_fini);
diff --git a/arch/arm64/lib/crypto/.gitignore b/arch/arm64/lib/crypto/.gitignore
index 0d47d4f21c6de..12d74d8b03d0a 100644
--- a/arch/arm64/lib/crypto/.gitignore
+++ b/arch/arm64/lib/crypto/.gitignore
@@ -1,2 +1,3 @@
# SPDX-License-Identifier: GPL-2.0-only
poly1305-core.S
+sha256-core.S
diff --git a/arch/arm64/lib/crypto/Kconfig b/arch/arm64/lib/crypto/Kconfig
index 0b903ef524d85..49e57bfdb5b52 100644
--- a/arch/arm64/lib/crypto/Kconfig
+++ b/arch/arm64/lib/crypto/Kconfig
@@ -10,5 +10,10 @@ config CRYPTO_CHACHA20_NEON
config CRYPTO_POLY1305_NEON
tristate
depends on KERNEL_MODE_NEON
default CRYPTO_LIB_POLY1305
select CRYPTO_ARCH_HAVE_LIB_POLY1305
+
+config CRYPTO_SHA256_ARM64
+ tristate
+ default CRYPTO_LIB_SHA256
+ select CRYPTO_ARCH_HAVE_LIB_SHA256
diff --git a/arch/arm64/lib/crypto/Makefile b/arch/arm64/lib/crypto/Makefile
index ac624c3effdaf..141efe07155c0 100644
--- a/arch/arm64/lib/crypto/Makefile
+++ b/arch/arm64/lib/crypto/Makefile
@@ -5,12 +5,19 @@ chacha-neon-y := chacha-neon-core.o chacha-neon-glue.o
obj-$(CONFIG_CRYPTO_POLY1305_NEON) += poly1305-neon.o
poly1305-neon-y := poly1305-core.o poly1305-glue.o
AFLAGS_poly1305-core.o += -Dpoly1305_init=poly1305_init_arm64
+obj-$(CONFIG_CRYPTO_SHA256_ARM64) += sha256-arm64.o
+sha256-arm64-y := sha256.o sha256-core.o
+sha256-arm64-$(CONFIG_KERNEL_MODE_NEON) += sha256-ce.o
+
quiet_cmd_perlasm = PERLASM $@
cmd_perlasm = $(PERL) $(<) void $(@)
$(obj)/%-core.S: $(src)/%-armv8.pl
$(call cmd,perlasm)
-clean-files += poly1305-core.S
+$(obj)/sha256-core.S: $(src)/sha2-armv8.pl
+ $(call cmd,perlasm)
+
+clean-files += poly1305-core.S sha256-core.S
diff --git a/arch/arm64/crypto/sha512-armv8.pl b/arch/arm64/lib/crypto/sha2-armv8.pl
similarity index 100%
rename from arch/arm64/crypto/sha512-armv8.pl
rename to arch/arm64/lib/crypto/sha2-armv8.pl
diff --git a/arch/arm64/crypto/sha2-ce-core.S b/arch/arm64/lib/crypto/sha256-ce.S
similarity index 80%
rename from arch/arm64/crypto/sha2-ce-core.S
rename to arch/arm64/lib/crypto/sha256-ce.S
index fce84d88ddb2c..a8461d6dad634 100644
--- a/arch/arm64/crypto/sha2-ce-core.S
+++ b/arch/arm64/lib/crypto/sha256-ce.S
@@ -69,12 +69,12 @@
.word 0x391c0cb3, 0x4ed8aa4a, 0x5b9cca4f, 0x682e6ff3
.word 0x748f82ee, 0x78a5636f, 0x84c87814, 0x8cc70208
.word 0x90befffa, 0xa4506ceb, 0xbef9a3f7, 0xc67178f2
/*
- * int __sha256_ce_transform(struct sha256_ce_state *sst, u8 const *src,
- * int blocks)
+ * size_t __sha256_ce_transform(u32 state[SHA256_STATE_WORDS],
+ * const u8 *data, size_t nblocks);
*/
.text
SYM_FUNC_START(__sha256_ce_transform)
/* load round constants */
adr_l x8, .Lsha2_rcon
@@ -84,24 +84,20 @@ SYM_FUNC_START(__sha256_ce_transform)
ld1 {v12.4s-v15.4s}, [x8]
/* load state */
ld1 {dgav.4s, dgbv.4s}, [x0]
- /* load sha256_ce_state::finalize */
- ldr_l w4, sha256_ce_offsetof_finalize, x4
- ldr w4, [x0, x4]
-
/* load input */
0: ld1 {v16.4s-v19.4s}, [x1], #64
- sub w2, w2, #1
+ sub x2, x2, #1
CPU_LE( rev32 v16.16b, v16.16b )
CPU_LE( rev32 v17.16b, v17.16b )
CPU_LE( rev32 v18.16b, v18.16b )
CPU_LE( rev32 v19.16b, v19.16b )
-1: add t0.4s, v16.4s, v0.4s
+ add t0.4s, v16.4s, v0.4s
mov dg0v.16b, dgav.16b
mov dg1v.16b, dgbv.16b
add_update 0, v1, 16, 17, 18, 19
add_update 1, v2, 17, 18, 19, 16
@@ -126,32 +122,14 @@ CPU_LE( rev32 v19.16b, v19.16b )
/* update state */
add dgav.4s, dgav.4s, dg0v.4s
add dgbv.4s, dgbv.4s, dg1v.4s
/* handled all input blocks? */
- cbz w2, 2f
+ cbz x2, 1f
cond_yield 3f, x5, x6
b 0b
- /*
- * Final block: add padding and total bit count.
- * Skip if the input size was not a round multiple of the block size,
- * the padding is handled by the C code in that case.
- */
-2: cbz x4, 3f
- ldr_l w4, sha256_ce_offsetof_count, x4
- ldr x4, [x0, x4]
- movi v17.2d, #0
- mov x8, #0x80000000
- movi v18.2d, #0
- ror x7, x4, #29 // ror(lsl(x4, 3), 32)
- fmov d16, x8
- mov x4, #0
- mov v19.d[0], xzr
- mov v19.d[1], x7
- b 1b
-
/* store new state */
-3: st1 {dgav.4s, dgbv.4s}, [x0]
- mov w0, w2
+1: st1 {dgav.4s, dgbv.4s}, [x0]
+ mov x0, x2
ret
SYM_FUNC_END(__sha256_ce_transform)
diff --git a/arch/arm64/lib/crypto/sha256.c b/arch/arm64/lib/crypto/sha256.c
new file mode 100644
index 0000000000000..2bd413c586d27
--- /dev/null
+++ b/arch/arm64/lib/crypto/sha256.c
@@ -0,0 +1,75 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * SHA-256 optimized for ARM64
+ *
+ * Copyright 2025 Google LLC
+ */
+#include <asm/neon.h>
+#include <crypto/internal/sha2.h>
+#include <crypto/internal/simd.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+asmlinkage void sha256_block_data_order(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+asmlinkage void sha256_block_neon(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+asmlinkage size_t __sha256_ce_transform(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_neon);
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_ce);
+
+void sha256_blocks_arch(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks)
+{
+ if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) &&
+ static_branch_likely(&have_neon) && crypto_simd_usable()) {
+ if (static_branch_likely(&have_ce)) {
+ do {
+ size_t rem;
+
+ kernel_neon_begin();
+ rem = __sha256_ce_transform(state,
+ data, nblocks);
+ kernel_neon_end();
+ data += (nblocks - rem) * SHA256_BLOCK_SIZE;
+ nblocks = rem;
+ } while (nblocks);
+ } else {
+ kernel_neon_begin();
+ sha256_block_neon(state, data, nblocks);
+ kernel_neon_end();
+ }
+ } else {
+ sha256_block_data_order(state, data, nblocks);
+ }
+}
+EXPORT_SYMBOL(sha256_blocks_arch);
+
+bool sha256_is_arch_optimized(void)
+{
+ /* We always can use at least the ARM64 scalar implementation. */
+ return true;
+}
+EXPORT_SYMBOL(sha256_is_arch_optimized);
+
+static int __init sha256_arm64_mod_init(void)
+{
+ if (IS_ENABLED(CONFIG_KERNEL_MODE_NEON) &&
+ cpu_have_named_feature(ASIMD)) {
+ static_branch_enable(&have_neon);
+ if (cpu_have_named_feature(SHA2))
+ static_branch_enable(&have_ce);
+ }
+ return 0;
+}
+arch_initcall(sha256_arm64_mod_init);
+
+static void __exit sha256_arm64_mod_exit(void)
+{
+}
+module_exit(sha256_arm64_mod_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("SHA-256 optimized for ARM64");
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 05/13] crypto: mips/sha256 - implement library instead of shash
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
` (3 preceding siblings ...)
2025-04-28 17:00 ` [PATCH v4 04/13] crypto: arm64/sha256 - implement library instead of shash Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-04-28 17:00 ` [PATCH v4 06/13] crypto: powerpc/sha256 " Eric Biggers
` (8 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Instead of providing crypto_shash algorithms for the arch-optimized
SHA-256 code, instead implement the SHA-256 library. This is much
simpler, it makes the SHA-256 library functions be arch-optimized, and
it fixes the longstanding issue where the arch-optimized SHA-256 was
disabled by default. SHA-256 still remains available through
crypto_shash, but individual architectures no longer need to handle it.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/mips/cavium-octeon/Kconfig | 6 +
.../mips/cavium-octeon/crypto/octeon-sha256.c | 135 ++++--------------
arch/mips/configs/cavium_octeon_defconfig | 1 -
arch/mips/crypto/Kconfig | 10 --
4 files changed, 33 insertions(+), 119 deletions(-)
diff --git a/arch/mips/cavium-octeon/Kconfig b/arch/mips/cavium-octeon/Kconfig
index 450e979ef5d93..11f4aa6e80e9b 100644
--- a/arch/mips/cavium-octeon/Kconfig
+++ b/arch/mips/cavium-octeon/Kconfig
@@ -21,10 +21,16 @@ config CAVIUM_OCTEON_CVMSEG_SIZE
local memory; the larger CVMSEG is, the smaller the cache is.
This selects the size of CVMSEG LM, which is in cache blocks. The
legally range is from zero to 54 cache blocks (i.e. CVMSEG LM is
between zero and 6192 bytes).
+config CRYPTO_SHA256_OCTEON
+ tristate
+ default CRYPTO_LIB_SHA256
+ select CRYPTO_ARCH_HAVE_LIB_SHA256
+ select CRYPTO_LIB_SHA256_GENERIC
+
endif # CPU_CAVIUM_OCTEON
if CAVIUM_OCTEON_SOC
config CAVIUM_OCTEON_LOCK_L2
diff --git a/arch/mips/cavium-octeon/crypto/octeon-sha256.c b/arch/mips/cavium-octeon/crypto/octeon-sha256.c
index 8e85ea65387c8..f169054852bcb 100644
--- a/arch/mips/cavium-octeon/crypto/octeon-sha256.c
+++ b/arch/mips/cavium-octeon/crypto/octeon-sha256.c
@@ -1,10 +1,8 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
- * Cryptographic API.
- *
- * SHA-224 and SHA-256 Secure Hash Algorithm.
+ * SHA-256 Secure Hash Algorithm.
*
* Adapted for OCTEON by Aaro Koskinen <aaro.koskinen@iki.fi>.
*
* Based on crypto/sha256_generic.c, which is:
*
@@ -13,142 +11,63 @@
* Copyright (c) 2002 James Morris <jmorris@intercode.com.au>
* SHA224 Support Copyright 2007 Intel Corporation <jonathan.lynch@intel.com>
*/
#include <asm/octeon/octeon.h>
-#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
-#include <crypto/sha256_base.h>
+#include <crypto/internal/sha2.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include "octeon-crypto.h"
/*
* We pass everything as 64-bit. OCTEON can handle misaligned data.
*/
-static void octeon_sha256_store_hash(struct crypto_sha256_state *sctx)
+void sha256_blocks_arch(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks)
{
- u64 *hash = (u64 *)sctx->state;
-
- write_octeon_64bit_hash_dword(hash[0], 0);
- write_octeon_64bit_hash_dword(hash[1], 1);
- write_octeon_64bit_hash_dword(hash[2], 2);
- write_octeon_64bit_hash_dword(hash[3], 3);
-}
+ struct octeon_cop2_state cop2_state;
+ u64 *state64 = (u64 *)state;
+ unsigned long flags;
-static void octeon_sha256_read_hash(struct crypto_sha256_state *sctx)
-{
- u64 *hash = (u64 *)sctx->state;
+ if (!octeon_has_crypto())
+ return sha256_blocks_generic(state, data, nblocks);
- hash[0] = read_octeon_64bit_hash_dword(0);
- hash[1] = read_octeon_64bit_hash_dword(1);
- hash[2] = read_octeon_64bit_hash_dword(2);
- hash[3] = read_octeon_64bit_hash_dword(3);
-}
+ flags = octeon_crypto_enable(&cop2_state);
+ write_octeon_64bit_hash_dword(state64[0], 0);
+ write_octeon_64bit_hash_dword(state64[1], 1);
+ write_octeon_64bit_hash_dword(state64[2], 2);
+ write_octeon_64bit_hash_dword(state64[3], 3);
-static void octeon_sha256_transform(struct crypto_sha256_state *sctx,
- const u8 *src, int blocks)
-{
do {
- const u64 *block = (const u64 *)src;
+ const u64 *block = (const u64 *)data;
write_octeon_64bit_block_dword(block[0], 0);
write_octeon_64bit_block_dword(block[1], 1);
write_octeon_64bit_block_dword(block[2], 2);
write_octeon_64bit_block_dword(block[3], 3);
write_octeon_64bit_block_dword(block[4], 4);
write_octeon_64bit_block_dword(block[5], 5);
write_octeon_64bit_block_dword(block[6], 6);
octeon_sha256_start(block[7]);
- src += SHA256_BLOCK_SIZE;
- } while (--blocks);
-}
-
-static int octeon_sha256_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- struct crypto_sha256_state *sctx = shash_desc_ctx(desc);
- struct octeon_cop2_state state;
- unsigned long flags;
- int remain;
-
- flags = octeon_crypto_enable(&state);
- octeon_sha256_store_hash(sctx);
-
- remain = sha256_base_do_update_blocks(desc, data, len,
- octeon_sha256_transform);
+ data += SHA256_BLOCK_SIZE;
+ } while (--nblocks);
- octeon_sha256_read_hash(sctx);
- octeon_crypto_disable(&state, flags);
- return remain;
+ state64[0] = read_octeon_64bit_hash_dword(0);
+ state64[1] = read_octeon_64bit_hash_dword(1);
+ state64[2] = read_octeon_64bit_hash_dword(2);
+ state64[3] = read_octeon_64bit_hash_dword(3);
+ octeon_crypto_disable(&cop2_state, flags);
}
+EXPORT_SYMBOL(sha256_blocks_arch);
-static int octeon_sha256_finup(struct shash_desc *desc, const u8 *src,
- unsigned int len, u8 *out)
+bool sha256_is_arch_optimized(void)
{
- struct crypto_sha256_state *sctx = shash_desc_ctx(desc);
- struct octeon_cop2_state state;
- unsigned long flags;
-
- flags = octeon_crypto_enable(&state);
- octeon_sha256_store_hash(sctx);
-
- sha256_base_do_finup(desc, src, len, octeon_sha256_transform);
-
- octeon_sha256_read_hash(sctx);
- octeon_crypto_disable(&state, flags);
- return sha256_base_finish(desc, out);
+ return octeon_has_crypto();
}
-
-static struct shash_alg octeon_sha256_algs[2] = { {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = octeon_sha256_update,
- .finup = octeon_sha256_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha256",
- .cra_driver_name= "octeon-sha256",
- .cra_priority = OCTEON_CR_OPCODE_PRIORITY,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-}, {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = octeon_sha256_update,
- .finup = octeon_sha256_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha224",
- .cra_driver_name= "octeon-sha224",
- .cra_priority = OCTEON_CR_OPCODE_PRIORITY,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-} };
-
-static int __init octeon_sha256_mod_init(void)
-{
- if (!octeon_has_crypto())
- return -ENOTSUPP;
- return crypto_register_shashes(octeon_sha256_algs,
- ARRAY_SIZE(octeon_sha256_algs));
-}
-
-static void __exit octeon_sha256_mod_fini(void)
-{
- crypto_unregister_shashes(octeon_sha256_algs,
- ARRAY_SIZE(octeon_sha256_algs));
-}
-
-module_init(octeon_sha256_mod_init);
-module_exit(octeon_sha256_mod_fini);
+EXPORT_SYMBOL(sha256_is_arch_optimized);
MODULE_LICENSE("GPL");
-MODULE_DESCRIPTION("SHA-224 and SHA-256 Secure Hash Algorithm (OCTEON)");
+MODULE_DESCRIPTION("SHA-256 Secure Hash Algorithm (OCTEON)");
MODULE_AUTHOR("Aaro Koskinen <aaro.koskinen@iki.fi>");
diff --git a/arch/mips/configs/cavium_octeon_defconfig b/arch/mips/configs/cavium_octeon_defconfig
index f523ee6f25bfe..88ae0aa85364b 100644
--- a/arch/mips/configs/cavium_octeon_defconfig
+++ b/arch/mips/configs/cavium_octeon_defconfig
@@ -155,11 +155,10 @@ CONFIG_SECURITY=y
CONFIG_SECURITY_NETWORK=y
CONFIG_CRYPTO_CBC=y
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_MD5_OCTEON=y
CONFIG_CRYPTO_SHA1_OCTEON=m
-CONFIG_CRYPTO_SHA256_OCTEON=m
CONFIG_CRYPTO_SHA512_OCTEON=m
CONFIG_CRYPTO_DES=y
CONFIG_DEBUG_INFO_DWARF_TOOLCHAIN_DEFAULT=y
CONFIG_DEBUG_FS=y
CONFIG_MAGIC_SYSRQ=y
diff --git a/arch/mips/crypto/Kconfig b/arch/mips/crypto/Kconfig
index 9db1fd6d9f0e0..6bf073ae7613f 100644
--- a/arch/mips/crypto/Kconfig
+++ b/arch/mips/crypto/Kconfig
@@ -20,20 +20,10 @@ config CRYPTO_SHA1_OCTEON
help
SHA-1 secure hash algorithm (FIPS 180)
Architecture: mips OCTEON
-config CRYPTO_SHA256_OCTEON
- tristate "Hash functions: SHA-224 and SHA-256 (OCTEON)"
- depends on CPU_CAVIUM_OCTEON
- select CRYPTO_SHA256
- select CRYPTO_HASH
- help
- SHA-224 and SHA-256 secure hash algorithms (FIPS 180)
-
- Architecture: mips OCTEON using crypto instructions, when available
-
config CRYPTO_SHA512_OCTEON
tristate "Hash functions: SHA-384 and SHA-512 (OCTEON)"
depends on CPU_CAVIUM_OCTEON
select CRYPTO_SHA512
select CRYPTO_HASH
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 06/13] crypto: powerpc/sha256 - implement library instead of shash
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
` (4 preceding siblings ...)
2025-04-28 17:00 ` [PATCH v4 05/13] crypto: mips/sha256 " Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-04-28 17:00 ` [PATCH v4 07/13] crypto: riscv/sha256 " Eric Biggers
` (7 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Instead of providing crypto_shash algorithms for the arch-optimized
SHA-256 code, instead implement the SHA-256 library. This is much
simpler, it makes the SHA-256 library functions be arch-optimized, and
it fixes the longstanding issue where the arch-optimized SHA-256 was
disabled by default. SHA-256 still remains available through
crypto_shash, but individual architectures no longer need to handle it.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/powerpc/crypto/Kconfig | 11 --
arch/powerpc/crypto/Makefile | 2 -
arch/powerpc/crypto/sha256-spe-glue.c | 128 ------------------
arch/powerpc/lib/crypto/Kconfig | 6 +
arch/powerpc/lib/crypto/Makefile | 3 +
.../powerpc/{ => lib}/crypto/sha256-spe-asm.S | 0
arch/powerpc/lib/crypto/sha256.c | 70 ++++++++++
7 files changed, 79 insertions(+), 141 deletions(-)
delete mode 100644 arch/powerpc/crypto/sha256-spe-glue.c
rename arch/powerpc/{ => lib}/crypto/sha256-spe-asm.S (100%)
create mode 100644 arch/powerpc/lib/crypto/sha256.c
diff --git a/arch/powerpc/crypto/Kconfig b/arch/powerpc/crypto/Kconfig
index 4bf7b01228e72..caaa359f47420 100644
--- a/arch/powerpc/crypto/Kconfig
+++ b/arch/powerpc/crypto/Kconfig
@@ -37,21 +37,10 @@ config CRYPTO_SHA1_PPC_SPE
SHA-1 secure hash algorithm (FIPS 180)
Architecture: powerpc using
- SPE (Signal Processing Engine) extensions
-config CRYPTO_SHA256_PPC_SPE
- tristate "Hash functions: SHA-224 and SHA-256 (SPE)"
- depends on SPE
- select CRYPTO_SHA256
- select CRYPTO_HASH
- help
- SHA-224 and SHA-256 secure hash algorithms (FIPS 180)
-
- Architecture: powerpc using
- - SPE (Signal Processing Engine) extensions
-
config CRYPTO_AES_PPC_SPE
tristate "Ciphers: AES, modes: ECB/CBC/CTR/XTS (SPE)"
depends on SPE
select CRYPTO_SKCIPHER
help
diff --git a/arch/powerpc/crypto/Makefile b/arch/powerpc/crypto/Makefile
index f13aec8a18335..8c2936ae466fc 100644
--- a/arch/powerpc/crypto/Makefile
+++ b/arch/powerpc/crypto/Makefile
@@ -7,20 +7,18 @@
obj-$(CONFIG_CRYPTO_AES_PPC_SPE) += aes-ppc-spe.o
obj-$(CONFIG_CRYPTO_MD5_PPC) += md5-ppc.o
obj-$(CONFIG_CRYPTO_SHA1_PPC) += sha1-powerpc.o
obj-$(CONFIG_CRYPTO_SHA1_PPC_SPE) += sha1-ppc-spe.o
-obj-$(CONFIG_CRYPTO_SHA256_PPC_SPE) += sha256-ppc-spe.o
obj-$(CONFIG_CRYPTO_AES_GCM_P10) += aes-gcm-p10-crypto.o
obj-$(CONFIG_CRYPTO_DEV_VMX_ENCRYPT) += vmx-crypto.o
obj-$(CONFIG_CRYPTO_CURVE25519_PPC64) += curve25519-ppc64le.o
aes-ppc-spe-y := aes-spe-core.o aes-spe-keys.o aes-tab-4k.o aes-spe-modes.o aes-spe-glue.o
md5-ppc-y := md5-asm.o md5-glue.o
sha1-powerpc-y := sha1-powerpc-asm.o sha1.o
sha1-ppc-spe-y := sha1-spe-asm.o sha1-spe-glue.o
-sha256-ppc-spe-y := sha256-spe-asm.o sha256-spe-glue.o
aes-gcm-p10-crypto-y := aes-gcm-p10-glue.o aes-gcm-p10.o ghashp10-ppc.o aesp10-ppc.o
vmx-crypto-objs := vmx.o aesp8-ppc.o ghashp8-ppc.o aes.o aes_cbc.o aes_ctr.o aes_xts.o ghash.o
curve25519-ppc64le-y := curve25519-ppc64le-core.o curve25519-ppc64le_asm.o
ifeq ($(CONFIG_CPU_LITTLE_ENDIAN),y)
diff --git a/arch/powerpc/crypto/sha256-spe-glue.c b/arch/powerpc/crypto/sha256-spe-glue.c
deleted file mode 100644
index 42c76bf8062dc..0000000000000
--- a/arch/powerpc/crypto/sha256-spe-glue.c
+++ /dev/null
@@ -1,128 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Glue code for SHA-256 implementation for SPE instructions (PPC)
- *
- * Based on generic implementation. The assembler module takes care
- * about the SPE registers so it can run from interrupt context.
- *
- * Copyright (c) 2015 Markus Stockhausen <stockhausen@collogia.de>
- */
-
-#include <asm/switch_to.h>
-#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
-#include <crypto/sha256_base.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/preempt.h>
-
-/*
- * MAX_BYTES defines the number of bytes that are allowed to be processed
- * between preempt_disable() and preempt_enable(). SHA256 takes ~2,000
- * operations per 64 bytes. e500 cores can issue two arithmetic instructions
- * per clock cycle using one 32/64 bit unit (SU1) and one 32 bit unit (SU2).
- * Thus 1KB of input data will need an estimated maximum of 18,000 cycles.
- * Headroom for cache misses included. Even with the low end model clocked
- * at 667 MHz this equals to a critical time window of less than 27us.
- *
- */
-#define MAX_BYTES 1024
-
-extern void ppc_spe_sha256_transform(u32 *state, const u8 *src, u32 blocks);
-
-static void spe_begin(void)
-{
- /* We just start SPE operations and will save SPE registers later. */
- preempt_disable();
- enable_kernel_spe();
-}
-
-static void spe_end(void)
-{
- disable_kernel_spe();
- /* reenable preemption */
- preempt_enable();
-}
-
-static void ppc_spe_sha256_block(struct crypto_sha256_state *sctx,
- const u8 *src, int blocks)
-{
- do {
- /* cut input data into smaller blocks */
- int unit = min(blocks, MAX_BYTES / SHA256_BLOCK_SIZE);
-
- spe_begin();
- ppc_spe_sha256_transform(sctx->state, src, unit);
- spe_end();
-
- src += unit * SHA256_BLOCK_SIZE;
- blocks -= unit;
- } while (blocks);
-}
-
-static int ppc_spe_sha256_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- return sha256_base_do_update_blocks(desc, data, len,
- ppc_spe_sha256_block);
-}
-
-static int ppc_spe_sha256_finup(struct shash_desc *desc, const u8 *src,
- unsigned int len, u8 *out)
-{
- sha256_base_do_finup(desc, src, len, ppc_spe_sha256_block);
- return sha256_base_finish(desc, out);
-}
-
-static struct shash_alg algs[2] = { {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = ppc_spe_sha256_update,
- .finup = ppc_spe_sha256_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha256",
- .cra_driver_name= "sha256-ppc-spe",
- .cra_priority = 300,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-}, {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = ppc_spe_sha256_update,
- .finup = ppc_spe_sha256_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha224",
- .cra_driver_name= "sha224-ppc-spe",
- .cra_priority = 300,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-} };
-
-static int __init ppc_spe_sha256_mod_init(void)
-{
- return crypto_register_shashes(algs, ARRAY_SIZE(algs));
-}
-
-static void __exit ppc_spe_sha256_mod_fini(void)
-{
- crypto_unregister_shashes(algs, ARRAY_SIZE(algs));
-}
-
-module_init(ppc_spe_sha256_mod_init);
-module_exit(ppc_spe_sha256_mod_fini);
-
-MODULE_LICENSE("GPL");
-MODULE_DESCRIPTION("SHA-224 and SHA-256 Secure Hash Algorithm, SPE optimized");
-
-MODULE_ALIAS_CRYPTO("sha224");
-MODULE_ALIAS_CRYPTO("sha224-ppc-spe");
-MODULE_ALIAS_CRYPTO("sha256");
-MODULE_ALIAS_CRYPTO("sha256-ppc-spe");
diff --git a/arch/powerpc/lib/crypto/Kconfig b/arch/powerpc/lib/crypto/Kconfig
index bf6d0ab22c27d..ffa541ad6d5da 100644
--- a/arch/powerpc/lib/crypto/Kconfig
+++ b/arch/powerpc/lib/crypto/Kconfig
@@ -11,5 +11,11 @@ config CRYPTO_POLY1305_P10
tristate
depends on PPC64 && CPU_LITTLE_ENDIAN && VSX
default CRYPTO_LIB_POLY1305
select CRYPTO_ARCH_HAVE_LIB_POLY1305
select CRYPTO_LIB_POLY1305_GENERIC
+
+config CRYPTO_SHA256_PPC_SPE
+ tristate
+ depends on SPE
+ default CRYPTO_LIB_SHA256
+ select CRYPTO_ARCH_HAVE_LIB_SHA256
diff --git a/arch/powerpc/lib/crypto/Makefile b/arch/powerpc/lib/crypto/Makefile
index 5709ae14258a0..27f231f8e334a 100644
--- a/arch/powerpc/lib/crypto/Makefile
+++ b/arch/powerpc/lib/crypto/Makefile
@@ -3,5 +3,8 @@
obj-$(CONFIG_CRYPTO_CHACHA20_P10) += chacha-p10-crypto.o
chacha-p10-crypto-y := chacha-p10-glue.o chacha-p10le-8x.o
obj-$(CONFIG_CRYPTO_POLY1305_P10) += poly1305-p10-crypto.o
poly1305-p10-crypto-y := poly1305-p10-glue.o poly1305-p10le_64.o
+
+obj-$(CONFIG_CRYPTO_SHA256_PPC_SPE) += sha256-ppc-spe.o
+sha256-ppc-spe-y := sha256.o sha256-spe-asm.o
diff --git a/arch/powerpc/crypto/sha256-spe-asm.S b/arch/powerpc/lib/crypto/sha256-spe-asm.S
similarity index 100%
rename from arch/powerpc/crypto/sha256-spe-asm.S
rename to arch/powerpc/lib/crypto/sha256-spe-asm.S
diff --git a/arch/powerpc/lib/crypto/sha256.c b/arch/powerpc/lib/crypto/sha256.c
new file mode 100644
index 0000000000000..c05023c5acdd4
--- /dev/null
+++ b/arch/powerpc/lib/crypto/sha256.c
@@ -0,0 +1,70 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * SHA-256 Secure Hash Algorithm, SPE optimized
+ *
+ * Based on generic implementation. The assembler module takes care
+ * about the SPE registers so it can run from interrupt context.
+ *
+ * Copyright (c) 2015 Markus Stockhausen <stockhausen@collogia.de>
+ */
+
+#include <asm/switch_to.h>
+#include <crypto/internal/sha2.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/preempt.h>
+
+/*
+ * MAX_BYTES defines the number of bytes that are allowed to be processed
+ * between preempt_disable() and preempt_enable(). SHA256 takes ~2,000
+ * operations per 64 bytes. e500 cores can issue two arithmetic instructions
+ * per clock cycle using one 32/64 bit unit (SU1) and one 32 bit unit (SU2).
+ * Thus 1KB of input data will need an estimated maximum of 18,000 cycles.
+ * Headroom for cache misses included. Even with the low end model clocked
+ * at 667 MHz this equals to a critical time window of less than 27us.
+ *
+ */
+#define MAX_BYTES 1024
+
+extern void ppc_spe_sha256_transform(u32 *state, const u8 *src, u32 blocks);
+
+static void spe_begin(void)
+{
+ /* We just start SPE operations and will save SPE registers later. */
+ preempt_disable();
+ enable_kernel_spe();
+}
+
+static void spe_end(void)
+{
+ disable_kernel_spe();
+ /* reenable preemption */
+ preempt_enable();
+}
+
+void sha256_blocks_arch(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks)
+{
+ do {
+ /* cut input data into smaller blocks */
+ u32 unit = min_t(size_t, nblocks,
+ MAX_BYTES / SHA256_BLOCK_SIZE);
+
+ spe_begin();
+ ppc_spe_sha256_transform(state, data, unit);
+ spe_end();
+
+ data += unit * SHA256_BLOCK_SIZE;
+ nblocks -= unit;
+ } while (nblocks);
+}
+EXPORT_SYMBOL(sha256_blocks_arch);
+
+bool sha256_is_arch_optimized(void)
+{
+ return true;
+}
+EXPORT_SYMBOL(sha256_is_arch_optimized);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("SHA-256 Secure Hash Algorithm, SPE optimized");
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 07/13] crypto: riscv/sha256 - implement library instead of shash
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
` (5 preceding siblings ...)
2025-04-28 17:00 ` [PATCH v4 06/13] crypto: powerpc/sha256 " Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-05-08 17:45 ` Palmer Dabbelt
2025-04-28 17:00 ` [PATCH v4 08/13] crypto: s390/sha256 " Eric Biggers
` (6 subsequent siblings)
13 siblings, 1 reply; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Instead of providing crypto_shash algorithms for the arch-optimized
SHA-256 code, instead implement the SHA-256 library. This is much
simpler, it makes the SHA-256 library functions be arch-optimized, and
it fixes the longstanding issue where the arch-optimized SHA-256 was
disabled by default. SHA-256 still remains available through
crypto_shash, but individual architectures no longer need to handle it.
To match sha256_blocks_arch(), change the type of the nblocks parameter
of the assembly function from int to size_t. The assembly function
actually already treated it as size_t.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/riscv/crypto/Kconfig | 11 --
arch/riscv/crypto/Makefile | 3 -
arch/riscv/crypto/sha256-riscv64-glue.c | 125 ------------------
arch/riscv/lib/crypto/Kconfig | 7 +
arch/riscv/lib/crypto/Makefile | 3 +
.../sha256-riscv64-zvknha_or_zvknhb-zvkb.S | 4 +-
arch/riscv/lib/crypto/sha256.c | 62 +++++++++
7 files changed, 74 insertions(+), 141 deletions(-)
delete mode 100644 arch/riscv/crypto/sha256-riscv64-glue.c
rename arch/riscv/{ => lib}/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S (98%)
create mode 100644 arch/riscv/lib/crypto/sha256.c
diff --git a/arch/riscv/crypto/Kconfig b/arch/riscv/crypto/Kconfig
index 4863be2a4ec2f..cd9b776602f89 100644
--- a/arch/riscv/crypto/Kconfig
+++ b/arch/riscv/crypto/Kconfig
@@ -26,21 +26,10 @@ config CRYPTO_GHASH_RISCV64
GCM GHASH function (NIST SP 800-38D)
Architecture: riscv64 using:
- Zvkg vector crypto extension
-config CRYPTO_SHA256_RISCV64
- tristate "Hash functions: SHA-224 and SHA-256"
- depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO
- select CRYPTO_SHA256
- help
- SHA-224 and SHA-256 secure hash algorithm (FIPS 180)
-
- Architecture: riscv64 using:
- - Zvknha or Zvknhb vector crypto extensions
- - Zvkb vector crypto extension
-
config CRYPTO_SHA512_RISCV64
tristate "Hash functions: SHA-384 and SHA-512"
depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO
select CRYPTO_SHA512
help
diff --git a/arch/riscv/crypto/Makefile b/arch/riscv/crypto/Makefile
index 4ae9bf762e907..e10e8257734e3 100644
--- a/arch/riscv/crypto/Makefile
+++ b/arch/riscv/crypto/Makefile
@@ -5,13 +5,10 @@ aes-riscv64-y := aes-riscv64-glue.o aes-riscv64-zvkned.o \
aes-riscv64-zvkned-zvbb-zvkg.o aes-riscv64-zvkned-zvkb.o
obj-$(CONFIG_CRYPTO_GHASH_RISCV64) += ghash-riscv64.o
ghash-riscv64-y := ghash-riscv64-glue.o ghash-riscv64-zvkg.o
-obj-$(CONFIG_CRYPTO_SHA256_RISCV64) += sha256-riscv64.o
-sha256-riscv64-y := sha256-riscv64-glue.o sha256-riscv64-zvknha_or_zvknhb-zvkb.o
-
obj-$(CONFIG_CRYPTO_SHA512_RISCV64) += sha512-riscv64.o
sha512-riscv64-y := sha512-riscv64-glue.o sha512-riscv64-zvknhb-zvkb.o
obj-$(CONFIG_CRYPTO_SM3_RISCV64) += sm3-riscv64.o
sm3-riscv64-y := sm3-riscv64-glue.o sm3-riscv64-zvksh-zvkb.o
diff --git a/arch/riscv/crypto/sha256-riscv64-glue.c b/arch/riscv/crypto/sha256-riscv64-glue.c
deleted file mode 100644
index c998300ab8435..0000000000000
--- a/arch/riscv/crypto/sha256-riscv64-glue.c
+++ /dev/null
@@ -1,125 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * SHA-256 and SHA-224 using the RISC-V vector crypto extensions
- *
- * Copyright (C) 2022 VRULL GmbH
- * Author: Heiko Stuebner <heiko.stuebner@vrull.eu>
- *
- * Copyright (C) 2023 SiFive, Inc.
- * Author: Jerry Shih <jerry.shih@sifive.com>
- */
-
-#include <asm/simd.h>
-#include <asm/vector.h>
-#include <crypto/internal/hash.h>
-#include <crypto/internal/simd.h>
-#include <crypto/sha256_base.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-
-/*
- * Note: the asm function only uses the 'state' field of struct sha256_state.
- * It is assumed to be the first field.
- */
-asmlinkage void sha256_transform_zvknha_or_zvknhb_zvkb(
- struct crypto_sha256_state *state, const u8 *data, int num_blocks);
-
-static void sha256_block(struct crypto_sha256_state *state, const u8 *data,
- int num_blocks)
-{
- /*
- * Ensure struct crypto_sha256_state begins directly with the SHA-256
- * 256-bit internal state, as this is what the asm function expects.
- */
- BUILD_BUG_ON(offsetof(struct crypto_sha256_state, state) != 0);
-
- if (crypto_simd_usable()) {
- kernel_vector_begin();
- sha256_transform_zvknha_or_zvknhb_zvkb(state, data, num_blocks);
- kernel_vector_end();
- } else
- sha256_transform_blocks(state, data, num_blocks);
-}
-
-static int riscv64_sha256_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- return sha256_base_do_update_blocks(desc, data, len, sha256_block);
-}
-
-static int riscv64_sha256_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- sha256_base_do_finup(desc, data, len, sha256_block);
- return sha256_base_finish(desc, out);
-}
-
-static int riscv64_sha256_digest(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- return sha256_base_init(desc) ?:
- riscv64_sha256_finup(desc, data, len, out);
-}
-
-static struct shash_alg riscv64_sha256_algs[] = {
- {
- .init = sha256_base_init,
- .update = riscv64_sha256_update,
- .finup = riscv64_sha256_finup,
- .digest = riscv64_sha256_digest,
- .descsize = sizeof(struct crypto_sha256_state),
- .digestsize = SHA256_DIGEST_SIZE,
- .base = {
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_priority = 300,
- .cra_name = "sha256",
- .cra_driver_name = "sha256-riscv64-zvknha_or_zvknhb-zvkb",
- .cra_module = THIS_MODULE,
- },
- }, {
- .init = sha224_base_init,
- .update = riscv64_sha256_update,
- .finup = riscv64_sha256_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .digestsize = SHA224_DIGEST_SIZE,
- .base = {
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_priority = 300,
- .cra_name = "sha224",
- .cra_driver_name = "sha224-riscv64-zvknha_or_zvknhb-zvkb",
- .cra_module = THIS_MODULE,
- },
- },
-};
-
-static int __init riscv64_sha256_mod_init(void)
-{
- /* Both zvknha and zvknhb provide the SHA-256 instructions. */
- if ((riscv_isa_extension_available(NULL, ZVKNHA) ||
- riscv_isa_extension_available(NULL, ZVKNHB)) &&
- riscv_isa_extension_available(NULL, ZVKB) &&
- riscv_vector_vlen() >= 128)
- return crypto_register_shashes(riscv64_sha256_algs,
- ARRAY_SIZE(riscv64_sha256_algs));
-
- return -ENODEV;
-}
-
-static void __exit riscv64_sha256_mod_exit(void)
-{
- crypto_unregister_shashes(riscv64_sha256_algs,
- ARRAY_SIZE(riscv64_sha256_algs));
-}
-
-module_init(riscv64_sha256_mod_init);
-module_exit(riscv64_sha256_mod_exit);
-
-MODULE_DESCRIPTION("SHA-256 (RISC-V accelerated)");
-MODULE_AUTHOR("Heiko Stuebner <heiko.stuebner@vrull.eu>");
-MODULE_LICENSE("GPL");
-MODULE_ALIAS_CRYPTO("sha256");
-MODULE_ALIAS_CRYPTO("sha224");
diff --git a/arch/riscv/lib/crypto/Kconfig b/arch/riscv/lib/crypto/Kconfig
index bc7a43f33eb3a..c100571feb7e8 100644
--- a/arch/riscv/lib/crypto/Kconfig
+++ b/arch/riscv/lib/crypto/Kconfig
@@ -4,5 +4,12 @@ config CRYPTO_CHACHA_RISCV64
tristate
depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO
default CRYPTO_LIB_CHACHA
select CRYPTO_ARCH_HAVE_LIB_CHACHA
select CRYPTO_LIB_CHACHA_GENERIC
+
+config CRYPTO_SHA256_RISCV64
+ tristate
+ depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO
+ default CRYPTO_LIB_SHA256
+ select CRYPTO_ARCH_HAVE_LIB_SHA256
+ select CRYPTO_LIB_SHA256_GENERIC
diff --git a/arch/riscv/lib/crypto/Makefile b/arch/riscv/lib/crypto/Makefile
index e27b78f317fc8..b7cb877a2c07e 100644
--- a/arch/riscv/lib/crypto/Makefile
+++ b/arch/riscv/lib/crypto/Makefile
@@ -1,4 +1,7 @@
# SPDX-License-Identifier: GPL-2.0-only
obj-$(CONFIG_CRYPTO_CHACHA_RISCV64) += chacha-riscv64.o
chacha-riscv64-y := chacha-riscv64-glue.o chacha-riscv64-zvkb.o
+
+obj-$(CONFIG_CRYPTO_SHA256_RISCV64) += sha256-riscv64.o
+sha256-riscv64-y := sha256.o sha256-riscv64-zvknha_or_zvknhb-zvkb.o
diff --git a/arch/riscv/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S b/arch/riscv/lib/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S
similarity index 98%
rename from arch/riscv/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S
rename to arch/riscv/lib/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S
index f1f5779e47323..fad501ad06171 100644
--- a/arch/riscv/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S
+++ b/arch/riscv/lib/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S
@@ -104,12 +104,12 @@
sha256_4rounds \last, \k1, W1, W2, W3, W0
sha256_4rounds \last, \k2, W2, W3, W0, W1
sha256_4rounds \last, \k3, W3, W0, W1, W2
.endm
-// void sha256_transform_zvknha_or_zvknhb_zvkb(u32 state[8], const u8 *data,
-// int num_blocks);
+// void sha256_transform_zvknha_or_zvknhb_zvkb(u32 state[SHA256_STATE_WORDS],
+// const u8 *data, size_t nblocks);
SYM_FUNC_START(sha256_transform_zvknha_or_zvknhb_zvkb)
// Load the round constants into K0-K15.
vsetivli zero, 4, e32, m1, ta, ma
la t0, K256
diff --git a/arch/riscv/lib/crypto/sha256.c b/arch/riscv/lib/crypto/sha256.c
new file mode 100644
index 0000000000000..18b84030f0b39
--- /dev/null
+++ b/arch/riscv/lib/crypto/sha256.c
@@ -0,0 +1,62 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * SHA-256 (RISC-V accelerated)
+ *
+ * Copyright (C) 2022 VRULL GmbH
+ * Author: Heiko Stuebner <heiko.stuebner@vrull.eu>
+ *
+ * Copyright (C) 2023 SiFive, Inc.
+ * Author: Jerry Shih <jerry.shih@sifive.com>
+ */
+
+#include <asm/simd.h>
+#include <asm/vector.h>
+#include <crypto/internal/sha2.h>
+#include <crypto/internal/simd.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+asmlinkage void sha256_transform_zvknha_or_zvknhb_zvkb(
+ u32 state[SHA256_STATE_WORDS], const u8 *data, size_t nblocks);
+
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_extensions);
+
+void sha256_blocks_arch(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks)
+{
+ if (static_branch_likely(&have_extensions) && crypto_simd_usable()) {
+ kernel_vector_begin();
+ sha256_transform_zvknha_or_zvknhb_zvkb(state, data, nblocks);
+ kernel_vector_end();
+ } else {
+ sha256_blocks_generic(state, data, nblocks);
+ }
+}
+EXPORT_SYMBOL(sha256_blocks_arch);
+
+bool sha256_is_arch_optimized(void)
+{
+ return static_key_enabled(&have_extensions);
+}
+EXPORT_SYMBOL(sha256_is_arch_optimized);
+
+static int __init riscv64_sha256_mod_init(void)
+{
+ /* Both zvknha and zvknhb provide the SHA-256 instructions. */
+ if ((riscv_isa_extension_available(NULL, ZVKNHA) ||
+ riscv_isa_extension_available(NULL, ZVKNHB)) &&
+ riscv_isa_extension_available(NULL, ZVKB) &&
+ riscv_vector_vlen() >= 128)
+ static_branch_enable(&have_extensions);
+ return 0;
+}
+arch_initcall(riscv64_sha256_mod_init);
+
+static void __exit riscv64_sha256_mod_exit(void)
+{
+}
+module_exit(riscv64_sha256_mod_exit);
+
+MODULE_DESCRIPTION("SHA-256 (RISC-V accelerated)");
+MODULE_AUTHOR("Heiko Stuebner <heiko.stuebner@vrull.eu>");
+MODULE_LICENSE("GPL");
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
` (6 preceding siblings ...)
2025-04-28 17:00 ` [PATCH v4 07/13] crypto: riscv/sha256 " Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-05-29 17:05 ` Alex Williamson
2025-04-28 17:00 ` [PATCH v4 09/13] crypto: sparc - move opcodes.h into asm directory Eric Biggers
` (5 subsequent siblings)
13 siblings, 1 reply; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Instead of providing crypto_shash algorithms for the arch-optimized
SHA-256 code, instead implement the SHA-256 library. This is much
simpler, it makes the SHA-256 library functions be arch-optimized, and
it fixes the longstanding issue where the arch-optimized SHA-256 was
disabled by default. SHA-256 still remains available through
crypto_shash, but individual architectures no longer need to handle it.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/s390/configs/debug_defconfig | 1 -
arch/s390/configs/defconfig | 1 -
arch/s390/crypto/Kconfig | 10 ---
arch/s390/crypto/Makefile | 1 -
arch/s390/crypto/sha256_s390.c | 144 ------------------------------
arch/s390/lib/crypto/Kconfig | 6 ++
arch/s390/lib/crypto/Makefile | 2 +
arch/s390/lib/crypto/sha256.c | 47 ++++++++++
8 files changed, 55 insertions(+), 157 deletions(-)
delete mode 100644 arch/s390/crypto/sha256_s390.c
create mode 100644 arch/s390/lib/crypto/sha256.c
diff --git a/arch/s390/configs/debug_defconfig b/arch/s390/configs/debug_defconfig
index 6f2c9ce1b1548..de69faa4d94f3 100644
--- a/arch/s390/configs/debug_defconfig
+++ b/arch/s390/configs/debug_defconfig
@@ -793,11 +793,10 @@ CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CRYPTO_USER_API_SKCIPHER=m
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_USER_API_AEAD=m
CONFIG_CRYPTO_SHA512_S390=m
CONFIG_CRYPTO_SHA1_S390=m
-CONFIG_CRYPTO_SHA256_S390=m
CONFIG_CRYPTO_SHA3_256_S390=m
CONFIG_CRYPTO_SHA3_512_S390=m
CONFIG_CRYPTO_GHASH_S390=m
CONFIG_CRYPTO_AES_S390=m
CONFIG_CRYPTO_DES_S390=m
diff --git a/arch/s390/configs/defconfig b/arch/s390/configs/defconfig
index f18a7d97ac216..f12679448e976 100644
--- a/arch/s390/configs/defconfig
+++ b/arch/s390/configs/defconfig
@@ -780,11 +780,10 @@ CONFIG_CRYPTO_USER_API_HASH=m
CONFIG_CRYPTO_USER_API_SKCIPHER=m
CONFIG_CRYPTO_USER_API_RNG=m
CONFIG_CRYPTO_USER_API_AEAD=m
CONFIG_CRYPTO_SHA512_S390=m
CONFIG_CRYPTO_SHA1_S390=m
-CONFIG_CRYPTO_SHA256_S390=m
CONFIG_CRYPTO_SHA3_256_S390=m
CONFIG_CRYPTO_SHA3_512_S390=m
CONFIG_CRYPTO_GHASH_S390=m
CONFIG_CRYPTO_AES_S390=m
CONFIG_CRYPTO_DES_S390=m
diff --git a/arch/s390/crypto/Kconfig b/arch/s390/crypto/Kconfig
index a2bfd6eef0ca3..e2c27588b21a9 100644
--- a/arch/s390/crypto/Kconfig
+++ b/arch/s390/crypto/Kconfig
@@ -20,20 +20,10 @@ config CRYPTO_SHA1_S390
Architecture: s390
It is available as of z990.
-config CRYPTO_SHA256_S390
- tristate "Hash functions: SHA-224 and SHA-256"
- select CRYPTO_HASH
- help
- SHA-224 and SHA-256 secure hash algorithms (FIPS 180)
-
- Architecture: s390
-
- It is available as of z9.
-
config CRYPTO_SHA3_256_S390
tristate "Hash functions: SHA3-224 and SHA3-256"
select CRYPTO_HASH
help
SHA3-224 and SHA3-256 secure hash algorithms (FIPS 202)
diff --git a/arch/s390/crypto/Makefile b/arch/s390/crypto/Makefile
index e3853774e1a3a..21757d86cd499 100644
--- a/arch/s390/crypto/Makefile
+++ b/arch/s390/crypto/Makefile
@@ -2,11 +2,10 @@
#
# Cryptographic API
#
obj-$(CONFIG_CRYPTO_SHA1_S390) += sha1_s390.o sha_common.o
-obj-$(CONFIG_CRYPTO_SHA256_S390) += sha256_s390.o sha_common.o
obj-$(CONFIG_CRYPTO_SHA512_S390) += sha512_s390.o sha_common.o
obj-$(CONFIG_CRYPTO_SHA3_256_S390) += sha3_256_s390.o sha_common.o
obj-$(CONFIG_CRYPTO_SHA3_512_S390) += sha3_512_s390.o sha_common.o
obj-$(CONFIG_CRYPTO_DES_S390) += des_s390.o
obj-$(CONFIG_CRYPTO_AES_S390) += aes_s390.o
diff --git a/arch/s390/crypto/sha256_s390.c b/arch/s390/crypto/sha256_s390.c
deleted file mode 100644
index e6876c49414d5..0000000000000
--- a/arch/s390/crypto/sha256_s390.c
+++ /dev/null
@@ -1,144 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0+
-/*
- * Cryptographic API.
- *
- * s390 implementation of the SHA256 and SHA224 Secure Hash Algorithm.
- *
- * s390 Version:
- * Copyright IBM Corp. 2005, 2011
- * Author(s): Jan Glauber (jang@de.ibm.com)
- */
-#include <asm/cpacf.h>
-#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
-#include <linux/cpufeature.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/string.h>
-
-#include "sha.h"
-
-static int s390_sha256_init(struct shash_desc *desc)
-{
- struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
-
- sctx->state[0] = SHA256_H0;
- sctx->state[1] = SHA256_H1;
- sctx->state[2] = SHA256_H2;
- sctx->state[3] = SHA256_H3;
- sctx->state[4] = SHA256_H4;
- sctx->state[5] = SHA256_H5;
- sctx->state[6] = SHA256_H6;
- sctx->state[7] = SHA256_H7;
- sctx->count = 0;
- sctx->func = CPACF_KIMD_SHA_256;
-
- return 0;
-}
-
-static int sha256_export(struct shash_desc *desc, void *out)
-{
- struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
- struct crypto_sha256_state *octx = out;
-
- octx->count = sctx->count;
- memcpy(octx->state, sctx->state, sizeof(octx->state));
- return 0;
-}
-
-static int sha256_import(struct shash_desc *desc, const void *in)
-{
- struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
- const struct crypto_sha256_state *ictx = in;
-
- sctx->count = ictx->count;
- memcpy(sctx->state, ictx->state, sizeof(ictx->state));
- sctx->func = CPACF_KIMD_SHA_256;
- return 0;
-}
-
-static struct shash_alg sha256_alg = {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = s390_sha256_init,
- .update = s390_sha_update_blocks,
- .finup = s390_sha_finup,
- .export = sha256_export,
- .import = sha256_import,
- .descsize = S390_SHA_CTX_SIZE,
- .statesize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha256",
- .cra_driver_name= "sha256-s390",
- .cra_priority = 300,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-};
-
-static int s390_sha224_init(struct shash_desc *desc)
-{
- struct s390_sha_ctx *sctx = shash_desc_ctx(desc);
-
- sctx->state[0] = SHA224_H0;
- sctx->state[1] = SHA224_H1;
- sctx->state[2] = SHA224_H2;
- sctx->state[3] = SHA224_H3;
- sctx->state[4] = SHA224_H4;
- sctx->state[5] = SHA224_H5;
- sctx->state[6] = SHA224_H6;
- sctx->state[7] = SHA224_H7;
- sctx->count = 0;
- sctx->func = CPACF_KIMD_SHA_256;
-
- return 0;
-}
-
-static struct shash_alg sha224_alg = {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = s390_sha224_init,
- .update = s390_sha_update_blocks,
- .finup = s390_sha_finup,
- .export = sha256_export,
- .import = sha256_import,
- .descsize = S390_SHA_CTX_SIZE,
- .statesize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha224",
- .cra_driver_name= "sha224-s390",
- .cra_priority = 300,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY,
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-};
-
-static int __init sha256_s390_init(void)
-{
- int ret;
-
- if (!cpacf_query_func(CPACF_KIMD, CPACF_KIMD_SHA_256))
- return -ENODEV;
- ret = crypto_register_shash(&sha256_alg);
- if (ret < 0)
- goto out;
- ret = crypto_register_shash(&sha224_alg);
- if (ret < 0)
- crypto_unregister_shash(&sha256_alg);
-out:
- return ret;
-}
-
-static void __exit sha256_s390_fini(void)
-{
- crypto_unregister_shash(&sha224_alg);
- crypto_unregister_shash(&sha256_alg);
-}
-
-module_cpu_feature_match(S390_CPU_FEATURE_MSA, sha256_s390_init);
-module_exit(sha256_s390_fini);
-
-MODULE_ALIAS_CRYPTO("sha256");
-MODULE_ALIAS_CRYPTO("sha224");
-MODULE_LICENSE("GPL");
-MODULE_DESCRIPTION("SHA256 and SHA224 Secure Hash Algorithm");
diff --git a/arch/s390/lib/crypto/Kconfig b/arch/s390/lib/crypto/Kconfig
index 069b355fe51aa..e3f855ef43934 100644
--- a/arch/s390/lib/crypto/Kconfig
+++ b/arch/s390/lib/crypto/Kconfig
@@ -3,5 +3,11 @@
config CRYPTO_CHACHA_S390
tristate
default CRYPTO_LIB_CHACHA
select CRYPTO_LIB_CHACHA_GENERIC
select CRYPTO_ARCH_HAVE_LIB_CHACHA
+
+config CRYPTO_SHA256_S390
+ tristate
+ default CRYPTO_LIB_SHA256
+ select CRYPTO_ARCH_HAVE_LIB_SHA256
+ select CRYPTO_LIB_SHA256_GENERIC
diff --git a/arch/s390/lib/crypto/Makefile b/arch/s390/lib/crypto/Makefile
index 06c2cf77178ef..920197967f463 100644
--- a/arch/s390/lib/crypto/Makefile
+++ b/arch/s390/lib/crypto/Makefile
@@ -1,4 +1,6 @@
# SPDX-License-Identifier: GPL-2.0-only
obj-$(CONFIG_CRYPTO_CHACHA_S390) += chacha_s390.o
chacha_s390-y := chacha-glue.o chacha-s390.o
+
+obj-$(CONFIG_CRYPTO_SHA256_S390) += sha256.o
diff --git a/arch/s390/lib/crypto/sha256.c b/arch/s390/lib/crypto/sha256.c
new file mode 100644
index 0000000000000..50c592ce7a5de
--- /dev/null
+++ b/arch/s390/lib/crypto/sha256.c
@@ -0,0 +1,47 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * SHA-256 optimized using the CP Assist for Cryptographic Functions (CPACF)
+ *
+ * Copyright 2025 Google LLC
+ */
+#include <asm/cpacf.h>
+#include <crypto/internal/sha2.h>
+#include <linux/cpufeature.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_cpacf_sha256);
+
+void sha256_blocks_arch(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks)
+{
+ if (static_branch_likely(&have_cpacf_sha256))
+ cpacf_kimd(CPACF_KIMD_SHA_256, state, data,
+ nblocks * SHA256_BLOCK_SIZE);
+ else
+ sha256_blocks_generic(state, data, nblocks);
+}
+EXPORT_SYMBOL(sha256_blocks_arch);
+
+bool sha256_is_arch_optimized(void)
+{
+ return static_key_enabled(&have_cpacf_sha256);
+}
+EXPORT_SYMBOL(sha256_is_arch_optimized);
+
+static int __init sha256_s390_mod_init(void)
+{
+ if (cpu_have_feature(S390_CPU_FEATURE_MSA) &&
+ cpacf_query_func(CPACF_KIMD, CPACF_KIMD_SHA_256))
+ static_branch_enable(&have_cpacf_sha256);
+ return 0;
+}
+arch_initcall(sha256_s390_mod_init);
+
+static void __exit sha256_s390_mod_exit(void)
+{
+}
+module_exit(sha256_s390_mod_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("SHA-256 using the CP Assist for Cryptographic Functions (CPACF)");
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 09/13] crypto: sparc - move opcodes.h into asm directory
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
` (7 preceding siblings ...)
2025-04-28 17:00 ` [PATCH v4 08/13] crypto: s390/sha256 " Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-04-28 17:00 ` [PATCH v4 10/13] crypto: sparc/sha256 - implement library instead of shash Eric Biggers
` (4 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Since arch/sparc/crypto/opcodes.h is now needed outside the
arch/sparc/crypto/ directory, move it into arch/sparc/include/asm/ so
that it can be included as <asm/opcodes.h>.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/sparc/crypto/aes_asm.S | 3 +--
arch/sparc/crypto/aes_glue.c | 3 +--
arch/sparc/crypto/camellia_asm.S | 3 +--
arch/sparc/crypto/camellia_glue.c | 3 +--
arch/sparc/crypto/des_asm.S | 3 +--
arch/sparc/crypto/des_glue.c | 3 +--
arch/sparc/crypto/md5_asm.S | 3 +--
arch/sparc/crypto/md5_glue.c | 3 +--
arch/sparc/crypto/sha1_asm.S | 3 +--
arch/sparc/crypto/sha1_glue.c | 3 +--
arch/sparc/crypto/sha256_asm.S | 3 +--
arch/sparc/crypto/sha256_glue.c | 3 +--
arch/sparc/crypto/sha512_asm.S | 3 +--
arch/sparc/crypto/sha512_glue.c | 3 +--
arch/sparc/{crypto => include/asm}/opcodes.h | 6 +++---
arch/sparc/lib/crc32c_asm.S | 3 +--
16 files changed, 18 insertions(+), 33 deletions(-)
rename arch/sparc/{crypto => include/asm}/opcodes.h (96%)
diff --git a/arch/sparc/crypto/aes_asm.S b/arch/sparc/crypto/aes_asm.S
index 155cefb98520e..f291174a72a1d 100644
--- a/arch/sparc/crypto/aes_asm.S
+++ b/arch/sparc/crypto/aes_asm.S
@@ -1,11 +1,10 @@
/* SPDX-License-Identifier: GPL-2.0 */
#include <linux/linkage.h>
+#include <asm/opcodes.h>
#include <asm/visasm.h>
-#include "opcodes.h"
-
#define ENCRYPT_TWO_ROUNDS(KEY_BASE, I0, I1, T0, T1) \
AES_EROUND01(KEY_BASE + 0, I0, I1, T0) \
AES_EROUND23(KEY_BASE + 2, I0, I1, T1) \
AES_EROUND01(KEY_BASE + 4, T0, T1, I0) \
AES_EROUND23(KEY_BASE + 6, T0, T1, I1)
diff --git a/arch/sparc/crypto/aes_glue.c b/arch/sparc/crypto/aes_glue.c
index 6831508303562..359f22643b051 100644
--- a/arch/sparc/crypto/aes_glue.c
+++ b/arch/sparc/crypto/aes_glue.c
@@ -25,15 +25,14 @@
#include <crypto/algapi.h>
#include <crypto/aes.h>
#include <crypto/internal/skcipher.h>
#include <asm/fpumacro.h>
+#include <asm/opcodes.h>
#include <asm/pstate.h>
#include <asm/elf.h>
-#include "opcodes.h"
-
struct aes_ops {
void (*encrypt)(const u64 *key, const u32 *input, u32 *output);
void (*decrypt)(const u64 *key, const u32 *input, u32 *output);
void (*load_encrypt_keys)(const u64 *key);
void (*load_decrypt_keys)(const u64 *key);
diff --git a/arch/sparc/crypto/camellia_asm.S b/arch/sparc/crypto/camellia_asm.S
index dcdc9193fcd72..8471b346ef548 100644
--- a/arch/sparc/crypto/camellia_asm.S
+++ b/arch/sparc/crypto/camellia_asm.S
@@ -1,11 +1,10 @@
/* SPDX-License-Identifier: GPL-2.0 */
#include <linux/linkage.h>
+#include <asm/opcodes.h>
#include <asm/visasm.h>
-#include "opcodes.h"
-
#define CAMELLIA_6ROUNDS(KEY_BASE, I0, I1) \
CAMELLIA_F(KEY_BASE + 0, I1, I0, I1) \
CAMELLIA_F(KEY_BASE + 2, I0, I1, I0) \
CAMELLIA_F(KEY_BASE + 4, I1, I0, I1) \
CAMELLIA_F(KEY_BASE + 6, I0, I1, I0) \
diff --git a/arch/sparc/crypto/camellia_glue.c b/arch/sparc/crypto/camellia_glue.c
index aaa9714378e66..e7a1e1c42b996 100644
--- a/arch/sparc/crypto/camellia_glue.c
+++ b/arch/sparc/crypto/camellia_glue.c
@@ -13,15 +13,14 @@
#include <linux/types.h>
#include <crypto/algapi.h>
#include <crypto/internal/skcipher.h>
#include <asm/fpumacro.h>
+#include <asm/opcodes.h>
#include <asm/pstate.h>
#include <asm/elf.h>
-#include "opcodes.h"
-
#define CAMELLIA_MIN_KEY_SIZE 16
#define CAMELLIA_MAX_KEY_SIZE 32
#define CAMELLIA_BLOCK_SIZE 16
#define CAMELLIA_TABLE_BYTE_LEN 272
diff --git a/arch/sparc/crypto/des_asm.S b/arch/sparc/crypto/des_asm.S
index 7157468a679df..d534446cbef9a 100644
--- a/arch/sparc/crypto/des_asm.S
+++ b/arch/sparc/crypto/des_asm.S
@@ -1,11 +1,10 @@
/* SPDX-License-Identifier: GPL-2.0 */
#include <linux/linkage.h>
+#include <asm/opcodes.h>
#include <asm/visasm.h>
-#include "opcodes.h"
-
.align 32
ENTRY(des_sparc64_key_expand)
/* %o0=input_key, %o1=output_key */
VISEntryHalf
ld [%o0 + 0x00], %f0
diff --git a/arch/sparc/crypto/des_glue.c b/arch/sparc/crypto/des_glue.c
index a499102bf7065..e50ec4cd57cde 100644
--- a/arch/sparc/crypto/des_glue.c
+++ b/arch/sparc/crypto/des_glue.c
@@ -14,15 +14,14 @@
#include <crypto/algapi.h>
#include <crypto/internal/des.h>
#include <crypto/internal/skcipher.h>
#include <asm/fpumacro.h>
+#include <asm/opcodes.h>
#include <asm/pstate.h>
#include <asm/elf.h>
-#include "opcodes.h"
-
struct des_sparc64_ctx {
u64 encrypt_expkey[DES_EXPKEY_WORDS / 2];
u64 decrypt_expkey[DES_EXPKEY_WORDS / 2];
};
diff --git a/arch/sparc/crypto/md5_asm.S b/arch/sparc/crypto/md5_asm.S
index 7a6637455f37a..60b544e4d205b 100644
--- a/arch/sparc/crypto/md5_asm.S
+++ b/arch/sparc/crypto/md5_asm.S
@@ -1,11 +1,10 @@
/* SPDX-License-Identifier: GPL-2.0 */
#include <linux/linkage.h>
+#include <asm/opcodes.h>
#include <asm/visasm.h>
-#include "opcodes.h"
-
ENTRY(md5_sparc64_transform)
/* %o0 = digest, %o1 = data, %o2 = rounds */
VISEntryHalf
ld [%o0 + 0x00], %f0
ld [%o0 + 0x04], %f1
diff --git a/arch/sparc/crypto/md5_glue.c b/arch/sparc/crypto/md5_glue.c
index 5b018c6a376c4..b3615f0cdf626 100644
--- a/arch/sparc/crypto/md5_glue.c
+++ b/arch/sparc/crypto/md5_glue.c
@@ -13,21 +13,20 @@
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <asm/elf.h>
+#include <asm/opcodes.h>
#include <asm/pstate.h>
#include <crypto/internal/hash.h>
#include <crypto/md5.h>
#include <linux/errno.h>
#include <linux/kernel.h>
#include <linux/module.h>
#include <linux/string.h>
#include <linux/unaligned.h>
-#include "opcodes.h"
-
struct sparc_md5_state {
__le32 hash[MD5_HASH_WORDS];
u64 byte_count;
};
diff --git a/arch/sparc/crypto/sha1_asm.S b/arch/sparc/crypto/sha1_asm.S
index 7d8bf354f0e79..00b46bac1b08f 100644
--- a/arch/sparc/crypto/sha1_asm.S
+++ b/arch/sparc/crypto/sha1_asm.S
@@ -1,11 +1,10 @@
/* SPDX-License-Identifier: GPL-2.0 */
#include <linux/linkage.h>
+#include <asm/opcodes.h>
#include <asm/visasm.h>
-#include "opcodes.h"
-
ENTRY(sha1_sparc64_transform)
/* %o0 = digest, %o1 = data, %o2 = rounds */
VISEntryHalf
ld [%o0 + 0x00], %f0
ld [%o0 + 0x04], %f1
diff --git a/arch/sparc/crypto/sha1_glue.c b/arch/sparc/crypto/sha1_glue.c
index ec5a06948e0d4..ef19d5023b1bc 100644
--- a/arch/sparc/crypto/sha1_glue.c
+++ b/arch/sparc/crypto/sha1_glue.c
@@ -10,19 +10,18 @@
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <asm/elf.h>
+#include <asm/opcodes.h>
#include <asm/pstate.h>
#include <crypto/internal/hash.h>
#include <crypto/sha1.h>
#include <crypto/sha1_base.h>
#include <linux/kernel.h>
#include <linux/module.h>
-#include "opcodes.h"
-
asmlinkage void sha1_sparc64_transform(struct sha1_state *digest,
const u8 *data, int rounds);
static int sha1_sparc64_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
diff --git a/arch/sparc/crypto/sha256_asm.S b/arch/sparc/crypto/sha256_asm.S
index 0b39ec7d7ca29..8ce88611e98ad 100644
--- a/arch/sparc/crypto/sha256_asm.S
+++ b/arch/sparc/crypto/sha256_asm.S
@@ -1,11 +1,10 @@
/* SPDX-License-Identifier: GPL-2.0 */
#include <linux/linkage.h>
+#include <asm/opcodes.h>
#include <asm/visasm.h>
-#include "opcodes.h"
-
ENTRY(sha256_sparc64_transform)
/* %o0 = digest, %o1 = data, %o2 = rounds */
VISEntryHalf
ld [%o0 + 0x00], %f0
ld [%o0 + 0x04], %f1
diff --git a/arch/sparc/crypto/sha256_glue.c b/arch/sparc/crypto/sha256_glue.c
index ddb250242faf4..25008603a9868 100644
--- a/arch/sparc/crypto/sha256_glue.c
+++ b/arch/sparc/crypto/sha256_glue.c
@@ -10,19 +10,18 @@
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <asm/elf.h>
+#include <asm/opcodes.h>
#include <asm/pstate.h>
#include <crypto/internal/hash.h>
#include <crypto/sha2.h>
#include <crypto/sha256_base.h>
#include <linux/kernel.h>
#include <linux/module.h>
-#include "opcodes.h"
-
asmlinkage void sha256_sparc64_transform(u32 *digest, const char *data,
unsigned int rounds);
static void sha256_block(struct crypto_sha256_state *sctx, const u8 *src,
int blocks)
diff --git a/arch/sparc/crypto/sha512_asm.S b/arch/sparc/crypto/sha512_asm.S
index b2f6e67288023..9932b4fe1b599 100644
--- a/arch/sparc/crypto/sha512_asm.S
+++ b/arch/sparc/crypto/sha512_asm.S
@@ -1,11 +1,10 @@
/* SPDX-License-Identifier: GPL-2.0 */
#include <linux/linkage.h>
+#include <asm/opcodes.h>
#include <asm/visasm.h>
-#include "opcodes.h"
-
ENTRY(sha512_sparc64_transform)
/* %o0 = digest, %o1 = data, %o2 = rounds */
VISEntry
ldd [%o0 + 0x00], %f0
ldd [%o0 + 0x08], %f2
diff --git a/arch/sparc/crypto/sha512_glue.c b/arch/sparc/crypto/sha512_glue.c
index 1d0e1f98ca461..47b9277b6877a 100644
--- a/arch/sparc/crypto/sha512_glue.c
+++ b/arch/sparc/crypto/sha512_glue.c
@@ -9,19 +9,18 @@
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
#include <asm/elf.h>
+#include <asm/opcodes.h>
#include <asm/pstate.h>
#include <crypto/internal/hash.h>
#include <crypto/sha2.h>
#include <crypto/sha512_base.h>
#include <linux/kernel.h>
#include <linux/module.h>
-#include "opcodes.h"
-
asmlinkage void sha512_sparc64_transform(u64 *digest, const char *data,
unsigned int rounds);
static void sha512_block(struct sha512_state *sctx, const u8 *src, int blocks)
{
diff --git a/arch/sparc/crypto/opcodes.h b/arch/sparc/include/asm/opcodes.h
similarity index 96%
rename from arch/sparc/crypto/opcodes.h
rename to arch/sparc/include/asm/opcodes.h
index 417b6a10a337a..ebfda6eb49b26 100644
--- a/arch/sparc/crypto/opcodes.h
+++ b/arch/sparc/include/asm/opcodes.h
@@ -1,8 +1,8 @@
/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _OPCODES_H
-#define _OPCODES_H
+#ifndef _SPARC_ASM_OPCODES_H
+#define _SPARC_ASM_OPCODES_H
#define SPARC_CR_OPCODE_PRIORITY 300
#define F3F(x,y,z) (((x)<<30)|((y)<<19)|((z)<<5))
@@ -95,6 +95,6 @@
#define MOVXTOD_G3_F60 \
.word 0xbbb02303;
#define MOVXTOD_G7_F62 \
.word 0xbfb02307;
-#endif /* _OPCODES_H */
+#endif /* _SPARC_ASM_OPCODES_H */
diff --git a/arch/sparc/lib/crc32c_asm.S b/arch/sparc/lib/crc32c_asm.S
index ee454fa6aed68..4db873850f44c 100644
--- a/arch/sparc/lib/crc32c_asm.S
+++ b/arch/sparc/lib/crc32c_asm.S
@@ -1,12 +1,11 @@
/* SPDX-License-Identifier: GPL-2.0 */
#include <linux/linkage.h>
+#include <asm/opcodes.h>
#include <asm/visasm.h>
#include <asm/asi.h>
-#include "../crypto/opcodes.h"
-
ENTRY(crc32c_sparc64)
/* %o0=crc32p, %o1=data_ptr, %o2=len */
VISEntryHalf
lda [%o0] ASI_PL, %f1
1: ldd [%o1], %f2
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 10/13] crypto: sparc/sha256 - implement library instead of shash
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
` (8 preceding siblings ...)
2025-04-28 17:00 ` [PATCH v4 09/13] crypto: sparc - move opcodes.h into asm directory Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-04-28 17:00 ` [PATCH v4 11/13] crypto: x86/sha256 " Eric Biggers
` (3 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Instead of providing crypto_shash algorithms for the arch-optimized
SHA-256 code, instead implement the SHA-256 library. This is much
simpler, it makes the SHA-256 library functions be arch-optimized, and
it fixes the longstanding issue where the arch-optimized SHA-256 was
disabled by default. SHA-256 still remains available through
crypto_shash, but individual architectures no longer need to handle it.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/sparc/crypto/Kconfig | 10 --
arch/sparc/crypto/Makefile | 2 -
arch/sparc/crypto/sha256_glue.c | 128 -----------------------
arch/sparc/lib/Makefile | 1 +
arch/sparc/lib/crypto/Kconfig | 8 ++
arch/sparc/lib/crypto/Makefile | 4 +
arch/sparc/lib/crypto/sha256.c | 64 ++++++++++++
arch/sparc/{ => lib}/crypto/sha256_asm.S | 2 +-
lib/crypto/Kconfig | 3 +
9 files changed, 81 insertions(+), 141 deletions(-)
delete mode 100644 arch/sparc/crypto/sha256_glue.c
create mode 100644 arch/sparc/lib/crypto/Kconfig
create mode 100644 arch/sparc/lib/crypto/Makefile
create mode 100644 arch/sparc/lib/crypto/sha256.c
rename arch/sparc/{ => lib}/crypto/sha256_asm.S (96%)
diff --git a/arch/sparc/crypto/Kconfig b/arch/sparc/crypto/Kconfig
index e858597de89db..a6ba319c42dce 100644
--- a/arch/sparc/crypto/Kconfig
+++ b/arch/sparc/crypto/Kconfig
@@ -34,20 +34,10 @@ config CRYPTO_SHA1_SPARC64
help
SHA-1 secure hash algorithm (FIPS 180)
Architecture: sparc64
-config CRYPTO_SHA256_SPARC64
- tristate "Hash functions: SHA-224 and SHA-256"
- depends on SPARC64
- select CRYPTO_SHA256
- select CRYPTO_HASH
- help
- SHA-224 and SHA-256 secure hash algorithms (FIPS 180)
-
- Architecture: sparc64 using crypto instructions, when available
-
config CRYPTO_SHA512_SPARC64
tristate "Hash functions: SHA-384 and SHA-512"
depends on SPARC64
select CRYPTO_SHA512
select CRYPTO_HASH
diff --git a/arch/sparc/crypto/Makefile b/arch/sparc/crypto/Makefile
index a2d7fca40cb4b..701c39edb0d73 100644
--- a/arch/sparc/crypto/Makefile
+++ b/arch/sparc/crypto/Makefile
@@ -2,20 +2,18 @@
#
# Arch-specific CryptoAPI modules.
#
obj-$(CONFIG_CRYPTO_SHA1_SPARC64) += sha1-sparc64.o
-obj-$(CONFIG_CRYPTO_SHA256_SPARC64) += sha256-sparc64.o
obj-$(CONFIG_CRYPTO_SHA512_SPARC64) += sha512-sparc64.o
obj-$(CONFIG_CRYPTO_MD5_SPARC64) += md5-sparc64.o
obj-$(CONFIG_CRYPTO_AES_SPARC64) += aes-sparc64.o
obj-$(CONFIG_CRYPTO_DES_SPARC64) += des-sparc64.o
obj-$(CONFIG_CRYPTO_CAMELLIA_SPARC64) += camellia-sparc64.o
sha1-sparc64-y := sha1_asm.o sha1_glue.o
-sha256-sparc64-y := sha256_asm.o sha256_glue.o
sha512-sparc64-y := sha512_asm.o sha512_glue.o
md5-sparc64-y := md5_asm.o md5_glue.o
aes-sparc64-y := aes_asm.o aes_glue.o
des-sparc64-y := des_asm.o des_glue.o
diff --git a/arch/sparc/crypto/sha256_glue.c b/arch/sparc/crypto/sha256_glue.c
deleted file mode 100644
index 25008603a9868..0000000000000
--- a/arch/sparc/crypto/sha256_glue.c
+++ /dev/null
@@ -1,128 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/* Glue code for SHA256 hashing optimized for sparc64 crypto opcodes.
- *
- * This is based largely upon crypto/sha256_generic.c
- *
- * Copyright (c) Jean-Luc Cooke <jlcooke@certainkey.com>
- * Copyright (c) Andrew McDonald <andrew@mcdonald.org.uk>
- * Copyright (c) 2002 James Morris <jmorris@intercode.com.au>
- * SHA224 Support Copyright 2007 Intel Corporation <jonathan.lynch@intel.com>
- */
-
-#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-
-#include <asm/elf.h>
-#include <asm/opcodes.h>
-#include <asm/pstate.h>
-#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
-#include <crypto/sha256_base.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-
-asmlinkage void sha256_sparc64_transform(u32 *digest, const char *data,
- unsigned int rounds);
-
-static void sha256_block(struct crypto_sha256_state *sctx, const u8 *src,
- int blocks)
-{
- sha256_sparc64_transform(sctx->state, src, blocks);
-}
-
-static int sha256_sparc64_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- return sha256_base_do_update_blocks(desc, data, len, sha256_block);
-}
-
-static int sha256_sparc64_finup(struct shash_desc *desc, const u8 *src,
- unsigned int len, u8 *out)
-{
- sha256_base_do_finup(desc, src, len, sha256_block);
- return sha256_base_finish(desc, out);
-}
-
-static struct shash_alg sha256_alg = {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = sha256_sparc64_update,
- .finup = sha256_sparc64_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha256",
- .cra_driver_name= "sha256-sparc64",
- .cra_priority = SPARC_CR_OPCODE_PRIORITY,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-};
-
-static struct shash_alg sha224_alg = {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = sha256_sparc64_update,
- .finup = sha256_sparc64_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha224",
- .cra_driver_name= "sha224-sparc64",
- .cra_priority = SPARC_CR_OPCODE_PRIORITY,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-};
-
-static bool __init sparc64_has_sha256_opcode(void)
-{
- unsigned long cfr;
-
- if (!(sparc64_elf_hwcap & HWCAP_SPARC_CRYPTO))
- return false;
-
- __asm__ __volatile__("rd %%asr26, %0" : "=r" (cfr));
- if (!(cfr & CFR_SHA256))
- return false;
-
- return true;
-}
-
-static int __init sha256_sparc64_mod_init(void)
-{
- if (sparc64_has_sha256_opcode()) {
- int ret = crypto_register_shash(&sha224_alg);
- if (ret < 0)
- return ret;
-
- ret = crypto_register_shash(&sha256_alg);
- if (ret < 0) {
- crypto_unregister_shash(&sha224_alg);
- return ret;
- }
-
- pr_info("Using sparc64 sha256 opcode optimized SHA-256/SHA-224 implementation\n");
- return 0;
- }
- pr_info("sparc64 sha256 opcode not available.\n");
- return -ENODEV;
-}
-
-static void __exit sha256_sparc64_mod_fini(void)
-{
- crypto_unregister_shash(&sha224_alg);
- crypto_unregister_shash(&sha256_alg);
-}
-
-module_init(sha256_sparc64_mod_init);
-module_exit(sha256_sparc64_mod_fini);
-
-MODULE_LICENSE("GPL");
-MODULE_DESCRIPTION("SHA-224 and SHA-256 Secure Hash Algorithm, sparc64 sha256 opcode accelerated");
-
-MODULE_ALIAS_CRYPTO("sha224");
-MODULE_ALIAS_CRYPTO("sha256");
-
-#include "crop_devid.c"
diff --git a/arch/sparc/lib/Makefile b/arch/sparc/lib/Makefile
index 5724d0f356eb5..98887dc295a1e 100644
--- a/arch/sparc/lib/Makefile
+++ b/arch/sparc/lib/Makefile
@@ -2,10 +2,11 @@
# Makefile for Sparc library files..
#
asflags-y := -ansi -DST_DIV0=0x02
+obj-y += crypto/
lib-$(CONFIG_SPARC32) += ashrdi3.o
lib-$(CONFIG_SPARC32) += memcpy.o memset.o
lib-y += strlen.o
lib-y += checksum_$(BITS).o
lib-$(CONFIG_SPARC32) += blockops.o
diff --git a/arch/sparc/lib/crypto/Kconfig b/arch/sparc/lib/crypto/Kconfig
new file mode 100644
index 0000000000000..e5c3e4d3dba62
--- /dev/null
+++ b/arch/sparc/lib/crypto/Kconfig
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+config CRYPTO_SHA256_SPARC64
+ tristate
+ depends on SPARC64
+ default CRYPTO_LIB_SHA256
+ select CRYPTO_ARCH_HAVE_LIB_SHA256
+ select CRYPTO_LIB_SHA256_GENERIC
diff --git a/arch/sparc/lib/crypto/Makefile b/arch/sparc/lib/crypto/Makefile
new file mode 100644
index 0000000000000..75ee244ad6f79
--- /dev/null
+++ b/arch/sparc/lib/crypto/Makefile
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+obj-$(CONFIG_CRYPTO_SHA256_SPARC64) += sha256-sparc64.o
+sha256-sparc64-y := sha256.o sha256_asm.o
diff --git a/arch/sparc/lib/crypto/sha256.c b/arch/sparc/lib/crypto/sha256.c
new file mode 100644
index 0000000000000..6f118a23d210a
--- /dev/null
+++ b/arch/sparc/lib/crypto/sha256.c
@@ -0,0 +1,64 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * SHA-256 accelerated using the sparc64 sha256 opcodes
+ *
+ * Copyright (c) Jean-Luc Cooke <jlcooke@certainkey.com>
+ * Copyright (c) Andrew McDonald <andrew@mcdonald.org.uk>
+ * Copyright (c) 2002 James Morris <jmorris@intercode.com.au>
+ * SHA224 Support Copyright 2007 Intel Corporation <jonathan.lynch@intel.com>
+ */
+
+#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
+
+#include <asm/elf.h>
+#include <asm/opcodes.h>
+#include <asm/pstate.h>
+#include <crypto/internal/sha2.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_sha256_opcodes);
+
+asmlinkage void sha256_sparc64_transform(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+
+void sha256_blocks_arch(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks)
+{
+ if (static_branch_likely(&have_sha256_opcodes))
+ sha256_sparc64_transform(state, data, nblocks);
+ else
+ sha256_blocks_generic(state, data, nblocks);
+}
+EXPORT_SYMBOL(sha256_blocks_arch);
+
+bool sha256_is_arch_optimized(void)
+{
+ return static_key_enabled(&have_sha256_opcodes);
+}
+EXPORT_SYMBOL(sha256_is_arch_optimized);
+
+static int __init sha256_sparc64_mod_init(void)
+{
+ unsigned long cfr;
+
+ if (!(sparc64_elf_hwcap & HWCAP_SPARC_CRYPTO))
+ return 0;
+
+ __asm__ __volatile__("rd %%asr26, %0" : "=r" (cfr));
+ if (!(cfr & CFR_SHA256))
+ return 0;
+
+ static_branch_enable(&have_sha256_opcodes);
+ pr_info("Using sparc64 sha256 opcode optimized SHA-256/SHA-224 implementation\n");
+ return 0;
+}
+arch_initcall(sha256_sparc64_mod_init);
+
+static void __exit sha256_sparc64_mod_exit(void)
+{
+}
+module_exit(sha256_sparc64_mod_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("SHA-256 accelerated using the sparc64 sha256 opcodes");
diff --git a/arch/sparc/crypto/sha256_asm.S b/arch/sparc/lib/crypto/sha256_asm.S
similarity index 96%
rename from arch/sparc/crypto/sha256_asm.S
rename to arch/sparc/lib/crypto/sha256_asm.S
index 8ce88611e98ad..ddcdd3daf31e3 100644
--- a/arch/sparc/crypto/sha256_asm.S
+++ b/arch/sparc/lib/crypto/sha256_asm.S
@@ -2,11 +2,11 @@
#include <linux/linkage.h>
#include <asm/opcodes.h>
#include <asm/visasm.h>
ENTRY(sha256_sparc64_transform)
- /* %o0 = digest, %o1 = data, %o2 = rounds */
+ /* %o0 = state, %o1 = data, %o2 = nblocks */
VISEntryHalf
ld [%o0 + 0x00], %f0
ld [%o0 + 0x04], %f1
ld [%o0 + 0x08], %f2
ld [%o0 + 0x0c], %f3
diff --git a/lib/crypto/Kconfig b/lib/crypto/Kconfig
index 7fe678047939b..6319358b38c20 100644
--- a/lib/crypto/Kconfig
+++ b/lib/crypto/Kconfig
@@ -179,10 +179,13 @@ if RISCV
source "arch/riscv/lib/crypto/Kconfig"
endif
if S390
source "arch/s390/lib/crypto/Kconfig"
endif
+if SPARC
+source "arch/sparc/lib/crypto/Kconfig"
+endif
if X86
source "arch/x86/lib/crypto/Kconfig"
endif
endif
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 11/13] crypto: x86/sha256 - implement library instead of shash
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
` (9 preceding siblings ...)
2025-04-28 17:00 ` [PATCH v4 10/13] crypto: sparc/sha256 - implement library instead of shash Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-04-28 17:00 ` [PATCH v4 12/13] crypto: sha256 - remove sha256_base.h Eric Biggers
` (2 subsequent siblings)
13 siblings, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Instead of providing crypto_shash algorithms for the arch-optimized
SHA-256 code, instead implement the SHA-256 library. This is much
simpler, it makes the SHA-256 library functions be arch-optimized, and
it fixes the longstanding issue where the arch-optimized SHA-256 was
disabled by default. SHA-256 still remains available through
crypto_shash, but individual architectures no longer need to handle it.
To match sha256_blocks_arch(), change the type of the nblocks parameter
of the assembly functions from int to size_t. The assembly functions
actually already treated it as size_t.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/x86/crypto/Kconfig | 14 -
arch/x86/crypto/Makefile | 3 -
arch/x86/crypto/sha256_ssse3_glue.c | 432 ------------------
arch/x86/lib/crypto/Kconfig | 7 +
arch/x86/lib/crypto/Makefile | 3 +
arch/x86/{ => lib}/crypto/sha256-avx-asm.S | 12 +-
arch/x86/{ => lib}/crypto/sha256-avx2-asm.S | 12 +-
.../crypto/sha256-ni-asm.S} | 36 +-
arch/x86/{ => lib}/crypto/sha256-ssse3-asm.S | 14 +-
arch/x86/lib/crypto/sha256.c | 74 +++
10 files changed, 118 insertions(+), 489 deletions(-)
delete mode 100644 arch/x86/crypto/sha256_ssse3_glue.c
rename arch/x86/{ => lib}/crypto/sha256-avx-asm.S (98%)
rename arch/x86/{ => lib}/crypto/sha256-avx2-asm.S (98%)
rename arch/x86/{crypto/sha256_ni_asm.S => lib/crypto/sha256-ni-asm.S} (85%)
rename arch/x86/{ => lib}/crypto/sha256-ssse3-asm.S (98%)
create mode 100644 arch/x86/lib/crypto/sha256.c
diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig
index 9e941362e4cd5..56cfdc79e2c66 100644
--- a/arch/x86/crypto/Kconfig
+++ b/arch/x86/crypto/Kconfig
@@ -388,24 +388,10 @@ config CRYPTO_SHA1_SSSE3
- SSSE3 (Supplemental SSE3)
- AVX (Advanced Vector Extensions)
- AVX2 (Advanced Vector Extensions 2)
- SHA-NI (SHA Extensions New Instructions)
-config CRYPTO_SHA256_SSSE3
- tristate "Hash functions: SHA-224 and SHA-256 (SSSE3/AVX/AVX2/SHA-NI)"
- depends on 64BIT
- select CRYPTO_SHA256
- select CRYPTO_HASH
- help
- SHA-224 and SHA-256 secure hash algorithms (FIPS 180)
-
- Architecture: x86_64 using:
- - SSSE3 (Supplemental SSE3)
- - AVX (Advanced Vector Extensions)
- - AVX2 (Advanced Vector Extensions 2)
- - SHA-NI (SHA Extensions New Instructions)
-
config CRYPTO_SHA512_SSSE3
tristate "Hash functions: SHA-384 and SHA-512 (SSSE3/AVX/AVX2)"
depends on 64BIT
select CRYPTO_SHA512
select CRYPTO_HASH
diff --git a/arch/x86/crypto/Makefile b/arch/x86/crypto/Makefile
index fad59a6c6c26f..aa289a9e0153b 100644
--- a/arch/x86/crypto/Makefile
+++ b/arch/x86/crypto/Makefile
@@ -52,13 +52,10 @@ aesni-intel-$(CONFIG_64BIT) += aes-gcm-avx10-x86_64.o
endif
obj-$(CONFIG_CRYPTO_SHA1_SSSE3) += sha1-ssse3.o
sha1-ssse3-y := sha1_avx2_x86_64_asm.o sha1_ssse3_asm.o sha1_ni_asm.o sha1_ssse3_glue.o
-obj-$(CONFIG_CRYPTO_SHA256_SSSE3) += sha256-ssse3.o
-sha256-ssse3-y := sha256-ssse3-asm.o sha256-avx-asm.o sha256-avx2-asm.o sha256_ni_asm.o sha256_ssse3_glue.o
-
obj-$(CONFIG_CRYPTO_SHA512_SSSE3) += sha512-ssse3.o
sha512-ssse3-y := sha512-ssse3-asm.o sha512-avx-asm.o sha512-avx2-asm.o sha512_ssse3_glue.o
obj-$(CONFIG_CRYPTO_GHASH_CLMUL_NI_INTEL) += ghash-clmulni-intel.o
ghash-clmulni-intel-y := ghash-clmulni-intel_asm.o ghash-clmulni-intel_glue.o
diff --git a/arch/x86/crypto/sha256_ssse3_glue.c b/arch/x86/crypto/sha256_ssse3_glue.c
deleted file mode 100644
index a5d3be00550b8..0000000000000
--- a/arch/x86/crypto/sha256_ssse3_glue.c
+++ /dev/null
@@ -1,432 +0,0 @@
-/*
- * Cryptographic API.
- *
- * Glue code for the SHA256 Secure Hash Algorithm assembler implementations
- * using SSSE3, AVX, AVX2, and SHA-NI instructions.
- *
- * This file is based on sha256_generic.c
- *
- * Copyright (C) 2013 Intel Corporation.
- *
- * Author:
- * Tim Chen <tim.c.chen@linux.intel.com>
- *
- * This program is free software; you can redistribute it and/or modify it
- * under the terms of the GNU General Public License as published by the Free
- * Software Foundation; either version 2 of the License, or (at your option)
- * any later version.
- *
- * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
- * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
- * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
- * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS
- * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN
- * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN
- * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
- * SOFTWARE.
- */
-
-
-#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-
-#include <asm/cpu_device_id.h>
-#include <asm/fpu/api.h>
-#include <crypto/internal/hash.h>
-#include <crypto/sha2.h>
-#include <crypto/sha256_base.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-
-asmlinkage void sha256_transform_ssse3(struct crypto_sha256_state *state,
- const u8 *data, int blocks);
-
-static const struct x86_cpu_id module_cpu_ids[] = {
- X86_MATCH_FEATURE(X86_FEATURE_SHA_NI, NULL),
- X86_MATCH_FEATURE(X86_FEATURE_AVX2, NULL),
- X86_MATCH_FEATURE(X86_FEATURE_AVX, NULL),
- X86_MATCH_FEATURE(X86_FEATURE_SSSE3, NULL),
- {}
-};
-MODULE_DEVICE_TABLE(x86cpu, module_cpu_ids);
-
-static int _sha256_update(struct shash_desc *desc, const u8 *data,
- unsigned int len,
- sha256_block_fn *sha256_xform)
-{
- int remain;
-
- /*
- * Make sure struct crypto_sha256_state begins directly with the SHA256
- * 256-bit internal state, as this is what the asm functions expect.
- */
- BUILD_BUG_ON(offsetof(struct crypto_sha256_state, state) != 0);
-
- kernel_fpu_begin();
- remain = sha256_base_do_update_blocks(desc, data, len, sha256_xform);
- kernel_fpu_end();
-
- return remain;
-}
-
-static int sha256_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out, sha256_block_fn *sha256_xform)
-{
- kernel_fpu_begin();
- sha256_base_do_finup(desc, data, len, sha256_xform);
- kernel_fpu_end();
-
- return sha256_base_finish(desc, out);
-}
-
-static int sha256_ssse3_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- return _sha256_update(desc, data, len, sha256_transform_ssse3);
-}
-
-static int sha256_ssse3_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- return sha256_finup(desc, data, len, out, sha256_transform_ssse3);
-}
-
-static int sha256_ssse3_digest(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- return sha256_base_init(desc) ?:
- sha256_ssse3_finup(desc, data, len, out);
-}
-
-static struct shash_alg sha256_ssse3_algs[] = { {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = sha256_ssse3_update,
- .finup = sha256_ssse3_finup,
- .digest = sha256_ssse3_digest,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha256",
- .cra_driver_name = "sha256-ssse3",
- .cra_priority = 150,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-}, {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = sha256_ssse3_update,
- .finup = sha256_ssse3_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha224",
- .cra_driver_name = "sha224-ssse3",
- .cra_priority = 150,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-} };
-
-static int register_sha256_ssse3(void)
-{
- if (boot_cpu_has(X86_FEATURE_SSSE3))
- return crypto_register_shashes(sha256_ssse3_algs,
- ARRAY_SIZE(sha256_ssse3_algs));
- return 0;
-}
-
-static void unregister_sha256_ssse3(void)
-{
- if (boot_cpu_has(X86_FEATURE_SSSE3))
- crypto_unregister_shashes(sha256_ssse3_algs,
- ARRAY_SIZE(sha256_ssse3_algs));
-}
-
-asmlinkage void sha256_transform_avx(struct crypto_sha256_state *state,
- const u8 *data, int blocks);
-
-static int sha256_avx_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- return _sha256_update(desc, data, len, sha256_transform_avx);
-}
-
-static int sha256_avx_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- return sha256_finup(desc, data, len, out, sha256_transform_avx);
-}
-
-static int sha256_avx_digest(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- return sha256_base_init(desc) ?:
- sha256_avx_finup(desc, data, len, out);
-}
-
-static struct shash_alg sha256_avx_algs[] = { {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = sha256_avx_update,
- .finup = sha256_avx_finup,
- .digest = sha256_avx_digest,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha256",
- .cra_driver_name = "sha256-avx",
- .cra_priority = 160,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-}, {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = sha256_avx_update,
- .finup = sha256_avx_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha224",
- .cra_driver_name = "sha224-avx",
- .cra_priority = 160,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-} };
-
-static bool avx_usable(void)
-{
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL)) {
- if (boot_cpu_has(X86_FEATURE_AVX))
- pr_info("AVX detected but unusable.\n");
- return false;
- }
-
- return true;
-}
-
-static int register_sha256_avx(void)
-{
- if (avx_usable())
- return crypto_register_shashes(sha256_avx_algs,
- ARRAY_SIZE(sha256_avx_algs));
- return 0;
-}
-
-static void unregister_sha256_avx(void)
-{
- if (avx_usable())
- crypto_unregister_shashes(sha256_avx_algs,
- ARRAY_SIZE(sha256_avx_algs));
-}
-
-asmlinkage void sha256_transform_rorx(struct crypto_sha256_state *state,
- const u8 *data, int blocks);
-
-static int sha256_avx2_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- return _sha256_update(desc, data, len, sha256_transform_rorx);
-}
-
-static int sha256_avx2_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- return sha256_finup(desc, data, len, out, sha256_transform_rorx);
-}
-
-static int sha256_avx2_digest(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- return sha256_base_init(desc) ?:
- sha256_avx2_finup(desc, data, len, out);
-}
-
-static struct shash_alg sha256_avx2_algs[] = { {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = sha256_avx2_update,
- .finup = sha256_avx2_finup,
- .digest = sha256_avx2_digest,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha256",
- .cra_driver_name = "sha256-avx2",
- .cra_priority = 170,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-}, {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = sha256_avx2_update,
- .finup = sha256_avx2_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha224",
- .cra_driver_name = "sha224-avx2",
- .cra_priority = 170,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-} };
-
-static bool avx2_usable(void)
-{
- if (avx_usable() && boot_cpu_has(X86_FEATURE_AVX2) &&
- boot_cpu_has(X86_FEATURE_BMI2))
- return true;
-
- return false;
-}
-
-static int register_sha256_avx2(void)
-{
- if (avx2_usable())
- return crypto_register_shashes(sha256_avx2_algs,
- ARRAY_SIZE(sha256_avx2_algs));
- return 0;
-}
-
-static void unregister_sha256_avx2(void)
-{
- if (avx2_usable())
- crypto_unregister_shashes(sha256_avx2_algs,
- ARRAY_SIZE(sha256_avx2_algs));
-}
-
-asmlinkage void sha256_ni_transform(struct crypto_sha256_state *digest,
- const u8 *data, int rounds);
-
-static int sha256_ni_update(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- return _sha256_update(desc, data, len, sha256_ni_transform);
-}
-
-static int sha256_ni_finup(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- return sha256_finup(desc, data, len, out, sha256_ni_transform);
-}
-
-static int sha256_ni_digest(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- return sha256_base_init(desc) ?:
- sha256_ni_finup(desc, data, len, out);
-}
-
-static struct shash_alg sha256_ni_algs[] = { {
- .digestsize = SHA256_DIGEST_SIZE,
- .init = sha256_base_init,
- .update = sha256_ni_update,
- .finup = sha256_ni_finup,
- .digest = sha256_ni_digest,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha256",
- .cra_driver_name = "sha256-ni",
- .cra_priority = 250,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA256_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-}, {
- .digestsize = SHA224_DIGEST_SIZE,
- .init = sha224_base_init,
- .update = sha256_ni_update,
- .finup = sha256_ni_finup,
- .descsize = sizeof(struct crypto_sha256_state),
- .base = {
- .cra_name = "sha224",
- .cra_driver_name = "sha224-ni",
- .cra_priority = 250,
- .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
- CRYPTO_AHASH_ALG_FINUP_MAX,
- .cra_blocksize = SHA224_BLOCK_SIZE,
- .cra_module = THIS_MODULE,
- }
-} };
-
-static int register_sha256_ni(void)
-{
- if (boot_cpu_has(X86_FEATURE_SHA_NI))
- return crypto_register_shashes(sha256_ni_algs,
- ARRAY_SIZE(sha256_ni_algs));
- return 0;
-}
-
-static void unregister_sha256_ni(void)
-{
- if (boot_cpu_has(X86_FEATURE_SHA_NI))
- crypto_unregister_shashes(sha256_ni_algs,
- ARRAY_SIZE(sha256_ni_algs));
-}
-
-static int __init sha256_ssse3_mod_init(void)
-{
- if (!x86_match_cpu(module_cpu_ids))
- return -ENODEV;
-
- if (register_sha256_ssse3())
- goto fail;
-
- if (register_sha256_avx()) {
- unregister_sha256_ssse3();
- goto fail;
- }
-
- if (register_sha256_avx2()) {
- unregister_sha256_avx();
- unregister_sha256_ssse3();
- goto fail;
- }
-
- if (register_sha256_ni()) {
- unregister_sha256_avx2();
- unregister_sha256_avx();
- unregister_sha256_ssse3();
- goto fail;
- }
-
- return 0;
-fail:
- return -ENODEV;
-}
-
-static void __exit sha256_ssse3_mod_fini(void)
-{
- unregister_sha256_ni();
- unregister_sha256_avx2();
- unregister_sha256_avx();
- unregister_sha256_ssse3();
-}
-
-module_init(sha256_ssse3_mod_init);
-module_exit(sha256_ssse3_mod_fini);
-
-MODULE_LICENSE("GPL");
-MODULE_DESCRIPTION("SHA256 Secure Hash Algorithm, Supplemental SSE3 accelerated");
-
-MODULE_ALIAS_CRYPTO("sha256");
-MODULE_ALIAS_CRYPTO("sha256-ssse3");
-MODULE_ALIAS_CRYPTO("sha256-avx");
-MODULE_ALIAS_CRYPTO("sha256-avx2");
-MODULE_ALIAS_CRYPTO("sha224");
-MODULE_ALIAS_CRYPTO("sha224-ssse3");
-MODULE_ALIAS_CRYPTO("sha224-avx");
-MODULE_ALIAS_CRYPTO("sha224-avx2");
-MODULE_ALIAS_CRYPTO("sha256-ni");
-MODULE_ALIAS_CRYPTO("sha224-ni");
diff --git a/arch/x86/lib/crypto/Kconfig b/arch/x86/lib/crypto/Kconfig
index 546fe2afe0b51..e344579db3d85 100644
--- a/arch/x86/lib/crypto/Kconfig
+++ b/arch/x86/lib/crypto/Kconfig
@@ -22,5 +22,12 @@ config CRYPTO_CHACHA20_X86_64
config CRYPTO_POLY1305_X86_64
tristate
depends on 64BIT
default CRYPTO_LIB_POLY1305
select CRYPTO_ARCH_HAVE_LIB_POLY1305
+
+config CRYPTO_SHA256_X86_64
+ tristate
+ depends on 64BIT
+ default CRYPTO_LIB_SHA256
+ select CRYPTO_ARCH_HAVE_LIB_SHA256
+ select CRYPTO_LIB_SHA256_GENERIC
diff --git a/arch/x86/lib/crypto/Makefile b/arch/x86/lib/crypto/Makefile
index c2ff8c5f1046e..abceca3d31c01 100644
--- a/arch/x86/lib/crypto/Makefile
+++ b/arch/x86/lib/crypto/Makefile
@@ -8,10 +8,13 @@ chacha-x86_64-y := chacha-avx2-x86_64.o chacha-ssse3-x86_64.o chacha-avx512vl-x8
obj-$(CONFIG_CRYPTO_POLY1305_X86_64) += poly1305-x86_64.o
poly1305-x86_64-y := poly1305-x86_64-cryptogams.o poly1305_glue.o
targets += poly1305-x86_64-cryptogams.S
+obj-$(CONFIG_CRYPTO_SHA256_X86_64) += sha256-x86_64.o
+sha256-x86_64-y := sha256.o sha256-ssse3-asm.o sha256-avx-asm.o sha256-avx2-asm.o sha256-ni-asm.o
+
quiet_cmd_perlasm = PERLASM $@
cmd_perlasm = $(PERL) $< > $@
$(obj)/%.S: $(src)/%.pl FORCE
$(call if_changed,perlasm)
diff --git a/arch/x86/crypto/sha256-avx-asm.S b/arch/x86/lib/crypto/sha256-avx-asm.S
similarity index 98%
rename from arch/x86/crypto/sha256-avx-asm.S
rename to arch/x86/lib/crypto/sha256-avx-asm.S
index 53de72bdd851e..0d7b2c3e45d9a 100644
--- a/arch/x86/crypto/sha256-avx-asm.S
+++ b/arch/x86/lib/crypto/sha256-avx-asm.S
@@ -46,11 +46,11 @@
########################################################################
# This code schedules 1 block at a time, with 4 lanes per block
########################################################################
#include <linux/linkage.h>
-#include <linux/cfi_types.h>
+#include <linux/objtool.h>
## assume buffers not aligned
#define VMOVDQ vmovdqu
################################ Define Macros
@@ -339,17 +339,17 @@ a = TMP_
add y0, h # h = h + S1 + CH + k + w + S0 + MAJ
ROTATE_ARGS
.endm
########################################################################
-## void sha256_transform_avx(state sha256_state *state, const u8 *data, int blocks)
-## arg 1 : pointer to state
-## arg 2 : pointer to input data
-## arg 3 : Num blocks
+## void sha256_transform_avx(u32 state[SHA256_STATE_WORDS],
+## const u8 *data, size_t nblocks);
########################################################################
.text
-SYM_TYPED_FUNC_START(sha256_transform_avx)
+SYM_FUNC_START(sha256_transform_avx)
+ ANNOTATE_NOENDBR # since this is called only via static_call
+
pushq %rbx
pushq %r12
pushq %r13
pushq %r14
pushq %r15
diff --git a/arch/x86/crypto/sha256-avx2-asm.S b/arch/x86/lib/crypto/sha256-avx2-asm.S
similarity index 98%
rename from arch/x86/crypto/sha256-avx2-asm.S
rename to arch/x86/lib/crypto/sha256-avx2-asm.S
index 0bbec1c75cd0b..25d3380321ec3 100644
--- a/arch/x86/crypto/sha256-avx2-asm.S
+++ b/arch/x86/lib/crypto/sha256-avx2-asm.S
@@ -47,11 +47,11 @@
########################################################################
# This code schedules 2 blocks at a time, with 4 lanes per block
########################################################################
#include <linux/linkage.h>
-#include <linux/cfi_types.h>
+#include <linux/objtool.h>
## assume buffers not aligned
#define VMOVDQ vmovdqu
################################ Define Macros
@@ -516,17 +516,17 @@ STACK_SIZE = _CTX + _CTX_SIZE
ROTATE_ARGS
.endm
########################################################################
-## void sha256_transform_rorx(struct sha256_state *state, const u8 *data, int blocks)
-## arg 1 : pointer to state
-## arg 2 : pointer to input data
-## arg 3 : Num blocks
+## void sha256_transform_rorx(u32 state[SHA256_STATE_WORDS],
+## const u8 *data, size_t nblocks);
########################################################################
.text
-SYM_TYPED_FUNC_START(sha256_transform_rorx)
+SYM_FUNC_START(sha256_transform_rorx)
+ ANNOTATE_NOENDBR # since this is called only via static_call
+
pushq %rbx
pushq %r12
pushq %r13
pushq %r14
pushq %r15
diff --git a/arch/x86/crypto/sha256_ni_asm.S b/arch/x86/lib/crypto/sha256-ni-asm.S
similarity index 85%
rename from arch/x86/crypto/sha256_ni_asm.S
rename to arch/x86/lib/crypto/sha256-ni-asm.S
index d515a55a3bc1d..d3548206cf3d4 100644
--- a/arch/x86/crypto/sha256_ni_asm.S
+++ b/arch/x86/lib/crypto/sha256-ni-asm.S
@@ -52,13 +52,13 @@
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
*/
#include <linux/linkage.h>
-#include <linux/cfi_types.h>
+#include <linux/objtool.h>
-#define DIGEST_PTR %rdi /* 1st arg */
+#define STATE_PTR %rdi /* 1st arg */
#define DATA_PTR %rsi /* 2nd arg */
#define NUM_BLKS %rdx /* 3rd arg */
#define SHA256CONSTANTS %rax
@@ -96,40 +96,36 @@
sha256msg1 \m0, \m3
.endif
.endm
/*
- * Intel SHA Extensions optimized implementation of a SHA-256 update function
+ * Intel SHA Extensions optimized implementation of a SHA-256 block function
*
- * The function takes a pointer to the current hash values, a pointer to the
- * input data, and a number of 64 byte blocks to process. Once all blocks have
- * been processed, the digest pointer is updated with the resulting hash value.
- * The function only processes complete blocks, there is no functionality to
- * store partial blocks. All message padding and hash value initialization must
- * be done outside the update function.
+ * This function takes a pointer to the current SHA-256 state, a pointer to the
+ * input data, and the number of 64-byte blocks to process. Once all blocks
+ * have been processed, the state is updated with the new state. This function
+ * only processes complete blocks. State initialization, buffering of partial
+ * blocks, and digest finalization is expected to be handled elsewhere.
*
- * void sha256_ni_transform(uint32_t *digest, const void *data,
- uint32_t numBlocks);
- * digest : pointer to digest
- * data: pointer to input data
- * numBlocks: Number of blocks to process
+ * void sha256_ni_transform(u32 state[SHA256_STATE_WORDS],
+ * const u8 *data, size_t nblocks);
*/
-
.text
-SYM_TYPED_FUNC_START(sha256_ni_transform)
+SYM_FUNC_START(sha256_ni_transform)
+ ANNOTATE_NOENDBR # since this is called only via static_call
shl $6, NUM_BLKS /* convert to bytes */
jz .Ldone_hash
add DATA_PTR, NUM_BLKS /* pointer to end of data */
/*
* load initial hash values
* Need to reorder these appropriately
* DCBA, HGFE -> ABEF, CDGH
*/
- movdqu 0*16(DIGEST_PTR), STATE0 /* DCBA */
- movdqu 1*16(DIGEST_PTR), STATE1 /* HGFE */
+ movdqu 0*16(STATE_PTR), STATE0 /* DCBA */
+ movdqu 1*16(STATE_PTR), STATE1 /* HGFE */
movdqa STATE0, TMP
punpcklqdq STATE1, STATE0 /* FEBA */
punpckhqdq TMP, STATE1 /* DCHG */
pshufd $0x1B, STATE0, STATE0 /* ABEF */
@@ -164,12 +160,12 @@ SYM_TYPED_FUNC_START(sha256_ni_transform)
punpcklqdq STATE1, STATE0 /* GHEF */
punpckhqdq TMP, STATE1 /* ABCD */
pshufd $0xB1, STATE0, STATE0 /* HGFE */
pshufd $0x1B, STATE1, STATE1 /* DCBA */
- movdqu STATE1, 0*16(DIGEST_PTR)
- movdqu STATE0, 1*16(DIGEST_PTR)
+ movdqu STATE1, 0*16(STATE_PTR)
+ movdqu STATE0, 1*16(STATE_PTR)
.Ldone_hash:
RET
SYM_FUNC_END(sha256_ni_transform)
diff --git a/arch/x86/crypto/sha256-ssse3-asm.S b/arch/x86/lib/crypto/sha256-ssse3-asm.S
similarity index 98%
rename from arch/x86/crypto/sha256-ssse3-asm.S
rename to arch/x86/lib/crypto/sha256-ssse3-asm.S
index 93264ee445432..7f24a4cdcb257 100644
--- a/arch/x86/crypto/sha256-ssse3-asm.S
+++ b/arch/x86/lib/crypto/sha256-ssse3-asm.S
@@ -45,11 +45,11 @@
# and search for that title.
#
########################################################################
#include <linux/linkage.h>
-#include <linux/cfi_types.h>
+#include <linux/objtool.h>
## assume buffers not aligned
#define MOVDQ movdqu
################################ Define Macros
@@ -346,19 +346,17 @@ a = TMP_
add y0, h # h = h + S1 + CH + k + w + S0 + MAJ
ROTATE_ARGS
.endm
########################################################################
-## void sha256_transform_ssse3(struct sha256_state *state, const u8 *data,
-## int blocks);
-## arg 1 : pointer to state
-## (struct sha256_state is assumed to begin with u32 state[8])
-## arg 2 : pointer to input data
-## arg 3 : Num blocks
+## void sha256_transform_ssse3(u32 state[SHA256_STATE_WORDS],
+## const u8 *data, size_t nblocks);
########################################################################
.text
-SYM_TYPED_FUNC_START(sha256_transform_ssse3)
+SYM_FUNC_START(sha256_transform_ssse3)
+ ANNOTATE_NOENDBR # since this is called only via static_call
+
pushq %rbx
pushq %r12
pushq %r13
pushq %r14
pushq %r15
diff --git a/arch/x86/lib/crypto/sha256.c b/arch/x86/lib/crypto/sha256.c
new file mode 100644
index 0000000000000..47865b5cd94be
--- /dev/null
+++ b/arch/x86/lib/crypto/sha256.c
@@ -0,0 +1,74 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * SHA-256 optimized for x86_64
+ *
+ * Copyright 2025 Google LLC
+ */
+#include <asm/fpu/api.h>
+#include <crypto/internal/sha2.h>
+#include <crypto/internal/simd.h>
+#include <linux/kernel.h>
+#include <linux/module.h>
+#include <linux/static_call.h>
+
+asmlinkage void sha256_transform_ssse3(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+asmlinkage void sha256_transform_avx(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+asmlinkage void sha256_transform_rorx(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+asmlinkage void sha256_ni_transform(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks);
+
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_sha256_x86);
+
+DEFINE_STATIC_CALL(sha256_blocks_x86, sha256_transform_ssse3);
+
+void sha256_blocks_arch(u32 state[SHA256_STATE_WORDS],
+ const u8 *data, size_t nblocks)
+{
+ if (static_branch_likely(&have_sha256_x86) && crypto_simd_usable()) {
+ kernel_fpu_begin();
+ static_call(sha256_blocks_x86)(state, data, nblocks);
+ kernel_fpu_end();
+ } else {
+ sha256_blocks_generic(state, data, nblocks);
+ }
+}
+EXPORT_SYMBOL(sha256_blocks_arch);
+
+bool sha256_is_arch_optimized(void)
+{
+ return static_key_enabled(&have_sha256_x86);
+}
+EXPORT_SYMBOL(sha256_is_arch_optimized);
+
+static int __init sha256_x86_mod_init(void)
+{
+ if (boot_cpu_has(X86_FEATURE_SHA_NI)) {
+ static_call_update(sha256_blocks_x86, sha256_ni_transform);
+ } else if (cpu_has_xfeatures(XFEATURE_MASK_SSE |
+ XFEATURE_MASK_YMM, NULL) &&
+ boot_cpu_has(X86_FEATURE_AVX)) {
+ if (boot_cpu_has(X86_FEATURE_AVX2) &&
+ boot_cpu_has(X86_FEATURE_BMI2))
+ static_call_update(sha256_blocks_x86,
+ sha256_transform_rorx);
+ else
+ static_call_update(sha256_blocks_x86,
+ sha256_transform_avx);
+ } else if (!boot_cpu_has(X86_FEATURE_SSSE3)) {
+ return 0;
+ }
+ static_branch_enable(&have_sha256_x86);
+ return 0;
+}
+arch_initcall(sha256_x86_mod_init);
+
+static void __exit sha256_x86_mod_exit(void)
+{
+}
+module_exit(sha256_x86_mod_exit);
+
+MODULE_LICENSE("GPL");
+MODULE_DESCRIPTION("SHA-256 optimized for x86_64");
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 12/13] crypto: sha256 - remove sha256_base.h
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
` (10 preceding siblings ...)
2025-04-28 17:00 ` [PATCH v4 11/13] crypto: x86/sha256 " Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-04-28 17:00 ` [PATCH v4 13/13] crypto: lib/sha256 - improve function prototypes Eric Biggers
2025-05-05 12:24 ` [PATCH v4 00/13] Architecture-optimized SHA-256 library API Herbert Xu
13 siblings, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
sha256_base.h is no longer used, so remove it.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
include/crypto/sha256_base.h | 183 -----------------------------------
1 file changed, 183 deletions(-)
delete mode 100644 include/crypto/sha256_base.h
diff --git a/include/crypto/sha256_base.h b/include/crypto/sha256_base.h
deleted file mode 100644
index 6878fb9c26c04..0000000000000
--- a/include/crypto/sha256_base.h
+++ /dev/null
@@ -1,183 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-only */
-/*
- * sha256_base.h - core logic for SHA-256 implementations
- *
- * Copyright (C) 2015 Linaro Ltd <ard.biesheuvel@linaro.org>
- */
-
-#ifndef _CRYPTO_SHA256_BASE_H
-#define _CRYPTO_SHA256_BASE_H
-
-#include <crypto/internal/hash.h>
-#include <crypto/internal/sha2.h>
-#include <linux/math.h>
-#include <linux/string.h>
-#include <linux/types.h>
-#include <linux/unaligned.h>
-
-typedef void (sha256_block_fn)(struct crypto_sha256_state *sst, u8 const *src,
- int blocks);
-
-static inline int sha224_base_init(struct shash_desc *desc)
-{
- struct sha256_state *sctx = shash_desc_ctx(desc);
-
- sha224_init(sctx);
- return 0;
-}
-
-static inline int sha256_base_init(struct shash_desc *desc)
-{
- struct sha256_state *sctx = shash_desc_ctx(desc);
-
- sha256_init(sctx);
- return 0;
-}
-
-static inline int lib_sha256_base_do_update(struct sha256_state *sctx,
- const u8 *data,
- unsigned int len,
- sha256_block_fn *block_fn)
-{
- unsigned int partial = sctx->count % SHA256_BLOCK_SIZE;
- struct crypto_sha256_state *state = (void *)sctx;
-
- sctx->count += len;
-
- if (unlikely((partial + len) >= SHA256_BLOCK_SIZE)) {
- int blocks;
-
- if (partial) {
- int p = SHA256_BLOCK_SIZE - partial;
-
- memcpy(sctx->buf + partial, data, p);
- data += p;
- len -= p;
-
- block_fn(state, sctx->buf, 1);
- }
-
- blocks = len / SHA256_BLOCK_SIZE;
- len %= SHA256_BLOCK_SIZE;
-
- if (blocks) {
- block_fn(state, data, blocks);
- data += blocks * SHA256_BLOCK_SIZE;
- }
- partial = 0;
- }
- if (len)
- memcpy(sctx->buf + partial, data, len);
-
- return 0;
-}
-
-static inline int lib_sha256_base_do_update_blocks(
- struct crypto_sha256_state *sctx, const u8 *data, unsigned int len,
- sha256_block_fn *block_fn)
-{
- unsigned int remain = len - round_down(len, SHA256_BLOCK_SIZE);
-
- sctx->count += len - remain;
- block_fn(sctx, data, len / SHA256_BLOCK_SIZE);
- return remain;
-}
-
-static inline int sha256_base_do_update_blocks(
- struct shash_desc *desc, const u8 *data, unsigned int len,
- sha256_block_fn *block_fn)
-{
- return lib_sha256_base_do_update_blocks(shash_desc_ctx(desc), data,
- len, block_fn);
-}
-
-static inline int lib_sha256_base_do_finup(struct crypto_sha256_state *sctx,
- const u8 *src, unsigned int len,
- sha256_block_fn *block_fn)
-{
- unsigned int bit_offset = SHA256_BLOCK_SIZE / 8 - 1;
- union {
- __be64 b64[SHA256_BLOCK_SIZE / 4];
- u8 u8[SHA256_BLOCK_SIZE * 2];
- } block = {};
-
- if (len >= bit_offset * 8)
- bit_offset += SHA256_BLOCK_SIZE / 8;
- memcpy(&block, src, len);
- block.u8[len] = 0x80;
- sctx->count += len;
- block.b64[bit_offset] = cpu_to_be64(sctx->count << 3);
- block_fn(sctx, block.u8, (bit_offset + 1) * 8 / SHA256_BLOCK_SIZE);
- memzero_explicit(&block, sizeof(block));
-
- return 0;
-}
-
-static inline int sha256_base_do_finup(struct shash_desc *desc,
- const u8 *src, unsigned int len,
- sha256_block_fn *block_fn)
-{
- struct crypto_sha256_state *sctx = shash_desc_ctx(desc);
-
- if (len >= SHA256_BLOCK_SIZE) {
- int remain;
-
- remain = lib_sha256_base_do_update_blocks(sctx, src, len,
- block_fn);
- src += len - remain;
- len = remain;
- }
- return lib_sha256_base_do_finup(sctx, src, len, block_fn);
-}
-
-static inline int lib_sha256_base_do_finalize(struct sha256_state *sctx,
- sha256_block_fn *block_fn)
-{
- unsigned int partial = sctx->count % SHA256_BLOCK_SIZE;
- struct crypto_sha256_state *state = (void *)sctx;
-
- sctx->count -= partial;
- return lib_sha256_base_do_finup(state, sctx->buf, partial, block_fn);
-}
-
-static inline int sha256_base_do_finalize(struct shash_desc *desc,
- sha256_block_fn *block_fn)
-{
- struct sha256_state *sctx = shash_desc_ctx(desc);
-
- return lib_sha256_base_do_finalize(sctx, block_fn);
-}
-
-static inline int __sha256_base_finish(u32 state[SHA256_DIGEST_SIZE / 4],
- u8 *out, unsigned int digest_size)
-{
- __be32 *digest = (__be32 *)out;
- int i;
-
- for (i = 0; digest_size > 0; i++, digest_size -= sizeof(__be32))
- put_unaligned_be32(state[i], digest++);
- return 0;
-}
-
-static inline void lib_sha256_base_finish(struct sha256_state *sctx, u8 *out,
- unsigned int digest_size)
-{
- __sha256_base_finish(sctx->state, out, digest_size);
- memzero_explicit(sctx, sizeof(*sctx));
-}
-
-static inline int sha256_base_finish(struct shash_desc *desc, u8 *out)
-{
- unsigned int digest_size = crypto_shash_digestsize(desc->tfm);
- struct crypto_sha256_state *sctx = shash_desc_ctx(desc);
-
- return __sha256_base_finish(sctx->state, out, digest_size);
-}
-
-static inline void sha256_transform_blocks(struct crypto_sha256_state *sst,
- const u8 *input, int blocks)
-{
- sha256_blocks_generic(sst->state, input, blocks);
-}
-
-#endif /* _CRYPTO_SHA256_BASE_H */
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* [PATCH v4 13/13] crypto: lib/sha256 - improve function prototypes
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
` (11 preceding siblings ...)
2025-04-28 17:00 ` [PATCH v4 12/13] crypto: sha256 - remove sha256_base.h Eric Biggers
@ 2025-04-28 17:00 ` Eric Biggers
2025-05-05 12:24 ` [PATCH v4 00/13] Architecture-optimized SHA-256 library API Herbert Xu
13 siblings, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-04-28 17:00 UTC (permalink / raw)
To: linux-crypto
Cc: linux-kernel, linux-arch, linux-arm-kernel, linux-mips,
linuxppc-dev, linux-riscv, sparclinux, linux-s390, x86,
Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Follow best practices by changing the length parameters to size_t and
explicitly specifying the length of the output digest arrays.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
include/crypto/sha2.h | 8 ++++----
lib/crypto/sha256.c | 8 ++++----
2 files changed, 8 insertions(+), 8 deletions(-)
diff --git a/include/crypto/sha2.h b/include/crypto/sha2.h
index 444484d1b1cfa..7dfc560daa2c7 100644
--- a/include/crypto/sha2.h
+++ b/include/crypto/sha2.h
@@ -93,13 +93,13 @@ static inline void sha256_init(struct sha256_state *sctx)
sctx->state[5] = SHA256_H5;
sctx->state[6] = SHA256_H6;
sctx->state[7] = SHA256_H7;
sctx->count = 0;
}
-void sha256_update(struct sha256_state *sctx, const u8 *data, unsigned int len);
-void sha256_final(struct sha256_state *sctx, u8 *out);
-void sha256(const u8 *data, unsigned int len, u8 *out);
+void sha256_update(struct sha256_state *sctx, const u8 *data, size_t len);
+void sha256_final(struct sha256_state *sctx, u8 out[SHA256_DIGEST_SIZE]);
+void sha256(const u8 *data, size_t len, u8 out[SHA256_DIGEST_SIZE]);
static inline void sha224_init(struct sha256_state *sctx)
{
sctx->state[0] = SHA224_H0;
sctx->state[1] = SHA224_H1;
@@ -110,8 +110,8 @@ static inline void sha224_init(struct sha256_state *sctx)
sctx->state[6] = SHA224_H6;
sctx->state[7] = SHA224_H7;
sctx->count = 0;
}
/* Simply use sha256_update as it is equivalent to sha224_update. */
-void sha224_final(struct sha256_state *sctx, u8 *out);
+void sha224_final(struct sha256_state *sctx, u8 out[SHA224_DIGEST_SIZE]);
#endif /* _CRYPTO_SHA2_H */
diff --git a/lib/crypto/sha256.c b/lib/crypto/sha256.c
index 4b19cf977ef1b..563f09c9f3815 100644
--- a/lib/crypto/sha256.c
+++ b/lib/crypto/sha256.c
@@ -68,11 +68,11 @@ static inline void __sha256_update(struct sha256_state *sctx, const u8 *data,
}
if (len)
memcpy(&sctx->buf[partial], data, len);
}
-void sha256_update(struct sha256_state *sctx, const u8 *data, unsigned int len)
+void sha256_update(struct sha256_state *sctx, const u8 *data, size_t len)
{
__sha256_update(sctx, data, len, false);
}
EXPORT_SYMBOL(sha256_update);
@@ -99,23 +99,23 @@ static inline void __sha256_final(struct sha256_state *sctx, u8 *out,
put_unaligned_be32(sctx->state[i / 4], out + i);
memzero_explicit(sctx, sizeof(*sctx));
}
-void sha256_final(struct sha256_state *sctx, u8 *out)
+void sha256_final(struct sha256_state *sctx, u8 out[SHA256_DIGEST_SIZE])
{
__sha256_final(sctx, out, SHA256_DIGEST_SIZE, false);
}
EXPORT_SYMBOL(sha256_final);
-void sha224_final(struct sha256_state *sctx, u8 *out)
+void sha224_final(struct sha256_state *sctx, u8 out[SHA224_DIGEST_SIZE])
{
__sha256_final(sctx, out, SHA224_DIGEST_SIZE, false);
}
EXPORT_SYMBOL(sha224_final);
-void sha256(const u8 *data, unsigned int len, u8 *out)
+void sha256(const u8 *data, size_t len, u8 out[SHA256_DIGEST_SIZE])
{
struct sha256_state sctx;
sha256_init(&sctx);
sha256_update(&sctx, data, len);
--
2.49.0
^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: [PATCH v4 01/13] crypto: sha256 - support arch-optimized lib and expose through shash
2025-04-28 17:00 ` [PATCH v4 01/13] crypto: sha256 - support arch-optimized lib and expose through shash Eric Biggers
@ 2025-04-30 3:48 ` Herbert Xu
0 siblings, 0 replies; 27+ messages in thread
From: Herbert Xu @ 2025-04-30 3:48 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, linux-kernel, linux-arch, linux-arm-kernel,
linux-mips, linuxppc-dev, linux-riscv, sparclinux, linux-s390,
x86, Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
On Mon, Apr 28, 2025 at 10:00:26AM -0700, Eric Biggers wrote:
> From: Eric Biggers <ebiggers@google.com>
>
> As has been done for various other algorithms, rework the design of the
> SHA-256 library to support arch-optimized implementations, and make
> crypto/sha256.c expose both generic and arch-optimized shash algorithms
> that wrap the library functions.
>
> This allows users of the SHA-256 library functions to take advantage of
> the arch-optimized code, and this makes it much simpler to integrate
> SHA-256 for each architecture.
>
> Note that sha256_base.h is not used in the new design. It will be
> removed once all the architecture-specific code has been updated.
>
> Move the generic block function into its own module to avoid a circular
> dependency from libsha256.ko => sha256-$ARCH.ko => libsha256.ko.
>
> Signed-off-by: Eric Biggers <ebiggers@google.com>
> ---
> crypto/Kconfig | 1 +
> crypto/Makefile | 3 +-
> crypto/sha256.c | 201 +++++++++++++++++++++++++++++++++
> crypto/sha256_generic.c | 102 -----------------
> include/crypto/internal/sha2.h | 28 +++++
> include/crypto/sha2.h | 15 +--
> include/crypto/sha256_base.h | 9 +-
> lib/crypto/Kconfig | 19 ++++
> lib/crypto/Makefile | 3 +
> lib/crypto/sha256-generic.c | 137 ++++++++++++++++++++++
> lib/crypto/sha256.c | 196 ++++++++++++++------------------
> 11 files changed, 487 insertions(+), 227 deletions(-)
> create mode 100644 crypto/sha256.c
> delete mode 100644 crypto/sha256_generic.c
> create mode 100644 include/crypto/internal/sha2.h
> create mode 100644 lib/crypto/sha256-generic.c
This is the patch that I will fold in here to maintain the existing
export format:
diff --git a/crypto/sha256.c b/crypto/sha256.c
index 1c2edcf9453d..c2588d08ee3e 100644
--- a/crypto/sha256.c
+++ b/crypto/sha256.c
@@ -116,6 +116,32 @@ static int crypto_sha224_final_arch(struct shash_desc *desc, u8 *out)
return 0;
}
+static int crypto_sha256_import_lib(struct shash_desc *desc, const void *in)
+{
+ struct sha256_state *sctx = shash_desc_ctx(desc);
+ const u8 *p = in;
+
+ memcpy(sctx, p, sizeof(*sctx));
+ p += sizeof(*sctx);
+ sctx->count += *p;
+ return 0;
+}
+
+static int crypto_sha256_export_lib(struct shash_desc *desc, void *out)
+{
+ struct sha256_state *sctx0 = shash_desc_ctx(desc);
+ struct sha256_state sctx = *sctx0;
+ unsigned int partial;
+ u8 *p = out;
+
+ partial = sctx.count % SHA256_BLOCK_SIZE;
+ sctx.count -= partial;
+ memcpy(p, &sctx, sizeof(sctx));
+ p += sizeof(sctx);
+ *p = partial;
+ return 0;
+}
+
static struct shash_alg algs[] = {
{
.base.cra_name = "sha256",
@@ -130,6 +156,10 @@ static struct shash_alg algs[] = {
.finup = crypto_sha256_finup_generic,
.digest = crypto_sha256_digest_generic,
.descsize = sizeof(struct sha256_state),
+ .statesize = sizeof(struct crypto_sha256_state) +
+ SHA256_BLOCK_SIZE + 1,
+ .import = crypto_sha256_import_lib,
+ .export = crypto_sha256_export_lib,
},
{
.base.cra_name = "sha224",
@@ -142,6 +172,10 @@ static struct shash_alg algs[] = {
.update = crypto_sha256_update_generic,
.final = crypto_sha224_final_generic,
.descsize = sizeof(struct sha256_state),
+ .statesize = sizeof(struct crypto_sha256_state) +
+ SHA256_BLOCK_SIZE + 1,
+ .import = crypto_sha256_import_lib,
+ .export = crypto_sha256_export_lib,
},
{
.base.cra_name = "sha256",
@@ -156,6 +190,10 @@ static struct shash_alg algs[] = {
.finup = crypto_sha256_finup_arch,
.digest = crypto_sha256_digest_arch,
.descsize = sizeof(struct sha256_state),
+ .statesize = sizeof(struct crypto_sha256_state) +
+ SHA256_BLOCK_SIZE + 1,
+ .import = crypto_sha256_import_lib,
+ .export = crypto_sha256_export_lib,
},
{
.base.cra_name = "sha224",
@@ -168,6 +206,10 @@ static struct shash_alg algs[] = {
.update = crypto_sha256_update_arch,
.final = crypto_sha224_final_arch,
.descsize = sizeof(struct sha256_state),
+ .statesize = sizeof(struct crypto_sha256_state) +
+ SHA256_BLOCK_SIZE + 1,
+ .import = crypto_sha256_import_lib,
+ .export = crypto_sha256_export_lib,
},
};
Cheers,
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply related [flat|nested] 27+ messages in thread
* Re: [PATCH v4 00/13] Architecture-optimized SHA-256 library API
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
` (12 preceding siblings ...)
2025-04-28 17:00 ` [PATCH v4 13/13] crypto: lib/sha256 - improve function prototypes Eric Biggers
@ 2025-05-05 12:24 ` Herbert Xu
13 siblings, 0 replies; 27+ messages in thread
From: Herbert Xu @ 2025-05-05 12:24 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, linux-kernel, linux-arch, linux-arm-kernel,
linux-mips, linuxppc-dev, linux-riscv, sparclinux, linux-s390,
x86, ardb, Jason, torvalds
Eric Biggers <ebiggers@kernel.org> wrote:
> This is based on cryptodev commit 2dfc7cd74a5e062a. It can also be
> retrieved from:
>
> git fetch https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git sha256-lib-v4
>
> Following the example of several other algorithms (e.g. CRC32, ChaCha,
> Poly1305, BLAKE2s), this series refactors the kernel's existing
> architecture-optimized SHA-256 code to be available via the library API,
> instead of just via the crypto_shash API as it was before. It also
> reimplements the SHA-256 crypto_shash API on top of the library API.
>
> This makes it possible to use the SHA-256 library in
> performance-critical cases. The new design is also much simpler, with a
> negative diffstat of almost 1200 lines. Finally, this also fixes the
> longstanding issue where the arch-optimized SHA-256 was disabled by
> default, so people often forgot to enable it.
>
> For now the SHA-256 library is well-covered by the crypto_shash
> self-tests, but I plan to add a test for the library directly later.
> I've fully tested this series on arm, arm64, riscv, and x86. On mips,
> powerpc, s390, and sparc I've only been able to partially test it, since
> QEMU does not support the SHA-256 instructions on those platforms. If
> anyone with access to a mips, powerpc, s390, or sparc system that has
> SHA-256 instructions can verify that the crypto self-tests still pass,
>
> Changed v1 => v4:
> - Moved sha256_generic_blocks() into its own module to avoid a
> circular module dependency.
> - Added Ard's Reviewed-by tags.
> - Rebased onto cryptodev.
>
> Eric Biggers (13):
> crypto: sha256 - support arch-optimized lib and expose through shash
> crypto: arm/sha256 - implement library instead of shash
> crypto: arm64/sha256 - remove obsolete chunking logic
> crypto: arm64/sha256 - implement library instead of shash
> crypto: mips/sha256 - implement library instead of shash
> crypto: powerpc/sha256 - implement library instead of shash
> crypto: riscv/sha256 - implement library instead of shash
> crypto: s390/sha256 - implement library instead of shash
> crypto: sparc - move opcodes.h into asm directory
> crypto: sparc/sha256 - implement library instead of shash
> crypto: x86/sha256 - implement library instead of shash
> crypto: sha256 - remove sha256_base.h
> crypto: lib/sha256 - improve function prototypes
>
> arch/arm/configs/exynos_defconfig | 1 -
> arch/arm/configs/milbeaut_m10v_defconfig | 1 -
> arch/arm/configs/multi_v7_defconfig | 1 -
> arch/arm/configs/omap2plus_defconfig | 1 -
> arch/arm/configs/pxa_defconfig | 1 -
> arch/arm/crypto/Kconfig | 21 -
> arch/arm/crypto/Makefile | 8 +-
> arch/arm/crypto/sha2-ce-glue.c | 87 ----
> arch/arm/crypto/sha256_glue.c | 107 -----
> arch/arm/crypto/sha256_glue.h | 9 -
> arch/arm/crypto/sha256_neon_glue.c | 75 ---
> arch/arm/lib/crypto/.gitignore | 1 +
> arch/arm/lib/crypto/Kconfig | 6 +
> arch/arm/lib/crypto/Makefile | 8 +-
> arch/arm/{ => lib}/crypto/sha256-armv4.pl | 0
> .../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 10 +-
> arch/arm/lib/crypto/sha256.c | 64 +++
> arch/arm64/configs/defconfig | 1 -
> arch/arm64/crypto/Kconfig | 19 -
> arch/arm64/crypto/Makefile | 13 +-
> arch/arm64/crypto/sha2-ce-glue.c | 138 ------
> arch/arm64/crypto/sha256-glue.c | 171 -------
> arch/arm64/lib/crypto/.gitignore | 1 +
> arch/arm64/lib/crypto/Kconfig | 5 +
> arch/arm64/lib/crypto/Makefile | 9 +-
> .../crypto/sha2-armv8.pl} | 0
> .../sha2-ce-core.S => lib/crypto/sha256-ce.S} | 36 +-
> arch/arm64/lib/crypto/sha256.c | 75 +++
> arch/mips/cavium-octeon/Kconfig | 6 +
> .../mips/cavium-octeon/crypto/octeon-sha256.c | 135 ++----
> arch/mips/configs/cavium_octeon_defconfig | 1 -
> arch/mips/crypto/Kconfig | 10 -
> arch/powerpc/crypto/Kconfig | 11 -
> arch/powerpc/crypto/Makefile | 2 -
> arch/powerpc/crypto/sha256-spe-glue.c | 128 ------
> arch/powerpc/lib/crypto/Kconfig | 6 +
> arch/powerpc/lib/crypto/Makefile | 3 +
> .../powerpc/{ => lib}/crypto/sha256-spe-asm.S | 0
> arch/powerpc/lib/crypto/sha256.c | 70 +++
> arch/riscv/crypto/Kconfig | 11 -
> arch/riscv/crypto/Makefile | 3 -
> arch/riscv/crypto/sha256-riscv64-glue.c | 125 -----
> arch/riscv/lib/crypto/Kconfig | 7 +
> arch/riscv/lib/crypto/Makefile | 3 +
> .../sha256-riscv64-zvknha_or_zvknhb-zvkb.S | 4 +-
> arch/riscv/lib/crypto/sha256.c | 62 +++
> arch/s390/configs/debug_defconfig | 1 -
> arch/s390/configs/defconfig | 1 -
> arch/s390/crypto/Kconfig | 10 -
> arch/s390/crypto/Makefile | 1 -
> arch/s390/crypto/sha256_s390.c | 144 ------
> arch/s390/lib/crypto/Kconfig | 6 +
> arch/s390/lib/crypto/Makefile | 2 +
> arch/s390/lib/crypto/sha256.c | 47 ++
> arch/sparc/crypto/Kconfig | 10 -
> arch/sparc/crypto/Makefile | 2 -
> arch/sparc/crypto/aes_asm.S | 3 +-
> arch/sparc/crypto/aes_glue.c | 3 +-
> arch/sparc/crypto/camellia_asm.S | 3 +-
> arch/sparc/crypto/camellia_glue.c | 3 +-
> arch/sparc/crypto/des_asm.S | 3 +-
> arch/sparc/crypto/des_glue.c | 3 +-
> arch/sparc/crypto/md5_asm.S | 3 +-
> arch/sparc/crypto/md5_glue.c | 3 +-
> arch/sparc/crypto/sha1_asm.S | 3 +-
> arch/sparc/crypto/sha1_glue.c | 3 +-
> arch/sparc/crypto/sha256_glue.c | 129 ------
> arch/sparc/crypto/sha512_asm.S | 3 +-
> arch/sparc/crypto/sha512_glue.c | 3 +-
> arch/sparc/{crypto => include/asm}/opcodes.h | 6 +-
> arch/sparc/lib/Makefile | 1 +
> arch/sparc/lib/crc32c_asm.S | 3 +-
> arch/sparc/lib/crypto/Kconfig | 8 +
> arch/sparc/lib/crypto/Makefile | 4 +
> arch/sparc/lib/crypto/sha256.c | 64 +++
> arch/sparc/{ => lib}/crypto/sha256_asm.S | 5 +-
> arch/x86/crypto/Kconfig | 14 -
> arch/x86/crypto/Makefile | 3 -
> arch/x86/crypto/sha256_ssse3_glue.c | 432 ------------------
> arch/x86/lib/crypto/Kconfig | 7 +
> arch/x86/lib/crypto/Makefile | 3 +
> arch/x86/{ => lib}/crypto/sha256-avx-asm.S | 12 +-
> arch/x86/{ => lib}/crypto/sha256-avx2-asm.S | 12 +-
> .../crypto/sha256-ni-asm.S} | 36 +-
> arch/x86/{ => lib}/crypto/sha256-ssse3-asm.S | 14 +-
> arch/x86/lib/crypto/sha256.c | 74 +++
> crypto/Kconfig | 1 +
> crypto/Makefile | 3 +-
> crypto/sha256.c | 201 ++++++++
> crypto/sha256_generic.c | 102 -----
> include/crypto/internal/sha2.h | 28 ++
> include/crypto/sha2.h | 23 +-
> include/crypto/sha256_base.h | 180 --------
> lib/crypto/Kconfig | 22 +
> lib/crypto/Makefile | 3 +
> lib/crypto/sha256-generic.c | 137 ++++++
> lib/crypto/sha256.c | 204 ++++-----
> 97 files changed, 1128 insertions(+), 2319 deletions(-)
> delete mode 100644 arch/arm/crypto/sha2-ce-glue.c
> delete mode 100644 arch/arm/crypto/sha256_glue.c
> delete mode 100644 arch/arm/crypto/sha256_glue.h
> delete mode 100644 arch/arm/crypto/sha256_neon_glue.c
> rename arch/arm/{ => lib}/crypto/sha256-armv4.pl (100%)
> rename arch/arm/{crypto/sha2-ce-core.S => lib/crypto/sha256-ce.S} (91%)
> create mode 100644 arch/arm/lib/crypto/sha256.c
> delete mode 100644 arch/arm64/crypto/sha2-ce-glue.c
> delete mode 100644 arch/arm64/crypto/sha256-glue.c
> rename arch/arm64/{crypto/sha512-armv8.pl => lib/crypto/sha2-armv8.pl} (100%)
> rename arch/arm64/{crypto/sha2-ce-core.S => lib/crypto/sha256-ce.S} (80%)
> create mode 100644 arch/arm64/lib/crypto/sha256.c
> delete mode 100644 arch/powerpc/crypto/sha256-spe-glue.c
> rename arch/powerpc/{ => lib}/crypto/sha256-spe-asm.S (100%)
> create mode 100644 arch/powerpc/lib/crypto/sha256.c
> delete mode 100644 arch/riscv/crypto/sha256-riscv64-glue.c
> rename arch/riscv/{ => lib}/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S (98%)
> create mode 100644 arch/riscv/lib/crypto/sha256.c
> delete mode 100644 arch/s390/crypto/sha256_s390.c
> create mode 100644 arch/s390/lib/crypto/sha256.c
> delete mode 100644 arch/sparc/crypto/sha256_glue.c
> rename arch/sparc/{crypto => include/asm}/opcodes.h (96%)
> create mode 100644 arch/sparc/lib/crypto/Kconfig
> create mode 100644 arch/sparc/lib/crypto/Makefile
> create mode 100644 arch/sparc/lib/crypto/sha256.c
> rename arch/sparc/{ => lib}/crypto/sha256_asm.S (95%)
> delete mode 100644 arch/x86/crypto/sha256_ssse3_glue.c
> rename arch/x86/{ => lib}/crypto/sha256-avx-asm.S (98%)
> rename arch/x86/{ => lib}/crypto/sha256-avx2-asm.S (98%)
> rename arch/x86/{crypto/sha256_ni_asm.S => lib/crypto/sha256-ni-asm.S} (85%)
> rename arch/x86/{ => lib}/crypto/sha256-ssse3-asm.S (98%)
> create mode 100644 arch/x86/lib/crypto/sha256.c
> create mode 100644 crypto/sha256.c
> delete mode 100644 crypto/sha256_generic.c
> create mode 100644 include/crypto/internal/sha2.h
> delete mode 100644 include/crypto/sha256_base.h
> create mode 100644 lib/crypto/sha256-generic.c
>
>
> base-commit: 2dfc7cd74a5e062a5405560447517e7aab1c7341
All applied with export/import addition to patch 1. Thanks.
--
Email: Herbert Xu <herbert@gondor.apana.org.au>
Home Page: http://gondor.apana.org.au/~herbert/
PGP Key: http://gondor.apana.org.au/~herbert/pubkey.txt
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v4 07/13] crypto: riscv/sha256 - implement library instead of shash
2025-04-28 17:00 ` [PATCH v4 07/13] crypto: riscv/sha256 " Eric Biggers
@ 2025-05-08 17:45 ` Palmer Dabbelt
2025-05-08 18:06 ` Eric Biggers
0 siblings, 1 reply; 27+ messages in thread
From: Palmer Dabbelt @ 2025-05-08 17:45 UTC (permalink / raw)
To: ebiggers
Cc: linux-crypto, linux-kernel, linux-arch, linux-arm-kernel,
linux-mips, linuxppc-dev, linux-riscv, sparclinux, linux-s390,
x86, Ard Biesheuvel, Jason, Linus Torvalds
On Mon, 28 Apr 2025 10:00:32 PDT (-0700), ebiggers@kernel.org wrote:
> From: Eric Biggers <ebiggers@google.com>
>
> Instead of providing crypto_shash algorithms for the arch-optimized
> SHA-256 code, instead implement the SHA-256 library. This is much
> simpler, it makes the SHA-256 library functions be arch-optimized, and
> it fixes the longstanding issue where the arch-optimized SHA-256 was
> disabled by default. SHA-256 still remains available through
> crypto_shash, but individual architectures no longer need to handle it.
>
> To match sha256_blocks_arch(), change the type of the nblocks parameter
> of the assembly function from int to size_t. The assembly function
> actually already treated it as size_t.
>
> Signed-off-by: Eric Biggers <ebiggers@google.com>
> ---
> arch/riscv/crypto/Kconfig | 11 --
> arch/riscv/crypto/Makefile | 3 -
> arch/riscv/crypto/sha256-riscv64-glue.c | 125 ------------------
> arch/riscv/lib/crypto/Kconfig | 7 +
> arch/riscv/lib/crypto/Makefile | 3 +
> .../sha256-riscv64-zvknha_or_zvknhb-zvkb.S | 4 +-
> arch/riscv/lib/crypto/sha256.c | 62 +++++++++
> 7 files changed, 74 insertions(+), 141 deletions(-)
> delete mode 100644 arch/riscv/crypto/sha256-riscv64-glue.c
> rename arch/riscv/{ => lib}/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S (98%)
> create mode 100644 arch/riscv/lib/crypto/sha256.c
>
> diff --git a/arch/riscv/crypto/Kconfig b/arch/riscv/crypto/Kconfig
> index 4863be2a4ec2f..cd9b776602f89 100644
> --- a/arch/riscv/crypto/Kconfig
> +++ b/arch/riscv/crypto/Kconfig
> @@ -26,21 +26,10 @@ config CRYPTO_GHASH_RISCV64
> GCM GHASH function (NIST SP 800-38D)
>
> Architecture: riscv64 using:
> - Zvkg vector crypto extension
>
> -config CRYPTO_SHA256_RISCV64
> - tristate "Hash functions: SHA-224 and SHA-256"
> - depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO
> - select CRYPTO_SHA256
> - help
> - SHA-224 and SHA-256 secure hash algorithm (FIPS 180)
> -
> - Architecture: riscv64 using:
> - - Zvknha or Zvknhb vector crypto extensions
> - - Zvkb vector crypto extension
> -
> config CRYPTO_SHA512_RISCV64
> tristate "Hash functions: SHA-384 and SHA-512"
> depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO
> select CRYPTO_SHA512
> help
> diff --git a/arch/riscv/crypto/Makefile b/arch/riscv/crypto/Makefile
> index 4ae9bf762e907..e10e8257734e3 100644
> --- a/arch/riscv/crypto/Makefile
> +++ b/arch/riscv/crypto/Makefile
> @@ -5,13 +5,10 @@ aes-riscv64-y := aes-riscv64-glue.o aes-riscv64-zvkned.o \
> aes-riscv64-zvkned-zvbb-zvkg.o aes-riscv64-zvkned-zvkb.o
>
> obj-$(CONFIG_CRYPTO_GHASH_RISCV64) += ghash-riscv64.o
> ghash-riscv64-y := ghash-riscv64-glue.o ghash-riscv64-zvkg.o
>
> -obj-$(CONFIG_CRYPTO_SHA256_RISCV64) += sha256-riscv64.o
> -sha256-riscv64-y := sha256-riscv64-glue.o sha256-riscv64-zvknha_or_zvknhb-zvkb.o
> -
> obj-$(CONFIG_CRYPTO_SHA512_RISCV64) += sha512-riscv64.o
> sha512-riscv64-y := sha512-riscv64-glue.o sha512-riscv64-zvknhb-zvkb.o
>
> obj-$(CONFIG_CRYPTO_SM3_RISCV64) += sm3-riscv64.o
> sm3-riscv64-y := sm3-riscv64-glue.o sm3-riscv64-zvksh-zvkb.o
> diff --git a/arch/riscv/crypto/sha256-riscv64-glue.c b/arch/riscv/crypto/sha256-riscv64-glue.c
> deleted file mode 100644
> index c998300ab8435..0000000000000
> --- a/arch/riscv/crypto/sha256-riscv64-glue.c
> +++ /dev/null
> @@ -1,125 +0,0 @@
> -// SPDX-License-Identifier: GPL-2.0-or-later
> -/*
> - * SHA-256 and SHA-224 using the RISC-V vector crypto extensions
> - *
> - * Copyright (C) 2022 VRULL GmbH
> - * Author: Heiko Stuebner <heiko.stuebner@vrull.eu>
> - *
> - * Copyright (C) 2023 SiFive, Inc.
> - * Author: Jerry Shih <jerry.shih@sifive.com>
> - */
> -
> -#include <asm/simd.h>
> -#include <asm/vector.h>
> -#include <crypto/internal/hash.h>
> -#include <crypto/internal/simd.h>
> -#include <crypto/sha256_base.h>
> -#include <linux/kernel.h>
> -#include <linux/module.h>
> -
> -/*
> - * Note: the asm function only uses the 'state' field of struct sha256_state.
> - * It is assumed to be the first field.
> - */
> -asmlinkage void sha256_transform_zvknha_or_zvknhb_zvkb(
> - struct crypto_sha256_state *state, const u8 *data, int num_blocks);
> -
> -static void sha256_block(struct crypto_sha256_state *state, const u8 *data,
> - int num_blocks)
> -{
> - /*
> - * Ensure struct crypto_sha256_state begins directly with the SHA-256
> - * 256-bit internal state, as this is what the asm function expects.
> - */
> - BUILD_BUG_ON(offsetof(struct crypto_sha256_state, state) != 0);
> -
> - if (crypto_simd_usable()) {
> - kernel_vector_begin();
> - sha256_transform_zvknha_or_zvknhb_zvkb(state, data, num_blocks);
> - kernel_vector_end();
> - } else
> - sha256_transform_blocks(state, data, num_blocks);
> -}
> -
> -static int riscv64_sha256_update(struct shash_desc *desc, const u8 *data,
> - unsigned int len)
> -{
> - return sha256_base_do_update_blocks(desc, data, len, sha256_block);
> -}
> -
> -static int riscv64_sha256_finup(struct shash_desc *desc, const u8 *data,
> - unsigned int len, u8 *out)
> -{
> - sha256_base_do_finup(desc, data, len, sha256_block);
> - return sha256_base_finish(desc, out);
> -}
> -
> -static int riscv64_sha256_digest(struct shash_desc *desc, const u8 *data,
> - unsigned int len, u8 *out)
> -{
> - return sha256_base_init(desc) ?:
> - riscv64_sha256_finup(desc, data, len, out);
> -}
> -
> -static struct shash_alg riscv64_sha256_algs[] = {
> - {
> - .init = sha256_base_init,
> - .update = riscv64_sha256_update,
> - .finup = riscv64_sha256_finup,
> - .digest = riscv64_sha256_digest,
> - .descsize = sizeof(struct crypto_sha256_state),
> - .digestsize = SHA256_DIGEST_SIZE,
> - .base = {
> - .cra_blocksize = SHA256_BLOCK_SIZE,
> - .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
> - CRYPTO_AHASH_ALG_FINUP_MAX,
> - .cra_priority = 300,
> - .cra_name = "sha256",
> - .cra_driver_name = "sha256-riscv64-zvknha_or_zvknhb-zvkb",
> - .cra_module = THIS_MODULE,
> - },
> - }, {
> - .init = sha224_base_init,
> - .update = riscv64_sha256_update,
> - .finup = riscv64_sha256_finup,
> - .descsize = sizeof(struct crypto_sha256_state),
> - .digestsize = SHA224_DIGEST_SIZE,
> - .base = {
> - .cra_blocksize = SHA224_BLOCK_SIZE,
> - .cra_flags = CRYPTO_AHASH_ALG_BLOCK_ONLY |
> - CRYPTO_AHASH_ALG_FINUP_MAX,
> - .cra_priority = 300,
> - .cra_name = "sha224",
> - .cra_driver_name = "sha224-riscv64-zvknha_or_zvknhb-zvkb",
> - .cra_module = THIS_MODULE,
> - },
> - },
> -};
> -
> -static int __init riscv64_sha256_mod_init(void)
> -{
> - /* Both zvknha and zvknhb provide the SHA-256 instructions. */
> - if ((riscv_isa_extension_available(NULL, ZVKNHA) ||
> - riscv_isa_extension_available(NULL, ZVKNHB)) &&
> - riscv_isa_extension_available(NULL, ZVKB) &&
> - riscv_vector_vlen() >= 128)
> - return crypto_register_shashes(riscv64_sha256_algs,
> - ARRAY_SIZE(riscv64_sha256_algs));
> -
> - return -ENODEV;
> -}
> -
> -static void __exit riscv64_sha256_mod_exit(void)
> -{
> - crypto_unregister_shashes(riscv64_sha256_algs,
> - ARRAY_SIZE(riscv64_sha256_algs));
> -}
> -
> -module_init(riscv64_sha256_mod_init);
> -module_exit(riscv64_sha256_mod_exit);
> -
> -MODULE_DESCRIPTION("SHA-256 (RISC-V accelerated)");
> -MODULE_AUTHOR("Heiko Stuebner <heiko.stuebner@vrull.eu>");
> -MODULE_LICENSE("GPL");
> -MODULE_ALIAS_CRYPTO("sha256");
> -MODULE_ALIAS_CRYPTO("sha224");
> diff --git a/arch/riscv/lib/crypto/Kconfig b/arch/riscv/lib/crypto/Kconfig
> index bc7a43f33eb3a..c100571feb7e8 100644
> --- a/arch/riscv/lib/crypto/Kconfig
> +++ b/arch/riscv/lib/crypto/Kconfig
> @@ -4,5 +4,12 @@ config CRYPTO_CHACHA_RISCV64
> tristate
> depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO
> default CRYPTO_LIB_CHACHA
> select CRYPTO_ARCH_HAVE_LIB_CHACHA
> select CRYPTO_LIB_CHACHA_GENERIC
> +
> +config CRYPTO_SHA256_RISCV64
> + tristate
> + depends on 64BIT && RISCV_ISA_V && TOOLCHAIN_HAS_VECTOR_CRYPTO
> + default CRYPTO_LIB_SHA256
> + select CRYPTO_ARCH_HAVE_LIB_SHA256
> + select CRYPTO_LIB_SHA256_GENERIC
> diff --git a/arch/riscv/lib/crypto/Makefile b/arch/riscv/lib/crypto/Makefile
> index e27b78f317fc8..b7cb877a2c07e 100644
> --- a/arch/riscv/lib/crypto/Makefile
> +++ b/arch/riscv/lib/crypto/Makefile
> @@ -1,4 +1,7 @@
> # SPDX-License-Identifier: GPL-2.0-only
>
> obj-$(CONFIG_CRYPTO_CHACHA_RISCV64) += chacha-riscv64.o
> chacha-riscv64-y := chacha-riscv64-glue.o chacha-riscv64-zvkb.o
> +
> +obj-$(CONFIG_CRYPTO_SHA256_RISCV64) += sha256-riscv64.o
> +sha256-riscv64-y := sha256.o sha256-riscv64-zvknha_or_zvknhb-zvkb.o
> diff --git a/arch/riscv/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S b/arch/riscv/lib/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S
> similarity index 98%
> rename from arch/riscv/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S
> rename to arch/riscv/lib/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S
> index f1f5779e47323..fad501ad06171 100644
> --- a/arch/riscv/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S
> +++ b/arch/riscv/lib/crypto/sha256-riscv64-zvknha_or_zvknhb-zvkb.S
> @@ -104,12 +104,12 @@
> sha256_4rounds \last, \k1, W1, W2, W3, W0
> sha256_4rounds \last, \k2, W2, W3, W0, W1
> sha256_4rounds \last, \k3, W3, W0, W1, W2
> .endm
>
> -// void sha256_transform_zvknha_or_zvknhb_zvkb(u32 state[8], const u8 *data,
> -// int num_blocks);
> +// void sha256_transform_zvknha_or_zvknhb_zvkb(u32 state[SHA256_STATE_WORDS],
> +// const u8 *data, size_t nblocks);
> SYM_FUNC_START(sha256_transform_zvknha_or_zvknhb_zvkb)
>
> // Load the round constants into K0-K15.
> vsetivli zero, 4, e32, m1, ta, ma
> la t0, K256
> diff --git a/arch/riscv/lib/crypto/sha256.c b/arch/riscv/lib/crypto/sha256.c
> new file mode 100644
> index 0000000000000..18b84030f0b39
> --- /dev/null
> +++ b/arch/riscv/lib/crypto/sha256.c
> @@ -0,0 +1,62 @@
> +// SPDX-License-Identifier: GPL-2.0-or-later
> +/*
> + * SHA-256 (RISC-V accelerated)
> + *
> + * Copyright (C) 2022 VRULL GmbH
> + * Author: Heiko Stuebner <heiko.stuebner@vrull.eu>
> + *
> + * Copyright (C) 2023 SiFive, Inc.
> + * Author: Jerry Shih <jerry.shih@sifive.com>
> + */
> +
> +#include <asm/simd.h>
> +#include <asm/vector.h>
> +#include <crypto/internal/sha2.h>
> +#include <crypto/internal/simd.h>
> +#include <linux/kernel.h>
> +#include <linux/module.h>
> +
> +asmlinkage void sha256_transform_zvknha_or_zvknhb_zvkb(
> + u32 state[SHA256_STATE_WORDS], const u8 *data, size_t nblocks);
> +
> +static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_extensions);
> +
> +void sha256_blocks_arch(u32 state[SHA256_STATE_WORDS],
> + const u8 *data, size_t nblocks)
> +{
> + if (static_branch_likely(&have_extensions) && crypto_simd_usable()) {
> + kernel_vector_begin();
> + sha256_transform_zvknha_or_zvknhb_zvkb(state, data, nblocks);
> + kernel_vector_end();
> + } else {
> + sha256_blocks_generic(state, data, nblocks);
> + }
> +}
> +EXPORT_SYMBOL(sha256_blocks_arch);
> +
> +bool sha256_is_arch_optimized(void)
> +{
> + return static_key_enabled(&have_extensions);
> +}
> +EXPORT_SYMBOL(sha256_is_arch_optimized);
> +
> +static int __init riscv64_sha256_mod_init(void)
> +{
> + /* Both zvknha and zvknhb provide the SHA-256 instructions. */
> + if ((riscv_isa_extension_available(NULL, ZVKNHA) ||
> + riscv_isa_extension_available(NULL, ZVKNHB)) &&
> + riscv_isa_extension_available(NULL, ZVKB) &&
> + riscv_vector_vlen() >= 128)
> + static_branch_enable(&have_extensions);
> + return 0;
> +}
> +arch_initcall(riscv64_sha256_mod_init);
> +
> +static void __exit riscv64_sha256_mod_exit(void)
> +{
> +}
> +module_exit(riscv64_sha256_mod_exit);
> +
> +MODULE_DESCRIPTION("SHA-256 (RISC-V accelerated)");
> +MODULE_AUTHOR("Heiko Stuebner <heiko.stuebner@vrull.eu>");
> +MODULE_LICENSE("GPL");
Acked-by: Palmer Dabbelt <palmer@rivosinc.com>
I assume you want to keep these all together somewhere, so I'm going to
drop it from the RISC-V patchwork.
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v4 07/13] crypto: riscv/sha256 - implement library instead of shash
2025-05-08 17:45 ` Palmer Dabbelt
@ 2025-05-08 18:06 ` Eric Biggers
0 siblings, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-05-08 18:06 UTC (permalink / raw)
To: Palmer Dabbelt
Cc: linux-crypto, linux-kernel, linux-arch, linux-arm-kernel,
linux-mips, linuxppc-dev, linux-riscv, sparclinux, linux-s390,
x86, Ard Biesheuvel, Jason, Linus Torvalds
On Thu, May 08, 2025 at 10:45:39AM -0700, Palmer Dabbelt wrote:
>
> Acked-by: Palmer Dabbelt <palmer@rivosinc.com>
>
> I assume you want to keep these all together somewhere, so I'm going to
> drop it from the RISC-V patchwork.
Thanks! Yes, this series was already applied to "cryptodev"
(https://git.kernel.org/pub/scm/linux/kernel/git/herbert/cryptodev-2.6.git/log/).
- Eric
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
2025-04-28 17:00 ` [PATCH v4 08/13] crypto: s390/sha256 " Eric Biggers
@ 2025-05-29 17:05 ` Alex Williamson
2025-05-29 17:37 ` Eric Biggers
0 siblings, 1 reply; 27+ messages in thread
From: Alex Williamson @ 2025-05-29 17:05 UTC (permalink / raw)
To: Eric Biggers
Cc: linux-crypto, linux-kernel, linux-arch, linux-arm-kernel,
linux-mips, linuxppc-dev, linux-riscv, sparclinux, linux-s390,
x86, Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
On Mon, 28 Apr 2025 10:00:33 -0700
Eric Biggers <ebiggers@kernel.org> wrote:
> From: Eric Biggers <ebiggers@google.com>
>
> Instead of providing crypto_shash algorithms for the arch-optimized
> SHA-256 code, instead implement the SHA-256 library. This is much
> simpler, it makes the SHA-256 library functions be arch-optimized, and
> it fixes the longstanding issue where the arch-optimized SHA-256 was
> disabled by default. SHA-256 still remains available through
> crypto_shash, but individual architectures no longer need to handle it.
I can get to the following error after this patch, now merged as commit
b9eac03edcf8 ("crypto: s390/sha256 - implement library instead of shash"):
error: the following would cause module name conflict:
crypto/sha256.ko
arch/s390/lib/crypto/sha256.ko
Base config file is generated from:
$ CONFIG=$(mktemp)
$ cat << EOF > $CONFIG
CONFIG_MODULES=y
CONFIG_CRYPTO=y
CONFIG_CRYPTO_SHA256=m
EOF
Base config applied to allnoconfig:
$ KCONFIG_ALLCONFIG=$CONFIG make ARCH=s390 CROSS_COMPILE=s390x-linux-gnu- allnoconfig
Resulting in:
$ grep SHA256 .config
CONFIG_CRYPTO_SHA256=m
CONFIG_CRYPTO_LIB_SHA256=m
CONFIG_CRYPTO_ARCH_HAVE_LIB_SHA256=y
CONFIG_CRYPTO_LIB_SHA256_GENERIC=m
CONFIG_CRYPTO_SHA256_S390=m
Thanks,
Alex
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
2025-05-29 17:05 ` Alex Williamson
@ 2025-05-29 17:37 ` Eric Biggers
2025-05-29 19:00 ` Eric Biggers
2025-05-29 20:14 ` Linus Torvalds
0 siblings, 2 replies; 27+ messages in thread
From: Eric Biggers @ 2025-05-29 17:37 UTC (permalink / raw)
To: Alex Williamson
Cc: linux-crypto, linux-kernel, linux-arch, linux-arm-kernel,
linux-mips, linuxppc-dev, linux-riscv, sparclinux, linux-s390,
x86, Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
On Thu, May 29, 2025 at 11:05:26AM -0600, Alex Williamson wrote:
> On Mon, 28 Apr 2025 10:00:33 -0700
> Eric Biggers <ebiggers@kernel.org> wrote:
>
> > From: Eric Biggers <ebiggers@google.com>
> >
> > Instead of providing crypto_shash algorithms for the arch-optimized
> > SHA-256 code, instead implement the SHA-256 library. This is much
> > simpler, it makes the SHA-256 library functions be arch-optimized, and
> > it fixes the longstanding issue where the arch-optimized SHA-256 was
> > disabled by default. SHA-256 still remains available through
> > crypto_shash, but individual architectures no longer need to handle it.
>
> I can get to the following error after this patch, now merged as commit
> b9eac03edcf8 ("crypto: s390/sha256 - implement library instead of shash"):
>
> error: the following would cause module name conflict:
> crypto/sha256.ko
> arch/s390/lib/crypto/sha256.ko
Thanks for reporting this. For now the s390 one should be renamed to
sha256-s390, similar to how the other architectures' sha256 modules are named.
I'll send a patch.
Long-term, I'd like to find a clean way to consolidate the library code for each
algorithm into a single module. So instead of e.g. libsha256.ko,
libsha256-generic.ko, and sha256-s390.ko (all of which get loaded when the
SHA-256 library is needed), we'd just have libsha256.ko. (Or just sha256.ko,
with the old-school crypto API one renamed to sha256-cryptoapi.ko.) A lot of
these weird build problems we've been having are caused by the unnecessary
separation into multiple modules.
- Eric
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
2025-05-29 17:37 ` Eric Biggers
@ 2025-05-29 19:00 ` Eric Biggers
2025-05-29 20:14 ` Linus Torvalds
1 sibling, 0 replies; 27+ messages in thread
From: Eric Biggers @ 2025-05-29 19:00 UTC (permalink / raw)
To: Alex Williamson
Cc: linux-crypto, linux-kernel, linux-arch, linux-arm-kernel,
linux-mips, linuxppc-dev, linux-riscv, sparclinux, linux-s390,
x86, Ard Biesheuvel, Jason A . Donenfeld, Linus Torvalds
On Thu, May 29, 2025 at 05:37:02PM +0000, Eric Biggers wrote:
> On Thu, May 29, 2025 at 11:05:26AM -0600, Alex Williamson wrote:
> > On Mon, 28 Apr 2025 10:00:33 -0700
> > Eric Biggers <ebiggers@kernel.org> wrote:
> >
> > > From: Eric Biggers <ebiggers@google.com>
> > >
> > > Instead of providing crypto_shash algorithms for the arch-optimized
> > > SHA-256 code, instead implement the SHA-256 library. This is much
> > > simpler, it makes the SHA-256 library functions be arch-optimized, and
> > > it fixes the longstanding issue where the arch-optimized SHA-256 was
> > > disabled by default. SHA-256 still remains available through
> > > crypto_shash, but individual architectures no longer need to handle it.
> >
> > I can get to the following error after this patch, now merged as commit
> > b9eac03edcf8 ("crypto: s390/sha256 - implement library instead of shash"):
> >
> > error: the following would cause module name conflict:
> > crypto/sha256.ko
> > arch/s390/lib/crypto/sha256.ko
>
> Thanks for reporting this. For now the s390 one should be renamed to
> sha256-s390, similar to how the other architectures' sha256 modules are named.
> I'll send a patch.
>
> Long-term, I'd like to find a clean way to consolidate the library code for each
> algorithm into a single module. So instead of e.g. libsha256.ko,
> libsha256-generic.ko, and sha256-s390.ko (all of which get loaded when the
> SHA-256 library is needed), we'd just have libsha256.ko. (Or just sha256.ko,
> with the old-school crypto API one renamed to sha256-cryptoapi.ko.) A lot of
> these weird build problems we've been having are caused by the unnecessary
> separation into multiple modules.
>
> - Eric
>
Patch sent: https://lore.kernel.org/r/20250529185913.25091-1-ebiggers@kernel.org
- Eric
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
2025-05-29 17:37 ` Eric Biggers
2025-05-29 19:00 ` Eric Biggers
@ 2025-05-29 20:14 ` Linus Torvalds
2025-05-29 21:16 ` Eric Biggers
1 sibling, 1 reply; 27+ messages in thread
From: Linus Torvalds @ 2025-05-29 20:14 UTC (permalink / raw)
To: Eric Biggers
Cc: Alex Williamson, linux-crypto, linux-kernel, linux-arch,
linux-arm-kernel, linux-mips, linuxppc-dev, linux-riscv,
sparclinux, linux-s390, x86, Ard Biesheuvel, Jason A . Donenfeld
On Thu, 29 May 2025 at 10:37, Eric Biggers <ebiggers@kernel.org> wrote:
>
> Long-term, I'd like to find a clean way to consolidate the library code for each
> algorithm into a single module.
No, while I think the current situation isn't great, I think the "make
it one single module" is even worse.
For most architectures - including s390 - you end up being in the
situation that these kinds of hw accelerated crypto things depend on
some CPU capability, and aren't necessarily statically always
available.
So these things end up having stupid extra overhead due to having some
conditional.
That extra overhead is then in turn minimized with tricks like static
branches, but that's all just just piling more ugly hacks on top
because it picked a bad choice to begin with.
So what's the *right* thing to do?
The right thing to do is to just link the right routine in the first
place, and *not* have static branch hackery at all. Because you didn't
need it.
And we already do runtime linking at module loading time. So if it's a
module, if the hardware acceleration doesn't exist, the module load
should just fail, and the loader should go on to load the next option.
Not any silly "one module to rule them all" hackery that only results
in worse code. Just a simple "if this module loads successfully,
you'll link the optimal hw acceleration".
Now, the problem with this all is the *non*modular case.
For modules, we already have the optimal solution in the form of
init-module error handling and runtime linking.
So I think the module case is "solved" (except the solution is not
what we actually do).
For the non-module case, the problem is that "I linked this
unconditionally, and now it turns out I run on hardware that doesn't
have the capability to run this".
And that's when you need to do things like static_call_update() to
basically do runtime re-linking of a static decision.
And currently we very much do this wrong. See how s390 and x86-64 (and
presumably others) basically have the *exact* same problems, but they
then mix static branches and static calls (in the case of x86-64) and
just have non-optimal code in general.
What I think the generic code should do (for the built-in case) is just have
DEFINE_STATIC_CALL(sha256_blocks_fn, sha256_blocks_generic);
and do
static_call(sha256_blocks_fn)(args..);
and then architecture code can do the static_call_update() to set
their optimal version.
And yeah, we'd presumably need multiple versions, since there's the
whole "is simd usable" thing. Although maybe that's going away?
Linus
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
2025-05-29 20:14 ` Linus Torvalds
@ 2025-05-29 21:16 ` Eric Biggers
2025-05-29 23:54 ` Linus Torvalds
0 siblings, 1 reply; 27+ messages in thread
From: Eric Biggers @ 2025-05-29 21:16 UTC (permalink / raw)
To: Linus Torvalds
Cc: Alex Williamson, linux-crypto, linux-kernel, linux-arch,
linux-arm-kernel, linux-mips, linuxppc-dev, linux-riscv,
sparclinux, linux-s390, x86, Ard Biesheuvel, Jason A . Donenfeld
On Thu, May 29, 2025 at 01:14:31PM -0700, Linus Torvalds wrote:
> On Thu, 29 May 2025 at 10:37, Eric Biggers <ebiggers@kernel.org> wrote:
> >
> > Long-term, I'd like to find a clean way to consolidate the library code for each
> > algorithm into a single module.
>
> No, while I think the current situation isn't great, I think the "make
> it one single module" is even worse.
>
> For most architectures - including s390 - you end up being in the
> situation that these kinds of hw accelerated crypto things depend on
> some CPU capability, and aren't necessarily statically always
> available.
>
> So these things end up having stupid extra overhead due to having some
> conditional.
>
> That extra overhead is then in turn minimized with tricks like static
> branches, but that's all just just piling more ugly hacks on top
> because it picked a bad choice to begin with.
>
> So what's the *right* thing to do?
>
> The right thing to do is to just link the right routine in the first
> place, and *not* have static branch hackery at all. Because you didn't
> need it.
>
> And we already do runtime linking at module loading time. So if it's a
> module, if the hardware acceleration doesn't exist, the module load
> should just fail, and the loader should go on to load the next option.
So using crc32c() + ext4 + x86 as an example (but SHA-256 would be very
similar), the current behavior is that ext4.ko depends on the crc32c_arch()
symbol. That causes crc32-x86.ko to be loaded, which then depends on the
crc32c_base() symbol as a fallback, which causes crc32.ko to be loaded too. My
idea is to consolidate the two crc32 modules into one (they always go together,
after all), keeping the same symbols. The main challenge is just the current
directory structure.
Your suggestion sounds like: ext4.ko would depend on the crc32c() symbol, which
would be defined in *both* crc32-x86.ko and crc32.ko. The module loader would
try to load crc32-x86.ko first. If the CPU does not support any of the x86
accelerated CRC32 code, then loading that module would fail. The module loader
would then load crc32.ko instead.
Does any of the infrastructure to handle "this symbol is in multiple modules and
they must be loaded in this particular order" actually exist, though?
And how do we avoid the issues the crypto API often has where the accelerated
modules don't get loaded, causing slow generic code to unnecessarily be used?
IMO this sounds questionable compared to just using static keys and/or branches,
which we'd need anyway to support the non-modular case.
> Not any silly "one module to rule them all" hackery that only results
> in worse code. Just a simple "if this module loads successfully,
> you'll link the optimal hw acceleration".
>
> Now, the problem with this all is the *non*modular case.
>
> For modules, we already have the optimal solution in the form of
> init-module error handling and runtime linking.
>
> So I think the module case is "solved" (except the solution is not
> what we actually do).
>
> For the non-module case, the problem is that "I linked this
> unconditionally, and now it turns out I run on hardware that doesn't
> have the capability to run this".
>
> And that's when you need to do things like static_call_update() to
> basically do runtime re-linking of a static decision.
>
> And currently we very much do this wrong. See how s390 and x86-64 (and
> presumably others) basically have the *exact* same problems, but they
> then mix static branches and static calls (in the case of x86-64) and
> just have non-optimal code in general.
>
> What I think the generic code should do (for the built-in case) is just have
>
> DEFINE_STATIC_CALL(sha256_blocks_fn, sha256_blocks_generic);
>
> and do
>
> static_call(sha256_blocks_fn)(args..);
>
> and then architecture code can do the static_call_update() to set
> their optimal version.
>
> And yeah, we'd presumably need multiple versions, since there's the
> whole "is simd usable" thing. Although maybe that's going away?
Moving the static_call into the generic code might make sense. I don't think
it's a win in all cases currently, though. Only x86 and PPC32 actually have a
real static_call implementation; everywhere else it's an indirect call which is
slower than a static branch. Also, some arch code is just usable
unconditionally without any CPU feature check, e.g. the MIPS ChaCha code. That
doesn't use (or need to use) a static call or branch at all.
Also, while the centralized static_call would *allow* for the generic code to be
loaded while the arch code is not, in the vast majority of cases that would be a
bug, not a feature. The generic crypto infrastructure has that bug, and this
has caused a huge amount of pain over the years. People have to go out of the
way to ensure that the arch-optimized crypto code gets loaded. And they often
forget, resulting in the slow generic code being used unnecessarily...
Making the arch-optimized code be loaded through a direct symbol dependency
solves that problem.
- Eric
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
2025-05-29 21:16 ` Eric Biggers
@ 2025-05-29 23:54 ` Linus Torvalds
2025-05-30 0:18 ` Eric Biggers
0 siblings, 1 reply; 27+ messages in thread
From: Linus Torvalds @ 2025-05-29 23:54 UTC (permalink / raw)
To: Eric Biggers
Cc: Alex Williamson, linux-crypto, linux-kernel, linux-arch,
linux-arm-kernel, linux-mips, linuxppc-dev, linux-riscv,
sparclinux, linux-s390, x86, Ard Biesheuvel, Jason A . Donenfeld
On Thu, 29 May 2025 at 14:16, Eric Biggers <ebiggers@kernel.org> wrote:
>
> So using crc32c() + ext4 + x86 as an example (but SHA-256 would be very
> similar), the current behavior is that ext4.ko depends on the crc32c_arch()
> symbol.
Yes, I think that's a good example.
I think it's an example of something that "works", but it certainly is
a bit hacky.
Wouldn't it be nicer if just plain "crc32c()" did the right thing,
instead of users having to do strange hacks just to get the optimized
version that they are looking for?
> Does any of the infrastructure to handle "this symbol is in multiple modules and
> they must be loaded in this particular order" actually exist, though?
Hmm. I was sure we already did that for other things, but looking
around, I'm not finding any cases.
Or rather, I _am_ finding cases where we export the same symbol from
different code, but all the ones I found were being careful to not be
active at the same time.
I really thought we had cases where depending on which module you
loaded you got different implementations, but it looks like it either
was some historical thing that no longer exists - or that I need to go
take my meds.
> IMO this sounds questionable compared to just using static keys and/or branches,
> which we'd need anyway to support the non-modular case.
I really wish the non-modular case used static calls, not static keys
like it does now.
In fact, that should work even for modular users.
Of course, not all architectures actually do the optimized thing, and
the generic fallback uses indirect calls through a function pointer,
but hey, if an architecture didn't bother with the rewriting code that
is fixable - if the architecture maintainer cares.
(On some architectures, indirect calls are not noticeably slower than
direct calls - because you have to load the address from some global
pointer area anyway - so not having the rewriting can be a "we don't
need it" thing)
Linus
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
2025-05-29 23:54 ` Linus Torvalds
@ 2025-05-30 0:18 ` Eric Biggers
2025-06-01 23:00 ` Eric Biggers
0 siblings, 1 reply; 27+ messages in thread
From: Eric Biggers @ 2025-05-30 0:18 UTC (permalink / raw)
To: Linus Torvalds
Cc: Alex Williamson, linux-crypto, linux-kernel, linux-arch,
linux-arm-kernel, linux-mips, linuxppc-dev, linux-riscv,
sparclinux, linux-s390, x86, Ard Biesheuvel, Jason A . Donenfeld
On Thu, May 29, 2025 at 04:54:34PM -0700, Linus Torvalds wrote:
> On Thu, 29 May 2025 at 14:16, Eric Biggers <ebiggers@kernel.org> wrote:
> >
> > So using crc32c() + ext4 + x86 as an example (but SHA-256 would be very
> > similar), the current behavior is that ext4.ko depends on the crc32c_arch()
> > symbol.
>
> Yes, I think that's a good example.
>
> I think it's an example of something that "works", but it certainly is
> a bit hacky.
>
> Wouldn't it be nicer if just plain "crc32c()" did the right thing,
> instead of users having to do strange hacks just to get the optimized
> version that they are looking for?
For crc32c() that's exactly how it works (since v6.14, when I implemented it).
The users call crc32c() which is an inline function, which then calls
crc32c_arch() or crc32c_base() depending on the kconfig. So that's why I said
the symbol dependency is currently on crc32c_arch. Sorry if I wasn't clear.
The SHA-256, ChaCha, and Poly1305 library code now has a similar design too.
If we merged the arch and generic modules together, then the symbol would become
crc32c. But in either case crc32c() is the API that all the users call.
- Eric
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
2025-05-30 0:18 ` Eric Biggers
@ 2025-06-01 23:00 ` Eric Biggers
2025-06-02 14:45 ` Linus Torvalds
0 siblings, 1 reply; 27+ messages in thread
From: Eric Biggers @ 2025-06-01 23:00 UTC (permalink / raw)
To: Linus Torvalds
Cc: Alex Williamson, linux-crypto, linux-kernel, linux-arch,
linux-arm-kernel, linux-mips, linuxppc-dev, linux-riscv,
sparclinux, linux-s390, x86, Ard Biesheuvel, Jason A . Donenfeld
On Fri, May 30, 2025 at 12:18:58AM +0000, Eric Biggers wrote:
> On Thu, May 29, 2025 at 04:54:34PM -0700, Linus Torvalds wrote:
> > On Thu, 29 May 2025 at 14:16, Eric Biggers <ebiggers@kernel.org> wrote:
> > >
> > > So using crc32c() + ext4 + x86 as an example (but SHA-256 would be very
> > > similar), the current behavior is that ext4.ko depends on the crc32c_arch()
> > > symbol.
> >
> > Yes, I think that's a good example.
> >
> > I think it's an example of something that "works", but it certainly is
> > a bit hacky.
> >
> > Wouldn't it be nicer if just plain "crc32c()" did the right thing,
> > instead of users having to do strange hacks just to get the optimized
> > version that they are looking for?
>
> For crc32c() that's exactly how it works (since v6.14, when I implemented it).
> The users call crc32c() which is an inline function, which then calls
> crc32c_arch() or crc32c_base() depending on the kconfig. So that's why I said
> the symbol dependency is currently on crc32c_arch. Sorry if I wasn't clear.
> The SHA-256, ChaCha, and Poly1305 library code now has a similar design too.
>
> If we merged the arch and generic modules together, then the symbol would become
> crc32c. But in either case crc32c() is the API that all the users call.
>
> - Eric
>
I implemented my proposal, for lib/crc first, in
https://lore.kernel.org/lkml/20250601224441.778374-1-ebiggers@kernel.org.
I think it's strictly better than the status quo, and once applied to lib/crypto
it will solve some of the problems we've been having there too. But let me know
if you still have misgivings.
- Eric
^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [PATCH v4 08/13] crypto: s390/sha256 - implement library instead of shash
2025-06-01 23:00 ` Eric Biggers
@ 2025-06-02 14:45 ` Linus Torvalds
0 siblings, 0 replies; 27+ messages in thread
From: Linus Torvalds @ 2025-06-02 14:45 UTC (permalink / raw)
To: Eric Biggers
Cc: Alex Williamson, linux-crypto, linux-kernel, linux-arch,
linux-arm-kernel, linux-mips, linuxppc-dev, linux-riscv,
sparclinux, linux-s390, x86, Ard Biesheuvel, Jason A . Donenfeld
On Sun, 1 Jun 2025 at 16:00, Eric Biggers <ebiggers@kernel.org> wrote:
>
> I implemented my proposal, for lib/crc first,
Ok, I scanned through that series, and it looks good to me. A clear improvement.
Linus
^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2025-06-02 14:46 UTC | newest]
Thread overview: 27+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-04-28 17:00 [PATCH v4 00/13] Architecture-optimized SHA-256 library API Eric Biggers
2025-04-28 17:00 ` [PATCH v4 01/13] crypto: sha256 - support arch-optimized lib and expose through shash Eric Biggers
2025-04-30 3:48 ` Herbert Xu
2025-04-28 17:00 ` [PATCH v4 02/13] crypto: arm/sha256 - implement library instead of shash Eric Biggers
2025-04-28 17:00 ` [PATCH v4 03/13] crypto: arm64/sha256 - remove obsolete chunking logic Eric Biggers
2025-04-28 17:00 ` [PATCH v4 04/13] crypto: arm64/sha256 - implement library instead of shash Eric Biggers
2025-04-28 17:00 ` [PATCH v4 05/13] crypto: mips/sha256 " Eric Biggers
2025-04-28 17:00 ` [PATCH v4 06/13] crypto: powerpc/sha256 " Eric Biggers
2025-04-28 17:00 ` [PATCH v4 07/13] crypto: riscv/sha256 " Eric Biggers
2025-05-08 17:45 ` Palmer Dabbelt
2025-05-08 18:06 ` Eric Biggers
2025-04-28 17:00 ` [PATCH v4 08/13] crypto: s390/sha256 " Eric Biggers
2025-05-29 17:05 ` Alex Williamson
2025-05-29 17:37 ` Eric Biggers
2025-05-29 19:00 ` Eric Biggers
2025-05-29 20:14 ` Linus Torvalds
2025-05-29 21:16 ` Eric Biggers
2025-05-29 23:54 ` Linus Torvalds
2025-05-30 0:18 ` Eric Biggers
2025-06-01 23:00 ` Eric Biggers
2025-06-02 14:45 ` Linus Torvalds
2025-04-28 17:00 ` [PATCH v4 09/13] crypto: sparc - move opcodes.h into asm directory Eric Biggers
2025-04-28 17:00 ` [PATCH v4 10/13] crypto: sparc/sha256 - implement library instead of shash Eric Biggers
2025-04-28 17:00 ` [PATCH v4 11/13] crypto: x86/sha256 " Eric Biggers
2025-04-28 17:00 ` [PATCH v4 12/13] crypto: sha256 - remove sha256_base.h Eric Biggers
2025-04-28 17:00 ` [PATCH v4 13/13] crypto: lib/sha256 - improve function prototypes Eric Biggers
2025-05-05 12:24 ` [PATCH v4 00/13] Architecture-optimized SHA-256 library API Herbert Xu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).