* [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated
@ 2025-06-01 22:44 Eric Biggers
2025-06-01 22:44 ` [PATCH 01/13] crypto/crc32: register only one shash_alg Eric Biggers
` (13 more replies)
0 siblings, 14 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
This series improves how lib/crc supports arch-optimized code. First,
instead of the arch-optimized CRC code being in arch/$(SRCARCH)/lib/, it
will now be in lib/crc/$(SRCARCH)/. Second, the API functions (e.g.
crc32c()), arch-optimized functions (e.g. crc32c_arch()), and generic
functions (e.g. crc32c_base()) will now be part of a single module for
each CRC type, allowing better inlining and dead code elimination. The
second change is made possible by the first.
As an example, consider CONFIG_CRC32=m on x86. We'll now have just
crc32.ko instead of both crc32-x86.ko and crc32.ko. The two modules
were already coupled together and always both got loaded together via
direct symbol dependency, so the separation provided no benefit.
Note: later I'd like to apply the same design to lib/crypto/ too, where
often the API functions are out-of-line so this will work even better.
In those cases, for each algorithm we currently have 3 modules all
coupled together, e.g. libsha256.ko, libsha256-generic.ko, and
sha256-x86.ko. We should have just one, inline things properly, and
rely on the compiler's dead code elimination to decide the inclusion of
the generic code instead of manually setting it via kconfig.
Having arch-specific code outside arch/ was somewhat controversial when
Zinc proposed it back in 2018. But I don't think the concerns are
warranted. It's better from a technical perspective, as it enables the
improvements mentioned above. This model is already successfully used
in other places in the kernel such as lib/raid6/. The community of each
architecture still remains free to work on the code, even if it's not in
arch/. At the time there was also a desire to put the library code in
the same files as the old-school crypto API, but that was a mistake; now
that the library is separate, that's no longer a constraint either.
Patches 1 and 2, which I previously sent out by themselves, are
prerequisites because they eliminate the need for the CRC32 library API
to expose the generic functions.
Eric Biggers (13):
crypto/crc32: register only one shash_alg
crypto/crc32c: register only one shash_alg
lib/crc: move files into lib/crc/
lib/crc: prepare for arch-optimized code in subdirs of lib/crc/
lib/crc/arm: migrate arm-optimized CRC code into lib/crc/
lib/crc/arm64: migrate arm64-optimized CRC code into lib/crc/
lib/crc/loongarch: migrate loongarch-optimized CRC code into lib/crc/
lib/crc/mips: migrate mips-optimized CRC code into lib/crc/
lib/crc/powerpc: migrate powerpc-optimized CRC code into lib/crc/
lib/crc/riscv: migrate riscv-optimized CRC code into lib/crc/
lib/crc/s390: migrate s390-optimized CRC code into lib/s390/
lib/crc/sparc: migrate sparc-optimized CRC code into lib/crc/
lib/crc/x86: migrate x86-optimized CRC code into lib/crc/
Documentation/core-api/kernel-api.rst | 14 +-
MAINTAINERS | 4 +-
arch/arm/Kconfig | 2 -
arch/arm/lib/Makefile | 6 -
arch/arm64/Kconfig | 2 -
arch/arm64/lib/Makefile | 6 -
arch/loongarch/Kconfig | 1 -
arch/loongarch/lib/Makefile | 2 -
arch/mips/Kconfig | 1 -
arch/mips/lib/Makefile | 2 -
arch/powerpc/Kconfig | 2 -
arch/powerpc/lib/Makefile | 6 -
arch/riscv/Kconfig | 3 -
arch/riscv/lib/Makefile | 6 -
arch/s390/Kconfig | 1 -
arch/s390/lib/Makefile | 3 -
arch/sparc/Kconfig | 1 -
arch/sparc/lib/Makefile | 2 -
arch/x86/Kconfig | 3 -
arch/x86/lib/Makefile | 10 --
crypto/crc32.c | 69 ++--------
crypto/crc32c.c | 70 ++--------
include/linux/crc-t10dif.h | 10 +-
include/linux/crc32.h | 30 +----
include/linux/crc64.h | 22 +---
lib/Kconfig | 87 +------------
lib/Kconfig.debug | 21 ---
lib/Makefile | 32 +----
lib/crc/.gitignore | 5 +
lib/crc/Kconfig | 121 ++++++++++++++++++
lib/crc/Makefile | 62 +++++++++
.../arm/lib => lib/crc/arm}/crc-t10dif-core.S | 0
.../crc-t10dif.c => lib/crc/arm/crc-t10dif.h | 23 +---
{arch/arm/lib => lib/crc/arm}/crc32-core.S | 0
arch/arm/lib/crc32.c => lib/crc/arm/crc32.h | 40 ++----
.../lib => lib/crc/arm64}/crc-t10dif-core.S | 0
.../crc/arm64/crc-t10dif.h | 22 +---
.../arm64/lib => lib/crc/arm64}/crc32-core.S | 0
.../lib/crc32.c => lib/crc/arm64/crc32.h | 22 +---
lib/{ => crc}/crc-ccitt.c | 3 -
lib/{ => crc}/crc-itu-t.c | 0
lib/{crc-t10dif.c => crc/crc-t10dif-main.c} | 37 +++++-
lib/{ => crc}/crc16.c | 0
lib/{crc32.c => crc/crc32-main.c} | 77 +++++++++--
lib/{ => crc}/crc4.c | 0
lib/{crc64.c => crc/crc64-main.c} | 47 ++++++-
lib/{ => crc}/crc7.c | 0
lib/{ => crc}/crc8.c | 0
lib/{ => crc}/gen_crc32table.c | 4 +-
lib/{ => crc}/gen_crc64table.c | 11 +-
.../crc/loongarch/crc32.h | 34 +----
.../lib/crc32-mips.c => lib/crc/mips/crc32.h | 35 +----
.../crc/powerpc/crc-t10dif.h | 20 +--
.../crc/powerpc}/crc-vpmsum-template.S | 0
.../lib/crc32.c => lib/crc/powerpc/crc32.h | 36 +-----
.../crc/powerpc}/crc32c-vpmsum_asm.S | 0
.../crc/powerpc}/crct10dif-vpmsum_asm.S | 0
.../lib => lib/crc/riscv}/crc-clmul-consts.h | 0
.../crc/riscv}/crc-clmul-template.h | 0
{arch/riscv/lib => lib/crc/riscv}/crc-clmul.h | 0
.../crc/riscv/crc-t10dif.h | 8 +-
{arch/riscv/lib => lib/crc/riscv}/crc16_msb.c | 0
.../lib/crc32.c => lib/crc/riscv/crc32.h | 20 +--
{arch/riscv/lib => lib/crc/riscv}/crc32_lsb.c | 0
{arch/riscv/lib => lib/crc/riscv}/crc32_msb.c | 0
.../lib/crc64.c => lib/crc/riscv/crc64.h | 11 +-
{arch/riscv/lib => lib/crc/riscv}/crc64_lsb.c | 0
{arch/riscv/lib => lib/crc/riscv}/crc64_msb.c | 0
{arch/s390/lib => lib/crc/s390}/crc32-vx.h | 0
arch/s390/lib/crc32.c => lib/crc/s390/crc32.h | 19 +--
{arch/s390/lib => lib/crc/s390}/crc32be-vx.c | 0
{arch/s390/lib => lib/crc/s390}/crc32le-vx.c | 0
.../lib/crc32.c => lib/crc/sparc/crc32.h | 40 +-----
.../sparc/lib => lib/crc/sparc}/crc32c_asm.S | 0
lib/crc/tests/Makefile | 2 +
lib/{ => crc}/tests/crc_kunit.c | 0
.../lib => lib/crc/x86}/crc-pclmul-consts.h | 0
.../lib => lib/crc/x86}/crc-pclmul-template.S | 0
.../lib => lib/crc/x86}/crc-pclmul-template.h | 0
.../crc-t10dif.c => lib/crc/x86/crc-t10dif.h | 18 +--
.../lib => lib/crc/x86}/crc16-msb-pclmul.S | 0
{arch/x86/lib => lib/crc/x86}/crc32-pclmul.S | 0
arch/x86/lib/crc32.c => lib/crc/x86/crc32.h | 32 +----
{arch/x86/lib => lib/crc/x86}/crc32c-3way.S | 0
{arch/x86/lib => lib/crc/x86}/crc64-pclmul.S | 0
arch/x86/lib/crc64.c => lib/crc/x86/crc64.h | 21 +--
lib/tests/Makefile | 1 -
87 files changed, 446 insertions(+), 743 deletions(-)
create mode 100644 lib/crc/.gitignore
create mode 100644 lib/crc/Kconfig
create mode 100644 lib/crc/Makefile
rename {arch/arm/lib => lib/crc/arm}/crc-t10dif-core.S (100%)
rename arch/arm/lib/crc-t10dif.c => lib/crc/arm/crc-t10dif.h (70%)
rename {arch/arm/lib => lib/crc/arm}/crc32-core.S (100%)
rename arch/arm/lib/crc32.c => lib/crc/arm/crc32.h (69%)
rename {arch/arm64/lib => lib/crc/arm64}/crc-t10dif-core.S (100%)
rename arch/arm64/lib/crc-t10dif.c => lib/crc/arm64/crc-t10dif.h (70%)
rename {arch/arm64/lib => lib/crc/arm64}/crc32-core.S (100%)
rename arch/arm64/lib/crc32.c => lib/crc/arm64/crc32.h (81%)
rename lib/{ => crc}/crc-ccitt.c (98%)
rename lib/{ => crc}/crc-itu-t.c (100%)
rename lib/{crc-t10dif.c => crc/crc-t10dif-main.c} (78%)
rename lib/{ => crc}/crc16.c (100%)
rename lib/{crc32.c => crc/crc32-main.c} (73%)
rename lib/{ => crc}/crc4.c (100%)
rename lib/{crc64.c => crc/crc64-main.c} (66%)
rename lib/{ => crc}/crc7.c (100%)
rename lib/{ => crc}/crc8.c (100%)
rename lib/{ => crc}/gen_crc32table.c (95%)
rename lib/{ => crc}/gen_crc64table.c (81%)
rename arch/loongarch/lib/crc32-loongarch.c => lib/crc/loongarch/crc32.h (71%)
rename arch/mips/lib/crc32-mips.c => lib/crc/mips/crc32.h (82%)
rename arch/powerpc/lib/crc-t10dif.c => lib/crc/powerpc/crc-t10dif.h (75%)
rename {arch/powerpc/lib => lib/crc/powerpc}/crc-vpmsum-template.S (100%)
rename arch/powerpc/lib/crc32.c => lib/crc/powerpc/crc32.h (64%)
rename {arch/powerpc/lib => lib/crc/powerpc}/crc32c-vpmsum_asm.S (100%)
rename {arch/powerpc/lib => lib/crc/powerpc}/crct10dif-vpmsum_asm.S (100%)
rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul-consts.h (100%)
rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul-template.h (100%)
rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul.h (100%)
rename arch/riscv/lib/crc-t10dif.c => lib/crc/riscv/crc-t10dif.h (62%)
rename {arch/riscv/lib => lib/crc/riscv}/crc16_msb.c (100%)
rename arch/riscv/lib/crc32.c => lib/crc/riscv/crc32.h (66%)
rename {arch/riscv/lib => lib/crc/riscv}/crc32_lsb.c (100%)
rename {arch/riscv/lib => lib/crc/riscv}/crc32_msb.c (100%)
rename arch/riscv/lib/crc64.c => lib/crc/riscv/crc64.h (65%)
rename {arch/riscv/lib => lib/crc/riscv}/crc64_lsb.c (100%)
rename {arch/riscv/lib => lib/crc/riscv}/crc64_msb.c (100%)
rename {arch/s390/lib => lib/crc/s390}/crc32-vx.h (100%)
rename arch/s390/lib/crc32.c => lib/crc/s390/crc32.h (81%)
rename {arch/s390/lib => lib/crc/s390}/crc32be-vx.c (100%)
rename {arch/s390/lib => lib/crc/s390}/crc32le-vx.c (100%)
rename arch/sparc/lib/crc32.c => lib/crc/sparc/crc32.h (62%)
rename {arch/sparc/lib => lib/crc/sparc}/crc32c_asm.S (100%)
create mode 100644 lib/crc/tests/Makefile
rename lib/{ => crc}/tests/crc_kunit.c (100%)
rename {arch/x86/lib => lib/crc/x86}/crc-pclmul-consts.h (100%)
rename {arch/x86/lib => lib/crc/x86}/crc-pclmul-template.S (100%)
rename {arch/x86/lib => lib/crc/x86}/crc-pclmul-template.h (100%)
rename arch/x86/lib/crc-t10dif.c => lib/crc/x86/crc-t10dif.h (56%)
rename {arch/x86/lib => lib/crc/x86}/crc16-msb-pclmul.S (100%)
rename {arch/x86/lib => lib/crc/x86}/crc32-pclmul.S (100%)
rename arch/x86/lib/crc32.c => lib/crc/x86/crc32.h (76%)
rename {arch/x86/lib => lib/crc/x86}/crc32c-3way.S (100%)
rename {arch/x86/lib => lib/crc/x86}/crc64-pclmul.S (100%)
rename arch/x86/lib/crc64.c => lib/crc/x86/crc64.h (61%)
base-commit: cd2e103d57e5615f9bb027d772f93b9efd567224
--
2.49.0
^ permalink raw reply [flat|nested] 15+ messages in thread
* [PATCH 01/13] crypto/crc32: register only one shash_alg
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 02/13] crypto/crc32c: " Eric Biggers
` (12 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Stop unnecessarily registering a "crc32-generic" shash_alg when a
"crc32-$(ARCH)" shash_alg is registered too.
While every algorithm does need to have a generic implementation to
ensure uniformity of support across platforms, that doesn't mean that we
need to make the generic implementation available through crypto_shash
when an optimized implementation is also available.
Registering the generic shash_alg did allow users of the crypto_shash or
crypto_ahash APIs to request the generic implementation specifically,
instead of an optimized one. However, the only known use case for that
was the differential fuzz tests in crypto/testmgr.c. Equivalent test
coverage is now provided by crc_kunit.
Besides simplifying crypto/crc32.c, this change eliminates the need for
the library to provide crc32_le_base() as part of its interface. Later
patches will make crc32_le_base() be internal to the library.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
crypto/crc32.c | 69 ++++++++------------------------------------------
1 file changed, 11 insertions(+), 58 deletions(-)
diff --git a/crypto/crc32.c b/crypto/crc32.c
index cc371d42601fd..b61d5663d0bac 100644
--- a/crypto/crc32.c
+++ b/crypto/crc32.c
@@ -57,33 +57,16 @@ static int crc32_init(struct shash_desc *desc)
static int crc32_update(struct shash_desc *desc, const u8 *data,
unsigned int len)
{
u32 *crcp = shash_desc_ctx(desc);
- *crcp = crc32_le_base(*crcp, data, len);
- return 0;
-}
-
-static int crc32_update_arch(struct shash_desc *desc, const u8 *data,
- unsigned int len)
-{
- u32 *crcp = shash_desc_ctx(desc);
-
*crcp = crc32_le(*crcp, data, len);
return 0;
}
/* No final XOR 0xFFFFFFFF, like crc32_le */
-static int __crc32_finup(u32 *crcp, const u8 *data, unsigned int len,
- u8 *out)
-{
- put_unaligned_le32(crc32_le_base(*crcp, data, len), out);
- return 0;
-}
-
-static int __crc32_finup_arch(u32 *crcp, const u8 *data, unsigned int len,
- u8 *out)
+static int __crc32_finup(u32 *crcp, const u8 *data, unsigned int len, u8 *out)
{
put_unaligned_le32(crc32_le(*crcp, data, len), out);
return 0;
}
@@ -91,16 +74,10 @@ static int crc32_finup(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
return __crc32_finup(shash_desc_ctx(desc), data, len, out);
}
-static int crc32_finup_arch(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- return __crc32_finup_arch(shash_desc_ctx(desc), data, len, out);
-}
-
static int crc32_final(struct shash_desc *desc, u8 *out)
{
u32 *crcp = shash_desc_ctx(desc);
put_unaligned_le32(*crcp, out);
@@ -111,72 +88,48 @@ static int crc32_digest(struct shash_desc *desc, const u8 *data,
unsigned int len, u8 *out)
{
return __crc32_finup(crypto_shash_ctx(desc->tfm), data, len, out);
}
-static int crc32_digest_arch(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- return __crc32_finup_arch(crypto_shash_ctx(desc->tfm), data, len, out);
-}
-
-static struct shash_alg algs[] = {{
+static struct shash_alg alg = {
.setkey = crc32_setkey,
.init = crc32_init,
.update = crc32_update,
.final = crc32_final,
.finup = crc32_finup,
.digest = crc32_digest,
.descsize = sizeof(u32),
.digestsize = CHKSUM_DIGEST_SIZE,
.base.cra_name = "crc32",
- .base.cra_driver_name = "crc32-generic",
.base.cra_priority = 100,
.base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY,
.base.cra_blocksize = CHKSUM_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(u32),
.base.cra_module = THIS_MODULE,
.base.cra_init = crc32_cra_init,
-}, {
- .setkey = crc32_setkey,
- .init = crc32_init,
- .update = crc32_update_arch,
- .final = crc32_final,
- .finup = crc32_finup_arch,
- .digest = crc32_digest_arch,
- .descsize = sizeof(u32),
- .digestsize = CHKSUM_DIGEST_SIZE,
-
- .base.cra_name = "crc32",
- .base.cra_driver_name = "crc32-" __stringify(ARCH),
- .base.cra_priority = 150,
- .base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY,
- .base.cra_blocksize = CHKSUM_BLOCK_SIZE,
- .base.cra_ctxsize = sizeof(u32),
- .base.cra_module = THIS_MODULE,
- .base.cra_init = crc32_cra_init,
-}};
-
-static int num_algs;
+};
static int __init crc32_mod_init(void)
{
- /* register the arch flavor only if it differs from the generic one */
- num_algs = 1 + ((crc32_optimizations() & CRC32_LE_OPTIMIZATION) != 0);
+ const char *driver_name =
+ (crc32_optimizations() & CRC32_LE_OPTIMIZATION) ?
+ "crc32-" __stringify(ARCH) :
+ "crc32-generic";
+
+ strscpy(alg.base.cra_driver_name, driver_name, CRYPTO_MAX_ALG_NAME);
- return crypto_register_shashes(algs, num_algs);
+ return crypto_register_shash(&alg);
}
static void __exit crc32_mod_fini(void)
{
- crypto_unregister_shashes(algs, num_algs);
+ crypto_unregister_shash(&alg);
}
module_init(crc32_mod_init);
module_exit(crc32_mod_fini);
MODULE_AUTHOR("Alexander Boyko <alexander_boyko@xyratex.com>");
MODULE_DESCRIPTION("CRC32 calculations wrapper for lib/crc32");
MODULE_LICENSE("GPL");
MODULE_ALIAS_CRYPTO("crc32");
-MODULE_ALIAS_CRYPTO("crc32-generic");
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 02/13] crypto/crc32c: register only one shash_alg
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
2025-06-01 22:44 ` [PATCH 01/13] crypto/crc32: register only one shash_alg Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 03/13] lib/crc: move files into lib/crc/ Eric Biggers
` (11 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Stop unnecessarily registering a "crc32c-generic" shash_alg when a
"crc32c-$(ARCH)" shash_alg is registered too.
While every algorithm does need to have a generic implementation to
ensure uniformity of support across platforms, that doesn't mean that we
need to make the generic implementation available through crypto_shash
when an optimized implementation is also available.
Registering the generic shash_alg did allow users of the crypto_shash or
crypto_ahash APIs to request the generic implementation specifically,
instead of an optimized one. However, the only known use case for that
was the differential fuzz tests in crypto/testmgr.c. Equivalent test
coverage is now provided by crc_kunit.
Besides simplifying crypto/crc32c.c, this change eliminates the need for
the library to provide crc32c_base() as part of its interface. Later
patches will make crc32c_base() be internal to the library.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
crypto/crc32c.c | 70 +++++++------------------------------------------
1 file changed, 10 insertions(+), 60 deletions(-)
diff --git a/crypto/crc32c.c b/crypto/crc32c.c
index e5377898414a2..e160695682f16 100644
--- a/crypto/crc32c.c
+++ b/crypto/crc32c.c
@@ -83,19 +83,10 @@ static int chksum_setkey(struct crypto_shash *tfm, const u8 *key,
static int chksum_update(struct shash_desc *desc, const u8 *data,
unsigned int length)
{
struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
- ctx->crc = crc32c_base(ctx->crc, data, length);
- return 0;
-}
-
-static int chksum_update_arch(struct shash_desc *desc, const u8 *data,
- unsigned int length)
-{
- struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
-
ctx->crc = crc32c(ctx->crc, data, length);
return 0;
}
static int chksum_final(struct shash_desc *desc, u8 *out)
@@ -105,17 +96,10 @@ static int chksum_final(struct shash_desc *desc, u8 *out)
put_unaligned_le32(~ctx->crc, out);
return 0;
}
static int __chksum_finup(u32 *crcp, const u8 *data, unsigned int len, u8 *out)
-{
- put_unaligned_le32(~crc32c_base(*crcp, data, len), out);
- return 0;
-}
-
-static int __chksum_finup_arch(u32 *crcp, const u8 *data, unsigned int len,
- u8 *out)
{
put_unaligned_le32(~crc32c(*crcp, data, len), out);
return 0;
}
@@ -125,98 +109,64 @@ static int chksum_finup(struct shash_desc *desc, const u8 *data,
struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
return __chksum_finup(&ctx->crc, data, len, out);
}
-static int chksum_finup_arch(struct shash_desc *desc, const u8 *data,
- unsigned int len, u8 *out)
-{
- struct chksum_desc_ctx *ctx = shash_desc_ctx(desc);
-
- return __chksum_finup_arch(&ctx->crc, data, len, out);
-}
-
static int chksum_digest(struct shash_desc *desc, const u8 *data,
unsigned int length, u8 *out)
{
struct chksum_ctx *mctx = crypto_shash_ctx(desc->tfm);
return __chksum_finup(&mctx->key, data, length, out);
}
-static int chksum_digest_arch(struct shash_desc *desc, const u8 *data,
- unsigned int length, u8 *out)
-{
- struct chksum_ctx *mctx = crypto_shash_ctx(desc->tfm);
-
- return __chksum_finup_arch(&mctx->key, data, length, out);
-}
-
static int crc32c_cra_init(struct crypto_tfm *tfm)
{
struct chksum_ctx *mctx = crypto_tfm_ctx(tfm);
mctx->key = ~0;
return 0;
}
-static struct shash_alg algs[] = {{
+static struct shash_alg alg = {
.digestsize = CHKSUM_DIGEST_SIZE,
.setkey = chksum_setkey,
.init = chksum_init,
.update = chksum_update,
.final = chksum_final,
.finup = chksum_finup,
.digest = chksum_digest,
.descsize = sizeof(struct chksum_desc_ctx),
.base.cra_name = "crc32c",
- .base.cra_driver_name = "crc32c-generic",
.base.cra_priority = 100,
.base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY,
.base.cra_blocksize = CHKSUM_BLOCK_SIZE,
.base.cra_ctxsize = sizeof(struct chksum_ctx),
.base.cra_module = THIS_MODULE,
.base.cra_init = crc32c_cra_init,
-}, {
- .digestsize = CHKSUM_DIGEST_SIZE,
- .setkey = chksum_setkey,
- .init = chksum_init,
- .update = chksum_update_arch,
- .final = chksum_final,
- .finup = chksum_finup_arch,
- .digest = chksum_digest_arch,
- .descsize = sizeof(struct chksum_desc_ctx),
-
- .base.cra_name = "crc32c",
- .base.cra_driver_name = "crc32c-" __stringify(ARCH),
- .base.cra_priority = 150,
- .base.cra_flags = CRYPTO_ALG_OPTIONAL_KEY,
- .base.cra_blocksize = CHKSUM_BLOCK_SIZE,
- .base.cra_ctxsize = sizeof(struct chksum_ctx),
- .base.cra_module = THIS_MODULE,
- .base.cra_init = crc32c_cra_init,
-}};
-
-static int num_algs;
+};
static int __init crc32c_mod_init(void)
{
- /* register the arch flavor only if it differs from the generic one */
- num_algs = 1 + ((crc32_optimizations() & CRC32C_OPTIMIZATION) != 0);
+ const char *driver_name =
+ (crc32_optimizations() & CRC32C_OPTIMIZATION) ?
+ "crc32c-" __stringify(ARCH) :
+ "crc32c-generic";
+
+ strscpy(alg.base.cra_driver_name, driver_name, CRYPTO_MAX_ALG_NAME);
- return crypto_register_shashes(algs, num_algs);
+ return crypto_register_shash(&alg);
}
static void __exit crc32c_mod_fini(void)
{
- crypto_unregister_shashes(algs, num_algs);
+ crypto_unregister_shash(&alg);
}
module_init(crc32c_mod_init);
module_exit(crc32c_mod_fini);
MODULE_AUTHOR("Clay Haapala <chaapala@cisco.com>");
MODULE_DESCRIPTION("CRC32c (Castagnoli) calculations wrapper for lib/crc32c");
MODULE_LICENSE("GPL");
MODULE_ALIAS_CRYPTO("crc32c");
-MODULE_ALIAS_CRYPTO("crc32c-generic");
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 03/13] lib/crc: move files into lib/crc/
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
2025-06-01 22:44 ` [PATCH 01/13] crypto/crc32: register only one shash_alg Eric Biggers
2025-06-01 22:44 ` [PATCH 02/13] crypto/crc32c: " Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 04/13] lib/crc: prepare for arch-optimized code in subdirs of lib/crc/ Eric Biggers
` (10 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Move all CRC files in lib/ into a subdirectory lib/crc/ to keep them
from cluttering up the main lib/ directory.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
Documentation/core-api/kernel-api.rst | 14 ++--
MAINTAINERS | 3 +-
include/linux/crc64.h | 3 -
lib/Kconfig | 87 +-------------------
lib/Kconfig.debug | 21 -----
lib/Makefile | 32 +-------
lib/crc/.gitignore | 5 ++
lib/crc/Kconfig | 111 ++++++++++++++++++++++++++
lib/crc/Makefile | 32 ++++++++
lib/{ => crc}/crc-ccitt.c | 3 -
lib/{ => crc}/crc-itu-t.c | 0
lib/{ => crc}/crc-t10dif.c | 0
lib/{ => crc}/crc16.c | 0
lib/{ => crc}/crc32.c | 0
lib/{ => crc}/crc4.c | 0
lib/{ => crc}/crc64.c | 0
lib/{ => crc}/crc7.c | 0
lib/{ => crc}/crc8.c | 0
lib/{ => crc}/gen_crc32table.c | 4 +-
lib/{ => crc}/gen_crc64table.c | 11 +--
lib/crc/tests/Makefile | 2 +
lib/{ => crc}/tests/crc_kunit.c | 0
lib/tests/Makefile | 1 -
23 files changed, 164 insertions(+), 165 deletions(-)
create mode 100644 lib/crc/.gitignore
create mode 100644 lib/crc/Kconfig
create mode 100644 lib/crc/Makefile
rename lib/{ => crc}/crc-ccitt.c (98%)
rename lib/{ => crc}/crc-itu-t.c (100%)
rename lib/{ => crc}/crc-t10dif.c (100%)
rename lib/{ => crc}/crc16.c (100%)
rename lib/{ => crc}/crc32.c (100%)
rename lib/{ => crc}/crc4.c (100%)
rename lib/{ => crc}/crc64.c (100%)
rename lib/{ => crc}/crc7.c (100%)
rename lib/{ => crc}/crc8.c (100%)
rename lib/{ => crc}/gen_crc32table.c (95%)
rename lib/{ => crc}/gen_crc64table.c (81%)
create mode 100644 lib/crc/tests/Makefile
rename lib/{ => crc}/tests/crc_kunit.c (100%)
diff --git a/Documentation/core-api/kernel-api.rst b/Documentation/core-api/kernel-api.rst
index ae92a2571388a..c4642d9f13a9c 100644
--- a/Documentation/core-api/kernel-api.rst
+++ b/Documentation/core-api/kernel-api.rst
@@ -134,28 +134,28 @@ Arithmetic Overflow Checking
:internal:
CRC Functions
-------------
-.. kernel-doc:: lib/crc4.c
+.. kernel-doc:: lib/crc/crc4.c
:export:
-.. kernel-doc:: lib/crc7.c
+.. kernel-doc:: lib/crc/crc7.c
:export:
-.. kernel-doc:: lib/crc8.c
+.. kernel-doc:: lib/crc/crc8.c
:export:
-.. kernel-doc:: lib/crc16.c
+.. kernel-doc:: lib/crc/crc16.c
:export:
-.. kernel-doc:: lib/crc32.c
+.. kernel-doc:: lib/crc/crc32.c
-.. kernel-doc:: lib/crc-ccitt.c
+.. kernel-doc:: lib/crc/crc-ccitt.c
:export:
-.. kernel-doc:: lib/crc-itu-t.c
+.. kernel-doc:: lib/crc/crc-itu-t.c
:export:
Base 2 log and power Functions
------------------------------
diff --git a/MAINTAINERS b/MAINTAINERS
index 75da23034e18d..724f46458a480 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6335,12 +6335,11 @@ L: linux-crypto@vger.kernel.org
S: Maintained
T: git https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git crc-next
F: Documentation/staging/crc*
F: arch/*/lib/crc*
F: include/linux/crc*
-F: lib/crc*
-F: lib/tests/crc_kunit.c
+F: lib/crc/
F: scripts/gen-crc-consts.py
CREATIVE SB0540
M: Bastien Nocera <hadess@hadess.net>
L: linux-input@vger.kernel.org
diff --git a/include/linux/crc64.h b/include/linux/crc64.h
index 41de30b907dff..b6aa290a79312 100644
--- a/include/linux/crc64.h
+++ b/include/linux/crc64.h
@@ -1,9 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0 */
-/*
- * See lib/crc64.c for the related specification and polynomial arithmetic.
- */
#ifndef _LINUX_CRC64_H
#define _LINUX_CRC64_H
#include <linux/types.h>
diff --git a/lib/Kconfig b/lib/Kconfig
index 6c1b8f1842678..3db88ab0d2b4c 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -134,98 +134,13 @@ config TRACE_MMIO_ACCESS
depends on TRACING && ARCH_HAVE_TRACE_MMIO_ACCESS
help
Create tracepoints for MMIO read/write operations. These trace events
can be used for logging all MMIO read/write operations.
+source "lib/crc/Kconfig"
source "lib/crypto/Kconfig"
-config CRC_CCITT
- tristate
- help
- The CRC-CCITT library functions. Select this if your module uses any
- of the functions from <linux/crc-ccitt.h>.
-
-config CRC16
- tristate
- help
- The CRC16 library functions. Select this if your module uses any of
- the functions from <linux/crc16.h>.
-
-config CRC_T10DIF
- tristate
- help
- The CRC-T10DIF library functions. Select this if your module uses
- any of the functions from <linux/crc-t10dif.h>.
-
-config ARCH_HAS_CRC_T10DIF
- bool
-
-config CRC_T10DIF_ARCH
- tristate
- default CRC_T10DIF if ARCH_HAS_CRC_T10DIF && CRC_OPTIMIZATIONS
-
-config CRC_ITU_T
- tristate
- help
- The CRC-ITU-T library functions. Select this if your module uses
- any of the functions from <linux/crc-itu-t.h>.
-
-config CRC32
- tristate
- select BITREVERSE
- help
- The CRC32 library functions. Select this if your module uses any of
- the functions from <linux/crc32.h> or <linux/crc32c.h>.
-
-config ARCH_HAS_CRC32
- bool
-
-config CRC32_ARCH
- tristate
- default CRC32 if ARCH_HAS_CRC32 && CRC_OPTIMIZATIONS
-
-config CRC64
- tristate
- help
- The CRC64 library functions. Select this if your module uses any of
- the functions from <linux/crc64.h>.
-
-config ARCH_HAS_CRC64
- bool
-
-config CRC64_ARCH
- tristate
- default CRC64 if ARCH_HAS_CRC64 && CRC_OPTIMIZATIONS
-
-config CRC4
- tristate
- help
- The CRC4 library functions. Select this if your module uses any of
- the functions from <linux/crc4.h>.
-
-config CRC7
- tristate
- help
- The CRC7 library functions. Select this if your module uses any of
- the functions from <linux/crc7.h>.
-
-config CRC8
- tristate
- help
- The CRC8 library functions. Select this if your module uses any of
- the functions from <linux/crc8.h>.
-
-config CRC_OPTIMIZATIONS
- bool "Enable optimized CRC implementations" if EXPERT
- default y
- help
- Disabling this option reduces code size slightly by disabling the
- architecture-optimized implementations of any CRC variants that are
- enabled. CRC checksumming performance may get much slower.
-
- Keep this enabled unless you're really trying to minimize the size of
- the kernel.
config XXHASH
tristate
config AUDIT_GENERIC
diff --git a/lib/Kconfig.debug b/lib/Kconfig.debug
index ebe33181b6e6e..3fda96761adbc 100644
--- a/lib/Kconfig.debug
+++ b/lib/Kconfig.debug
@@ -2899,31 +2899,10 @@ config HW_BREAKPOINT_KUNIT_TEST
help
Tests for hw_breakpoint constraints accounting.
If unsure, say N.
-config CRC_KUNIT_TEST
- tristate "KUnit tests for CRC functions" if !KUNIT_ALL_TESTS
- depends on KUNIT
- default KUNIT_ALL_TESTS
- select CRC7
- select CRC16
- select CRC_T10DIF
- select CRC32
- select CRC64
- help
- Unit tests for the CRC library functions.
-
- This is intended to help people writing architecture-specific
- optimized versions. If unsure, say N.
-
-config CRC_BENCHMARK
- bool "Benchmark for the CRC functions"
- depends on CRC_KUNIT_TEST
- help
- Include benchmarks in the KUnit test suite for the CRC functions.
-
config SIPHASH_KUNIT_TEST
tristate "Perform selftest on siphash functions" if !KUNIT_ALL_TESTS
depends on KUNIT
default KUNIT_ALL_TESTS
help
diff --git a/lib/Makefile b/lib/Makefile
index c38582f187dd8..14a5928bb57f5 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -120,11 +120,11 @@ CFLAGS_kobject_uevent.o += -DDEBUG
endif
obj-$(CONFIG_DEBUG_INFO_REDUCED) += debug_info.o
CFLAGS_debug_info.o += $(call cc-option, -femit-struct-debug-detailed=any)
-obj-y += math/ crypto/ tests/ vdso/
+obj-y += math/ crc/ crypto/ tests/ vdso/
obj-$(CONFIG_GENERIC_IOMAP) += iomap.o
obj-$(CONFIG_HAS_IOMEM) += iomap_copy.o devres.o
obj-$(CONFIG_CHECK_SIGNATURE) += check_signature.o
obj-$(CONFIG_DEBUG_LOCKING_API_SELFTESTS) += locking-selftest.o
@@ -146,19 +146,10 @@ obj-$(CONFIG_DEBUG_OBJECTS) += debugobjects.o
obj-$(CONFIG_BITREVERSE) += bitrev.o
obj-$(CONFIG_LINEAR_RANGES) += linear_ranges.o
obj-$(CONFIG_PACKING) += packing.o
obj-$(CONFIG_PACKING_KUNIT_TEST) += packing_test.o
-obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o
-obj-$(CONFIG_CRC16) += crc16.o
-obj-$(CONFIG_CRC_T10DIF)+= crc-t10dif.o
-obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o
-obj-$(CONFIG_CRC32) += crc32.o
-obj-$(CONFIG_CRC64) += crc64.o
-obj-$(CONFIG_CRC4) += crc4.o
-obj-$(CONFIG_CRC7) += crc7.o
-obj-$(CONFIG_CRC8) += crc8.o
obj-$(CONFIG_XXHASH) += xxhash.o
obj-$(CONFIG_GENERIC_ALLOCATOR) += genalloc.o
obj-$(CONFIG_842_COMPRESS) += 842/
obj-$(CONFIG_842_DECOMPRESS) += 842/
@@ -292,31 +283,10 @@ obj-$(CONFIG_PERCPU_TEST) += percpu_test.o
obj-$(CONFIG_ASN1) += asn1_decoder.o
obj-$(CONFIG_ASN1_ENCODER) += asn1_encoder.o
obj-$(CONFIG_FONT_SUPPORT) += fonts/
-hostprogs := gen_crc32table
-hostprogs += gen_crc64table
-clean-files := crc32table.h
-clean-files += crc64table.h
-
-$(obj)/crc32.o: $(obj)/crc32table.h
-
-quiet_cmd_crc32 = GEN $@
- cmd_crc32 = $< > $@
-
-$(obj)/crc32table.h: $(obj)/gen_crc32table
- $(call cmd,crc32)
-
-$(obj)/crc64.o: $(obj)/crc64table.h
-
-quiet_cmd_crc64 = GEN $@
- cmd_crc64 = $< > $@
-
-$(obj)/crc64table.h: $(obj)/gen_crc64table
- $(call cmd,crc64)
-
#
# Build a fast OID lookip registry from include/linux/oid_registry.h
#
obj-$(CONFIG_OID_REGISTRY) += oid_registry.o
diff --git a/lib/crc/.gitignore b/lib/crc/.gitignore
new file mode 100644
index 0000000000000..a9e48103c9fb1
--- /dev/null
+++ b/lib/crc/.gitignore
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0-only
+/crc32table.h
+/crc64table.h
+/gen_crc32table
+/gen_crc64table
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
new file mode 100644
index 0000000000000..e0e7168b74c75
--- /dev/null
+++ b/lib/crc/Kconfig
@@ -0,0 +1,111 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+# Kconfig for the kernel's cyclic redundancy check (CRC) library code
+
+config CRC4
+ tristate
+ help
+ The CRC4 library functions. Select this if your module uses any of
+ the functions from <linux/crc4.h>.
+
+config CRC7
+ tristate
+ help
+ The CRC7 library functions. Select this if your module uses any of
+ the functions from <linux/crc7.h>.
+
+config CRC8
+ tristate
+ help
+ The CRC8 library functions. Select this if your module uses any of
+ the functions from <linux/crc8.h>.
+
+config CRC16
+ tristate
+ help
+ The CRC16 library functions. Select this if your module uses any of
+ the functions from <linux/crc16.h>.
+
+config CRC_CCITT
+ tristate
+ help
+ The CRC-CCITT library functions. Select this if your module uses any
+ of the functions from <linux/crc-ccitt.h>.
+
+config CRC_ITU_T
+ tristate
+ help
+ The CRC-ITU-T library functions. Select this if your module uses
+ any of the functions from <linux/crc-itu-t.h>.
+
+config CRC_T10DIF
+ tristate
+ help
+ The CRC-T10DIF library functions. Select this if your module uses
+ any of the functions from <linux/crc-t10dif.h>.
+
+config ARCH_HAS_CRC_T10DIF
+ bool
+
+config CRC_T10DIF_ARCH
+ tristate
+ default CRC_T10DIF if ARCH_HAS_CRC_T10DIF && CRC_OPTIMIZATIONS
+
+config CRC32
+ tristate
+ select BITREVERSE
+ help
+ The CRC32 library functions. Select this if your module uses any of
+ the functions from <linux/crc32.h> or <linux/crc32c.h>.
+
+config ARCH_HAS_CRC32
+ bool
+
+config CRC32_ARCH
+ tristate
+ default CRC32 if ARCH_HAS_CRC32 && CRC_OPTIMIZATIONS
+
+config CRC64
+ tristate
+ help
+ The CRC64 library functions. Select this if your module uses any of
+ the functions from <linux/crc64.h>.
+
+config ARCH_HAS_CRC64
+ bool
+
+config CRC64_ARCH
+ tristate
+ default CRC64 if ARCH_HAS_CRC64 && CRC_OPTIMIZATIONS
+
+config CRC_OPTIMIZATIONS
+ bool "Enable optimized CRC implementations" if EXPERT
+ default y
+ help
+ Disabling this option reduces code size slightly by disabling the
+ architecture-optimized implementations of any CRC variants that are
+ enabled. CRC checksumming performance may get much slower.
+
+ Keep this enabled unless you're really trying to minimize the size of
+ the kernel.
+
+config CRC_KUNIT_TEST
+ tristate "KUnit tests for CRC functions" if !KUNIT_ALL_TESTS
+ depends on KUNIT
+ default KUNIT_ALL_TESTS
+ select CRC7
+ select CRC16
+ select CRC_T10DIF
+ select CRC32
+ select CRC64
+ help
+ Unit tests for the CRC library functions.
+
+ This is intended to help people writing architecture-specific
+ optimized versions. If unsure, say N.
+
+config CRC_BENCHMARK
+ bool "Benchmark for the CRC functions"
+ depends on CRC_KUNIT_TEST
+ help
+ Include benchmarks in the KUnit test suite for the CRC functions.
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
new file mode 100644
index 0000000000000..ff4c30dda4528
--- /dev/null
+++ b/lib/crc/Makefile
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: GPL-2.0-only
+
+# Makefile for the kernel's cyclic redundancy check (CRC) library code
+
+obj-$(CONFIG_CRC4) += crc4.o
+obj-$(CONFIG_CRC7) += crc7.o
+obj-$(CONFIG_CRC8) += crc8.o
+obj-$(CONFIG_CRC16) += crc16.o
+obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o
+obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o
+obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o
+obj-$(CONFIG_CRC32) += crc32.o
+obj-$(CONFIG_CRC64) += crc64.o
+obj-y += tests/
+
+hostprogs := gen_crc32table gen_crc64table
+clean-files := crc32table.h crc64table.h
+
+$(obj)/crc32.o: $(obj)/crc32table.h
+$(obj)/crc64.o: $(obj)/crc64table.h
+
+quiet_cmd_crc32 = GEN $@
+ cmd_crc32 = $< > $@
+
+quiet_cmd_crc64 = GEN $@
+ cmd_crc64 = $< > $@
+
+$(obj)/crc32table.h: $(obj)/gen_crc32table
+ $(call cmd,crc32)
+
+$(obj)/crc64table.h: $(obj)/gen_crc64table
+ $(call cmd,crc64)
diff --git a/lib/crc-ccitt.c b/lib/crc/crc-ccitt.c
similarity index 98%
rename from lib/crc-ccitt.c
rename to lib/crc/crc-ccitt.c
index 9cddf35d3b66e..8d2bc419230b3 100644
--- a/lib/crc-ccitt.c
+++ b/lib/crc/crc-ccitt.c
@@ -1,9 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
-/*
- * linux/lib/crc-ccitt.c
- */
#include <linux/types.h>
#include <linux/module.h>
#include <linux/crc-ccitt.h>
diff --git a/lib/crc-itu-t.c b/lib/crc/crc-itu-t.c
similarity index 100%
rename from lib/crc-itu-t.c
rename to lib/crc/crc-itu-t.c
diff --git a/lib/crc-t10dif.c b/lib/crc/crc-t10dif.c
similarity index 100%
rename from lib/crc-t10dif.c
rename to lib/crc/crc-t10dif.c
diff --git a/lib/crc16.c b/lib/crc/crc16.c
similarity index 100%
rename from lib/crc16.c
rename to lib/crc/crc16.c
diff --git a/lib/crc32.c b/lib/crc/crc32.c
similarity index 100%
rename from lib/crc32.c
rename to lib/crc/crc32.c
diff --git a/lib/crc4.c b/lib/crc/crc4.c
similarity index 100%
rename from lib/crc4.c
rename to lib/crc/crc4.c
diff --git a/lib/crc64.c b/lib/crc/crc64.c
similarity index 100%
rename from lib/crc64.c
rename to lib/crc/crc64.c
diff --git a/lib/crc7.c b/lib/crc/crc7.c
similarity index 100%
rename from lib/crc7.c
rename to lib/crc/crc7.c
diff --git a/lib/crc8.c b/lib/crc/crc8.c
similarity index 100%
rename from lib/crc8.c
rename to lib/crc/crc8.c
diff --git a/lib/gen_crc32table.c b/lib/crc/gen_crc32table.c
similarity index 95%
rename from lib/gen_crc32table.c
rename to lib/crc/gen_crc32table.c
index 6d03425b849e4..9a7f31658e355 100644
--- a/lib/gen_crc32table.c
+++ b/lib/crc/gen_crc32table.c
@@ -1,9 +1,9 @@
// SPDX-License-Identifier: GPL-2.0
#include <stdio.h>
-#include "../include/linux/crc32poly.h"
-#include "../include/generated/autoconf.h"
+#include "../../include/linux/crc32poly.h"
+#include "../../include/generated/autoconf.h"
#include <inttypes.h>
static uint32_t crc32table_le[256];
static uint32_t crc32table_be[256];
static uint32_t crc32ctable_le[256];
diff --git a/lib/gen_crc64table.c b/lib/crc/gen_crc64table.c
similarity index 81%
rename from lib/gen_crc64table.c
rename to lib/crc/gen_crc64table.c
index e05a4230a0a08..f2be9f62bab71 100644
--- a/lib/gen_crc64table.c
+++ b/lib/crc/gen_crc64table.c
@@ -1,16 +1,9 @@
// SPDX-License-Identifier: GPL-2.0
/*
- * Generate lookup table for the table-driven CRC64 calculation.
- *
- * gen_crc64table is executed in kernel build time and generates
- * lib/crc64table.h. This header is included by lib/crc64.c for
- * the table-driven CRC64 calculation.
- *
- * See lib/crc64.c for more information about which specification
- * and polynomial arithmetic that gen_crc64table.c follows to
- * generate the lookup table.
+ * This host program runs at kernel build time and generates the lookup tables
+ * used by the generic CRC64 code.
*
* Copyright 2018 SUSE Linux.
* Author: Coly Li <colyli@suse.de>
*/
#include <inttypes.h>
diff --git a/lib/crc/tests/Makefile b/lib/crc/tests/Makefile
new file mode 100644
index 0000000000000..65f63c318ef5e
--- /dev/null
+++ b/lib/crc/tests/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0-only
+obj-$(CONFIG_CRC_KUNIT_TEST) += crc_kunit.o
diff --git a/lib/tests/crc_kunit.c b/lib/crc/tests/crc_kunit.c
similarity index 100%
rename from lib/tests/crc_kunit.c
rename to lib/crc/tests/crc_kunit.c
diff --git a/lib/tests/Makefile b/lib/tests/Makefile
index 56d6450144828..741d3ac2cba25 100644
--- a/lib/tests/Makefile
+++ b/lib/tests/Makefile
@@ -8,11 +8,10 @@ obj-$(CONFIG_BITFIELD_KUNIT) += bitfield_kunit.o
obj-$(CONFIG_BITS_TEST) += test_bits.o
obj-$(CONFIG_BLACKHOLE_DEV_KUNIT_TEST) += blackhole_dev_kunit.o
obj-$(CONFIG_CHECKSUM_KUNIT) += checksum_kunit.o
obj-$(CONFIG_CMDLINE_KUNIT_TEST) += cmdline_kunit.o
obj-$(CONFIG_CPUMASK_KUNIT_TEST) += cpumask_kunit.o
-obj-$(CONFIG_CRC_KUNIT_TEST) += crc_kunit.o
CFLAGS_fortify_kunit.o += $(call cc-disable-warning, unsequenced)
CFLAGS_fortify_kunit.o += $(call cc-disable-warning, stringop-overread)
CFLAGS_fortify_kunit.o += $(call cc-disable-warning, stringop-truncation)
CFLAGS_fortify_kunit.o += $(DISABLE_STRUCTLEAK_PLUGIN)
obj-$(CONFIG_FORTIFY_KUNIT_TEST) += fortify_kunit.o
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 04/13] lib/crc: prepare for arch-optimized code in subdirs of lib/crc/
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
` (2 preceding siblings ...)
2025-06-01 22:44 ` [PATCH 03/13] lib/crc: move files into lib/crc/ Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 05/13] lib/crc/arm: migrate arm-optimized CRC code into lib/crc/ Eric Biggers
` (9 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Rework how lib/crc/ supports arch-optimized code. First, instead of the
arch-optimized code being in arch/$(SRCARCH)/lib/, it will now be in
lib/crc/$(SRCARCH)/. Second, the API functions (e.g. crc32c()),
arch-optimized functions (e.g. crc32c_arch()), and generic functions
(e.g. crc32c_base()) will now be part of a single module for each CRC
type, allowing better inlining and dead code elimination. The second
change is made possible by the first.
As an example, consider CONFIG_CRC32=m on x86. We'll now have just
crc32.ko instead of both crc32-x86.ko and crc32.ko. The two modules
were already coupled together and always both got loaded together via
direct symbol dependency, so the separation provided no benefit.
Note: later I'd like to apply the same design to lib/crypto/ too, where
often the API functions are out-of-line so this will work even better.
In those cases, for each algorithm we currently have 3 modules all
coupled together, e.g. libsha256.ko, libsha256-generic.ko, and
sha256-x86.ko. We should have just one, inline things properly, and
rely on the compiler's dead code elimination to decide the inclusion of
the generic code instead of manually setting it via kconfig.
Having arch-specific code outside arch/ was somewhat controversial when
Zinc proposed it back in 2018. But I don't think the concerns are
warranted. It's better from a technical perspective, as it enables the
improvements mentioned above. This model is already successfully used
in other places in the kernel such as lib/raid6/. The community of each
architecture still remains free to work on the code, even if it's not in
arch/. At the time there was also a desire to put the library code in
the same files as the old-school crypto API, but that was a mistake; now
that the library is separate, that's no longer a constraint either.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
Documentation/core-api/kernel-api.rst | 2 +-
MAINTAINERS | 1 -
include/linux/crc-t10dif.h | 10 +--
include/linux/crc32.h | 30 +-------
include/linux/crc64.h | 19 +----
lib/crc/Kconfig | 18 ++---
lib/crc/Makefile | 19 ++++-
lib/crc/{crc-t10dif.c => crc-t10dif-main.c} | 37 ++++++++--
lib/crc/{crc32.c => crc32-main.c} | 77 +++++++++++++++++----
lib/crc/{crc64.c => crc64-main.c} | 47 +++++++++++--
10 files changed, 164 insertions(+), 96 deletions(-)
rename lib/crc/{crc-t10dif.c => crc-t10dif-main.c} (78%)
rename lib/crc/{crc32.c => crc32-main.c} (73%)
rename lib/crc/{crc64.c => crc64-main.c} (66%)
diff --git a/Documentation/core-api/kernel-api.rst b/Documentation/core-api/kernel-api.rst
index c4642d9f13a9c..9c8370891a39b 100644
--- a/Documentation/core-api/kernel-api.rst
+++ b/Documentation/core-api/kernel-api.rst
@@ -146,11 +146,11 @@ CRC Functions
:export:
.. kernel-doc:: lib/crc/crc16.c
:export:
-.. kernel-doc:: lib/crc/crc32.c
+.. kernel-doc:: lib/crc/crc32-main.c
.. kernel-doc:: lib/crc/crc-ccitt.c
:export:
.. kernel-doc:: lib/crc/crc-itu-t.c
diff --git a/MAINTAINERS b/MAINTAINERS
index 724f46458a480..2aefd2bdbded2 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -6333,11 +6333,10 @@ M: Eric Biggers <ebiggers@kernel.org>
R: Ard Biesheuvel <ardb@kernel.org>
L: linux-crypto@vger.kernel.org
S: Maintained
T: git https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git crc-next
F: Documentation/staging/crc*
-F: arch/*/lib/crc*
F: include/linux/crc*
F: lib/crc/
F: scripts/gen-crc-consts.py
CREATIVE SB0540
diff --git a/include/linux/crc-t10dif.h b/include/linux/crc-t10dif.h
index a559fdff3f7e2..ecc8bc2dd7f4c 100644
--- a/include/linux/crc-t10dif.h
+++ b/include/linux/crc-t10dif.h
@@ -2,19 +2,11 @@
#ifndef _LINUX_CRC_T10DIF_H
#define _LINUX_CRC_T10DIF_H
#include <linux/types.h>
-u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len);
-u16 crc_t10dif_generic(u16 crc, const u8 *p, size_t len);
-
-static inline u16 crc_t10dif_update(u16 crc, const u8 *p, size_t len)
-{
- if (IS_ENABLED(CONFIG_CRC_T10DIF_ARCH))
- return crc_t10dif_arch(crc, p, len);
- return crc_t10dif_generic(crc, p, len);
-}
+u16 crc_t10dif_update(u16 crc, const u8 *p, size_t len);
static inline u16 crc_t10dif(const u8 *p, size_t len)
{
return crc_t10dif_update(0, p, len);
}
diff --git a/include/linux/crc32.h b/include/linux/crc32.h
index 8c1883b81b420..87b71bff5951b 100644
--- a/include/linux/crc32.h
+++ b/include/linux/crc32.h
@@ -3,37 +3,13 @@
#define _LINUX_CRC32_H
#include <linux/types.h>
#include <linux/bitrev.h>
-u32 crc32_le_arch(u32 crc, const u8 *p, size_t len);
-u32 crc32_le_base(u32 crc, const u8 *p, size_t len);
-u32 crc32_be_arch(u32 crc, const u8 *p, size_t len);
-u32 crc32_be_base(u32 crc, const u8 *p, size_t len);
-u32 crc32c_arch(u32 crc, const u8 *p, size_t len);
-u32 crc32c_base(u32 crc, const u8 *p, size_t len);
-
-static inline u32 crc32_le(u32 crc, const void *p, size_t len)
-{
- if (IS_ENABLED(CONFIG_CRC32_ARCH))
- return crc32_le_arch(crc, p, len);
- return crc32_le_base(crc, p, len);
-}
-
-static inline u32 crc32_be(u32 crc, const void *p, size_t len)
-{
- if (IS_ENABLED(CONFIG_CRC32_ARCH))
- return crc32_be_arch(crc, p, len);
- return crc32_be_base(crc, p, len);
-}
-
-static inline u32 crc32c(u32 crc, const void *p, size_t len)
-{
- if (IS_ENABLED(CONFIG_CRC32_ARCH))
- return crc32c_arch(crc, p, len);
- return crc32c_base(crc, p, len);
-}
+u32 crc32_le(u32 crc, const void *p, size_t len);
+u32 crc32_be(u32 crc, const void *p, size_t len);
+u32 crc32c(u32 crc, const void *p, size_t len);
/*
* crc32_optimizations() returns flags that indicate which CRC32 library
* functions are using architecture-specific optimizations. Unlike
* IS_ENABLED(CONFIG_CRC32_ARCH) it takes into account the different CRC32
diff --git a/include/linux/crc64.h b/include/linux/crc64.h
index b6aa290a79312..fc0c06ab1993c 100644
--- a/include/linux/crc64.h
+++ b/include/linux/crc64.h
@@ -2,28 +2,18 @@
#ifndef _LINUX_CRC64_H
#define _LINUX_CRC64_H
#include <linux/types.h>
-u64 crc64_be_arch(u64 crc, const u8 *p, size_t len);
-u64 crc64_be_generic(u64 crc, const u8 *p, size_t len);
-u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len);
-u64 crc64_nvme_generic(u64 crc, const u8 *p, size_t len);
-
/**
* crc64_be - Calculate bitwise big-endian ECMA-182 CRC64
* @crc: seed value for computation. 0 or (u64)~0 for a new CRC calculation,
* or the previous crc64 value if computing incrementally.
* @p: pointer to buffer over which CRC64 is run
* @len: length of buffer @p
*/
-static inline u64 crc64_be(u64 crc, const void *p, size_t len)
-{
- if (IS_ENABLED(CONFIG_CRC64_ARCH))
- return crc64_be_arch(crc, p, len);
- return crc64_be_generic(crc, p, len);
-}
+u64 crc64_be(u64 crc, const void *p, size_t len);
/**
* crc64_nvme - Calculate CRC64-NVME
* @crc: seed value for computation. 0 for a new CRC calculation, or the
* previous crc64 value if computing incrementally.
@@ -31,13 +21,8 @@ static inline u64 crc64_be(u64 crc, const void *p, size_t len)
* @len: length of buffer @p
*
* This computes the CRC64 defined in the NVME NVM Command Set Specification,
* *including the bitwise inversion at the beginning and end*.
*/
-static inline u64 crc64_nvme(u64 crc, const void *p, size_t len)
-{
- if (IS_ENABLED(CONFIG_CRC64_ARCH))
- return ~crc64_nvme_arch(~crc, p, len);
- return ~crc64_nvme_generic(~crc, p, len);
-}
+u64 crc64_nvme(u64 crc, const void *p, size_t len);
#endif /* _LINUX_CRC64_H */
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index e0e7168b74c75..85b0dc774d722 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -46,39 +46,33 @@ config CRC_T10DIF
config ARCH_HAS_CRC_T10DIF
bool
config CRC_T10DIF_ARCH
- tristate
- default CRC_T10DIF if ARCH_HAS_CRC_T10DIF && CRC_OPTIMIZATIONS
+ bool
+ depends on CRC_T10DIF && CRC_OPTIMIZATIONS
config CRC32
tristate
select BITREVERSE
help
The CRC32 library functions. Select this if your module uses any of
the functions from <linux/crc32.h> or <linux/crc32c.h>.
-config ARCH_HAS_CRC32
- bool
-
config CRC32_ARCH
- tristate
- default CRC32 if ARCH_HAS_CRC32 && CRC_OPTIMIZATIONS
+ bool
+ depends on CRC32 && CRC_OPTIMIZATIONS
config CRC64
tristate
help
The CRC64 library functions. Select this if your module uses any of
the functions from <linux/crc64.h>.
-config ARCH_HAS_CRC64
- bool
-
config CRC64_ARCH
- tristate
- default CRC64 if ARCH_HAS_CRC64 && CRC_OPTIMIZATIONS
+ bool
+ depends on CRC64 && CRC_OPTIMIZATIONS
config CRC_OPTIMIZATIONS
bool "Enable optimized CRC implementations" if EXPERT
default y
help
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index ff4c30dda4528..2f39385f97a5c 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -1,25 +1,40 @@
# SPDX-License-Identifier: GPL-2.0-only
# Makefile for the kernel's cyclic redundancy check (CRC) library code
+ccflags-y := -I$(src)/$(SRCARCH)
+
obj-$(CONFIG_CRC4) += crc4.o
obj-$(CONFIG_CRC7) += crc7.o
obj-$(CONFIG_CRC8) += crc8.o
obj-$(CONFIG_CRC16) += crc16.o
obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o
obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o
+
obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o
+crc-t10dif-y := crc-t10dif-main.o
+ifeq ($(CONFIG_CRC_T10DIF_ARCH),y)
+endif
+
obj-$(CONFIG_CRC32) += crc32.o
+crc32-y := crc32-main.o
+ifeq ($(CONFIG_CRC32_ARCH),y)
+endif
+
obj-$(CONFIG_CRC64) += crc64.o
+crc64-y := crc64-main.o
+ifeq ($(CONFIG_CRC64_ARCH),y)
+endif
+
obj-y += tests/
hostprogs := gen_crc32table gen_crc64table
clean-files := crc32table.h crc64table.h
-$(obj)/crc32.o: $(obj)/crc32table.h
-$(obj)/crc64.o: $(obj)/crc64table.h
+$(obj)/crc32-main.o: $(obj)/crc32table.h
+$(obj)/crc64-main.o: $(obj)/crc64table.h
quiet_cmd_crc32 = GEN $@
cmd_crc32 = $< > $@
quiet_cmd_crc64 = GEN $@
diff --git a/lib/crc/crc-t10dif.c b/lib/crc/crc-t10dif-main.c
similarity index 78%
rename from lib/crc/crc-t10dif.c
rename to lib/crc/crc-t10dif-main.c
index 311c2ab829f15..5d7f298c267ed 100644
--- a/lib/crc/crc-t10dif.c
+++ b/lib/crc/crc-t10dif-main.c
@@ -48,18 +48,41 @@ static const u16 t10_dif_crc_table[256] = {
0xA415, 0x2FA2, 0x38CC, 0xB37B, 0x1610, 0x9DA7, 0x8AC9, 0x017E,
0x1F65, 0x94D2, 0x83BC, 0x080B, 0xAD60, 0x26D7, 0x31B9, 0xBA0E,
0xF0D8, 0x7B6F, 0x6C01, 0xE7B6, 0x42DD, 0xC96A, 0xDE04, 0x55B3
};
-u16 crc_t10dif_generic(u16 crc, const u8 *p, size_t len)
+static inline __maybe_unused u16
+crc_t10dif_generic(u16 crc, const u8 *p, size_t len)
{
- size_t i;
+ while (len--)
+ crc = (crc << 8) ^ t10_dif_crc_table[(crc >> 8) ^ *p++];
+ return crc;
+}
- for (i = 0; i < len; i++)
- crc = (crc << 8) ^ t10_dif_crc_table[(crc >> 8) ^ p[i]];
+#ifdef CONFIG_CRC_T10DIF_ARCH
+#include "crc-t10dif.h" /* $(SRCARCH)/crc-t10dif.h */
+#else
+#define crc_t10dif_arch crc_t10dif_generic
+#endif
- return crc;
+u16 crc_t10dif_update(u16 crc, const u8 *p, size_t len)
+{
+ return crc_t10dif_arch(crc, p, len);
+}
+EXPORT_SYMBOL(crc_t10dif_update);
+
+#ifdef crc_t10dif_mod_init_arch
+static int __init crc_t10dif_mod_init(void)
+{
+ crc_t10dif_mod_init_arch();
+ return 0;
+}
+subsys_initcall(crc_t10dif_mod_init);
+
+static void __exit crc_t10dif_mod_exit(void)
+{
}
-EXPORT_SYMBOL(crc_t10dif_generic);
+module_exit(crc_t10dif_mod_exit);
+#endif
-MODULE_DESCRIPTION("T10 DIF CRC calculation");
+MODULE_DESCRIPTION("CRC-T10DIF library functions");
MODULE_LICENSE("GPL");
diff --git a/lib/crc/crc32.c b/lib/crc/crc32-main.c
similarity index 73%
rename from lib/crc/crc32.c
rename to lib/crc/crc32-main.c
index 95429861d3aca..a86b4bed0def7 100644
--- a/lib/crc/crc32.c
+++ b/lib/crc/crc32-main.c
@@ -29,29 +29,71 @@
#include <linux/module.h>
#include <linux/types.h>
#include "crc32table.h"
-MODULE_AUTHOR("Matt Domsch <Matt_Domsch@dell.com>");
-MODULE_DESCRIPTION("Various CRC32 calculations");
-MODULE_LICENSE("GPL");
-
-u32 crc32_le_base(u32 crc, const u8 *p, size_t len)
+static inline __maybe_unused u32
+crc32_le_base(u32 crc, const u8 *p, size_t len)
{
while (len--)
crc = (crc >> 8) ^ crc32table_le[(crc & 255) ^ *p++];
return crc;
}
-EXPORT_SYMBOL(crc32_le_base);
-u32 crc32c_base(u32 crc, const u8 *p, size_t len)
+static inline __maybe_unused u32
+crc32_be_base(u32 crc, const u8 *p, size_t len)
+{
+ while (len--)
+ crc = (crc << 8) ^ crc32table_be[(crc >> 24) ^ *p++];
+ return crc;
+}
+
+static inline __maybe_unused u32
+crc32c_base(u32 crc, const u8 *p, size_t len)
{
while (len--)
crc = (crc >> 8) ^ crc32ctable_le[(crc & 255) ^ *p++];
return crc;
}
-EXPORT_SYMBOL(crc32c_base);
+
+#ifdef CONFIG_CRC32_ARCH
+#include "crc32.h" /* $(SRCARCH)/crc32.h */
+
+u32 crc32_optimizations(void)
+{
+ return crc32_optimizations_arch();
+}
+EXPORT_SYMBOL(crc32_optimizations);
+#endif
+
+#ifndef crc32_le_arch
+#define crc32_le_arch crc32_le_base
+#endif
+#ifndef crc32_be_arch
+#define crc32_be_arch crc32_be_base
+#endif
+#ifndef crc32c_arch
+#define crc32c_arch crc32c_base
+#endif
+
+u32 crc32_le(u32 crc, const void *p, size_t len)
+{
+ return crc32_le_arch(crc, p, len);
+}
+EXPORT_SYMBOL(crc32_le);
+
+u32 crc32_be(u32 crc, const void *p, size_t len)
+{
+ return crc32_be_arch(crc, p, len);
+}
+EXPORT_SYMBOL(crc32_be);
+
+u32 crc32c(u32 crc, const void *p, size_t len)
+{
+ return crc32c_arch(crc, p, len);
+}
+EXPORT_SYMBOL(crc32c);
/*
* This multiplies the polynomials x and y modulo the given modulus.
* This follows the "little-endian" CRC convention that the lsbit
* represents the highest power of x, and the msbit represents x^0.
@@ -115,12 +157,21 @@ u32 crc32_le_shift(u32 crc, size_t len)
{
return crc32_generic_shift(crc, len, CRC32_POLY_LE);
}
EXPORT_SYMBOL(crc32_le_shift);
-u32 crc32_be_base(u32 crc, const u8 *p, size_t len)
+#ifdef crc32_mod_init_arch
+static int __init crc32_mod_init(void)
{
- while (len--)
- crc = (crc << 8) ^ crc32table_be[(crc >> 24) ^ *p++];
- return crc;
+ crc32_mod_init_arch();
+ return 0;
}
-EXPORT_SYMBOL(crc32_be_base);
+subsys_initcall(crc32_mod_init);
+
+static void __exit crc32_mod_exit(void)
+{
+}
+module_exit(crc32_mod_exit);
+#endif
+
+MODULE_DESCRIPTION("CRC32 library functions");
+MODULE_LICENSE("GPL");
diff --git a/lib/crc/crc64.c b/lib/crc/crc64-main.c
similarity index 66%
rename from lib/crc/crc64.c
rename to lib/crc/crc64-main.c
index 5b1b17057f0ae..e4a6d879e84c3 100644
--- a/lib/crc/crc64.c
+++ b/lib/crc/crc64-main.c
@@ -36,23 +36,56 @@
#include <linux/module.h>
#include <linux/types.h>
#include <linux/crc64.h>
#include "crc64table.h"
-MODULE_DESCRIPTION("CRC64 calculations");
-MODULE_LICENSE("GPL v2");
-
-u64 crc64_be_generic(u64 crc, const u8 *p, size_t len)
+static inline __maybe_unused u64
+crc64_be_generic(u64 crc, const u8 *p, size_t len)
{
while (len--)
crc = (crc << 8) ^ crc64table[(crc >> 56) ^ *p++];
return crc;
}
-EXPORT_SYMBOL_GPL(crc64_be_generic);
-u64 crc64_nvme_generic(u64 crc, const u8 *p, size_t len)
+static inline __maybe_unused u64
+crc64_nvme_generic(u64 crc, const u8 *p, size_t len)
{
while (len--)
crc = (crc >> 8) ^ crc64nvmetable[(crc & 0xff) ^ *p++];
return crc;
}
-EXPORT_SYMBOL_GPL(crc64_nvme_generic);
+
+#ifdef CONFIG_CRC64_ARCH
+#include "crc64.h" /* $(SRCARCH)/crc64.h */
+#else
+#define crc64_be_arch crc64_be_generic
+#define crc64_nvme_arch crc64_nvme_generic
+#endif
+
+u64 crc64_be(u64 crc, const void *p, size_t len)
+{
+ return crc64_be_arch(crc, p, len);
+}
+EXPORT_SYMBOL_GPL(crc64_be);
+
+u64 crc64_nvme(u64 crc, const void *p, size_t len)
+{
+ return ~crc64_nvme_arch(~crc, p, len);
+}
+EXPORT_SYMBOL_GPL(crc64_nvme);
+
+#ifdef crc64_mod_init_arch
+static int __init crc64_mod_init(void)
+{
+ crc64_mod_init_arch();
+ return 0;
+}
+subsys_initcall(crc64_mod_init);
+
+static void __exit crc64_mod_exit(void)
+{
+}
+module_exit(crc64_mod_exit);
+#endif
+
+MODULE_DESCRIPTION("CRC64 library functions");
+MODULE_LICENSE("GPL");
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 05/13] lib/crc/arm: migrate arm-optimized CRC code into lib/crc/
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
` (3 preceding siblings ...)
2025-06-01 22:44 ` [PATCH 04/13] lib/crc: prepare for arch-optimized code in subdirs of lib/crc/ Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 06/13] lib/crc/arm64: migrate arm64-optimized " Eric Biggers
` (8 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Move the arm-optimized CRC code from arch/arm/lib/crc* into its new
location in lib/crc/arm/, and wire it up in the new way. For a detailed
explanation of why this change is being made, see the commit that
introduced the new way of integrating arch-specific code into lib/crc/.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/arm/Kconfig | 2 -
arch/arm/lib/Makefile | 6 ---
lib/crc/Kconfig | 2 +
lib/crc/Makefile | 2 +
.../arm/lib => lib/crc/arm}/crc-t10dif-core.S | 0
.../crc-t10dif.c => lib/crc/arm/crc-t10dif.h | 23 ++---------
{arch/arm/lib => lib/crc/arm}/crc32-core.S | 0
arch/arm/lib/crc32.c => lib/crc/arm/crc32.h | 40 +++++--------------
8 files changed, 16 insertions(+), 59 deletions(-)
rename {arch/arm/lib => lib/crc/arm}/crc-t10dif-core.S (100%)
rename arch/arm/lib/crc-t10dif.c => lib/crc/arm/crc-t10dif.h (70%)
rename {arch/arm/lib => lib/crc/arm}/crc32-core.S (100%)
rename arch/arm/lib/crc32.c => lib/crc/arm/crc32.h (69%)
diff --git a/arch/arm/Kconfig b/arch/arm/Kconfig
index 3072731fe09c5..c531b49aa98ea 100644
--- a/arch/arm/Kconfig
+++ b/arch/arm/Kconfig
@@ -6,12 +6,10 @@ config ARM
select ARCH_CORRECT_STACKTRACE_ON_KRETPROBE if HAVE_KRETPROBES && FRAME_POINTER && !ARM_UNWIND
select ARCH_HAS_BINFMT_FLAT
select ARCH_HAS_CACHE_LINE_SIZE if OF
select ARCH_HAS_CPU_CACHE_ALIASING
select ARCH_HAS_CPU_FINALIZE_INIT if MMU
- select ARCH_HAS_CRC32 if KERNEL_MODE_NEON
- select ARCH_HAS_CRC_T10DIF if KERNEL_MODE_NEON
select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_HAS_DEBUG_VIRTUAL if MMU
select ARCH_HAS_DMA_ALLOC if MMU
select ARCH_HAS_DMA_OPS
select ARCH_HAS_DMA_WRITE_COMBINE if !ARM_DMA_MEM_BUFFERABLE
diff --git a/arch/arm/lib/Makefile b/arch/arm/lib/Makefile
index 91ea0e29107af..d1b9ea2023648 100644
--- a/arch/arm/lib/Makefile
+++ b/arch/arm/lib/Makefile
@@ -45,11 +45,5 @@ ifeq ($(CONFIG_KERNEL_MODE_NEON),y)
CFLAGS_xor-neon.o += $(CC_FLAGS_FPU)
obj-$(CONFIG_XOR_BLOCKS) += xor-neon.o
endif
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
-
-obj-$(CONFIG_CRC32_ARCH) += crc32-arm.o
-crc32-arm-y := crc32.o crc32-core.o
-
-obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-arm.o
-crc-t10dif-arm-y := crc-t10dif.o crc-t10dif-core.o
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index 85b0dc774d722..b34a52cdba5d7 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -48,10 +48,11 @@ config ARCH_HAS_CRC_T10DIF
bool
config CRC_T10DIF_ARCH
bool
depends on CRC_T10DIF && CRC_OPTIMIZATIONS
+ default y if ARM && KERNEL_MODE_NEON
config CRC32
tristate
select BITREVERSE
help
@@ -59,10 +60,11 @@ config CRC32
the functions from <linux/crc32.h> or <linux/crc32c.h>.
config CRC32_ARCH
bool
depends on CRC32 && CRC_OPTIMIZATIONS
+ default y if ARM && KERNEL_MODE_NEON
config CRC64
tristate
help
The CRC64 library functions. Select this if your module uses any of
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index 2f39385f97a5c..1069edb81a481 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -12,15 +12,17 @@ obj-$(CONFIG_CRC_CCITT) += crc-ccitt.o
obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o
obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o
crc-t10dif-y := crc-t10dif-main.o
ifeq ($(CONFIG_CRC_T10DIF_ARCH),y)
+crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o
endif
obj-$(CONFIG_CRC32) += crc32.o
crc32-y := crc32-main.o
ifeq ($(CONFIG_CRC32_ARCH),y)
+crc32-$(CONFIG_ARM) += arm/crc32-core.o
endif
obj-$(CONFIG_CRC64) += crc64.o
crc64-y := crc64-main.o
ifeq ($(CONFIG_CRC64_ARCH),y)
diff --git a/arch/arm/lib/crc-t10dif-core.S b/lib/crc/arm/crc-t10dif-core.S
similarity index 100%
rename from arch/arm/lib/crc-t10dif-core.S
rename to lib/crc/arm/crc-t10dif-core.S
diff --git a/arch/arm/lib/crc-t10dif.c b/lib/crc/arm/crc-t10dif.h
similarity index 70%
rename from arch/arm/lib/crc-t10dif.c
rename to lib/crc/arm/crc-t10dif.h
index 1093f8ec13b0b..2edf7e9681d05 100644
--- a/arch/arm/lib/crc-t10dif.c
+++ b/lib/crc/arm/crc-t10dif.h
@@ -3,16 +3,10 @@
* Accelerated CRC-T10DIF using ARM NEON and Crypto Extensions instructions
*
* Copyright (C) 2016 Linaro Ltd <ard.biesheuvel@linaro.org>
*/
-#include <linux/crc-t10dif.h>
-#include <linux/init.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/string.h>
-
#include <crypto/internal/simd.h>
#include <asm/neon.h>
#include <asm/simd.h>
@@ -23,11 +17,11 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull);
asmlinkage u16 crc_t10dif_pmull64(u16 init_crc, const u8 *buf, size_t len);
asmlinkage void crc_t10dif_pmull8(u16 init_crc, const u8 *buf, size_t len,
u8 out[16]);
-u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length)
+static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length)
{
if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE) {
if (static_branch_likely(&have_pmull)) {
if (crypto_simd_usable()) {
kernel_neon_begin();
@@ -47,26 +41,15 @@ u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length)
return crc_t10dif_generic(0, buf, sizeof(buf));
}
}
return crc_t10dif_generic(crc, data, length);
}
-EXPORT_SYMBOL(crc_t10dif_arch);
-static int __init crc_t10dif_arm_init(void)
+#define crc_t10dif_mod_init_arch crc_t10dif_mod_init_arch
+static inline void crc_t10dif_mod_init_arch(void)
{
if (elf_hwcap & HWCAP_NEON) {
static_branch_enable(&have_neon);
if (elf_hwcap2 & HWCAP2_PMULL)
static_branch_enable(&have_pmull);
}
- return 0;
}
-subsys_initcall(crc_t10dif_arm_init);
-
-static void __exit crc_t10dif_arm_exit(void)
-{
-}
-module_exit(crc_t10dif_arm_exit);
-
-MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
-MODULE_DESCRIPTION("Accelerated CRC-T10DIF using ARM NEON and Crypto Extensions");
-MODULE_LICENSE("GPL v2");
diff --git a/arch/arm/lib/crc32-core.S b/lib/crc/arm/crc32-core.S
similarity index 100%
rename from arch/arm/lib/crc32-core.S
rename to lib/crc/arm/crc32-core.S
diff --git a/arch/arm/lib/crc32.c b/lib/crc/arm/crc32.h
similarity index 69%
rename from arch/arm/lib/crc32.c
rename to lib/crc/arm/crc32.h
index f2bef8849c7c3..19e6e35f6418f 100644
--- a/arch/arm/lib/crc32.c
+++ b/lib/crc/arm/crc32.h
@@ -4,15 +4,10 @@
*
* Copyright (C) 2016 Linaro Ltd <ard.biesheuvel@linaro.org>
*/
#include <linux/cpufeature.h>
-#include <linux/crc32.h>
-#include <linux/init.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/string.h>
#include <crypto/internal/simd.h>
#include <asm/hwcap.h>
#include <asm/neon.h>
@@ -27,18 +22,19 @@ asmlinkage u32 crc32_pmull_le(const u8 buf[], u32 len, u32 init_crc);
asmlinkage u32 crc32_armv8_le(u32 init_crc, const u8 buf[], u32 len);
asmlinkage u32 crc32c_pmull_le(const u8 buf[], u32 len, u32 init_crc);
asmlinkage u32 crc32c_armv8_le(u32 init_crc, const u8 buf[], u32 len);
-static u32 crc32_le_scalar(u32 crc, const u8 *p, size_t len)
+static inline u32 crc32_le_scalar(u32 crc, const u8 *p, size_t len)
{
if (static_branch_likely(&have_crc32))
return crc32_armv8_le(crc, p, len);
return crc32_le_base(crc, p, len);
}
-u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
+#define crc32_le_arch crc32_le_arch
+static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
{
if (len >= PMULL_MIN_LEN + 15 &&
static_branch_likely(&have_pmull) && crypto_simd_usable()) {
size_t n = -(uintptr_t)p & 15;
@@ -55,20 +51,20 @@ u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
p += n;
len -= n;
}
return crc32_le_scalar(crc, p, len);
}
-EXPORT_SYMBOL(crc32_le_arch);
-static u32 crc32c_scalar(u32 crc, const u8 *p, size_t len)
+static inline u32 crc32c_scalar(u32 crc, const u8 *p, size_t len)
{
if (static_branch_likely(&have_crc32))
return crc32c_armv8_le(crc, p, len);
return crc32c_base(crc, p, len);
}
-u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
+#define crc32c_arch crc32c_arch
+static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
{
if (len >= PMULL_MIN_LEN + 15 &&
static_branch_likely(&have_pmull) && crypto_simd_usable()) {
size_t n = -(uintptr_t)p & 15;
@@ -85,39 +81,21 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
p += n;
len -= n;
}
return crc32c_scalar(crc, p, len);
}
-EXPORT_SYMBOL(crc32c_arch);
-u32 crc32_be_arch(u32 crc, const u8 *p, size_t len)
-{
- return crc32_be_base(crc, p, len);
-}
-EXPORT_SYMBOL(crc32_be_arch);
-
-static int __init crc32_arm_init(void)
+#define crc32_mod_init_arch crc32_mod_init_arch
+static inline void crc32_mod_init_arch(void)
{
if (elf_hwcap2 & HWCAP2_CRC32)
static_branch_enable(&have_crc32);
if (elf_hwcap2 & HWCAP2_PMULL)
static_branch_enable(&have_pmull);
- return 0;
}
-subsys_initcall(crc32_arm_init);
-static void __exit crc32_arm_exit(void)
-{
-}
-module_exit(crc32_arm_exit);
-
-u32 crc32_optimizations(void)
+static inline u32 crc32_optimizations_arch(void)
{
if (elf_hwcap2 & (HWCAP2_CRC32 | HWCAP2_PMULL))
return CRC32_LE_OPTIMIZATION | CRC32C_OPTIMIZATION;
return 0;
}
-EXPORT_SYMBOL(crc32_optimizations);
-
-MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
-MODULE_DESCRIPTION("Accelerated CRC32(C) using ARM CRC, NEON and Crypto Extensions");
-MODULE_LICENSE("GPL v2");
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 06/13] lib/crc/arm64: migrate arm64-optimized CRC code into lib/crc/
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
` (4 preceding siblings ...)
2025-06-01 22:44 ` [PATCH 05/13] lib/crc/arm: migrate arm-optimized CRC code into lib/crc/ Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 07/13] lib/crc/loongarch: migrate loongarch-optimized " Eric Biggers
` (7 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Move the arm64-optimized CRC code from arch/arm64/lib/crc* into its new
location in lib/crc/arm64/, and wire it up in the new way. For a
detailed explanation of why this change is being made, see the commit
that introduced the new way of integrating arch-specific code into
lib/crc/.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/arm64/Kconfig | 2 --
arch/arm64/lib/Makefile | 6 -----
lib/crc/Kconfig | 2 ++
lib/crc/Makefile | 2 ++
.../lib => lib/crc/arm64}/crc-t10dif-core.S | 0
.../crc/arm64/crc-t10dif.h | 22 +++----------------
.../arm64/lib => lib/crc/arm64}/crc32-core.S | 0
.../lib/crc32.c => lib/crc/arm64/crc32.h | 22 ++++++-------------
8 files changed, 14 insertions(+), 42 deletions(-)
rename {arch/arm64/lib => lib/crc/arm64}/crc-t10dif-core.S (100%)
rename arch/arm64/lib/crc-t10dif.c => lib/crc/arm64/crc-t10dif.h (70%)
rename {arch/arm64/lib => lib/crc/arm64}/crc32-core.S (100%)
rename arch/arm64/lib/crc32.c => lib/crc/arm64/crc32.h (81%)
diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig
index 55fc331af3371..cbeac225903f7 100644
--- a/arch/arm64/Kconfig
+++ b/arch/arm64/Kconfig
@@ -19,12 +19,10 @@ config ARM64
select ARCH_ENABLE_MEMORY_HOTREMOVE
select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
select ARCH_HAS_CACHE_LINE_SIZE
select ARCH_HAS_CC_PLATFORM
- select ARCH_HAS_CRC32
- select ARCH_HAS_CRC_T10DIF if KERNEL_MODE_NEON
select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_HAS_DEBUG_VIRTUAL
select ARCH_HAS_DEBUG_VM_PGTABLE
select ARCH_HAS_DMA_OPS if XEN
select ARCH_HAS_DMA_PREP_COHERENT
diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
index 027bfa9689c6a..9b255d9332479 100644
--- a/arch/arm64/lib/Makefile
+++ b/arch/arm64/lib/Makefile
@@ -14,16 +14,10 @@ CFLAGS_xor-neon.o += $(CC_FLAGS_FPU)
CFLAGS_REMOVE_xor-neon.o += $(CC_FLAGS_NO_FPU)
endif
lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o
-obj-$(CONFIG_CRC32_ARCH) += crc32-arm64.o
-crc32-arm64-y := crc32.o crc32-core.o
-
-obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-arm64.o
-crc-t10dif-arm64-y := crc-t10dif.o crc-t10dif-core.o
-
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
obj-$(CONFIG_ARM64_MTE) += mte.o
obj-$(CONFIG_KASAN_SW_TAGS) += kasan_sw_tags.o
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index b34a52cdba5d7..663f4dca2bda1 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -49,10 +49,11 @@ config ARCH_HAS_CRC_T10DIF
config CRC_T10DIF_ARCH
bool
depends on CRC_T10DIF && CRC_OPTIMIZATIONS
default y if ARM && KERNEL_MODE_NEON
+ default y if ARM64 && KERNEL_MODE_NEON
config CRC32
tristate
select BITREVERSE
help
@@ -61,10 +62,11 @@ config CRC32
config CRC32_ARCH
bool
depends on CRC32 && CRC_OPTIMIZATIONS
default y if ARM && KERNEL_MODE_NEON
+ default y if ARM64
config CRC64
tristate
help
The CRC64 library functions. Select this if your module uses any of
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index 1069edb81a481..822d18b57b432 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -13,16 +13,18 @@ obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o
obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o
crc-t10dif-y := crc-t10dif-main.o
ifeq ($(CONFIG_CRC_T10DIF_ARCH),y)
crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o
+crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o
endif
obj-$(CONFIG_CRC32) += crc32.o
crc32-y := crc32-main.o
ifeq ($(CONFIG_CRC32_ARCH),y)
crc32-$(CONFIG_ARM) += arm/crc32-core.o
+crc32-$(CONFIG_ARM64) += arm64/crc32-core.o
endif
obj-$(CONFIG_CRC64) += crc64.o
crc64-y := crc64-main.o
ifeq ($(CONFIG_CRC64_ARCH),y)
diff --git a/arch/arm64/lib/crc-t10dif-core.S b/lib/crc/arm64/crc-t10dif-core.S
similarity index 100%
rename from arch/arm64/lib/crc-t10dif-core.S
rename to lib/crc/arm64/crc-t10dif-core.S
diff --git a/arch/arm64/lib/crc-t10dif.c b/lib/crc/arm64/crc-t10dif.h
similarity index 70%
rename from arch/arm64/lib/crc-t10dif.c
rename to lib/crc/arm64/crc-t10dif.h
index c2ffe4fdb59d1..c4521a7f1ee9b 100644
--- a/arch/arm64/lib/crc-t10dif.c
+++ b/lib/crc/arm64/crc-t10dif.h
@@ -4,15 +4,10 @@
*
* Copyright (C) 2016 - 2017 Linaro Ltd <ard.biesheuvel@linaro.org>
*/
#include <linux/cpufeature.h>
-#include <linux/crc-t10dif.h>
-#include <linux/init.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
-#include <linux/string.h>
#include <crypto/internal/simd.h>
#include <asm/neon.h>
#include <asm/simd.h>
@@ -24,11 +19,11 @@ static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull);
asmlinkage void crc_t10dif_pmull_p8(u16 init_crc, const u8 *buf, size_t len,
u8 out[16]);
asmlinkage u16 crc_t10dif_pmull_p64(u16 init_crc, const u8 *buf, size_t len);
-u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length)
+static inline u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length)
{
if (length >= CRC_T10DIF_PMULL_CHUNK_SIZE) {
if (static_branch_likely(&have_pmull)) {
if (crypto_simd_usable()) {
kernel_neon_begin();
@@ -48,26 +43,15 @@ u16 crc_t10dif_arch(u16 crc, const u8 *data, size_t length)
return crc_t10dif_generic(0, buf, sizeof(buf));
}
}
return crc_t10dif_generic(crc, data, length);
}
-EXPORT_SYMBOL(crc_t10dif_arch);
-static int __init crc_t10dif_arm64_init(void)
+#define crc_t10dif_mod_init_arch crc_t10dif_mod_init_arch
+static inline void crc_t10dif_mod_init_arch(void)
{
if (cpu_have_named_feature(ASIMD)) {
static_branch_enable(&have_asimd);
if (cpu_have_named_feature(PMULL))
static_branch_enable(&have_pmull);
}
- return 0;
}
-subsys_initcall(crc_t10dif_arm64_init);
-
-static void __exit crc_t10dif_arm64_exit(void)
-{
-}
-module_exit(crc_t10dif_arm64_exit);
-
-MODULE_AUTHOR("Ard Biesheuvel <ard.biesheuvel@linaro.org>");
-MODULE_DESCRIPTION("CRC-T10DIF using arm64 NEON and Crypto Extensions");
-MODULE_LICENSE("GPL v2");
diff --git a/arch/arm64/lib/crc32-core.S b/lib/crc/arm64/crc32-core.S
similarity index 100%
rename from arch/arm64/lib/crc32-core.S
rename to lib/crc/arm64/crc32-core.S
diff --git a/arch/arm64/lib/crc32.c b/lib/crc/arm64/crc32.h
similarity index 81%
rename from arch/arm64/lib/crc32.c
rename to lib/crc/arm64/crc32.h
index ed3acd71178f8..98f3b3fcc729a 100644
--- a/arch/arm64/lib/crc32.c
+++ b/lib/crc/arm64/crc32.h
@@ -1,11 +1,7 @@
// SPDX-License-Identifier: GPL-2.0-only
-#include <linux/crc32.h>
-#include <linux/linkage.h>
-#include <linux/module.h>
-
#include <asm/alternative.h>
#include <asm/cpufeature.h>
#include <asm/neon.h>
#include <asm/simd.h>
@@ -20,11 +16,12 @@ asmlinkage u32 crc32_be_arm64(u32 crc, unsigned char const *p, size_t len);
asmlinkage u32 crc32_le_arm64_4way(u32 crc, unsigned char const *p, size_t len);
asmlinkage u32 crc32c_le_arm64_4way(u32 crc, unsigned char const *p, size_t len);
asmlinkage u32 crc32_be_arm64_4way(u32 crc, unsigned char const *p, size_t len);
-u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
+#define crc32_le_arch crc32_le_arch
+static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
{
if (!alternative_has_cap_likely(ARM64_HAS_CRC32))
return crc32_le_base(crc, p, len);
if (len >= min_len && cpu_have_named_feature(PMULL) && crypto_simd_usable()) {
@@ -39,13 +36,13 @@ u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
return crc;
}
return crc32_le_arm64(crc, p, len);
}
-EXPORT_SYMBOL(crc32_le_arch);
-u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
+#define crc32c_arch crc32c_arch
+static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
{
if (!alternative_has_cap_likely(ARM64_HAS_CRC32))
return crc32c_base(crc, p, len);
if (len >= min_len && cpu_have_named_feature(PMULL) && crypto_simd_usable()) {
@@ -60,13 +57,13 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
return crc;
}
return crc32c_le_arm64(crc, p, len);
}
-EXPORT_SYMBOL(crc32c_arch);
-u32 crc32_be_arch(u32 crc, const u8 *p, size_t len)
+#define crc32_be_arch crc32_be_arch
+static inline u32 crc32_be_arch(u32 crc, const u8 *p, size_t len)
{
if (!alternative_has_cap_likely(ARM64_HAS_CRC32))
return crc32_be_base(crc, p, len);
if (len >= min_len && cpu_have_named_feature(PMULL) && crypto_simd_usable()) {
@@ -81,19 +78,14 @@ u32 crc32_be_arch(u32 crc, const u8 *p, size_t len)
return crc;
}
return crc32_be_arm64(crc, p, len);
}
-EXPORT_SYMBOL(crc32_be_arch);
-u32 crc32_optimizations(void)
+static inline u32 crc32_optimizations_arch(void)
{
if (alternative_has_cap_likely(ARM64_HAS_CRC32))
return CRC32_LE_OPTIMIZATION |
CRC32_BE_OPTIMIZATION |
CRC32C_OPTIMIZATION;
return 0;
}
-EXPORT_SYMBOL(crc32_optimizations);
-
-MODULE_LICENSE("GPL");
-MODULE_DESCRIPTION("arm64-optimized CRC32 functions");
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 07/13] lib/crc/loongarch: migrate loongarch-optimized CRC code into lib/crc/
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
` (5 preceding siblings ...)
2025-06-01 22:44 ` [PATCH 06/13] lib/crc/arm64: migrate arm64-optimized " Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 08/13] lib/crc/mips: migrate mips-optimized " Eric Biggers
` (6 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Move the loongarch-optimized CRC code from arch/loongarch/lib/crc* into
its new location in lib/crc/loongarch/, and wire it up in the new way.
For a detailed explanation of why this change is being made, see the
commit that introduced the new way of integrating arch-specific code
into lib/crc/.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/loongarch/Kconfig | 1 -
arch/loongarch/lib/Makefile | 2 --
lib/crc/Kconfig | 1 +
.../crc/loongarch/crc32.h | 34 ++++---------------
4 files changed, 8 insertions(+), 30 deletions(-)
rename arch/loongarch/lib/crc32-loongarch.c => lib/crc/loongarch/crc32.h (71%)
diff --git a/arch/loongarch/Kconfig b/arch/loongarch/Kconfig
index 1a2cf012b8f2f..1b19893a6bdf0 100644
--- a/arch/loongarch/Kconfig
+++ b/arch/loongarch/Kconfig
@@ -13,11 +13,10 @@ config LOONGARCH
select ARCH_ENABLE_MEMORY_HOTPLUG
select ARCH_ENABLE_MEMORY_HOTREMOVE
select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI
select ARCH_HAS_CPU_FINALIZE_INIT
- select ARCH_HAS_CRC32
select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_HAS_DEBUG_VM_PGTABLE
select ARCH_HAS_FAST_MULTIPLIER
select ARCH_HAS_FORTIFY_SOURCE
select ARCH_HAS_KCOV
diff --git a/arch/loongarch/lib/Makefile b/arch/loongarch/lib/Makefile
index fae77809048b8..ccea3bbd43531 100644
--- a/arch/loongarch/lib/Makefile
+++ b/arch/loongarch/lib/Makefile
@@ -9,7 +9,5 @@ lib-y += delay.o memset.o memcpy.o memmove.o \
obj-$(CONFIG_ARCH_SUPPORTS_INT128) += tishift.o
obj-$(CONFIG_CPU_HAS_LSX) += xor_simd.o xor_simd_glue.o
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
-
-obj-$(CONFIG_CRC32_ARCH) += crc32-loongarch.o
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index 663f4dca2bda1..ee320ae19ca85 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -63,10 +63,11 @@ config CRC32
config CRC32_ARCH
bool
depends on CRC32 && CRC_OPTIMIZATIONS
default y if ARM && KERNEL_MODE_NEON
default y if ARM64
+ default y if LOONGARCH
config CRC64
tristate
help
The CRC64 library functions. Select this if your module uses any of
diff --git a/arch/loongarch/lib/crc32-loongarch.c b/lib/crc/loongarch/crc32.h
similarity index 71%
rename from arch/loongarch/lib/crc32-loongarch.c
rename to lib/crc/loongarch/crc32.h
index b37cd8537b459..d5aff7ca96ae9 100644
--- a/arch/loongarch/lib/crc32-loongarch.c
+++ b/lib/crc/loongarch/crc32.h
@@ -8,12 +8,10 @@
* Copyright (C) 2018 MIPS Tech, LLC
* Copyright (C) 2020-2023 Loongson Technology Corporation Limited
*/
#include <asm/cpu-features.h>
-#include <linux/crc32.h>
-#include <linux/module.h>
#include <linux/unaligned.h>
#define _CRC32(crc, value, size, type) \
do { \
__asm__ __volatile__( \
@@ -26,11 +24,12 @@ do { \
#define CRC32(crc, value, size) _CRC32(crc, value, size, crc)
#define CRC32C(crc, value, size) _CRC32(crc, value, size, crcc)
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32);
-u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
+#define crc32_le_arch crc32_le_arch
+static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
{
if (!static_branch_likely(&have_crc32))
return crc32_le_base(crc, p, len);
while (len >= sizeof(u64)) {
@@ -61,13 +60,13 @@ u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
CRC32(crc, value, b);
}
return crc;
}
-EXPORT_SYMBOL(crc32_le_arch);
-u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
+#define crc32c_arch crc32c_arch
+static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
{
if (!static_branch_likely(&have_crc32))
return crc32c_base(crc, p, len);
while (len >= sizeof(u64)) {
@@ -98,38 +97,19 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
CRC32C(crc, value, b);
}
return crc;
}
-EXPORT_SYMBOL(crc32c_arch);
-u32 crc32_be_arch(u32 crc, const u8 *p, size_t len)
-{
- return crc32_be_base(crc, p, len);
-}
-EXPORT_SYMBOL(crc32_be_arch);
-
-static int __init crc32_loongarch_init(void)
+#define crc32_mod_init_arch crc32_mod_init_arch
+static inline void crc32_mod_init_arch(void)
{
if (cpu_has_crc32)
static_branch_enable(&have_crc32);
- return 0;
}
-subsys_initcall(crc32_loongarch_init);
-static void __exit crc32_loongarch_exit(void)
-{
-}
-module_exit(crc32_loongarch_exit);
-
-u32 crc32_optimizations(void)
+static inline u32 crc32_optimizations_arch(void)
{
if (static_key_enabled(&have_crc32))
return CRC32_LE_OPTIMIZATION | CRC32C_OPTIMIZATION;
return 0;
}
-EXPORT_SYMBOL(crc32_optimizations);
-
-MODULE_AUTHOR("Min Zhou <zhoumin@loongson.cn>");
-MODULE_AUTHOR("Huacai Chen <chenhuacai@loongson.cn>");
-MODULE_DESCRIPTION("CRC32 and CRC32C using LoongArch crc* instructions");
-MODULE_LICENSE("GPL v2");
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 08/13] lib/crc/mips: migrate mips-optimized CRC code into lib/crc/
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
` (6 preceding siblings ...)
2025-06-01 22:44 ` [PATCH 07/13] lib/crc/loongarch: migrate loongarch-optimized " Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 09/13] lib/crc/powerpc: migrate powerpc-optimized " Eric Biggers
` (5 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Move the mips-optimized CRC code from arch/mips/lib/crc* into its new
location in lib/crc/mips/, and wire it up in the new way. For a
detailed explanation of why this change is being made, see the commit
that introduced the new way of integrating arch-specific code into
lib/crc/.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/mips/Kconfig | 1 -
arch/mips/lib/Makefile | 2 --
lib/crc/Kconfig | 1 +
.../lib/crc32-mips.c => lib/crc/mips/crc32.h | 35 ++++---------------
4 files changed, 8 insertions(+), 31 deletions(-)
rename arch/mips/lib/crc32-mips.c => lib/crc/mips/crc32.h (82%)
diff --git a/arch/mips/Kconfig b/arch/mips/Kconfig
index fc0772c1bad4a..31d6975674fd6 100644
--- a/arch/mips/Kconfig
+++ b/arch/mips/Kconfig
@@ -1995,11 +1995,10 @@ config CPU_MIPSR5
select MIPS_SPRAM
config CPU_MIPSR6
bool
default y if CPU_MIPS32_R6 || CPU_MIPS64_R6
- select ARCH_HAS_CRC32
select CPU_HAS_RIXI
select CPU_HAS_DIEI if !CPU_DIEI_BROKEN
select HAVE_ARCH_BITREVERSE
select MIPS_ASID_BITS_VARIABLE
select MIPS_SPRAM
diff --git a/arch/mips/lib/Makefile b/arch/mips/lib/Makefile
index 9d75845ef78e1..8c40ffb09c420 100644
--- a/arch/mips/lib/Makefile
+++ b/arch/mips/lib/Makefile
@@ -14,9 +14,7 @@ obj-$(CONFIG_PCI) += iomap-pci.o
lib-$(CONFIG_GENERIC_CSUM) := $(filter-out csum_partial.o, $(lib-y))
obj-$(CONFIG_CPU_GENERIC_DUMP_TLB) += dump_tlb.o
obj-$(CONFIG_CPU_R3000) += r3k_dump_tlb.o
-obj-$(CONFIG_CRC32_ARCH) += crc32-mips.o
-
# libgcc-style stuff needed in the kernel
obj-y += bswapsi.o bswapdi.o multi3.o
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index ee320ae19ca85..2d99aab4f838d 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -64,10 +64,11 @@ config CRC32_ARCH
bool
depends on CRC32 && CRC_OPTIMIZATIONS
default y if ARM && KERNEL_MODE_NEON
default y if ARM64
default y if LOONGARCH
+ default y if MIPS && CPU_MIPSR6
config CRC64
tristate
help
The CRC64 library functions. Select this if your module uses any of
diff --git a/arch/mips/lib/crc32-mips.c b/lib/crc/mips/crc32.h
similarity index 82%
rename from arch/mips/lib/crc32-mips.c
rename to lib/crc/mips/crc32.h
index 45e4d2c9fbf54..aa7f3228b7db3 100644
--- a/arch/mips/lib/crc32-mips.c
+++ b/lib/crc/mips/crc32.h
@@ -7,14 +7,10 @@
* Copyright (C) 2014 Linaro Ltd <yazen.ghannam@linaro.org>
* Copyright (C) 2018 MIPS Tech, LLC
*/
#include <linux/cpufeature.h>
-#include <linux/crc32.h>
-#include <linux/init.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
#include <asm/mipsregs.h>
#include <linux/unaligned.h>
#ifndef TOOLCHAIN_SUPPORTS_CRC
#define _ASM_SET_CRC(OP, SZ, TYPE) \
@@ -62,11 +58,12 @@ do { \
#define CRC32C(crc, value, size) \
_CRC32(crc, value, size, crc32c)
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32);
-u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
+#define crc32_le_arch crc32_le_arch
+static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
{
if (!static_branch_likely(&have_crc32))
return crc32_le_base(crc, p, len);
if (IS_ENABLED(CONFIG_64BIT)) {
@@ -104,13 +101,13 @@ u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
CRC32(crc, value, b);
}
return crc;
}
-EXPORT_SYMBOL(crc32_le_arch);
-u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
+#define crc32c_arch crc32c_arch
+static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
{
if (!static_branch_likely(&have_crc32))
return crc32c_base(crc, p, len);
if (IS_ENABLED(CONFIG_64BIT)) {
@@ -147,37 +144,19 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
CRC32C(crc, value, b);
}
return crc;
}
-EXPORT_SYMBOL(crc32c_arch);
-u32 crc32_be_arch(u32 crc, const u8 *p, size_t len)
-{
- return crc32_be_base(crc, p, len);
-}
-EXPORT_SYMBOL(crc32_be_arch);
-
-static int __init crc32_mips_init(void)
+#define crc32_mod_init_arch crc32_mod_init_arch
+static inline void crc32_mod_init_arch(void)
{
if (cpu_have_feature(cpu_feature(MIPS_CRC32)))
static_branch_enable(&have_crc32);
- return 0;
}
-subsys_initcall(crc32_mips_init);
-static void __exit crc32_mips_exit(void)
-{
-}
-module_exit(crc32_mips_exit);
-
-u32 crc32_optimizations(void)
+static inline u32 crc32_optimizations_arch(void)
{
if (static_key_enabled(&have_crc32))
return CRC32_LE_OPTIMIZATION | CRC32C_OPTIMIZATION;
return 0;
}
-EXPORT_SYMBOL(crc32_optimizations);
-
-MODULE_AUTHOR("Marcin Nowakowski <marcin.nowakowski@mips.com");
-MODULE_DESCRIPTION("CRC32 and CRC32C using optional MIPS instructions");
-MODULE_LICENSE("GPL v2");
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 09/13] lib/crc/powerpc: migrate powerpc-optimized CRC code into lib/crc/
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
` (7 preceding siblings ...)
2025-06-01 22:44 ` [PATCH 08/13] lib/crc/mips: migrate mips-optimized " Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 10/13] lib/crc/riscv: migrate riscv-optimized " Eric Biggers
` (4 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Move the powerpc-optimized CRC code from arch/powerpc/lib/crc* into its
new location in lib/crc/powerpc/, and wire it up in the new way. For a
detailed explanation of why this change is being made, see the commit
that introduced the new way of integrating arch-specific code into
lib/crc/.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/powerpc/Kconfig | 2 --
arch/powerpc/lib/Makefile | 6 ----
lib/crc/Kconfig | 2 ++
lib/crc/Makefile | 2 ++
.../crc/powerpc/crc-t10dif.h | 20 ++---------
.../crc/powerpc}/crc-vpmsum-template.S | 0
.../lib/crc32.c => lib/crc/powerpc/crc32.h | 36 +++----------------
.../crc/powerpc}/crc32c-vpmsum_asm.S | 0
.../crc/powerpc}/crct10dif-vpmsum_asm.S | 0
9 files changed, 12 insertions(+), 56 deletions(-)
rename arch/powerpc/lib/crc-t10dif.c => lib/crc/powerpc/crc-t10dif.h (75%)
rename {arch/powerpc/lib => lib/crc/powerpc}/crc-vpmsum-template.S (100%)
rename arch/powerpc/lib/crc32.c => lib/crc/powerpc/crc32.h (64%)
rename {arch/powerpc/lib => lib/crc/powerpc}/crc32c-vpmsum_asm.S (100%)
rename {arch/powerpc/lib => lib/crc/powerpc}/crct10dif-vpmsum_asm.S (100%)
diff --git a/arch/powerpc/Kconfig b/arch/powerpc/Kconfig
index c3e0cc83f1205..45b4fa7b9b02f 100644
--- a/arch/powerpc/Kconfig
+++ b/arch/powerpc/Kconfig
@@ -125,12 +125,10 @@ config PPC
select ARCH_DISABLE_KASAN_INLINE if PPC_RADIX_MMU
select ARCH_DMA_DEFAULT_COHERENT if !NOT_COHERENT_CACHE
select ARCH_ENABLE_MEMORY_HOTPLUG
select ARCH_ENABLE_MEMORY_HOTREMOVE
select ARCH_HAS_COPY_MC if PPC64
- select ARCH_HAS_CRC32 if PPC64 && ALTIVEC
- select ARCH_HAS_CRC_T10DIF if PPC64 && ALTIVEC
select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_HAS_DEBUG_VIRTUAL
select ARCH_HAS_DEBUG_VM_PGTABLE
select ARCH_HAS_DEBUG_WX if STRICT_KERNEL_RWX
select ARCH_HAS_DEVMEM_IS_ALLOWED
diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile
index 481f968e42c7b..59de2e2232df6 100644
--- a/arch/powerpc/lib/Makefile
+++ b/arch/powerpc/lib/Makefile
@@ -78,12 +78,6 @@ obj-$(CONFIG_FTR_FIXUP_SELFTEST) += feature-fixups-test.o
obj-$(CONFIG_ALTIVEC) += xor_vmx.o xor_vmx_glue.o
CFLAGS_xor_vmx.o += -mhard-float -maltivec $(call cc-option,-mabi=altivec)
# Enable <altivec.h>
CFLAGS_xor_vmx.o += -isystem $(shell $(CC) -print-file-name=include)
-obj-$(CONFIG_CRC32_ARCH) += crc32-powerpc.o
-crc32-powerpc-y := crc32.o crc32c-vpmsum_asm.o
-
-obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-powerpc.o
-crc-t10dif-powerpc-y := crc-t10dif.o crct10dif-vpmsum_asm.o
-
obj-$(CONFIG_PPC64) += $(obj64-y)
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index 2d99aab4f838d..06611610e478f 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -50,10 +50,11 @@ config ARCH_HAS_CRC_T10DIF
config CRC_T10DIF_ARCH
bool
depends on CRC_T10DIF && CRC_OPTIMIZATIONS
default y if ARM && KERNEL_MODE_NEON
default y if ARM64 && KERNEL_MODE_NEON
+ default y if PPC64 && ALTIVEC
config CRC32
tristate
select BITREVERSE
help
@@ -65,10 +66,11 @@ config CRC32_ARCH
depends on CRC32 && CRC_OPTIMIZATIONS
default y if ARM && KERNEL_MODE_NEON
default y if ARM64
default y if LOONGARCH
default y if MIPS && CPU_MIPSR6
+ default y if PPC64 && ALTIVEC
config CRC64
tristate
help
The CRC64 library functions. Select this if your module uses any of
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index 822d18b57b432..f9109b71c1264 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -14,17 +14,19 @@ obj-$(CONFIG_CRC_ITU_T) += crc-itu-t.o
obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o
crc-t10dif-y := crc-t10dif-main.o
ifeq ($(CONFIG_CRC_T10DIF_ARCH),y)
crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o
crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o
+crc-t10dif-$(CONFIG_PPC) += powerpc/crct10dif-vpmsum_asm.o
endif
obj-$(CONFIG_CRC32) += crc32.o
crc32-y := crc32-main.o
ifeq ($(CONFIG_CRC32_ARCH),y)
crc32-$(CONFIG_ARM) += arm/crc32-core.o
crc32-$(CONFIG_ARM64) += arm64/crc32-core.o
+crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o
endif
obj-$(CONFIG_CRC64) += crc64.o
crc64-y := crc64-main.o
ifeq ($(CONFIG_CRC64_ARCH),y)
diff --git a/arch/powerpc/lib/crc-t10dif.c b/lib/crc/powerpc/crc-t10dif.h
similarity index 75%
rename from arch/powerpc/lib/crc-t10dif.c
rename to lib/crc/powerpc/crc-t10dif.h
index be23ded3a9df6..59e16804a6eae 100644
--- a/arch/powerpc/lib/crc-t10dif.c
+++ b/lib/crc/powerpc/crc-t10dif.h
@@ -7,14 +7,11 @@
*/
#include <asm/switch_to.h>
#include <crypto/internal/simd.h>
#include <linux/cpufeature.h>
-#include <linux/crc-t10dif.h>
#include <linux/jump_label.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
#include <linux/preempt.h>
#include <linux/uaccess.h>
#define VMX_ALIGN 16
#define VMX_ALIGN_MASK (VMX_ALIGN-1)
@@ -23,11 +20,11 @@
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_vec_crypto);
u32 __crct10dif_vpmsum(u32 crc, unsigned char const *p, size_t len);
-u16 crc_t10dif_arch(u16 crci, const u8 *p, size_t len)
+static inline u16 crc_t10dif_arch(u16 crci, const u8 *p, size_t len)
{
unsigned int prealign;
unsigned int tail;
u32 crc = crci;
@@ -60,24 +57,13 @@ u16 crc_t10dif_arch(u16 crci, const u8 *p, size_t len)
crc = crc_t10dif_generic(crc, p, tail);
}
return crc & 0xffff;
}
-EXPORT_SYMBOL(crc_t10dif_arch);
-static int __init crc_t10dif_powerpc_init(void)
+#define crc_t10dif_mod_init_arch crc_t10dif_mod_init_arch
+static inline void crc_t10dif_mod_init_arch(void)
{
if (cpu_has_feature(CPU_FTR_ARCH_207S) &&
(cur_cpu_spec->cpu_user_features2 & PPC_FEATURE2_VEC_CRYPTO))
static_branch_enable(&have_vec_crypto);
- return 0;
}
-subsys_initcall(crc_t10dif_powerpc_init);
-
-static void __exit crc_t10dif_powerpc_exit(void)
-{
-}
-module_exit(crc_t10dif_powerpc_exit);
-
-MODULE_AUTHOR("Daniel Axtens <dja@axtens.net>");
-MODULE_DESCRIPTION("CRCT10DIF using vector polynomial multiply-sum instructions");
-MODULE_LICENSE("GPL");
diff --git a/arch/powerpc/lib/crc-vpmsum-template.S b/lib/crc/powerpc/crc-vpmsum-template.S
similarity index 100%
rename from arch/powerpc/lib/crc-vpmsum-template.S
rename to lib/crc/powerpc/crc-vpmsum-template.S
diff --git a/arch/powerpc/lib/crc32.c b/lib/crc/powerpc/crc32.h
similarity index 64%
rename from arch/powerpc/lib/crc32.c
rename to lib/crc/powerpc/crc32.h
index 0d9befb6e7b83..0dceb04d0ee94 100644
--- a/arch/powerpc/lib/crc32.c
+++ b/lib/crc/powerpc/crc32.h
@@ -1,13 +1,10 @@
// SPDX-License-Identifier: GPL-2.0-only
#include <asm/switch_to.h>
#include <crypto/internal/simd.h>
#include <linux/cpufeature.h>
-#include <linux/crc32.h>
#include <linux/jump_label.h>
-#include <linux/kernel.h>
-#include <linux/module.h>
#include <linux/preempt.h>
#include <linux/uaccess.h>
#define VMX_ALIGN 16
#define VMX_ALIGN_MASK (VMX_ALIGN-1)
@@ -16,17 +13,12 @@
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_vec_crypto);
u32 __crc32c_vpmsum(u32 crc, const u8 *p, size_t len);
-u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
-{
- return crc32_le_base(crc, p, len);
-}
-EXPORT_SYMBOL(crc32_le_arch);
-
-u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
+#define crc32c_arch crc32c_arch
+static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
{
unsigned int prealign;
unsigned int tail;
if (len < (VECTOR_BREAKPOINT + VMX_ALIGN) ||
@@ -56,38 +48,20 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
crc = crc32c_base(crc, p, tail);
}
return crc;
}
-EXPORT_SYMBOL(crc32c_arch);
-
-u32 crc32_be_arch(u32 crc, const u8 *p, size_t len)
-{
- return crc32_be_base(crc, p, len);
-}
-EXPORT_SYMBOL(crc32_be_arch);
-static int __init crc32_powerpc_init(void)
+#define crc32_mod_init_arch crc32_mod_init_arch
+static inline void crc32_mod_init_arch(void)
{
if (cpu_has_feature(CPU_FTR_ARCH_207S) &&
(cur_cpu_spec->cpu_user_features2 & PPC_FEATURE2_VEC_CRYPTO))
static_branch_enable(&have_vec_crypto);
- return 0;
}
-subsys_initcall(crc32_powerpc_init);
-static void __exit crc32_powerpc_exit(void)
-{
-}
-module_exit(crc32_powerpc_exit);
-
-u32 crc32_optimizations(void)
+static inline u32 crc32_optimizations_arch(void)
{
if (static_key_enabled(&have_vec_crypto))
return CRC32C_OPTIMIZATION;
return 0;
}
-EXPORT_SYMBOL(crc32_optimizations);
-
-MODULE_AUTHOR("Anton Blanchard <anton@samba.org>");
-MODULE_DESCRIPTION("CRC32C using vector polynomial multiply-sum instructions");
-MODULE_LICENSE("GPL");
diff --git a/arch/powerpc/lib/crc32c-vpmsum_asm.S b/lib/crc/powerpc/crc32c-vpmsum_asm.S
similarity index 100%
rename from arch/powerpc/lib/crc32c-vpmsum_asm.S
rename to lib/crc/powerpc/crc32c-vpmsum_asm.S
diff --git a/arch/powerpc/lib/crct10dif-vpmsum_asm.S b/lib/crc/powerpc/crct10dif-vpmsum_asm.S
similarity index 100%
rename from arch/powerpc/lib/crct10dif-vpmsum_asm.S
rename to lib/crc/powerpc/crct10dif-vpmsum_asm.S
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 10/13] lib/crc/riscv: migrate riscv-optimized CRC code into lib/crc/
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
` (8 preceding siblings ...)
2025-06-01 22:44 ` [PATCH 09/13] lib/crc/powerpc: migrate powerpc-optimized " Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 11/13] lib/crc/s390: migrate s390-optimized CRC code into lib/s390/ Eric Biggers
` (3 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Move the riscv-optimized CRC code from arch/riscv/lib/crc* into its new
location in lib/crc/riscv/, and wire it up in the new way. For a
detailed explanation of why this change is being made, see the commit
that introduced the new way of integrating arch-specific code into
lib/crc/.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/riscv/Kconfig | 3 ---
arch/riscv/lib/Makefile | 6 ------
lib/crc/Kconfig | 3 +++
lib/crc/Makefile | 3 +++
.../lib => lib/crc/riscv}/crc-clmul-consts.h | 0
.../crc/riscv}/crc-clmul-template.h | 0
{arch/riscv/lib => lib/crc/riscv}/crc-clmul.h | 0
.../crc/riscv/crc-t10dif.h | 8 +-------
{arch/riscv/lib => lib/crc/riscv}/crc16_msb.c | 0
.../lib/crc32.c => lib/crc/riscv/crc32.h | 20 +++++++------------
{arch/riscv/lib => lib/crc/riscv}/crc32_lsb.c | 0
{arch/riscv/lib => lib/crc/riscv}/crc32_msb.c | 0
.../lib/crc64.c => lib/crc/riscv/crc64.h | 11 ++--------
{arch/riscv/lib => lib/crc/riscv}/crc64_lsb.c | 0
{arch/riscv/lib => lib/crc/riscv}/crc64_msb.c | 0
15 files changed, 16 insertions(+), 38 deletions(-)
rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul-consts.h (100%)
rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul-template.h (100%)
rename {arch/riscv/lib => lib/crc/riscv}/crc-clmul.h (100%)
rename arch/riscv/lib/crc-t10dif.c => lib/crc/riscv/crc-t10dif.h (62%)
rename {arch/riscv/lib => lib/crc/riscv}/crc16_msb.c (100%)
rename arch/riscv/lib/crc32.c => lib/crc/riscv/crc32.h (66%)
rename {arch/riscv/lib => lib/crc/riscv}/crc32_lsb.c (100%)
rename {arch/riscv/lib => lib/crc/riscv}/crc32_msb.c (100%)
rename arch/riscv/lib/crc64.c => lib/crc/riscv/crc64.h (65%)
rename {arch/riscv/lib => lib/crc/riscv}/crc64_lsb.c (100%)
rename {arch/riscv/lib => lib/crc/riscv}/crc64_msb.c (100%)
diff --git a/arch/riscv/Kconfig b/arch/riscv/Kconfig
index bbec87b793099..b533d9e5086a9 100644
--- a/arch/riscv/Kconfig
+++ b/arch/riscv/Kconfig
@@ -22,13 +22,10 @@ config RISCV
select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM_VMEMMAP
select ARCH_ENABLE_MEMORY_HOTREMOVE if MEMORY_HOTPLUG
select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
select ARCH_ENABLE_THP_MIGRATION if TRANSPARENT_HUGEPAGE
select ARCH_HAS_BINFMT_FLAT
- select ARCH_HAS_CRC32 if RISCV_ISA_ZBC
- select ARCH_HAS_CRC64 if 64BIT && RISCV_ISA_ZBC
- select ARCH_HAS_CRC_T10DIF if RISCV_ISA_ZBC
select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_HAS_DEBUG_VIRTUAL if MMU
select ARCH_HAS_DEBUG_VM_PGTABLE
select ARCH_HAS_DEBUG_WX
select ARCH_HAS_FAST_MULTIPLIER
diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile
index 0baec92d2f55b..a4f4b48ed3a47 100644
--- a/arch/riscv/lib/Makefile
+++ b/arch/riscv/lib/Makefile
@@ -14,14 +14,8 @@ ifeq ($(CONFIG_MMU), y)
lib-$(CONFIG_RISCV_ISA_V) += uaccess_vector.o
endif
lib-$(CONFIG_MMU) += uaccess.o
lib-$(CONFIG_64BIT) += tishift.o
lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o
-obj-$(CONFIG_CRC32_ARCH) += crc32-riscv.o
-crc32-riscv-y := crc32.o crc32_msb.o crc32_lsb.o
-obj-$(CONFIG_CRC64_ARCH) += crc64-riscv.o
-crc64-riscv-y := crc64.o crc64_msb.o crc64_lsb.o
-obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-riscv.o
-crc-t10dif-riscv-y := crc-t10dif.o crc16_msb.o
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
lib-$(CONFIG_RISCV_ISA_V) += xor.o
lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index 06611610e478f..4044e08cb9739 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -51,10 +51,11 @@ config CRC_T10DIF_ARCH
bool
depends on CRC_T10DIF && CRC_OPTIMIZATIONS
default y if ARM && KERNEL_MODE_NEON
default y if ARM64 && KERNEL_MODE_NEON
default y if PPC64 && ALTIVEC
+ default y if RISCV && RISCV_ISA_ZBC
config CRC32
tristate
select BITREVERSE
help
@@ -67,20 +68,22 @@ config CRC32_ARCH
default y if ARM && KERNEL_MODE_NEON
default y if ARM64
default y if LOONGARCH
default y if MIPS && CPU_MIPSR6
default y if PPC64 && ALTIVEC
+ default y if RISCV && RISCV_ISA_ZBC
config CRC64
tristate
help
The CRC64 library functions. Select this if your module uses any of
the functions from <linux/crc64.h>.
config CRC64_ARCH
bool
depends on CRC64 && CRC_OPTIMIZATIONS
+ default y if RISCV && RISCV_ISA_ZBC && 64BIT
config CRC_OPTIMIZATIONS
bool "Enable optimized CRC implementations" if EXPERT
default y
help
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index f9109b71c1264..e46b35a2ffc04 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -15,23 +15,26 @@ obj-$(CONFIG_CRC_T10DIF) += crc-t10dif.o
crc-t10dif-y := crc-t10dif-main.o
ifeq ($(CONFIG_CRC_T10DIF_ARCH),y)
crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o
crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o
crc-t10dif-$(CONFIG_PPC) += powerpc/crct10dif-vpmsum_asm.o
+crc-t10dif-$(CONFIG_RISCV) += riscv/crc16_msb.o
endif
obj-$(CONFIG_CRC32) += crc32.o
crc32-y := crc32-main.o
ifeq ($(CONFIG_CRC32_ARCH),y)
crc32-$(CONFIG_ARM) += arm/crc32-core.o
crc32-$(CONFIG_ARM64) += arm64/crc32-core.o
crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o
+crc32-$(CONFIG_RISCV) += riscv/crc32_lsb.o riscv/crc32_msb.o
endif
obj-$(CONFIG_CRC64) += crc64.o
crc64-y := crc64-main.o
ifeq ($(CONFIG_CRC64_ARCH),y)
+crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o
endif
obj-y += tests/
hostprogs := gen_crc32table gen_crc64table
diff --git a/arch/riscv/lib/crc-clmul-consts.h b/lib/crc/riscv/crc-clmul-consts.h
similarity index 100%
rename from arch/riscv/lib/crc-clmul-consts.h
rename to lib/crc/riscv/crc-clmul-consts.h
diff --git a/arch/riscv/lib/crc-clmul-template.h b/lib/crc/riscv/crc-clmul-template.h
similarity index 100%
rename from arch/riscv/lib/crc-clmul-template.h
rename to lib/crc/riscv/crc-clmul-template.h
diff --git a/arch/riscv/lib/crc-clmul.h b/lib/crc/riscv/crc-clmul.h
similarity index 100%
rename from arch/riscv/lib/crc-clmul.h
rename to lib/crc/riscv/crc-clmul.h
diff --git a/arch/riscv/lib/crc-t10dif.c b/lib/crc/riscv/crc-t10dif.h
similarity index 62%
rename from arch/riscv/lib/crc-t10dif.c
rename to lib/crc/riscv/crc-t10dif.h
index e6b0051ccd86c..cd6136cbfda1c 100644
--- a/arch/riscv/lib/crc-t10dif.c
+++ b/lib/crc/riscv/crc-t10dif.h
@@ -5,20 +5,14 @@
* Copyright 2025 Google LLC
*/
#include <asm/hwcap.h>
#include <asm/alternative-macros.h>
-#include <linux/crc-t10dif.h>
-#include <linux/module.h>
#include "crc-clmul.h"
-u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len)
+static inline u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len)
{
if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC))
return crc16_msb_clmul(crc, p, len, &crc16_msb_0x8bb7_consts);
return crc_t10dif_generic(crc, p, len);
}
-EXPORT_SYMBOL(crc_t10dif_arch);
-
-MODULE_DESCRIPTION("RISC-V optimized CRC-T10DIF function");
-MODULE_LICENSE("GPL");
diff --git a/arch/riscv/lib/crc16_msb.c b/lib/crc/riscv/crc16_msb.c
similarity index 100%
rename from arch/riscv/lib/crc16_msb.c
rename to lib/crc/riscv/crc16_msb.c
diff --git a/arch/riscv/lib/crc32.c b/lib/crc/riscv/crc32.h
similarity index 66%
rename from arch/riscv/lib/crc32.c
rename to lib/crc/riscv/crc32.h
index a3188b7d9c403..08c33357d25cd 100644
--- a/arch/riscv/lib/crc32.c
+++ b/lib/crc/riscv/crc32.h
@@ -5,49 +5,43 @@
* Copyright 2025 Google LLC
*/
#include <asm/hwcap.h>
#include <asm/alternative-macros.h>
-#include <linux/crc32.h>
-#include <linux/module.h>
#include "crc-clmul.h"
-u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
+#define crc32_le_arch crc32_le_arch
+static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
{
if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC))
return crc32_lsb_clmul(crc, p, len,
&crc32_lsb_0xedb88320_consts);
return crc32_le_base(crc, p, len);
}
-EXPORT_SYMBOL(crc32_le_arch);
-u32 crc32_be_arch(u32 crc, const u8 *p, size_t len)
+#define crc32_be_arch crc32_be_arch
+static inline u32 crc32_be_arch(u32 crc, const u8 *p, size_t len)
{
if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC))
return crc32_msb_clmul(crc, p, len,
&crc32_msb_0x04c11db7_consts);
return crc32_be_base(crc, p, len);
}
-EXPORT_SYMBOL(crc32_be_arch);
-u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
+#define crc32c_arch crc32c_arch
+static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
{
if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC))
return crc32_lsb_clmul(crc, p, len,
&crc32_lsb_0x82f63b78_consts);
return crc32c_base(crc, p, len);
}
-EXPORT_SYMBOL(crc32c_arch);
-u32 crc32_optimizations(void)
+static inline u32 crc32_optimizations_arch(void)
{
if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC))
return CRC32_LE_OPTIMIZATION |
CRC32_BE_OPTIMIZATION |
CRC32C_OPTIMIZATION;
return 0;
}
-EXPORT_SYMBOL(crc32_optimizations);
-
-MODULE_DESCRIPTION("RISC-V optimized CRC32 functions");
-MODULE_LICENSE("GPL");
diff --git a/arch/riscv/lib/crc32_lsb.c b/lib/crc/riscv/crc32_lsb.c
similarity index 100%
rename from arch/riscv/lib/crc32_lsb.c
rename to lib/crc/riscv/crc32_lsb.c
diff --git a/arch/riscv/lib/crc32_msb.c b/lib/crc/riscv/crc32_msb.c
similarity index 100%
rename from arch/riscv/lib/crc32_msb.c
rename to lib/crc/riscv/crc32_msb.c
diff --git a/arch/riscv/lib/crc64.c b/lib/crc/riscv/crc64.h
similarity index 65%
rename from arch/riscv/lib/crc64.c
rename to lib/crc/riscv/crc64.h
index f0015a27836a4..a1b7873fde579 100644
--- a/arch/riscv/lib/crc64.c
+++ b/lib/crc/riscv/crc64.h
@@ -5,30 +5,23 @@
* Copyright 2025 Google LLC
*/
#include <asm/hwcap.h>
#include <asm/alternative-macros.h>
-#include <linux/crc64.h>
-#include <linux/module.h>
#include "crc-clmul.h"
-u64 crc64_be_arch(u64 crc, const u8 *p, size_t len)
+static inline u64 crc64_be_arch(u64 crc, const u8 *p, size_t len)
{
if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC))
return crc64_msb_clmul(crc, p, len,
&crc64_msb_0x42f0e1eba9ea3693_consts);
return crc64_be_generic(crc, p, len);
}
-EXPORT_SYMBOL(crc64_be_arch);
-u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)
+static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)
{
if (riscv_has_extension_likely(RISCV_ISA_EXT_ZBC))
return crc64_lsb_clmul(crc, p, len,
&crc64_lsb_0x9a6c9329ac4bc9b5_consts);
return crc64_nvme_generic(crc, p, len);
}
-EXPORT_SYMBOL(crc64_nvme_arch);
-
-MODULE_DESCRIPTION("RISC-V optimized CRC64 functions");
-MODULE_LICENSE("GPL");
diff --git a/arch/riscv/lib/crc64_lsb.c b/lib/crc/riscv/crc64_lsb.c
similarity index 100%
rename from arch/riscv/lib/crc64_lsb.c
rename to lib/crc/riscv/crc64_lsb.c
diff --git a/arch/riscv/lib/crc64_msb.c b/lib/crc/riscv/crc64_msb.c
similarity index 100%
rename from arch/riscv/lib/crc64_msb.c
rename to lib/crc/riscv/crc64_msb.c
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 11/13] lib/crc/s390: migrate s390-optimized CRC code into lib/s390/
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
` (9 preceding siblings ...)
2025-06-01 22:44 ` [PATCH 10/13] lib/crc/riscv: migrate riscv-optimized " Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 12/13] lib/crc/sparc: migrate sparc-optimized CRC code into lib/crc/ Eric Biggers
` (2 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Move the s390-optimized CRC code from arch/s390/lib/crc* into its new
location in lib/crc/s390/, and wire it up in the new way. For a
detailed explanation of why this change is being made, see the commit
that introduced the new way of integrating arch-specific code into
lib/crc/.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/s390/Kconfig | 1 -
arch/s390/lib/Makefile | 3 ---
lib/crc/Kconfig | 1 +
lib/crc/Makefile | 1 +
{arch/s390/lib => lib/crc/s390}/crc32-vx.h | 0
arch/s390/lib/crc32.c => lib/crc/s390/crc32.h | 19 ++++++-------------
{arch/s390/lib => lib/crc/s390}/crc32be-vx.c | 0
{arch/s390/lib => lib/crc/s390}/crc32le-vx.c | 0
8 files changed, 8 insertions(+), 17 deletions(-)
rename {arch/s390/lib => lib/crc/s390}/crc32-vx.h (100%)
rename arch/s390/lib/crc32.c => lib/crc/s390/crc32.h (81%)
rename {arch/s390/lib => lib/crc/s390}/crc32be-vx.c (100%)
rename {arch/s390/lib => lib/crc/s390}/crc32le-vx.c (100%)
diff --git a/arch/s390/Kconfig b/arch/s390/Kconfig
index 0c16dc443e2f6..22b90f6aa1a09 100644
--- a/arch/s390/Kconfig
+++ b/arch/s390/Kconfig
@@ -73,11 +73,10 @@ config S390
select ARCH_CORRECT_STACKTRACE_ON_KRETPROBE
select ARCH_ENABLE_MEMORY_HOTPLUG if SPARSEMEM
select ARCH_ENABLE_MEMORY_HOTREMOVE
select ARCH_ENABLE_SPLIT_PMD_PTLOCK if PGTABLE_LEVELS > 2
select ARCH_HAS_CPU_FINALIZE_INIT
- select ARCH_HAS_CRC32
select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_HAS_DEBUG_VIRTUAL
select ARCH_HAS_DEBUG_VM_PGTABLE
select ARCH_HAS_DEBUG_WX
select ARCH_HAS_DEVMEM_IS_ALLOWED
diff --git a/arch/s390/lib/Makefile b/arch/s390/lib/Makefile
index cd35cdbfa8713..7c8583d46eca1 100644
--- a/arch/s390/lib/Makefile
+++ b/arch/s390/lib/Makefile
@@ -23,8 +23,5 @@ obj-$(CONFIG_S390_MODULES_SANITY_TEST) += test_modules.o
obj-$(CONFIG_S390_MODULES_SANITY_TEST_HELPERS) += test_modules_helpers.o
lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
obj-$(CONFIG_EXPOLINE_EXTERN) += expoline.o
-
-obj-$(CONFIG_CRC32_ARCH) += crc32-s390.o
-crc32-s390-y := crc32.o crc32le-vx.o crc32be-vx.o
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index 4044e08cb9739..1ed6e11bef909 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -69,10 +69,11 @@ config CRC32_ARCH
default y if ARM64
default y if LOONGARCH
default y if MIPS && CPU_MIPSR6
default y if PPC64 && ALTIVEC
default y if RISCV && RISCV_ISA_ZBC
+ default y if S390
config CRC64
tristate
help
The CRC64 library functions. Select this if your module uses any of
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index e46b35a2ffc04..1b83262b6987f 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -25,10 +25,11 @@ crc32-y := crc32-main.o
ifeq ($(CONFIG_CRC32_ARCH),y)
crc32-$(CONFIG_ARM) += arm/crc32-core.o
crc32-$(CONFIG_ARM64) += arm64/crc32-core.o
crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o
crc32-$(CONFIG_RISCV) += riscv/crc32_lsb.o riscv/crc32_msb.o
+crc32-$(CONFIG_S390) += s390/crc32le-vx.o s390/crc32be-vx.o
endif
obj-$(CONFIG_CRC64) += crc64.o
crc64-y := crc64-main.o
ifeq ($(CONFIG_CRC64_ARCH),y)
diff --git a/arch/s390/lib/crc32-vx.h b/lib/crc/s390/crc32-vx.h
similarity index 100%
rename from arch/s390/lib/crc32-vx.h
rename to lib/crc/s390/crc32-vx.h
diff --git a/arch/s390/lib/crc32.c b/lib/crc/s390/crc32.h
similarity index 81%
rename from arch/s390/lib/crc32.c
rename to lib/crc/s390/crc32.h
index 3c4b344417c11..8dbb07b9ea64c 100644
--- a/arch/s390/lib/crc32.c
+++ b/lib/crc/s390/crc32.h
@@ -3,16 +3,12 @@
* CRC-32 implemented with the z/Architecture Vector Extension Facility.
*
* Copyright IBM Corp. 2015
* Author(s): Hendrik Brueckner <brueckner@linux.vnet.ibm.com>
*/
-#define KMSG_COMPONENT "crc32-vx"
-#define pr_fmt(fmt) KMSG_COMPONENT ": " fmt
-#include <linux/module.h>
#include <linux/cpufeature.h>
-#include <linux/crc32.h>
#include <asm/fpu.h>
#include "crc32-vx.h"
#define VX_MIN_LEN 64
#define VX_ALIGNMENT 16L
@@ -25,11 +21,11 @@
* on the message buffer, the hardware-accelerated or software implementation
* is used. Note that the message buffer is aligned to improve fetch
* operations of VECTOR LOAD MULTIPLE instructions.
*/
#define DEFINE_CRC32_VX(___fname, ___crc32_vx, ___crc32_sw) \
- u32 ___fname(u32 crc, const u8 *data, size_t datalen) \
+ static inline u32 ___fname(u32 crc, const u8 *data, size_t datalen) \
{ \
unsigned long prealign, aligned, remaining; \
DECLARE_KERNEL_FPU_ONSTACK16(vxstate); \
\
if (datalen < VX_MIN_LEN + VX_ALIGN_MASK || !cpu_has_vx()) \
@@ -52,26 +48,23 @@
\
if (remaining) \
crc = ___crc32_sw(crc, data + aligned, remaining); \
\
return crc; \
- } \
- EXPORT_SYMBOL(___fname);
+ }
+#define crc32_le_arch crc32_le_arch
+#define crc32_be_arch crc32_be_arch
+#define crc32c_arch crc32c_arch
DEFINE_CRC32_VX(crc32_le_arch, crc32_le_vgfm_16, crc32_le_base)
DEFINE_CRC32_VX(crc32_be_arch, crc32_be_vgfm_16, crc32_be_base)
DEFINE_CRC32_VX(crc32c_arch, crc32c_le_vgfm_16, crc32c_base)
-u32 crc32_optimizations(void)
+static inline u32 crc32_optimizations_arch(void)
{
if (cpu_has_vx()) {
return CRC32_LE_OPTIMIZATION |
CRC32_BE_OPTIMIZATION |
CRC32C_OPTIMIZATION;
}
return 0;
}
-EXPORT_SYMBOL(crc32_optimizations);
-
-MODULE_AUTHOR("Hendrik Brueckner <brueckner@linux.vnet.ibm.com>");
-MODULE_DESCRIPTION("CRC-32 algorithms using z/Architecture Vector Extension Facility");
-MODULE_LICENSE("GPL");
diff --git a/arch/s390/lib/crc32be-vx.c b/lib/crc/s390/crc32be-vx.c
similarity index 100%
rename from arch/s390/lib/crc32be-vx.c
rename to lib/crc/s390/crc32be-vx.c
diff --git a/arch/s390/lib/crc32le-vx.c b/lib/crc/s390/crc32le-vx.c
similarity index 100%
rename from arch/s390/lib/crc32le-vx.c
rename to lib/crc/s390/crc32le-vx.c
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 12/13] lib/crc/sparc: migrate sparc-optimized CRC code into lib/crc/
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
` (10 preceding siblings ...)
2025-06-01 22:44 ` [PATCH 11/13] lib/crc/s390: migrate s390-optimized CRC code into lib/s390/ Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-01 22:44 ` [PATCH 13/13] lib/crc/x86: migrate x86-optimized " Eric Biggers
2025-06-08 23:46 ` [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Move the sparc-optimized CRC code from arch/sparc/lib/crc* into its new
location in lib/crc/sparc/, and wire it up in the new way. For a
detailed explanation of why this change is being made, see the commit
that introduced the new way of integrating arch-specific code into
lib/crc/.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/sparc/Kconfig | 1 -
arch/sparc/lib/Makefile | 2 -
lib/crc/Kconfig | 1 +
lib/crc/Makefile | 1 +
.../lib/crc32.c => lib/crc/sparc/crc32.h | 40 ++++---------------
.../sparc/lib => lib/crc/sparc}/crc32c_asm.S | 0
6 files changed, 9 insertions(+), 36 deletions(-)
rename arch/sparc/lib/crc32.c => lib/crc/sparc/crc32.h (62%)
rename {arch/sparc/lib => lib/crc/sparc}/crc32c_asm.S (100%)
diff --git a/arch/sparc/Kconfig b/arch/sparc/Kconfig
index 0f88123925a4f..dcfdb7f1dae97 100644
--- a/arch/sparc/Kconfig
+++ b/arch/sparc/Kconfig
@@ -108,11 +108,10 @@ config SPARC64
select ARCH_HAS_GIGANTIC_PAGE
select HAVE_SOFTIRQ_ON_OWN_STACK
select HAVE_SETUP_PER_CPU_AREA
select NEED_PER_CPU_EMBED_FIRST_CHUNK
select NEED_PER_CPU_PAGE_FIRST_CHUNK
- select ARCH_HAS_CRC32
config ARCH_PROC_KCORE_TEXT
def_bool y
config CPU_BIG_ENDIAN
diff --git a/arch/sparc/lib/Makefile b/arch/sparc/lib/Makefile
index 5cf9781d68b40..2d6c3c5352734 100644
--- a/arch/sparc/lib/Makefile
+++ b/arch/sparc/lib/Makefile
@@ -52,7 +52,5 @@ lib-$(CONFIG_SPARC64) += copy_in_user.o memmove.o
lib-$(CONFIG_SPARC64) += mcount.o ipcsum.o xor.o hweight.o ffs.o
obj-$(CONFIG_SPARC64) += iomap.o
obj-$(CONFIG_SPARC32) += atomic32.o
obj-$(CONFIG_SPARC64) += PeeCeeI.o
-obj-$(CONFIG_CRC32_ARCH) += crc32-sparc.o
-crc32-sparc-y := crc32.o crc32c_asm.o
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index 1ed6e11bef909..c0b36d03b66dc 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -70,10 +70,11 @@ config CRC32_ARCH
default y if LOONGARCH
default y if MIPS && CPU_MIPSR6
default y if PPC64 && ALTIVEC
default y if RISCV && RISCV_ISA_ZBC
default y if S390
+ default y if SPARC64
config CRC64
tristate
help
The CRC64 library functions. Select this if your module uses any of
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index 1b83262b6987f..37dd4a776d1b4 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -26,10 +26,11 @@ ifeq ($(CONFIG_CRC32_ARCH),y)
crc32-$(CONFIG_ARM) += arm/crc32-core.o
crc32-$(CONFIG_ARM64) += arm64/crc32-core.o
crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o
crc32-$(CONFIG_RISCV) += riscv/crc32_lsb.o riscv/crc32_msb.o
crc32-$(CONFIG_S390) += s390/crc32le-vx.o s390/crc32be-vx.o
+crc32-$(CONFIG_SPARC) += sparc/crc32c_asm.o
endif
obj-$(CONFIG_CRC64) += crc64.o
crc64-y := crc64-main.o
ifeq ($(CONFIG_CRC64_ARCH),y)
diff --git a/arch/sparc/lib/crc32.c b/lib/crc/sparc/crc32.h
similarity index 62%
rename from arch/sparc/lib/crc32.c
rename to lib/crc/sparc/crc32.h
index 40d4720a42a1b..43d52e60b2960 100644
--- a/arch/sparc/lib/crc32.c
+++ b/lib/crc/sparc/crc32.h
@@ -8,28 +8,19 @@
* Kent Liu <kent.liu@intel.com>
*/
#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt
-#include <linux/init.h>
-#include <linux/module.h>
-#include <linux/kernel.h>
-#include <linux/crc32.h>
#include <asm/pstate.h>
#include <asm/elf.h>
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32c_opcode);
-u32 crc32_le_arch(u32 crc, const u8 *data, size_t len)
-{
- return crc32_le_base(crc, data, len);
-}
-EXPORT_SYMBOL(crc32_le_arch);
-
void crc32c_sparc64(u32 *crcp, const u64 *data, size_t len);
-u32 crc32c_arch(u32 crc, const u8 *data, size_t len)
+#define crc32c_arch crc32c_arch
+static inline u32 crc32c_arch(u32 crc, const u8 *data, size_t len)
{
size_t n = -(uintptr_t)data & 7;
if (!static_branch_likely(&have_crc32c_opcode))
return crc32c_base(crc, data, len);
@@ -49,45 +40,28 @@ u32 crc32c_arch(u32 crc, const u8 *data, size_t len)
}
if (len)
crc = crc32c_base(crc, data, len);
return crc;
}
-EXPORT_SYMBOL(crc32c_arch);
-
-u32 crc32_be_arch(u32 crc, const u8 *data, size_t len)
-{
- return crc32_be_base(crc, data, len);
-}
-EXPORT_SYMBOL(crc32_be_arch);
-static int __init crc32_sparc_init(void)
+#define crc32_mod_init_arch crc32_mod_init_arch
+static inline void crc32_mod_init_arch(void)
{
unsigned long cfr;
if (!(sparc64_elf_hwcap & HWCAP_SPARC_CRYPTO))
- return 0;
+ return;
__asm__ __volatile__("rd %%asr26, %0" : "=r" (cfr));
if (!(cfr & CFR_CRC32C))
- return 0;
+ return;
static_branch_enable(&have_crc32c_opcode);
pr_info("Using sparc64 crc32c opcode optimized CRC32C implementation\n");
- return 0;
}
-subsys_initcall(crc32_sparc_init);
-static void __exit crc32_sparc_exit(void)
-{
-}
-module_exit(crc32_sparc_exit);
-
-u32 crc32_optimizations(void)
+static inline u32 crc32_optimizations_arch(void)
{
if (static_key_enabled(&have_crc32c_opcode))
return CRC32C_OPTIMIZATION;
return 0;
}
-EXPORT_SYMBOL(crc32_optimizations);
-
-MODULE_LICENSE("GPL");
-MODULE_DESCRIPTION("CRC32c (Castagnoli), sparc64 crc32c opcode accelerated");
diff --git a/arch/sparc/lib/crc32c_asm.S b/lib/crc/sparc/crc32c_asm.S
similarity index 100%
rename from arch/sparc/lib/crc32c_asm.S
rename to lib/crc/sparc/crc32c_asm.S
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* [PATCH 13/13] lib/crc/x86: migrate x86-optimized CRC code into lib/crc/
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
` (11 preceding siblings ...)
2025-06-01 22:44 ` [PATCH 12/13] lib/crc/sparc: migrate sparc-optimized CRC code into lib/crc/ Eric Biggers
@ 2025-06-01 22:44 ` Eric Biggers
2025-06-08 23:46 ` [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-01 22:44 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
From: Eric Biggers <ebiggers@google.com>
Move the x86-optimized CRC code from arch/x86/lib/crc* into its new
location in lib/crc/x86/, and wire it up in the new way. For a detailed
explanation of why this change is being made, see the commit that
introduced the new way of integrating arch-specific code into lib/crc/.
Signed-off-by: Eric Biggers <ebiggers@google.com>
---
arch/x86/Kconfig | 3 --
arch/x86/lib/Makefile | 10 ------
lib/crc/Kconfig | 3 ++
lib/crc/Makefile | 4 +++
.../lib => lib/crc/x86}/crc-pclmul-consts.h | 0
.../lib => lib/crc/x86}/crc-pclmul-template.S | 0
.../lib => lib/crc/x86}/crc-pclmul-template.h | 0
.../crc-t10dif.c => lib/crc/x86/crc-t10dif.h | 18 ++---------
.../lib => lib/crc/x86}/crc16-msb-pclmul.S | 0
{arch/x86/lib => lib/crc/x86}/crc32-pclmul.S | 0
arch/x86/lib/crc32.c => lib/crc/x86/crc32.h | 32 ++++---------------
{arch/x86/lib => lib/crc/x86}/crc32c-3way.S | 0
{arch/x86/lib => lib/crc/x86}/crc64-pclmul.S | 0
arch/x86/lib/crc64.c => lib/crc/x86/crc64.h | 21 +++---------
14 files changed, 21 insertions(+), 70 deletions(-)
rename {arch/x86/lib => lib/crc/x86}/crc-pclmul-consts.h (100%)
rename {arch/x86/lib => lib/crc/x86}/crc-pclmul-template.S (100%)
rename {arch/x86/lib => lib/crc/x86}/crc-pclmul-template.h (100%)
rename arch/x86/lib/crc-t10dif.c => lib/crc/x86/crc-t10dif.h (56%)
rename {arch/x86/lib => lib/crc/x86}/crc16-msb-pclmul.S (100%)
rename {arch/x86/lib => lib/crc/x86}/crc32-pclmul.S (100%)
rename arch/x86/lib/crc32.c => lib/crc/x86/crc32.h (76%)
rename {arch/x86/lib => lib/crc/x86}/crc32c-3way.S (100%)
rename {arch/x86/lib => lib/crc/x86}/crc64-pclmul.S (100%)
rename arch/x86/lib/crc64.c => lib/crc/x86/crc64.h (61%)
diff --git a/arch/x86/Kconfig b/arch/x86/Kconfig
index 340e5468980e0..6181f414f87ec 100644
--- a/arch/x86/Kconfig
+++ b/arch/x86/Kconfig
@@ -77,13 +77,10 @@ config X86
select ARCH_HAS_ACPI_TABLE_UPGRADE if ACPI
select ARCH_HAS_CACHE_LINE_SIZE
select ARCH_HAS_CPU_CACHE_INVALIDATE_MEMREGION
select ARCH_HAS_CPU_FINALIZE_INIT
select ARCH_HAS_CPU_PASID if IOMMU_SVA
- select ARCH_HAS_CRC32
- select ARCH_HAS_CRC64 if X86_64
- select ARCH_HAS_CRC_T10DIF
select ARCH_HAS_CURRENT_STACK_POINTER
select ARCH_HAS_DEBUG_VIRTUAL
select ARCH_HAS_DEBUG_VM_PGTABLE if !X86_PAE
select ARCH_HAS_DEVMEM_IS_ALLOWED
select ARCH_HAS_DMA_OPS if GART_IOMMU || XEN
diff --git a/arch/x86/lib/Makefile b/arch/x86/lib/Makefile
index 4fa5c4e1ba8a0..dc5ee2a6938c4 100644
--- a/arch/x86/lib/Makefile
+++ b/arch/x86/lib/Makefile
@@ -38,20 +38,10 @@ lib-$(CONFIG_ARCH_HAS_COPY_MC) += copy_mc.o copy_mc_64.o
lib-$(CONFIG_INSTRUCTION_DECODER) += insn.o inat.o insn-eval.o
lib-$(CONFIG_RANDOMIZE_BASE) += kaslr.o
lib-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
lib-$(CONFIG_MITIGATION_RETPOLINE) += retpoline.o
-obj-$(CONFIG_CRC32_ARCH) += crc32-x86.o
-crc32-x86-y := crc32.o crc32-pclmul.o
-crc32-x86-$(CONFIG_64BIT) += crc32c-3way.o
-
-obj-$(CONFIG_CRC64_ARCH) += crc64-x86.o
-crc64-x86-y := crc64.o crc64-pclmul.o
-
-obj-$(CONFIG_CRC_T10DIF_ARCH) += crc-t10dif-x86.o
-crc-t10dif-x86-y := crc-t10dif.o crc16-msb-pclmul.o
-
obj-y += msr.o msr-reg.o msr-reg-export.o hweight.o
obj-y += iomem.o
ifeq ($(CONFIG_X86_32),y)
obj-y += atomic64_32.o
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index c0b36d03b66dc..450c8290cbc1f 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -52,10 +52,11 @@ config CRC_T10DIF_ARCH
depends on CRC_T10DIF && CRC_OPTIMIZATIONS
default y if ARM && KERNEL_MODE_NEON
default y if ARM64 && KERNEL_MODE_NEON
default y if PPC64 && ALTIVEC
default y if RISCV && RISCV_ISA_ZBC
+ default y if X86
config CRC32
tristate
select BITREVERSE
help
@@ -71,10 +72,11 @@ config CRC32_ARCH
default y if MIPS && CPU_MIPSR6
default y if PPC64 && ALTIVEC
default y if RISCV && RISCV_ISA_ZBC
default y if S390
default y if SPARC64
+ default y if X86
config CRC64
tristate
help
The CRC64 library functions. Select this if your module uses any of
@@ -82,10 +84,11 @@ config CRC64
config CRC64_ARCH
bool
depends on CRC64 && CRC_OPTIMIZATIONS
default y if RISCV && RISCV_ISA_ZBC && 64BIT
+ default y if X86_64
config CRC_OPTIMIZATIONS
bool "Enable optimized CRC implementations" if EXPERT
default y
help
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index 37dd4a776d1b4..80cfc94f5180d 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -16,10 +16,11 @@ crc-t10dif-y := crc-t10dif-main.o
ifeq ($(CONFIG_CRC_T10DIF_ARCH),y)
crc-t10dif-$(CONFIG_ARM) += arm/crc-t10dif-core.o
crc-t10dif-$(CONFIG_ARM64) += arm64/crc-t10dif-core.o
crc-t10dif-$(CONFIG_PPC) += powerpc/crct10dif-vpmsum_asm.o
crc-t10dif-$(CONFIG_RISCV) += riscv/crc16_msb.o
+crc-t10dif-$(CONFIG_X86) += x86/crc16-msb-pclmul.o
endif
obj-$(CONFIG_CRC32) += crc32.o
crc32-y := crc32-main.o
ifeq ($(CONFIG_CRC32_ARCH),y)
@@ -27,16 +28,19 @@ crc32-$(CONFIG_ARM) += arm/crc32-core.o
crc32-$(CONFIG_ARM64) += arm64/crc32-core.o
crc32-$(CONFIG_PPC) += powerpc/crc32c-vpmsum_asm.o
crc32-$(CONFIG_RISCV) += riscv/crc32_lsb.o riscv/crc32_msb.o
crc32-$(CONFIG_S390) += s390/crc32le-vx.o s390/crc32be-vx.o
crc32-$(CONFIG_SPARC) += sparc/crc32c_asm.o
+crc32-$(CONFIG_X86) += x86/crc32-pclmul.o
+crc32-$(CONFIG_X86_64) += x86/crc32c-3way.o
endif
obj-$(CONFIG_CRC64) += crc64.o
crc64-y := crc64-main.o
ifeq ($(CONFIG_CRC64_ARCH),y)
crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o
+crc64-$(CONFIG_X86) += x86/crc64-pclmul.o
endif
obj-y += tests/
hostprogs := gen_crc32table gen_crc64table
diff --git a/arch/x86/lib/crc-pclmul-consts.h b/lib/crc/x86/crc-pclmul-consts.h
similarity index 100%
rename from arch/x86/lib/crc-pclmul-consts.h
rename to lib/crc/x86/crc-pclmul-consts.h
diff --git a/arch/x86/lib/crc-pclmul-template.S b/lib/crc/x86/crc-pclmul-template.S
similarity index 100%
rename from arch/x86/lib/crc-pclmul-template.S
rename to lib/crc/x86/crc-pclmul-template.S
diff --git a/arch/x86/lib/crc-pclmul-template.h b/lib/crc/x86/crc-pclmul-template.h
similarity index 100%
rename from arch/x86/lib/crc-pclmul-template.h
rename to lib/crc/x86/crc-pclmul-template.h
diff --git a/arch/x86/lib/crc-t10dif.c b/lib/crc/x86/crc-t10dif.h
similarity index 56%
rename from arch/x86/lib/crc-t10dif.c
rename to lib/crc/x86/crc-t10dif.h
index db7ce59c31ace..eb1f23db4daa1 100644
--- a/arch/x86/lib/crc-t10dif.c
+++ b/lib/crc/x86/crc-t10dif.h
@@ -3,38 +3,26 @@
* CRC-T10DIF using [V]PCLMULQDQ instructions
*
* Copyright 2024 Google LLC
*/
-#include <linux/crc-t10dif.h>
-#include <linux/module.h>
#include "crc-pclmul-template.h"
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pclmulqdq);
DECLARE_CRC_PCLMUL_FUNCS(crc16_msb, u16);
-u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len)
+static inline u16 crc_t10dif_arch(u16 crc, const u8 *p, size_t len)
{
CRC_PCLMUL(crc, p, len, crc16_msb, crc16_msb_0x8bb7_consts,
have_pclmulqdq);
return crc_t10dif_generic(crc, p, len);
}
-EXPORT_SYMBOL(crc_t10dif_arch);
-static int __init crc_t10dif_x86_init(void)
+#define crc_t10dif_mod_init_arch crc_t10dif_mod_init_arch
+static inline void crc_t10dif_mod_init_arch(void)
{
if (boot_cpu_has(X86_FEATURE_PCLMULQDQ)) {
static_branch_enable(&have_pclmulqdq);
INIT_CRC_PCLMUL(crc16_msb);
}
- return 0;
}
-subsys_initcall(crc_t10dif_x86_init);
-
-static void __exit crc_t10dif_x86_exit(void)
-{
-}
-module_exit(crc_t10dif_x86_exit);
-
-MODULE_DESCRIPTION("CRC-T10DIF using [V]PCLMULQDQ instructions");
-MODULE_LICENSE("GPL");
diff --git a/arch/x86/lib/crc16-msb-pclmul.S b/lib/crc/x86/crc16-msb-pclmul.S
similarity index 100%
rename from arch/x86/lib/crc16-msb-pclmul.S
rename to lib/crc/x86/crc16-msb-pclmul.S
diff --git a/arch/x86/lib/crc32-pclmul.S b/lib/crc/x86/crc32-pclmul.S
similarity index 100%
rename from arch/x86/lib/crc32-pclmul.S
rename to lib/crc/x86/crc32-pclmul.S
diff --git a/arch/x86/lib/crc32.c b/lib/crc/x86/crc32.h
similarity index 76%
rename from arch/x86/lib/crc32.c
rename to lib/crc/x86/crc32.h
index d09343e2cea93..ad7fc96fd6bc9 100644
--- a/arch/x86/lib/crc32.c
+++ b/lib/crc/x86/crc32.h
@@ -5,26 +5,24 @@
* Copyright (C) 2008 Intel Corporation
* Copyright 2012 Xyratex Technology Limited
* Copyright 2024 Google LLC
*/
-#include <linux/crc32.h>
-#include <linux/module.h>
#include "crc-pclmul-template.h"
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_crc32);
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pclmulqdq);
DECLARE_CRC_PCLMUL_FUNCS(crc32_lsb, u32);
-u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
+#define crc32_le_arch crc32_le_arch
+static inline u32 crc32_le_arch(u32 crc, const u8 *p, size_t len)
{
CRC_PCLMUL(crc, p, len, crc32_lsb, crc32_lsb_0xedb88320_consts,
have_pclmulqdq);
return crc32_le_base(crc, p, len);
}
-EXPORT_SYMBOL(crc32_le_arch);
#ifdef CONFIG_X86_64
#define CRC32_INST "crc32q %1, %q0"
#else
#define CRC32_INST "crc32l %1, %0"
@@ -36,11 +34,12 @@ EXPORT_SYMBOL(crc32_le_arch);
*/
#define CRC32C_PCLMUL_BREAKEVEN 512
asmlinkage u32 crc32c_x86_3way(u32 crc, const u8 *buffer, size_t len);
-u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
+#define crc32c_arch crc32c_arch
+static inline u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
{
size_t num_longs;
if (!static_branch_likely(&have_crc32))
return crc32c_base(crc, p, len);
@@ -68,44 +67,27 @@ u32 crc32c_arch(u32 crc, const u8 *p, size_t len)
if (len & 1)
asm("crc32b %1, %0" : "+r" (crc) : ASM_INPUT_RM (*p));
return crc;
}
-EXPORT_SYMBOL(crc32c_arch);
-u32 crc32_be_arch(u32 crc, const u8 *p, size_t len)
-{
- return crc32_be_base(crc, p, len);
-}
-EXPORT_SYMBOL(crc32_be_arch);
-
-static int __init crc32_x86_init(void)
+#define crc32_mod_init_arch crc32_mod_init_arch
+static inline void crc32_mod_init_arch(void)
{
if (boot_cpu_has(X86_FEATURE_XMM4_2))
static_branch_enable(&have_crc32);
if (boot_cpu_has(X86_FEATURE_PCLMULQDQ)) {
static_branch_enable(&have_pclmulqdq);
INIT_CRC_PCLMUL(crc32_lsb);
}
- return 0;
}
-subsys_initcall(crc32_x86_init);
-static void __exit crc32_x86_exit(void)
-{
-}
-module_exit(crc32_x86_exit);
-
-u32 crc32_optimizations(void)
+static inline u32 crc32_optimizations_arch(void)
{
u32 optimizations = 0;
if (static_key_enabled(&have_crc32))
optimizations |= CRC32C_OPTIMIZATION;
if (static_key_enabled(&have_pclmulqdq))
optimizations |= CRC32_LE_OPTIMIZATION;
return optimizations;
}
-EXPORT_SYMBOL(crc32_optimizations);
-
-MODULE_DESCRIPTION("x86-optimized CRC32 functions");
-MODULE_LICENSE("GPL");
diff --git a/arch/x86/lib/crc32c-3way.S b/lib/crc/x86/crc32c-3way.S
similarity index 100%
rename from arch/x86/lib/crc32c-3way.S
rename to lib/crc/x86/crc32c-3way.S
diff --git a/arch/x86/lib/crc64-pclmul.S b/lib/crc/x86/crc64-pclmul.S
similarity index 100%
rename from arch/x86/lib/crc64-pclmul.S
rename to lib/crc/x86/crc64-pclmul.S
diff --git a/arch/x86/lib/crc64.c b/lib/crc/x86/crc64.h
similarity index 61%
rename from arch/x86/lib/crc64.c
rename to lib/crc/x86/crc64.h
index 351a09f5813e2..54aca3a9475c9 100644
--- a/arch/x86/lib/crc64.c
+++ b/lib/crc/x86/crc64.h
@@ -3,48 +3,35 @@
* CRC64 using [V]PCLMULQDQ instructions
*
* Copyright 2025 Google LLC
*/
-#include <linux/crc64.h>
-#include <linux/module.h>
#include "crc-pclmul-template.h"
static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pclmulqdq);
DECLARE_CRC_PCLMUL_FUNCS(crc64_msb, u64);
DECLARE_CRC_PCLMUL_FUNCS(crc64_lsb, u64);
-u64 crc64_be_arch(u64 crc, const u8 *p, size_t len)
+static inline u64 crc64_be_arch(u64 crc, const u8 *p, size_t len)
{
CRC_PCLMUL(crc, p, len, crc64_msb, crc64_msb_0x42f0e1eba9ea3693_consts,
have_pclmulqdq);
return crc64_be_generic(crc, p, len);
}
-EXPORT_SYMBOL_GPL(crc64_be_arch);
-u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)
+static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)
{
CRC_PCLMUL(crc, p, len, crc64_lsb, crc64_lsb_0x9a6c9329ac4bc9b5_consts,
have_pclmulqdq);
return crc64_nvme_generic(crc, p, len);
}
-EXPORT_SYMBOL_GPL(crc64_nvme_arch);
-static int __init crc64_x86_init(void)
+#define crc64_mod_init_arch crc64_mod_init_arch
+static inline void crc64_mod_init_arch(void)
{
if (boot_cpu_has(X86_FEATURE_PCLMULQDQ)) {
static_branch_enable(&have_pclmulqdq);
INIT_CRC_PCLMUL(crc64_msb);
INIT_CRC_PCLMUL(crc64_lsb);
}
- return 0;
}
-subsys_initcall(crc64_x86_init);
-
-static void __exit crc64_x86_exit(void)
-{
-}
-module_exit(crc64_x86_exit);
-
-MODULE_DESCRIPTION("CRC64 using [V]PCLMULQDQ instructions");
-MODULE_LICENSE("GPL");
--
2.49.0
^ permalink raw reply related [flat|nested] 15+ messages in thread
* Re: [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
` (12 preceding siblings ...)
2025-06-01 22:44 ` [PATCH 13/13] lib/crc/x86: migrate x86-optimized " Eric Biggers
@ 2025-06-08 23:46 ` Eric Biggers
13 siblings, 0 replies; 15+ messages in thread
From: Eric Biggers @ 2025-06-08 23:46 UTC (permalink / raw)
To: linux-kernel
Cc: linux-crypto, linux-arm-kernel, loongarch, linux-mips,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, x86,
linux-arch, Ard Biesheuvel, Jason A . Donenfeld , Linus Torvalds
On Sun, Jun 01, 2025 at 03:44:28PM -0700, Eric Biggers wrote:
> This series improves how lib/crc supports arch-optimized code. First,
> instead of the arch-optimized CRC code being in arch/$(SRCARCH)/lib/, it
> will now be in lib/crc/$(SRCARCH)/. Second, the API functions (e.g.
> crc32c()), arch-optimized functions (e.g. crc32c_arch()), and generic
> functions (e.g. crc32c_base()) will now be part of a single module for
> each CRC type, allowing better inlining and dead code elimination. The
> second change is made possible by the first.
>
> As an example, consider CONFIG_CRC32=m on x86. We'll now have just
> crc32.ko instead of both crc32-x86.ko and crc32.ko. The two modules
> were already coupled together and always both got loaded together via
> direct symbol dependency, so the separation provided no benefit.
>
> Note: later I'd like to apply the same design to lib/crypto/ too, where
> often the API functions are out-of-line so this will work even better.
> In those cases, for each algorithm we currently have 3 modules all
> coupled together, e.g. libsha256.ko, libsha256-generic.ko, and
> sha256-x86.ko. We should have just one, inline things properly, and
> rely on the compiler's dead code elimination to decide the inclusion of
> the generic code instead of manually setting it via kconfig.
>
> Having arch-specific code outside arch/ was somewhat controversial when
> Zinc proposed it back in 2018. But I don't think the concerns are
> warranted. It's better from a technical perspective, as it enables the
> improvements mentioned above. This model is already successfully used
> in other places in the kernel such as lib/raid6/. The community of each
> architecture still remains free to work on the code, even if it's not in
> arch/. At the time there was also a desire to put the library code in
> the same files as the old-school crypto API, but that was a mistake; now
> that the library is separate, that's no longer a constraint either.
>
> Patches 1 and 2, which I previously sent out by themselves, are
> prerequisites because they eliminate the need for the CRC32 library API
> to expose the generic functions.
>
> Eric Biggers (13):
> crypto/crc32: register only one shash_alg
> crypto/crc32c: register only one shash_alg
Applied the first 2 patches to
https://git.kernel.org/pub/scm/linux/kernel/git/ebiggers/linux.git/log/?h=crc-next
The remaining patches have new versions in the v2 series.
- Eric
^ permalink raw reply [flat|nested] 15+ messages in thread
end of thread, other threads:[~2025-06-08 23:47 UTC | newest]
Thread overview: 15+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2025-06-01 22:44 [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
2025-06-01 22:44 ` [PATCH 01/13] crypto/crc32: register only one shash_alg Eric Biggers
2025-06-01 22:44 ` [PATCH 02/13] crypto/crc32c: " Eric Biggers
2025-06-01 22:44 ` [PATCH 03/13] lib/crc: move files into lib/crc/ Eric Biggers
2025-06-01 22:44 ` [PATCH 04/13] lib/crc: prepare for arch-optimized code in subdirs of lib/crc/ Eric Biggers
2025-06-01 22:44 ` [PATCH 05/13] lib/crc/arm: migrate arm-optimized CRC code into lib/crc/ Eric Biggers
2025-06-01 22:44 ` [PATCH 06/13] lib/crc/arm64: migrate arm64-optimized " Eric Biggers
2025-06-01 22:44 ` [PATCH 07/13] lib/crc/loongarch: migrate loongarch-optimized " Eric Biggers
2025-06-01 22:44 ` [PATCH 08/13] lib/crc/mips: migrate mips-optimized " Eric Biggers
2025-06-01 22:44 ` [PATCH 09/13] lib/crc/powerpc: migrate powerpc-optimized " Eric Biggers
2025-06-01 22:44 ` [PATCH 10/13] lib/crc/riscv: migrate riscv-optimized " Eric Biggers
2025-06-01 22:44 ` [PATCH 11/13] lib/crc/s390: migrate s390-optimized CRC code into lib/s390/ Eric Biggers
2025-06-01 22:44 ` [PATCH 12/13] lib/crc/sparc: migrate sparc-optimized CRC code into lib/crc/ Eric Biggers
2025-06-01 22:44 ` [PATCH 13/13] lib/crc/x86: migrate x86-optimized " Eric Biggers
2025-06-08 23:46 ` [PATCH 00/13] lib/crc: improve how arch-optimized code is integrated Eric Biggers
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).