* [PATCH v3 1/8] x86/fpu: Check for missing AVX and AVX-512 xstate bits
2026-08-13 2:14 [PATCH v3 0/8] x86: Remove cpu_has_xfeatures() and add AVX-512 xor_gen() Eric Biggers
@ 2026-08-13 2:14 ` Eric Biggers
2026-08-13 2:15 ` [PATCH v3 2/8] um: " Eric Biggers
` (6 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Eric Biggers @ 2026-08-13 2:14 UTC (permalink / raw)
To: x86
Cc: linux-um, linux-raid, linux-crypto, linux-kernel,
Christoph Hellwig, Andrew Morton, Eric Biggers
If the CPU declares AVX or AVX-512 support, verify that the
corresponding xstate bits are also set. If not, warn and clear them.
This eliminates the perceived need for AVX and AVX-512 optimized code in
the kernel to call cpu_has_xfeatures(). That has never been universally
done, which strongly suggests that it has never really been needed in
practice, but this should remove any remaining doubt.
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
arch/x86/kernel/fpu/xstate.c | 25 +++++++++++++++++++++++--
1 file changed, 23 insertions(+), 2 deletions(-)
diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c
index a7b6524a9dea..97cfd4fb6cc0 100644
--- a/arch/x86/kernel/fpu/xstate.c
+++ b/arch/x86/kernel/fpu/xstate.c
@@ -806,7 +806,7 @@ static u64 __init guest_default_mask(void)
void __init fpu__init_system_xstate(unsigned int legacy_size)
{
unsigned int eax, ebx, ecx, edx;
- u64 xfeatures;
+ u64 xfeatures, mask;
int err;
int i;
@@ -818,6 +818,8 @@ void __init fpu__init_system_xstate(unsigned int legacy_size)
if (!boot_cpu_has(X86_FEATURE_XSAVE)) {
pr_info("x86/fpu: x87 FPU will use %s\n",
boot_cpu_has(X86_FEATURE_FXSR) ? "FXSAVE" : "FSAVE");
+ /* Disable all dependent flags too */
+ setup_clear_cpu_cap(X86_FEATURE_XSAVE);
return;
}
@@ -833,7 +835,8 @@ void __init fpu__init_system_xstate(unsigned int legacy_size)
cpuid_count(CPUID_LEAF_XSTATE, 1, &eax, &ebx, &ecx, &edx);
fpu_kernel_cfg.max_features |= ecx + ((u64)edx << 32);
- if ((fpu_kernel_cfg.max_features & XFEATURE_MASK_FPSSE) != XFEATURE_MASK_FPSSE) {
+ mask = XFEATURE_MASK_FPSSE;
+ if ((fpu_kernel_cfg.max_features & mask) != mask) {
/*
* This indicates that something really unexpected happened
* with the enumeration. Disable XSAVE and try to continue
@@ -844,6 +847,24 @@ void __init fpu__init_system_xstate(unsigned int legacy_size)
goto out_disable;
}
+ mask |= XFEATURE_MASK_YMM;
+ if (boot_cpu_has(X86_FEATURE_AVX)) {
+ if ((fpu_kernel_cfg.max_features & mask) != mask) {
+ pr_err(FW_BUG
+ "x86/fpu: Disabling AVX support due to missing xstate features\n");
+ setup_clear_cpu_cap(X86_FEATURE_AVX);
+ }
+ }
+
+ mask |= XFEATURE_MASK_AVX512;
+ if (boot_cpu_has(X86_FEATURE_AVX512F)) {
+ if ((fpu_kernel_cfg.max_features & mask) != mask) {
+ pr_err(FW_BUG
+ "x86/fpu: Disabling AVX-512 support due to missing xstate features\n");
+ setup_clear_cpu_cap(X86_FEATURE_AVX512F);
+ }
+ }
+
if (fpu_kernel_cfg.max_features & XFEATURE_MASK_APX &&
fpu_kernel_cfg.max_features & (XFEATURE_MASK_BNDREGS | XFEATURE_MASK_BNDCSR)) {
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v3 2/8] um: Check for missing AVX and AVX-512 xstate bits
2026-08-13 2:14 [PATCH v3 0/8] x86: Remove cpu_has_xfeatures() and add AVX-512 xor_gen() Eric Biggers
2026-08-13 2:14 ` [PATCH v3 1/8] x86/fpu: Check for missing AVX and AVX-512 xstate bits Eric Biggers
@ 2026-08-13 2:15 ` Eric Biggers
2026-08-13 2:15 ` [PATCH v3 3/8] crypto: x86 - Stop using cpu_has_xfeatures() Eric Biggers
` (5 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Eric Biggers @ 2026-08-13 2:15 UTC (permalink / raw)
To: x86
Cc: linux-um, linux-raid, linux-crypto, linux-kernel,
Christoph Hellwig, Andrew Morton, Eric Biggers
If the CPU declares AVX or AVX-512 support, verify that all the
corresponding bits are set in the XCR0 register. If any are missing,
warn and don't set the corresponding X86_FEATURE_* flags.
This eliminates the perceived need for UML-supporting AVX and AVX-512
optimized code in the kernel (that is, lib/raid/ currently) to start
checking the XCR0 bits in addition to X86_FEATURE_AVX*.
This aligns UML with the vast majority of userspace programs, which
check the XCR0 bits before considering AVX and AVX-512 to be supported,
as per the procedure documented in Intel's CPU manual.
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
arch/um/kernel/um_arch.c | 78 +++++++++++++++++++++++++++++++++++++++-
1 file changed, 77 insertions(+), 1 deletion(-)
diff --git a/arch/um/kernel/um_arch.c b/arch/um/kernel/um_arch.c
index 2141f5f1f5a2..aafbaef2ae82 100644
--- a/arch/um/kernel/um_arch.c
+++ b/arch/um/kernel/um_arch.c
@@ -264,12 +264,88 @@ unsigned long brk_start;
#define MIN_VMALLOC (32 * 1024 * 1024)
+static u64 __init read_xcr0(void)
+{
+ u32 a, b, c, d;
+
+ asm volatile("cpuid"
+ : "=a"(a), "=b"(b), "=c"(c), "=d"(d)
+ : "a"(0), "c"(0));
+ if (a >= 1) { /* max_leaf >= 1 */
+ asm volatile("cpuid"
+ : "=a"(a), "=b"(b), "=c"(c), "=d"(d)
+ : "a"(1), "c"(0));
+ if (c & (1 << 27)) { /* XSAVE enabled by OS */
+ asm volatile("xgetbv" : "=d"(d), "=a"(a) : "c"(0));
+ return ((u64)d << 32) | a;
+ }
+ }
+ return 0;
+}
+
+static void __init validate_and_set_cpu_cap(int cap, u64 xcr0)
+{
+ /*
+ * Check for missing xstate features right away, so that there's no
+ * perceived need for all optimized code in the kernel to do so.
+ */
+ switch (cap) {
+ case X86_FEATURE_AVX:
+ case X86_FEATURE_AVX2:
+ case X86_FEATURE_AVX_VNNI:
+ case X86_FEATURE_FMA:
+ case X86_FEATURE_VAES:
+ case X86_FEATURE_VPCLMULQDQ:
+ if ((xcr0 & 0x7) != 0x7) {
+ static bool warned;
+
+ if (!warned) {
+ os_warn("Disabling AVX support due to missing xstate features\n");
+ warned = true;
+ }
+ return;
+ }
+ break;
+ case X86_FEATURE_AVX512F:
+ case X86_FEATURE_AVX512BW:
+ case X86_FEATURE_AVX512CD:
+ case X86_FEATURE_AVX512DQ:
+ case X86_FEATURE_AVX512ER:
+ case X86_FEATURE_AVX512IFMA:
+ case X86_FEATURE_AVX512PF:
+ case X86_FEATURE_AVX512VBMI:
+ case X86_FEATURE_AVX512VL:
+ case X86_FEATURE_AVX512_4FMAPS:
+ case X86_FEATURE_AVX512_4VNNIW:
+ case X86_FEATURE_AVX512_BF16:
+ case X86_FEATURE_AVX512_BITALG:
+ case X86_FEATURE_AVX512_FP16:
+ case X86_FEATURE_AVX512_VBMI2:
+ case X86_FEATURE_AVX512_VNNI:
+ case X86_FEATURE_AVX512_VP2INTERSECT:
+ case X86_FEATURE_AVX512_VPOPCNTDQ:
+ if ((xcr0 & 0xe7) != 0xe7) {
+ static bool warned;
+
+ if (!warned) {
+ os_warn("Disabling AVX-512 support due to missing xstate features\n");
+ warned = true;
+ }
+ return;
+ }
+ break;
+ }
+ set_cpu_cap(&boot_cpu_data, cap);
+}
+
static void __init parse_host_cpu_flags(char *line)
{
+ u64 xcr0 = read_xcr0();
int i;
+
for (i = 0; i < 32*NCAPINTS; i++) {
if ((x86_cap_flags[i] != NULL) && strstr(line, x86_cap_flags[i]))
- set_cpu_cap(&boot_cpu_data, i);
+ validate_and_set_cpu_cap(i, xcr0);
}
}
--
2.55.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v3 3/8] crypto: x86 - Stop using cpu_has_xfeatures()
2026-08-13 2:14 [PATCH v3 0/8] x86: Remove cpu_has_xfeatures() and add AVX-512 xor_gen() Eric Biggers
2026-08-13 2:14 ` [PATCH v3 1/8] x86/fpu: Check for missing AVX and AVX-512 xstate bits Eric Biggers
2026-08-13 2:15 ` [PATCH v3 2/8] um: " Eric Biggers
@ 2026-08-13 2:15 ` Eric Biggers
2026-08-13 2:15 ` [PATCH v3 4/8] lib/crypto: x86: " Eric Biggers
` (4 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Eric Biggers @ 2026-08-13 2:15 UTC (permalink / raw)
To: x86
Cc: linux-um, linux-raid, linux-crypto, linux-kernel,
Christoph Hellwig, Andrew Morton, Eric Biggers
Checking both boot_cpu_has(X86_FEATURE_AVX*) and cpu_has_xfeatures() has
never really been needed in practice, and it's never been universally
done (e.g., lib/raid/ omits cpu_has_xfeatures()). Nevertheless, both
x86 and UML now explicitly clear the AVX and AVX-512 flags if their
xfeatures are missing, which should remove any remaining doubts.
Thus, remove all the calls to cpu_has_xfeatures(), as well as the
related checks of boot_cpu_has(X86_FEATURE_OSXSAVE).
In a few cases there was no corresponding boot_cpu_has(X86_FEATURE_AVX*)
check, so add the missing ones.
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
arch/x86/crypto/aegis128-aesni-glue.c | 3 +--
arch/x86/crypto/aesni-intel_glue.c | 7 ++-----
arch/x86/crypto/aria_aesni_avx2_glue.c | 11 +----------
arch/x86/crypto/aria_aesni_avx_glue.c | 11 +----------
arch/x86/crypto/aria_gfni_avx512_glue.c | 11 +----------
arch/x86/crypto/camellia_aesni_avx2_glue.c | 11 +----------
arch/x86/crypto/camellia_aesni_avx_glue.c | 11 +----------
arch/x86/crypto/cast5_avx_glue.c | 7 ++-----
arch/x86/crypto/cast6_avx_glue.c | 7 ++-----
arch/x86/crypto/serpent_avx2_glue.c | 9 +--------
arch/x86/crypto/serpent_avx_glue.c | 7 ++-----
arch/x86/crypto/sm4_aesni_avx2_glue.c | 11 +----------
arch/x86/crypto/sm4_aesni_avx_glue.c | 11 +----------
arch/x86/crypto/twofish_avx_glue.c | 6 ++----
14 files changed, 19 insertions(+), 104 deletions(-)
diff --git a/arch/x86/crypto/aegis128-aesni-glue.c b/arch/x86/crypto/aegis128-aesni-glue.c
index f1adfba1a76e..09fc0b15b0e9 100644
--- a/arch/x86/crypto/aegis128-aesni-glue.c
+++ b/arch/x86/crypto/aegis128-aesni-glue.c
@@ -265,8 +265,7 @@ static struct aead_alg crypto_aegis128_aesni_alg = {
static int __init crypto_aegis128_aesni_module_init(void)
{
if (!boot_cpu_has(X86_FEATURE_XMM4_1) ||
- !boot_cpu_has(X86_FEATURE_AES) ||
- !cpu_has_xfeatures(XFEATURE_MASK_SSE, NULL))
+ !boot_cpu_has(X86_FEATURE_AES))
return -ENODEV;
return crypto_register_aead(&crypto_aegis128_aesni_alg);
diff --git a/arch/x86/crypto/aesni-intel_glue.c b/arch/x86/crypto/aesni-intel_glue.c
index f522fff9231e..f6f899db7482 100644
--- a/arch/x86/crypto/aesni-intel_glue.c
+++ b/arch/x86/crypto/aesni-intel_glue.c
@@ -1548,8 +1548,7 @@ static int __init register_avx_algs(void)
if (!boot_cpu_has(X86_FEATURE_AVX2) ||
!boot_cpu_has(X86_FEATURE_VAES) ||
!boot_cpu_has(X86_FEATURE_VPCLMULQDQ) ||
- !boot_cpu_has(X86_FEATURE_PCLMULQDQ) ||
- !cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL))
+ !boot_cpu_has(X86_FEATURE_PCLMULQDQ))
return 0;
err = crypto_register_skciphers(skcipher_algs_vaes_avx2,
ARRAY_SIZE(skcipher_algs_vaes_avx2));
@@ -1562,9 +1561,7 @@ static int __init register_avx_algs(void)
if (!boot_cpu_has(X86_FEATURE_AVX512BW) ||
!boot_cpu_has(X86_FEATURE_AVX512VL) ||
- !boot_cpu_has(X86_FEATURE_BMI2) ||
- !cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM |
- XFEATURE_MASK_AVX512, NULL))
+ !boot_cpu_has(X86_FEATURE_BMI2))
return 0;
if (boot_cpu_has(X86_FEATURE_PREFER_YMM)) {
diff --git a/arch/x86/crypto/aria_aesni_avx2_glue.c b/arch/x86/crypto/aria_aesni_avx2_glue.c
index 1487a49bfbac..371be2fb6469 100644
--- a/arch/x86/crypto/aria_aesni_avx2_glue.c
+++ b/arch/x86/crypto/aria_aesni_avx2_glue.c
@@ -195,22 +195,13 @@ static struct skcipher_alg aria_algs[] = {
static int __init aria_avx2_init(void)
{
- const char *feature_name;
-
if (!boot_cpu_has(X86_FEATURE_AVX) ||
!boot_cpu_has(X86_FEATURE_AVX2) ||
- !boot_cpu_has(X86_FEATURE_AES) ||
- !boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ !boot_cpu_has(X86_FEATURE_AES)) {
pr_info("AVX2 or AES-NI instructions are not detected.\n");
return -ENODEV;
}
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
- return -ENODEV;
- }
-
if (boot_cpu_has(X86_FEATURE_GFNI)) {
aria_ops.aria_encrypt_16way = aria_aesni_avx_gfni_encrypt_16way;
aria_ops.aria_decrypt_16way = aria_aesni_avx_gfni_decrypt_16way;
diff --git a/arch/x86/crypto/aria_aesni_avx_glue.c b/arch/x86/crypto/aria_aesni_avx_glue.c
index e4e3d78915a5..d23fc91c0ebd 100644
--- a/arch/x86/crypto/aria_aesni_avx_glue.c
+++ b/arch/x86/crypto/aria_aesni_avx_glue.c
@@ -182,21 +182,12 @@ static struct skcipher_alg aria_algs[] = {
static int __init aria_avx_init(void)
{
- const char *feature_name;
-
if (!boot_cpu_has(X86_FEATURE_AVX) ||
- !boot_cpu_has(X86_FEATURE_AES) ||
- !boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ !boot_cpu_has(X86_FEATURE_AES)) {
pr_info("AVX or AES-NI instructions are not detected.\n");
return -ENODEV;
}
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
- return -ENODEV;
- }
-
if (boot_cpu_has(X86_FEATURE_GFNI)) {
aria_ops.aria_encrypt_16way = aria_aesni_avx_gfni_encrypt_16way;
aria_ops.aria_decrypt_16way = aria_aesni_avx_gfni_decrypt_16way;
diff --git a/arch/x86/crypto/aria_gfni_avx512_glue.c b/arch/x86/crypto/aria_gfni_avx512_glue.c
index 363cbf4399cc..e05bbeb22d4a 100644
--- a/arch/x86/crypto/aria_gfni_avx512_glue.c
+++ b/arch/x86/crypto/aria_gfni_avx512_glue.c
@@ -196,24 +196,15 @@ static struct skcipher_alg aria_algs[] = {
static int __init aria_avx512_init(void)
{
- const char *feature_name;
-
if (!boot_cpu_has(X86_FEATURE_AVX) ||
!boot_cpu_has(X86_FEATURE_AVX2) ||
!boot_cpu_has(X86_FEATURE_AVX512F) ||
!boot_cpu_has(X86_FEATURE_AVX512VL) ||
- !boot_cpu_has(X86_FEATURE_GFNI) ||
- !boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ !boot_cpu_has(X86_FEATURE_GFNI)) {
pr_info("AVX512/GFNI instructions are not detected.\n");
return -ENODEV;
}
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM |
- XFEATURE_MASK_AVX512, &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
- return -ENODEV;
- }
-
aria_ops.aria_encrypt_16way = aria_aesni_avx_gfni_encrypt_16way;
aria_ops.aria_decrypt_16way = aria_aesni_avx_gfni_decrypt_16way;
aria_ops.aria_ctr_crypt_16way = aria_aesni_avx_gfni_ctr_crypt_16way;
diff --git a/arch/x86/crypto/camellia_aesni_avx2_glue.c b/arch/x86/crypto/camellia_aesni_avx2_glue.c
index 2d2f4e16537c..073fa3bb8388 100644
--- a/arch/x86/crypto/camellia_aesni_avx2_glue.c
+++ b/arch/x86/crypto/camellia_aesni_avx2_glue.c
@@ -97,22 +97,13 @@ static struct skcipher_alg camellia_algs[] = {
static int __init camellia_aesni_init(void)
{
- const char *feature_name;
-
if (!boot_cpu_has(X86_FEATURE_AVX) ||
!boot_cpu_has(X86_FEATURE_AVX2) ||
- !boot_cpu_has(X86_FEATURE_AES) ||
- !boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ !boot_cpu_has(X86_FEATURE_AES)) {
pr_info("AVX2 or AES-NI instructions are not detected.\n");
return -ENODEV;
}
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
- return -ENODEV;
- }
-
return crypto_register_skciphers(camellia_algs,
ARRAY_SIZE(camellia_algs));
}
diff --git a/arch/x86/crypto/camellia_aesni_avx_glue.c b/arch/x86/crypto/camellia_aesni_avx_glue.c
index 5c321f255eb7..872e5e07220f 100644
--- a/arch/x86/crypto/camellia_aesni_avx_glue.c
+++ b/arch/x86/crypto/camellia_aesni_avx_glue.c
@@ -98,21 +98,12 @@ static struct skcipher_alg camellia_algs[] = {
static int __init camellia_aesni_init(void)
{
- const char *feature_name;
-
if (!boot_cpu_has(X86_FEATURE_AVX) ||
- !boot_cpu_has(X86_FEATURE_AES) ||
- !boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ !boot_cpu_has(X86_FEATURE_AES)) {
pr_info("AVX or AES-NI instructions are not detected.\n");
return -ENODEV;
}
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
- return -ENODEV;
- }
-
return crypto_register_skciphers(camellia_algs,
ARRAY_SIZE(camellia_algs));
}
diff --git a/arch/x86/crypto/cast5_avx_glue.c b/arch/x86/crypto/cast5_avx_glue.c
index 3aca04d43b34..5de35e863370 100644
--- a/arch/x86/crypto/cast5_avx_glue.c
+++ b/arch/x86/crypto/cast5_avx_glue.c
@@ -92,11 +92,8 @@ static struct skcipher_alg cast5_algs[] = {
static int __init cast5_init(void)
{
- const char *feature_name;
-
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
+ if (!boot_cpu_has(X86_FEATURE_AVX)) {
+ pr_info("AVX instructions are not detected.\n");
return -ENODEV;
}
diff --git a/arch/x86/crypto/cast6_avx_glue.c b/arch/x86/crypto/cast6_avx_glue.c
index c4dd28c30303..3d7ea48007bc 100644
--- a/arch/x86/crypto/cast6_avx_glue.c
+++ b/arch/x86/crypto/cast6_avx_glue.c
@@ -92,11 +92,8 @@ static struct skcipher_alg cast6_algs[] = {
static int __init cast6_init(void)
{
- const char *feature_name;
-
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
+ if (!boot_cpu_has(X86_FEATURE_AVX)) {
+ pr_info("AVX instructions are not detected.\n");
return -ENODEV;
}
diff --git a/arch/x86/crypto/serpent_avx2_glue.c b/arch/x86/crypto/serpent_avx2_glue.c
index f5f2121b7956..72a9e2b306d6 100644
--- a/arch/x86/crypto/serpent_avx2_glue.c
+++ b/arch/x86/crypto/serpent_avx2_glue.c
@@ -93,17 +93,10 @@ static struct skcipher_alg serpent_algs[] = {
static int __init serpent_avx2_init(void)
{
- const char *feature_name;
-
- if (!boot_cpu_has(X86_FEATURE_AVX2) || !boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ if (!boot_cpu_has(X86_FEATURE_AVX2)) {
pr_info("AVX2 instructions are not detected.\n");
return -ENODEV;
}
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
- return -ENODEV;
- }
return crypto_register_skciphers(serpent_algs,
ARRAY_SIZE(serpent_algs));
diff --git a/arch/x86/crypto/serpent_avx_glue.c b/arch/x86/crypto/serpent_avx_glue.c
index 9c8b3a335d5c..42c4e1569674 100644
--- a/arch/x86/crypto/serpent_avx_glue.c
+++ b/arch/x86/crypto/serpent_avx_glue.c
@@ -100,11 +100,8 @@ static struct skcipher_alg serpent_algs[] = {
static int __init serpent_init(void)
{
- const char *feature_name;
-
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
+ if (!boot_cpu_has(X86_FEATURE_AVX)) {
+ pr_info("AVX instructions are not detected.\n");
return -ENODEV;
}
diff --git a/arch/x86/crypto/sm4_aesni_avx2_glue.c b/arch/x86/crypto/sm4_aesni_avx2_glue.c
index fec0ab7a63dd..eef73894e777 100644
--- a/arch/x86/crypto/sm4_aesni_avx2_glue.c
+++ b/arch/x86/crypto/sm4_aesni_avx2_glue.c
@@ -98,22 +98,13 @@ static struct skcipher_alg sm4_aesni_avx2_skciphers[] = {
static int __init sm4_init(void)
{
- const char *feature_name;
-
if (!boot_cpu_has(X86_FEATURE_AVX) ||
!boot_cpu_has(X86_FEATURE_AVX2) ||
- !boot_cpu_has(X86_FEATURE_AES) ||
- !boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ !boot_cpu_has(X86_FEATURE_AES)) {
pr_info("AVX2 or AES-NI instructions are not detected.\n");
return -ENODEV;
}
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
- return -ENODEV;
- }
-
return crypto_register_skciphers(sm4_aesni_avx2_skciphers,
ARRAY_SIZE(sm4_aesni_avx2_skciphers));
}
diff --git a/arch/x86/crypto/sm4_aesni_avx_glue.c b/arch/x86/crypto/sm4_aesni_avx_glue.c
index 88caf418a06f..ed383da5ff46 100644
--- a/arch/x86/crypto/sm4_aesni_avx_glue.c
+++ b/arch/x86/crypto/sm4_aesni_avx_glue.c
@@ -314,21 +314,12 @@ static struct skcipher_alg sm4_aesni_avx_skciphers[] = {
static int __init sm4_init(void)
{
- const char *feature_name;
-
if (!boot_cpu_has(X86_FEATURE_AVX) ||
- !boot_cpu_has(X86_FEATURE_AES) ||
- !boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ !boot_cpu_has(X86_FEATURE_AES)) {
pr_info("AVX or AES-NI instructions are not detected.\n");
return -ENODEV;
}
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
- return -ENODEV;
- }
-
return crypto_register_skciphers(sm4_aesni_avx_skciphers,
ARRAY_SIZE(sm4_aesni_avx_skciphers));
}
diff --git a/arch/x86/crypto/twofish_avx_glue.c b/arch/x86/crypto/twofish_avx_glue.c
index 9e20db013750..985bc54a2340 100644
--- a/arch/x86/crypto/twofish_avx_glue.c
+++ b/arch/x86/crypto/twofish_avx_glue.c
@@ -102,10 +102,8 @@ static struct skcipher_alg twofish_algs[] = {
static int __init twofish_init(void)
{
- const char *feature_name;
-
- if (!cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, &feature_name)) {
- pr_info("CPU feature '%s' is not supported.\n", feature_name);
+ if (!boot_cpu_has(X86_FEATURE_AVX)) {
+ pr_info("AVX instructions are not detected.\n");
return -ENODEV;
}
--
2.55.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v3 4/8] lib/crypto: x86: Stop using cpu_has_xfeatures()
2026-08-13 2:14 [PATCH v3 0/8] x86: Remove cpu_has_xfeatures() and add AVX-512 xor_gen() Eric Biggers
` (2 preceding siblings ...)
2026-08-13 2:15 ` [PATCH v3 3/8] crypto: x86 - Stop using cpu_has_xfeatures() Eric Biggers
@ 2026-08-13 2:15 ` Eric Biggers
2026-08-13 2:15 ` [PATCH v3 5/8] lib/crc: " Eric Biggers
` (3 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Eric Biggers @ 2026-08-13 2:15 UTC (permalink / raw)
To: x86
Cc: linux-um, linux-raid, linux-crypto, linux-kernel,
Christoph Hellwig, Andrew Morton, Eric Biggers
Checking both boot_cpu_has() and cpu_has_xfeatures() has never really
been needed in practice, and it's never been universally done (e.g.,
lib/raid/ omits cpu_has_xfeatures()). Nevertheless, both x86 and UML
now explicitly clear the AVX and AVX-512 flags if their xfeatures are
missing, which should remove any remaining doubts.
Thus, remove all the calls to cpu_has_xfeatures().
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
lib/crypto/x86/blake2s.h | 4 +---
lib/crypto/x86/chacha.h | 3 +--
lib/crypto/x86/nh.h | 4 +---
lib/crypto/x86/poly1305.h | 7 ++-----
lib/crypto/x86/sha1.h | 4 +---
lib/crypto/x86/sha256.h | 4 +---
lib/crypto/x86/sha512.h | 3 +--
lib/crypto/x86/sm3.h | 3 +--
8 files changed, 9 insertions(+), 23 deletions(-)
diff --git a/lib/crypto/x86/blake2s.h b/lib/crypto/x86/blake2s.h
index f8eed6cb042e..0f7c51f055c8 100644
--- a/lib/crypto/x86/blake2s.h
+++ b/lib/crypto/x86/blake2s.h
@@ -55,8 +55,6 @@ static void blake2s_mod_init_arch(void)
if (boot_cpu_has(X86_FEATURE_AVX) &&
boot_cpu_has(X86_FEATURE_AVX2) &&
boot_cpu_has(X86_FEATURE_AVX512F) &&
- boot_cpu_has(X86_FEATURE_AVX512VL) &&
- cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM |
- XFEATURE_MASK_AVX512, NULL))
+ boot_cpu_has(X86_FEATURE_AVX512VL))
static_branch_enable(&blake2s_use_avx512);
}
diff --git a/lib/crypto/x86/chacha.h b/lib/crypto/x86/chacha.h
index 10cf8f1c569d..c79562aac56b 100644
--- a/lib/crypto/x86/chacha.h
+++ b/lib/crypto/x86/chacha.h
@@ -165,8 +165,7 @@ static void chacha_mod_init_arch(void)
static_branch_enable(&chacha_use_simd);
if (boot_cpu_has(X86_FEATURE_AVX) &&
- boot_cpu_has(X86_FEATURE_AVX2) &&
- cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL)) {
+ boot_cpu_has(X86_FEATURE_AVX2)) {
static_branch_enable(&chacha_use_avx2);
if (boot_cpu_has(X86_FEATURE_AVX512VL) &&
diff --git a/lib/crypto/x86/nh.h b/lib/crypto/x86/nh.h
index 83361c2e9783..342636dcb750 100644
--- a/lib/crypto/x86/nh.h
+++ b/lib/crypto/x86/nh.h
@@ -37,9 +37,7 @@ static void nh_mod_init_arch(void)
{
if (boot_cpu_has(X86_FEATURE_XMM2)) {
static_branch_enable(&have_sse2);
- if (boot_cpu_has(X86_FEATURE_AVX2) &&
- cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- NULL))
+ if (boot_cpu_has(X86_FEATURE_AVX2))
static_branch_enable(&have_avx2);
}
}
diff --git a/lib/crypto/x86/poly1305.h b/lib/crypto/x86/poly1305.h
index ee92e3740a78..b061b9926fa5 100644
--- a/lib/crypto/x86/poly1305.h
+++ b/lib/crypto/x86/poly1305.h
@@ -143,15 +143,12 @@ static void poly1305_emit(const struct poly1305_state *ctx,
#define poly1305_mod_init_arch poly1305_mod_init_arch
static void poly1305_mod_init_arch(void)
{
- if (boot_cpu_has(X86_FEATURE_AVX) &&
- cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL))
+ if (boot_cpu_has(X86_FEATURE_AVX))
static_branch_enable(&poly1305_use_avx);
- if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_AVX2) &&
- cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL))
+ if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_AVX2))
static_branch_enable(&poly1305_use_avx2);
if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_AVX2) &&
boot_cpu_has(X86_FEATURE_AVX512F) &&
- cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM | XFEATURE_MASK_AVX512, NULL) &&
/* Skylake downclocks unacceptably much when using zmm, but later generations are fast. */
boot_cpu_data.x86_vfm != INTEL_SKYLAKE_X)
static_branch_enable(&poly1305_use_avx512);
diff --git a/lib/crypto/x86/sha1.h b/lib/crypto/x86/sha1.h
index c48a0131fd12..6aff433466e7 100644
--- a/lib/crypto/x86/sha1.h
+++ b/lib/crypto/x86/sha1.h
@@ -59,9 +59,7 @@ static void sha1_mod_init_arch(void)
{
if (boot_cpu_has(X86_FEATURE_SHA_NI)) {
static_call_update(sha1_blocks_x86, sha1_blocks_ni);
- } else if (cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- NULL) &&
- boot_cpu_has(X86_FEATURE_AVX)) {
+ } else if (boot_cpu_has(X86_FEATURE_AVX)) {
if (boot_cpu_has(X86_FEATURE_AVX2) &&
boot_cpu_has(X86_FEATURE_BMI1) &&
boot_cpu_has(X86_FEATURE_BMI2))
diff --git a/lib/crypto/x86/sha256.h b/lib/crypto/x86/sha256.h
index 0ee69d8e39fe..e98ffdaf4b14 100644
--- a/lib/crypto/x86/sha256.h
+++ b/lib/crypto/x86/sha256.h
@@ -104,9 +104,7 @@ static void sha256_mod_init_arch(void)
boot_cpu_has(X86_FEATURE_PHE_EN) &&
boot_cpu_data.x86 >= 0x07) {
static_call_update(sha256_blocks_x86, sha256_blocks_phe);
- } else if (cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM,
- NULL) &&
- boot_cpu_has(X86_FEATURE_AVX)) {
+ } else if (boot_cpu_has(X86_FEATURE_AVX)) {
if (boot_cpu_has(X86_FEATURE_AVX2) &&
boot_cpu_has(X86_FEATURE_BMI2))
static_call_update(sha256_blocks_x86,
diff --git a/lib/crypto/x86/sha512.h b/lib/crypto/x86/sha512.h
index 0213c70cedd0..4e177b4606bd 100644
--- a/lib/crypto/x86/sha512.h
+++ b/lib/crypto/x86/sha512.h
@@ -37,8 +37,7 @@ static void sha512_blocks(struct sha512_block_state *state,
#define sha512_mod_init_arch sha512_mod_init_arch
static void sha512_mod_init_arch(void)
{
- if (cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL) &&
- boot_cpu_has(X86_FEATURE_AVX)) {
+ if (boot_cpu_has(X86_FEATURE_AVX)) {
if (boot_cpu_has(X86_FEATURE_AVX2) &&
boot_cpu_has(X86_FEATURE_BMI2))
static_call_update(sha512_blocks_x86,
diff --git a/lib/crypto/x86/sm3.h b/lib/crypto/x86/sm3.h
index 3834780f2f6a..e06d4a22e4fa 100644
--- a/lib/crypto/x86/sm3.h
+++ b/lib/crypto/x86/sm3.h
@@ -33,7 +33,6 @@ static void sm3_blocks(struct sm3_block_state *state,
#define sm3_mod_init_arch sm3_mod_init_arch
static void sm3_mod_init_arch(void)
{
- if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_BMI2) &&
- cpu_has_xfeatures(XFEATURE_MASK_SSE | XFEATURE_MASK_YMM, NULL))
+ if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_BMI2))
static_call_update(sm3_blocks_x86, sm3_blocks_avx);
}
--
2.55.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v3 5/8] lib/crc: x86: Stop using cpu_has_xfeatures()
2026-08-13 2:14 [PATCH v3 0/8] x86: Remove cpu_has_xfeatures() and add AVX-512 xor_gen() Eric Biggers
` (3 preceding siblings ...)
2026-08-13 2:15 ` [PATCH v3 4/8] lib/crypto: x86: " Eric Biggers
@ 2026-08-13 2:15 ` Eric Biggers
2026-08-13 2:15 ` [PATCH v3 6/8] x86/fpu: Remove cpu_has_xfeatures() Eric Biggers
` (2 subsequent siblings)
7 siblings, 0 replies; 10+ messages in thread
From: Eric Biggers @ 2026-08-13 2:15 UTC (permalink / raw)
To: x86
Cc: linux-um, linux-raid, linux-crypto, linux-kernel,
Christoph Hellwig, Andrew Morton, Eric Biggers
Checking both boot_cpu_has() and cpu_has_xfeatures() has never really
been needed in practice, and it's never been universally done (e.g.,
lib/raid/ omits cpu_has_xfeatures()). Nevertheless, both x86 and UML
now explicitly clear the AVX and AVX-512 flags if their xfeatures are
missing, which should remove any remaining doubts.
Thus, remove all the calls to cpu_has_xfeatures().
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
lib/crc/x86/crc-pclmul-template.h | 6 ++----
1 file changed, 2 insertions(+), 4 deletions(-)
diff --git a/lib/crc/x86/crc-pclmul-template.h b/lib/crc/x86/crc-pclmul-template.h
index 02744831c6fa..893119bb7c07 100644
--- a/lib/crc/x86/crc-pclmul-template.h
+++ b/lib/crc/x86/crc-pclmul-template.h
@@ -27,16 +27,14 @@ DEFINE_STATIC_CALL(prefix##_pclmul, prefix##_pclmul_sse)
static inline bool have_vpclmul(void)
{
return boot_cpu_has(X86_FEATURE_VPCLMULQDQ) &&
- boot_cpu_has(X86_FEATURE_AVX2) &&
- cpu_has_xfeatures(XFEATURE_MASK_YMM, NULL);
+ boot_cpu_has(X86_FEATURE_AVX2);
}
static inline bool have_avx512(void)
{
return boot_cpu_has(X86_FEATURE_AVX512BW) &&
boot_cpu_has(X86_FEATURE_AVX512VL) &&
- !boot_cpu_has(X86_FEATURE_PREFER_YMM) &&
- cpu_has_xfeatures(XFEATURE_MASK_AVX512, NULL);
+ !boot_cpu_has(X86_FEATURE_PREFER_YMM);
}
/*
--
2.55.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v3 6/8] x86/fpu: Remove cpu_has_xfeatures()
2026-08-13 2:14 [PATCH v3 0/8] x86: Remove cpu_has_xfeatures() and add AVX-512 xor_gen() Eric Biggers
` (4 preceding siblings ...)
2026-08-13 2:15 ` [PATCH v3 5/8] lib/crc: " Eric Biggers
@ 2026-08-13 2:15 ` Eric Biggers
2026-08-13 2:15 ` [PATCH v3 7/8] xor: Remove redundant X86_FEATURE_OSXSAVE check Eric Biggers
2026-08-13 2:15 ` [PATCH v3 8/8] xor: Add AVX-512 optimized xor_gen() Eric Biggers
7 siblings, 0 replies; 10+ messages in thread
From: Eric Biggers @ 2026-08-13 2:15 UTC (permalink / raw)
To: x86
Cc: linux-um, linux-raid, linux-crypto, linux-kernel,
Christoph Hellwig, Andrew Morton, Eric Biggers
The only remaining caller of cpu_has_xfeatures() is
print_xstate_features(), which uses it only to check and get the name of
a single feature.
Remove it and just inline the needed code into print_xstate_features().
This also makes the "unknown xstate feature" entry at index XFEATURE_MAX
of xfeature_names[] unnecessary, so remove that too.
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
arch/x86/include/asm/fpu/api.h | 9 -------
arch/x86/kernel/fpu/xstate.c | 44 +++-------------------------------
2 files changed, 3 insertions(+), 50 deletions(-)
diff --git a/arch/x86/include/asm/fpu/api.h b/arch/x86/include/asm/fpu/api.h
index 90c63fe19c0f..cfed8b24d64f 100644
--- a/arch/x86/include/asm/fpu/api.h
+++ b/arch/x86/include/asm/fpu/api.h
@@ -99,15 +99,6 @@ static inline void fpregs_assert_state_consistent(void) { }
*/
extern void switch_fpu_return(void);
-/*
- * Query the presence of one or more xfeatures. Works on any legacy CPU as well.
- *
- * If 'feature_name' is set then put a human-readable description of
- * the feature there as well - this can be used to print error (or success)
- * messages.
- */
-extern int cpu_has_xfeatures(u64 xfeatures_mask, const char **feature_name);
-
/* Trap handling */
extern int fpu__exception_code(struct fpu *fpu, int trap_nr);
extern void fpu_sync_fpstate(struct fpu *fpu);
diff --git a/arch/x86/kernel/fpu/xstate.c b/arch/x86/kernel/fpu/xstate.c
index 97cfd4fb6cc0..0b17452d1788 100644
--- a/arch/x86/kernel/fpu/xstate.c
+++ b/arch/x86/kernel/fpu/xstate.c
@@ -66,8 +66,8 @@ static const char *xfeature_names[] =
"AMX Tile config",
"AMX Tile data",
"APX registers",
- "unknown xstate feature",
};
+static_assert(ARRAY_SIZE(xfeature_names) == XFEATURE_MAX);
static unsigned short xsave_cpuid_features[] __initdata = {
[XFEATURE_FP] = X86_FEATURE_FPU,
@@ -122,44 +122,6 @@ static inline unsigned int next_xfeature_order(unsigned int i, u64 mask)
#define XSTATE_FLAG_SUPERVISOR BIT(0)
#define XSTATE_FLAG_ALIGNED64 BIT(1)
-/*
- * Return whether the system supports a given xfeature.
- *
- * Also return the name of the (most advanced) feature that the caller requested:
- */
-int cpu_has_xfeatures(u64 xfeatures_needed, const char **feature_name)
-{
- u64 xfeatures_missing = xfeatures_needed & ~fpu_kernel_cfg.max_features;
-
- if (unlikely(feature_name)) {
- long xfeature_idx, max_idx;
- u64 xfeatures_print;
- /*
- * So we use FLS here to be able to print the most advanced
- * feature that was requested but is missing. So if a driver
- * asks about "XFEATURE_MASK_SSE | XFEATURE_MASK_YMM" we'll print the
- * missing AVX feature - this is the most informative message
- * to users:
- */
- if (xfeatures_missing)
- xfeatures_print = xfeatures_missing;
- else
- xfeatures_print = xfeatures_needed;
-
- xfeature_idx = fls64(xfeatures_print)-1;
- max_idx = ARRAY_SIZE(xfeature_names)-1;
- xfeature_idx = min(xfeature_idx, max_idx);
-
- *feature_name = xfeature_names[xfeature_idx];
- }
-
- if (xfeatures_missing)
- return 0;
-
- return 1;
-}
-EXPORT_SYMBOL_GPL(cpu_has_xfeatures);
-
static bool xfeature_is_aligned64(int xfeature_nr)
{
return xstate_flags[xfeature_nr] & XSTATE_FLAG_ALIGNED64;
@@ -302,9 +264,9 @@ static void __init print_xstate_features(void)
for (i = 0; i < XFEATURE_MAX; i++) {
u64 mask = BIT_ULL(i);
- const char *name;
+ const char *name = xfeature_names[i];
- if (cpu_has_xfeatures(mask, &name))
+ if (fpu_kernel_cfg.max_features & mask)
pr_info("x86/fpu: Supporting XSAVE feature 0x%03Lx: '%s'\n", mask, name);
}
}
--
2.55.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v3 7/8] xor: Remove redundant X86_FEATURE_OSXSAVE check
2026-08-13 2:14 [PATCH v3 0/8] x86: Remove cpu_has_xfeatures() and add AVX-512 xor_gen() Eric Biggers
` (5 preceding siblings ...)
2026-08-13 2:15 ` [PATCH v3 6/8] x86/fpu: Remove cpu_has_xfeatures() Eric Biggers
@ 2026-08-13 2:15 ` Eric Biggers
2026-08-13 2:15 ` [PATCH v3 8/8] xor: Add AVX-512 optimized xor_gen() Eric Biggers
7 siblings, 0 replies; 10+ messages in thread
From: Eric Biggers @ 2026-08-13 2:15 UTC (permalink / raw)
To: x86
Cc: linux-um, linux-raid, linux-crypto, linux-kernel,
Christoph Hellwig, Andrew Morton, Eric Biggers
X86_FEATURE_AVX implies X86_FEATURE_OSXSAVE already.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
lib/raid/xor/x86/xor_arch.h | 3 +--
1 file changed, 1 insertion(+), 2 deletions(-)
diff --git a/lib/raid/xor/x86/xor_arch.h b/lib/raid/xor/x86/xor_arch.h
index 99fe85a213c6..991abe3f4bbd 100644
--- a/lib/raid/xor/x86/xor_arch.h
+++ b/lib/raid/xor/x86/xor_arch.h
@@ -18,8 +18,7 @@ extern struct xor_block_template xor_block_avx;
*/
static __always_inline void __init arch_xor_init(void)
{
- if (boot_cpu_has(X86_FEATURE_AVX) &&
- boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ if (boot_cpu_has(X86_FEATURE_AVX)) {
xor_force(&xor_block_avx);
} else if (IS_ENABLED(CONFIG_X86_64) || boot_cpu_has(X86_FEATURE_XMM)) {
xor_register(&xor_block_sse);
--
2.55.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* [PATCH v3 8/8] xor: Add AVX-512 optimized xor_gen()
2026-08-13 2:14 [PATCH v3 0/8] x86: Remove cpu_has_xfeatures() and add AVX-512 xor_gen() Eric Biggers
` (6 preceding siblings ...)
2026-08-13 2:15 ` [PATCH v3 7/8] xor: Remove redundant X86_FEATURE_OSXSAVE check Eric Biggers
@ 2026-08-13 2:15 ` Eric Biggers
2026-08-13 2:39 ` sashiko-bot
7 siblings, 1 reply; 10+ messages in thread
From: Eric Biggers @ 2026-08-13 2:15 UTC (permalink / raw)
To: x86
Cc: linux-um, linux-raid, linux-crypto, linux-kernel,
Christoph Hellwig, Andrew Morton, Eric Biggers, David Laight
Add an implementation of xor_gen() using AVX-512.
It uses 512-bit vectors, i.e. ZMM registers. It also uses the
vpternlogq instruction to do three-input XORs when applicable.
It's enabled on x86_64 CPUs that have AVX512F && !PREFER_YMM. In
practice that means:
- AMD Zen 4 and later (client and server)
- Intel Sapphire Rapids and later (server)
- Intel Rocket Lake (client)
- Intel Nova Lake and later (client)
The !PREFER_YMM condition excludes the older AVX-512 implementations in
Intel Skylake Server and Intel Ice Lake. They could run this code, but
they're known to have overly-eager downclocking when ZMM registers are
used. This is the same policy that the crypto and CRC code uses.
Benchmark on AMD Ryzen 9 9950X (Zen 5):
src_cnt avx avx512 Improvement
======= ========== ========== ===========
1 56353 MB/s 75388 MB/s 33%
2 54274 MB/s 68409 MB/s 26%
3 44649 MB/s 64042 MB/s 43%
4 41315 MB/s 55002 MB/s 33%
Reviewed-by: David Laight <david.laight.linux@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Acked-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
---
lib/raid/xor/Makefile | 2 +-
lib/raid/xor/x86/xor-avx512.c | 121 ++++++++++++++++++++++++++++++++++
lib/raid/xor/x86/xor_arch.h | 30 ++++++---
3 files changed, 142 insertions(+), 11 deletions(-)
create mode 100644 lib/raid/xor/x86/xor-avx512.c
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index e8ecec3c09f9..4a0e5c6d8298 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -29,7 +29,7 @@ xor-$(CONFIG_SPARC32) += sparc/xor-sparc32.o
xor-$(CONFIG_SPARC64) += sparc/xor-sparc64.o sparc/xor-sparc64-glue.o
xor-$(CONFIG_S390) += s390/xor.o
xor-$(CONFIG_X86_32) += x86/xor-avx.o x86/xor-sse.o x86/xor-mmx.o
-xor-$(CONFIG_X86_64) += x86/xor-avx.o x86/xor-sse.o
+xor-$(CONFIG_X86_64) += x86/xor-avx.o x86/xor-sse.o x86/xor-avx512.o
obj-y += tests/
CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH)
diff --git a/lib/raid/xor/x86/xor-avx512.c b/lib/raid/xor/x86/xor-avx512.c
new file mode 100644
index 000000000000..17f57900d827
--- /dev/null
+++ b/lib/raid/xor/x86/xor-avx512.c
@@ -0,0 +1,121 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * AVX-512 optimized implementation of xor_gen()
+ *
+ * Copyright 2026 Google LLC
+ */
+
+#include <linux/types.h>
+#include <asm/fpu/api.h>
+#include "xor_impl.h"
+#include "xor_arch.h"
+
+/*
+ * Implementation notes:
+ *
+ * Unrolling by the number of buffers (2-5) is very important.
+ *
+ * Unrolling by length is less important, especially when using register-indexed
+ * addressing with negative indices from the end of the buffers. That approach
+ * results in just two loop control instructions being needed per iteration,
+ * regardless of the number of buffers.
+ *
+ * In fact, benchmarks showed that the 2 and 3 buffer cases require only 2x
+ * unrolling by length, while the 4 and 5 buffer cases don't require any
+ * unrolling by length. Benchmarks also showed that the register-indexed
+ * addressing isn't a bottleneck either; i.e., we can't do any better by
+ * incrementing the pointers as we go along, even with more unrolling.
+ */
+
+static void xor_avx512_2(long bytes, u8 *p1, const u8 *p2)
+{
+ long i = -bytes;
+
+ asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n"
+ "vmovdqa64 64(%1,%0), %%zmm1\n"
+ "vpxorq (%2,%0), %%zmm0, %%zmm0\n"
+ "vpxorq 64(%2,%0), %%zmm1, %%zmm1\n"
+ "vmovdqa64 %%zmm0, (%1,%0)\n"
+ "vmovdqa64 %%zmm1, 64(%1,%0)\n"
+ "add $128, %0\n"
+ "jnz 1b\n"
+ : "+&r"(i)
+ : "r"(p1 + bytes), "r"(p2 + bytes)
+ : "memory", "cc");
+}
+
+static void xor_avx512_3(long bytes, u8 *p1, const u8 *p2, const u8 *p3)
+{
+ long i = -bytes;
+
+ asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n"
+ "vmovdqa64 64(%1,%0), %%zmm1\n"
+ "vmovdqa64 (%2,%0), %%zmm2\n"
+ "vmovdqa64 64(%2,%0), %%zmm3\n"
+ "vpternlogq $0x96, (%3,%0), %%zmm2, %%zmm0\n"
+ "vpternlogq $0x96, 64(%3,%0), %%zmm3, %%zmm1\n"
+ "vmovdqa64 %%zmm0, (%1,%0)\n"
+ "vmovdqa64 %%zmm1, 64(%1,%0)\n"
+ "add $128, %0\n"
+ "jnz 1b\n"
+ : "+&r"(i)
+ : "r"(p1 + bytes), "r"(p2 + bytes), "r"(p3 + bytes)
+ : "memory", "cc");
+}
+
+static void xor_avx512_4(long bytes, u8 *p1, const u8 *p2, const u8 *p3,
+ const u8 *p4)
+{
+ long i = -bytes;
+
+ asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n"
+ "vmovdqa64 (%2,%0), %%zmm1\n"
+ "vpxorq (%3,%0), %%zmm0, %%zmm0\n"
+ "vpternlogq $0x96, (%4,%0), %%zmm1, %%zmm0\n"
+ "vmovdqa64 %%zmm0, (%1,%0)\n"
+ "add $64, %0\n"
+ "jnz 1b\n"
+ : "+&r"(i)
+ : "r"(p1 + bytes), "r"(p2 + bytes), "r"(p3 + bytes),
+ "r"(p4 + bytes)
+ : "memory", "cc");
+}
+
+static void xor_avx512_5(long bytes, u8 *p1, const u8 *p2, const u8 *p3,
+ const u8 *p4, const u8 *p5)
+{
+ long i = -bytes;
+
+ asm volatile("1: vmovdqa64 (%1,%0), %%zmm0\n"
+ "vmovdqa64 (%2,%0), %%zmm1\n"
+ "vpternlogq $0x96, (%3,%0), %%zmm1, %%zmm0\n"
+ "vmovdqa64 (%4,%0), %%zmm1\n"
+ "vpternlogq $0x96, (%5,%0), %%zmm1, %%zmm0\n"
+ "vmovdqa64 %%zmm0, (%1,%0)\n"
+ "add $64, %0\n"
+ "jnz 1b\n"
+ : "+&r"(i)
+ : "r"(p1 + bytes), "r"(p2 + bytes), "r"(p3 + bytes),
+ "r"(p4 + bytes), "r"(p5 + bytes)
+ : "memory", "cc");
+}
+
+DO_XOR_BLOCKS(avx512_inner, xor_avx512_2, xor_avx512_3, xor_avx512_4,
+ xor_avx512_5);
+
+/*
+ * Preconditions: bytes is a nonzero multiple of 512, and all buffers are
+ * 64-byte aligned.
+ */
+static void xor_gen_avx512(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes)
+{
+ kernel_fpu_begin();
+ xor_gen_avx512_inner(dest, srcs, src_cnt, bytes);
+ kernel_fpu_end();
+}
+
+struct xor_block_template xor_block_avx512 = {
+ .name = "avx512",
+ .xor_gen = xor_gen_avx512,
+};
diff --git a/lib/raid/xor/x86/xor_arch.h b/lib/raid/xor/x86/xor_arch.h
index 991abe3f4bbd..ed5921d2e2aa 100644
--- a/lib/raid/xor/x86/xor_arch.h
+++ b/lib/raid/xor/x86/xor_arch.h
@@ -6,21 +6,31 @@ extern struct xor_block_template xor_block_p5_mmx;
extern struct xor_block_template xor_block_sse;
extern struct xor_block_template xor_block_sse_pf64;
extern struct xor_block_template xor_block_avx;
+extern struct xor_block_template xor_block_avx512;
-/*
- * When SSE is available, use it as it can write around L2. We may also be able
- * to load into the L1 only depending on how the cpu deals with a load to a line
- * that is being prefetched.
- *
- * When AVX2 is available, force using it as it is better by all measures.
- *
- * 32-bit without MMX can fall back to the generic routines.
- */
static __always_inline void __init arch_xor_init(void)
{
- if (boot_cpu_has(X86_FEATURE_AVX)) {
+ if (IS_ENABLED(CONFIG_X86_64) && boot_cpu_has(X86_FEATURE_AVX512F) &&
+ !boot_cpu_has(X86_FEATURE_PREFER_YMM)) {
+ /*
+ * Use the AVX-512 code on CPUs that support AVX-512 without
+ * overly-eager downclocking. On such CPUs the AVX-512 code
+ * should always work at least as well as the AVX code, so
+ * runtime selection is unnecessary.
+ *
+ * The AVX-512 code can work on X86_32. However, due to lack of
+ * use case for that, for now it's built only for X86_64.
+ */
+ xor_force(&xor_block_avx512);
+ } else if (boot_cpu_has(X86_FEATURE_AVX)) {
+ /* AVX will be the best; no need to try others. */
xor_force(&xor_block_avx);
} else if (IS_ENABLED(CONFIG_X86_64) || boot_cpu_has(X86_FEATURE_XMM)) {
+ /*
+ * When SSE is available, use it as it can write around L2. We
+ * may also be able to load into the L1 only depending on how
+ * the cpu deals with a load to a line that is being prefetched.
+ */
xor_register(&xor_block_sse);
xor_register(&xor_block_sse_pf64);
} else if (boot_cpu_has(X86_FEATURE_MMX)) {
--
2.55.0
^ permalink raw reply related [flat|nested] 10+ messages in thread* Re: [PATCH v3 8/8] xor: Add AVX-512 optimized xor_gen()
2026-08-13 2:15 ` [PATCH v3 8/8] xor: Add AVX-512 optimized xor_gen() Eric Biggers
@ 2026-08-13 2:39 ` sashiko-bot
0 siblings, 0 replies; 10+ messages in thread
From: sashiko-bot @ 2026-08-13 2:39 UTC (permalink / raw)
To: Eric Biggers; +Cc: linux-raid, yukuai
Thank you for your contribution! Sashiko AI review found 1 potential issue(s) to consider:
- [Medium] Missing `vzeroupper` instruction after using AVX-512 (ZMM) registers, leading to severe AVX-to-SSE transition penalties in kernel threads.
--
--- Patch [8]: [PATCH v3 8/8] xor: Add AVX-512 optimized xor_gen() ---
commit 0074b2adffd00d86b2a03e4d30495b21794ecb20
Author: Eric Biggers <ebiggers@kernel.org>
xor: Add AVX-512 optimized xor_gen()
This commit introduces an AVX-512 optimized implementation of xor_gen()
using 512-bit ZMM registers and the vpternlogq instruction. It is enabled
on x86_64 CPUs with AVX512F that do not have the PREFER_YMM feature set.
> diff --git a/lib/raid/xor/x86/xor-avx512.c b/lib/raid/xor/x86/xor-avx512.c
> new file mode 100644
> index 0000000000000..17f57900d8274
> --- /dev/null
> +++ b/lib/raid/xor/x86/xor-avx512.c
[ ... ]
> +static void xor_gen_avx512(void *dest, void **srcs, unsigned int src_cnt,
> + unsigned int bytes)
> +{
> + kernel_fpu_begin();
> + xor_gen_avx512_inner(dest, srcs, src_cnt, bytes);
[Severity: Medium]
Does this code need a vzeroupper instruction before releasing the FPU to avoid
AVX-to-SSE transition penalties? If a kernel thread modifies the ZMM registers
here and subsequently executes legacy SSE instructions (for example, in kcryptd)
without an intervening user-space context switch to clear the state, the CPU
could incur a severe transition penalty on affected architectures.
While this problem already existed in the xor-avx.c implementation using YMM
registers, the newly introduced xor-avx512.c appears to repeat the defect.
> + kernel_fpu_end();
> +}
--
Sashiko AI review · https://sashiko.dev/#/patchset/20260813021506.55129-1-ebiggers@kernel.org?part=8
^ permalink raw reply [flat|nested] 10+ messages in thread