* [PATCH 0/2] raid6/arm64: Use aggregation and 3-way XOR
@ 2026-11-18 18:33 Ard Biesheuvel
2026-11-18 18:33 ` [PATCH 1/2] raid6: neon: process four data disks per Horner step Ard Biesheuvel
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Ard Biesheuvel @ 2026-11-18 18:33 UTC (permalink / raw)
To: linux-arm-kernel
Cc: Ard Biesheuvel, Christoph Hellwig, Eric Biggers, Demian Shulhan
From: Ard Biesheuvel <ardb@kernel.org>
Tweak the arm64/ARM P/Q gen/xor code to use 3-way XOR instructions where
available. Details are in the commit logs.
Cc: Christoph Hellwig <hch@lst.de>
Cc: Eric Biggers <ebiggers@kernel.org>
Cc: Demian Shulhan <demyansh@gmail.com>
Ard Biesheuvel (2):
raid6: neon: process four data disks per Horner step
raid6: arm64: use the EOR3 instruction where available
lib/raid/raid6/.gitignore | 1 +
lib/raid/raid6/Makefile | 19 ++
lib/raid/raid6/arm/neon.c | 44 +++--
lib/raid/raid6/arm/neon.h | 28 ++-
lib/raid/raid6/arm/neon.uc | 196 ++++++++++++++------
lib/raid/raid6/arm/neon_compat.h | 22 +++
lib/raid/raid6/arm/recov_neon_inner.c | 19 +-
lib/raid/raid6/arm64/pq_arch.h | 34 +++-
8 files changed, 267 insertions(+), 96 deletions(-)
create mode 100644 lib/raid/raid6/arm/neon_compat.h
--
2.56.1.708.g77570a6a1d-goog
^ permalink raw reply [flat|nested] 9+ messages in thread* [PATCH 1/2] raid6: neon: process four data disks per Horner step 2026-11-18 18:33 [PATCH 0/2] raid6/arm64: Use aggregation and 3-way XOR Ard Biesheuvel @ 2026-11-18 18:33 ` Ard Biesheuvel 2026-11-19 22:44 ` Eric Biggers 2026-11-19 23:51 ` Demian Shulhan 2026-11-18 18:33 ` [PATCH 2/2] raid6: arm64: use the EOR3 instruction where available Ard Biesheuvel ` (2 subsequent siblings) 3 siblings, 2 replies; 9+ messages in thread From: Ard Biesheuvel @ 2026-11-18 18:33 UTC (permalink / raw) To: linux-arm-kernel Cc: Ard Biesheuvel, Christoph Hellwig, Eric Biggers, Demian Shulhan From: Ard Biesheuvel <ardb@kernel.org> The NEON code evaluates the Q syndrome using Horner's rule, multiplying by x = {02} once for each data disk. This takes six instructions per data disk and vector: four for the multiplication by x, and two XORs to add the data to P and Q. It also creates a dependency chain through Q that is four instructions long for each data disk. Instead, process four data disks at a time, i.e., Q' = x^4.Q + x^3.D[z+3] + x^2.D[z+2] + x.D[z+1] + D[z] Multiplying by x^n amounts to shifting each byte left by n bits, and reducing the n bits that are shifted out modulo the RAID-6 polynomial. Since this reduction is linear, the bits shifted out of all four terms can be XORed together first, and reduced using a single polynomial multiplication (PMUL) by 0x1d, which is exact as long as no more than 4 bits are shifted out. This reduces the instruction count to 5.25 per data disk and vector, and the length of the dependency chain through Q to that of a single step per four data disks. In xor_syndrome(), replace the loop that scales Q by x^4 at a time for the unmodified disks on the 'left side' with a single multiplication by g^start, using the nibble based table lookups that the recovery code uses already. This is much faster when only a few high numbered disks are modified, which is the common case for partial stripe writes. Put the code that performs the Horner steps for each lane in a helper function, so that unroll.awk keeps the instructions for each lane together. This prevents the 8-way unrolled version from running out of registers. While at it, use the ^ and & operators rather than the veorq_u8() and vandq_u8() intrinsics, which makes the code easier to read, and allows the compiler to use EOR3 instructions by itself, as long as the SHA3 extension is enabled (see the next patch). Results on an Apple M2 running Linux, best of the 1x/2x/4x/8x versions, in MB/s, using a userspace build of the same code (4 KiB per disk): gen_syndrome() 2 data 4 data 8 data 16 data 32 data Avalanche (P) old 55203 42463 37529 35354 34744 new 54099 46028 41733 40484 39569 Blizzard (E) old 19790 15393 14050 13333 12887 new 21670 16197 15524 13915 12475 xor_syndrome(), 16 data disks, stop = 15, in GB/s by value of start: 0 4 8 12 15 Avalanche (P) old 34 33 30 24 11 new 39 37 36 33 25 Blizzard (E) old 12 12 11 9 4 new 13 14 13 12 9 Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- lib/raid/raid6/arm/neon.c | 6 +- lib/raid/raid6/arm/neon.h | 12 +- lib/raid/raid6/arm/neon.uc | 180 ++++++++++++++------ lib/raid/raid6/arm/neon_compat.h | 22 +++ lib/raid/raid6/arm/recov_neon_inner.c | 19 +-- 5 files changed, 161 insertions(+), 78 deletions(-) diff --git a/lib/raid/raid6/arm/neon.c b/lib/raid/raid6/arm/neon.c index af90869aaffc..2f8e98f5556e 100644 --- a/lib/raid/raid6/arm/neon.c +++ b/lib/raid/raid6/arm/neon.c @@ -34,10 +34,12 @@ size_t bytes, void **ptrs) \ { \ void raid6_neon ## _n ## _xor_syndrome_real(int, \ - int, int, unsigned long, void**); \ + int, int, unsigned long, void**, \ + const u8 *); \ scoped_ksimd() \ raid6_neon ## _n ## _xor_syndrome_real(disks, \ - start, stop, (unsigned long)bytes, ptrs);\ + start, stop, (unsigned long)bytes, ptrs,\ + raid6_vgfmul[raid6_gfexp[start]]); \ } \ struct raid6_calls const raid6_neonx ## _n = { \ .gen_syndrome = raid6_neon ## _n ## _gen_syndrome, \ diff --git a/lib/raid/raid6/arm/neon.h b/lib/raid/raid6/arm/neon.h index 2ca41ee9b499..a9977ec57f27 100644 --- a/lib/raid/raid6/arm/neon.h +++ b/lib/raid/raid6/arm/neon.h @@ -2,16 +2,20 @@ void raid6_neon1_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); void raid6_neon1_xor_syndrome_real(int disks, int start, int stop, - unsigned long bytes, void **ptrs); + unsigned long bytes, void **ptrs, + const uint8_t *gmul); void raid6_neon2_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); void raid6_neon2_xor_syndrome_real(int disks, int start, int stop, - unsigned long bytes, void **ptrs); + unsigned long bytes, void **ptrs, + const uint8_t *gmul); void raid6_neon4_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); void raid6_neon4_xor_syndrome_real(int disks, int start, int stop, - unsigned long bytes, void **ptrs); + unsigned long bytes, void **ptrs, + const uint8_t *gmul); void raid6_neon8_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); void raid6_neon8_xor_syndrome_real(int disks, int start, int stop, - unsigned long bytes, void **ptrs); + unsigned long bytes, void **ptrs, + const uint8_t *gmul); void __raid6_2data_recov_neon(int bytes, uint8_t *p, uint8_t *q, uint8_t *dp, uint8_t *dq, const uint8_t *pbmul, const uint8_t *qmul); diff --git a/lib/raid/raid6/arm/neon.uc b/lib/raid/raid6/arm/neon.uc index 14a9fc2c60fa..6f74e2b82fc4 100644 --- a/lib/raid/raid6/arm/neon.uc +++ b/lib/raid/raid6/arm/neon.uc @@ -22,15 +22,29 @@ * $#-way unrolled NEON intrinsics math RAID-6 instruction set * * This file is postprocessed using unroll.awk + * + * The Q syndrome is evaluated using Horner's rule, but four data disks at a + * time, i.e., + * + * Q' = x^4 * Q + x^3 * D[z+3] + x^2 * D[z+2] + x * D[z+1] + D[z] + * + * Multiplying by x^n amounts to shifting each byte left by n bits, and + * reducing the n bits that are shifted out modulo the RAID-6 polynomial. + * Since this reduction is linear, the bits shifted out of all terms can be + * XORed together first, and reduced using a single polynomial multiplication + * by 0x1d, which is exact as long as no more than 4 bits are shifted out. */ #include <arm_neon.h> #include "arm/neon.h" +#include "arm/neon_compat.h" typedef uint8x16_t unative_t; #define NSIZE sizeof(unative_t) +#define RAID6_NEON_FN(n, fn) raid6_neon ## n ## _ ## fn + /* * The SHLBYTE() operation shifts each byte left by 1, *not* * rolling over into the next byte @@ -54,13 +68,85 @@ static inline unative_t PMUL(unative_t v, unative_t u) return (unative_t)vmulq_p8((poly8x16_t)v, (poly8x16_t)u); } -void raid6_neon$#_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs) +/* Q * x + D */ +static inline unative_t HORNER1(unative_t q, unative_t d, unative_t x1d) +{ + return SHLBYTE(q) ^ (MASK(q) & x1d) ^ d; +} + +/* Q * x^2 + D1 * x + D0 */ +static inline unative_t HORNER2(unative_t q, unative_t d1, unative_t d0, + unative_t x1d) +{ + unative_t lo = vshlq_n_u8(q, 2) ^ vshlq_n_u8(d1, 1) ^ d0; + unative_t hi = vshrq_n_u8(q, 6) ^ vshrq_n_u8(d1, 7); + + return lo ^ PMUL(hi, x1d); +} + +/* Q * x^4 + D3 * x^3 + D2 * x^2 + D1 * x + D0 */ +static inline unative_t HORNER4(unative_t q, unative_t d3, unative_t d2, + unative_t d1, unative_t d0, unative_t x1d) +{ + unative_t lo = vshlq_n_u8(q, 4) ^ vshlq_n_u8(d3, 3) ^ + vshlq_n_u8(d2, 2) ^ vshlq_n_u8(d1, 1) ^ d0; + unative_t hi = vshrq_n_u8(q, 4) ^ vshrq_n_u8(d3, 5) ^ + vshrq_n_u8(d2, 6) ^ vshrq_n_u8(d1, 7); + + return lo ^ PMUL(hi, x1d); +} + +/* + * Perform a Horner step for a single lane. Each step is a single statement + * so that unroll.awk keeps the code for each lane together, which avoids + * running out of registers with the wider unrolls. + */ +static inline void STEP1(unative_t *wp, unative_t *wq, const uint8_t *s, + unative_t x1d) +{ + unative_t d = vld1q_u8(s); + + *wp ^= d; + *wq = HORNER1(*wq, d, x1d); +} + +static inline void STEP2(unative_t *wp, unative_t *wq, const uint8_t *s1, + const uint8_t *s0, unative_t x1d) +{ + unative_t d1 = vld1q_u8(s1); + unative_t d0 = vld1q_u8(s0); + + *wp ^= d1 ^ d0; + *wq = HORNER2(*wq, d1, d0, x1d); +} + +static inline void STEP4(unative_t *wp, unative_t *wq, const uint8_t *s3, + const uint8_t *s2, const uint8_t *s1, + const uint8_t *s0, unative_t x1d) +{ + unative_t d3 = vld1q_u8(s3); + unative_t d2 = vld1q_u8(s2); + unative_t d1 = vld1q_u8(s1); + unative_t d0 = vld1q_u8(s0); + + *wp ^= d3 ^ d2 ^ d1 ^ d0; + *wq = HORNER4(*wq, d3, d2, d1, d0, x1d); +} + +/* Multiply by the constant described by the nibble tables in m0 and m1 */ +static inline unative_t GFMUL(unative_t v, unative_t m0, unative_t m1) +{ + return vqtbl1q_u8(m0, v & 0x0f) ^ vqtbl1q_u8(m1, vshrq_n_u8(v, 4)); +} + +void RAID6_NEON_FN($#, gen_syndrome_real)(int disks, unsigned long bytes, + void **ptrs) { uint8_t **dptr = (uint8_t **)ptrs; uint8_t *p, *q; int d, z, z0; - register unative_t wd$$, wq$$, wp$$, w1$$, w2$$; + unative_t wq$$, wp$$; const unative_t x1d = vdupq_n_u8(0x1d); z0 = disks - 3; /* Highest data disk */ @@ -69,30 +155,41 @@ void raid6_neon$#_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs) for ( d = 0 ; d < bytes ; d += NSIZE*$# ) { wq$$ = wp$$ = vld1q_u8(&dptr[z0][d+$$*NSIZE]); - for ( z = z0-1 ; z >= 0 ; z-- ) { - wd$$ = vld1q_u8(&dptr[z][d+$$*NSIZE]); - wp$$ = veorq_u8(wp$$, wd$$); - w2$$ = MASK(wq$$); - w1$$ = SHLBYTE(wq$$); - - w2$$ = vandq_u8(w2$$, x1d); - w1$$ = veorq_u8(w1$$, w2$$); - wq$$ = veorq_u8(w1$$, wd$$); + z = z0 - 1; + + /* Process 1 and/or 2 disks so that a multiple of 4 remains */ + if (z0 & 1) { + STEP1(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], x1d); + z--; + } + if (z0 & 2) { + STEP2(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], &dptr[z-1][d+$$*NSIZE], x1d); + z -= 2; + } + for ( ; z >= 0 ; z -= 4 ) { + STEP4(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], &dptr[z-1][d+$$*NSIZE], &dptr[z-2][d+$$*NSIZE], &dptr[z-3][d+$$*NSIZE], x1d); } vst1q_u8(&p[d+NSIZE*$$], wp$$); vst1q_u8(&q[d+NSIZE*$$], wq$$); } } -void raid6_neon$#_xor_syndrome_real(int disks, int start, int stop, - unsigned long bytes, void **ptrs) +/* + * @gmul points to the 32 byte nibble table (see raid6_vgfmul[]) that + * multiplies by g^start. + */ +void RAID6_NEON_FN($#, xor_syndrome_real)(int disks, int start, int stop, + unsigned long bytes, void **ptrs, + const uint8_t *gmul) { uint8_t **dptr = (uint8_t **)ptrs; uint8_t *p, *q; int d, z, z0; - register unative_t wd$$, wq$$, wp$$, w1$$, w2$$; + unative_t wq$$, wp$$; const unative_t x1d = vdupq_n_u8(0x1d); + const unative_t gm0 = vld1q_u8(gmul); + const unative_t gm1 = vld1q_u8(gmul + 16); z0 = stop; /* P/Q right side optimization */ p = dptr[disks-2]; /* XOR parity */ @@ -100,53 +197,28 @@ void raid6_neon$#_xor_syndrome_real(int disks, int start, int stop, for ( d = 0 ; d < bytes ; d += NSIZE*$# ) { wq$$ = vld1q_u8(&dptr[z0][d+$$*NSIZE]); - wp$$ = veorq_u8(vld1q_u8(&p[d+$$*NSIZE]), wq$$); + wp$$ = vld1q_u8(&p[d+$$*NSIZE]) ^ wq$$; + z = z0 - 1; /* P/Q data pages */ - for ( z = z0-1 ; z >= start ; z-- ) { - wd$$ = vld1q_u8(&dptr[z][d+$$*NSIZE]); - wp$$ = veorq_u8(wp$$, wd$$); - w2$$ = MASK(wq$$); - w1$$ = SHLBYTE(wq$$); - - w2$$ = vandq_u8(w2$$, x1d); - w1$$ = veorq_u8(w1$$, w2$$); - wq$$ = veorq_u8(w1$$, wd$$); + if ((z0 - start) & 1) { + STEP1(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], x1d); + z--; } - /* P/Q left side optimization */ - for ( z = start-1 ; z >= 3 ; z -= 4 ) { - w2$$ = vshrq_n_u8(wq$$, 4); - w1$$ = vshlq_n_u8(wq$$, 4); - - w2$$ = PMUL(w2$$, x1d); - wq$$ = veorq_u8(w1$$, w2$$); + if ((z0 - start) & 2) { + STEP2(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], &dptr[z-1][d+$$*NSIZE], x1d); + z -= 2; + } + for ( ; z >= start ; z -= 4 ) { + STEP4(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], &dptr[z-1][d+$$*NSIZE], &dptr[z-2][d+$$*NSIZE], &dptr[z-3][d+$$*NSIZE], x1d); } - switch (z) { - case 2: - w2$$ = vshrq_n_u8(wq$$, 5); - w1$$ = vshlq_n_u8(wq$$, 3); - - w2$$ = PMUL(w2$$, x1d); - wq$$ = veorq_u8(w1$$, w2$$); - break; - case 1: - w2$$ = vshrq_n_u8(wq$$, 6); - w1$$ = vshlq_n_u8(wq$$, 2); - - w2$$ = PMUL(w2$$, x1d); - wq$$ = veorq_u8(w1$$, w2$$); - break; - case 0: - w2$$ = MASK(wq$$); - w1$$ = SHLBYTE(wq$$); - - w2$$ = vandq_u8(w2$$, x1d); - wq$$ = veorq_u8(w1$$, w2$$); + /* P/Q left side optimization: multiply by g^start */ + if (start) { + wq$$ = GFMUL(wq$$, gm0, gm1); } - w1$$ = vld1q_u8(&q[d+NSIZE*$$]); - wq$$ = veorq_u8(wq$$, w1$$); + wq$$ ^= vld1q_u8(&q[d+NSIZE*$$]); vst1q_u8(&p[d+NSIZE*$$], wp$$); vst1q_u8(&q[d+NSIZE*$$], wq$$); } diff --git a/lib/raid/raid6/arm/neon_compat.h b/lib/raid/raid6/arm/neon_compat.h new file mode 100644 index 000000000000..81410b2c2c99 --- /dev/null +++ b/lib/raid/raid6/arm/neon_compat.h @@ -0,0 +1,22 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ +/* + * Copyright (C) 2017 Linaro Ltd. <ard.biesheuvel@linaro.org> + */ + +#ifdef CONFIG_ARM +/* + * AArch32 does not provide this intrinsic natively because it does not + * implement the underlying instruction. AArch32 only provides a 64-bit + * wide vtbl.8 instruction, so use that instead. + */ +static inline uint8x16_t vqtbl1q_u8(uint8x16_t a, uint8x16_t b) +{ + union { + uint8x16_t val; + uint8x8x2_t pair; + } __a = { a }; + + return vcombine_u8(vtbl2_u8(__a.pair, vget_low_u8(b)), + vtbl2_u8(__a.pair, vget_high_u8(b))); +} +#endif diff --git a/lib/raid/raid6/arm/recov_neon_inner.c b/lib/raid/raid6/arm/recov_neon_inner.c index 53c355efa7ff..66f687e5dea8 100644 --- a/lib/raid/raid6/arm/recov_neon_inner.c +++ b/lib/raid/raid6/arm/recov_neon_inner.c @@ -6,24 +6,7 @@ #include <arm_neon.h> #include "arm/neon.h" - -#ifdef CONFIG_ARM -/* - * AArch32 does not provide this intrinsic natively because it does not - * implement the underlying instruction. AArch32 only provides a 64-bit - * wide vtbl.8 instruction, so use that instead. - */ -static uint8x16_t vqtbl1q_u8(uint8x16_t a, uint8x16_t b) -{ - union { - uint8x16_t val; - uint8x8x2_t pair; - } __a = { a }; - - return vcombine_u8(vtbl2_u8(__a.pair, vget_low_u8(b)), - vtbl2_u8(__a.pair, vget_high_u8(b))); -} -#endif +#include "arm/neon_compat.h" void __raid6_2data_recov_neon(int bytes, uint8_t *p, uint8_t *q, uint8_t *dp, uint8_t *dq, const uint8_t *pbmul, -- 2.56.1.708.g77570a6a1d-goog ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] raid6: neon: process four data disks per Horner step 2026-11-18 18:33 ` [PATCH 1/2] raid6: neon: process four data disks per Horner step Ard Biesheuvel @ 2026-11-19 22:44 ` Eric Biggers 2026-11-19 23:51 ` Demian Shulhan 1 sibling, 0 replies; 9+ messages in thread From: Eric Biggers @ 2026-11-19 22:44 UTC (permalink / raw) To: Ard Biesheuvel Cc: linux-arm-kernel, Ard Biesheuvel, Christoph Hellwig, Demian Shulhan On Wed, Nov 18, 2026 at 07:33:37PM +0100, Ard Biesheuvel wrote: > diff --git a/lib/raid/raid6/arm/neon.uc b/lib/raid/raid6/arm/neon.uc > index 14a9fc2c60fa..6f74e2b82fc4 100644 > --- a/lib/raid/raid6/arm/neon.uc > +++ b/lib/raid/raid6/arm/neon.uc > @@ -22,15 +22,29 @@ > * $#-way unrolled NEON intrinsics math RAID-6 instruction set > * > * This file is postprocessed using unroll.awk > + * > + * The Q syndrome is evaluated using Horner's rule, but four data disks at a > + * time, i.e., > + * > + * Q' = x^4 * Q + x^3 * D[z+3] + x^2 * D[z+2] + x * D[z+1] + D[z] In gen_syndrome_real() and xor_syndrome_real() later in the file, z is the highest disk index in each 4-disk step rather than the lowest. So it might make more sense to write this as: Q' = x^4 * Q + x^3 * D[z] + x^2 * D[z-1] + x * D[z-2] + D[z-3] > +/* Q * x^4 + D3 * x^3 + D2 * x^2 + D1 * x + D0 */ > +static inline unative_t HORNER4(unative_t q, unative_t d3, unative_t d2, > + unative_t d1, unative_t d0, unative_t x1d) > +{ > + unative_t lo = vshlq_n_u8(q, 4) ^ vshlq_n_u8(d3, 3) ^ > + vshlq_n_u8(d2, 2) ^ vshlq_n_u8(d1, 1) ^ d0; > + unative_t hi = vshrq_n_u8(q, 4) ^ vshrq_n_u8(d3, 5) ^ > + vshrq_n_u8(d2, 6) ^ vshrq_n_u8(d1, 7); > + > + return lo ^ PMUL(hi, x1d); > +} This is called right after 'q' is updated, so it will take a bit longer to be available than the other inputs, and ideally the generated code would XOR together the other inputs first. Putting 'q' last might make it more likely for the compiler to do that: unative_t lo = vshlq_n_u8(d3, 3) ^ vshlq_n_u8(d2, 2) ^ vshlq_n_u8(d1, 1) ^ d0 ^ vshlq_n_u8(q, 4); unative_t hi = vshrq_n_u8(d3, 5) ^ vshrq_n_u8(d2, 6) ^ vshrq_n_u8(d1, 7) ^ vshrq_n_u8(q, 4); - Eric ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 1/2] raid6: neon: process four data disks per Horner step 2026-11-18 18:33 ` [PATCH 1/2] raid6: neon: process four data disks per Horner step Ard Biesheuvel 2026-11-19 22:44 ` Eric Biggers @ 2026-11-19 23:51 ` Demian Shulhan 1 sibling, 0 replies; 9+ messages in thread From: Demian Shulhan @ 2026-11-19 23:51 UTC (permalink / raw) To: Ard Biesheuvel Cc: linux-arm-kernel, Ard Biesheuvel, Christoph Hellwig, Eric Biggers Hi Ard, I tested this on ARM Neoverse-V2 (GCC 15.2 and Clang 21.1) and have a couple of suggestions: > static inline unative_t PMUL(unative_t v, unative_t u) > { > return (unative_t)vmulq_p8((poly8x16_t)v, (poly8x16_t)u); > } Since 'hi' in HORNER2/HORNER4 only holds 4-bit values in [0..15], reducing via vqtbl1q_u8() against the 16-byte low-nibble table in raid6_vgfmul[0x1d] (on !CONFIG_ARM, where single-register TBL is native) is exact and avoids the 2-pipe PMUL+shift bottleneck on cores like Neoverse-V2, where single-register TBL issues across all 4 ASIMD pipes. > +void RAID6_NEON_FN($#, gen_syndrome_real)(int disks, unsigned long bytes, > + void **ptrs) > { > uint8_t **dptr = (uint8_t **)ptrs; > uint8_t *p, *q; > int d, z, z0; While touching this, making 'd' unsigned long (here and in xor_syndrome_real()) avoids 32-bit sign-extension (sxtw) of the loop index and lets GCC form paired ldp/stp q, q accesses. BR, Demian ср, 18 лист. 2026 р. о 19:33 Ard Biesheuvel <ardb+git@google.com> пише: > > From: Ard Biesheuvel <ardb@kernel.org> > > The NEON code evaluates the Q syndrome using Horner's rule, multiplying by > x = {02} once for each data disk. This takes six instructions per data > disk and vector: four for the multiplication by x, and two XORs to add the > data to P and Q. It also creates a dependency chain through Q that is four > instructions long for each data disk. > > Instead, process four data disks at a time, i.e., > > Q' = x^4.Q + x^3.D[z+3] + x^2.D[z+2] + x.D[z+1] + D[z] > > Multiplying by x^n amounts to shifting each byte left by n bits, and > reducing the n bits that are shifted out modulo the RAID-6 polynomial. > Since this reduction is linear, the bits shifted out of all four terms can > be XORed together first, and reduced using a single polynomial > multiplication (PMUL) by 0x1d, which is exact as long as no more than 4 > bits are shifted out. This reduces the instruction count to 5.25 per data > disk and vector, and the length of the dependency chain through Q to that > of a single step per four data disks. > > In xor_syndrome(), replace the loop that scales Q by x^4 at a time for the > unmodified disks on the 'left side' with a single multiplication by > g^start, using the nibble based table lookups that the recovery code uses > already. This is much faster when only a few high numbered disks are > modified, which is the common case for partial stripe writes. > > Put the code that performs the Horner steps for each lane in a helper > function, so that unroll.awk keeps the instructions for each lane > together. This prevents the 8-way unrolled version from running out of > registers. > > While at it, use the ^ and & operators rather than the veorq_u8() and > vandq_u8() intrinsics, which makes the code easier to read, and allows > the compiler to use EOR3 instructions by itself, as long as the SHA3 > extension is enabled (see the next patch). > > Results on an Apple M2 running Linux, best of the 1x/2x/4x/8x versions, > in MB/s, using a userspace build of the same code (4 KiB per disk): > > gen_syndrome() 2 data 4 data 8 data 16 data 32 data > Avalanche (P) old 55203 42463 37529 35354 34744 > new 54099 46028 41733 40484 39569 > Blizzard (E) old 19790 15393 14050 13333 12887 > new 21670 16197 15524 13915 12475 > > xor_syndrome(), 16 data disks, stop = 15, in GB/s by value of start: > 0 4 8 12 15 > Avalanche (P) old 34 33 30 24 11 > new 39 37 36 33 25 > Blizzard (E) old 12 12 11 9 4 > new 13 14 13 12 9 > > Signed-off-by: Ard Biesheuvel <ardb@kernel.org> > --- > lib/raid/raid6/arm/neon.c | 6 +- > lib/raid/raid6/arm/neon.h | 12 +- > lib/raid/raid6/arm/neon.uc | 180 ++++++++++++++------ > lib/raid/raid6/arm/neon_compat.h | 22 +++ > lib/raid/raid6/arm/recov_neon_inner.c | 19 +-- > 5 files changed, 161 insertions(+), 78 deletions(-) > > diff --git a/lib/raid/raid6/arm/neon.c b/lib/raid/raid6/arm/neon.c > index af90869aaffc..2f8e98f5556e 100644 > --- a/lib/raid/raid6/arm/neon.c > +++ b/lib/raid/raid6/arm/neon.c > @@ -34,10 +34,12 @@ > size_t bytes, void **ptrs) \ > { \ > void raid6_neon ## _n ## _xor_syndrome_real(int, \ > - int, int, unsigned long, void**); \ > + int, int, unsigned long, void**, \ > + const u8 *); \ > scoped_ksimd() \ > raid6_neon ## _n ## _xor_syndrome_real(disks, \ > - start, stop, (unsigned long)bytes, ptrs);\ > + start, stop, (unsigned long)bytes, ptrs,\ > + raid6_vgfmul[raid6_gfexp[start]]); \ > } \ > struct raid6_calls const raid6_neonx ## _n = { \ > .gen_syndrome = raid6_neon ## _n ## _gen_syndrome, \ > diff --git a/lib/raid/raid6/arm/neon.h b/lib/raid/raid6/arm/neon.h > index 2ca41ee9b499..a9977ec57f27 100644 > --- a/lib/raid/raid6/arm/neon.h > +++ b/lib/raid/raid6/arm/neon.h > @@ -2,16 +2,20 @@ > > void raid6_neon1_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); > void raid6_neon1_xor_syndrome_real(int disks, int start, int stop, > - unsigned long bytes, void **ptrs); > + unsigned long bytes, void **ptrs, > + const uint8_t *gmul); > void raid6_neon2_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); > void raid6_neon2_xor_syndrome_real(int disks, int start, int stop, > - unsigned long bytes, void **ptrs); > + unsigned long bytes, void **ptrs, > + const uint8_t *gmul); > void raid6_neon4_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); > void raid6_neon4_xor_syndrome_real(int disks, int start, int stop, > - unsigned long bytes, void **ptrs); > + unsigned long bytes, void **ptrs, > + const uint8_t *gmul); > void raid6_neon8_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); > void raid6_neon8_xor_syndrome_real(int disks, int start, int stop, > - unsigned long bytes, void **ptrs); > + unsigned long bytes, void **ptrs, > + const uint8_t *gmul); > void __raid6_2data_recov_neon(int bytes, uint8_t *p, uint8_t *q, uint8_t *dp, > uint8_t *dq, const uint8_t *pbmul, > const uint8_t *qmul); > diff --git a/lib/raid/raid6/arm/neon.uc b/lib/raid/raid6/arm/neon.uc > index 14a9fc2c60fa..6f74e2b82fc4 100644 > --- a/lib/raid/raid6/arm/neon.uc > +++ b/lib/raid/raid6/arm/neon.uc > @@ -22,15 +22,29 @@ > * $#-way unrolled NEON intrinsics math RAID-6 instruction set > * > * This file is postprocessed using unroll.awk > + * > + * The Q syndrome is evaluated using Horner's rule, but four data disks at a > + * time, i.e., > + * > + * Q' = x^4 * Q + x^3 * D[z+3] + x^2 * D[z+2] + x * D[z+1] + D[z] > + * > + * Multiplying by x^n amounts to shifting each byte left by n bits, and > + * reducing the n bits that are shifted out modulo the RAID-6 polynomial. > + * Since this reduction is linear, the bits shifted out of all terms can be > + * XORed together first, and reduced using a single polynomial multiplication > + * by 0x1d, which is exact as long as no more than 4 bits are shifted out. > */ > > #include <arm_neon.h> > #include "arm/neon.h" > +#include "arm/neon_compat.h" > > typedef uint8x16_t unative_t; > > #define NSIZE sizeof(unative_t) > > +#define RAID6_NEON_FN(n, fn) raid6_neon ## n ## _ ## fn > + > /* > * The SHLBYTE() operation shifts each byte left by 1, *not* > * rolling over into the next byte > @@ -54,13 +68,85 @@ static inline unative_t PMUL(unative_t v, unative_t u) > return (unative_t)vmulq_p8((poly8x16_t)v, (poly8x16_t)u); > } > > -void raid6_neon$#_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs) > +/* Q * x + D */ > +static inline unative_t HORNER1(unative_t q, unative_t d, unative_t x1d) > +{ > + return SHLBYTE(q) ^ (MASK(q) & x1d) ^ d; > +} > + > +/* Q * x^2 + D1 * x + D0 */ > +static inline unative_t HORNER2(unative_t q, unative_t d1, unative_t d0, > + unative_t x1d) > +{ > + unative_t lo = vshlq_n_u8(q, 2) ^ vshlq_n_u8(d1, 1) ^ d0; > + unative_t hi = vshrq_n_u8(q, 6) ^ vshrq_n_u8(d1, 7); > + > + return lo ^ PMUL(hi, x1d); > +} > + > +/* Q * x^4 + D3 * x^3 + D2 * x^2 + D1 * x + D0 */ > +static inline unative_t HORNER4(unative_t q, unative_t d3, unative_t d2, > + unative_t d1, unative_t d0, unative_t x1d) > +{ > + unative_t lo = vshlq_n_u8(q, 4) ^ vshlq_n_u8(d3, 3) ^ > + vshlq_n_u8(d2, 2) ^ vshlq_n_u8(d1, 1) ^ d0; > + unative_t hi = vshrq_n_u8(q, 4) ^ vshrq_n_u8(d3, 5) ^ > + vshrq_n_u8(d2, 6) ^ vshrq_n_u8(d1, 7); > + > + return lo ^ PMUL(hi, x1d); > +} > + > +/* > + * Perform a Horner step for a single lane. Each step is a single statement > + * so that unroll.awk keeps the code for each lane together, which avoids > + * running out of registers with the wider unrolls. > + */ > +static inline void STEP1(unative_t *wp, unative_t *wq, const uint8_t *s, > + unative_t x1d) > +{ > + unative_t d = vld1q_u8(s); > + > + *wp ^= d; > + *wq = HORNER1(*wq, d, x1d); > +} > + > +static inline void STEP2(unative_t *wp, unative_t *wq, const uint8_t *s1, > + const uint8_t *s0, unative_t x1d) > +{ > + unative_t d1 = vld1q_u8(s1); > + unative_t d0 = vld1q_u8(s0); > + > + *wp ^= d1 ^ d0; > + *wq = HORNER2(*wq, d1, d0, x1d); > +} > + > +static inline void STEP4(unative_t *wp, unative_t *wq, const uint8_t *s3, > + const uint8_t *s2, const uint8_t *s1, > + const uint8_t *s0, unative_t x1d) > +{ > + unative_t d3 = vld1q_u8(s3); > + unative_t d2 = vld1q_u8(s2); > + unative_t d1 = vld1q_u8(s1); > + unative_t d0 = vld1q_u8(s0); > + > + *wp ^= d3 ^ d2 ^ d1 ^ d0; > + *wq = HORNER4(*wq, d3, d2, d1, d0, x1d); > +} > + > +/* Multiply by the constant described by the nibble tables in m0 and m1 */ > +static inline unative_t GFMUL(unative_t v, unative_t m0, unative_t m1) > +{ > + return vqtbl1q_u8(m0, v & 0x0f) ^ vqtbl1q_u8(m1, vshrq_n_u8(v, 4)); > +} > + > +void RAID6_NEON_FN($#, gen_syndrome_real)(int disks, unsigned long bytes, > + void **ptrs) > { > uint8_t **dptr = (uint8_t **)ptrs; > uint8_t *p, *q; > int d, z, z0; > > - register unative_t wd$$, wq$$, wp$$, w1$$, w2$$; > + unative_t wq$$, wp$$; > const unative_t x1d = vdupq_n_u8(0x1d); > > z0 = disks - 3; /* Highest data disk */ > @@ -69,30 +155,41 @@ void raid6_neon$#_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs) > > for ( d = 0 ; d < bytes ; d += NSIZE*$# ) { > wq$$ = wp$$ = vld1q_u8(&dptr[z0][d+$$*NSIZE]); > - for ( z = z0-1 ; z >= 0 ; z-- ) { > - wd$$ = vld1q_u8(&dptr[z][d+$$*NSIZE]); > - wp$$ = veorq_u8(wp$$, wd$$); > - w2$$ = MASK(wq$$); > - w1$$ = SHLBYTE(wq$$); > - > - w2$$ = vandq_u8(w2$$, x1d); > - w1$$ = veorq_u8(w1$$, w2$$); > - wq$$ = veorq_u8(w1$$, wd$$); > + z = z0 - 1; > + > + /* Process 1 and/or 2 disks so that a multiple of 4 remains */ > + if (z0 & 1) { > + STEP1(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], x1d); > + z--; > + } > + if (z0 & 2) { > + STEP2(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], &dptr[z-1][d+$$*NSIZE], x1d); > + z -= 2; > + } > + for ( ; z >= 0 ; z -= 4 ) { > + STEP4(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], &dptr[z-1][d+$$*NSIZE], &dptr[z-2][d+$$*NSIZE], &dptr[z-3][d+$$*NSIZE], x1d); > } > vst1q_u8(&p[d+NSIZE*$$], wp$$); > vst1q_u8(&q[d+NSIZE*$$], wq$$); > } > } > > -void raid6_neon$#_xor_syndrome_real(int disks, int start, int stop, > - unsigned long bytes, void **ptrs) > +/* > + * @gmul points to the 32 byte nibble table (see raid6_vgfmul[]) that > + * multiplies by g^start. > + */ > +void RAID6_NEON_FN($#, xor_syndrome_real)(int disks, int start, int stop, > + unsigned long bytes, void **ptrs, > + const uint8_t *gmul) > { > uint8_t **dptr = (uint8_t **)ptrs; > uint8_t *p, *q; > int d, z, z0; > > - register unative_t wd$$, wq$$, wp$$, w1$$, w2$$; > + unative_t wq$$, wp$$; > const unative_t x1d = vdupq_n_u8(0x1d); > + const unative_t gm0 = vld1q_u8(gmul); > + const unative_t gm1 = vld1q_u8(gmul + 16); > > z0 = stop; /* P/Q right side optimization */ > p = dptr[disks-2]; /* XOR parity */ > @@ -100,53 +197,28 @@ void raid6_neon$#_xor_syndrome_real(int disks, int start, int stop, > > for ( d = 0 ; d < bytes ; d += NSIZE*$# ) { > wq$$ = vld1q_u8(&dptr[z0][d+$$*NSIZE]); > - wp$$ = veorq_u8(vld1q_u8(&p[d+$$*NSIZE]), wq$$); > + wp$$ = vld1q_u8(&p[d+$$*NSIZE]) ^ wq$$; > + z = z0 - 1; > > /* P/Q data pages */ > - for ( z = z0-1 ; z >= start ; z-- ) { > - wd$$ = vld1q_u8(&dptr[z][d+$$*NSIZE]); > - wp$$ = veorq_u8(wp$$, wd$$); > - w2$$ = MASK(wq$$); > - w1$$ = SHLBYTE(wq$$); > - > - w2$$ = vandq_u8(w2$$, x1d); > - w1$$ = veorq_u8(w1$$, w2$$); > - wq$$ = veorq_u8(w1$$, wd$$); > + if ((z0 - start) & 1) { > + STEP1(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], x1d); > + z--; > } > - /* P/Q left side optimization */ > - for ( z = start-1 ; z >= 3 ; z -= 4 ) { > - w2$$ = vshrq_n_u8(wq$$, 4); > - w1$$ = vshlq_n_u8(wq$$, 4); > - > - w2$$ = PMUL(w2$$, x1d); > - wq$$ = veorq_u8(w1$$, w2$$); > + if ((z0 - start) & 2) { > + STEP2(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], &dptr[z-1][d+$$*NSIZE], x1d); > + z -= 2; > + } > + for ( ; z >= start ; z -= 4 ) { > + STEP4(&wp$$, &wq$$, &dptr[z][d+$$*NSIZE], &dptr[z-1][d+$$*NSIZE], &dptr[z-2][d+$$*NSIZE], &dptr[z-3][d+$$*NSIZE], x1d); > } > > - switch (z) { > - case 2: > - w2$$ = vshrq_n_u8(wq$$, 5); > - w1$$ = vshlq_n_u8(wq$$, 3); > - > - w2$$ = PMUL(w2$$, x1d); > - wq$$ = veorq_u8(w1$$, w2$$); > - break; > - case 1: > - w2$$ = vshrq_n_u8(wq$$, 6); > - w1$$ = vshlq_n_u8(wq$$, 2); > - > - w2$$ = PMUL(w2$$, x1d); > - wq$$ = veorq_u8(w1$$, w2$$); > - break; > - case 0: > - w2$$ = MASK(wq$$); > - w1$$ = SHLBYTE(wq$$); > - > - w2$$ = vandq_u8(w2$$, x1d); > - wq$$ = veorq_u8(w1$$, w2$$); > + /* P/Q left side optimization: multiply by g^start */ > + if (start) { > + wq$$ = GFMUL(wq$$, gm0, gm1); > } > - w1$$ = vld1q_u8(&q[d+NSIZE*$$]); > - wq$$ = veorq_u8(wq$$, w1$$); > > + wq$$ ^= vld1q_u8(&q[d+NSIZE*$$]); > vst1q_u8(&p[d+NSIZE*$$], wp$$); > vst1q_u8(&q[d+NSIZE*$$], wq$$); > } > diff --git a/lib/raid/raid6/arm/neon_compat.h b/lib/raid/raid6/arm/neon_compat.h > new file mode 100644 > index 000000000000..81410b2c2c99 > --- /dev/null > +++ b/lib/raid/raid6/arm/neon_compat.h > @@ -0,0 +1,22 @@ > +/* SPDX-License-Identifier: GPL-2.0-only */ > +/* > + * Copyright (C) 2017 Linaro Ltd. <ard.biesheuvel@linaro.org> > + */ > + > +#ifdef CONFIG_ARM > +/* > + * AArch32 does not provide this intrinsic natively because it does not > + * implement the underlying instruction. AArch32 only provides a 64-bit > + * wide vtbl.8 instruction, so use that instead. > + */ > +static inline uint8x16_t vqtbl1q_u8(uint8x16_t a, uint8x16_t b) > +{ > + union { > + uint8x16_t val; > + uint8x8x2_t pair; > + } __a = { a }; > + > + return vcombine_u8(vtbl2_u8(__a.pair, vget_low_u8(b)), > + vtbl2_u8(__a.pair, vget_high_u8(b))); > +} > +#endif > diff --git a/lib/raid/raid6/arm/recov_neon_inner.c b/lib/raid/raid6/arm/recov_neon_inner.c > index 53c355efa7ff..66f687e5dea8 100644 > --- a/lib/raid/raid6/arm/recov_neon_inner.c > +++ b/lib/raid/raid6/arm/recov_neon_inner.c > @@ -6,24 +6,7 @@ > > #include <arm_neon.h> > #include "arm/neon.h" > - > -#ifdef CONFIG_ARM > -/* > - * AArch32 does not provide this intrinsic natively because it does not > - * implement the underlying instruction. AArch32 only provides a 64-bit > - * wide vtbl.8 instruction, so use that instead. > - */ > -static uint8x16_t vqtbl1q_u8(uint8x16_t a, uint8x16_t b) > -{ > - union { > - uint8x16_t val; > - uint8x8x2_t pair; > - } __a = { a }; > - > - return vcombine_u8(vtbl2_u8(__a.pair, vget_low_u8(b)), > - vtbl2_u8(__a.pair, vget_high_u8(b))); > -} > -#endif > +#include "arm/neon_compat.h" > > void __raid6_2data_recov_neon(int bytes, uint8_t *p, uint8_t *q, uint8_t *dp, > uint8_t *dq, const uint8_t *pbmul, > -- > 2.56.1.708.g77570a6a1d-goog > ^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH 2/2] raid6: arm64: use the EOR3 instruction where available 2026-11-18 18:33 [PATCH 0/2] raid6/arm64: Use aggregation and 3-way XOR Ard Biesheuvel 2026-11-18 18:33 ` [PATCH 1/2] raid6: neon: process four data disks per Horner step Ard Biesheuvel @ 2026-11-18 18:33 ` Ard Biesheuvel 2026-11-19 22:35 ` Eric Biggers 2026-11-19 23:55 ` Demian Shulhan 2026-11-19 22:50 ` [PATCH 0/2] raid6/arm64: Use aggregation and 3-way XOR Eric Biggers 2026-11-19 23:58 ` Demian Shulhan 3 siblings, 2 replies; 9+ messages in thread From: Ard Biesheuvel @ 2026-11-18 18:33 UTC (permalink / raw) To: linux-arm-kernel Cc: Ard Biesheuvel, Christoph Hellwig, Eric Biggers, Demian Shulhan From: Ard Biesheuvel <ardb@kernel.org> The SHA3 extension to the arm64 instruction set provides EOR3, which XORs three vectors in a single instruction. Use it to fold the XORs in the Horner steps of the NEON syndrome code, which reduces the instruction count from 5.25 to 4 per data disk and vector for the 4-disk step, i.e., a reduction of a third compared to the original code. Build the EOR3 versions from the same neon.uc source, with RAID6_NEON_EOR3 defined and the SHA3 extension enabled, and use them instead of the plain NEON versions on CPUs that implement SHA3. Use -march=armv8.2-a+sha3 for this: SHA3 is an optional extension of ARMv8.2, so any CPU that implements it implements ARMv8.2 as well. This is also what older toolchains (e.g., GCC 8.5 with binutils 2.36) need to accept EOR3. With SHA3 enabled, GCC and Clang fold a ^ b ^ c into EOR3 by themselves. The exception is the loop carried P accumulator: recent GCC versions reassociate the XORs into it in a way that only permits a single EOR3 instruction where two can be used, so use the veor3q_u8() intrinsic explicitly for that case. Results on an Apple M2 running Linux, best of the 1x/2x/4x/8x versions, in MB/s, using a userspace build of the same code (4 KiB per disk): gen_syndrome() 2 data 4 data 8 data 16 data 32 data Avalanche (P) neon 54099 46028 41733 40484 39569 eor3 63505 56936 53229 51807 51380 Blizzard (E) neon 21670 16197 15524 13915 12475 eor3 23388 19800 19583 17278 15609 xor_syndrome(), 16 data disks, stop = 15, in GB/s by value of start: 0 4 8 12 15 Avalanche (P) neon 39 37 36 33 25 eor3 50 46 44 38 25 Blizzard (E) neon 13 14 13 12 9 eor3 16 17 16 14 9 Compared to the code before the previous patch, this is 15-48% faster for gen_syndrome() on the performance cores, and 18-39% on the efficiency cores. Signed-off-by: Ard Biesheuvel <ardb@kernel.org> --- lib/raid/raid6/.gitignore | 1 + lib/raid/raid6/Makefile | 19 +++++++++ lib/raid/raid6/arm/neon.c | 42 +++++++++++--------- lib/raid/raid6/arm/neon.h | 16 ++++++++ lib/raid/raid6/arm/neon.uc | 20 +++++++++- lib/raid/raid6/arm64/pq_arch.h | 34 +++++++++++++++- 6 files changed, 110 insertions(+), 22 deletions(-) diff --git a/lib/raid/raid6/.gitignore b/lib/raid/raid6/.gitignore index 6be57745afd1..2e9bd6299837 100644 --- a/lib/raid/raid6/.gitignore +++ b/lib/raid/raid6/.gitignore @@ -4,5 +4,6 @@ altivec*.c int*.c tables.c neon?.c +eor3_?.c s390vx?.c vpermxor*.c diff --git a/lib/raid/raid6/Makefile b/lib/raid/raid6/Makefile index 5cdb7223de2a..314ed1012cc0 100644 --- a/lib/raid/raid6/Makefile +++ b/lib/raid/raid6/Makefile @@ -24,6 +24,10 @@ raid6_pq-$(CONFIG_KERNEL_MODE_NEON) += arm/neon.o \ arm/neon8.o \ arm/recov_neon.o \ arm/recov_neon_inner.o +raid6_pq-$(CONFIG_ARM64) += arm64/eor3_1.o \ + arm64/eor3_2.o \ + arm64/eor3_4.o \ + arm64/eor3_8.o raid6_pq-$(CONFIG_LOONGARCH) += loongarch/loongarch_simd.o \ loongarch/recov_loongarch_simd.o raid6_pq-$(CONFIG_ALTIVEC) += powerpc/altivec1.o \ @@ -56,11 +60,21 @@ CFLAGS_arm/neon2.o += $(CC_FLAGS_FPU) CFLAGS_arm/neon4.o += $(CC_FLAGS_FPU) CFLAGS_arm/neon8.o += $(CC_FLAGS_FPU) CFLAGS_arm/recov_neon_inner.o += $(CC_FLAGS_FPU) +# SHA3 is an optional extension of ARMv8.2, so any CPU that implements it +# implements ARMv8.2 as well +CFLAGS_arm64/eor3_1.o += $(CC_FLAGS_FPU) -DRAID6_NEON_EOR3 -march=armv8.2-a+sha3 +CFLAGS_arm64/eor3_2.o += $(CC_FLAGS_FPU) -DRAID6_NEON_EOR3 -march=armv8.2-a+sha3 +CFLAGS_arm64/eor3_4.o += $(CC_FLAGS_FPU) -DRAID6_NEON_EOR3 -march=armv8.2-a+sha3 +CFLAGS_arm64/eor3_8.o += $(CC_FLAGS_FPU) -DRAID6_NEON_EOR3 -march=armv8.2-a+sha3 CFLAGS_REMOVE_arm/neon1.o += $(CC_FLAGS_NO_FPU) CFLAGS_REMOVE_arm/neon2.o += $(CC_FLAGS_NO_FPU) CFLAGS_REMOVE_arm/neon4.o += $(CC_FLAGS_NO_FPU) CFLAGS_REMOVE_arm/neon8.o += $(CC_FLAGS_NO_FPU) CFLAGS_REMOVE_arm/recov_neon_inner.o += $(CC_FLAGS_NO_FPU) +CFLAGS_REMOVE_arm64/eor3_1.o += $(CC_FLAGS_NO_FPU) +CFLAGS_REMOVE_arm64/eor3_2.o += $(CC_FLAGS_NO_FPU) +CFLAGS_REMOVE_arm64/eor3_4.o += $(CC_FLAGS_NO_FPU) +CFLAGS_REMOVE_arm64/eor3_8.o += $(CC_FLAGS_NO_FPU) ifeq ($(CONFIG_ALTIVEC),y) altivec_flags := -maltivec $(call cc-option,-mabi=altivec) @@ -109,6 +123,11 @@ targets += arm/neon1.c arm/neon2.c arm/neon4.c arm/neon8.c $(obj)/arm/neon%.c: $(src)/arm/neon.uc $(src)/unroll.awk FORCE $(call if_changed,unroll) +# the arm64 EOR3 versions are built from the same source +targets += arm64/eor3_1.c arm64/eor3_2.c arm64/eor3_4.c arm64/eor3_8.c +$(obj)/arm64/eor3_%.c: $(src)/arm/neon.uc $(src)/unroll.awk FORCE + $(call if_changed,unroll) + targets += powerpc/altivec1.c \ powerpc/altivec2.c \ powerpc/altivec4.c \ diff --git a/lib/raid/raid6/arm/neon.c b/lib/raid/raid6/arm/neon.c index 2f8e98f5556e..722f15f2a6ad 100644 --- a/lib/raid/raid6/arm/neon.c +++ b/lib/raid/raid6/arm/neon.c @@ -7,6 +7,7 @@ #include <asm/simd.h> #include "algos.h" +#include "arm/neon.h" /* * There are 2 reasons these wrappers are kept in a separate compilation unit @@ -19,35 +20,38 @@ * outside a kernel_neon_begin()/kernel_neon_end() pair. */ -#define RAID6_NEON_WRAPPER(_n) \ - static void raid6_neon ## _n ## _gen_syndrome(int disks, \ +#define RAID6_NEON_WRAPPER(_pfx, _n, _name) \ + static void _pfx ## _n ## _gen_syndrome(int disks, \ size_t bytes, void **ptrs) \ { \ - void raid6_neon ## _n ## _gen_syndrome_real(int, \ - unsigned long, void**); \ scoped_ksimd() \ - raid6_neon ## _n ## _gen_syndrome_real(disks, \ + _pfx ## _n ## _gen_syndrome_real(disks, \ (unsigned long)bytes, ptrs); \ } \ - static void raid6_neon ## _n ## _xor_syndrome(int disks, \ - int start, int stop, \ + static void _pfx ## _n ## _xor_syndrome(int disks, \ + int start, int stop, \ size_t bytes, void **ptrs) \ { \ - void raid6_neon ## _n ## _xor_syndrome_real(int, \ - int, int, unsigned long, void**, \ - const u8 *); \ scoped_ksimd() \ - raid6_neon ## _n ## _xor_syndrome_real(disks, \ + _pfx ## _n ## _xor_syndrome_real(disks, \ start, stop, (unsigned long)bytes, ptrs,\ raid6_vgfmul[raid6_gfexp[start]]); \ } \ - struct raid6_calls const raid6_neonx ## _n = { \ - .gen_syndrome = raid6_neon ## _n ## _gen_syndrome, \ - .xor_syndrome = raid6_neon ## _n ## _xor_syndrome, \ - .name = "neonx" #_n, \ + struct raid6_calls const raid6_ ## _name = { \ + .gen_syndrome = _pfx ## _n ## _gen_syndrome, \ + .xor_syndrome = _pfx ## _n ## _xor_syndrome, \ + .name = #_name, \ } -RAID6_NEON_WRAPPER(1); -RAID6_NEON_WRAPPER(2); -RAID6_NEON_WRAPPER(4); -RAID6_NEON_WRAPPER(8); +RAID6_NEON_WRAPPER(raid6_neon, 1, neonx1); +RAID6_NEON_WRAPPER(raid6_neon, 2, neonx2); +RAID6_NEON_WRAPPER(raid6_neon, 4, neonx4); +RAID6_NEON_WRAPPER(raid6_neon, 8, neonx8); + +#ifdef CONFIG_ARM64 +/* versions using the EOR3 instruction from the SHA3 extension */ +RAID6_NEON_WRAPPER(raid6_eor3_, 1, eor3x1); +RAID6_NEON_WRAPPER(raid6_eor3_, 2, eor3x2); +RAID6_NEON_WRAPPER(raid6_eor3_, 4, eor3x4); +RAID6_NEON_WRAPPER(raid6_eor3_, 8, eor3x8); +#endif diff --git a/lib/raid/raid6/arm/neon.h b/lib/raid/raid6/arm/neon.h index a9977ec57f27..a8fc4ba06944 100644 --- a/lib/raid/raid6/arm/neon.h +++ b/lib/raid/raid6/arm/neon.h @@ -16,6 +16,22 @@ void raid6_neon8_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); void raid6_neon8_xor_syndrome_real(int disks, int start, int stop, unsigned long bytes, void **ptrs, const uint8_t *gmul); +void raid6_eor3_1_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); +void raid6_eor3_1_xor_syndrome_real(int disks, int start, int stop, + unsigned long bytes, void **ptrs, + const uint8_t *gmul); +void raid6_eor3_2_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); +void raid6_eor3_2_xor_syndrome_real(int disks, int start, int stop, + unsigned long bytes, void **ptrs, + const uint8_t *gmul); +void raid6_eor3_4_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); +void raid6_eor3_4_xor_syndrome_real(int disks, int start, int stop, + unsigned long bytes, void **ptrs, + const uint8_t *gmul); +void raid6_eor3_8_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); +void raid6_eor3_8_xor_syndrome_real(int disks, int start, int stop, + unsigned long bytes, void **ptrs, + const uint8_t *gmul); void __raid6_2data_recov_neon(int bytes, uint8_t *p, uint8_t *q, uint8_t *dp, uint8_t *dq, const uint8_t *pbmul, const uint8_t *qmul); diff --git a/lib/raid/raid6/arm/neon.uc b/lib/raid/raid6/arm/neon.uc index 6f74e2b82fc4..e1084a8d3a18 100644 --- a/lib/raid/raid6/arm/neon.uc +++ b/lib/raid/raid6/arm/neon.uc @@ -33,6 +33,10 @@ * Since this reduction is linear, the bits shifted out of all terms can be * XORed together first, and reduced using a single polynomial multiplication * by 0x1d, which is exact as long as no more than 4 bits are shifted out. + * + * When built with RAID6_NEON_EOR3 defined, this file is compiled with the + * arm64 SHA3 extension enabled, allowing the compiler to fold the additions + * into three-way XOR (EOR3) instructions. */ #include <arm_neon.h> @@ -43,7 +47,19 @@ typedef uint8x16_t unative_t; #define NSIZE sizeof(unative_t) +#ifdef RAID6_NEON_EOR3 +#define RAID6_NEON_FN(n, fn) raid6_eor3_ ## n ## _ ## fn + +/* + * The compiler emits EOR3 for a ^ b ^ c by itself, but recent GCC versions + * reassociate the XORs into the loop carried P accumulator such that EOR3 + * can no longer be used, so spell it out for that case. + */ +#define XOR3(a, b, c) veor3q_u8(a, b, c) +#else #define RAID6_NEON_FN(n, fn) raid6_neon ## n ## _ ## fn +#define XOR3(a, b, c) ((a) ^ (b) ^ (c)) +#endif /* * The SHLBYTE() operation shifts each byte left by 1, *not* @@ -116,7 +132,7 @@ static inline void STEP2(unative_t *wp, unative_t *wq, const uint8_t *s1, unative_t d1 = vld1q_u8(s1); unative_t d0 = vld1q_u8(s0); - *wp ^= d1 ^ d0; + *wp = XOR3(*wp, d1, d0); *wq = HORNER2(*wq, d1, d0, x1d); } @@ -129,7 +145,7 @@ static inline void STEP4(unative_t *wp, unative_t *wq, const uint8_t *s3, unative_t d1 = vld1q_u8(s1); unative_t d0 = vld1q_u8(s0); - *wp ^= d3 ^ d2 ^ d1 ^ d0; + *wp = XOR3(XOR3(*wp, d3, d2), d1, d0); *wq = HORNER4(*wq, d3, d2, d1, d0, x1d); } diff --git a/lib/raid/raid6/arm64/pq_arch.h b/lib/raid/raid6/arm64/pq_arch.h index 27ff564d7594..4e397aa36480 100644 --- a/lib/raid/raid6/arm64/pq_arch.h +++ b/lib/raid/raid6/arm64/pq_arch.h @@ -1 +1,33 @@ -#include "arm/pq_arch.h" +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +#include <linux/cpufeature.h> +#include <asm/neon.h> + +extern const struct raid6_calls raid6_neonx1; +extern const struct raid6_calls raid6_neonx2; +extern const struct raid6_calls raid6_neonx4; +extern const struct raid6_calls raid6_neonx8; +extern const struct raid6_calls raid6_eor3x1; +extern const struct raid6_calls raid6_eor3x2; +extern const struct raid6_calls raid6_eor3x4; +extern const struct raid6_calls raid6_eor3x8; +extern const struct raid6_recov_calls raid6_recov_neon; + +static __always_inline void __init arch_raid6_init(void) +{ + raid6_algo_add_default(); + if (cpu_has_neon()) { + if (cpu_have_named_feature(SHA3)) { + raid6_algo_add(&raid6_eor3x1); + raid6_algo_add(&raid6_eor3x2); + raid6_algo_add(&raid6_eor3x4); + raid6_algo_add(&raid6_eor3x8); + } else { + raid6_algo_add(&raid6_neonx1); + raid6_algo_add(&raid6_neonx2); + raid6_algo_add(&raid6_neonx4); + raid6_algo_add(&raid6_neonx8); + } + raid6_recov_algo_add(&raid6_recov_neon); + } +} -- 2.56.1.708.g77570a6a1d-goog ^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] raid6: arm64: use the EOR3 instruction where available 2026-11-18 18:33 ` [PATCH 2/2] raid6: arm64: use the EOR3 instruction where available Ard Biesheuvel @ 2026-11-19 22:35 ` Eric Biggers 2026-11-19 23:55 ` Demian Shulhan 1 sibling, 0 replies; 9+ messages in thread From: Eric Biggers @ 2026-11-19 22:35 UTC (permalink / raw) To: Ard Biesheuvel Cc: linux-arm-kernel, Ard Biesheuvel, Christoph Hellwig, Demian Shulhan On Wed, Nov 18, 2026 at 07:33:38PM +0100, Ard Biesheuvel wrote: > Build the EOR3 versions from the same neon.uc source, with > RAID6_NEON_EOR3 defined and the SHA3 extension enabled, and use them > instead of the plain NEON versions on CPUs that implement SHA3. Use > -march=armv8.2-a+sha3 for this: SHA3 is an optional extension of > ARMv8.2, so any CPU that implements it implements ARMv8.2 as well. This > is also what older toolchains (e.g., GCC 8.5 with binutils 2.36) need to > accept EOR3. gcc 8.1 supports the sha3 feature itself, but the veor3q_u8() intrinsic wasn't added until gcc 9.1 and gcc 8.5. Since the minimum gcc version to build the kernel is 8.1, we probably need a gcc version check. > - struct raid6_calls const raid6_neonx ## _n = { \ > - .gen_syndrome = raid6_neon ## _n ## _gen_syndrome, \ > - .xor_syndrome = raid6_neon ## _n ## _xor_syndrome, \ > - .name = "neonx" #_n, \ > + struct raid6_calls const raid6_ ## _name = { \ > + .gen_syndrome = _pfx ## _n ## _gen_syndrome, \ > + .xor_syndrome = _pfx ## _n ## _xor_syndrome, \ > + .name = #_name, \ > } Nit: 'const' should go before 'struct' > @@ -129,7 +145,7 @@ static inline void STEP4(unative_t *wp, unative_t *wq, const uint8_t *s3, > unative_t d1 = vld1q_u8(s1); > unative_t d0 = vld1q_u8(s0); > > - *wp ^= d3 ^ d2 ^ d1 ^ d0; > + *wp = XOR3(XOR3(*wp, d3, d2), d1, d0); > *wq = HORNER4(*wq, d3, d2, d1, d0, x1d); > } The following might be better, since it cuts down the length of the loop-carried dependency chain on *wp. *wp = XOR3(*wp, XOR3(d3, d2, d1), d0); > diff --git a/lib/raid/raid6/arm64/pq_arch.h b/lib/raid/raid6/arm64/pq_arch.h > index 27ff564d7594..4e397aa36480 100644 > --- a/lib/raid/raid6/arm64/pq_arch.h > +++ b/lib/raid/raid6/arm64/pq_arch.h > @@ -1 +1,33 @@ > -#include "arm/pq_arch.h" > +/* SPDX-License-Identifier: GPL-2.0-or-later */ > + > +#include <linux/cpufeature.h> > +#include <asm/neon.h> > + > +extern const struct raid6_calls raid6_neonx1; > +extern const struct raid6_calls raid6_neonx2; > +extern const struct raid6_calls raid6_neonx4; > +extern const struct raid6_calls raid6_neonx8; > +extern const struct raid6_calls raid6_eor3x1; > +extern const struct raid6_calls raid6_eor3x2; > +extern const struct raid6_calls raid6_eor3x4; > +extern const struct raid6_calls raid6_eor3x8; > +extern const struct raid6_recov_calls raid6_recov_neon; These declarations are duplicated in 2 places, but they're still not in-scope in the .c files that actually define them. Maybe they should go in lib/raid/raid6/arm/neon.h, and lib/raid/raid6/arm/pq_arch.h and lib/raid/raid6/arm64/pq_arch.h should include that? - Eric ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 2/2] raid6: arm64: use the EOR3 instruction where available 2026-11-18 18:33 ` [PATCH 2/2] raid6: arm64: use the EOR3 instruction where available Ard Biesheuvel 2026-11-19 22:35 ` Eric Biggers @ 2026-11-19 23:55 ` Demian Shulhan 1 sibling, 0 replies; 9+ messages in thread From: Demian Shulhan @ 2026-11-19 23:55 UTC (permalink / raw) To: Ard Biesheuvel Cc: linux-arm-kernel, Ard Biesheuvel, Christoph Hellwig, Eric Biggers Hi Ard, A few EOR3-specific suggestions on top of patch 2: > @@ -129,7 +145,7 @@ static inline void STEP4(unative_t *wp, unative_t *wq, const uint8_t *s3, > unative_t d1 = vld1q_u8(s1); > unative_t d0 = vld1q_u8(s0); > > - *wp ^= d3 ^ d2 ^ d1 ^ d0; > + *wp = XOR3(XOR3(*wp, d3, d2), d1, d0); > *wq = HORNER4(*wq, d3, d2, d1, d0, x1d); > } Writing this as: *wp = XOR3(*wp, XOR3(d3, d2, d1), d0); keeps *wp out of the inner EOR3 so d3 ^ d2 ^ d1 can issue in parallel with the previous iteration's P update, cutting the loop-carried latency on *wp in half. Similarly, in HORNER4(), keeping 'q >> 4' at the outer XOR of 'hi' and folding 'q << 4' and PMUL(hi, x1d) together in a single final XOR3(): unative_t hi = vshrq_n_u8(q, 4) ^ XOR3(vshrq_n_u8(d3, 5), vshrq_n_u8(d2, 6), vshrq_n_u8(d1, 7)); unative_t lo_d = XOR3(vshlq_n_u8(d3, 3), vshlq_n_u8(d2, 2), vshlq_n_u8(d1, 1)) ^ d0; return XOR3(vshlq_n_u8(q, 4), PMUL(hi, x1d), lo_d); shortens the critical loop-carried Q chain (+7-8% for 16-32 data disks with GCC). Finally, in xor_syndrome_real(), neither GCC nor Clang folds GFMUL() with the final q[d] XOR across the 'if (start)' merge point. Passing q[d] into a GFMUL_XOR() helper using XOR3(): static inline unative_t GFMUL_XOR(unative_t v, unative_t m0, unative_t m1, unative_t q) { return XOR3(vqtbl1q_u8(m0, v & 0x0f), vqtbl1q_u8(m1, vshrq_n_u8(v, 4)), q); } and calling on a single line (for unroll.awk): if (start) { wq$$ = GFMUL_XOR(wq$$, gm0, gm1, vld1q_u8(&q[d+NSIZE*$$])); } else { wq$$ ^= vld1q_u8(&q[d+NSIZE*$$]); } saves one vector XOR per lane when start > 0 (+3-4% on partial stripe updates with GCC). BR, Demian ср, 18 лист. 2026 р. о 19:33 Ard Biesheuvel <ardb+git@google.com> пише: > > From: Ard Biesheuvel <ardb@kernel.org> > > The SHA3 extension to the arm64 instruction set provides EOR3, which XORs > three vectors in a single instruction. Use it to fold the XORs in the > Horner steps of the NEON syndrome code, which reduces the instruction > count from 5.25 to 4 per data disk and vector for the 4-disk step, i.e., > a reduction of a third compared to the original code. > > Build the EOR3 versions from the same neon.uc source, with > RAID6_NEON_EOR3 defined and the SHA3 extension enabled, and use them > instead of the plain NEON versions on CPUs that implement SHA3. Use > -march=armv8.2-a+sha3 for this: SHA3 is an optional extension of > ARMv8.2, so any CPU that implements it implements ARMv8.2 as well. This > is also what older toolchains (e.g., GCC 8.5 with binutils 2.36) need to > accept EOR3. > > With SHA3 enabled, GCC and Clang fold a ^ b ^ c into EOR3 by themselves. > The exception is the loop carried P accumulator: recent GCC versions > reassociate the XORs into it in a way that only permits a single EOR3 > instruction where two can be used, so use the veor3q_u8() intrinsic > explicitly for that case. > > Results on an Apple M2 running Linux, best of the 1x/2x/4x/8x versions, > in MB/s, using a userspace build of the same code (4 KiB per disk): > > gen_syndrome() 2 data 4 data 8 data 16 data 32 data > Avalanche (P) neon 54099 46028 41733 40484 39569 > eor3 63505 56936 53229 51807 51380 > Blizzard (E) neon 21670 16197 15524 13915 12475 > eor3 23388 19800 19583 17278 15609 > > xor_syndrome(), 16 data disks, stop = 15, in GB/s by value of start: > 0 4 8 12 15 > Avalanche (P) neon 39 37 36 33 25 > eor3 50 46 44 38 25 > Blizzard (E) neon 13 14 13 12 9 > eor3 16 17 16 14 9 > > Compared to the code before the previous patch, this is 15-48% faster for > gen_syndrome() on the performance cores, and 18-39% on the efficiency > cores. > > Signed-off-by: Ard Biesheuvel <ardb@kernel.org> > --- > lib/raid/raid6/.gitignore | 1 + > lib/raid/raid6/Makefile | 19 +++++++++ > lib/raid/raid6/arm/neon.c | 42 +++++++++++--------- > lib/raid/raid6/arm/neon.h | 16 ++++++++ > lib/raid/raid6/arm/neon.uc | 20 +++++++++- > lib/raid/raid6/arm64/pq_arch.h | 34 +++++++++++++++- > 6 files changed, 110 insertions(+), 22 deletions(-) > > diff --git a/lib/raid/raid6/.gitignore b/lib/raid/raid6/.gitignore > index 6be57745afd1..2e9bd6299837 100644 > --- a/lib/raid/raid6/.gitignore > +++ b/lib/raid/raid6/.gitignore > @@ -4,5 +4,6 @@ altivec*.c > int*.c > tables.c > neon?.c > +eor3_?.c > s390vx?.c > vpermxor*.c > diff --git a/lib/raid/raid6/Makefile b/lib/raid/raid6/Makefile > index 5cdb7223de2a..314ed1012cc0 100644 > --- a/lib/raid/raid6/Makefile > +++ b/lib/raid/raid6/Makefile > @@ -24,6 +24,10 @@ raid6_pq-$(CONFIG_KERNEL_MODE_NEON) += arm/neon.o \ > arm/neon8.o \ > arm/recov_neon.o \ > arm/recov_neon_inner.o > +raid6_pq-$(CONFIG_ARM64) += arm64/eor3_1.o \ > + arm64/eor3_2.o \ > + arm64/eor3_4.o \ > + arm64/eor3_8.o > raid6_pq-$(CONFIG_LOONGARCH) += loongarch/loongarch_simd.o \ > loongarch/recov_loongarch_simd.o > raid6_pq-$(CONFIG_ALTIVEC) += powerpc/altivec1.o \ > @@ -56,11 +60,21 @@ CFLAGS_arm/neon2.o += $(CC_FLAGS_FPU) > CFLAGS_arm/neon4.o += $(CC_FLAGS_FPU) > CFLAGS_arm/neon8.o += $(CC_FLAGS_FPU) > CFLAGS_arm/recov_neon_inner.o += $(CC_FLAGS_FPU) > +# SHA3 is an optional extension of ARMv8.2, so any CPU that implements it > +# implements ARMv8.2 as well > +CFLAGS_arm64/eor3_1.o += $(CC_FLAGS_FPU) -DRAID6_NEON_EOR3 -march=armv8.2-a+sha3 > +CFLAGS_arm64/eor3_2.o += $(CC_FLAGS_FPU) -DRAID6_NEON_EOR3 -march=armv8.2-a+sha3 > +CFLAGS_arm64/eor3_4.o += $(CC_FLAGS_FPU) -DRAID6_NEON_EOR3 -march=armv8.2-a+sha3 > +CFLAGS_arm64/eor3_8.o += $(CC_FLAGS_FPU) -DRAID6_NEON_EOR3 -march=armv8.2-a+sha3 > CFLAGS_REMOVE_arm/neon1.o += $(CC_FLAGS_NO_FPU) > CFLAGS_REMOVE_arm/neon2.o += $(CC_FLAGS_NO_FPU) > CFLAGS_REMOVE_arm/neon4.o += $(CC_FLAGS_NO_FPU) > CFLAGS_REMOVE_arm/neon8.o += $(CC_FLAGS_NO_FPU) > CFLAGS_REMOVE_arm/recov_neon_inner.o += $(CC_FLAGS_NO_FPU) > +CFLAGS_REMOVE_arm64/eor3_1.o += $(CC_FLAGS_NO_FPU) > +CFLAGS_REMOVE_arm64/eor3_2.o += $(CC_FLAGS_NO_FPU) > +CFLAGS_REMOVE_arm64/eor3_4.o += $(CC_FLAGS_NO_FPU) > +CFLAGS_REMOVE_arm64/eor3_8.o += $(CC_FLAGS_NO_FPU) > > ifeq ($(CONFIG_ALTIVEC),y) > altivec_flags := -maltivec $(call cc-option,-mabi=altivec) > @@ -109,6 +123,11 @@ targets += arm/neon1.c arm/neon2.c arm/neon4.c arm/neon8.c > $(obj)/arm/neon%.c: $(src)/arm/neon.uc $(src)/unroll.awk FORCE > $(call if_changed,unroll) > > +# the arm64 EOR3 versions are built from the same source > +targets += arm64/eor3_1.c arm64/eor3_2.c arm64/eor3_4.c arm64/eor3_8.c > +$(obj)/arm64/eor3_%.c: $(src)/arm/neon.uc $(src)/unroll.awk FORCE > + $(call if_changed,unroll) > + > targets += powerpc/altivec1.c \ > powerpc/altivec2.c \ > powerpc/altivec4.c \ > diff --git a/lib/raid/raid6/arm/neon.c b/lib/raid/raid6/arm/neon.c > index 2f8e98f5556e..722f15f2a6ad 100644 > --- a/lib/raid/raid6/arm/neon.c > +++ b/lib/raid/raid6/arm/neon.c > @@ -7,6 +7,7 @@ > > #include <asm/simd.h> > #include "algos.h" > +#include "arm/neon.h" > > /* > * There are 2 reasons these wrappers are kept in a separate compilation unit > @@ -19,35 +20,38 @@ > * outside a kernel_neon_begin()/kernel_neon_end() pair. > */ > > -#define RAID6_NEON_WRAPPER(_n) \ > - static void raid6_neon ## _n ## _gen_syndrome(int disks, \ > +#define RAID6_NEON_WRAPPER(_pfx, _n, _name) \ > + static void _pfx ## _n ## _gen_syndrome(int disks, \ > size_t bytes, void **ptrs) \ > { \ > - void raid6_neon ## _n ## _gen_syndrome_real(int, \ > - unsigned long, void**); \ > scoped_ksimd() \ > - raid6_neon ## _n ## _gen_syndrome_real(disks, \ > + _pfx ## _n ## _gen_syndrome_real(disks, \ > (unsigned long)bytes, ptrs); \ > } \ > - static void raid6_neon ## _n ## _xor_syndrome(int disks, \ > - int start, int stop, \ > + static void _pfx ## _n ## _xor_syndrome(int disks, \ > + int start, int stop, \ > size_t bytes, void **ptrs) \ > { \ > - void raid6_neon ## _n ## _xor_syndrome_real(int, \ > - int, int, unsigned long, void**, \ > - const u8 *); \ > scoped_ksimd() \ > - raid6_neon ## _n ## _xor_syndrome_real(disks, \ > + _pfx ## _n ## _xor_syndrome_real(disks, \ > start, stop, (unsigned long)bytes, ptrs,\ > raid6_vgfmul[raid6_gfexp[start]]); \ > } \ > - struct raid6_calls const raid6_neonx ## _n = { \ > - .gen_syndrome = raid6_neon ## _n ## _gen_syndrome, \ > - .xor_syndrome = raid6_neon ## _n ## _xor_syndrome, \ > - .name = "neonx" #_n, \ > + struct raid6_calls const raid6_ ## _name = { \ > + .gen_syndrome = _pfx ## _n ## _gen_syndrome, \ > + .xor_syndrome = _pfx ## _n ## _xor_syndrome, \ > + .name = #_name, \ > } > > -RAID6_NEON_WRAPPER(1); > -RAID6_NEON_WRAPPER(2); > -RAID6_NEON_WRAPPER(4); > -RAID6_NEON_WRAPPER(8); > +RAID6_NEON_WRAPPER(raid6_neon, 1, neonx1); > +RAID6_NEON_WRAPPER(raid6_neon, 2, neonx2); > +RAID6_NEON_WRAPPER(raid6_neon, 4, neonx4); > +RAID6_NEON_WRAPPER(raid6_neon, 8, neonx8); > + > +#ifdef CONFIG_ARM64 > +/* versions using the EOR3 instruction from the SHA3 extension */ > +RAID6_NEON_WRAPPER(raid6_eor3_, 1, eor3x1); > +RAID6_NEON_WRAPPER(raid6_eor3_, 2, eor3x2); > +RAID6_NEON_WRAPPER(raid6_eor3_, 4, eor3x4); > +RAID6_NEON_WRAPPER(raid6_eor3_, 8, eor3x8); > +#endif > diff --git a/lib/raid/raid6/arm/neon.h b/lib/raid/raid6/arm/neon.h > index a9977ec57f27..a8fc4ba06944 100644 > --- a/lib/raid/raid6/arm/neon.h > +++ b/lib/raid/raid6/arm/neon.h > @@ -16,6 +16,22 @@ void raid6_neon8_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); > void raid6_neon8_xor_syndrome_real(int disks, int start, int stop, > unsigned long bytes, void **ptrs, > const uint8_t *gmul); > +void raid6_eor3_1_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); > +void raid6_eor3_1_xor_syndrome_real(int disks, int start, int stop, > + unsigned long bytes, void **ptrs, > + const uint8_t *gmul); > +void raid6_eor3_2_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); > +void raid6_eor3_2_xor_syndrome_real(int disks, int start, int stop, > + unsigned long bytes, void **ptrs, > + const uint8_t *gmul); > +void raid6_eor3_4_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); > +void raid6_eor3_4_xor_syndrome_real(int disks, int start, int stop, > + unsigned long bytes, void **ptrs, > + const uint8_t *gmul); > +void raid6_eor3_8_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs); > +void raid6_eor3_8_xor_syndrome_real(int disks, int start, int stop, > + unsigned long bytes, void **ptrs, > + const uint8_t *gmul); > void __raid6_2data_recov_neon(int bytes, uint8_t *p, uint8_t *q, uint8_t *dp, > uint8_t *dq, const uint8_t *pbmul, > const uint8_t *qmul); > diff --git a/lib/raid/raid6/arm/neon.uc b/lib/raid/raid6/arm/neon.uc > index 6f74e2b82fc4..e1084a8d3a18 100644 > --- a/lib/raid/raid6/arm/neon.uc > +++ b/lib/raid/raid6/arm/neon.uc > @@ -33,6 +33,10 @@ > * Since this reduction is linear, the bits shifted out of all terms can be > * XORed together first, and reduced using a single polynomial multiplication > * by 0x1d, which is exact as long as no more than 4 bits are shifted out. > + * > + * When built with RAID6_NEON_EOR3 defined, this file is compiled with the > + * arm64 SHA3 extension enabled, allowing the compiler to fold the additions > + * into three-way XOR (EOR3) instructions. > */ > > #include <arm_neon.h> > @@ -43,7 +47,19 @@ typedef uint8x16_t unative_t; > > #define NSIZE sizeof(unative_t) > > +#ifdef RAID6_NEON_EOR3 > +#define RAID6_NEON_FN(n, fn) raid6_eor3_ ## n ## _ ## fn > + > +/* > + * The compiler emits EOR3 for a ^ b ^ c by itself, but recent GCC versions > + * reassociate the XORs into the loop carried P accumulator such that EOR3 > + * can no longer be used, so spell it out for that case. > + */ > +#define XOR3(a, b, c) veor3q_u8(a, b, c) > +#else > #define RAID6_NEON_FN(n, fn) raid6_neon ## n ## _ ## fn > +#define XOR3(a, b, c) ((a) ^ (b) ^ (c)) > +#endif > > /* > * The SHLBYTE() operation shifts each byte left by 1, *not* > @@ -116,7 +132,7 @@ static inline void STEP2(unative_t *wp, unative_t *wq, const uint8_t *s1, > unative_t d1 = vld1q_u8(s1); > unative_t d0 = vld1q_u8(s0); > > - *wp ^= d1 ^ d0; > + *wp = XOR3(*wp, d1, d0); > *wq = HORNER2(*wq, d1, d0, x1d); > } > > @@ -129,7 +145,7 @@ static inline void STEP4(unative_t *wp, unative_t *wq, const uint8_t *s3, > unative_t d1 = vld1q_u8(s1); > unative_t d0 = vld1q_u8(s0); > > - *wp ^= d3 ^ d2 ^ d1 ^ d0; > + *wp = XOR3(XOR3(*wp, d3, d2), d1, d0); > *wq = HORNER4(*wq, d3, d2, d1, d0, x1d); > } > > diff --git a/lib/raid/raid6/arm64/pq_arch.h b/lib/raid/raid6/arm64/pq_arch.h > index 27ff564d7594..4e397aa36480 100644 > --- a/lib/raid/raid6/arm64/pq_arch.h > +++ b/lib/raid/raid6/arm64/pq_arch.h > @@ -1 +1,33 @@ > -#include "arm/pq_arch.h" > +/* SPDX-License-Identifier: GPL-2.0-or-later */ > + > +#include <linux/cpufeature.h> > +#include <asm/neon.h> > + > +extern const struct raid6_calls raid6_neonx1; > +extern const struct raid6_calls raid6_neonx2; > +extern const struct raid6_calls raid6_neonx4; > +extern const struct raid6_calls raid6_neonx8; > +extern const struct raid6_calls raid6_eor3x1; > +extern const struct raid6_calls raid6_eor3x2; > +extern const struct raid6_calls raid6_eor3x4; > +extern const struct raid6_calls raid6_eor3x8; > +extern const struct raid6_recov_calls raid6_recov_neon; > + > +static __always_inline void __init arch_raid6_init(void) > +{ > + raid6_algo_add_default(); > + if (cpu_has_neon()) { > + if (cpu_have_named_feature(SHA3)) { > + raid6_algo_add(&raid6_eor3x1); > + raid6_algo_add(&raid6_eor3x2); > + raid6_algo_add(&raid6_eor3x4); > + raid6_algo_add(&raid6_eor3x8); > + } else { > + raid6_algo_add(&raid6_neonx1); > + raid6_algo_add(&raid6_neonx2); > + raid6_algo_add(&raid6_neonx4); > + raid6_algo_add(&raid6_neonx8); > + } > + raid6_recov_algo_add(&raid6_recov_neon); > + } > +} > -- > 2.56.1.708.g77570a6a1d-goog > ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 0/2] raid6/arm64: Use aggregation and 3-way XOR 2026-11-18 18:33 [PATCH 0/2] raid6/arm64: Use aggregation and 3-way XOR Ard Biesheuvel 2026-11-18 18:33 ` [PATCH 1/2] raid6: neon: process four data disks per Horner step Ard Biesheuvel 2026-11-18 18:33 ` [PATCH 2/2] raid6: arm64: use the EOR3 instruction where available Ard Biesheuvel @ 2026-11-19 22:50 ` Eric Biggers 2026-11-19 23:58 ` Demian Shulhan 3 siblings, 0 replies; 9+ messages in thread From: Eric Biggers @ 2026-11-19 22:50 UTC (permalink / raw) To: Ard Biesheuvel Cc: linux-arm-kernel, Ard Biesheuvel, Christoph Hellwig, Demian Shulhan On Wed, Nov 18, 2026 at 07:33:36PM +0100, Ard Biesheuvel wrote: > From: Ard Biesheuvel <ardb@kernel.org> > > Tweak the arm64/ARM P/Q gen/xor code to use 3-way XOR instructions where > available. Details are in the commit logs. > > Cc: Christoph Hellwig <hch@lst.de> > Cc: Eric Biggers <ebiggers@kernel.org> > Cc: Demian Shulhan <demyansh@gmail.com> > > Ard Biesheuvel (2): > raid6: neon: process four data disks per Horner step > raid6: arm64: use the EOR3 instruction where available I left some minor comments, and I don't know this code very well. But overall this looks good. Acked-by: Eric Biggers <ebiggers@kernel.org> linux-raid@vger.kernel.org is supposed to be Cc'ed on lib/raid/ changes. (Though I don't know whether anyone from there pays attention to these.) Andrew Morton has been taking lib/raid/ changes for now, so you should Cc him too unless you're trying to get someone else to take these. - Eric ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH 0/2] raid6/arm64: Use aggregation and 3-way XOR 2026-11-18 18:33 [PATCH 0/2] raid6/arm64: Use aggregation and 3-way XOR Ard Biesheuvel ` (2 preceding siblings ...) 2026-11-19 22:50 ` [PATCH 0/2] raid6/arm64: Use aggregation and 3-way XOR Eric Biggers @ 2026-11-19 23:58 ` Demian Shulhan 3 siblings, 0 replies; 9+ messages in thread From: Demian Shulhan @ 2026-11-19 23:58 UTC (permalink / raw) To: Ard Biesheuvel Cc: linux-arm-kernel, Ard Biesheuvel, Christoph Hellwig, Eric Biggers Hi Ard, Overall, the changes look great, I added a few suggestions how we can improve it a little bit. BR, Demian ср, 18 лист. 2026 р. о 19:33 Ard Biesheuvel <ardb+git@google.com> пише: > > From: Ard Biesheuvel <ardb@kernel.org> > > Tweak the arm64/ARM P/Q gen/xor code to use 3-way XOR instructions where > available. Details are in the commit logs. > > Cc: Christoph Hellwig <hch@lst.de> > Cc: Eric Biggers <ebiggers@kernel.org> > Cc: Demian Shulhan <demyansh@gmail.com> > > Ard Biesheuvel (2): > raid6: neon: process four data disks per Horner step > raid6: arm64: use the EOR3 instruction where available > > lib/raid/raid6/.gitignore | 1 + > lib/raid/raid6/Makefile | 19 ++ > lib/raid/raid6/arm/neon.c | 44 +++-- > lib/raid/raid6/arm/neon.h | 28 ++- > lib/raid/raid6/arm/neon.uc | 196 ++++++++++++++------ > lib/raid/raid6/arm/neon_compat.h | 22 +++ > lib/raid/raid6/arm/recov_neon_inner.c | 19 +- > lib/raid/raid6/arm64/pq_arch.h | 34 +++- > 8 files changed, 267 insertions(+), 96 deletions(-) > create mode 100644 lib/raid/raid6/arm/neon_compat.h > > -- > 2.56.1.708.g77570a6a1d-goog > ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2026-11-19 23:59 UTC | newest] Thread overview: 9+ messages (download: mbox.gz follow: Atom feed -- links below jump to the message on this page -- 2026-11-18 18:33 [PATCH 0/2] raid6/arm64: Use aggregation and 3-way XOR Ard Biesheuvel 2026-11-18 18:33 ` [PATCH 1/2] raid6: neon: process four data disks per Horner step Ard Biesheuvel 2026-11-19 22:44 ` Eric Biggers 2026-11-19 23:51 ` Demian Shulhan 2026-11-18 18:33 ` [PATCH 2/2] raid6: arm64: use the EOR3 instruction where available Ard Biesheuvel 2026-11-19 22:35 ` Eric Biggers 2026-11-19 23:55 ` Demian Shulhan 2026-11-19 22:50 ` [PATCH 0/2] raid6/arm64: Use aggregation and 3-way XOR Eric Biggers 2026-11-19 23:58 ` Demian Shulhan
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.