* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Demian Shulhan @ 2026-03-31 13:18 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Ard Biesheuvel, Mark Rutland, Song Liu, Yu Kuai, Will Deacon,
Catalin Marinas, Mark Brown, linux-arm-kernel, robin.murphy,
Li Nan, linux-raid, linux-kernel
In-Reply-To: <20260331063659.GA2061@lst.de>
Hi all,
Ard, your questions regarding real-world I/O bottlenecks and SVE power
efficiency versus raw throughput are entirely valid. I agree that
introducing SVE support requires solid real-world data to justify the
added complexity.
Due to my current workload, I won't be able to run the necessary
hardware tests and prepare the benchmark code immediately. I will get
back to the list in about 1 week with the requested source code,
unmangled test results, and further analysis.
Thanks!
вт, 31 бер. 2026 р. о 09:37 Christoph Hellwig <hch@lst.de> пише:
>
> On Mon, Mar 30, 2026 at 06:39:49PM +0200, Ard Biesheuvel wrote:
> > I think the results are impressive, but I'd like to better understand
> > its implications on a real-world scenario. Is this code only a
> > bottleneck when rebuilding an array?
>
> The syndrome generation is run every time you write data to a RAID6
> array, and if you do partial stripe writes it (or rather the XOR
> variant) is run twice. So this is the most performance critical
> path for writing to RAID6.
>
> Rebuild usually runs totally different code, but can end up here as well
> when both parity disks are lost.
>
> > > Furthermore, as Christoph suggested, I tested scalability on wider
> > > arrays since the default kernel benchmark is hardcoded to 8 disks,
> > > which doesn't give the unrolled SVE loop enough data to shine. On a
> > > 16-disk array, svex4 hits 15.1 GB/s compared to 8.0 GB/s for neonx4.
> > > On a 24-disk array, while neonx4 chokes and drops to 7.8 GB/s, svex4
> > > maintains a stable 15.0 GB/s — effectively doubling the throughput.
> >
> > Does this mean the kernel benchmark is no longer fit for purpose? If
> > it cannot distinguish between implementations that differ in performance
> > by a factor of 2, I don't think we can rely on it to pick the optimal one.
>
> It is not good, and we should either fix it or run more than one.
> The current setup is not really representative of real-life array.
> It also leads to wrong selections on x86, but only at the which unroll
> level to pick level, and only for minor differences so far. I plan
> to add this to the next version of the raid6 lib patches.
>
^ permalink raw reply
* [PATCH v2 5/5] ARM: Remove hacked-up asm/types.h header
From: Ard Biesheuvel @ 2026-03-31 7:49 UTC (permalink / raw)
To: linux-raid
Cc: linux-arm-kernel, linux-crypto, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260331074940.55502-7-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
ARM has a special version of asm/types.h which contains overrides for
certain #define's related to the C types used to back C99 types such as
uint32_t and uintptr_t.
This is only needed when pulling in system headers such as stdint.h
during the build, and this only happens when using NEON intrinsics,
for which there is now a dedicated header file.
So drop this header entirely, and revert to the asm-generic one.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/include/uapi/asm/types.h | 41 --------------------
1 file changed, 41 deletions(-)
diff --git a/arch/arm/include/uapi/asm/types.h b/arch/arm/include/uapi/asm/types.h
deleted file mode 100644
index 1a667bc26510..000000000000
--- a/arch/arm/include/uapi/asm/types.h
+++ /dev/null
@@ -1,41 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
-#ifndef _UAPI_ASM_TYPES_H
-#define _UAPI_ASM_TYPES_H
-
-#include <asm-generic/int-ll64.h>
-
-/*
- * The C99 types uintXX_t that are usually defined in 'stdint.h' are not as
- * unambiguous on ARM as you would expect. For the types below, there is a
- * difference on ARM between GCC built for bare metal ARM, GCC built for glibc
- * and the kernel itself, which results in build errors if you try to build with
- * -ffreestanding and include 'stdint.h' (such as when you include 'arm_neon.h'
- * in order to use NEON intrinsics)
- *
- * As the typedefs for these types in 'stdint.h' are based on builtin defines
- * supplied by GCC, we can tweak these to align with the kernel's idea of those
- * types, so 'linux/types.h' and 'stdint.h' can be safely included from the same
- * source file (provided that -ffreestanding is used).
- *
- * int32_t uint32_t uintptr_t
- * bare metal GCC long unsigned long unsigned int
- * glibc GCC int unsigned int unsigned int
- * kernel int unsigned int unsigned long
- */
-
-#ifdef __INT32_TYPE__
-#undef __INT32_TYPE__
-#define __INT32_TYPE__ int
-#endif
-
-#ifdef __UINT32_TYPE__
-#undef __UINT32_TYPE__
-#define __UINT32_TYPE__ unsigned int
-#endif
-
-#ifdef __UINTPTR_TYPE__
-#undef __UINTPTR_TYPE__
-#define __UINTPTR_TYPE__ unsigned long
-#endif
-
-#endif /* _UAPI_ASM_TYPES_H */
--
2.53.0.1018.g2bb0e51243-goog
^ permalink raw reply related
* [PATCH v2 4/5] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
From: Ard Biesheuvel @ 2026-03-31 7:49 UTC (permalink / raw)
To: linux-raid
Cc: linux-arm-kernel, linux-crypto, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260331074940.55502-7-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Tweak the arm64 code so that the pure NEON intrinsics implementation of
XOR is shared between arm64 and ARM.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/raid/xor/Makefile | 3 +-
lib/raid/xor/arm/xor-neon.c | 4 +
lib/raid/xor/arm64/xor-neon.c | 172 +-------------------
3 files changed, 9 insertions(+), 170 deletions(-)
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index 4d633dfd5b90..b27bf5156784 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -19,7 +19,8 @@ xor-$(CONFIG_ARM) += arm/xor.o
ifeq ($(CONFIG_ARM),y)
xor-$(CONFIG_KERNEL_MODE_NEON) += arm/xor-neon.o arm/xor-neon-glue.o
endif
-xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o
+xor-$(CONFIG_ARM64) += arm/xor-neon.o arm64/xor-neon.o \
+ arm64/xor-neon-glue.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd_glue.o
xor-$(CONFIG_ALTIVEC) += powerpc/xor_vmx.o powerpc/xor_vmx_glue.o
diff --git a/lib/raid/xor/arm/xor-neon.c b/lib/raid/xor/arm/xor-neon.c
index a3e2b4af8d36..c7c3cf634e23 100644
--- a/lib/raid/xor/arm/xor-neon.c
+++ b/lib/raid/xor/arm/xor-neon.c
@@ -173,3 +173,7 @@ static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
__DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4,
__xor_neon_5);
+
+#ifdef CONFIG_ARM64
+extern typeof(__xor_neon_2) __xor_eor3_2 __alias(__xor_neon_2);
+#endif
diff --git a/lib/raid/xor/arm64/xor-neon.c b/lib/raid/xor/arm64/xor-neon.c
index 97ef3cb92496..e44016c363f1 100644
--- a/lib/raid/xor/arm64/xor-neon.c
+++ b/lib/raid/xor/arm64/xor-neon.c
@@ -1,8 +1,4 @@
// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Authors: Jackie Liu <liuyun01@kylinos.cn>
- * Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd.
- */
#include <linux/cache.h>
#include <asm/neon-intrinsics.h>
@@ -10,170 +6,8 @@
#include "xor_arch.h"
#include "xor-neon.h"
-static void __xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* p1 ^= p3 */
- v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
- uint64_t *dp4 = (uint64_t *)p4;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* p1 ^= p3 */
- v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
-
- /* p1 ^= p4 */
- v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- dp4 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
- uint64_t *dp4 = (uint64_t *)p4;
- uint64_t *dp5 = (uint64_t *)p5;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* p1 ^= p3 */
- v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
-
- /* p1 ^= p4 */
- v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
-
- /* p1 ^= p5 */
- v0 = veorq_u64(v0, vld1q_u64(dp5 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp5 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp5 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp5 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- dp4 += 8;
- dp5 += 8;
- } while (--lines > 0);
-}
-
-__DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4,
- __xor_neon_5);
+extern void __xor_eor3_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2);
static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r)
{
@@ -308,5 +142,5 @@ static void __xor_eor3_5(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
-__DO_XOR_BLOCKS(eor3_inner, __xor_neon_2, __xor_eor3_3, __xor_eor3_4,
+__DO_XOR_BLOCKS(eor3_inner, __xor_eor3_2, __xor_eor3_3, __xor_eor3_4,
__xor_eor3_5);
--
2.53.0.1018.g2bb0e51243-goog
^ permalink raw reply related
* [PATCH v2 3/5] xor/arm: Replace vectorized implementation with arm64's intrinsics
From: Ard Biesheuvel @ 2026-03-31 7:49 UTC (permalink / raw)
To: linux-raid
Cc: linux-arm-kernel, linux-crypto, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260331074940.55502-7-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Drop the XOR implementation generated by the vectorizer: this has always
been a bit of a hack, and now that arm64 has an intrinsics version that
works on ARM too, let's use that instead.
So copy the part of the arm64 code that can be shared (so not the EOR3
version). The arm64 code will be updated in a subsequent patch to share
this implementation.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/raid/xor/arm/xor-neon.c | 183 ++++++++++++++++++--
lib/raid/xor/arm/xor-neon.h | 7 +
lib/raid/xor/arm/xor_arch.h | 7 +-
lib/raid/xor/xor-8regs.c | 2 -
4 files changed, 174 insertions(+), 25 deletions(-)
diff --git a/lib/raid/xor/arm/xor-neon.c b/lib/raid/xor/arm/xor-neon.c
index 23147e3a7904..a3e2b4af8d36 100644
--- a/lib/raid/xor/arm/xor-neon.c
+++ b/lib/raid/xor/arm/xor-neon.c
@@ -1,26 +1,175 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
- * Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
+ * Authors: Jackie Liu <liuyun01@kylinos.cn>
+ * Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd.
*/
#include "xor_impl.h"
-#include "xor_arch.h"
+#include "xor-neon.h"
-#ifndef __ARM_NEON__
-#error You should compile this file with '-march=armv7-a -mfloat-abi=softfp -mfpu=neon'
-#endif
+#include <asm/neon-intrinsics.h>
-/*
- * Pull in the reference implementations while instructing GCC (through
- * -ftree-vectorize) to attempt to exploit implicit parallelism and emit
- * NEON instructions. Clang does this by default at O2 so no pragma is
- * needed.
- */
-#ifdef CONFIG_CC_IS_GCC
-#pragma GCC optimize "tree-vectorize"
-#endif
+static void __xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* p1 ^= p3 */
+ v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+ uint64_t *dp4 = (uint64_t *)p4;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* p1 ^= p3 */
+ v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
+
+ /* p1 ^= p4 */
+ v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ dp4 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+ uint64_t *dp4 = (uint64_t *)p4;
+ uint64_t *dp5 = (uint64_t *)p5;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* p1 ^= p3 */
+ v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
+
+ /* p1 ^= p4 */
+ v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
+
+ /* p1 ^= p5 */
+ v0 = veorq_u64(v0, vld1q_u64(dp5 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp5 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp5 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp5 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
-#define NO_TEMPLATE
-#include "../xor-8regs.c"
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ dp4 += 8;
+ dp5 += 8;
+ } while (--lines > 0);
+}
-__DO_XOR_BLOCKS(neon_inner, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);
+__DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4,
+ __xor_neon_5);
diff --git a/lib/raid/xor/arm/xor-neon.h b/lib/raid/xor/arm/xor-neon.h
new file mode 100644
index 000000000000..406e0356f05b
--- /dev/null
+++ b/lib/raid/xor/arm/xor-neon.h
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+extern struct xor_block_template xor_block_arm4regs;
+extern struct xor_block_template xor_block_neon;
+
+void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes);
diff --git a/lib/raid/xor/arm/xor_arch.h b/lib/raid/xor/arm/xor_arch.h
index 775ff835df65..f1ddb64fe62a 100644
--- a/lib/raid/xor/arm/xor_arch.h
+++ b/lib/raid/xor/arm/xor_arch.h
@@ -3,12 +3,7 @@
* Copyright (C) 2001 Russell King
*/
#include <asm/neon.h>
-
-extern struct xor_block_template xor_block_arm4regs;
-extern struct xor_block_template xor_block_neon;
-
-void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt,
- unsigned int bytes);
+#include "xor-neon.h"
static __always_inline void __init arch_xor_init(void)
{
diff --git a/lib/raid/xor/xor-8regs.c b/lib/raid/xor/xor-8regs.c
index 1edaed8acffe..46b3c8bdc27f 100644
--- a/lib/raid/xor/xor-8regs.c
+++ b/lib/raid/xor/xor-8regs.c
@@ -93,11 +93,9 @@ xor_8regs_5(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
-#ifndef NO_TEMPLATE
DO_XOR_BLOCKS(8regs, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);
struct xor_block_template xor_block_8regs = {
.name = "8regs",
.xor_gen = xor_gen_8regs,
};
-#endif /* NO_TEMPLATE */
--
2.53.0.1018.g2bb0e51243-goog
^ permalink raw reply related
* [PATCH v2 2/5] crypto: aegis128 - Use neon-intrinsics.h on ARM too
From: Ard Biesheuvel @ 2026-03-31 7:49 UTC (permalink / raw)
To: linux-raid
Cc: linux-arm-kernel, linux-crypto, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260331074940.55502-7-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Use the asm/neon-intrinsics.h header on ARM as well as arm64, so that
the calling code does not have to know the difference.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
crypto/aegis128-neon-inner.c | 4 +---
1 file changed, 1 insertion(+), 3 deletions(-)
diff --git a/crypto/aegis128-neon-inner.c b/crypto/aegis128-neon-inner.c
index b6a52a386b22..56b534eeb680 100644
--- a/crypto/aegis128-neon-inner.c
+++ b/crypto/aegis128-neon-inner.c
@@ -3,13 +3,11 @@
* Copyright (C) 2019 Linaro, Ltd. <ard.biesheuvel@linaro.org>
*/
-#ifdef CONFIG_ARM64
#include <asm/neon-intrinsics.h>
+#ifdef CONFIG_ARM64
#define AES_ROUND "aese %0.16b, %1.16b \n\t aesmc %0.16b, %0.16b"
#else
-#include <arm_neon.h>
-
#define AES_ROUND "aese.8 %q0, %q1 \n\t aesmc.8 %q0, %q0"
#endif
--
2.53.0.1018.g2bb0e51243-goog
^ permalink raw reply related
* [PATCH v2 1/5] ARM: Add a neon-intrinsics.h header like on arm64
From: Ard Biesheuvel @ 2026-03-31 7:49 UTC (permalink / raw)
To: linux-raid
Cc: linux-arm-kernel, linux-crypto, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260331074940.55502-7-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Add a header asm/neon-intrinsics.h similar to the one that arm64 has.
This makes it possible for NEON intrinsics code to be shared seamlessly
between ARM and arm64.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
Documentation/arch/arm/kernel_mode_neon.rst | 4 +-
arch/arm/include/asm/neon-intrinsics.h | 64 ++++++++++++++++++++
2 files changed, 67 insertions(+), 1 deletion(-)
diff --git a/Documentation/arch/arm/kernel_mode_neon.rst b/Documentation/arch/arm/kernel_mode_neon.rst
index 9bfb71a2a9b9..1efb6d35b7bd 100644
--- a/Documentation/arch/arm/kernel_mode_neon.rst
+++ b/Documentation/arch/arm/kernel_mode_neon.rst
@@ -121,4 +121,6 @@ observe the following in addition to the rules above:
* Compile the unit containing the NEON intrinsics with '-ffreestanding' so GCC
uses its builtin version of <stdint.h> (this is a C99 header which the kernel
does not supply);
-* Include <arm_neon.h> last, or at least after <linux/types.h>
+* Do not include <arm_neon.h> directly: instead, include <asm/neon-intrinsics.h>,
+ which tweaks some macro definitions so that system headers can be included
+ safely.
diff --git a/arch/arm/include/asm/neon-intrinsics.h b/arch/arm/include/asm/neon-intrinsics.h
new file mode 100644
index 000000000000..3fe0b5ab9659
--- /dev/null
+++ b/arch/arm/include/asm/neon-intrinsics.h
@@ -0,0 +1,64 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef __ASM_NEON_INTRINSICS_H
+#define __ASM_NEON_INTRINSICS_H
+
+#ifndef __ARM_NEON__
+#error You should compile this file with '-march=armv7-a -mfloat-abi=softfp -mfpu=neon'
+#endif
+
+#include <asm-generic/int-ll64.h>
+
+/*
+ * The C99 types uintXX_t that are usually defined in 'stdint.h' are not as
+ * unambiguous on ARM as you would expect. For the types below, there is a
+ * difference on ARM between GCC built for bare metal ARM, GCC built for glibc
+ * and the kernel itself, which results in build errors if you try to build
+ * with -ffreestanding and include 'stdint.h' (such as when you include
+ * 'arm_neon.h' in order to use NEON intrinsics)
+ *
+ * As the typedefs for these types in 'stdint.h' are based on builtin defines
+ * supplied by GCC, we can tweak these to align with the kernel's idea of those
+ * types, so 'linux/types.h' and 'stdint.h' can be safely included from the
+ * same source file (provided that -ffreestanding is used).
+ *
+ * int32_t uint32_t intptr_t uintptr_t
+ * bare metal GCC long unsigned long int unsigned int
+ * glibc GCC int unsigned int int unsigned int
+ * kernel int unsigned int long unsigned long
+ */
+
+#ifdef __INT32_TYPE__
+#undef __INT32_TYPE__
+#define __INT32_TYPE__ int
+#endif
+
+#ifdef __UINT32_TYPE__
+#undef __UINT32_TYPE__
+#define __UINT32_TYPE__ unsigned int
+#endif
+
+#ifdef __INTPTR_TYPE__
+#undef __INTPTR_TYPE__
+#define __INTPTR_TYPE__ long
+#endif
+
+#ifdef __UINTPTR_TYPE__
+#undef __UINTPTR_TYPE__
+#define __UINTPTR_TYPE__ unsigned long
+#endif
+
+/*
+ * genksyms chokes on the ARM NEON instrinsics system header, but we
+ * don't export anything it defines anyway, so just disregard when
+ * genksyms execute.
+ */
+#ifndef __GENKSYMS__
+#include <arm_neon.h>
+#endif
+
+#ifdef CONFIG_CC_IS_CLANG
+#pragma clang diagnostic ignored "-Wincompatible-pointer-types"
+#endif
+
+#endif /* __ASM_NEON_INTRINSICS_H */
--
2.53.0.1018.g2bb0e51243-goog
^ permalink raw reply related
* [PATCH v2 0/5] xor/arm: Replace vectorized version with intrinsics
From: Ard Biesheuvel @ 2026-03-31 7:49 UTC (permalink / raw)
To: linux-raid
Cc: linux-arm-kernel, linux-crypto, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
From: Ard Biesheuvel <ardb@kernel.org>
Replace the compiler vectorized XOR implementation for ARM with the
existing NEON intrinsics implementation used by arm64. This is slightly
faster, and allows some minor cleanups of the type hacks in the headers
now that intrinsics are the only C code permitted to use FP/SIMD
instructions.
Changes since v1:
- Update kernel_mode_neon.rst to state that arm_neon.h must not be
included directly, but the new asm/neon-intrinsics.h should be used
instead
- Avoid #include's of .c files - instead, build arm/xor-neon.c for arm64
as a separate compilation unit, and export the symbol that is shared
between the EOR and EOR3 implementations.
Performance (QEMU mach-virt VM running on Synquacer [Cortex-A53 @ 1 GHz]
Before:
[ 3.519687] xor: measuring software checksum speed
[ 3.521725] neon : 1660 MB/sec
[ 3.524733] 32regs : 1105 MB/sec
[ 3.527751] 8regs : 1098 MB/sec
[ 3.529911] arm4regs : 1540 MB/sec
After:
[ 3.517654] xor: measuring software checksum speed
[ 3.519454] neon : 1896 MB/sec
[ 3.522499] 32regs : 1090 MB/sec
[ 3.525560] 8regs : 1083 MB/sec
[ 3.527700] arm4regs : 1556 MB/sec
This applies onto Christoph's XOR cleanup series.
Cc: Christoph Hellwig <hch@lst.de>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Eric Biggers <ebiggers@kernel.org>
Ard Biesheuvel (5):
ARM: Add a neon-intrinsics.h header like on arm64
crypto: aegis128 - Use neon-intrinsics.h on ARM too
xor/arm: Replace vectorized implementation with arm64's intrinsics
xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
ARM: Remove hacked-up asm/types.h header
Documentation/arch/arm/kernel_mode_neon.rst | 4 +-
arch/arm/include/asm/neon-intrinsics.h | 64 +++++++
arch/arm/include/uapi/asm/types.h | 41 -----
crypto/aegis128-neon-inner.c | 4 +-
lib/raid/xor/Makefile | 3 +-
lib/raid/xor/arm/xor-neon.c | 187 ++++++++++++++++++--
lib/raid/xor/arm/xor-neon.h | 7 +
lib/raid/xor/arm/xor_arch.h | 7 +-
lib/raid/xor/arm64/xor-neon.c | 172 +-----------------
lib/raid/xor/xor-8regs.c | 2 -
10 files changed, 251 insertions(+), 240 deletions(-)
create mode 100644 arch/arm/include/asm/neon-intrinsics.h
delete mode 100644 arch/arm/include/uapi/asm/types.h
create mode 100644 lib/raid/xor/arm/xor-neon.h
--
2.53.0.1018.g2bb0e51243-goog
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Christoph Hellwig @ 2026-03-31 6:36 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Demian Shulhan, Mark Rutland, Christoph Hellwig, Song Liu,
Yu Kuai, Will Deacon, Catalin Marinas, Mark Brown,
linux-arm-kernel, robin.murphy, Li Nan, linux-raid, linux-kernel
In-Reply-To: <9a12e043-8200-4650-bfe2-cbece57a4f87@app.fastmail.com>
On Mon, Mar 30, 2026 at 06:39:49PM +0200, Ard Biesheuvel wrote:
> I think the results are impressive, but I'd like to better understand
> its implications on a real-world scenario. Is this code only a
> bottleneck when rebuilding an array?
The syndrome generation is run every time you write data to a RAID6
array, and if you do partial stripe writes it (or rather the XOR
variant) is run twice. So this is the most performance critical
path for writing to RAID6.
Rebuild usually runs totally different code, but can end up here as well
when both parity disks are lost.
> > Furthermore, as Christoph suggested, I tested scalability on wider
> > arrays since the default kernel benchmark is hardcoded to 8 disks,
> > which doesn't give the unrolled SVE loop enough data to shine. On a
> > 16-disk array, svex4 hits 15.1 GB/s compared to 8.0 GB/s for neonx4.
> > On a 24-disk array, while neonx4 chokes and drops to 7.8 GB/s, svex4
> > maintains a stable 15.0 GB/s — effectively doubling the throughput.
>
> Does this mean the kernel benchmark is no longer fit for purpose? If
> it cannot distinguish between implementations that differ in performance
> by a factor of 2, I don't think we can rely on it to pick the optimal one.
It is not good, and we should either fix it or run more than one.
The current setup is not really representative of real-life array.
It also leads to wrong selections on x86, but only at the which unroll
level to pick level, and only for minor differences so far. I plan
to add this to the next version of the raid6 lib patches.
^ permalink raw reply
* Re: [RFC PATCH] md/raid5: Fix UAF on IO across the reshape position
From: Xiao Ni @ 2026-03-31 2:24 UTC (permalink / raw)
To: Benjamin Marzinski, Yu Kuai
Cc: Song Liu, Li Nan, linux-raid, dm-devel, Nigel Croxon
In-Reply-To: <acrMIhAjB-HKszbs@redhat.com>
On Tue, Mar 31, 2026 at 3:16 AM Benjamin Marzinski <bmarzins@redhat.com> wrote:
>
> On Mon, Mar 30, 2026 at 11:44:54PM +0800, Xiao Ni wrote:
> > On Tue, Mar 24, 2026 at 6:58 AM Benjamin Marzinski <bmarzins@redhat.com> wrote:
> > >
> > > If make_stripe_request() returns STRIPE_WAIT_RESHAPE,
> > > raid5_make_request() will free the cloned bio. But raid5_make_request()
> > > can call make_stripe_request() multiple times, writing to the various
> > > stripes. If that bio got added to the toread or towrite lists of a
> > > stripe disk in an earlier call to make_stripe_request(), then it's not
> > > safe to just free the bio if a later part of it is found to cross the
> > > reshape position. Doing so can lead to a UAF error, when bio_endio()
> > > is called on the bio for the earlier stripes.
> > >
> > > Instead, raid5_make_request() needs to wait until all parts of the bio
> > > have called bio_endio(). To do this, bios that cross the reshape
> > > position while the reshape can't make progress are flagged as needing a
> > > retry, and mddev tracks the number of bios needing a retry which have
> > > not yet completed. When raid5_make_request() has a bio that failed
> > > make_stripe_request() with STRIPE_WAIT_RESHAPE, it waits for this
> > > counter to reach zero. When the bio_endio() is called for the last time
> > > on a bio needing a retry, it decrements mddev's count of outstanding
> > > bios needing a retry. This guarantees that raid5_make_request() doesn't
> > > return until the cloned bio needing a retry for io across the reshape
> > > boundary is safely cleaned up.
> > >
> > > There is a simple reproducer available at [1]. Compile the kernel with
> > > KASAN for more useful reporting when the error is triggered (this is not
> > > necessary to see the bug).
> > >
> > > [1] https://gist.github.com/bmarzins/e48598824305cf2171289e47d7241fa5
> > >
> > > Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
> > > ---
> > >
> > > I've tested this for regressions with the lvm2-testsuite raid tests. I
> > > have not run any md-specific tests on it.
> > >
> > >
> > > drivers/md/md.c | 30 +++++++++---------------------
> > > drivers/md/md.h | 5 ++++-
> > > drivers/md/raid5.c | 8 +++++++-
> > > 3 files changed, 20 insertions(+), 23 deletions(-)
> > >
> > > diff --git a/drivers/md/md.c b/drivers/md/md.c
> > > index 3ce6f9e9d38e..5ec116b9da32 100644
> > > --- a/drivers/md/md.c
> > > +++ b/drivers/md/md.c
> > > @@ -776,9 +776,11 @@ int mddev_init(struct mddev *mddev)
> > > atomic_set(&mddev->active, 1);
> > > atomic_set(&mddev->openers, 0);
> > > atomic_set(&mddev->sync_seq, 0);
> > > + atomic_set(&mddev->pending_retry_bios, 0);
> > > spin_lock_init(&mddev->lock);
> > > init_waitqueue_head(&mddev->sb_wait);
> > > init_waitqueue_head(&mddev->recovery_wait);
> > > + init_waitqueue_head(&mddev->retry_bios_wait);
> > > mddev->reshape_position = MaxSector;
> > > mddev->reshape_backwards = 0;
> > > mddev->last_sync_action = ACTION_IDLE;
> > > @@ -9218,6 +9220,7 @@ static void md_end_clone_io(struct bio *bio)
> > > struct md_io_clone *md_io_clone = bio->bi_private;
> > > struct bio *orig_bio = md_io_clone->orig_bio;
> > > struct mddev *mddev = md_io_clone->mddev;
> > > + unsigned int must_retry = md_io_clone->must_retry;
> > >
> > > if (bio_data_dir(orig_bio) == WRITE && md_bitmap_enabled(mddev, false))
> > > md_bitmap_end(mddev, md_io_clone);
> > > @@ -9229,7 +9232,11 @@ static void md_end_clone_io(struct bio *bio)
> > > bio_end_io_acct(orig_bio, md_io_clone->start_time);
> > >
> > > bio_put(bio);
> > > - bio_endio(orig_bio);
> > > + if (unlikely(must_retry)) {
> > > + if (atomic_dec_and_test(&mddev->pending_retry_bios))
> > > + wake_up(&mddev->retry_bios_wait);
> > > + } else
> > > + bio_endio(orig_bio);
> > > percpu_ref_put(&mddev->active_io);
> > > }
> > >
> > > @@ -9243,6 +9250,7 @@ static void md_clone_bio(struct mddev *mddev, struct bio **bio)
> > > md_io_clone = container_of(clone, struct md_io_clone, bio_clone);
> > > md_io_clone->orig_bio = *bio;
> > > md_io_clone->mddev = mddev;
> > > + md_io_clone->must_retry = 0;
> > > if (blk_queue_io_stat(bdev->bd_disk->queue))
> > > md_io_clone->start_time = bio_start_io_acct(*bio);
> > >
> > > @@ -9265,26 +9273,6 @@ void md_account_bio(struct mddev *mddev, struct bio **bio)
> > > }
> > > EXPORT_SYMBOL_GPL(md_account_bio);
> > >
> > > -void md_free_cloned_bio(struct bio *bio)
> > > -{
> > > - struct md_io_clone *md_io_clone = bio->bi_private;
> > > - struct bio *orig_bio = md_io_clone->orig_bio;
> > > - struct mddev *mddev = md_io_clone->mddev;
> > > -
> > > - if (bio_data_dir(orig_bio) == WRITE && md_bitmap_enabled(mddev, false))
> > > - md_bitmap_end(mddev, md_io_clone);
> > > -
> > > - if (bio->bi_status && !orig_bio->bi_status)
> > > - orig_bio->bi_status = bio->bi_status;
> > > -
> > > - if (md_io_clone->start_time)
> > > - bio_end_io_acct(orig_bio, md_io_clone->start_time);
> > > -
> > > - bio_put(bio);
> > > - percpu_ref_put(&mddev->active_io);
> > > -}
> > > -EXPORT_SYMBOL_GPL(md_free_cloned_bio);
> > > -
> > > /* md_allow_write(mddev)
> > > * Calling this ensures that the array is marked 'active' so that writes
> > > * may proceed without blocking. It is important to call this before
> > > diff --git a/drivers/md/md.h b/drivers/md/md.h
> > > index ac84289664cd..49a231f11676 100644
> > > --- a/drivers/md/md.h
> > > +++ b/drivers/md/md.h
> > > @@ -626,6 +626,9 @@ struct mddev {
> > >
> > > /* The sequence number for sync thread */
> > > atomic_t sync_seq;
> > > +
> > > + wait_queue_head_t retry_bios_wait;
> > > + atomic_t pending_retry_bios;
> > > };
> > >
> > > enum recovery_flags {
> > > @@ -877,6 +880,7 @@ struct md_io_clone {
> > > sector_t offset;
> > > unsigned long sectors;
> > > enum stat_group rw;
> > > + unsigned int must_retry;
> > > struct bio bio_clone;
> > > };
> > >
> > > @@ -917,7 +921,6 @@ extern void md_finish_reshape(struct mddev *mddev);
> > > void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,
> > > struct bio *bio, sector_t start, sector_t size);
> > > void md_account_bio(struct mddev *mddev, struct bio **bio);
> > > -void md_free_cloned_bio(struct bio *bio);
> > >
> > > extern bool __must_check md_flush_request(struct mddev *mddev, struct bio *bio);
> > > void md_write_metadata(struct mddev *mddev, struct md_rdev *rdev,
> > > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> > > index a8e8d431071b..fb78a757f2fd 100644
> > > --- a/drivers/md/raid5.c
> > > +++ b/drivers/md/raid5.c
> > > @@ -6217,7 +6217,13 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
> > >
> > > mempool_free(ctx, conf->ctx_pool);
> > > if (res == STRIPE_WAIT_RESHAPE) {
> > > - md_free_cloned_bio(bi);
> > > + struct md_io_clone *md_io_clone = bi->bi_private;
> > > +
> > > + md_io_clone->must_retry = 1;
> > > + atomic_inc(&mddev->pending_retry_bios);
> > > + bio_endio(bi);
> > > + wait_event(mddev->retry_bios_wait,
> > > + atomic_read(&mddev->pending_retry_bios)==0);
> >
> > Hi Ben
> >
> > There is a problem here. The new counter pending_retry_bios above
> > doesn't represent the bios which have been added to stripes. So uaf
> > still can happen.
>
> I don't think it can. That counter won't be zero until there are no bios
> that need to get retried. So we way end up waiting too long, but we
> shouldn't ever end up not waiting long enough.
Ah, yes, you're right. I thought wrongly. The counter tracks multiple
threads that encounter BLK_STS_RESOURCE. For each thread,
md_end_clone_bio is called only when all bios return. But why do
different threads need to sync at raid5_make_request?
>
> > Do we need to add a new counter?
>
> I did it that way because I wanted to avoid growing md_io_clone
> ("must_retry" fit nicely in a hole in the structure, so it didn't take
> up any extra space). But if you are fine with adding the extra
Nice design.
> reshape_completion pointer to md_io_clone, then your way avoids the
> chance that we might wait when we don't need to.
If you mention the threads sync here, yes.
>
> > Because
> > md_end_clone_io is only called when all bios return. How about
> > something like:
> >
> > md_end_clone_io
> > + if (unlikely(READ_ONCE(md_io_clone->waiting_reshape))) {
> > + complete(md_io_clone->reshape_completion);
> > + } else {
> > + bio_endio(orig_bio);
> > + }
> >
> > raid5_make_request:
> >
> > if (res == STRIPE_WAIT_RESHAPE) {
> > - md_free_cloned_bio(bi);
> > + struct md_io_clone *md_io_clone = bi->bi_private;
> > + struct completion done;
> > +
> > + init_completion(&done);
> > + md_io_clone->reshape_completion = &done;
> > + WRITE_ONCE(md_io_clone->waiting_reshape, 1);
> > +
> > + bio_endio(bi);
> > +
> > + wait_for_completion(&done);
>
> This looks fine.
>
> I'm pretty sure that the READ_ONCE() and WRITE_ONCE() are unnecessary.
> First, theres no chance that these operations will ever happen at the
> same time and we're just writing a 1, so there's no chance of tearing.
For the first point, I have a question. They can't happen
simultaneously. But the cache of different CPUs may contain different
values.
>
> The compiler can't reorder setting md_io_clone->waiting_reshape to afer
> bio_endio(), since bio_endio can free md_io_clone. Also the compiler
> can't reorder reading it to before calling md_end_clone_io() from
> bio_endio(), obviously.
Thanks very much for pointing this out.
>
> If the bio was never chained, then there can't be a race, since only
> raid5_make_request() will be calling bio_endio(). waiting_reshape will
> be read by the same process that sets it.
>
> If the bio was chained, then like you said, md_end_clone_io() will only
> get called by the final caller of bio_endio(). This is determined by
> bio_remaining_done(), which guarantees a full memory barrier for
> atomic_dec_and_test(&bio->__bi_remaining).
Nice analysis! Thanks.
>
> Since we know the compiler will keep these instructions on the proper
> size of a full memory barrier, I'm pretty sure that this is concurrency
> race free, even without the READ_ONCE() and WRITE_ONCE(). Like with
> trying to avoid growing md_io_clone, I was trying to keep
> md_end_clone_io() as fast as possible (at least on architectures where
> READ_ONCE can have more of a performance impact) Again, if I'm
> overthinking this (or if my concurrency argument is flawed) feel free to
> ignore me.
Thanks for your patience in explaining the concurrency problem. I
agree with you. And it's better to add comments that describe the
reason and it will help people understand it better.
By the way, the md raid maintainer's email has changed. I fixed the
email address in to list.
Regards
Xiao
>
> -Ben
>
> > Best Regards
> > Xiao
> >
> >
> > > return false;
> > > }
> >
> > >
> > > --
> > > 2.53.0
> > >
>
^ permalink raw reply
* Re: [RFC PATCH] md/raid5: Fix UAF on IO across the reshape position
From: Benjamin Marzinski @ 2026-03-30 19:16 UTC (permalink / raw)
To: Xiao Ni; +Cc: Song Liu, Yu Kuai, Li Nan, linux-raid, dm-devel, Nigel Croxon
In-Reply-To: <CALTww29P_jZ-n9GMgFG0EW_N22vUsGAPJ5OGn5Lw8yfEpEoGOw@mail.gmail.com>
On Mon, Mar 30, 2026 at 11:44:54PM +0800, Xiao Ni wrote:
> On Tue, Mar 24, 2026 at 6:58 AM Benjamin Marzinski <bmarzins@redhat.com> wrote:
> >
> > If make_stripe_request() returns STRIPE_WAIT_RESHAPE,
> > raid5_make_request() will free the cloned bio. But raid5_make_request()
> > can call make_stripe_request() multiple times, writing to the various
> > stripes. If that bio got added to the toread or towrite lists of a
> > stripe disk in an earlier call to make_stripe_request(), then it's not
> > safe to just free the bio if a later part of it is found to cross the
> > reshape position. Doing so can lead to a UAF error, when bio_endio()
> > is called on the bio for the earlier stripes.
> >
> > Instead, raid5_make_request() needs to wait until all parts of the bio
> > have called bio_endio(). To do this, bios that cross the reshape
> > position while the reshape can't make progress are flagged as needing a
> > retry, and mddev tracks the number of bios needing a retry which have
> > not yet completed. When raid5_make_request() has a bio that failed
> > make_stripe_request() with STRIPE_WAIT_RESHAPE, it waits for this
> > counter to reach zero. When the bio_endio() is called for the last time
> > on a bio needing a retry, it decrements mddev's count of outstanding
> > bios needing a retry. This guarantees that raid5_make_request() doesn't
> > return until the cloned bio needing a retry for io across the reshape
> > boundary is safely cleaned up.
> >
> > There is a simple reproducer available at [1]. Compile the kernel with
> > KASAN for more useful reporting when the error is triggered (this is not
> > necessary to see the bug).
> >
> > [1] https://gist.github.com/bmarzins/e48598824305cf2171289e47d7241fa5
> >
> > Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
> > ---
> >
> > I've tested this for regressions with the lvm2-testsuite raid tests. I
> > have not run any md-specific tests on it.
> >
> >
> > drivers/md/md.c | 30 +++++++++---------------------
> > drivers/md/md.h | 5 ++++-
> > drivers/md/raid5.c | 8 +++++++-
> > 3 files changed, 20 insertions(+), 23 deletions(-)
> >
> > diff --git a/drivers/md/md.c b/drivers/md/md.c
> > index 3ce6f9e9d38e..5ec116b9da32 100644
> > --- a/drivers/md/md.c
> > +++ b/drivers/md/md.c
> > @@ -776,9 +776,11 @@ int mddev_init(struct mddev *mddev)
> > atomic_set(&mddev->active, 1);
> > atomic_set(&mddev->openers, 0);
> > atomic_set(&mddev->sync_seq, 0);
> > + atomic_set(&mddev->pending_retry_bios, 0);
> > spin_lock_init(&mddev->lock);
> > init_waitqueue_head(&mddev->sb_wait);
> > init_waitqueue_head(&mddev->recovery_wait);
> > + init_waitqueue_head(&mddev->retry_bios_wait);
> > mddev->reshape_position = MaxSector;
> > mddev->reshape_backwards = 0;
> > mddev->last_sync_action = ACTION_IDLE;
> > @@ -9218,6 +9220,7 @@ static void md_end_clone_io(struct bio *bio)
> > struct md_io_clone *md_io_clone = bio->bi_private;
> > struct bio *orig_bio = md_io_clone->orig_bio;
> > struct mddev *mddev = md_io_clone->mddev;
> > + unsigned int must_retry = md_io_clone->must_retry;
> >
> > if (bio_data_dir(orig_bio) == WRITE && md_bitmap_enabled(mddev, false))
> > md_bitmap_end(mddev, md_io_clone);
> > @@ -9229,7 +9232,11 @@ static void md_end_clone_io(struct bio *bio)
> > bio_end_io_acct(orig_bio, md_io_clone->start_time);
> >
> > bio_put(bio);
> > - bio_endio(orig_bio);
> > + if (unlikely(must_retry)) {
> > + if (atomic_dec_and_test(&mddev->pending_retry_bios))
> > + wake_up(&mddev->retry_bios_wait);
> > + } else
> > + bio_endio(orig_bio);
> > percpu_ref_put(&mddev->active_io);
> > }
> >
> > @@ -9243,6 +9250,7 @@ static void md_clone_bio(struct mddev *mddev, struct bio **bio)
> > md_io_clone = container_of(clone, struct md_io_clone, bio_clone);
> > md_io_clone->orig_bio = *bio;
> > md_io_clone->mddev = mddev;
> > + md_io_clone->must_retry = 0;
> > if (blk_queue_io_stat(bdev->bd_disk->queue))
> > md_io_clone->start_time = bio_start_io_acct(*bio);
> >
> > @@ -9265,26 +9273,6 @@ void md_account_bio(struct mddev *mddev, struct bio **bio)
> > }
> > EXPORT_SYMBOL_GPL(md_account_bio);
> >
> > -void md_free_cloned_bio(struct bio *bio)
> > -{
> > - struct md_io_clone *md_io_clone = bio->bi_private;
> > - struct bio *orig_bio = md_io_clone->orig_bio;
> > - struct mddev *mddev = md_io_clone->mddev;
> > -
> > - if (bio_data_dir(orig_bio) == WRITE && md_bitmap_enabled(mddev, false))
> > - md_bitmap_end(mddev, md_io_clone);
> > -
> > - if (bio->bi_status && !orig_bio->bi_status)
> > - orig_bio->bi_status = bio->bi_status;
> > -
> > - if (md_io_clone->start_time)
> > - bio_end_io_acct(orig_bio, md_io_clone->start_time);
> > -
> > - bio_put(bio);
> > - percpu_ref_put(&mddev->active_io);
> > -}
> > -EXPORT_SYMBOL_GPL(md_free_cloned_bio);
> > -
> > /* md_allow_write(mddev)
> > * Calling this ensures that the array is marked 'active' so that writes
> > * may proceed without blocking. It is important to call this before
> > diff --git a/drivers/md/md.h b/drivers/md/md.h
> > index ac84289664cd..49a231f11676 100644
> > --- a/drivers/md/md.h
> > +++ b/drivers/md/md.h
> > @@ -626,6 +626,9 @@ struct mddev {
> >
> > /* The sequence number for sync thread */
> > atomic_t sync_seq;
> > +
> > + wait_queue_head_t retry_bios_wait;
> > + atomic_t pending_retry_bios;
> > };
> >
> > enum recovery_flags {
> > @@ -877,6 +880,7 @@ struct md_io_clone {
> > sector_t offset;
> > unsigned long sectors;
> > enum stat_group rw;
> > + unsigned int must_retry;
> > struct bio bio_clone;
> > };
> >
> > @@ -917,7 +921,6 @@ extern void md_finish_reshape(struct mddev *mddev);
> > void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,
> > struct bio *bio, sector_t start, sector_t size);
> > void md_account_bio(struct mddev *mddev, struct bio **bio);
> > -void md_free_cloned_bio(struct bio *bio);
> >
> > extern bool __must_check md_flush_request(struct mddev *mddev, struct bio *bio);
> > void md_write_metadata(struct mddev *mddev, struct md_rdev *rdev,
> > diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> > index a8e8d431071b..fb78a757f2fd 100644
> > --- a/drivers/md/raid5.c
> > +++ b/drivers/md/raid5.c
> > @@ -6217,7 +6217,13 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
> >
> > mempool_free(ctx, conf->ctx_pool);
> > if (res == STRIPE_WAIT_RESHAPE) {
> > - md_free_cloned_bio(bi);
> > + struct md_io_clone *md_io_clone = bi->bi_private;
> > +
> > + md_io_clone->must_retry = 1;
> > + atomic_inc(&mddev->pending_retry_bios);
> > + bio_endio(bi);
> > + wait_event(mddev->retry_bios_wait,
> > + atomic_read(&mddev->pending_retry_bios)==0);
>
> Hi Ben
>
> There is a problem here. The new counter pending_retry_bios above
> doesn't represent the bios which have been added to stripes. So uaf
> still can happen.
I don't think it can. That counter won't be zero until there are no bios
that need to get retried. So we way end up waiting too long, but we
shouldn't ever end up not waiting long enough.
> Do we need to add a new counter?
I did it that way because I wanted to avoid growing md_io_clone
("must_retry" fit nicely in a hole in the structure, so it didn't take
up any extra space). But if you are fine with adding the extra
reshape_completion pointer to md_io_clone, then your way avoids the
chance that we might wait when we don't need to.
> Because
> md_end_clone_io is only called when all bios return. How about
> something like:
>
> md_end_clone_io
> + if (unlikely(READ_ONCE(md_io_clone->waiting_reshape))) {
> + complete(md_io_clone->reshape_completion);
> + } else {
> + bio_endio(orig_bio);
> + }
>
> raid5_make_request:
>
> if (res == STRIPE_WAIT_RESHAPE) {
> - md_free_cloned_bio(bi);
> + struct md_io_clone *md_io_clone = bi->bi_private;
> + struct completion done;
> +
> + init_completion(&done);
> + md_io_clone->reshape_completion = &done;
> + WRITE_ONCE(md_io_clone->waiting_reshape, 1);
> +
> + bio_endio(bi);
> +
> + wait_for_completion(&done);
This looks fine.
I'm pretty sure that the READ_ONCE() and WRITE_ONCE() are unnecessary.
First, theres no chance that these operations will ever happen at the
same time and we're just writing a 1, so there's no chance of tearing.
The compiler can't reorder setting md_io_clone->waiting_reshape to afer
bio_endio(), since bio_endio can free md_io_clone. Also the compiler
can't reorder reading it to before calling md_end_clone_io() from
bio_endio(), obviously.
If the bio was never chained, then there can't be a race, since only
raid5_make_request() will be calling bio_endio(). waiting_reshape will
be read by the same process that sets it.
If the bio was chained, then like you said, md_end_clone_io() will only
get called by the final caller of bio_endio(). This is determined by
bio_remaining_done(), which guarantees a full memory barrier for
atomic_dec_and_test(&bio->__bi_remaining).
Since we know the compiler will keep these instructions on the proper
size of a full memory barrier, I'm pretty sure that this is concurrency
race free, even without the READ_ONCE() and WRITE_ONCE(). Like with
trying to avoid growing md_io_clone, I was trying to keep
md_end_clone_io() as fast as possible (at least on architectures where
READ_ONCE can have more of a performance impact) Again, if I'm
overthinking this (or if my concurrency argument is flawed) feel free to
ignore me.
-Ben
> Best Regards
> Xiao
>
>
> > return false;
> > }
>
> >
> > --
> > 2.53.0
> >
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Ard Biesheuvel @ 2026-03-30 16:39 UTC (permalink / raw)
To: Demian Shulhan, Mark Rutland
Cc: Christoph Hellwig, Song Liu, Yu Kuai, Will Deacon,
Catalin Marinas, Mark Brown, linux-arm-kernel, robin.murphy,
Li Nan, linux-raid, linux-kernel
In-Reply-To: <CAOLeWCsxhzdxQviizJ4X4VOp_28LCtO-RjWoCcZG29rQw86NVg@mail.gmail.com>
Hi Demian,
On Sun, 29 Mar 2026, at 15:01, Demian Shulhan wrote:
> I want to address the comment about the marginal 0.3% speedup on the
> 8-disk benchmark. While the pure memory bandwidth on a small array is
> indeed bottlenecked, it doesn't reveal the whole picture. I extracted
> the SVE and NEON implementations into a user-space benchmark to
> measure the actual hardware efficiency using perf stat, running on the
> same AWS Graviton3 (Neoverse-V1) instance.The results show a massive
> difference in CPU efficiency. For the same 8-disk workload, the svex4
> implementation requires about 35% fewer instructions and 46% fewer CPU
> cycles compared to neonx4 (7.58 billion instructions vs 11.62
> billion). This translates directly into significant energy savings and
> reduced pressure on the CPU frontend, which would leave more compute
> resources available for network and NVMe queues during an array
> rebuild.
>
I think the results are impressive, but I'd like to better understand
its implications on a real-world scenario. Is this code only a
bottleneck when rebuilding an array? Is it really that much more power
efficient, given that the registers (and ALU paths) are twice the size?
And given the I/O load of rebuilding a 24+ disk array, how much CPU
throughput can we make use of meaningfully in such a scenario?
Supporting SVE in the kernel primarily impacts the size of the per-task
buffers that we need to preserve/restore the context. Fortunately,
these are no longer allocated for the lifetime of the task, but
dynamically (by scoped_ksimd()), and so the main impediment has been
recently removed. But as Mark pointed out, there are other things to
take into account. Nonetheless, our position has always been that a
compelling use case could convince us that the additional complexity
of in-kernel SVE is justified.
> Furthermore, as Christoph suggested, I tested scalability on wider
> arrays since the default kernel benchmark is hardcoded to 8 disks,
> which doesn't give the unrolled SVE loop enough data to shine. On a
> 16-disk array, svex4 hits 15.1 GB/s compared to 8.0 GB/s for neonx4.
> On a 24-disk array, while neonx4 chokes and drops to 7.8 GB/s, svex4
> maintains a stable 15.0 GB/s — effectively doubling the throughput.
Does this mean the kernel benchmark is no longer fit for purpose? If
it cannot distinguish between implementations that differ in performance
by a factor of 2, I don't think we can rely on it to pick the optimal one.
> I agree this patch should be put on hold for now. My intention is to
> leave these numbers here as evidence that implementing SVE context
> preservation in the kernel (the "good use case") is highly justifiable
> from both a power-efficiency and a wide-array throughput perspective
> for modern ARM64 hardware.
>
Could you please summarize the results? The output below seems to have
become mangled a bit. Please also include the command line, a link to
the test source, and the vector length of the implementation.
> Thanks again for your time and time and review!
>
> ---------------------------------------------------
> User space test results:
> ==================================================
> RAID6 SVE Benchmark Results (AWS Graviton3)
> ==================================================
> Instance Details:
> Linux ip-172-31-87-234 6.8.0-1047-aws #50~22.04.1-Ubuntu SMP Thu Feb
> 19 20:49:25 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux
> --------------------------------------------------
>
> [Test 1: Energy Efficiency / Instruction Count (8 disks)]
> Running baseline (neonx4)...
> algo=neonx4 ndisks=8 iterations=1000000 time=2.681s MB/s=8741.36
>
> Running SVE (svex1)...
>
> Performance counter stats for './raid6_bench neonx4 8 1000000':
>
> 11626717224 instructions # 1.67
> insn per cycle
> 6946699489 cycles
> 257013219 L1-dcache-load-misses
>
> 2.681213149 seconds time elapsed
>
> 2.676771000 seconds user
> 0.002000000 seconds sys
>
>
> algo=svex1 ndisks=8 iterations=1000000 time=1.688s MB/s=13885.23
>
> Performance counter stats for './raid6_bench svex1 8 1000000':
>
> 10527277490
> Running SVE unrolled x4 (svex4)...
> instructions # 2.40 insn per cycle
> 4379539835 cycles
> 175695656 L1-dcache-load-misses
>
> 1.688852006 seconds time elapsed
>
> 1.687298000 seconds user
> 0.000999000 seconds sys
>
>
> algo=svex4 ndisks=8 iterations=1000000 time=1.445s MB/s=16215.04
>
> Performance counter stats for './raid6_bench svex4 8 1000000':
>
> 7587813392 instructions
> ==================================================
> [Test 2: Scalability on Wide RAID Arrays (MB/s)]
> --- 16 Disks ---
> # 2.02 insn per cycle
> 3748486131 cycles
> 213816184 L1-dcache-load-misses
>
> 1.446032415 seconds time elapsed
>
> 1.442412000 seconds user
> 0.002996000 seconds sys
>
>
> algo=neonx4 ndisks=16 iterations=1000000 time=6.783s MB/s=8062.33
> algo=svex1 ndisks=16 iterations=1000000 time=4.912s MB/s=11132.90
> algo=svex4 ndisks=16 iterations=1000000 time=3.601s MB/s=15188.85
>
> --- 24 Disks ---
> algo=neonx4 ndisks=24 iterations=1000000 time=11.011s MB/s=7805.02
> algo=svex1 ndisks=24 iterations=1000000 time=8.843s MB/s=9718.26
> algo=svex4 ndisks=24 iterations=1000000 time=5.719s MB/s=15026.92
>
> Extra tests:
> --- 48 Disks ---
> algo=neonx4 ndisks=48 iterations=500000 time=11.826s MB/s=7597.25
> algo=svex4 ndisks=48 iterations=500000 time=5.808s MB/s=15468.10
> --- 96 Disks ---
> algo=neonx4 ndisks=96 iterations=200000 time=9.783s MB/s=7507.01
> algo=svex4 ndisks=96 iterations=200000 time=4.701s MB/s=15621.17
> ==================================================
>
^ permalink raw reply
* Re: [PATCH] md: fix array_state=clear sysfs deadlock
From: Xiao Ni @ 2026-03-30 15:47 UTC (permalink / raw)
To: Yu Kuai; +Cc: song, linan122, linux-raid, linux-kernel
In-Reply-To: <20260330055213.3976052-1-yukuai@fnnas.com>
On Mon, Mar 30, 2026 at 1:55 PM Yu Kuai <yukuai@fnnas.com> wrote:
>
> From: Yu Kuai <yukuai3@huawei.com>
>
> When "clear" is written to array_state, md_attr_store() breaks sysfs
> active protection so the array can delete itself from its own sysfs
> store method.
>
> However, md_attr_store() currently drops the mddev reference before
> calling sysfs_unbreak_active_protection(). Once do_md_stop(..., 0)
> has made the mddev eligible for delayed deletion, the temporary
> kobject reference taken by sysfs_break_active_protection() can become
> the last kobject reference protecting the md kobject.
>
> That allows sysfs_unbreak_active_protection() to drop the last
> kobject reference from the current sysfs writer context. kobject
> teardown then recurses into kernfs removal while the current sysfs
> node is still being unwound, and lockdep reports recursive locking on
> kn->active with kernfs_drain() in the call chain.
>
> Reproducer on an existing level:
> 1. Create an md0 linear array and activate it:
> mknod /dev/md0 b 9 0
> echo none > /sys/block/md0/md/metadata_version
> echo linear > /sys/block/md0/md/level
> echo 1 > /sys/block/md0/md/raid_disks
> echo "$(cat /sys/class/block/sdb/dev)" > /sys/block/md0/md/new_dev
> echo "$(($(cat /sys/class/block/sdb/size) / 2))" > \
> /sys/block/md0/md/dev-sdb/size
> echo 0 > /sys/block/md0/md/dev-sdb/slot
> echo active > /sys/block/md0/md/array_state
> 2. Wait briefly for the array to settle, then clear it:
> sleep 2
> echo clear > /sys/block/md0/md/array_state
>
> The warning looks like:
>
> WARNING: possible recursive locking detected
> bash/588 is trying to acquire lock:
> (kn->active#65) at __kernfs_remove+0x157/0x1d0
> but task is already holding lock:
> (kn->active#65) at sysfs_unbreak_active_protection+0x1f/0x40
> ...
> Call Trace:
> kernfs_drain
> __kernfs_remove
> kernfs_remove_by_name_ns
> sysfs_remove_group
> sysfs_remove_groups
> __kobject_del
> kobject_put
> md_attr_store
> kernfs_fop_write_iter
> vfs_write
> ksys_write
>
> Restore active protection before mddev_put() so the extra sysfs
> kobject reference is dropped while the mddev is still held alive. The
> actual md kobject deletion is then deferred until after the sysfs
> write path has fully returned.
>
> Fixes: 9e59d609763f ("md: call del_gendisk in control path")
> Signed-off-by: Yu Kuai <yukuai3@huawei.com>
> ---
> drivers/md/md.c | 8 +++++++-
> 1 file changed, 7 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 521d9b34cd9e..02efe9700256 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -6130,10 +6130,16 @@ md_attr_store(struct kobject *kobj, struct attribute *attr,
> }
> spin_unlock(&all_mddevs_lock);
> rv = entry->store(mddev, page, length);
> - mddev_put(mddev);
>
> + /*
> + * For "array_state=clear", dropping the extra kobject reference from
> + * sysfs_break_active_protection() can trigger md kobject deletion.
> + * Restore active protection before mddev_put() so deletion happens
> + * after the sysfs write path fully unwinds.
> + */
> if (kn)
> sysfs_unbreak_active_protection(kn);
> + mddev_put(mddev);
>
> return rv;
> }
> --
> 2.51.0
>
>
This patch looks good to me.
Reviewed-by: Xiao Ni <xni@redhat.com>
^ permalink raw reply
* Re: [RFC PATCH] md/raid5: Fix UAF on IO across the reshape position
From: Xiao Ni @ 2026-03-30 15:44 UTC (permalink / raw)
To: Benjamin Marzinski
Cc: Song Liu, Yu Kuai, Li Nan, linux-raid, dm-devel, Nigel Croxon
In-Reply-To: <20260323225836.1037060-1-bmarzins@redhat.com>
On Tue, Mar 24, 2026 at 6:58 AM Benjamin Marzinski <bmarzins@redhat.com> wrote:
>
> If make_stripe_request() returns STRIPE_WAIT_RESHAPE,
> raid5_make_request() will free the cloned bio. But raid5_make_request()
> can call make_stripe_request() multiple times, writing to the various
> stripes. If that bio got added to the toread or towrite lists of a
> stripe disk in an earlier call to make_stripe_request(), then it's not
> safe to just free the bio if a later part of it is found to cross the
> reshape position. Doing so can lead to a UAF error, when bio_endio()
> is called on the bio for the earlier stripes.
>
> Instead, raid5_make_request() needs to wait until all parts of the bio
> have called bio_endio(). To do this, bios that cross the reshape
> position while the reshape can't make progress are flagged as needing a
> retry, and mddev tracks the number of bios needing a retry which have
> not yet completed. When raid5_make_request() has a bio that failed
> make_stripe_request() with STRIPE_WAIT_RESHAPE, it waits for this
> counter to reach zero. When the bio_endio() is called for the last time
> on a bio needing a retry, it decrements mddev's count of outstanding
> bios needing a retry. This guarantees that raid5_make_request() doesn't
> return until the cloned bio needing a retry for io across the reshape
> boundary is safely cleaned up.
>
> There is a simple reproducer available at [1]. Compile the kernel with
> KASAN for more useful reporting when the error is triggered (this is not
> necessary to see the bug).
>
> [1] https://gist.github.com/bmarzins/e48598824305cf2171289e47d7241fa5
>
> Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
> ---
>
> I've tested this for regressions with the lvm2-testsuite raid tests. I
> have not run any md-specific tests on it.
>
>
> drivers/md/md.c | 30 +++++++++---------------------
> drivers/md/md.h | 5 ++++-
> drivers/md/raid5.c | 8 +++++++-
> 3 files changed, 20 insertions(+), 23 deletions(-)
>
> diff --git a/drivers/md/md.c b/drivers/md/md.c
> index 3ce6f9e9d38e..5ec116b9da32 100644
> --- a/drivers/md/md.c
> +++ b/drivers/md/md.c
> @@ -776,9 +776,11 @@ int mddev_init(struct mddev *mddev)
> atomic_set(&mddev->active, 1);
> atomic_set(&mddev->openers, 0);
> atomic_set(&mddev->sync_seq, 0);
> + atomic_set(&mddev->pending_retry_bios, 0);
> spin_lock_init(&mddev->lock);
> init_waitqueue_head(&mddev->sb_wait);
> init_waitqueue_head(&mddev->recovery_wait);
> + init_waitqueue_head(&mddev->retry_bios_wait);
> mddev->reshape_position = MaxSector;
> mddev->reshape_backwards = 0;
> mddev->last_sync_action = ACTION_IDLE;
> @@ -9218,6 +9220,7 @@ static void md_end_clone_io(struct bio *bio)
> struct md_io_clone *md_io_clone = bio->bi_private;
> struct bio *orig_bio = md_io_clone->orig_bio;
> struct mddev *mddev = md_io_clone->mddev;
> + unsigned int must_retry = md_io_clone->must_retry;
>
> if (bio_data_dir(orig_bio) == WRITE && md_bitmap_enabled(mddev, false))
> md_bitmap_end(mddev, md_io_clone);
> @@ -9229,7 +9232,11 @@ static void md_end_clone_io(struct bio *bio)
> bio_end_io_acct(orig_bio, md_io_clone->start_time);
>
> bio_put(bio);
> - bio_endio(orig_bio);
> + if (unlikely(must_retry)) {
> + if (atomic_dec_and_test(&mddev->pending_retry_bios))
> + wake_up(&mddev->retry_bios_wait);
> + } else
> + bio_endio(orig_bio);
> percpu_ref_put(&mddev->active_io);
> }
>
> @@ -9243,6 +9250,7 @@ static void md_clone_bio(struct mddev *mddev, struct bio **bio)
> md_io_clone = container_of(clone, struct md_io_clone, bio_clone);
> md_io_clone->orig_bio = *bio;
> md_io_clone->mddev = mddev;
> + md_io_clone->must_retry = 0;
> if (blk_queue_io_stat(bdev->bd_disk->queue))
> md_io_clone->start_time = bio_start_io_acct(*bio);
>
> @@ -9265,26 +9273,6 @@ void md_account_bio(struct mddev *mddev, struct bio **bio)
> }
> EXPORT_SYMBOL_GPL(md_account_bio);
>
> -void md_free_cloned_bio(struct bio *bio)
> -{
> - struct md_io_clone *md_io_clone = bio->bi_private;
> - struct bio *orig_bio = md_io_clone->orig_bio;
> - struct mddev *mddev = md_io_clone->mddev;
> -
> - if (bio_data_dir(orig_bio) == WRITE && md_bitmap_enabled(mddev, false))
> - md_bitmap_end(mddev, md_io_clone);
> -
> - if (bio->bi_status && !orig_bio->bi_status)
> - orig_bio->bi_status = bio->bi_status;
> -
> - if (md_io_clone->start_time)
> - bio_end_io_acct(orig_bio, md_io_clone->start_time);
> -
> - bio_put(bio);
> - percpu_ref_put(&mddev->active_io);
> -}
> -EXPORT_SYMBOL_GPL(md_free_cloned_bio);
> -
> /* md_allow_write(mddev)
> * Calling this ensures that the array is marked 'active' so that writes
> * may proceed without blocking. It is important to call this before
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index ac84289664cd..49a231f11676 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -626,6 +626,9 @@ struct mddev {
>
> /* The sequence number for sync thread */
> atomic_t sync_seq;
> +
> + wait_queue_head_t retry_bios_wait;
> + atomic_t pending_retry_bios;
> };
>
> enum recovery_flags {
> @@ -877,6 +880,7 @@ struct md_io_clone {
> sector_t offset;
> unsigned long sectors;
> enum stat_group rw;
> + unsigned int must_retry;
> struct bio bio_clone;
> };
>
> @@ -917,7 +921,6 @@ extern void md_finish_reshape(struct mddev *mddev);
> void md_submit_discard_bio(struct mddev *mddev, struct md_rdev *rdev,
> struct bio *bio, sector_t start, sector_t size);
> void md_account_bio(struct mddev *mddev, struct bio **bio);
> -void md_free_cloned_bio(struct bio *bio);
>
> extern bool __must_check md_flush_request(struct mddev *mddev, struct bio *bio);
> void md_write_metadata(struct mddev *mddev, struct md_rdev *rdev,
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index a8e8d431071b..fb78a757f2fd 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -6217,7 +6217,13 @@ static bool raid5_make_request(struct mddev *mddev, struct bio * bi)
>
> mempool_free(ctx, conf->ctx_pool);
> if (res == STRIPE_WAIT_RESHAPE) {
> - md_free_cloned_bio(bi);
> + struct md_io_clone *md_io_clone = bi->bi_private;
> +
> + md_io_clone->must_retry = 1;
> + atomic_inc(&mddev->pending_retry_bios);
> + bio_endio(bi);
> + wait_event(mddev->retry_bios_wait,
> + atomic_read(&mddev->pending_retry_bios)==0);
Hi Ben
There is a problem here. The new counter pending_retry_bios above
doesn't represent the bios which have been added to stripes. So uaf
still can happen. Do we need to add a new counter? Because
md_end_clone_io is only called when all bios return. How about
something like:
md_end_clone_io
+ if (unlikely(READ_ONCE(md_io_clone->waiting_reshape))) {
+ complete(md_io_clone->reshape_completion);
+ } else {
+ bio_endio(orig_bio);
+ }
raid5_make_request:
if (res == STRIPE_WAIT_RESHAPE) {
- md_free_cloned_bio(bi);
+ struct md_io_clone *md_io_clone = bi->bi_private;
+ struct completion done;
+
+ init_completion(&done);
+ md_io_clone->reshape_completion = &done;
+ WRITE_ONCE(md_io_clone->waiting_reshape, 1);
+
+ bio_endio(bi);
+
+ wait_for_completion(&done);
Best Regards
Xiao
> return false;
> }
>
> --
> 2.53.0
>
^ permalink raw reply
* Re: [PATCH 4/5] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
From: Christoph Hellwig @ 2026-03-30 13:02 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Christoph Hellwig, Ard Biesheuvel, linux-raid, linux-arm-kernel,
linux-crypto, Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <6bedf98e-a424-4baa-890c-806345c067c1@app.fastmail.com>
On Mon, Mar 30, 2026 at 11:38:15AM +0200, Ard Biesheuvel wrote:
> > This avoid the including of .c files which is always a bit ugly.
> > But if there is a strong argument to prefer including of the .c file I
> > can live with that as well.
> >
>
> I've respun it without the include. Instead, I've added this to arm/xor-neon.c
>
> +#ifdef CONFIG_ARM64
> +extern typeof(__xor_neon_2) __xor_eor3_2 __alias(__xor_neon_2);
> +#endif
>
> so that __xor_eor3_2() exists in the arm64 build as an alias. That way, the arm64-only EOR3 implementation can just remain a separate compilation unit.
Ok.
^ permalink raw reply
* Re: [PATCH 4/5] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
From: Ard Biesheuvel @ 2026-03-30 9:38 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Ard Biesheuvel, linux-raid, linux-arm-kernel, linux-crypto,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260330053233.GB4736@lst.de>
On Mon, 30 Mar 2026, at 07:32, Christoph Hellwig wrote:
> On Fri, Mar 27, 2026 at 03:45:56PM +0100, Ard Biesheuvel wrote:
>> On Fri, 27 Mar 2026, at 14:50, Christoph Hellwig wrote:
>> > On Fri, Mar 27, 2026 at 12:30:52PM +0100, Ard Biesheuvel wrote:
>> >> From: Ard Biesheuvel <ardb@kernel.org>
>> >>
>> >> Tweak the arm64 code so that the pure NEON intrinsics implementation of
>> >> XOR is shared between arm64 and ARM.
>> >
>> > Instead of hiding the implementation in a header, just split xor-neon.c
>> > into two .c files, one of which could be built by arm32 as well.
>>
>> That is what patch 3/5 does. This patch wires up that version into arm64, and drops the copy that has become redundant as a result.
>
> Yeah, sorry - I misread the series a little.
>
>>
>> > probably
>> > in the arm/ instead of the arm64/ subdirectory, but we can also add a
>> > new arm-common one if that's what the arm maintainers prefer.
>>
>> Having the shared pure NEON version in arm/ is perfectly fine.
>
> So here would be my preference:
>
> - keep all the arm/arm64 code in lib/raid/xor/arm
> - have the neon and EOR3 code in a single xor-neon.c file, with an
> ifdef CONFIG_ARM64 around the EOE3 routines
>
> This avoid the including of .c files which is always a bit ugly.
> But if there is a strong argument to prefer including of the .c file I
> can live with that as well.
>
I've respun it without the include. Instead, I've added this to arm/xor-neon.c
+#ifdef CONFIG_ARM64
+extern typeof(__xor_neon_2) __xor_eor3_2 __alias(__xor_neon_2);
+#endif
so that __xor_eor3_2() exists in the arm64 build as an alias. That way, the arm64-only EOR3 implementation can just remain a separate compilation unit.
I could move the eor3 code under arm/ too, but that seems a bit odd given that it is arm64 only, and a arm64/ sub-directory exists.
^ permalink raw reply
* [PATCH] md: fix array_state=clear sysfs deadlock
From: Yu Kuai @ 2026-03-30 5:52 UTC (permalink / raw)
To: song; +Cc: yukuai, linan122, xni, linux-raid, linux-kernel
From: Yu Kuai <yukuai3@huawei.com>
When "clear" is written to array_state, md_attr_store() breaks sysfs
active protection so the array can delete itself from its own sysfs
store method.
However, md_attr_store() currently drops the mddev reference before
calling sysfs_unbreak_active_protection(). Once do_md_stop(..., 0)
has made the mddev eligible for delayed deletion, the temporary
kobject reference taken by sysfs_break_active_protection() can become
the last kobject reference protecting the md kobject.
That allows sysfs_unbreak_active_protection() to drop the last
kobject reference from the current sysfs writer context. kobject
teardown then recurses into kernfs removal while the current sysfs
node is still being unwound, and lockdep reports recursive locking on
kn->active with kernfs_drain() in the call chain.
Reproducer on an existing level:
1. Create an md0 linear array and activate it:
mknod /dev/md0 b 9 0
echo none > /sys/block/md0/md/metadata_version
echo linear > /sys/block/md0/md/level
echo 1 > /sys/block/md0/md/raid_disks
echo "$(cat /sys/class/block/sdb/dev)" > /sys/block/md0/md/new_dev
echo "$(($(cat /sys/class/block/sdb/size) / 2))" > \
/sys/block/md0/md/dev-sdb/size
echo 0 > /sys/block/md0/md/dev-sdb/slot
echo active > /sys/block/md0/md/array_state
2. Wait briefly for the array to settle, then clear it:
sleep 2
echo clear > /sys/block/md0/md/array_state
The warning looks like:
WARNING: possible recursive locking detected
bash/588 is trying to acquire lock:
(kn->active#65) at __kernfs_remove+0x157/0x1d0
but task is already holding lock:
(kn->active#65) at sysfs_unbreak_active_protection+0x1f/0x40
...
Call Trace:
kernfs_drain
__kernfs_remove
kernfs_remove_by_name_ns
sysfs_remove_group
sysfs_remove_groups
__kobject_del
kobject_put
md_attr_store
kernfs_fop_write_iter
vfs_write
ksys_write
Restore active protection before mddev_put() so the extra sysfs
kobject reference is dropped while the mddev is still held alive. The
actual md kobject deletion is then deferred until after the sysfs
write path has fully returned.
Fixes: 9e59d609763f ("md: call del_gendisk in control path")
Signed-off-by: Yu Kuai <yukuai3@huawei.com>
---
drivers/md/md.c | 8 +++++++-
1 file changed, 7 insertions(+), 1 deletion(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 521d9b34cd9e..02efe9700256 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -6130,10 +6130,16 @@ md_attr_store(struct kobject *kobj, struct attribute *attr,
}
spin_unlock(&all_mddevs_lock);
rv = entry->store(mddev, page, length);
- mddev_put(mddev);
+ /*
+ * For "array_state=clear", dropping the extra kobject reference from
+ * sysfs_break_active_protection() can trigger md kobject deletion.
+ * Restore active protection before mddev_put() so deletion happens
+ * after the sysfs write path fully unwinds.
+ */
if (kn)
sysfs_unbreak_active_protection(kn);
+ mddev_put(mddev);
return rv;
}
--
2.51.0
^ permalink raw reply related
* Re: [PATCH 4/5] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
From: Christoph Hellwig @ 2026-03-30 5:32 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Christoph Hellwig, Ard Biesheuvel, linux-raid, linux-arm-kernel,
linux-crypto, Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <cca6facc-6c37-48d0-81e6-f8568f36b91d@app.fastmail.com>
On Fri, Mar 27, 2026 at 03:45:56PM +0100, Ard Biesheuvel wrote:
> On Fri, 27 Mar 2026, at 14:50, Christoph Hellwig wrote:
> > On Fri, Mar 27, 2026 at 12:30:52PM +0100, Ard Biesheuvel wrote:
> >> From: Ard Biesheuvel <ardb@kernel.org>
> >>
> >> Tweak the arm64 code so that the pure NEON intrinsics implementation of
> >> XOR is shared between arm64 and ARM.
> >
> > Instead of hiding the implementation in a header, just split xor-neon.c
> > into two .c files, one of which could be built by arm32 as well.
>
> That is what patch 3/5 does. This patch wires up that version into arm64, and drops the copy that has become redundant as a result.
Yeah, sorry - I misread the series a little.
>
> > probably
> > in the arm/ instead of the arm64/ subdirectory, but we can also add a
> > new arm-common one if that's what the arm maintainers prefer.
>
> Having the shared pure NEON version in arm/ is perfectly fine.
So here would be my preference:
- keep all the arm/arm64 code in lib/raid/xor/arm
- have the neon and EOR3 code in a single xor-neon.c file, with an
ifdef CONFIG_ARM64 around the EOE3 routines
This avoid the including of .c files which is always a bit ugly.
But if there is a strong argument to prefer including of the .c file I
can live with that as well.
>
> Building it as a separate compilation unit for arm64 should also be straight-forward, the only issue is that the 2-way NEON version needs to be shared with the EOR3 compilation unit.
>
---end quoted text---
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Christoph Hellwig @ 2026-03-30 5:30 UTC (permalink / raw)
To: Demian Shulhan
Cc: Mark Rutland, Ard Biesheuvel, Christoph Hellwig, Song Liu,
Yu Kuai, Will Deacon, Catalin Marinas, broonie, linux-arm-kernel,
robin.murphy, Li Nan, linux-raid, linux-kernel
In-Reply-To: <CAOLeWCsxhzdxQviizJ4X4VOp_28LCtO-RjWoCcZG29rQw86NVg@mail.gmail.com>
On Sun, Mar 29, 2026 at 04:01:06PM +0300, Demian Shulhan wrote:
> Furthermore, as Christoph suggested, I tested scalability on wider
> arrays since the default kernel benchmark is hardcoded to 8 disks,
> which doesn't give the unrolled SVE loop enough data to shine. On a
> 16-disk array, svex4 hits 15.1 GB/s compared to 8.0 GB/s for neonx4.
> On a 24-disk array, while neonx4 chokes and drops to 7.8 GB/s, svex4
> maintains a stable 15.0 GB/s — effectively doubling the throughput.I
> agree this patch should be put on hold for now. My intention is to
> leave these numbers here as evidence that implementing SVE context
> preservation in the kernel (the "good use case") is highly justifiable
> from both a power-efficiency and a wide-array throughput perspective
> for modern ARM64 hardware.
>
> Thanks again for your time and time and review!
To me this sounds like an interesting case for a SVE kernel API.
But I'm not relly knowledgeable enough to provide one to help
with testing this further.
^ permalink raw reply
* Re: cleanup the RAID5 XOR library v4
From: Christoph Hellwig @ 2026-03-30 5:26 UTC (permalink / raw)
To: Eric Biggers
Cc: Christoph Hellwig, Andrew Morton, Richard Henderson, Matt Turner,
Magnus Lindholm, Russell King, Catalin Marinas, Will Deacon,
Ard Biesheuvel, Huacai Chen, WANG Xuerui, Madhavan Srinivasan,
Michael Ellerman, Nicholas Piggin, Christophe Leroy (CS GROUP),
Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, David S. Miller,
Andreas Larsson, Richard Weinberger, Anton Ivanov, Johannes Berg,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Herbert Xu, Dan Williams, Chris Mason,
David Sterba, Arnd Bergmann, Song Liu, Yu Kuai, Li Nan,
Theodore Ts'o, Jason A. Donenfeld, linux-alpha, linux-kernel,
linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv,
linux-s390, sparclinux, linux-um, linux-crypto, linux-btrfs,
linux-arch, linux-raid
In-Reply-To: <20260329213119.GA2106@quark>
On Sun, Mar 29, 2026 at 02:31:19PM -0700, Eric Biggers wrote:
> But yes, as Andrew mentioned there are two "xor: add a better public
> API" patches. They should be folded together.
Yes. This is the ages old git bug of creating duplicate commits when
a rebase has conflicts biting again, sorry.
^ permalink raw reply
* Re: cleanup the RAID5 XOR library v4
From: Eric Biggers @ 2026-03-30 4:07 UTC (permalink / raw)
To: Andrew Morton
Cc: Christoph Hellwig, Richard Henderson, Matt Turner,
Magnus Lindholm, Russell King, Catalin Marinas, Will Deacon,
Ard Biesheuvel, Huacai Chen, WANG Xuerui, Madhavan Srinivasan,
Michael Ellerman, Nicholas Piggin, Christophe Leroy (CS GROUP),
Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, David S. Miller,
Andreas Larsson, Richard Weinberger, Anton Ivanov, Johannes Berg,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Herbert Xu, Dan Williams, Chris Mason,
David Sterba, Arnd Bergmann, Song Liu, Yu Kuai, Li Nan,
Theodore Ts'o, Jason A. Donenfeld, linux-alpha, linux-kernel,
linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv,
linux-s390, sparclinux, linux-um, linux-crypto, linux-btrfs,
linux-arch, linux-raid
In-Reply-To: <20260329155126.a01a5729b7d8376712182851@linux-foundation.org>
On Sun, Mar 29, 2026 at 03:51:26PM -0700, Andrew Morton wrote:
> >
> > Reviewed-by: Eric Biggers <ebiggers@kernel.org>
>
> Great, thanks, added to all changelogs.
>
> > But yes, as Andrew mentioned there are two "xor: add a better public
> > API" patches. They should be folded together.
>
> I folded them.
>
> I'm a bit wobbly about upstreaming all this for 7.1-rc1. It hits on a
> lot of stuff and I don't think we've heard a lot from the affected
> maintainers.
>
> otoh, we're unlikely to learn much from an additional nine weeks in
> linux-next so at some point one has to forge ahead and rely on seven
> weeks of -rc to address any remaining niggles. And I'm confident that
> Christoph will support his work well.
>
> But still, hearing some reassuring words about this would be
> appreciated ;)
The architecture-optimized crypto and CRC code has been the same way.
I've been working on it across architectures, and most of the arch
maintainers don't pay much attention to it.
I've seen engagement from a few of them, for example s390. But as a
general rule it's a separate group of people working on this code.
I think seeing the same for lib/raid/ is expected. So while the arch
maintainers are always welcome to chime in, I don't think we need to
wait for all of them, as then we'd be waiting forever.
Re testing, I've been running the crypto, CRC, and now the XOR KUnit
tests in QEMU for 8 architectures (arm, arm64, mips, powerpc, riscv,
s390, sparc, and x86), and over 40 variants within those (e.g. varying
CONFIG_64BIT, CONFIG_CPU_BIG_ENDIAN, and QEMU's "-cpu" flag). They are
all passing, including the XOR test that Christoph added in this series.
(So I guess feel free to add:
Tested-by: Eric Biggers <ebiggers@kernel.org>
to all the patches in this series as well.)
That still doesn't cover all the arch-optimized code, due to me missing
various combinations or QEMU not supporting them. But it's something.
I'm also hoping that with the move to standard KUnit tests, we'll get a
larger group of people, including projects like KernelCI, that simply
run *all* the kernel's KUnit tests on whatever platforms they care
about. That approach is more scalable.
- Eric
^ permalink raw reply
* Re: cleanup the RAID5 XOR library v4
From: Andrew Morton @ 2026-03-29 22:51 UTC (permalink / raw)
To: Eric Biggers
Cc: Christoph Hellwig, Richard Henderson, Matt Turner,
Magnus Lindholm, Russell King, Catalin Marinas, Will Deacon,
Ard Biesheuvel, Huacai Chen, WANG Xuerui, Madhavan Srinivasan,
Michael Ellerman, Nicholas Piggin, Christophe Leroy (CS GROUP),
Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, David S. Miller,
Andreas Larsson, Richard Weinberger, Anton Ivanov, Johannes Berg,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Herbert Xu, Dan Williams, Chris Mason,
David Sterba, Arnd Bergmann, Song Liu, Yu Kuai, Li Nan,
Theodore Ts'o, Jason A. Donenfeld, linux-alpha, linux-kernel,
linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv,
linux-s390, sparclinux, linux-um, linux-crypto, linux-btrfs,
linux-arch, linux-raid
In-Reply-To: <20260329213119.GA2106@quark>
On Sun, 29 Mar 2026 14:31:19 -0700 Eric Biggers <ebiggers@kernel.org> wrote:
> On Fri, Mar 27, 2026 at 07:16:32AM +0100, Christoph Hellwig wrote:
> > Hi all,
> >
> > the XOR library used for the RAID5 parity is a bit of a mess right now.
> > The main file sits in crypto/ despite not being cryptography and not
> > using the crypto API, with the generic implementations sitting in
> > include/asm-generic and the arch implementations sitting in an asm/
> > header in theory. The latter doesn't work for many cases, so
> > architectures often build the code directly into the core kernel, or
> > create another module for the architecture code.
> >
> > Changes this to a single module in lib/ that also contains the
> > architecture optimizations, similar to the library work Eric Biggers
> > has done for the CRC and crypto libraries later. After that it changes
> > to better calling conventions that allow for smarter architecture
> > implementations (although none is contained here yet), and uses
> > static_call to avoid indirection function call overhead.
> >
> > A git tree is also available here:
> >
> > git://git.infradead.org/users/hch/misc.git xor-improvements
> >
> > Gitweb:
> >
> > https://git.infradead.org/?p=users/hch/misc.git;a=shortlog;h=refs/heads/xor-improvements
> >
> > Changes since v3:
> > - switch away from lockdep_assert_preemption_enabled() again
> > - fix a @ reference in a kerneldoc comment.
> > - build the arm4regs implementation also without kernel-mode neon
> > support
> > - fix a pre-existing issue about mismatched attributes on arm64's
> > xor_block_inner_neon
> > - reject 0-sized xor request and adjust the kunit test case to not
> > generate them
>
> Reviewed-by: Eric Biggers <ebiggers@kernel.org>
Great, thanks, added to all changelogs.
> But yes, as Andrew mentioned there are two "xor: add a better public
> API" patches. They should be folded together.
I folded them.
I'm a bit wobbly about upstreaming all this for 7.1-rc1. It hits on a
lot of stuff and I don't think we've heard a lot from the affected
maintainers.
otoh, we're unlikely to learn much from an additional nine weeks in
linux-next so at some point one has to forge ahead and rely on seven
weeks of -rc to address any remaining niggles. And I'm confident that
Christoph will support his work well.
But still, hearing some reassuring words about this would be
appreciated ;)
^ permalink raw reply
* Re: cleanup the RAID5 XOR library v4
From: Eric Biggers @ 2026-03-29 21:31 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Ard Biesheuvel,
Huacai Chen, WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
On Fri, Mar 27, 2026 at 07:16:32AM +0100, Christoph Hellwig wrote:
> Hi all,
>
> the XOR library used for the RAID5 parity is a bit of a mess right now.
> The main file sits in crypto/ despite not being cryptography and not
> using the crypto API, with the generic implementations sitting in
> include/asm-generic and the arch implementations sitting in an asm/
> header in theory. The latter doesn't work for many cases, so
> architectures often build the code directly into the core kernel, or
> create another module for the architecture code.
>
> Changes this to a single module in lib/ that also contains the
> architecture optimizations, similar to the library work Eric Biggers
> has done for the CRC and crypto libraries later. After that it changes
> to better calling conventions that allow for smarter architecture
> implementations (although none is contained here yet), and uses
> static_call to avoid indirection function call overhead.
>
> A git tree is also available here:
>
> git://git.infradead.org/users/hch/misc.git xor-improvements
>
> Gitweb:
>
> https://git.infradead.org/?p=users/hch/misc.git;a=shortlog;h=refs/heads/xor-improvements
>
> Changes since v3:
> - switch away from lockdep_assert_preemption_enabled() again
> - fix a @ reference in a kerneldoc comment.
> - build the arm4regs implementation also without kernel-mode neon
> support
> - fix a pre-existing issue about mismatched attributes on arm64's
> xor_block_inner_neon
> - reject 0-sized xor request and adjust the kunit test case to not
> generate them
Reviewed-by: Eric Biggers <ebiggers@kernel.org>
But yes, as Andrew mentioned there are two "xor: add a better public
API" patches. They should be folded together.
- Eric
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Demian Shulhan @ 2026-03-29 13:01 UTC (permalink / raw)
To: Mark Rutland
Cc: Ard Biesheuvel, Christoph Hellwig, Song Liu, Yu Kuai, Will Deacon,
Catalin Marinas, broonie, linux-arm-kernel, robin.murphy, Li Nan,
linux-raid, linux-kernel
In-Reply-To: <acJhCCPfVxjFUZ5R@J2N7QTR9R3>
Hi all,
Thanks for the feedback and for clarifying the current limitations of
the kernel-mode SIMD API regarding SVE context preservation. I
completely understand why this patch cannot be merged in its current
state until the fundamental SVE infrastructure is in place.However, I
want to address the comment about the marginal 0.3% speedup on the
8-disk benchmark. While the pure memory bandwidth on a small array is
indeed bottlenecked, it doesn't reveal the whole picture. I extracted
the SVE and NEON implementations into a user-space benchmark to
measure the actual hardware efficiency using perf stat, running on the
same AWS Graviton3 (Neoverse-V1) instance.The results show a massive
difference in CPU efficiency. For the same 8-disk workload, the svex4
implementation requires about 35% fewer instructions and 46% fewer CPU
cycles compared to neonx4 (7.58 billion instructions vs 11.62
billion). This translates directly into significant energy savings and
reduced pressure on the CPU frontend, which would leave more compute
resources available for network and NVMe queues during an array
rebuild.
Furthermore, as Christoph suggested, I tested scalability on wider
arrays since the default kernel benchmark is hardcoded to 8 disks,
which doesn't give the unrolled SVE loop enough data to shine. On a
16-disk array, svex4 hits 15.1 GB/s compared to 8.0 GB/s for neonx4.
On a 24-disk array, while neonx4 chokes and drops to 7.8 GB/s, svex4
maintains a stable 15.0 GB/s — effectively doubling the throughput.I
agree this patch should be put on hold for now. My intention is to
leave these numbers here as evidence that implementing SVE context
preservation in the kernel (the "good use case") is highly justifiable
from both a power-efficiency and a wide-array throughput perspective
for modern ARM64 hardware.
Thanks again for your time and time and review!
---------------------------------------------------
User space test results:
==================================================
RAID6 SVE Benchmark Results (AWS Graviton3)
==================================================
Instance Details:
Linux ip-172-31-87-234 6.8.0-1047-aws #50~22.04.1-Ubuntu SMP Thu Feb
19 20:49:25 UTC 2026 aarch64 aarch64 aarch64 GNU/Linux
--------------------------------------------------
[Test 1: Energy Efficiency / Instruction Count (8 disks)]
Running baseline (neonx4)...
algo=neonx4 ndisks=8 iterations=1000000 time=2.681s MB/s=8741.36
Running SVE (svex1)...
Performance counter stats for './raid6_bench neonx4 8 1000000':
11626717224 instructions # 1.67
insn per cycle
6946699489 cycles
257013219 L1-dcache-load-misses
2.681213149 seconds time elapsed
2.676771000 seconds user
0.002000000 seconds sys
algo=svex1 ndisks=8 iterations=1000000 time=1.688s MB/s=13885.23
Performance counter stats for './raid6_bench svex1 8 1000000':
10527277490
Running SVE unrolled x4 (svex4)...
instructions # 2.40 insn per cycle
4379539835 cycles
175695656 L1-dcache-load-misses
1.688852006 seconds time elapsed
1.687298000 seconds user
0.000999000 seconds sys
algo=svex4 ndisks=8 iterations=1000000 time=1.445s MB/s=16215.04
Performance counter stats for './raid6_bench svex4 8 1000000':
7587813392 instructions
==================================================
[Test 2: Scalability on Wide RAID Arrays (MB/s)]
--- 16 Disks ---
# 2.02 insn per cycle
3748486131 cycles
213816184 L1-dcache-load-misses
1.446032415 seconds time elapsed
1.442412000 seconds user
0.002996000 seconds sys
algo=neonx4 ndisks=16 iterations=1000000 time=6.783s MB/s=8062.33
algo=svex1 ndisks=16 iterations=1000000 time=4.912s MB/s=11132.90
algo=svex4 ndisks=16 iterations=1000000 time=3.601s MB/s=15188.85
--- 24 Disks ---
algo=neonx4 ndisks=24 iterations=1000000 time=11.011s MB/s=7805.02
algo=svex1 ndisks=24 iterations=1000000 time=8.843s MB/s=9718.26
algo=svex4 ndisks=24 iterations=1000000 time=5.719s MB/s=15026.92
Extra tests:
--- 48 Disks ---
algo=neonx4 ndisks=48 iterations=500000 time=11.826s MB/s=7597.25
algo=svex4 ndisks=48 iterations=500000 time=5.808s MB/s=15468.10
--- 96 Disks ---
algo=neonx4 ndisks=96 iterations=200000 time=9.783s MB/s=7507.01
algo=svex4 ndisks=96 iterations=200000 time=4.701s MB/s=15621.17
==================================================
вт, 24 бер. 2026 р. о 12:05 Mark Rutland <mark.rutland@arm.com> пише:
>
> On Tue, Mar 24, 2026 at 09:00:16AM +0100, Ard Biesheuvel wrote:
> > On Wed, 18 Mar 2026, at 16:02, Demian Shulhan wrote:
> > > Implement Scalable Vector Extension (SVE) optimized routines for RAID6
> > > syndrome generation and recovery on ARM64.
> > >
> > > The SVE instruction set allows for variable vector lengths (from 128 to
> > > 2048 bits), scaling automatically with the hardware capabilities. This
> > > implementation handles arbitrary SVE vector lengths using the `cntb`
> > > instruction to determine the runtime vector length.
> > >
> > > The implementation introduces `svex1`, `svex2`, and `svex4` algorithms.
> > > The `svex4` algorithm utilizes loop unrolling by 4 blocks per iteration
> > > and manual software pipelining (interleaving memory loads with XORs)
> > > to minimize instruction dependency stalls and maximize CPU pipeline
> > > utilization and memory bandwidth.
> > >
> > > Performance was tested on an AWS Graviton3 (Neoverse-V1) instance which
> > > features 256-bit SVE vector length. The `svex4` implementation outperforms
> > > the existing 128-bit `neonx4` baseline for syndrome generation:
> > >
> > > raid6: svex4 gen() 19688 MB/s
> > ...
> > > raid6: neonx4 gen() 19612 MB/s
> >
> > You're being very generous characterising a 0.3% speedup as 'outperforms'
> >
> > But the real problem here is that the kernel-mode SIMD API only
> > supports NEON and not SVE, and preserves/restores only the 128-bit
> > view on the NEON/SVE register file. So any context switch or softirq
> > that uses kernel-mode SIMD too, and your SVE register values will get
> > truncated.
>
> Just to be a bit more explicit, since only the NEON register file is
> saved:
>
> * The vector registers will be truncated to 128-bit across
> preemption or softirq.
>
> * The predicates won't be saved/restored and will change arbitrarily
> across preemption.
>
> * The VL won't be saved/restored, and might change arbitrarily across
> preemption.
>
> * The VL to use hasn't been programmed, so performance might vary
> arbitrarily even in the absence of preemption.
>
> ... so this isn't even safe on machines with (only) a 128-bit VL, and
> there are big open design questions for the infrastructure we'd need.
>
> > Once we encounter a good use case for SVE in the kernel, we might
> > reconsider this, but as it stands, this patch should not be applied.
>
> I agree.
>
> Christoph, please do not pick this or any other in-kernel SVE patches.
> They cannot function correctly without additional infrastructure.
>
> Demian, for patches that use NEON/SVE/SME/etc, please Cc LAKML
> (linux-arm-kernel@lists.infradead.org), so that folk familiar with ARM
> see the patches.
>
> Mark
>
> > (leaving the reply untrimmed for the benefit of the cc'ees I added)
> >
> > > raid6: neonx2 gen() 16248 MB/s
> > > raid6: neonx1 gen() 13591 MB/s
> > > raid6: using algorithm svex4 gen() 19688 MB/s
> > > raid6: .... xor() 11212 MB/s, rmw enabled
> > > raid6: using neon recovery algorithm
> > >
> > > Note that for the recovery path (`xor_syndrome`), NEON may still be
> > > selected dynamically by the algorithm benchmark, as the recovery
> > > workload is heavily memory-bound.
> > >
> > > Signed-off-by: Demian Shulhan <demyansh@gmail.com>
> > > Reported-by: kernel test robot <lkp@intel.com>
> > > Closes:
> > > https://lore.kernel.org/oe-kbuild-all/202603181940.cFwYmYoi-lkp@intel.com/
> > > ---
> > > include/linux/raid/pq.h | 3 +
> > > lib/raid6/Makefile | 5 +
> > > lib/raid6/algos.c | 5 +
> > > lib/raid6/sve.c | 675 ++++++++++++++++++++++++++++++++++++++++
> > > 4 files changed, 688 insertions(+)
> > > create mode 100644 lib/raid6/sve.c
> > >
> > > diff --git a/include/linux/raid/pq.h b/include/linux/raid/pq.h
> > > index 2467b3be15c9..787cc57aea9d 100644
> > > --- a/include/linux/raid/pq.h
> > > +++ b/include/linux/raid/pq.h
> > > @@ -140,6 +140,9 @@ extern const struct raid6_calls raid6_neonx1;
> > > extern const struct raid6_calls raid6_neonx2;
> > > extern const struct raid6_calls raid6_neonx4;
> > > extern const struct raid6_calls raid6_neonx8;
> > > +extern const struct raid6_calls raid6_svex1;
> > > +extern const struct raid6_calls raid6_svex2;
> > > +extern const struct raid6_calls raid6_svex4;
> > >
> > > /* Algorithm list */
> > > extern const struct raid6_calls * const raid6_algos[];
> > > diff --git a/lib/raid6/Makefile b/lib/raid6/Makefile
> > > index 5be0a4e60ab1..6cdaa6f206fb 100644
> > > --- a/lib/raid6/Makefile
> > > +++ b/lib/raid6/Makefile
> > > @@ -8,6 +8,7 @@ raid6_pq-$(CONFIG_X86) += recov_ssse3.o recov_avx2.o
> > > mmx.o sse1.o sse2.o avx2.o
> > > raid6_pq-$(CONFIG_ALTIVEC) += altivec1.o altivec2.o altivec4.o
> > > altivec8.o \
> > > vpermxor1.o vpermxor2.o vpermxor4.o
> > > vpermxor8.o
> > > raid6_pq-$(CONFIG_KERNEL_MODE_NEON) += neon.o neon1.o neon2.o neon4.o
> > > neon8.o recov_neon.o recov_neon_inner.o
> > > +raid6_pq-$(CONFIG_ARM64_SVE) += sve.o
> > > raid6_pq-$(CONFIG_S390) += s390vx8.o recov_s390xc.o
> > > raid6_pq-$(CONFIG_LOONGARCH) += loongarch_simd.o recov_loongarch_simd.o
> > > raid6_pq-$(CONFIG_RISCV_ISA_V) += rvv.o recov_rvv.o
> > > @@ -67,6 +68,10 @@ CFLAGS_REMOVE_neon2.o += $(CC_FLAGS_NO_FPU)
> > > CFLAGS_REMOVE_neon4.o += $(CC_FLAGS_NO_FPU)
> > > CFLAGS_REMOVE_neon8.o += $(CC_FLAGS_NO_FPU)
> > > CFLAGS_REMOVE_recov_neon_inner.o += $(CC_FLAGS_NO_FPU)
> > > +
> > > +CFLAGS_sve.o += $(CC_FLAGS_FPU)
> > > +CFLAGS_REMOVE_sve.o += $(CC_FLAGS_NO_FPU)
> > > +
> > > targets += neon1.c neon2.c neon4.c neon8.c
> > > $(obj)/neon%.c: $(src)/neon.uc $(src)/unroll.awk FORCE
> > > $(call if_changed,unroll)
> > > diff --git a/lib/raid6/algos.c b/lib/raid6/algos.c
> > > index 799e0e5eac26..0ae73c3a4be3 100644
> > > --- a/lib/raid6/algos.c
> > > +++ b/lib/raid6/algos.c
> > > @@ -66,6 +66,11 @@ const struct raid6_calls * const raid6_algos[] = {
> > > &raid6_neonx2,
> > > &raid6_neonx1,
> > > #endif
> > > +#ifdef CONFIG_ARM64_SVE
> > > + &raid6_svex4,
> > > + &raid6_svex2,
> > > + &raid6_svex1,
> > > +#endif
> > > #ifdef CONFIG_LOONGARCH
> > > #ifdef CONFIG_CPU_HAS_LASX
> > > &raid6_lasx,
> > > diff --git a/lib/raid6/sve.c b/lib/raid6/sve.c
> > > new file mode 100644
> > > index 000000000000..d52937f806d4
> > > --- /dev/null
> > > +++ b/lib/raid6/sve.c
> > > @@ -0,0 +1,675 @@
> > > +// SPDX-License-Identifier: GPL-2.0-or-later
> > > +/*
> > > + * RAID-6 syndrome calculation using ARM SVE instructions
> > > + */
> > > +
> > > +#include <linux/raid/pq.h>
> > > +
> > > +#ifdef __KERNEL__
> > > +#include <asm/simd.h>
> > > +#include <linux/cpufeature.h>
> > > +#else
> > > +#define scoped_ksimd()
> > > +#define system_supports_sve() (1)
> > > +#endif
> > > +
> > > +static void raid6_sve1_gen_syndrome_real(int disks, unsigned long
> > > bytes, void **ptrs)
> > > +{
> > > + u8 **dptr = (u8 **)ptrs;
> > > + u8 *p, *q;
> > > + long z0 = disks - 3;
> > > +
> > > + p = dptr[z0 + 1];
> > > + q = dptr[z0 + 2];
> > > +
> > > + asm volatile(
> > > + ".arch armv8.2-a+sve\n"
> > > + "ptrue p0.b\n"
> > > + "cntb x3\n"
> > > + "mov w4, #0x1d\n"
> > > + "dup z4.b, w4\n"
> > > + "mov x5, #0\n"
> > > +
> > > + "0:\n"
> > > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > > + "ld1b z0.b, p0/z, [x6, x5]\n"
> > > + "mov z1.d, z0.d\n"
> > > +
> > > + "mov w7, %w[z0]\n"
> > > + "sub w7, w7, #1\n"
> > > +
> > > + "1:\n"
> > > + "cmp w7, #0\n"
> > > + "blt 2f\n"
> > > +
> > > + "mov z3.d, z1.d\n"
> > > + "asr z3.b, p0/m, z3.b, #7\n"
> > > + "lsl z1.b, p0/m, z1.b, #1\n"
> > > +
> > > + "and z3.d, z3.d, z4.d\n"
> > > + "eor z1.d, z1.d, z3.d\n"
> > > +
> > > + "sxtw x8, w7\n"
> > > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > > +
> > > + "eor z1.d, z1.d, z2.d\n"
> > > + "eor z0.d, z0.d, z2.d\n"
> > > +
> > > + "sub w7, w7, #1\n"
> > > + "b 1b\n"
> > > + "2:\n"
> > > +
> > > + "st1b z0.b, p0, [%[p], x5]\n"
> > > + "st1b z1.b, p0, [%[q], x5]\n"
> > > +
> > > + "add x5, x5, x3\n"
> > > + "cmp x5, %[bytes]\n"
> > > + "blt 0b\n"
> > > + :
> > > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > > + [p] "r" (p), [q] "r" (q)
> > > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > > + "z0", "z1", "z2", "z3", "z4"
> > > + );
> > > +}
> > > +
> > > +static void raid6_sve1_xor_syndrome_real(int disks, int start, int
> > > stop,
> > > + unsigned long bytes, void **ptrs)
> > > +{
> > > + u8 **dptr = (u8 **)ptrs;
> > > + u8 *p, *q;
> > > + long z0 = stop;
> > > +
> > > + p = dptr[disks - 2];
> > > + q = dptr[disks - 1];
> > > +
> > > + asm volatile(
> > > + ".arch armv8.2-a+sve\n"
> > > + "ptrue p0.b\n"
> > > + "cntb x3\n"
> > > + "mov w4, #0x1d\n"
> > > + "dup z4.b, w4\n"
> > > + "mov x5, #0\n"
> > > +
> > > + "0:\n"
> > > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > > + "ld1b z1.b, p0/z, [x6, x5]\n"
> > > + "ld1b z0.b, p0/z, [%[p], x5]\n"
> > > + "eor z0.d, z0.d, z1.d\n"
> > > +
> > > + "mov w7, %w[z0]\n"
> > > + "sub w7, w7, #1\n"
> > > +
> > > + "1:\n"
> > > + "cmp w7, %w[start]\n"
> > > + "blt 2f\n"
> > > +
> > > + "mov z3.d, z1.d\n"
> > > + "asr z3.b, p0/m, z3.b, #7\n"
> > > + "lsl z1.b, p0/m, z1.b, #1\n"
> > > + "and z3.d, z3.d, z4.d\n"
> > > + "eor z1.d, z1.d, z3.d\n"
> > > +
> > > + "sxtw x8, w7\n"
> > > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > > +
> > > + "eor z1.d, z1.d, z2.d\n"
> > > + "eor z0.d, z0.d, z2.d\n"
> > > +
> > > + "sub w7, w7, #1\n"
> > > + "b 1b\n"
> > > + "2:\n"
> > > +
> > > + "mov w7, %w[start]\n"
> > > + "sub w7, w7, #1\n"
> > > + "3:\n"
> > > + "cmp w7, #0\n"
> > > + "blt 4f\n"
> > > +
> > > + "mov z3.d, z1.d\n"
> > > + "asr z3.b, p0/m, z3.b, #7\n"
> > > + "lsl z1.b, p0/m, z1.b, #1\n"
> > > + "and z3.d, z3.d, z4.d\n"
> > > + "eor z1.d, z1.d, z3.d\n"
> > > +
> > > + "sub w7, w7, #1\n"
> > > + "b 3b\n"
> > > + "4:\n"
> > > +
> > > + "ld1b z2.b, p0/z, [%[q], x5]\n"
> > > + "eor z1.d, z1.d, z2.d\n"
> > > +
> > > + "st1b z0.b, p0, [%[p], x5]\n"
> > > + "st1b z1.b, p0, [%[q], x5]\n"
> > > +
> > > + "add x5, x5, x3\n"
> > > + "cmp x5, %[bytes]\n"
> > > + "blt 0b\n"
> > > + :
> > > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > > + [p] "r" (p), [q] "r" (q), [start] "r" (start)
> > > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > > + "z0", "z1", "z2", "z3", "z4"
> > > + );
> > > +}
> > > +
> > > +static void raid6_sve2_gen_syndrome_real(int disks, unsigned long
> > > bytes, void **ptrs)
> > > +{
> > > + u8 **dptr = (u8 **)ptrs;
> > > + u8 *p, *q;
> > > + long z0 = disks - 3;
> > > +
> > > + p = dptr[z0 + 1];
> > > + q = dptr[z0 + 2];
> > > +
> > > + asm volatile(
> > > + ".arch armv8.2-a+sve\n"
> > > + "ptrue p0.b\n"
> > > + "cntb x3\n"
> > > + "mov w4, #0x1d\n"
> > > + "dup z4.b, w4\n"
> > > + "mov x5, #0\n"
> > > +
> > > + "0:\n"
> > > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > > + "ld1b z0.b, p0/z, [x6, x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "ld1b z5.b, p0/z, [x6, x8]\n"
> > > + "mov z1.d, z0.d\n"
> > > + "mov z6.d, z5.d\n"
> > > +
> > > + "mov w7, %w[z0]\n"
> > > + "sub w7, w7, #1\n"
> > > +
> > > + "1:\n"
> > > + "cmp w7, #0\n"
> > > + "blt 2f\n"
> > > +
> > > + "mov z3.d, z1.d\n"
> > > + "asr z3.b, p0/m, z3.b, #7\n"
> > > + "lsl z1.b, p0/m, z1.b, #1\n"
> > > + "and z3.d, z3.d, z4.d\n"
> > > + "eor z1.d, z1.d, z3.d\n"
> > > +
> > > + "mov z8.d, z6.d\n"
> > > + "asr z8.b, p0/m, z8.b, #7\n"
> > > + "lsl z6.b, p0/m, z6.b, #1\n"
> > > + "and z8.d, z8.d, z4.d\n"
> > > + "eor z6.d, z6.d, z8.d\n"
> > > +
> > > + "sxtw x8, w7\n"
> > > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "ld1b z7.b, p0/z, [x6, x8]\n"
> > > +
> > > + "eor z1.d, z1.d, z2.d\n"
> > > + "eor z0.d, z0.d, z2.d\n"
> > > +
> > > + "eor z6.d, z6.d, z7.d\n"
> > > + "eor z5.d, z5.d, z7.d\n"
> > > +
> > > + "sub w7, w7, #1\n"
> > > + "b 1b\n"
> > > + "2:\n"
> > > +
> > > + "st1b z0.b, p0, [%[p], x5]\n"
> > > + "st1b z1.b, p0, [%[q], x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "st1b z5.b, p0, [%[p], x8]\n"
> > > + "st1b z6.b, p0, [%[q], x8]\n"
> > > +
> > > + "add x5, x5, x3\n"
> > > + "add x5, x5, x3\n"
> > > + "cmp x5, %[bytes]\n"
> > > + "blt 0b\n"
> > > + :
> > > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > > + [p] "r" (p), [q] "r" (q)
> > > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > > + "z0", "z1", "z2", "z3", "z4",
> > > + "z5", "z6", "z7", "z8"
> > > + );
> > > +}
> > > +
> > > +static void raid6_sve2_xor_syndrome_real(int disks, int start, int
> > > stop,
> > > + unsigned long bytes, void **ptrs)
> > > +{
> > > + u8 **dptr = (u8 **)ptrs;
> > > + u8 *p, *q;
> > > + long z0 = stop;
> > > +
> > > + p = dptr[disks - 2];
> > > + q = dptr[disks - 1];
> > > +
> > > + asm volatile(
> > > + ".arch armv8.2-a+sve\n"
> > > + "ptrue p0.b\n"
> > > + "cntb x3\n"
> > > + "mov w4, #0x1d\n"
> > > + "dup z4.b, w4\n"
> > > + "mov x5, #0\n"
> > > +
> > > + "0:\n"
> > > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > > + "ld1b z1.b, p0/z, [x6, x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "ld1b z6.b, p0/z, [x6, x8]\n"
> > > +
> > > + "ld1b z0.b, p0/z, [%[p], x5]\n"
> > > + "ld1b z5.b, p0/z, [%[p], x8]\n"
> > > +
> > > + "eor z0.d, z0.d, z1.d\n"
> > > + "eor z5.d, z5.d, z6.d\n"
> > > +
> > > + "mov w7, %w[z0]\n"
> > > + "sub w7, w7, #1\n"
> > > +
> > > + "1:\n"
> > > + "cmp w7, %w[start]\n"
> > > + "blt 2f\n"
> > > +
> > > + "mov z3.d, z1.d\n"
> > > + "asr z3.b, p0/m, z3.b, #7\n"
> > > + "lsl z1.b, p0/m, z1.b, #1\n"
> > > + "and z3.d, z3.d, z4.d\n"
> > > + "eor z1.d, z1.d, z3.d\n"
> > > +
> > > + "mov z8.d, z6.d\n"
> > > + "asr z8.b, p0/m, z8.b, #7\n"
> > > + "lsl z6.b, p0/m, z6.b, #1\n"
> > > + "and z8.d, z8.d, z4.d\n"
> > > + "eor z6.d, z6.d, z8.d\n"
> > > +
> > > + "sxtw x8, w7\n"
> > > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "ld1b z7.b, p0/z, [x6, x8]\n"
> > > +
> > > + "eor z1.d, z1.d, z2.d\n"
> > > + "eor z0.d, z0.d, z2.d\n"
> > > +
> > > + "eor z6.d, z6.d, z7.d\n"
> > > + "eor z5.d, z5.d, z7.d\n"
> > > +
> > > + "sub w7, w7, #1\n"
> > > + "b 1b\n"
> > > + "2:\n"
> > > +
> > > + "mov w7, %w[start]\n"
> > > + "sub w7, w7, #1\n"
> > > + "3:\n"
> > > + "cmp w7, #0\n"
> > > + "blt 4f\n"
> > > +
> > > + "mov z3.d, z1.d\n"
> > > + "asr z3.b, p0/m, z3.b, #7\n"
> > > + "lsl z1.b, p0/m, z1.b, #1\n"
> > > + "and z3.d, z3.d, z4.d\n"
> > > + "eor z1.d, z1.d, z3.d\n"
> > > +
> > > + "mov z8.d, z6.d\n"
> > > + "asr z8.b, p0/m, z8.b, #7\n"
> > > + "lsl z6.b, p0/m, z6.b, #1\n"
> > > + "and z8.d, z8.d, z4.d\n"
> > > + "eor z6.d, z6.d, z8.d\n"
> > > +
> > > + "sub w7, w7, #1\n"
> > > + "b 3b\n"
> > > + "4:\n"
> > > +
> > > + "ld1b z2.b, p0/z, [%[q], x5]\n"
> > > + "eor z1.d, z1.d, z2.d\n"
> > > + "st1b z0.b, p0, [%[p], x5]\n"
> > > + "st1b z1.b, p0, [%[q], x5]\n"
> > > +
> > > + "add x8, x5, x3\n"
> > > + "ld1b z7.b, p0/z, [%[q], x8]\n"
> > > + "eor z6.d, z6.d, z7.d\n"
> > > + "st1b z5.b, p0, [%[p], x8]\n"
> > > + "st1b z6.b, p0, [%[q], x8]\n"
> > > +
> > > + "add x5, x5, x3\n"
> > > + "add x5, x5, x3\n"
> > > + "cmp x5, %[bytes]\n"
> > > + "blt 0b\n"
> > > + :
> > > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > > + [p] "r" (p), [q] "r" (q), [start] "r" (start)
> > > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > > + "z0", "z1", "z2", "z3", "z4",
> > > + "z5", "z6", "z7", "z8"
> > > + );
> > > +}
> > > +
> > > +static void raid6_sve4_gen_syndrome_real(int disks, unsigned long
> > > bytes, void **ptrs)
> > > +{
> > > + u8 **dptr = (u8 **)ptrs;
> > > + u8 *p, *q;
> > > + long z0 = disks - 3;
> > > +
> > > + p = dptr[z0 + 1];
> > > + q = dptr[z0 + 2];
> > > +
> > > + asm volatile(
> > > + ".arch armv8.2-a+sve\n"
> > > + "ptrue p0.b\n"
> > > + "cntb x3\n"
> > > + "mov w4, #0x1d\n"
> > > + "dup z4.b, w4\n"
> > > + "mov x5, #0\n"
> > > +
> > > + "0:\n"
> > > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > > + "ld1b z0.b, p0/z, [x6, x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "ld1b z5.b, p0/z, [x6, x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z10.b, p0/z, [x6, x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z15.b, p0/z, [x6, x8]\n"
> > > +
> > > + "mov z1.d, z0.d\n"
> > > + "mov z6.d, z5.d\n"
> > > + "mov z11.d, z10.d\n"
> > > + "mov z16.d, z15.d\n"
> > > +
> > > + "mov w7, %w[z0]\n"
> > > + "sub w7, w7, #1\n"
> > > +
> > > + "1:\n"
> > > + "cmp w7, #0\n"
> > > + "blt 2f\n"
> > > +
> > > + // software pipelining: load data early
> > > + "sxtw x8, w7\n"
> > > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "ld1b z7.b, p0/z, [x6, x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z12.b, p0/z, [x6, x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z17.b, p0/z, [x6, x8]\n"
> > > +
> > > + // math block 1
> > > + "mov z3.d, z1.d\n"
> > > + "asr z3.b, p0/m, z3.b, #7\n"
> > > + "lsl z1.b, p0/m, z1.b, #1\n"
> > > + "and z3.d, z3.d, z4.d\n"
> > > + "eor z1.d, z1.d, z3.d\n"
> > > + "eor z1.d, z1.d, z2.d\n"
> > > + "eor z0.d, z0.d, z2.d\n"
> > > +
> > > + // math block 2
> > > + "mov z8.d, z6.d\n"
> > > + "asr z8.b, p0/m, z8.b, #7\n"
> > > + "lsl z6.b, p0/m, z6.b, #1\n"
> > > + "and z8.d, z8.d, z4.d\n"
> > > + "eor z6.d, z6.d, z8.d\n"
> > > + "eor z6.d, z6.d, z7.d\n"
> > > + "eor z5.d, z5.d, z7.d\n"
> > > +
> > > + // math block 3
> > > + "mov z13.d, z11.d\n"
> > > + "asr z13.b, p0/m, z13.b, #7\n"
> > > + "lsl z11.b, p0/m, z11.b, #1\n"
> > > + "and z13.d, z13.d, z4.d\n"
> > > + "eor z11.d, z11.d, z13.d\n"
> > > + "eor z11.d, z11.d, z12.d\n"
> > > + "eor z10.d, z10.d, z12.d\n"
> > > +
> > > + // math block 4
> > > + "mov z18.d, z16.d\n"
> > > + "asr z18.b, p0/m, z18.b, #7\n"
> > > + "lsl z16.b, p0/m, z16.b, #1\n"
> > > + "and z18.d, z18.d, z4.d\n"
> > > + "eor z16.d, z16.d, z18.d\n"
> > > + "eor z16.d, z16.d, z17.d\n"
> > > + "eor z15.d, z15.d, z17.d\n"
> > > +
> > > + "sub w7, w7, #1\n"
> > > + "b 1b\n"
> > > + "2:\n"
> > > +
> > > + "st1b z0.b, p0, [%[p], x5]\n"
> > > + "st1b z1.b, p0, [%[q], x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "st1b z5.b, p0, [%[p], x8]\n"
> > > + "st1b z6.b, p0, [%[q], x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "st1b z10.b, p0, [%[p], x8]\n"
> > > + "st1b z11.b, p0, [%[q], x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "st1b z15.b, p0, [%[p], x8]\n"
> > > + "st1b z16.b, p0, [%[q], x8]\n"
> > > +
> > > + "add x8, x3, x3\n"
> > > + "add x5, x5, x8, lsl #1\n"
> > > + "cmp x5, %[bytes]\n"
> > > + "blt 0b\n"
> > > + :
> > > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > > + [p] "r" (p), [q] "r" (q)
> > > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > > + "z0", "z1", "z2", "z3", "z4",
> > > + "z5", "z6", "z7", "z8",
> > > + "z10", "z11", "z12", "z13",
> > > + "z15", "z16", "z17", "z18"
> > > + );
> > > +}
> > > +
> > > +static void raid6_sve4_xor_syndrome_real(int disks, int start, int
> > > stop,
> > > + unsigned long bytes, void **ptrs)
> > > +{
> > > + u8 **dptr = (u8 **)ptrs;
> > > + u8 *p, *q;
> > > + long z0 = stop;
> > > +
> > > + p = dptr[disks - 2];
> > > + q = dptr[disks - 1];
> > > +
> > > + asm volatile(
> > > + ".arch armv8.2-a+sve\n"
> > > + "ptrue p0.b\n"
> > > + "cntb x3\n"
> > > + "mov w4, #0x1d\n"
> > > + "dup z4.b, w4\n"
> > > + "mov x5, #0\n"
> > > +
> > > + "0:\n"
> > > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > > + "ld1b z1.b, p0/z, [x6, x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "ld1b z6.b, p0/z, [x6, x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z11.b, p0/z, [x6, x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z16.b, p0/z, [x6, x8]\n"
> > > +
> > > + "ld1b z0.b, p0/z, [%[p], x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "ld1b z5.b, p0/z, [%[p], x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z10.b, p0/z, [%[p], x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z15.b, p0/z, [%[p], x8]\n"
> > > +
> > > + "eor z0.d, z0.d, z1.d\n"
> > > + "eor z5.d, z5.d, z6.d\n"
> > > + "eor z10.d, z10.d, z11.d\n"
> > > + "eor z15.d, z15.d, z16.d\n"
> > > +
> > > + "mov w7, %w[z0]\n"
> > > + "sub w7, w7, #1\n"
> > > +
> > > + "1:\n"
> > > + "cmp w7, %w[start]\n"
> > > + "blt 2f\n"
> > > +
> > > + // software pipelining: load data early
> > > + "sxtw x8, w7\n"
> > > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "ld1b z7.b, p0/z, [x6, x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z12.b, p0/z, [x6, x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z17.b, p0/z, [x6, x8]\n"
> > > +
> > > + // math block 1
> > > + "mov z3.d, z1.d\n"
> > > + "asr z3.b, p0/m, z3.b, #7\n"
> > > + "lsl z1.b, p0/m, z1.b, #1\n"
> > > + "and z3.d, z3.d, z4.d\n"
> > > + "eor z1.d, z1.d, z3.d\n"
> > > + "eor z1.d, z1.d, z2.d\n"
> > > + "eor z0.d, z0.d, z2.d\n"
> > > +
> > > + // math block 2
> > > + "mov z8.d, z6.d\n"
> > > + "asr z8.b, p0/m, z8.b, #7\n"
> > > + "lsl z6.b, p0/m, z6.b, #1\n"
> > > + "and z8.d, z8.d, z4.d\n"
> > > + "eor z6.d, z6.d, z8.d\n"
> > > + "eor z6.d, z6.d, z7.d\n"
> > > + "eor z5.d, z5.d, z7.d\n"
> > > +
> > > + // math block 3
> > > + "mov z13.d, z11.d\n"
> > > + "asr z13.b, p0/m, z13.b, #7\n"
> > > + "lsl z11.b, p0/m, z11.b, #1\n"
> > > + "and z13.d, z13.d, z4.d\n"
> > > + "eor z11.d, z11.d, z13.d\n"
> > > + "eor z11.d, z11.d, z12.d\n"
> > > + "eor z10.d, z10.d, z12.d\n"
> > > +
> > > + // math block 4
> > > + "mov z18.d, z16.d\n"
> > > + "asr z18.b, p0/m, z18.b, #7\n"
> > > + "lsl z16.b, p0/m, z16.b, #1\n"
> > > + "and z18.d, z18.d, z4.d\n"
> > > + "eor z16.d, z16.d, z18.d\n"
> > > + "eor z16.d, z16.d, z17.d\n"
> > > + "eor z15.d, z15.d, z17.d\n"
> > > +
> > > + "sub w7, w7, #1\n"
> > > + "b 1b\n"
> > > + "2:\n"
> > > +
> > > + "mov w7, %w[start]\n"
> > > + "sub w7, w7, #1\n"
> > > + "3:\n"
> > > + "cmp w7, #0\n"
> > > + "blt 4f\n"
> > > +
> > > + // math block 1
> > > + "mov z3.d, z1.d\n"
> > > + "asr z3.b, p0/m, z3.b, #7\n"
> > > + "lsl z1.b, p0/m, z1.b, #1\n"
> > > + "and z3.d, z3.d, z4.d\n"
> > > + "eor z1.d, z1.d, z3.d\n"
> > > +
> > > + // math block 2
> > > + "mov z8.d, z6.d\n"
> > > + "asr z8.b, p0/m, z8.b, #7\n"
> > > + "lsl z6.b, p0/m, z6.b, #1\n"
> > > + "and z8.d, z8.d, z4.d\n"
> > > + "eor z6.d, z6.d, z8.d\n"
> > > +
> > > + // math block 3
> > > + "mov z13.d, z11.d\n"
> > > + "asr z13.b, p0/m, z13.b, #7\n"
> > > + "lsl z11.b, p0/m, z11.b, #1\n"
> > > + "and z13.d, z13.d, z4.d\n"
> > > + "eor z11.d, z11.d, z13.d\n"
> > > +
> > > + // math block 4
> > > + "mov z18.d, z16.d\n"
> > > + "asr z18.b, p0/m, z18.b, #7\n"
> > > + "lsl z16.b, p0/m, z16.b, #1\n"
> > > + "and z18.d, z18.d, z4.d\n"
> > > + "eor z16.d, z16.d, z18.d\n"
> > > +
> > > + "sub w7, w7, #1\n"
> > > + "b 3b\n"
> > > + "4:\n"
> > > +
> > > + // Load q and XOR
> > > + "ld1b z2.b, p0/z, [%[q], x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "ld1b z7.b, p0/z, [%[q], x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z12.b, p0/z, [%[q], x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "ld1b z17.b, p0/z, [%[q], x8]\n"
> > > +
> > > + "eor z1.d, z1.d, z2.d\n"
> > > + "eor z6.d, z6.d, z7.d\n"
> > > + "eor z11.d, z11.d, z12.d\n"
> > > + "eor z16.d, z16.d, z17.d\n"
> > > +
> > > + // Store results
> > > + "st1b z0.b, p0, [%[p], x5]\n"
> > > + "st1b z1.b, p0, [%[q], x5]\n"
> > > + "add x8, x5, x3\n"
> > > + "st1b z5.b, p0, [%[p], x8]\n"
> > > + "st1b z6.b, p0, [%[q], x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "st1b z10.b, p0, [%[p], x8]\n"
> > > + "st1b z11.b, p0, [%[q], x8]\n"
> > > + "add x8, x8, x3\n"
> > > + "st1b z15.b, p0, [%[p], x8]\n"
> > > + "st1b z16.b, p0, [%[q], x8]\n"
> > > +
> > > + "add x8, x3, x3\n"
> > > + "add x5, x5, x8, lsl #1\n"
> > > + "cmp x5, %[bytes]\n"
> > > + "blt 0b\n"
> > > + :
> > > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > > + [p] "r" (p), [q] "r" (q), [start] "r" (start)
> > > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > > + "z0", "z1", "z2", "z3", "z4",
> > > + "z5", "z6", "z7", "z8",
> > > + "z10", "z11", "z12", "z13",
> > > + "z15", "z16", "z17", "z18"
> > > + );
> > > +}
> > > +
> > > +#define RAID6_SVE_WRAPPER(_n) \
> > > + static void raid6_sve ## _n ## _gen_syndrome(int disks, \
> > > + size_t bytes, void **ptrs) \
> > > + { \
> > > + scoped_ksimd() \
> > > + raid6_sve ## _n ## _gen_syndrome_real(disks, \
> > > + (unsigned long)bytes, ptrs); \
> > > + } \
> > > + static void raid6_sve ## _n ## _xor_syndrome(int disks, \
> > > + int start, int stop, \
> > > + size_t bytes, void **ptrs) \
> > > + { \
> > > + scoped_ksimd() \
> > > + raid6_sve ## _n ## _xor_syndrome_real(disks, \
> > > + start, stop, (unsigned long)bytes, ptrs);\
> > > + } \
> > > + struct raid6_calls const raid6_svex ## _n = { \
> > > + raid6_sve ## _n ## _gen_syndrome, \
> > > + raid6_sve ## _n ## _xor_syndrome, \
> > > + raid6_have_sve, \
> > > + "svex" #_n, \
> > > + 0 \
> > > + }
> > > +
> > > +static int raid6_have_sve(void)
> > > +{
> > > + return system_supports_sve();
> > > +}
> > > +
> > > +RAID6_SVE_WRAPPER(1);
> > > +RAID6_SVE_WRAPPER(2);
> > > +RAID6_SVE_WRAPPER(4);
> > > --
> > > 2.43.0
^ permalink raw reply
* [PATCH] md: remove unused static md_wq workqueue
From: Abd-Alrhman Masalkhi @ 2026-03-28 19:35 UTC (permalink / raw)
To: song, yukuai, linux-raid, linux-kernel; +Cc: Abd-Alrhman Masalkhi
The md_wq workqueue is defined as static and initialized in md_init(),
but it is not used anywhere within md.c.
All asynchronous and deferred work in this file is handled via
md_misc_wq or dedicated md threads.
Fixes: b75197e86e6d3 ("md: Remove flush handling")
Signed-off-by: Abd-Alrhman Masalkhi <abd.masalkhi@gmail.com>
---
drivers/md/md.c | 8 --------
1 file changed, 8 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index ac71640ff3a8..64bf08f16c0e 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -84,7 +84,6 @@ static DEFINE_XARRAY(md_submodule);
static const struct kobj_type md_ktype;
static DECLARE_WAIT_QUEUE_HEAD(resync_wait);
-static struct workqueue_struct *md_wq;
/*
* This workqueue is used for sync_work to register new sync_thread, and for
@@ -10495,10 +10494,6 @@ static int __init md_init(void)
goto err_bitmap;
ret = -ENOMEM;
- md_wq = alloc_workqueue("md", WQ_MEM_RECLAIM, 0);
- if (!md_wq)
- goto err_wq;
-
md_misc_wq = alloc_workqueue("md_misc", 0, 0);
if (!md_misc_wq)
goto err_misc_wq;
@@ -10523,8 +10518,6 @@ static int __init md_init(void)
err_md:
destroy_workqueue(md_misc_wq);
err_misc_wq:
- destroy_workqueue(md_wq);
-err_wq:
md_llbitmap_exit();
err_bitmap:
md_bitmap_exit();
@@ -10833,7 +10826,6 @@ static __exit void md_exit(void)
spin_unlock(&all_mddevs_lock);
destroy_workqueue(md_misc_wq);
- destroy_workqueue(md_wq);
md_bitmap_exit();
}
--
2.51.0
^ permalink raw reply related
* Re: [PATCH 23/28] xor: add a better public API
From: Andrew Morton @ 2026-03-27 17:51 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-24-hch@lst.de>
On Fri, 27 Mar 2026 07:16:55 +0100 Christoph Hellwig <hch@lst.de> wrote:
> xor_blocks is very annoying to use, because it is limited to 4 + 1
> sources / destinations, has an odd argument order and is completely
> undocumented.
>
> Lift the code that loops around it from btrfs and async_tx/async_xor into
> common code under the name xor_gen and properly document it.
Something funny here - two different patches with the same title and
changelog.
If you can send over new versions of title&changelog I can paste that in.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox