* [PATCH 18/28] s390: move the XOR code to lib/raid/
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Move the optimized XOR into lib/raid and include it it in xor.ko
instead of unconditionally building it into the main kernel image.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Heiko Carstens <hca@linux.ibm.com>
---
arch/s390/lib/Makefile | 2 +-
lib/raid/xor/Makefile | 1 +
{arch/s390/lib => lib/raid/xor/s390}/xor.c | 2 --
3 files changed, 2 insertions(+), 3 deletions(-)
rename {arch/s390/lib => lib/raid/xor/s390}/xor.c (98%)
diff --git a/arch/s390/lib/Makefile b/arch/s390/lib/Makefile
index f43f897d3fc0..2bf47204f6ab 100644
--- a/arch/s390/lib/Makefile
+++ b/arch/s390/lib/Makefile
@@ -5,7 +5,7 @@
lib-y += delay.o string.o uaccess.o find.o spinlock.o tishift.o
lib-y += csum-partial.o
-obj-y += mem.o xor.o
+obj-y += mem.o
lib-$(CONFIG_KPROBES) += probes.o
lib-$(CONFIG_UPROBES) += probes.o
obj-$(CONFIG_S390_KPROBES_SANITY_TEST) += test_kprobes_s390.o
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index 3a7c887d08ee..3db6c2b2f26a 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -20,6 +20,7 @@ xor-$(CONFIG_ALTIVEC) += powerpc/xor_vmx.o powerpc/xor_vmx_glue.o
xor-$(CONFIG_RISCV_ISA_V) += riscv/xor.o riscv/xor-glue.o
xor-$(CONFIG_SPARC32) += sparc/xor-sparc32.o
xor-$(CONFIG_SPARC64) += sparc/xor-sparc64.o sparc/xor-sparc64-glue.o
+xor-$(CONFIG_S390) += s390/xor.o
CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU)
diff --git a/arch/s390/lib/xor.c b/lib/raid/xor/s390/xor.c
similarity index 98%
rename from arch/s390/lib/xor.c
rename to lib/raid/xor/s390/xor.c
index 3bbe21b40e66..acbd268adfc8 100644
--- a/arch/s390/lib/xor.c
+++ b/lib/raid/xor/s390/xor.c
@@ -7,7 +7,6 @@
*/
#include <linux/types.h>
-#include <linux/export.h>
#include <linux/raid/xor_impl.h>
#include <asm/xor.h>
@@ -133,4 +132,3 @@ struct xor_block_template xor_block_xc = {
.do_4 = xor_xc_4,
.do_5 = xor_xc_5,
};
-EXPORT_SYMBOL(xor_block_xc);
--
2.47.3
^ permalink raw reply related
* [PATCH 17/28] sparc: move the XOR code to lib/raid/
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Move the optimized XOR into lib/raid and include it it in xor.ko
instead of always building it into the main kernel image.
The code should probably be split into separate files for the two
implementations, but for now this just does the trivial move.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
arch/sparc/include/asm/asm-prototypes.h | 1 -
arch/sparc/include/asm/xor.h | 45 ++++++++++++++++---
arch/sparc/lib/Makefile | 2 +-
lib/raid/xor/Makefile | 2 +
.../raid/xor/sparc/xor-sparc32.c | 23 ++--------
.../raid/xor/sparc/xor-sparc64-glue.c | 26 +++--------
.../xor.S => lib/raid/xor/sparc/xor-sparc64.S | 10 -----
7 files changed, 52 insertions(+), 57 deletions(-)
rename arch/sparc/include/asm/xor_32.h => lib/raid/xor/sparc/xor-sparc32.c (93%)
rename arch/sparc/include/asm/xor_64.h => lib/raid/xor/sparc/xor-sparc64-glue.c (74%)
rename arch/sparc/lib/xor.S => lib/raid/xor/sparc/xor-sparc64.S (98%)
diff --git a/arch/sparc/include/asm/asm-prototypes.h b/arch/sparc/include/asm/asm-prototypes.h
index 08810808ca6d..bbd1a8afaabf 100644
--- a/arch/sparc/include/asm/asm-prototypes.h
+++ b/arch/sparc/include/asm/asm-prototypes.h
@@ -14,7 +14,6 @@
#include <asm/oplib.h>
#include <asm/pgtable.h>
#include <asm/trap_block.h>
-#include <asm/xor.h>
void *__memscan_zero(void *, size_t);
void *__memscan_generic(void *, int, size_t);
diff --git a/arch/sparc/include/asm/xor.h b/arch/sparc/include/asm/xor.h
index f4c651e203c4..f923b009fc24 100644
--- a/arch/sparc/include/asm/xor.h
+++ b/arch/sparc/include/asm/xor.h
@@ -1,9 +1,44 @@
/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 1997, 1999 Jakub Jelinek (jj@ultra.linux.cz)
+ * Copyright (C) 2006 David S. Miller <davem@davemloft.net>
+ */
#ifndef ___ASM_SPARC_XOR_H
#define ___ASM_SPARC_XOR_H
+
#if defined(__sparc__) && defined(__arch64__)
-#include <asm/xor_64.h>
-#else
-#include <asm/xor_32.h>
-#endif
-#endif
+#include <asm/spitfire.h>
+
+extern struct xor_block_template xor_block_VIS;
+extern struct xor_block_template xor_block_niagara;
+
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ /* Force VIS for everything except Niagara. */
+ if (tlb_type == hypervisor &&
+ (sun4v_chip_type == SUN4V_CHIP_NIAGARA1 ||
+ sun4v_chip_type == SUN4V_CHIP_NIAGARA2 ||
+ sun4v_chip_type == SUN4V_CHIP_NIAGARA3 ||
+ sun4v_chip_type == SUN4V_CHIP_NIAGARA4 ||
+ sun4v_chip_type == SUN4V_CHIP_NIAGARA5))
+ xor_force(&xor_block_niagara);
+ else
+ xor_force(&xor_block_VIS);
+}
+#else /* sparc64 */
+
+/* For grins, also test the generic routines. */
+#include <asm-generic/xor.h>
+
+extern struct xor_block_template xor_block_SPARC;
+
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ xor_register(&xor_block_8regs);
+ xor_register(&xor_block_32regs);
+ xor_register(&xor_block_SPARC);
+}
+#endif /* !sparc64 */
+#endif /* ___ASM_SPARC_XOR_H */
diff --git a/arch/sparc/lib/Makefile b/arch/sparc/lib/Makefile
index 783bdec0d7be..dd10cdd6f062 100644
--- a/arch/sparc/lib/Makefile
+++ b/arch/sparc/lib/Makefile
@@ -48,7 +48,7 @@ lib-$(CONFIG_SPARC64) += GENmemcpy.o GENcopy_from_user.o GENcopy_to_user.o
lib-$(CONFIG_SPARC64) += GENpatch.o GENpage.o GENbzero.o
lib-$(CONFIG_SPARC64) += copy_in_user.o memmove.o
-lib-$(CONFIG_SPARC64) += mcount.o ipcsum.o xor.o hweight.o ffs.o
+lib-$(CONFIG_SPARC64) += mcount.o ipcsum.o hweight.o ffs.o
obj-$(CONFIG_SPARC64) += iomap.o
obj-$(CONFIG_SPARC32) += atomic32.o
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index 9e729b50e775..3a7c887d08ee 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -18,6 +18,8 @@ xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd_glue.o
xor-$(CONFIG_ALTIVEC) += powerpc/xor_vmx.o powerpc/xor_vmx_glue.o
xor-$(CONFIG_RISCV_ISA_V) += riscv/xor.o riscv/xor-glue.o
+xor-$(CONFIG_SPARC32) += sparc/xor-sparc32.o
+xor-$(CONFIG_SPARC64) += sparc/xor-sparc64.o sparc/xor-sparc64-glue.o
CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU)
diff --git a/arch/sparc/include/asm/xor_32.h b/lib/raid/xor/sparc/xor-sparc32.c
similarity index 93%
rename from arch/sparc/include/asm/xor_32.h
rename to lib/raid/xor/sparc/xor-sparc32.c
index 8fbf0c07ec28..b65a75a6e59d 100644
--- a/arch/sparc/include/asm/xor_32.h
+++ b/lib/raid/xor/sparc/xor-sparc32.c
@@ -1,16 +1,12 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * include/asm/xor.h
- *
- * Optimized RAID-5 checksumming functions for 32-bit Sparc.
- */
-
+// SPDX-License-Identifier: GPL-2.0-or-later
/*
* High speed xor_block operation for RAID4/5 utilizing the
* ldd/std SPARC instructions.
*
* Copyright (C) 1999 Jakub Jelinek (jj@ultra.linux.cz)
*/
+#include <linux/raid/xor_impl.h>
+#include <asm/xor.h>
static void
sparc_2(unsigned long bytes, unsigned long * __restrict p1,
@@ -248,21 +244,10 @@ sparc_5(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
-static struct xor_block_template xor_block_SPARC = {
+struct xor_block_template xor_block_SPARC = {
.name = "SPARC",
.do_2 = sparc_2,
.do_3 = sparc_3,
.do_4 = sparc_4,
.do_5 = sparc_5,
};
-
-/* For grins, also test the generic routines. */
-#include <asm-generic/xor.h>
-
-#define arch_xor_init arch_xor_init
-static __always_inline void __init arch_xor_init(void)
-{
- xor_register(&xor_block_8regs);
- xor_register(&xor_block_32regs);
- xor_register(&xor_block_SPARC);
-}
diff --git a/arch/sparc/include/asm/xor_64.h b/lib/raid/xor/sparc/xor-sparc64-glue.c
similarity index 74%
rename from arch/sparc/include/asm/xor_64.h
rename to lib/raid/xor/sparc/xor-sparc64-glue.c
index e0482ecc0a68..3c67c8c3a0e8 100644
--- a/arch/sparc/include/asm/xor_64.h
+++ b/lib/raid/xor/sparc/xor-sparc64-glue.c
@@ -1,7 +1,5 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
+// SPDX-License-Identifier: GPL-2.0-or-later
/*
- * include/asm/xor.h
- *
* High speed xor_block operation for RAID4/5 utilizing the
* UltraSparc Visual Instruction Set and Niagara block-init
* twin-load instructions.
@@ -10,7 +8,8 @@
* Copyright (C) 2006 David S. Miller <davem@davemloft.net>
*/
-#include <asm/spitfire.h>
+#include <linux/raid/xor_impl.h>
+#include <asm/xor.h>
void xor_vis_2(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p2);
@@ -29,7 +28,7 @@ void xor_vis_5(unsigned long bytes, unsigned long * __restrict p1,
/* XXX Ugh, write cheetah versions... -DaveM */
-static struct xor_block_template xor_block_VIS = {
+struct xor_block_template xor_block_VIS = {
.name = "VIS",
.do_2 = xor_vis_2,
.do_3 = xor_vis_3,
@@ -52,25 +51,10 @@ void xor_niagara_5(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p4,
const unsigned long * __restrict p5);
-static struct xor_block_template xor_block_niagara = {
+struct xor_block_template xor_block_niagara = {
.name = "Niagara",
.do_2 = xor_niagara_2,
.do_3 = xor_niagara_3,
.do_4 = xor_niagara_4,
.do_5 = xor_niagara_5,
};
-
-#define arch_xor_init arch_xor_init
-static __always_inline void __init arch_xor_init(void)
-{
- /* Force VIS for everything except Niagara. */
- if (tlb_type == hypervisor &&
- (sun4v_chip_type == SUN4V_CHIP_NIAGARA1 ||
- sun4v_chip_type == SUN4V_CHIP_NIAGARA2 ||
- sun4v_chip_type == SUN4V_CHIP_NIAGARA3 ||
- sun4v_chip_type == SUN4V_CHIP_NIAGARA4 ||
- sun4v_chip_type == SUN4V_CHIP_NIAGARA5))
- xor_force(&xor_block_niagara);
- else
- xor_force(&xor_block_VIS);
-}
diff --git a/arch/sparc/lib/xor.S b/lib/raid/xor/sparc/xor-sparc64.S
similarity index 98%
rename from arch/sparc/lib/xor.S
rename to lib/raid/xor/sparc/xor-sparc64.S
index 35461e3b2a9b..a7b74d473bd4 100644
--- a/arch/sparc/lib/xor.S
+++ b/lib/raid/xor/sparc/xor-sparc64.S
@@ -1,7 +1,5 @@
/* SPDX-License-Identifier: GPL-2.0 */
/*
- * arch/sparc64/lib/xor.S
- *
* High speed xor_block operation for RAID4/5 utilizing the
* UltraSparc Visual Instruction Set and Niagara store-init/twin-load.
*
@@ -92,7 +90,6 @@ ENTRY(xor_vis_2)
retl
wr %g0, 0, %fprs
ENDPROC(xor_vis_2)
-EXPORT_SYMBOL(xor_vis_2)
ENTRY(xor_vis_3)
rd %fprs, %o5
@@ -159,7 +156,6 @@ ENTRY(xor_vis_3)
retl
wr %g0, 0, %fprs
ENDPROC(xor_vis_3)
-EXPORT_SYMBOL(xor_vis_3)
ENTRY(xor_vis_4)
rd %fprs, %o5
@@ -245,7 +241,6 @@ ENTRY(xor_vis_4)
retl
wr %g0, 0, %fprs
ENDPROC(xor_vis_4)
-EXPORT_SYMBOL(xor_vis_4)
ENTRY(xor_vis_5)
save %sp, -192, %sp
@@ -352,7 +347,6 @@ ENTRY(xor_vis_5)
ret
restore
ENDPROC(xor_vis_5)
-EXPORT_SYMBOL(xor_vis_5)
/* Niagara versions. */
ENTRY(xor_niagara_2) /* %o0=bytes, %o1=dest, %o2=src */
@@ -399,7 +393,6 @@ ENTRY(xor_niagara_2) /* %o0=bytes, %o1=dest, %o2=src */
ret
restore
ENDPROC(xor_niagara_2)
-EXPORT_SYMBOL(xor_niagara_2)
ENTRY(xor_niagara_3) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2 */
save %sp, -192, %sp
@@ -461,7 +454,6 @@ ENTRY(xor_niagara_3) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2 */
ret
restore
ENDPROC(xor_niagara_3)
-EXPORT_SYMBOL(xor_niagara_3)
ENTRY(xor_niagara_4) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2, %o4=src3 */
save %sp, -192, %sp
@@ -544,7 +536,6 @@ ENTRY(xor_niagara_4) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2, %o4=src3 */
ret
restore
ENDPROC(xor_niagara_4)
-EXPORT_SYMBOL(xor_niagara_4)
ENTRY(xor_niagara_5) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2, %o4=src3, %o5=src4 */
save %sp, -192, %sp
@@ -643,4 +634,3 @@ ENTRY(xor_niagara_5) /* %o0=bytes, %o1=dest, %o2=src1, %o3=src2, %o4=src3, %o5=s
ret
restore
ENDPROC(xor_niagara_5)
-EXPORT_SYMBOL(xor_niagara_5)
--
2.47.3
^ permalink raw reply related
* [PATCH 16/28] riscv: move the XOR code to lib/raid/
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Move the optimized XOR into lib/raid and include it it in xor.ko
instead of always building it into the main kernel image.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
arch/riscv/include/asm/xor.h | 54 +------------------
arch/riscv/lib/Makefile | 1 -
lib/raid/xor/Makefile | 1 +
lib/raid/xor/riscv/xor-glue.c | 56 ++++++++++++++++++++
{arch/riscv/lib => lib/raid/xor/riscv}/xor.S | 4 --
5 files changed, 59 insertions(+), 57 deletions(-)
create mode 100644 lib/raid/xor/riscv/xor-glue.c
rename {arch/riscv/lib => lib/raid/xor/riscv}/xor.S (92%)
diff --git a/arch/riscv/include/asm/xor.h b/arch/riscv/include/asm/xor.h
index ed5f27903efc..614d9209d078 100644
--- a/arch/riscv/include/asm/xor.h
+++ b/arch/riscv/include/asm/xor.h
@@ -2,60 +2,10 @@
/*
* Copyright (C) 2021 SiFive
*/
-
-#include <linux/hardirq.h>
-#include <asm-generic/xor.h>
-#ifdef CONFIG_RISCV_ISA_V
#include <asm/vector.h>
-#include <asm/switch_to.h>
-#include <asm/asm-prototypes.h>
-
-static void xor_vector_2(unsigned long bytes, unsigned long *__restrict p1,
- const unsigned long *__restrict p2)
-{
- kernel_vector_begin();
- xor_regs_2_(bytes, p1, p2);
- kernel_vector_end();
-}
-
-static void xor_vector_3(unsigned long bytes, unsigned long *__restrict p1,
- const unsigned long *__restrict p2,
- const unsigned long *__restrict p3)
-{
- kernel_vector_begin();
- xor_regs_3_(bytes, p1, p2, p3);
- kernel_vector_end();
-}
-
-static void xor_vector_4(unsigned long bytes, unsigned long *__restrict p1,
- const unsigned long *__restrict p2,
- const unsigned long *__restrict p3,
- const unsigned long *__restrict p4)
-{
- kernel_vector_begin();
- xor_regs_4_(bytes, p1, p2, p3, p4);
- kernel_vector_end();
-}
-
-static void xor_vector_5(unsigned long bytes, unsigned long *__restrict p1,
- const unsigned long *__restrict p2,
- const unsigned long *__restrict p3,
- const unsigned long *__restrict p4,
- const unsigned long *__restrict p5)
-{
- kernel_vector_begin();
- xor_regs_5_(bytes, p1, p2, p3, p4, p5);
- kernel_vector_end();
-}
+#include <asm-generic/xor.h>
-static struct xor_block_template xor_block_rvv = {
- .name = "rvv",
- .do_2 = xor_vector_2,
- .do_3 = xor_vector_3,
- .do_4 = xor_vector_4,
- .do_5 = xor_vector_5
-};
-#endif /* CONFIG_RISCV_ISA_V */
+extern struct xor_block_template xor_block_rvv;
#define arch_xor_init arch_xor_init
static __always_inline void __init arch_xor_init(void)
diff --git a/arch/riscv/lib/Makefile b/arch/riscv/lib/Makefile
index bbc031124974..e220c35764eb 100644
--- a/arch/riscv/lib/Makefile
+++ b/arch/riscv/lib/Makefile
@@ -16,5 +16,4 @@ lib-$(CONFIG_MMU) += uaccess.o
lib-$(CONFIG_64BIT) += tishift.o
lib-$(CONFIG_RISCV_ISA_ZICBOZ) += clear_page.o
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
-lib-$(CONFIG_RISCV_ISA_V) += xor.o
lib-$(CONFIG_RISCV_ISA_V) += riscv_v_helpers.o
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index 006b44ce46bf..9e729b50e775 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -17,6 +17,7 @@ xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd_glue.o
xor-$(CONFIG_ALTIVEC) += powerpc/xor_vmx.o powerpc/xor_vmx_glue.o
+xor-$(CONFIG_RISCV_ISA_V) += riscv/xor.o riscv/xor-glue.o
CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU)
diff --git a/lib/raid/xor/riscv/xor-glue.c b/lib/raid/xor/riscv/xor-glue.c
new file mode 100644
index 000000000000..11666a4b6b68
--- /dev/null
+++ b/lib/raid/xor/riscv/xor-glue.c
@@ -0,0 +1,56 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Copyright (C) 2021 SiFive
+ */
+
+#include <linux/raid/xor_impl.h>
+#include <asm/vector.h>
+#include <asm/switch_to.h>
+#include <asm/asm-prototypes.h>
+#include <asm/xor.h>
+
+static void xor_vector_2(unsigned long bytes, unsigned long *__restrict p1,
+ const unsigned long *__restrict p2)
+{
+ kernel_vector_begin();
+ xor_regs_2_(bytes, p1, p2);
+ kernel_vector_end();
+}
+
+static void xor_vector_3(unsigned long bytes, unsigned long *__restrict p1,
+ const unsigned long *__restrict p2,
+ const unsigned long *__restrict p3)
+{
+ kernel_vector_begin();
+ xor_regs_3_(bytes, p1, p2, p3);
+ kernel_vector_end();
+}
+
+static void xor_vector_4(unsigned long bytes, unsigned long *__restrict p1,
+ const unsigned long *__restrict p2,
+ const unsigned long *__restrict p3,
+ const unsigned long *__restrict p4)
+{
+ kernel_vector_begin();
+ xor_regs_4_(bytes, p1, p2, p3, p4);
+ kernel_vector_end();
+}
+
+static void xor_vector_5(unsigned long bytes, unsigned long *__restrict p1,
+ const unsigned long *__restrict p2,
+ const unsigned long *__restrict p3,
+ const unsigned long *__restrict p4,
+ const unsigned long *__restrict p5)
+{
+ kernel_vector_begin();
+ xor_regs_5_(bytes, p1, p2, p3, p4, p5);
+ kernel_vector_end();
+}
+
+struct xor_block_template xor_block_rvv = {
+ .name = "rvv",
+ .do_2 = xor_vector_2,
+ .do_3 = xor_vector_3,
+ .do_4 = xor_vector_4,
+ .do_5 = xor_vector_5
+};
diff --git a/arch/riscv/lib/xor.S b/lib/raid/xor/riscv/xor.S
similarity index 92%
rename from arch/riscv/lib/xor.S
rename to lib/raid/xor/riscv/xor.S
index b28f2430e52f..56fb7fc1e2cd 100644
--- a/arch/riscv/lib/xor.S
+++ b/lib/raid/xor/riscv/xor.S
@@ -18,7 +18,6 @@ SYM_FUNC_START(xor_regs_2_)
bnez a0, xor_regs_2_
ret
SYM_FUNC_END(xor_regs_2_)
-EXPORT_SYMBOL(xor_regs_2_)
SYM_FUNC_START(xor_regs_3_)
vsetvli a4, a0, e8, m8, ta, ma
@@ -35,7 +34,6 @@ SYM_FUNC_START(xor_regs_3_)
bnez a0, xor_regs_3_
ret
SYM_FUNC_END(xor_regs_3_)
-EXPORT_SYMBOL(xor_regs_3_)
SYM_FUNC_START(xor_regs_4_)
vsetvli a5, a0, e8, m8, ta, ma
@@ -55,7 +53,6 @@ SYM_FUNC_START(xor_regs_4_)
bnez a0, xor_regs_4_
ret
SYM_FUNC_END(xor_regs_4_)
-EXPORT_SYMBOL(xor_regs_4_)
SYM_FUNC_START(xor_regs_5_)
vsetvli a6, a0, e8, m8, ta, ma
@@ -78,4 +75,3 @@ SYM_FUNC_START(xor_regs_5_)
bnez a0, xor_regs_5_
ret
SYM_FUNC_END(xor_regs_5_)
-EXPORT_SYMBOL(xor_regs_5_)
--
2.47.3
^ permalink raw reply related
* [PATCH 15/28] powerpc: move the XOR code to lib/raid/
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Move the optimized XOR into lib/raid and include it it in xor.ko
instead of always building it into the main kernel image.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
arch/powerpc/include/asm/xor.h | 17 +----
arch/powerpc/include/asm/xor_altivec.h | 22 ------
arch/powerpc/lib/Makefile | 5 --
arch/powerpc/lib/xor_vmx_glue.c | 63 -----------------
lib/raid/xor/Makefile | 5 ++
.../lib => lib/raid/xor/powerpc}/xor_vmx.c | 0
.../lib => lib/raid/xor/powerpc}/xor_vmx.h | 0
lib/raid/xor/powerpc/xor_vmx_glue.c | 67 +++++++++++++++++++
8 files changed, 74 insertions(+), 105 deletions(-)
delete mode 100644 arch/powerpc/include/asm/xor_altivec.h
delete mode 100644 arch/powerpc/lib/xor_vmx_glue.c
rename {arch/powerpc/lib => lib/raid/xor/powerpc}/xor_vmx.c (100%)
rename {arch/powerpc/lib => lib/raid/xor/powerpc}/xor_vmx.h (100%)
create mode 100644 lib/raid/xor/powerpc/xor_vmx_glue.c
diff --git a/arch/powerpc/include/asm/xor.h b/arch/powerpc/include/asm/xor.h
index 30224c5279c4..3293ac87181c 100644
--- a/arch/powerpc/include/asm/xor.h
+++ b/arch/powerpc/include/asm/xor.h
@@ -8,24 +8,11 @@
#ifndef _ASM_POWERPC_XOR_H
#define _ASM_POWERPC_XOR_H
-#ifdef CONFIG_ALTIVEC
-
-#include <asm/cputable.h>
#include <asm/cpu_has_feature.h>
-#include <asm/xor_altivec.h>
-
-static struct xor_block_template xor_block_altivec = {
- .name = "altivec",
- .do_2 = xor_altivec_2,
- .do_3 = xor_altivec_3,
- .do_4 = xor_altivec_4,
- .do_5 = xor_altivec_5,
-};
-#endif /* CONFIG_ALTIVEC */
-
-/* Also try the generic routines. */
#include <asm-generic/xor.h>
+extern struct xor_block_template xor_block_altivec;
+
#define arch_xor_init arch_xor_init
static __always_inline void __init arch_xor_init(void)
{
diff --git a/arch/powerpc/include/asm/xor_altivec.h b/arch/powerpc/include/asm/xor_altivec.h
deleted file mode 100644
index 294620a25f80..000000000000
--- a/arch/powerpc/include/asm/xor_altivec.h
+++ /dev/null
@@ -1,22 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 */
-#ifndef _ASM_POWERPC_XOR_ALTIVEC_H
-#define _ASM_POWERPC_XOR_ALTIVEC_H
-
-#ifdef CONFIG_ALTIVEC
-void xor_altivec_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2);
-void xor_altivec_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3);
-void xor_altivec_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4);
-void xor_altivec_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5);
-
-#endif
-#endif /* _ASM_POWERPC_XOR_ALTIVEC_H */
diff --git a/arch/powerpc/lib/Makefile b/arch/powerpc/lib/Makefile
index f14ecab674a3..002edc3f01d5 100644
--- a/arch/powerpc/lib/Makefile
+++ b/arch/powerpc/lib/Makefile
@@ -73,9 +73,4 @@ obj-$(CONFIG_PPC_LIB_RHEAP) += rheap.o
obj-$(CONFIG_FTR_FIXUP_SELFTEST) += feature-fixups-test.o
-obj-$(CONFIG_ALTIVEC) += xor_vmx.o xor_vmx_glue.o
-CFLAGS_xor_vmx.o += -mhard-float -maltivec $(call cc-option,-mabi=altivec)
-# Enable <altivec.h>
-CFLAGS_xor_vmx.o += -isystem $(shell $(CC) -print-file-name=include)
-
obj-$(CONFIG_PPC64) += $(obj64-y)
diff --git a/arch/powerpc/lib/xor_vmx_glue.c b/arch/powerpc/lib/xor_vmx_glue.c
deleted file mode 100644
index 35d917ece4d1..000000000000
--- a/arch/powerpc/lib/xor_vmx_glue.c
+++ /dev/null
@@ -1,63 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-or-later
-/*
- * Altivec XOR operations
- *
- * Copyright 2017 IBM Corp.
- */
-
-#include <linux/preempt.h>
-#include <linux/export.h>
-#include <linux/sched.h>
-#include <asm/switch_to.h>
-#include <asm/xor_altivec.h>
-#include "xor_vmx.h"
-
-void xor_altivec_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- preempt_disable();
- enable_kernel_altivec();
- __xor_altivec_2(bytes, p1, p2);
- disable_kernel_altivec();
- preempt_enable();
-}
-EXPORT_SYMBOL(xor_altivec_2);
-
-void xor_altivec_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- preempt_disable();
- enable_kernel_altivec();
- __xor_altivec_3(bytes, p1, p2, p3);
- disable_kernel_altivec();
- preempt_enable();
-}
-EXPORT_SYMBOL(xor_altivec_3);
-
-void xor_altivec_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- preempt_disable();
- enable_kernel_altivec();
- __xor_altivec_4(bytes, p1, p2, p3, p4);
- disable_kernel_altivec();
- preempt_enable();
-}
-EXPORT_SYMBOL(xor_altivec_4);
-
-void xor_altivec_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- preempt_disable();
- enable_kernel_altivec();
- __xor_altivec_5(bytes, p1, p2, p3, p4, p5);
- disable_kernel_altivec();
- preempt_enable();
-}
-EXPORT_SYMBOL(xor_altivec_5);
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index e8868f5fc396..006b44ce46bf 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -16,6 +16,7 @@ endif
xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd_glue.o
+xor-$(CONFIG_ALTIVEC) += powerpc/xor_vmx.o powerpc/xor_vmx_glue.o
CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU)
@@ -23,3 +24,7 @@ CFLAGS_REMOVE_arm/xor-neon.o += $(CC_FLAGS_NO_FPU)
CFLAGS_arm64/xor-neon.o += $(CC_FLAGS_FPU)
CFLAGS_REMOVE_arm64/xor-neon.o += $(CC_FLAGS_NO_FPU)
+
+CFLAGS_powerpc/xor_vmx.o += -mhard-float -maltivec \
+ $(call cc-option,-mabi=altivec) \
+ -isystem $(shell $(CC) -print-file-name=include)
diff --git a/arch/powerpc/lib/xor_vmx.c b/lib/raid/xor/powerpc/xor_vmx.c
similarity index 100%
rename from arch/powerpc/lib/xor_vmx.c
rename to lib/raid/xor/powerpc/xor_vmx.c
diff --git a/arch/powerpc/lib/xor_vmx.h b/lib/raid/xor/powerpc/xor_vmx.h
similarity index 100%
rename from arch/powerpc/lib/xor_vmx.h
rename to lib/raid/xor/powerpc/xor_vmx.h
diff --git a/lib/raid/xor/powerpc/xor_vmx_glue.c b/lib/raid/xor/powerpc/xor_vmx_glue.c
new file mode 100644
index 000000000000..c41e38340700
--- /dev/null
+++ b/lib/raid/xor/powerpc/xor_vmx_glue.c
@@ -0,0 +1,67 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Altivec XOR operations
+ *
+ * Copyright 2017 IBM Corp.
+ */
+
+#include <linux/preempt.h>
+#include <linux/sched.h>
+#include <linux/raid/xor_impl.h>
+#include <asm/switch_to.h>
+#include <asm/xor.h>
+#include "xor_vmx.h"
+
+static void xor_altivec_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2)
+{
+ preempt_disable();
+ enable_kernel_altivec();
+ __xor_altivec_2(bytes, p1, p2);
+ disable_kernel_altivec();
+ preempt_enable();
+}
+
+static void xor_altivec_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ preempt_disable();
+ enable_kernel_altivec();
+ __xor_altivec_3(bytes, p1, p2, p3);
+ disable_kernel_altivec();
+ preempt_enable();
+}
+
+static void xor_altivec_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ preempt_disable();
+ enable_kernel_altivec();
+ __xor_altivec_4(bytes, p1, p2, p3, p4);
+ disable_kernel_altivec();
+ preempt_enable();
+}
+
+static void xor_altivec_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ preempt_disable();
+ enable_kernel_altivec();
+ __xor_altivec_5(bytes, p1, p2, p3, p4, p5);
+ disable_kernel_altivec();
+ preempt_enable();
+}
+
+struct xor_block_template xor_block_altivec = {
+ .name = "altivec",
+ .do_2 = xor_altivec_2,
+ .do_3 = xor_altivec_3,
+ .do_4 = xor_altivec_4,
+ .do_5 = xor_altivec_5,
+};
--
2.47.3
^ permalink raw reply related
* [PATCH 14/28] loongarch: move the XOR code to lib/raid/
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Move the optimized XOR into lib/raid and include it it in xor.ko
instead of always building it into the main kernel image.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
arch/loongarch/include/asm/xor.h | 24 ++----------
arch/loongarch/include/asm/xor_simd.h | 34 ----------------
arch/loongarch/lib/Makefile | 2 -
lib/raid/xor/Makefile | 2 +
.../lib => lib/raid/xor/loongarch}/xor_simd.c | 0
.../lib => lib/raid/xor/loongarch}/xor_simd.h | 0
.../raid/xor/loongarch}/xor_simd_glue.c | 39 +++++++++++--------
.../raid/xor/loongarch}/xor_template.c | 0
8 files changed, 27 insertions(+), 74 deletions(-)
delete mode 100644 arch/loongarch/include/asm/xor_simd.h
rename {arch/loongarch/lib => lib/raid/xor/loongarch}/xor_simd.c (100%)
rename {arch/loongarch/lib => lib/raid/xor/loongarch}/xor_simd.h (100%)
rename {arch/loongarch/lib => lib/raid/xor/loongarch}/xor_simd_glue.c (64%)
rename {arch/loongarch/lib => lib/raid/xor/loongarch}/xor_template.c (100%)
diff --git a/arch/loongarch/include/asm/xor.h b/arch/loongarch/include/asm/xor.h
index d17c0e3b047f..7e32f72f8b03 100644
--- a/arch/loongarch/include/asm/xor.h
+++ b/arch/loongarch/include/asm/xor.h
@@ -6,27 +6,6 @@
#define _ASM_LOONGARCH_XOR_H
#include <asm/cpu-features.h>
-#include <asm/xor_simd.h>
-
-#ifdef CONFIG_CPU_HAS_LSX
-static struct xor_block_template xor_block_lsx = {
- .name = "lsx",
- .do_2 = xor_lsx_2,
- .do_3 = xor_lsx_3,
- .do_4 = xor_lsx_4,
- .do_5 = xor_lsx_5,
-};
-#endif /* CONFIG_CPU_HAS_LSX */
-
-#ifdef CONFIG_CPU_HAS_LASX
-static struct xor_block_template xor_block_lasx = {
- .name = "lasx",
- .do_2 = xor_lasx_2,
- .do_3 = xor_lasx_3,
- .do_4 = xor_lasx_4,
- .do_5 = xor_lasx_5,
-};
-#endif /* CONFIG_CPU_HAS_LASX */
/*
* For grins, also test the generic routines.
@@ -38,6 +17,9 @@ static struct xor_block_template xor_block_lasx = {
*/
#include <asm-generic/xor.h>
+extern struct xor_block_template xor_block_lsx;
+extern struct xor_block_template xor_block_lasx;
+
#define arch_xor_init arch_xor_init
static __always_inline void __init arch_xor_init(void)
{
diff --git a/arch/loongarch/include/asm/xor_simd.h b/arch/loongarch/include/asm/xor_simd.h
deleted file mode 100644
index 471b96332f38..000000000000
--- a/arch/loongarch/include/asm/xor_simd.h
+++ /dev/null
@@ -1,34 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * Copyright (C) 2023 WANG Xuerui <git@xen0n.name>
- */
-#ifndef _ASM_LOONGARCH_XOR_SIMD_H
-#define _ASM_LOONGARCH_XOR_SIMD_H
-
-#ifdef CONFIG_CPU_HAS_LSX
-void xor_lsx_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2);
-void xor_lsx_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2, const unsigned long * __restrict p3);
-void xor_lsx_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2, const unsigned long * __restrict p3,
- const unsigned long * __restrict p4);
-void xor_lsx_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2, const unsigned long * __restrict p3,
- const unsigned long * __restrict p4, const unsigned long * __restrict p5);
-#endif /* CONFIG_CPU_HAS_LSX */
-
-#ifdef CONFIG_CPU_HAS_LASX
-void xor_lasx_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2);
-void xor_lasx_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2, const unsigned long * __restrict p3);
-void xor_lasx_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2, const unsigned long * __restrict p3,
- const unsigned long * __restrict p4);
-void xor_lasx_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2, const unsigned long * __restrict p3,
- const unsigned long * __restrict p4, const unsigned long * __restrict p5);
-#endif /* CONFIG_CPU_HAS_LASX */
-
-#endif /* _ASM_LOONGARCH_XOR_SIMD_H */
diff --git a/arch/loongarch/lib/Makefile b/arch/loongarch/lib/Makefile
index ccea3bbd4353..827a88529a42 100644
--- a/arch/loongarch/lib/Makefile
+++ b/arch/loongarch/lib/Makefile
@@ -8,6 +8,4 @@ lib-y += delay.o memset.o memcpy.o memmove.o \
obj-$(CONFIG_ARCH_SUPPORTS_INT128) += tishift.o
-obj-$(CONFIG_CPU_HAS_LSX) += xor_simd.o xor_simd_glue.o
-
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index 4ab0e7411ff7..e8868f5fc396 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -14,6 +14,8 @@ ifeq ($(CONFIG_ARM),y)
xor-$(CONFIG_KERNEL_MODE_NEON) += arm/xor-neon.o arm/xor-neon-glue.o
endif
xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o
+xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o
+xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd_glue.o
CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU)
diff --git a/arch/loongarch/lib/xor_simd.c b/lib/raid/xor/loongarch/xor_simd.c
similarity index 100%
rename from arch/loongarch/lib/xor_simd.c
rename to lib/raid/xor/loongarch/xor_simd.c
diff --git a/arch/loongarch/lib/xor_simd.h b/lib/raid/xor/loongarch/xor_simd.h
similarity index 100%
rename from arch/loongarch/lib/xor_simd.h
rename to lib/raid/xor/loongarch/xor_simd.h
diff --git a/arch/loongarch/lib/xor_simd_glue.c b/lib/raid/xor/loongarch/xor_simd_glue.c
similarity index 64%
rename from arch/loongarch/lib/xor_simd_glue.c
rename to lib/raid/xor/loongarch/xor_simd_glue.c
index 393f689dbcf6..11fa3b47ba83 100644
--- a/arch/loongarch/lib/xor_simd_glue.c
+++ b/lib/raid/xor/loongarch/xor_simd_glue.c
@@ -5,24 +5,23 @@
* Copyright (C) 2023 WANG Xuerui <git@xen0n.name>
*/
-#include <linux/export.h>
#include <linux/sched.h>
+#include <linux/raid/xor_impl.h>
#include <asm/fpu.h>
-#include <asm/xor_simd.h>
+#include <asm/xor.h>
#include "xor_simd.h"
#define MAKE_XOR_GLUE_2(flavor) \
-void xor_##flavor##_2(unsigned long bytes, unsigned long * __restrict p1, \
+static void xor_##flavor##_2(unsigned long bytes, unsigned long * __restrict p1,\
const unsigned long * __restrict p2) \
{ \
kernel_fpu_begin(); \
__xor_##flavor##_2(bytes, p1, p2); \
kernel_fpu_end(); \
} \
-EXPORT_SYMBOL_GPL(xor_##flavor##_2)
#define MAKE_XOR_GLUE_3(flavor) \
-void xor_##flavor##_3(unsigned long bytes, unsigned long * __restrict p1, \
+static void xor_##flavor##_3(unsigned long bytes, unsigned long * __restrict p1,\
const unsigned long * __restrict p2, \
const unsigned long * __restrict p3) \
{ \
@@ -30,10 +29,9 @@ void xor_##flavor##_3(unsigned long bytes, unsigned long * __restrict p1, \
__xor_##flavor##_3(bytes, p1, p2, p3); \
kernel_fpu_end(); \
} \
-EXPORT_SYMBOL_GPL(xor_##flavor##_3)
#define MAKE_XOR_GLUE_4(flavor) \
-void xor_##flavor##_4(unsigned long bytes, unsigned long * __restrict p1, \
+static void xor_##flavor##_4(unsigned long bytes, unsigned long * __restrict p1,\
const unsigned long * __restrict p2, \
const unsigned long * __restrict p3, \
const unsigned long * __restrict p4) \
@@ -42,10 +40,9 @@ void xor_##flavor##_4(unsigned long bytes, unsigned long * __restrict p1, \
__xor_##flavor##_4(bytes, p1, p2, p3, p4); \
kernel_fpu_end(); \
} \
-EXPORT_SYMBOL_GPL(xor_##flavor##_4)
#define MAKE_XOR_GLUE_5(flavor) \
-void xor_##flavor##_5(unsigned long bytes, unsigned long * __restrict p1, \
+static void xor_##flavor##_5(unsigned long bytes, unsigned long * __restrict p1,\
const unsigned long * __restrict p2, \
const unsigned long * __restrict p3, \
const unsigned long * __restrict p4, \
@@ -55,18 +52,26 @@ void xor_##flavor##_5(unsigned long bytes, unsigned long * __restrict p1, \
__xor_##flavor##_5(bytes, p1, p2, p3, p4, p5); \
kernel_fpu_end(); \
} \
-EXPORT_SYMBOL_GPL(xor_##flavor##_5)
-#define MAKE_XOR_GLUES(flavor) \
- MAKE_XOR_GLUE_2(flavor); \
- MAKE_XOR_GLUE_3(flavor); \
- MAKE_XOR_GLUE_4(flavor); \
- MAKE_XOR_GLUE_5(flavor)
+#define MAKE_XOR_GLUES(flavor) \
+ MAKE_XOR_GLUE_2(flavor); \
+ MAKE_XOR_GLUE_3(flavor); \
+ MAKE_XOR_GLUE_4(flavor); \
+ MAKE_XOR_GLUE_5(flavor); \
+ \
+struct xor_block_template xor_block_##flavor = { \
+ .name = __stringify(flavor), \
+ .do_2 = xor_##flavor##_2, \
+ .do_3 = xor_##flavor##_3, \
+ .do_4 = xor_##flavor##_4, \
+ .do_5 = xor_##flavor##_5, \
+}
+
#ifdef CONFIG_CPU_HAS_LSX
MAKE_XOR_GLUES(lsx);
-#endif
+#endif /* CONFIG_CPU_HAS_LSX */
#ifdef CONFIG_CPU_HAS_LASX
MAKE_XOR_GLUES(lasx);
-#endif
+#endif /* CONFIG_CPU_HAS_LASX */
diff --git a/arch/loongarch/lib/xor_template.c b/lib/raid/xor/loongarch/xor_template.c
similarity index 100%
rename from arch/loongarch/lib/xor_template.c
rename to lib/raid/xor/loongarch/xor_template.c
--
2.47.3
^ permalink raw reply related
* [PATCH 13/28] arm64: move the XOR code to lib/raid/
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Move the optimized XOR into lib/raid and include it it in the main
xor.ko instead of building a separate module for it.
Note that this drops the CONFIG_KERNEL_MODE_NEON dependency, as that is
always set for arm64.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
arch/arm64/include/asm/xor.h | 58 +------------------
arch/arm64/lib/Makefile | 6 --
lib/raid/xor/Makefile | 4 ++
lib/raid/xor/arm64/xor-neon-glue.c | 57 ++++++++++++++++++
.../lib => lib/raid/xor/arm64}/xor-neon.c | 20 +------
5 files changed, 67 insertions(+), 78 deletions(-)
create mode 100644 lib/raid/xor/arm64/xor-neon-glue.c
rename {arch/arm64/lib => lib/raid/xor/arm64}/xor-neon.c (95%)
diff --git a/arch/arm64/include/asm/xor.h b/arch/arm64/include/asm/xor.h
index 3cee1eb86371..81718f010761 100644
--- a/arch/arm64/include/asm/xor.h
+++ b/arch/arm64/include/asm/xor.h
@@ -1,73 +1,21 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
- * arch/arm64/include/asm/xor.h
- *
* Authors: Jackie Liu <liuyun01@kylinos.cn>
* Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd.
*/
-#include <linux/hardirq.h>
#include <asm-generic/xor.h>
-#include <asm/hwcap.h>
#include <asm/simd.h>
-#ifdef CONFIG_KERNEL_MODE_NEON
-
-extern struct xor_block_template xor_block_inner_neon __ro_after_init;
-
-static void
-xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- scoped_ksimd()
- xor_block_inner_neon.do_2(bytes, p1, p2);
-}
-
-static void
-xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- scoped_ksimd()
- xor_block_inner_neon.do_3(bytes, p1, p2, p3);
-}
-
-static void
-xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- scoped_ksimd()
- xor_block_inner_neon.do_4(bytes, p1, p2, p3, p4);
-}
-
-static void
-xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- scoped_ksimd()
- xor_block_inner_neon.do_5(bytes, p1, p2, p3, p4, p5);
-}
-
-static struct xor_block_template xor_block_arm64 = {
- .name = "arm64_neon",
- .do_2 = xor_neon_2,
- .do_3 = xor_neon_3,
- .do_4 = xor_neon_4,
- .do_5 = xor_neon_5
-};
+extern struct xor_block_template xor_block_arm64;
+void __init xor_neon_init(void);
#define arch_xor_init arch_xor_init
static __always_inline void __init arch_xor_init(void)
{
+ xor_neon_init();
xor_register(&xor_block_8regs);
xor_register(&xor_block_32regs);
if (cpu_has_neon())
xor_register(&xor_block_arm64);
}
-
-#endif /* ! CONFIG_KERNEL_MODE_NEON */
diff --git a/arch/arm64/lib/Makefile b/arch/arm64/lib/Makefile
index 633e5223d944..448c917494f3 100644
--- a/arch/arm64/lib/Makefile
+++ b/arch/arm64/lib/Makefile
@@ -5,12 +5,6 @@ lib-y := clear_user.o delay.o copy_from_user.o \
memset.o memcmp.o strcmp.o strncmp.o strlen.o \
strnlen.o strchr.o strrchr.o tishift.o
-ifeq ($(CONFIG_KERNEL_MODE_NEON), y)
-obj-$(CONFIG_XOR_BLOCKS) += xor-neon.o
-CFLAGS_xor-neon.o += $(CC_FLAGS_FPU)
-CFLAGS_REMOVE_xor-neon.o += $(CC_FLAGS_NO_FPU)
-endif
-
lib-$(CONFIG_ARCH_HAS_UACCESS_FLUSHCACHE) += uaccess_flushcache.o
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index fb760edae54b..4ab0e7411ff7 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -13,7 +13,11 @@ xor-$(CONFIG_ARM) += arm/xor.o
ifeq ($(CONFIG_ARM),y)
xor-$(CONFIG_KERNEL_MODE_NEON) += arm/xor-neon.o arm/xor-neon-glue.o
endif
+xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o
CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU)
CFLAGS_REMOVE_arm/xor-neon.o += $(CC_FLAGS_NO_FPU)
+
+CFLAGS_arm64/xor-neon.o += $(CC_FLAGS_FPU)
+CFLAGS_REMOVE_arm64/xor-neon.o += $(CC_FLAGS_NO_FPU)
diff --git a/lib/raid/xor/arm64/xor-neon-glue.c b/lib/raid/xor/arm64/xor-neon-glue.c
new file mode 100644
index 000000000000..067a2095659a
--- /dev/null
+++ b/lib/raid/xor/arm64/xor-neon-glue.c
@@ -0,0 +1,57 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Authors: Jackie Liu <liuyun01@kylinos.cn>
+ * Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd.
+ */
+
+#include <linux/raid/xor_impl.h>
+#include <asm/simd.h>
+#include <asm/xor.h>
+
+extern struct xor_block_template const xor_block_inner_neon;
+
+static void
+xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2)
+{
+ scoped_ksimd()
+ xor_block_inner_neon.do_2(bytes, p1, p2);
+}
+
+static void
+xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ scoped_ksimd()
+ xor_block_inner_neon.do_3(bytes, p1, p2, p3);
+}
+
+static void
+xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ scoped_ksimd()
+ xor_block_inner_neon.do_4(bytes, p1, p2, p3, p4);
+}
+
+static void
+xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ scoped_ksimd()
+ xor_block_inner_neon.do_5(bytes, p1, p2, p3, p4, p5);
+}
+
+struct xor_block_template xor_block_arm64 = {
+ .name = "arm64_neon",
+ .do_2 = xor_neon_2,
+ .do_3 = xor_neon_3,
+ .do_4 = xor_neon_4,
+ .do_5 = xor_neon_5
+};
diff --git a/arch/arm64/lib/xor-neon.c b/lib/raid/xor/arm64/xor-neon.c
similarity index 95%
rename from arch/arm64/lib/xor-neon.c
rename to lib/raid/xor/arm64/xor-neon.c
index 351aba92d932..8d2d185090db 100644
--- a/arch/arm64/lib/xor-neon.c
+++ b/lib/raid/xor/arm64/xor-neon.c
@@ -1,15 +1,13 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
- * arch/arm64/lib/xor-neon.c
- *
* Authors: Jackie Liu <liuyun01@kylinos.cn>
* Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd.
*/
-#include <linux/raid/xor.h>
#include <linux/raid/xor_impl.h>
-#include <linux/module.h>
+#include <linux/cache.h>
#include <asm/neon-intrinsics.h>
+#include <asm/xor.h>
static void xor_arm64_neon_2(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p2)
@@ -180,7 +178,6 @@ struct xor_block_template xor_block_inner_neon __ro_after_init = {
.do_4 = xor_arm64_neon_4,
.do_5 = xor_arm64_neon_5,
};
-EXPORT_SYMBOL(xor_block_inner_neon);
static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r)
{
@@ -318,22 +315,11 @@ static void xor_arm64_eor3_5(unsigned long bytes,
} while (--lines > 0);
}
-static int __init xor_neon_init(void)
+void __init xor_neon_init(void)
{
if (cpu_have_named_feature(SHA3)) {
xor_block_inner_neon.do_3 = xor_arm64_eor3_3;
xor_block_inner_neon.do_4 = xor_arm64_eor3_4;
xor_block_inner_neon.do_5 = xor_arm64_eor3_5;
}
- return 0;
}
-module_init(xor_neon_init);
-
-static void __exit xor_neon_exit(void)
-{
-}
-module_exit(xor_neon_exit);
-
-MODULE_AUTHOR("Jackie Liu <liuyun01@kylinos.cn>");
-MODULE_DESCRIPTION("ARMv8 XOR Extensions");
-MODULE_LICENSE("GPL");
--
2.47.3
^ permalink raw reply related
* [PATCH 12/28] arm: move the XOR code to lib/raid/
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Move the optimized XOR into lib/raid and include it it in the main
xor.ko instead of building a separate module for it.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
arch/arm/include/asm/xor.h | 190 +-----------------
arch/arm/lib/Makefile | 5 -
lib/raid/xor/Makefile | 8 +
lib/raid/xor/arm/xor-neon-glue.c | 58 ++++++
{arch/arm/lib => lib/raid/xor/arm}/xor-neon.c | 10 +-
lib/raid/xor/arm/xor.c | 136 +++++++++++++
6 files changed, 205 insertions(+), 202 deletions(-)
create mode 100644 lib/raid/xor/arm/xor-neon-glue.c
rename {arch/arm/lib => lib/raid/xor/arm}/xor-neon.c (74%)
create mode 100644 lib/raid/xor/arm/xor.c
diff --git a/arch/arm/include/asm/xor.h b/arch/arm/include/asm/xor.h
index b2dcd49186e2..989c55872ef6 100644
--- a/arch/arm/include/asm/xor.h
+++ b/arch/arm/include/asm/xor.h
@@ -1,198 +1,12 @@
/* SPDX-License-Identifier: GPL-2.0-only */
/*
- * arch/arm/include/asm/xor.h
- *
* Copyright (C) 2001 Russell King
*/
#include <asm-generic/xor.h>
-#include <asm/hwcap.h>
#include <asm/neon.h>
-#define __XOR(a1, a2) a1 ^= a2
-
-#define GET_BLOCK_2(dst) \
- __asm__("ldmia %0, {%1, %2}" \
- : "=r" (dst), "=r" (a1), "=r" (a2) \
- : "0" (dst))
-
-#define GET_BLOCK_4(dst) \
- __asm__("ldmia %0, {%1, %2, %3, %4}" \
- : "=r" (dst), "=r" (a1), "=r" (a2), "=r" (a3), "=r" (a4) \
- : "0" (dst))
-
-#define XOR_BLOCK_2(src) \
- __asm__("ldmia %0!, {%1, %2}" \
- : "=r" (src), "=r" (b1), "=r" (b2) \
- : "0" (src)); \
- __XOR(a1, b1); __XOR(a2, b2);
-
-#define XOR_BLOCK_4(src) \
- __asm__("ldmia %0!, {%1, %2, %3, %4}" \
- : "=r" (src), "=r" (b1), "=r" (b2), "=r" (b3), "=r" (b4) \
- : "0" (src)); \
- __XOR(a1, b1); __XOR(a2, b2); __XOR(a3, b3); __XOR(a4, b4)
-
-#define PUT_BLOCK_2(dst) \
- __asm__ __volatile__("stmia %0!, {%2, %3}" \
- : "=r" (dst) \
- : "0" (dst), "r" (a1), "r" (a2))
-
-#define PUT_BLOCK_4(dst) \
- __asm__ __volatile__("stmia %0!, {%2, %3, %4, %5}" \
- : "=r" (dst) \
- : "0" (dst), "r" (a1), "r" (a2), "r" (a3), "r" (a4))
-
-static void
-xor_arm4regs_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- unsigned int lines = bytes / sizeof(unsigned long) / 4;
- register unsigned int a1 __asm__("r4");
- register unsigned int a2 __asm__("r5");
- register unsigned int a3 __asm__("r6");
- register unsigned int a4 __asm__("r10");
- register unsigned int b1 __asm__("r8");
- register unsigned int b2 __asm__("r9");
- register unsigned int b3 __asm__("ip");
- register unsigned int b4 __asm__("lr");
-
- do {
- GET_BLOCK_4(p1);
- XOR_BLOCK_4(p2);
- PUT_BLOCK_4(p1);
- } while (--lines);
-}
-
-static void
-xor_arm4regs_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- unsigned int lines = bytes / sizeof(unsigned long) / 4;
- register unsigned int a1 __asm__("r4");
- register unsigned int a2 __asm__("r5");
- register unsigned int a3 __asm__("r6");
- register unsigned int a4 __asm__("r10");
- register unsigned int b1 __asm__("r8");
- register unsigned int b2 __asm__("r9");
- register unsigned int b3 __asm__("ip");
- register unsigned int b4 __asm__("lr");
-
- do {
- GET_BLOCK_4(p1);
- XOR_BLOCK_4(p2);
- XOR_BLOCK_4(p3);
- PUT_BLOCK_4(p1);
- } while (--lines);
-}
-
-static void
-xor_arm4regs_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- unsigned int lines = bytes / sizeof(unsigned long) / 2;
- register unsigned int a1 __asm__("r8");
- register unsigned int a2 __asm__("r9");
- register unsigned int b1 __asm__("ip");
- register unsigned int b2 __asm__("lr");
-
- do {
- GET_BLOCK_2(p1);
- XOR_BLOCK_2(p2);
- XOR_BLOCK_2(p3);
- XOR_BLOCK_2(p4);
- PUT_BLOCK_2(p1);
- } while (--lines);
-}
-
-static void
-xor_arm4regs_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- unsigned int lines = bytes / sizeof(unsigned long) / 2;
- register unsigned int a1 __asm__("r8");
- register unsigned int a2 __asm__("r9");
- register unsigned int b1 __asm__("ip");
- register unsigned int b2 __asm__("lr");
-
- do {
- GET_BLOCK_2(p1);
- XOR_BLOCK_2(p2);
- XOR_BLOCK_2(p3);
- XOR_BLOCK_2(p4);
- XOR_BLOCK_2(p5);
- PUT_BLOCK_2(p1);
- } while (--lines);
-}
-
-static struct xor_block_template xor_block_arm4regs = {
- .name = "arm4regs",
- .do_2 = xor_arm4regs_2,
- .do_3 = xor_arm4regs_3,
- .do_4 = xor_arm4regs_4,
- .do_5 = xor_arm4regs_5,
-};
-
-#ifdef CONFIG_KERNEL_MODE_NEON
-
-extern struct xor_block_template const xor_block_neon_inner;
-
-static void
-xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- kernel_neon_begin();
- xor_block_neon_inner.do_2(bytes, p1, p2);
- kernel_neon_end();
-}
-
-static void
-xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- kernel_neon_begin();
- xor_block_neon_inner.do_3(bytes, p1, p2, p3);
- kernel_neon_end();
-}
-
-static void
-xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- kernel_neon_begin();
- xor_block_neon_inner.do_4(bytes, p1, p2, p3, p4);
- kernel_neon_end();
-}
-
-static void
-xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- kernel_neon_begin();
- xor_block_neon_inner.do_5(bytes, p1, p2, p3, p4, p5);
- kernel_neon_end();
-}
-
-static struct xor_block_template xor_block_neon = {
- .name = "neon",
- .do_2 = xor_neon_2,
- .do_3 = xor_neon_3,
- .do_4 = xor_neon_4,
- .do_5 = xor_neon_5
-};
-
-#endif /* CONFIG_KERNEL_MODE_NEON */
+extern struct xor_block_template xor_block_arm4regs;
+extern struct xor_block_template xor_block_neon;
#define arch_xor_init arch_xor_init
static __always_inline void __init arch_xor_init(void)
diff --git a/arch/arm/lib/Makefile b/arch/arm/lib/Makefile
index 0ca5aae1bcc3..9295055cdfc9 100644
--- a/arch/arm/lib/Makefile
+++ b/arch/arm/lib/Makefile
@@ -39,9 +39,4 @@ endif
$(obj)/csumpartialcopy.o: $(obj)/csumpartialcopygeneric.S
$(obj)/csumpartialcopyuser.o: $(obj)/csumpartialcopygeneric.S
-ifeq ($(CONFIG_KERNEL_MODE_NEON),y)
- CFLAGS_xor-neon.o += $(CC_FLAGS_FPU)
- obj-$(CONFIG_XOR_BLOCKS) += xor-neon.o
-endif
-
obj-$(CONFIG_FUNCTION_ERROR_INJECTION) += error-inject.o
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index 6d03c27c37c7..fb760edae54b 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -9,3 +9,11 @@ xor-y += xor-8regs-prefetch.o
xor-y += xor-32regs-prefetch.o
xor-$(CONFIG_ALPHA) += alpha/xor.o
+xor-$(CONFIG_ARM) += arm/xor.o
+ifeq ($(CONFIG_ARM),y)
+xor-$(CONFIG_KERNEL_MODE_NEON) += arm/xor-neon.o arm/xor-neon-glue.o
+endif
+
+
+CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU)
+CFLAGS_REMOVE_arm/xor-neon.o += $(CC_FLAGS_NO_FPU)
diff --git a/lib/raid/xor/arm/xor-neon-glue.c b/lib/raid/xor/arm/xor-neon-glue.c
new file mode 100644
index 000000000000..c7b162b383a2
--- /dev/null
+++ b/lib/raid/xor/arm/xor-neon-glue.c
@@ -0,0 +1,58 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2001 Russell King
+ */
+#include <linux/raid/xor_impl.h>
+#include <asm/xor.h>
+
+extern struct xor_block_template const xor_block_neon_inner;
+
+static void
+xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2)
+{
+ kernel_neon_begin();
+ xor_block_neon_inner.do_2(bytes, p1, p2);
+ kernel_neon_end();
+}
+
+static void
+xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ kernel_neon_begin();
+ xor_block_neon_inner.do_3(bytes, p1, p2, p3);
+ kernel_neon_end();
+}
+
+static void
+xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ kernel_neon_begin();
+ xor_block_neon_inner.do_4(bytes, p1, p2, p3, p4);
+ kernel_neon_end();
+}
+
+static void
+xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ kernel_neon_begin();
+ xor_block_neon_inner.do_5(bytes, p1, p2, p3, p4, p5);
+ kernel_neon_end();
+}
+
+struct xor_block_template xor_block_neon = {
+ .name = "neon",
+ .do_2 = xor_neon_2,
+ .do_3 = xor_neon_3,
+ .do_4 = xor_neon_4,
+ .do_5 = xor_neon_5
+};
diff --git a/arch/arm/lib/xor-neon.c b/lib/raid/xor/arm/xor-neon.c
similarity index 74%
rename from arch/arm/lib/xor-neon.c
rename to lib/raid/xor/arm/xor-neon.c
index b5be50567991..c9d4378b0f0e 100644
--- a/arch/arm/lib/xor-neon.c
+++ b/lib/raid/xor/arm/xor-neon.c
@@ -1,16 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/arch/arm/lib/xor-neon.c
- *
* Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
*/
-#include <linux/raid/xor.h>
#include <linux/raid/xor_impl.h>
-#include <linux/module.h>
-
-MODULE_DESCRIPTION("NEON accelerated XOR implementation");
-MODULE_LICENSE("GPL");
#ifndef __ARM_NEON__
#error You should compile this file with '-march=armv7-a -mfloat-abi=softfp -mfpu=neon'
@@ -27,7 +20,7 @@ MODULE_LICENSE("GPL");
#endif
#define NO_TEMPLATE
-#include "../../../lib/raid/xor/xor-8regs.c"
+#include "../xor-8regs.c"
struct xor_block_template const xor_block_neon_inner = {
.name = "__inner_neon__",
@@ -36,4 +29,3 @@ struct xor_block_template const xor_block_neon_inner = {
.do_4 = xor_8regs_4,
.do_5 = xor_8regs_5,
};
-EXPORT_SYMBOL(xor_block_neon_inner);
diff --git a/lib/raid/xor/arm/xor.c b/lib/raid/xor/arm/xor.c
new file mode 100644
index 000000000000..2263341dbbcd
--- /dev/null
+++ b/lib/raid/xor/arm/xor.c
@@ -0,0 +1,136 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Copyright (C) 2001 Russell King
+ */
+#include <linux/raid/xor_impl.h>
+#include <asm/xor.h>
+
+#define __XOR(a1, a2) a1 ^= a2
+
+#define GET_BLOCK_2(dst) \
+ __asm__("ldmia %0, {%1, %2}" \
+ : "=r" (dst), "=r" (a1), "=r" (a2) \
+ : "0" (dst))
+
+#define GET_BLOCK_4(dst) \
+ __asm__("ldmia %0, {%1, %2, %3, %4}" \
+ : "=r" (dst), "=r" (a1), "=r" (a2), "=r" (a3), "=r" (a4) \
+ : "0" (dst))
+
+#define XOR_BLOCK_2(src) \
+ __asm__("ldmia %0!, {%1, %2}" \
+ : "=r" (src), "=r" (b1), "=r" (b2) \
+ : "0" (src)); \
+ __XOR(a1, b1); __XOR(a2, b2);
+
+#define XOR_BLOCK_4(src) \
+ __asm__("ldmia %0!, {%1, %2, %3, %4}" \
+ : "=r" (src), "=r" (b1), "=r" (b2), "=r" (b3), "=r" (b4) \
+ : "0" (src)); \
+ __XOR(a1, b1); __XOR(a2, b2); __XOR(a3, b3); __XOR(a4, b4)
+
+#define PUT_BLOCK_2(dst) \
+ __asm__ __volatile__("stmia %0!, {%2, %3}" \
+ : "=r" (dst) \
+ : "0" (dst), "r" (a1), "r" (a2))
+
+#define PUT_BLOCK_4(dst) \
+ __asm__ __volatile__("stmia %0!, {%2, %3, %4, %5}" \
+ : "=r" (dst) \
+ : "0" (dst), "r" (a1), "r" (a2), "r" (a3), "r" (a4))
+
+static void
+xor_arm4regs_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2)
+{
+ unsigned int lines = bytes / sizeof(unsigned long) / 4;
+ register unsigned int a1 __asm__("r4");
+ register unsigned int a2 __asm__("r5");
+ register unsigned int a3 __asm__("r6");
+ register unsigned int a4 __asm__("r10");
+ register unsigned int b1 __asm__("r8");
+ register unsigned int b2 __asm__("r9");
+ register unsigned int b3 __asm__("ip");
+ register unsigned int b4 __asm__("lr");
+
+ do {
+ GET_BLOCK_4(p1);
+ XOR_BLOCK_4(p2);
+ PUT_BLOCK_4(p1);
+ } while (--lines);
+}
+
+static void
+xor_arm4regs_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ unsigned int lines = bytes / sizeof(unsigned long) / 4;
+ register unsigned int a1 __asm__("r4");
+ register unsigned int a2 __asm__("r5");
+ register unsigned int a3 __asm__("r6");
+ register unsigned int a4 __asm__("r10");
+ register unsigned int b1 __asm__("r8");
+ register unsigned int b2 __asm__("r9");
+ register unsigned int b3 __asm__("ip");
+ register unsigned int b4 __asm__("lr");
+
+ do {
+ GET_BLOCK_4(p1);
+ XOR_BLOCK_4(p2);
+ XOR_BLOCK_4(p3);
+ PUT_BLOCK_4(p1);
+ } while (--lines);
+}
+
+static void
+xor_arm4regs_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ unsigned int lines = bytes / sizeof(unsigned long) / 2;
+ register unsigned int a1 __asm__("r8");
+ register unsigned int a2 __asm__("r9");
+ register unsigned int b1 __asm__("ip");
+ register unsigned int b2 __asm__("lr");
+
+ do {
+ GET_BLOCK_2(p1);
+ XOR_BLOCK_2(p2);
+ XOR_BLOCK_2(p3);
+ XOR_BLOCK_2(p4);
+ PUT_BLOCK_2(p1);
+ } while (--lines);
+}
+
+static void
+xor_arm4regs_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ unsigned int lines = bytes / sizeof(unsigned long) / 2;
+ register unsigned int a1 __asm__("r8");
+ register unsigned int a2 __asm__("r9");
+ register unsigned int b1 __asm__("ip");
+ register unsigned int b2 __asm__("lr");
+
+ do {
+ GET_BLOCK_2(p1);
+ XOR_BLOCK_2(p2);
+ XOR_BLOCK_2(p3);
+ XOR_BLOCK_2(p4);
+ XOR_BLOCK_2(p5);
+ PUT_BLOCK_2(p1);
+ } while (--lines);
+}
+
+struct xor_block_template xor_block_arm4regs = {
+ .name = "arm4regs",
+ .do_2 = xor_arm4regs_2,
+ .do_3 = xor_arm4regs_3,
+ .do_4 = xor_arm4regs_4,
+ .do_5 = xor_arm4regs_5,
+};
--
2.47.3
^ permalink raw reply related
* [PATCH 11/28] alpha: move the XOR code to lib/raid/
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Move the optimized XOR code out of line into lib/raid.
Note that the giant inline assembly block might be better off as a
separate assembly source file now, but I'll leave that to the alpha
maintainers.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Magnus Lindholm <linmag7@gmail.com>
Tested-by: Magnus Lindholm <linmag7@gmail.com>
---
arch/alpha/include/asm/xor.h | 853 +----------------------------------
lib/raid/xor/Makefile | 2 +
lib/raid/xor/alpha/xor.c | 849 ++++++++++++++++++++++++++++++++++
3 files changed, 855 insertions(+), 849 deletions(-)
create mode 100644 lib/raid/xor/alpha/xor.c
diff --git a/arch/alpha/include/asm/xor.h b/arch/alpha/include/asm/xor.h
index 4c8085711df1..e517be577a09 100644
--- a/arch/alpha/include/asm/xor.h
+++ b/arch/alpha/include/asm/xor.h
@@ -1,856 +1,11 @@
/* SPDX-License-Identifier: GPL-2.0-or-later */
-/*
- * include/asm-alpha/xor.h
- *
- * Optimized RAID-5 checksumming functions for alpha EV5 and EV6
- */
-
-extern void
-xor_alpha_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2);
-extern void
-xor_alpha_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3);
-extern void
-xor_alpha_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4);
-extern void
-xor_alpha_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5);
-
-extern void
-xor_alpha_prefetch_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2);
-extern void
-xor_alpha_prefetch_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3);
-extern void
-xor_alpha_prefetch_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4);
-extern void
-xor_alpha_prefetch_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5);
-asm(" \n\
- .text \n\
- .align 3 \n\
- .ent xor_alpha_2 \n\
-xor_alpha_2: \n\
- .prologue 0 \n\
- srl $16, 6, $16 \n\
- .align 4 \n\
-2: \n\
- ldq $0,0($17) \n\
- ldq $1,0($18) \n\
- ldq $2,8($17) \n\
- ldq $3,8($18) \n\
- \n\
- ldq $4,16($17) \n\
- ldq $5,16($18) \n\
- ldq $6,24($17) \n\
- ldq $7,24($18) \n\
- \n\
- ldq $19,32($17) \n\
- ldq $20,32($18) \n\
- ldq $21,40($17) \n\
- ldq $22,40($18) \n\
- \n\
- ldq $23,48($17) \n\
- ldq $24,48($18) \n\
- ldq $25,56($17) \n\
- xor $0,$1,$0 # 7 cycles from $1 load \n\
- \n\
- ldq $27,56($18) \n\
- xor $2,$3,$2 \n\
- stq $0,0($17) \n\
- xor $4,$5,$4 \n\
- \n\
- stq $2,8($17) \n\
- xor $6,$7,$6 \n\
- stq $4,16($17) \n\
- xor $19,$20,$19 \n\
- \n\
- stq $6,24($17) \n\
- xor $21,$22,$21 \n\
- stq $19,32($17) \n\
- xor $23,$24,$23 \n\
- \n\
- stq $21,40($17) \n\
- xor $25,$27,$25 \n\
- stq $23,48($17) \n\
- subq $16,1,$16 \n\
- \n\
- stq $25,56($17) \n\
- addq $17,64,$17 \n\
- addq $18,64,$18 \n\
- bgt $16,2b \n\
- \n\
- ret \n\
- .end xor_alpha_2 \n\
- \n\
- .align 3 \n\
- .ent xor_alpha_3 \n\
-xor_alpha_3: \n\
- .prologue 0 \n\
- srl $16, 6, $16 \n\
- .align 4 \n\
-3: \n\
- ldq $0,0($17) \n\
- ldq $1,0($18) \n\
- ldq $2,0($19) \n\
- ldq $3,8($17) \n\
- \n\
- ldq $4,8($18) \n\
- ldq $6,16($17) \n\
- ldq $7,16($18) \n\
- ldq $21,24($17) \n\
- \n\
- ldq $22,24($18) \n\
- ldq $24,32($17) \n\
- ldq $25,32($18) \n\
- ldq $5,8($19) \n\
- \n\
- ldq $20,16($19) \n\
- ldq $23,24($19) \n\
- ldq $27,32($19) \n\
- nop \n\
- \n\
- xor $0,$1,$1 # 8 cycles from $0 load \n\
- xor $3,$4,$4 # 6 cycles from $4 load \n\
- xor $6,$7,$7 # 6 cycles from $7 load \n\
- xor $21,$22,$22 # 5 cycles from $22 load \n\
- \n\
- xor $1,$2,$2 # 9 cycles from $2 load \n\
- xor $24,$25,$25 # 5 cycles from $25 load \n\
- stq $2,0($17) \n\
- xor $4,$5,$5 # 6 cycles from $5 load \n\
- \n\
- stq $5,8($17) \n\
- xor $7,$20,$20 # 7 cycles from $20 load \n\
- stq $20,16($17) \n\
- xor $22,$23,$23 # 7 cycles from $23 load \n\
- \n\
- stq $23,24($17) \n\
- xor $25,$27,$27 # 7 cycles from $27 load \n\
- stq $27,32($17) \n\
- nop \n\
- \n\
- ldq $0,40($17) \n\
- ldq $1,40($18) \n\
- ldq $3,48($17) \n\
- ldq $4,48($18) \n\
- \n\
- ldq $6,56($17) \n\
- ldq $7,56($18) \n\
- ldq $2,40($19) \n\
- ldq $5,48($19) \n\
- \n\
- ldq $20,56($19) \n\
- xor $0,$1,$1 # 4 cycles from $1 load \n\
- xor $3,$4,$4 # 5 cycles from $4 load \n\
- xor $6,$7,$7 # 5 cycles from $7 load \n\
- \n\
- xor $1,$2,$2 # 4 cycles from $2 load \n\
- xor $4,$5,$5 # 5 cycles from $5 load \n\
- stq $2,40($17) \n\
- xor $7,$20,$20 # 4 cycles from $20 load \n\
- \n\
- stq $5,48($17) \n\
- subq $16,1,$16 \n\
- stq $20,56($17) \n\
- addq $19,64,$19 \n\
- \n\
- addq $18,64,$18 \n\
- addq $17,64,$17 \n\
- bgt $16,3b \n\
- ret \n\
- .end xor_alpha_3 \n\
- \n\
- .align 3 \n\
- .ent xor_alpha_4 \n\
-xor_alpha_4: \n\
- .prologue 0 \n\
- srl $16, 6, $16 \n\
- .align 4 \n\
-4: \n\
- ldq $0,0($17) \n\
- ldq $1,0($18) \n\
- ldq $2,0($19) \n\
- ldq $3,0($20) \n\
- \n\
- ldq $4,8($17) \n\
- ldq $5,8($18) \n\
- ldq $6,8($19) \n\
- ldq $7,8($20) \n\
- \n\
- ldq $21,16($17) \n\
- ldq $22,16($18) \n\
- ldq $23,16($19) \n\
- ldq $24,16($20) \n\
- \n\
- ldq $25,24($17) \n\
- xor $0,$1,$1 # 6 cycles from $1 load \n\
- ldq $27,24($18) \n\
- xor $2,$3,$3 # 6 cycles from $3 load \n\
- \n\
- ldq $0,24($19) \n\
- xor $1,$3,$3 \n\
- ldq $1,24($20) \n\
- xor $4,$5,$5 # 7 cycles from $5 load \n\
- \n\
- stq $3,0($17) \n\
- xor $6,$7,$7 \n\
- xor $21,$22,$22 # 7 cycles from $22 load \n\
- xor $5,$7,$7 \n\
- \n\
- stq $7,8($17) \n\
- xor $23,$24,$24 # 7 cycles from $24 load \n\
- ldq $2,32($17) \n\
- xor $22,$24,$24 \n\
- \n\
- ldq $3,32($18) \n\
- ldq $4,32($19) \n\
- ldq $5,32($20) \n\
- xor $25,$27,$27 # 8 cycles from $27 load \n\
- \n\
- ldq $6,40($17) \n\
- ldq $7,40($18) \n\
- ldq $21,40($19) \n\
- ldq $22,40($20) \n\
- \n\
- stq $24,16($17) \n\
- xor $0,$1,$1 # 9 cycles from $1 load \n\
- xor $2,$3,$3 # 5 cycles from $3 load \n\
- xor $27,$1,$1 \n\
- \n\
- stq $1,24($17) \n\
- xor $4,$5,$5 # 5 cycles from $5 load \n\
- ldq $23,48($17) \n\
- ldq $24,48($18) \n\
- \n\
- ldq $25,48($19) \n\
- xor $3,$5,$5 \n\
- ldq $27,48($20) \n\
- ldq $0,56($17) \n\
- \n\
- ldq $1,56($18) \n\
- ldq $2,56($19) \n\
- xor $6,$7,$7 # 8 cycles from $6 load \n\
- ldq $3,56($20) \n\
- \n\
- stq $5,32($17) \n\
- xor $21,$22,$22 # 8 cycles from $22 load \n\
- xor $7,$22,$22 \n\
- xor $23,$24,$24 # 5 cycles from $24 load \n\
- \n\
- stq $22,40($17) \n\
- xor $25,$27,$27 # 5 cycles from $27 load \n\
- xor $24,$27,$27 \n\
- xor $0,$1,$1 # 5 cycles from $1 load \n\
- \n\
- stq $27,48($17) \n\
- xor $2,$3,$3 # 4 cycles from $3 load \n\
- xor $1,$3,$3 \n\
- subq $16,1,$16 \n\
- \n\
- stq $3,56($17) \n\
- addq $20,64,$20 \n\
- addq $19,64,$19 \n\
- addq $18,64,$18 \n\
- \n\
- addq $17,64,$17 \n\
- bgt $16,4b \n\
- ret \n\
- .end xor_alpha_4 \n\
- \n\
- .align 3 \n\
- .ent xor_alpha_5 \n\
-xor_alpha_5: \n\
- .prologue 0 \n\
- srl $16, 6, $16 \n\
- .align 4 \n\
-5: \n\
- ldq $0,0($17) \n\
- ldq $1,0($18) \n\
- ldq $2,0($19) \n\
- ldq $3,0($20) \n\
- \n\
- ldq $4,0($21) \n\
- ldq $5,8($17) \n\
- ldq $6,8($18) \n\
- ldq $7,8($19) \n\
- \n\
- ldq $22,8($20) \n\
- ldq $23,8($21) \n\
- ldq $24,16($17) \n\
- ldq $25,16($18) \n\
- \n\
- ldq $27,16($19) \n\
- xor $0,$1,$1 # 6 cycles from $1 load \n\
- ldq $28,16($20) \n\
- xor $2,$3,$3 # 6 cycles from $3 load \n\
- \n\
- ldq $0,16($21) \n\
- xor $1,$3,$3 \n\
- ldq $1,24($17) \n\
- xor $3,$4,$4 # 7 cycles from $4 load \n\
- \n\
- stq $4,0($17) \n\
- xor $5,$6,$6 # 7 cycles from $6 load \n\
- xor $7,$22,$22 # 7 cycles from $22 load \n\
- xor $6,$23,$23 # 7 cycles from $23 load \n\
- \n\
- ldq $2,24($18) \n\
- xor $22,$23,$23 \n\
- ldq $3,24($19) \n\
- xor $24,$25,$25 # 8 cycles from $25 load \n\
- \n\
- stq $23,8($17) \n\
- xor $25,$27,$27 # 8 cycles from $27 load \n\
- ldq $4,24($20) \n\
- xor $28,$0,$0 # 7 cycles from $0 load \n\
- \n\
- ldq $5,24($21) \n\
- xor $27,$0,$0 \n\
- ldq $6,32($17) \n\
- ldq $7,32($18) \n\
- \n\
- stq $0,16($17) \n\
- xor $1,$2,$2 # 6 cycles from $2 load \n\
- ldq $22,32($19) \n\
- xor $3,$4,$4 # 4 cycles from $4 load \n\
- \n\
- ldq $23,32($20) \n\
- xor $2,$4,$4 \n\
- ldq $24,32($21) \n\
- ldq $25,40($17) \n\
- \n\
- ldq $27,40($18) \n\
- ldq $28,40($19) \n\
- ldq $0,40($20) \n\
- xor $4,$5,$5 # 7 cycles from $5 load \n\
- \n\
- stq $5,24($17) \n\
- xor $6,$7,$7 # 7 cycles from $7 load \n\
- ldq $1,40($21) \n\
- ldq $2,48($17) \n\
- \n\
- ldq $3,48($18) \n\
- xor $7,$22,$22 # 7 cycles from $22 load \n\
- ldq $4,48($19) \n\
- xor $23,$24,$24 # 6 cycles from $24 load \n\
- \n\
- ldq $5,48($20) \n\
- xor $22,$24,$24 \n\
- ldq $6,48($21) \n\
- xor $25,$27,$27 # 7 cycles from $27 load \n\
- \n\
- stq $24,32($17) \n\
- xor $27,$28,$28 # 8 cycles from $28 load \n\
- ldq $7,56($17) \n\
- xor $0,$1,$1 # 6 cycles from $1 load \n\
- \n\
- ldq $22,56($18) \n\
- ldq $23,56($19) \n\
- ldq $24,56($20) \n\
- ldq $25,56($21) \n\
- \n\
- xor $28,$1,$1 \n\
- xor $2,$3,$3 # 9 cycles from $3 load \n\
- xor $3,$4,$4 # 9 cycles from $4 load \n\
- xor $5,$6,$6 # 8 cycles from $6 load \n\
- \n\
- stq $1,40($17) \n\
- xor $4,$6,$6 \n\
- xor $7,$22,$22 # 7 cycles from $22 load \n\
- xor $23,$24,$24 # 6 cycles from $24 load \n\
- \n\
- stq $6,48($17) \n\
- xor $22,$24,$24 \n\
- subq $16,1,$16 \n\
- xor $24,$25,$25 # 8 cycles from $25 load \n\
- \n\
- stq $25,56($17) \n\
- addq $21,64,$21 \n\
- addq $20,64,$20 \n\
- addq $19,64,$19 \n\
- \n\
- addq $18,64,$18 \n\
- addq $17,64,$17 \n\
- bgt $16,5b \n\
- ret \n\
- .end xor_alpha_5 \n\
- \n\
- .align 3 \n\
- .ent xor_alpha_prefetch_2 \n\
-xor_alpha_prefetch_2: \n\
- .prologue 0 \n\
- srl $16, 6, $16 \n\
- \n\
- ldq $31, 0($17) \n\
- ldq $31, 0($18) \n\
- \n\
- ldq $31, 64($17) \n\
- ldq $31, 64($18) \n\
- \n\
- ldq $31, 128($17) \n\
- ldq $31, 128($18) \n\
- \n\
- ldq $31, 192($17) \n\
- ldq $31, 192($18) \n\
- .align 4 \n\
-2: \n\
- ldq $0,0($17) \n\
- ldq $1,0($18) \n\
- ldq $2,8($17) \n\
- ldq $3,8($18) \n\
- \n\
- ldq $4,16($17) \n\
- ldq $5,16($18) \n\
- ldq $6,24($17) \n\
- ldq $7,24($18) \n\
- \n\
- ldq $19,32($17) \n\
- ldq $20,32($18) \n\
- ldq $21,40($17) \n\
- ldq $22,40($18) \n\
- \n\
- ldq $23,48($17) \n\
- ldq $24,48($18) \n\
- ldq $25,56($17) \n\
- ldq $27,56($18) \n\
- \n\
- ldq $31,256($17) \n\
- xor $0,$1,$0 # 8 cycles from $1 load \n\
- ldq $31,256($18) \n\
- xor $2,$3,$2 \n\
- \n\
- stq $0,0($17) \n\
- xor $4,$5,$4 \n\
- stq $2,8($17) \n\
- xor $6,$7,$6 \n\
- \n\
- stq $4,16($17) \n\
- xor $19,$20,$19 \n\
- stq $6,24($17) \n\
- xor $21,$22,$21 \n\
- \n\
- stq $19,32($17) \n\
- xor $23,$24,$23 \n\
- stq $21,40($17) \n\
- xor $25,$27,$25 \n\
- \n\
- stq $23,48($17) \n\
- subq $16,1,$16 \n\
- stq $25,56($17) \n\
- addq $17,64,$17 \n\
- \n\
- addq $18,64,$18 \n\
- bgt $16,2b \n\
- ret \n\
- .end xor_alpha_prefetch_2 \n\
- \n\
- .align 3 \n\
- .ent xor_alpha_prefetch_3 \n\
-xor_alpha_prefetch_3: \n\
- .prologue 0 \n\
- srl $16, 6, $16 \n\
- \n\
- ldq $31, 0($17) \n\
- ldq $31, 0($18) \n\
- ldq $31, 0($19) \n\
- \n\
- ldq $31, 64($17) \n\
- ldq $31, 64($18) \n\
- ldq $31, 64($19) \n\
- \n\
- ldq $31, 128($17) \n\
- ldq $31, 128($18) \n\
- ldq $31, 128($19) \n\
- \n\
- ldq $31, 192($17) \n\
- ldq $31, 192($18) \n\
- ldq $31, 192($19) \n\
- .align 4 \n\
-3: \n\
- ldq $0,0($17) \n\
- ldq $1,0($18) \n\
- ldq $2,0($19) \n\
- ldq $3,8($17) \n\
- \n\
- ldq $4,8($18) \n\
- ldq $6,16($17) \n\
- ldq $7,16($18) \n\
- ldq $21,24($17) \n\
- \n\
- ldq $22,24($18) \n\
- ldq $24,32($17) \n\
- ldq $25,32($18) \n\
- ldq $5,8($19) \n\
- \n\
- ldq $20,16($19) \n\
- ldq $23,24($19) \n\
- ldq $27,32($19) \n\
- nop \n\
- \n\
- xor $0,$1,$1 # 8 cycles from $0 load \n\
- xor $3,$4,$4 # 7 cycles from $4 load \n\
- xor $6,$7,$7 # 6 cycles from $7 load \n\
- xor $21,$22,$22 # 5 cycles from $22 load \n\
- \n\
- xor $1,$2,$2 # 9 cycles from $2 load \n\
- xor $24,$25,$25 # 5 cycles from $25 load \n\
- stq $2,0($17) \n\
- xor $4,$5,$5 # 6 cycles from $5 load \n\
- \n\
- stq $5,8($17) \n\
- xor $7,$20,$20 # 7 cycles from $20 load \n\
- stq $20,16($17) \n\
- xor $22,$23,$23 # 7 cycles from $23 load \n\
- \n\
- stq $23,24($17) \n\
- xor $25,$27,$27 # 7 cycles from $27 load \n\
- stq $27,32($17) \n\
- nop \n\
- \n\
- ldq $0,40($17) \n\
- ldq $1,40($18) \n\
- ldq $3,48($17) \n\
- ldq $4,48($18) \n\
- \n\
- ldq $6,56($17) \n\
- ldq $7,56($18) \n\
- ldq $2,40($19) \n\
- ldq $5,48($19) \n\
- \n\
- ldq $20,56($19) \n\
- ldq $31,256($17) \n\
- ldq $31,256($18) \n\
- ldq $31,256($19) \n\
- \n\
- xor $0,$1,$1 # 6 cycles from $1 load \n\
- xor $3,$4,$4 # 5 cycles from $4 load \n\
- xor $6,$7,$7 # 5 cycles from $7 load \n\
- xor $1,$2,$2 # 4 cycles from $2 load \n\
- \n\
- xor $4,$5,$5 # 5 cycles from $5 load \n\
- xor $7,$20,$20 # 4 cycles from $20 load \n\
- stq $2,40($17) \n\
- subq $16,1,$16 \n\
- \n\
- stq $5,48($17) \n\
- addq $19,64,$19 \n\
- stq $20,56($17) \n\
- addq $18,64,$18 \n\
- \n\
- addq $17,64,$17 \n\
- bgt $16,3b \n\
- ret \n\
- .end xor_alpha_prefetch_3 \n\
- \n\
- .align 3 \n\
- .ent xor_alpha_prefetch_4 \n\
-xor_alpha_prefetch_4: \n\
- .prologue 0 \n\
- srl $16, 6, $16 \n\
- \n\
- ldq $31, 0($17) \n\
- ldq $31, 0($18) \n\
- ldq $31, 0($19) \n\
- ldq $31, 0($20) \n\
- \n\
- ldq $31, 64($17) \n\
- ldq $31, 64($18) \n\
- ldq $31, 64($19) \n\
- ldq $31, 64($20) \n\
- \n\
- ldq $31, 128($17) \n\
- ldq $31, 128($18) \n\
- ldq $31, 128($19) \n\
- ldq $31, 128($20) \n\
- \n\
- ldq $31, 192($17) \n\
- ldq $31, 192($18) \n\
- ldq $31, 192($19) \n\
- ldq $31, 192($20) \n\
- .align 4 \n\
-4: \n\
- ldq $0,0($17) \n\
- ldq $1,0($18) \n\
- ldq $2,0($19) \n\
- ldq $3,0($20) \n\
- \n\
- ldq $4,8($17) \n\
- ldq $5,8($18) \n\
- ldq $6,8($19) \n\
- ldq $7,8($20) \n\
- \n\
- ldq $21,16($17) \n\
- ldq $22,16($18) \n\
- ldq $23,16($19) \n\
- ldq $24,16($20) \n\
- \n\
- ldq $25,24($17) \n\
- xor $0,$1,$1 # 6 cycles from $1 load \n\
- ldq $27,24($18) \n\
- xor $2,$3,$3 # 6 cycles from $3 load \n\
- \n\
- ldq $0,24($19) \n\
- xor $1,$3,$3 \n\
- ldq $1,24($20) \n\
- xor $4,$5,$5 # 7 cycles from $5 load \n\
- \n\
- stq $3,0($17) \n\
- xor $6,$7,$7 \n\
- xor $21,$22,$22 # 7 cycles from $22 load \n\
- xor $5,$7,$7 \n\
- \n\
- stq $7,8($17) \n\
- xor $23,$24,$24 # 7 cycles from $24 load \n\
- ldq $2,32($17) \n\
- xor $22,$24,$24 \n\
- \n\
- ldq $3,32($18) \n\
- ldq $4,32($19) \n\
- ldq $5,32($20) \n\
- xor $25,$27,$27 # 8 cycles from $27 load \n\
- \n\
- ldq $6,40($17) \n\
- ldq $7,40($18) \n\
- ldq $21,40($19) \n\
- ldq $22,40($20) \n\
- \n\
- stq $24,16($17) \n\
- xor $0,$1,$1 # 9 cycles from $1 load \n\
- xor $2,$3,$3 # 5 cycles from $3 load \n\
- xor $27,$1,$1 \n\
- \n\
- stq $1,24($17) \n\
- xor $4,$5,$5 # 5 cycles from $5 load \n\
- ldq $23,48($17) \n\
- xor $3,$5,$5 \n\
- \n\
- ldq $24,48($18) \n\
- ldq $25,48($19) \n\
- ldq $27,48($20) \n\
- ldq $0,56($17) \n\
- \n\
- ldq $1,56($18) \n\
- ldq $2,56($19) \n\
- ldq $3,56($20) \n\
- xor $6,$7,$7 # 8 cycles from $6 load \n\
- \n\
- ldq $31,256($17) \n\
- xor $21,$22,$22 # 8 cycles from $22 load \n\
- ldq $31,256($18) \n\
- xor $7,$22,$22 \n\
- \n\
- ldq $31,256($19) \n\
- xor $23,$24,$24 # 6 cycles from $24 load \n\
- ldq $31,256($20) \n\
- xor $25,$27,$27 # 6 cycles from $27 load \n\
- \n\
- stq $5,32($17) \n\
- xor $24,$27,$27 \n\
- xor $0,$1,$1 # 7 cycles from $1 load \n\
- xor $2,$3,$3 # 6 cycles from $3 load \n\
- \n\
- stq $22,40($17) \n\
- xor $1,$3,$3 \n\
- stq $27,48($17) \n\
- subq $16,1,$16 \n\
- \n\
- stq $3,56($17) \n\
- addq $20,64,$20 \n\
- addq $19,64,$19 \n\
- addq $18,64,$18 \n\
- \n\
- addq $17,64,$17 \n\
- bgt $16,4b \n\
- ret \n\
- .end xor_alpha_prefetch_4 \n\
- \n\
- .align 3 \n\
- .ent xor_alpha_prefetch_5 \n\
-xor_alpha_prefetch_5: \n\
- .prologue 0 \n\
- srl $16, 6, $16 \n\
- \n\
- ldq $31, 0($17) \n\
- ldq $31, 0($18) \n\
- ldq $31, 0($19) \n\
- ldq $31, 0($20) \n\
- ldq $31, 0($21) \n\
- \n\
- ldq $31, 64($17) \n\
- ldq $31, 64($18) \n\
- ldq $31, 64($19) \n\
- ldq $31, 64($20) \n\
- ldq $31, 64($21) \n\
- \n\
- ldq $31, 128($17) \n\
- ldq $31, 128($18) \n\
- ldq $31, 128($19) \n\
- ldq $31, 128($20) \n\
- ldq $31, 128($21) \n\
- \n\
- ldq $31, 192($17) \n\
- ldq $31, 192($18) \n\
- ldq $31, 192($19) \n\
- ldq $31, 192($20) \n\
- ldq $31, 192($21) \n\
- .align 4 \n\
-5: \n\
- ldq $0,0($17) \n\
- ldq $1,0($18) \n\
- ldq $2,0($19) \n\
- ldq $3,0($20) \n\
- \n\
- ldq $4,0($21) \n\
- ldq $5,8($17) \n\
- ldq $6,8($18) \n\
- ldq $7,8($19) \n\
- \n\
- ldq $22,8($20) \n\
- ldq $23,8($21) \n\
- ldq $24,16($17) \n\
- ldq $25,16($18) \n\
- \n\
- ldq $27,16($19) \n\
- xor $0,$1,$1 # 6 cycles from $1 load \n\
- ldq $28,16($20) \n\
- xor $2,$3,$3 # 6 cycles from $3 load \n\
- \n\
- ldq $0,16($21) \n\
- xor $1,$3,$3 \n\
- ldq $1,24($17) \n\
- xor $3,$4,$4 # 7 cycles from $4 load \n\
- \n\
- stq $4,0($17) \n\
- xor $5,$6,$6 # 7 cycles from $6 load \n\
- xor $7,$22,$22 # 7 cycles from $22 load \n\
- xor $6,$23,$23 # 7 cycles from $23 load \n\
- \n\
- ldq $2,24($18) \n\
- xor $22,$23,$23 \n\
- ldq $3,24($19) \n\
- xor $24,$25,$25 # 8 cycles from $25 load \n\
- \n\
- stq $23,8($17) \n\
- xor $25,$27,$27 # 8 cycles from $27 load \n\
- ldq $4,24($20) \n\
- xor $28,$0,$0 # 7 cycles from $0 load \n\
- \n\
- ldq $5,24($21) \n\
- xor $27,$0,$0 \n\
- ldq $6,32($17) \n\
- ldq $7,32($18) \n\
- \n\
- stq $0,16($17) \n\
- xor $1,$2,$2 # 6 cycles from $2 load \n\
- ldq $22,32($19) \n\
- xor $3,$4,$4 # 4 cycles from $4 load \n\
- \n\
- ldq $23,32($20) \n\
- xor $2,$4,$4 \n\
- ldq $24,32($21) \n\
- ldq $25,40($17) \n\
- \n\
- ldq $27,40($18) \n\
- ldq $28,40($19) \n\
- ldq $0,40($20) \n\
- xor $4,$5,$5 # 7 cycles from $5 load \n\
- \n\
- stq $5,24($17) \n\
- xor $6,$7,$7 # 7 cycles from $7 load \n\
- ldq $1,40($21) \n\
- ldq $2,48($17) \n\
- \n\
- ldq $3,48($18) \n\
- xor $7,$22,$22 # 7 cycles from $22 load \n\
- ldq $4,48($19) \n\
- xor $23,$24,$24 # 6 cycles from $24 load \n\
- \n\
- ldq $5,48($20) \n\
- xor $22,$24,$24 \n\
- ldq $6,48($21) \n\
- xor $25,$27,$27 # 7 cycles from $27 load \n\
- \n\
- stq $24,32($17) \n\
- xor $27,$28,$28 # 8 cycles from $28 load \n\
- ldq $7,56($17) \n\
- xor $0,$1,$1 # 6 cycles from $1 load \n\
- \n\
- ldq $22,56($18) \n\
- ldq $23,56($19) \n\
- ldq $24,56($20) \n\
- ldq $25,56($21) \n\
- \n\
- ldq $31,256($17) \n\
- xor $28,$1,$1 \n\
- ldq $31,256($18) \n\
- xor $2,$3,$3 # 9 cycles from $3 load \n\
- \n\
- ldq $31,256($19) \n\
- xor $3,$4,$4 # 9 cycles from $4 load \n\
- ldq $31,256($20) \n\
- xor $5,$6,$6 # 8 cycles from $6 load \n\
- \n\
- stq $1,40($17) \n\
- xor $4,$6,$6 \n\
- xor $7,$22,$22 # 7 cycles from $22 load \n\
- xor $23,$24,$24 # 6 cycles from $24 load \n\
- \n\
- stq $6,48($17) \n\
- xor $22,$24,$24 \n\
- ldq $31,256($21) \n\
- xor $24,$25,$25 # 8 cycles from $25 load \n\
- \n\
- stq $25,56($17) \n\
- subq $16,1,$16 \n\
- addq $21,64,$21 \n\
- addq $20,64,$20 \n\
- \n\
- addq $19,64,$19 \n\
- addq $18,64,$18 \n\
- addq $17,64,$17 \n\
- bgt $16,5b \n\
- \n\
- ret \n\
- .end xor_alpha_prefetch_5 \n\
-");
-
-static struct xor_block_template xor_block_alpha = {
- .name = "alpha",
- .do_2 = xor_alpha_2,
- .do_3 = xor_alpha_3,
- .do_4 = xor_alpha_4,
- .do_5 = xor_alpha_5,
-};
-
-static struct xor_block_template xor_block_alpha_prefetch = {
- .name = "alpha prefetch",
- .do_2 = xor_alpha_prefetch_2,
- .do_3 = xor_alpha_prefetch_3,
- .do_4 = xor_alpha_prefetch_4,
- .do_5 = xor_alpha_prefetch_5,
-};
-
-/* For grins, also test the generic routines. */
+#include <asm/special_insns.h>
#include <asm-generic/xor.h>
+extern struct xor_block_template xor_block_alpha;
+extern struct xor_block_template xor_block_alpha_prefetch;
+
/*
* Force the use of alpha_prefetch if EV6, as it is significantly faster in the
* cold cache case.
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index 89a944c9f990..6d03c27c37c7 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -7,3 +7,5 @@ xor-y += xor-8regs.o
xor-y += xor-32regs.o
xor-y += xor-8regs-prefetch.o
xor-y += xor-32regs-prefetch.o
+
+xor-$(CONFIG_ALPHA) += alpha/xor.o
diff --git a/lib/raid/xor/alpha/xor.c b/lib/raid/xor/alpha/xor.c
new file mode 100644
index 000000000000..0964ac420604
--- /dev/null
+++ b/lib/raid/xor/alpha/xor.c
@@ -0,0 +1,849 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+/*
+ * Optimized XOR parity functions for alpha EV5 and EV6
+ */
+#include <linux/raid/xor_impl.h>
+#include <asm/xor.h>
+
+extern void
+xor_alpha_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2);
+extern void
+xor_alpha_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3);
+extern void
+xor_alpha_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4);
+extern void
+xor_alpha_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5);
+
+extern void
+xor_alpha_prefetch_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2);
+extern void
+xor_alpha_prefetch_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3);
+extern void
+xor_alpha_prefetch_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4);
+extern void
+xor_alpha_prefetch_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5);
+
+asm(" \n\
+ .text \n\
+ .align 3 \n\
+ .ent xor_alpha_2 \n\
+xor_alpha_2: \n\
+ .prologue 0 \n\
+ srl $16, 6, $16 \n\
+ .align 4 \n\
+2: \n\
+ ldq $0,0($17) \n\
+ ldq $1,0($18) \n\
+ ldq $2,8($17) \n\
+ ldq $3,8($18) \n\
+ \n\
+ ldq $4,16($17) \n\
+ ldq $5,16($18) \n\
+ ldq $6,24($17) \n\
+ ldq $7,24($18) \n\
+ \n\
+ ldq $19,32($17) \n\
+ ldq $20,32($18) \n\
+ ldq $21,40($17) \n\
+ ldq $22,40($18) \n\
+ \n\
+ ldq $23,48($17) \n\
+ ldq $24,48($18) \n\
+ ldq $25,56($17) \n\
+ xor $0,$1,$0 # 7 cycles from $1 load \n\
+ \n\
+ ldq $27,56($18) \n\
+ xor $2,$3,$2 \n\
+ stq $0,0($17) \n\
+ xor $4,$5,$4 \n\
+ \n\
+ stq $2,8($17) \n\
+ xor $6,$7,$6 \n\
+ stq $4,16($17) \n\
+ xor $19,$20,$19 \n\
+ \n\
+ stq $6,24($17) \n\
+ xor $21,$22,$21 \n\
+ stq $19,32($17) \n\
+ xor $23,$24,$23 \n\
+ \n\
+ stq $21,40($17) \n\
+ xor $25,$27,$25 \n\
+ stq $23,48($17) \n\
+ subq $16,1,$16 \n\
+ \n\
+ stq $25,56($17) \n\
+ addq $17,64,$17 \n\
+ addq $18,64,$18 \n\
+ bgt $16,2b \n\
+ \n\
+ ret \n\
+ .end xor_alpha_2 \n\
+ \n\
+ .align 3 \n\
+ .ent xor_alpha_3 \n\
+xor_alpha_3: \n\
+ .prologue 0 \n\
+ srl $16, 6, $16 \n\
+ .align 4 \n\
+3: \n\
+ ldq $0,0($17) \n\
+ ldq $1,0($18) \n\
+ ldq $2,0($19) \n\
+ ldq $3,8($17) \n\
+ \n\
+ ldq $4,8($18) \n\
+ ldq $6,16($17) \n\
+ ldq $7,16($18) \n\
+ ldq $21,24($17) \n\
+ \n\
+ ldq $22,24($18) \n\
+ ldq $24,32($17) \n\
+ ldq $25,32($18) \n\
+ ldq $5,8($19) \n\
+ \n\
+ ldq $20,16($19) \n\
+ ldq $23,24($19) \n\
+ ldq $27,32($19) \n\
+ nop \n\
+ \n\
+ xor $0,$1,$1 # 8 cycles from $0 load \n\
+ xor $3,$4,$4 # 6 cycles from $4 load \n\
+ xor $6,$7,$7 # 6 cycles from $7 load \n\
+ xor $21,$22,$22 # 5 cycles from $22 load \n\
+ \n\
+ xor $1,$2,$2 # 9 cycles from $2 load \n\
+ xor $24,$25,$25 # 5 cycles from $25 load \n\
+ stq $2,0($17) \n\
+ xor $4,$5,$5 # 6 cycles from $5 load \n\
+ \n\
+ stq $5,8($17) \n\
+ xor $7,$20,$20 # 7 cycles from $20 load \n\
+ stq $20,16($17) \n\
+ xor $22,$23,$23 # 7 cycles from $23 load \n\
+ \n\
+ stq $23,24($17) \n\
+ xor $25,$27,$27 # 7 cycles from $27 load \n\
+ stq $27,32($17) \n\
+ nop \n\
+ \n\
+ ldq $0,40($17) \n\
+ ldq $1,40($18) \n\
+ ldq $3,48($17) \n\
+ ldq $4,48($18) \n\
+ \n\
+ ldq $6,56($17) \n\
+ ldq $7,56($18) \n\
+ ldq $2,40($19) \n\
+ ldq $5,48($19) \n\
+ \n\
+ ldq $20,56($19) \n\
+ xor $0,$1,$1 # 4 cycles from $1 load \n\
+ xor $3,$4,$4 # 5 cycles from $4 load \n\
+ xor $6,$7,$7 # 5 cycles from $7 load \n\
+ \n\
+ xor $1,$2,$2 # 4 cycles from $2 load \n\
+ xor $4,$5,$5 # 5 cycles from $5 load \n\
+ stq $2,40($17) \n\
+ xor $7,$20,$20 # 4 cycles from $20 load \n\
+ \n\
+ stq $5,48($17) \n\
+ subq $16,1,$16 \n\
+ stq $20,56($17) \n\
+ addq $19,64,$19 \n\
+ \n\
+ addq $18,64,$18 \n\
+ addq $17,64,$17 \n\
+ bgt $16,3b \n\
+ ret \n\
+ .end xor_alpha_3 \n\
+ \n\
+ .align 3 \n\
+ .ent xor_alpha_4 \n\
+xor_alpha_4: \n\
+ .prologue 0 \n\
+ srl $16, 6, $16 \n\
+ .align 4 \n\
+4: \n\
+ ldq $0,0($17) \n\
+ ldq $1,0($18) \n\
+ ldq $2,0($19) \n\
+ ldq $3,0($20) \n\
+ \n\
+ ldq $4,8($17) \n\
+ ldq $5,8($18) \n\
+ ldq $6,8($19) \n\
+ ldq $7,8($20) \n\
+ \n\
+ ldq $21,16($17) \n\
+ ldq $22,16($18) \n\
+ ldq $23,16($19) \n\
+ ldq $24,16($20) \n\
+ \n\
+ ldq $25,24($17) \n\
+ xor $0,$1,$1 # 6 cycles from $1 load \n\
+ ldq $27,24($18) \n\
+ xor $2,$3,$3 # 6 cycles from $3 load \n\
+ \n\
+ ldq $0,24($19) \n\
+ xor $1,$3,$3 \n\
+ ldq $1,24($20) \n\
+ xor $4,$5,$5 # 7 cycles from $5 load \n\
+ \n\
+ stq $3,0($17) \n\
+ xor $6,$7,$7 \n\
+ xor $21,$22,$22 # 7 cycles from $22 load \n\
+ xor $5,$7,$7 \n\
+ \n\
+ stq $7,8($17) \n\
+ xor $23,$24,$24 # 7 cycles from $24 load \n\
+ ldq $2,32($17) \n\
+ xor $22,$24,$24 \n\
+ \n\
+ ldq $3,32($18) \n\
+ ldq $4,32($19) \n\
+ ldq $5,32($20) \n\
+ xor $25,$27,$27 # 8 cycles from $27 load \n\
+ \n\
+ ldq $6,40($17) \n\
+ ldq $7,40($18) \n\
+ ldq $21,40($19) \n\
+ ldq $22,40($20) \n\
+ \n\
+ stq $24,16($17) \n\
+ xor $0,$1,$1 # 9 cycles from $1 load \n\
+ xor $2,$3,$3 # 5 cycles from $3 load \n\
+ xor $27,$1,$1 \n\
+ \n\
+ stq $1,24($17) \n\
+ xor $4,$5,$5 # 5 cycles from $5 load \n\
+ ldq $23,48($17) \n\
+ ldq $24,48($18) \n\
+ \n\
+ ldq $25,48($19) \n\
+ xor $3,$5,$5 \n\
+ ldq $27,48($20) \n\
+ ldq $0,56($17) \n\
+ \n\
+ ldq $1,56($18) \n\
+ ldq $2,56($19) \n\
+ xor $6,$7,$7 # 8 cycles from $6 load \n\
+ ldq $3,56($20) \n\
+ \n\
+ stq $5,32($17) \n\
+ xor $21,$22,$22 # 8 cycles from $22 load \n\
+ xor $7,$22,$22 \n\
+ xor $23,$24,$24 # 5 cycles from $24 load \n\
+ \n\
+ stq $22,40($17) \n\
+ xor $25,$27,$27 # 5 cycles from $27 load \n\
+ xor $24,$27,$27 \n\
+ xor $0,$1,$1 # 5 cycles from $1 load \n\
+ \n\
+ stq $27,48($17) \n\
+ xor $2,$3,$3 # 4 cycles from $3 load \n\
+ xor $1,$3,$3 \n\
+ subq $16,1,$16 \n\
+ \n\
+ stq $3,56($17) \n\
+ addq $20,64,$20 \n\
+ addq $19,64,$19 \n\
+ addq $18,64,$18 \n\
+ \n\
+ addq $17,64,$17 \n\
+ bgt $16,4b \n\
+ ret \n\
+ .end xor_alpha_4 \n\
+ \n\
+ .align 3 \n\
+ .ent xor_alpha_5 \n\
+xor_alpha_5: \n\
+ .prologue 0 \n\
+ srl $16, 6, $16 \n\
+ .align 4 \n\
+5: \n\
+ ldq $0,0($17) \n\
+ ldq $1,0($18) \n\
+ ldq $2,0($19) \n\
+ ldq $3,0($20) \n\
+ \n\
+ ldq $4,0($21) \n\
+ ldq $5,8($17) \n\
+ ldq $6,8($18) \n\
+ ldq $7,8($19) \n\
+ \n\
+ ldq $22,8($20) \n\
+ ldq $23,8($21) \n\
+ ldq $24,16($17) \n\
+ ldq $25,16($18) \n\
+ \n\
+ ldq $27,16($19) \n\
+ xor $0,$1,$1 # 6 cycles from $1 load \n\
+ ldq $28,16($20) \n\
+ xor $2,$3,$3 # 6 cycles from $3 load \n\
+ \n\
+ ldq $0,16($21) \n\
+ xor $1,$3,$3 \n\
+ ldq $1,24($17) \n\
+ xor $3,$4,$4 # 7 cycles from $4 load \n\
+ \n\
+ stq $4,0($17) \n\
+ xor $5,$6,$6 # 7 cycles from $6 load \n\
+ xor $7,$22,$22 # 7 cycles from $22 load \n\
+ xor $6,$23,$23 # 7 cycles from $23 load \n\
+ \n\
+ ldq $2,24($18) \n\
+ xor $22,$23,$23 \n\
+ ldq $3,24($19) \n\
+ xor $24,$25,$25 # 8 cycles from $25 load \n\
+ \n\
+ stq $23,8($17) \n\
+ xor $25,$27,$27 # 8 cycles from $27 load \n\
+ ldq $4,24($20) \n\
+ xor $28,$0,$0 # 7 cycles from $0 load \n\
+ \n\
+ ldq $5,24($21) \n\
+ xor $27,$0,$0 \n\
+ ldq $6,32($17) \n\
+ ldq $7,32($18) \n\
+ \n\
+ stq $0,16($17) \n\
+ xor $1,$2,$2 # 6 cycles from $2 load \n\
+ ldq $22,32($19) \n\
+ xor $3,$4,$4 # 4 cycles from $4 load \n\
+ \n\
+ ldq $23,32($20) \n\
+ xor $2,$4,$4 \n\
+ ldq $24,32($21) \n\
+ ldq $25,40($17) \n\
+ \n\
+ ldq $27,40($18) \n\
+ ldq $28,40($19) \n\
+ ldq $0,40($20) \n\
+ xor $4,$5,$5 # 7 cycles from $5 load \n\
+ \n\
+ stq $5,24($17) \n\
+ xor $6,$7,$7 # 7 cycles from $7 load \n\
+ ldq $1,40($21) \n\
+ ldq $2,48($17) \n\
+ \n\
+ ldq $3,48($18) \n\
+ xor $7,$22,$22 # 7 cycles from $22 load \n\
+ ldq $4,48($19) \n\
+ xor $23,$24,$24 # 6 cycles from $24 load \n\
+ \n\
+ ldq $5,48($20) \n\
+ xor $22,$24,$24 \n\
+ ldq $6,48($21) \n\
+ xor $25,$27,$27 # 7 cycles from $27 load \n\
+ \n\
+ stq $24,32($17) \n\
+ xor $27,$28,$28 # 8 cycles from $28 load \n\
+ ldq $7,56($17) \n\
+ xor $0,$1,$1 # 6 cycles from $1 load \n\
+ \n\
+ ldq $22,56($18) \n\
+ ldq $23,56($19) \n\
+ ldq $24,56($20) \n\
+ ldq $25,56($21) \n\
+ \n\
+ xor $28,$1,$1 \n\
+ xor $2,$3,$3 # 9 cycles from $3 load \n\
+ xor $3,$4,$4 # 9 cycles from $4 load \n\
+ xor $5,$6,$6 # 8 cycles from $6 load \n\
+ \n\
+ stq $1,40($17) \n\
+ xor $4,$6,$6 \n\
+ xor $7,$22,$22 # 7 cycles from $22 load \n\
+ xor $23,$24,$24 # 6 cycles from $24 load \n\
+ \n\
+ stq $6,48($17) \n\
+ xor $22,$24,$24 \n\
+ subq $16,1,$16 \n\
+ xor $24,$25,$25 # 8 cycles from $25 load \n\
+ \n\
+ stq $25,56($17) \n\
+ addq $21,64,$21 \n\
+ addq $20,64,$20 \n\
+ addq $19,64,$19 \n\
+ \n\
+ addq $18,64,$18 \n\
+ addq $17,64,$17 \n\
+ bgt $16,5b \n\
+ ret \n\
+ .end xor_alpha_5 \n\
+ \n\
+ .align 3 \n\
+ .ent xor_alpha_prefetch_2 \n\
+xor_alpha_prefetch_2: \n\
+ .prologue 0 \n\
+ srl $16, 6, $16 \n\
+ \n\
+ ldq $31, 0($17) \n\
+ ldq $31, 0($18) \n\
+ \n\
+ ldq $31, 64($17) \n\
+ ldq $31, 64($18) \n\
+ \n\
+ ldq $31, 128($17) \n\
+ ldq $31, 128($18) \n\
+ \n\
+ ldq $31, 192($17) \n\
+ ldq $31, 192($18) \n\
+ .align 4 \n\
+2: \n\
+ ldq $0,0($17) \n\
+ ldq $1,0($18) \n\
+ ldq $2,8($17) \n\
+ ldq $3,8($18) \n\
+ \n\
+ ldq $4,16($17) \n\
+ ldq $5,16($18) \n\
+ ldq $6,24($17) \n\
+ ldq $7,24($18) \n\
+ \n\
+ ldq $19,32($17) \n\
+ ldq $20,32($18) \n\
+ ldq $21,40($17) \n\
+ ldq $22,40($18) \n\
+ \n\
+ ldq $23,48($17) \n\
+ ldq $24,48($18) \n\
+ ldq $25,56($17) \n\
+ ldq $27,56($18) \n\
+ \n\
+ ldq $31,256($17) \n\
+ xor $0,$1,$0 # 8 cycles from $1 load \n\
+ ldq $31,256($18) \n\
+ xor $2,$3,$2 \n\
+ \n\
+ stq $0,0($17) \n\
+ xor $4,$5,$4 \n\
+ stq $2,8($17) \n\
+ xor $6,$7,$6 \n\
+ \n\
+ stq $4,16($17) \n\
+ xor $19,$20,$19 \n\
+ stq $6,24($17) \n\
+ xor $21,$22,$21 \n\
+ \n\
+ stq $19,32($17) \n\
+ xor $23,$24,$23 \n\
+ stq $21,40($17) \n\
+ xor $25,$27,$25 \n\
+ \n\
+ stq $23,48($17) \n\
+ subq $16,1,$16 \n\
+ stq $25,56($17) \n\
+ addq $17,64,$17 \n\
+ \n\
+ addq $18,64,$18 \n\
+ bgt $16,2b \n\
+ ret \n\
+ .end xor_alpha_prefetch_2 \n\
+ \n\
+ .align 3 \n\
+ .ent xor_alpha_prefetch_3 \n\
+xor_alpha_prefetch_3: \n\
+ .prologue 0 \n\
+ srl $16, 6, $16 \n\
+ \n\
+ ldq $31, 0($17) \n\
+ ldq $31, 0($18) \n\
+ ldq $31, 0($19) \n\
+ \n\
+ ldq $31, 64($17) \n\
+ ldq $31, 64($18) \n\
+ ldq $31, 64($19) \n\
+ \n\
+ ldq $31, 128($17) \n\
+ ldq $31, 128($18) \n\
+ ldq $31, 128($19) \n\
+ \n\
+ ldq $31, 192($17) \n\
+ ldq $31, 192($18) \n\
+ ldq $31, 192($19) \n\
+ .align 4 \n\
+3: \n\
+ ldq $0,0($17) \n\
+ ldq $1,0($18) \n\
+ ldq $2,0($19) \n\
+ ldq $3,8($17) \n\
+ \n\
+ ldq $4,8($18) \n\
+ ldq $6,16($17) \n\
+ ldq $7,16($18) \n\
+ ldq $21,24($17) \n\
+ \n\
+ ldq $22,24($18) \n\
+ ldq $24,32($17) \n\
+ ldq $25,32($18) \n\
+ ldq $5,8($19) \n\
+ \n\
+ ldq $20,16($19) \n\
+ ldq $23,24($19) \n\
+ ldq $27,32($19) \n\
+ nop \n\
+ \n\
+ xor $0,$1,$1 # 8 cycles from $0 load \n\
+ xor $3,$4,$4 # 7 cycles from $4 load \n\
+ xor $6,$7,$7 # 6 cycles from $7 load \n\
+ xor $21,$22,$22 # 5 cycles from $22 load \n\
+ \n\
+ xor $1,$2,$2 # 9 cycles from $2 load \n\
+ xor $24,$25,$25 # 5 cycles from $25 load \n\
+ stq $2,0($17) \n\
+ xor $4,$5,$5 # 6 cycles from $5 load \n\
+ \n\
+ stq $5,8($17) \n\
+ xor $7,$20,$20 # 7 cycles from $20 load \n\
+ stq $20,16($17) \n\
+ xor $22,$23,$23 # 7 cycles from $23 load \n\
+ \n\
+ stq $23,24($17) \n\
+ xor $25,$27,$27 # 7 cycles from $27 load \n\
+ stq $27,32($17) \n\
+ nop \n\
+ \n\
+ ldq $0,40($17) \n\
+ ldq $1,40($18) \n\
+ ldq $3,48($17) \n\
+ ldq $4,48($18) \n\
+ \n\
+ ldq $6,56($17) \n\
+ ldq $7,56($18) \n\
+ ldq $2,40($19) \n\
+ ldq $5,48($19) \n\
+ \n\
+ ldq $20,56($19) \n\
+ ldq $31,256($17) \n\
+ ldq $31,256($18) \n\
+ ldq $31,256($19) \n\
+ \n\
+ xor $0,$1,$1 # 6 cycles from $1 load \n\
+ xor $3,$4,$4 # 5 cycles from $4 load \n\
+ xor $6,$7,$7 # 5 cycles from $7 load \n\
+ xor $1,$2,$2 # 4 cycles from $2 load \n\
+ \n\
+ xor $4,$5,$5 # 5 cycles from $5 load \n\
+ xor $7,$20,$20 # 4 cycles from $20 load \n\
+ stq $2,40($17) \n\
+ subq $16,1,$16 \n\
+ \n\
+ stq $5,48($17) \n\
+ addq $19,64,$19 \n\
+ stq $20,56($17) \n\
+ addq $18,64,$18 \n\
+ \n\
+ addq $17,64,$17 \n\
+ bgt $16,3b \n\
+ ret \n\
+ .end xor_alpha_prefetch_3 \n\
+ \n\
+ .align 3 \n\
+ .ent xor_alpha_prefetch_4 \n\
+xor_alpha_prefetch_4: \n\
+ .prologue 0 \n\
+ srl $16, 6, $16 \n\
+ \n\
+ ldq $31, 0($17) \n\
+ ldq $31, 0($18) \n\
+ ldq $31, 0($19) \n\
+ ldq $31, 0($20) \n\
+ \n\
+ ldq $31, 64($17) \n\
+ ldq $31, 64($18) \n\
+ ldq $31, 64($19) \n\
+ ldq $31, 64($20) \n\
+ \n\
+ ldq $31, 128($17) \n\
+ ldq $31, 128($18) \n\
+ ldq $31, 128($19) \n\
+ ldq $31, 128($20) \n\
+ \n\
+ ldq $31, 192($17) \n\
+ ldq $31, 192($18) \n\
+ ldq $31, 192($19) \n\
+ ldq $31, 192($20) \n\
+ .align 4 \n\
+4: \n\
+ ldq $0,0($17) \n\
+ ldq $1,0($18) \n\
+ ldq $2,0($19) \n\
+ ldq $3,0($20) \n\
+ \n\
+ ldq $4,8($17) \n\
+ ldq $5,8($18) \n\
+ ldq $6,8($19) \n\
+ ldq $7,8($20) \n\
+ \n\
+ ldq $21,16($17) \n\
+ ldq $22,16($18) \n\
+ ldq $23,16($19) \n\
+ ldq $24,16($20) \n\
+ \n\
+ ldq $25,24($17) \n\
+ xor $0,$1,$1 # 6 cycles from $1 load \n\
+ ldq $27,24($18) \n\
+ xor $2,$3,$3 # 6 cycles from $3 load \n\
+ \n\
+ ldq $0,24($19) \n\
+ xor $1,$3,$3 \n\
+ ldq $1,24($20) \n\
+ xor $4,$5,$5 # 7 cycles from $5 load \n\
+ \n\
+ stq $3,0($17) \n\
+ xor $6,$7,$7 \n\
+ xor $21,$22,$22 # 7 cycles from $22 load \n\
+ xor $5,$7,$7 \n\
+ \n\
+ stq $7,8($17) \n\
+ xor $23,$24,$24 # 7 cycles from $24 load \n\
+ ldq $2,32($17) \n\
+ xor $22,$24,$24 \n\
+ \n\
+ ldq $3,32($18) \n\
+ ldq $4,32($19) \n\
+ ldq $5,32($20) \n\
+ xor $25,$27,$27 # 8 cycles from $27 load \n\
+ \n\
+ ldq $6,40($17) \n\
+ ldq $7,40($18) \n\
+ ldq $21,40($19) \n\
+ ldq $22,40($20) \n\
+ \n\
+ stq $24,16($17) \n\
+ xor $0,$1,$1 # 9 cycles from $1 load \n\
+ xor $2,$3,$3 # 5 cycles from $3 load \n\
+ xor $27,$1,$1 \n\
+ \n\
+ stq $1,24($17) \n\
+ xor $4,$5,$5 # 5 cycles from $5 load \n\
+ ldq $23,48($17) \n\
+ xor $3,$5,$5 \n\
+ \n\
+ ldq $24,48($18) \n\
+ ldq $25,48($19) \n\
+ ldq $27,48($20) \n\
+ ldq $0,56($17) \n\
+ \n\
+ ldq $1,56($18) \n\
+ ldq $2,56($19) \n\
+ ldq $3,56($20) \n\
+ xor $6,$7,$7 # 8 cycles from $6 load \n\
+ \n\
+ ldq $31,256($17) \n\
+ xor $21,$22,$22 # 8 cycles from $22 load \n\
+ ldq $31,256($18) \n\
+ xor $7,$22,$22 \n\
+ \n\
+ ldq $31,256($19) \n\
+ xor $23,$24,$24 # 6 cycles from $24 load \n\
+ ldq $31,256($20) \n\
+ xor $25,$27,$27 # 6 cycles from $27 load \n\
+ \n\
+ stq $5,32($17) \n\
+ xor $24,$27,$27 \n\
+ xor $0,$1,$1 # 7 cycles from $1 load \n\
+ xor $2,$3,$3 # 6 cycles from $3 load \n\
+ \n\
+ stq $22,40($17) \n\
+ xor $1,$3,$3 \n\
+ stq $27,48($17) \n\
+ subq $16,1,$16 \n\
+ \n\
+ stq $3,56($17) \n\
+ addq $20,64,$20 \n\
+ addq $19,64,$19 \n\
+ addq $18,64,$18 \n\
+ \n\
+ addq $17,64,$17 \n\
+ bgt $16,4b \n\
+ ret \n\
+ .end xor_alpha_prefetch_4 \n\
+ \n\
+ .align 3 \n\
+ .ent xor_alpha_prefetch_5 \n\
+xor_alpha_prefetch_5: \n\
+ .prologue 0 \n\
+ srl $16, 6, $16 \n\
+ \n\
+ ldq $31, 0($17) \n\
+ ldq $31, 0($18) \n\
+ ldq $31, 0($19) \n\
+ ldq $31, 0($20) \n\
+ ldq $31, 0($21) \n\
+ \n\
+ ldq $31, 64($17) \n\
+ ldq $31, 64($18) \n\
+ ldq $31, 64($19) \n\
+ ldq $31, 64($20) \n\
+ ldq $31, 64($21) \n\
+ \n\
+ ldq $31, 128($17) \n\
+ ldq $31, 128($18) \n\
+ ldq $31, 128($19) \n\
+ ldq $31, 128($20) \n\
+ ldq $31, 128($21) \n\
+ \n\
+ ldq $31, 192($17) \n\
+ ldq $31, 192($18) \n\
+ ldq $31, 192($19) \n\
+ ldq $31, 192($20) \n\
+ ldq $31, 192($21) \n\
+ .align 4 \n\
+5: \n\
+ ldq $0,0($17) \n\
+ ldq $1,0($18) \n\
+ ldq $2,0($19) \n\
+ ldq $3,0($20) \n\
+ \n\
+ ldq $4,0($21) \n\
+ ldq $5,8($17) \n\
+ ldq $6,8($18) \n\
+ ldq $7,8($19) \n\
+ \n\
+ ldq $22,8($20) \n\
+ ldq $23,8($21) \n\
+ ldq $24,16($17) \n\
+ ldq $25,16($18) \n\
+ \n\
+ ldq $27,16($19) \n\
+ xor $0,$1,$1 # 6 cycles from $1 load \n\
+ ldq $28,16($20) \n\
+ xor $2,$3,$3 # 6 cycles from $3 load \n\
+ \n\
+ ldq $0,16($21) \n\
+ xor $1,$3,$3 \n\
+ ldq $1,24($17) \n\
+ xor $3,$4,$4 # 7 cycles from $4 load \n\
+ \n\
+ stq $4,0($17) \n\
+ xor $5,$6,$6 # 7 cycles from $6 load \n\
+ xor $7,$22,$22 # 7 cycles from $22 load \n\
+ xor $6,$23,$23 # 7 cycles from $23 load \n\
+ \n\
+ ldq $2,24($18) \n\
+ xor $22,$23,$23 \n\
+ ldq $3,24($19) \n\
+ xor $24,$25,$25 # 8 cycles from $25 load \n\
+ \n\
+ stq $23,8($17) \n\
+ xor $25,$27,$27 # 8 cycles from $27 load \n\
+ ldq $4,24($20) \n\
+ xor $28,$0,$0 # 7 cycles from $0 load \n\
+ \n\
+ ldq $5,24($21) \n\
+ xor $27,$0,$0 \n\
+ ldq $6,32($17) \n\
+ ldq $7,32($18) \n\
+ \n\
+ stq $0,16($17) \n\
+ xor $1,$2,$2 # 6 cycles from $2 load \n\
+ ldq $22,32($19) \n\
+ xor $3,$4,$4 # 4 cycles from $4 load \n\
+ \n\
+ ldq $23,32($20) \n\
+ xor $2,$4,$4 \n\
+ ldq $24,32($21) \n\
+ ldq $25,40($17) \n\
+ \n\
+ ldq $27,40($18) \n\
+ ldq $28,40($19) \n\
+ ldq $0,40($20) \n\
+ xor $4,$5,$5 # 7 cycles from $5 load \n\
+ \n\
+ stq $5,24($17) \n\
+ xor $6,$7,$7 # 7 cycles from $7 load \n\
+ ldq $1,40($21) \n\
+ ldq $2,48($17) \n\
+ \n\
+ ldq $3,48($18) \n\
+ xor $7,$22,$22 # 7 cycles from $22 load \n\
+ ldq $4,48($19) \n\
+ xor $23,$24,$24 # 6 cycles from $24 load \n\
+ \n\
+ ldq $5,48($20) \n\
+ xor $22,$24,$24 \n\
+ ldq $6,48($21) \n\
+ xor $25,$27,$27 # 7 cycles from $27 load \n\
+ \n\
+ stq $24,32($17) \n\
+ xor $27,$28,$28 # 8 cycles from $28 load \n\
+ ldq $7,56($17) \n\
+ xor $0,$1,$1 # 6 cycles from $1 load \n\
+ \n\
+ ldq $22,56($18) \n\
+ ldq $23,56($19) \n\
+ ldq $24,56($20) \n\
+ ldq $25,56($21) \n\
+ \n\
+ ldq $31,256($17) \n\
+ xor $28,$1,$1 \n\
+ ldq $31,256($18) \n\
+ xor $2,$3,$3 # 9 cycles from $3 load \n\
+ \n\
+ ldq $31,256($19) \n\
+ xor $3,$4,$4 # 9 cycles from $4 load \n\
+ ldq $31,256($20) \n\
+ xor $5,$6,$6 # 8 cycles from $6 load \n\
+ \n\
+ stq $1,40($17) \n\
+ xor $4,$6,$6 \n\
+ xor $7,$22,$22 # 7 cycles from $22 load \n\
+ xor $23,$24,$24 # 6 cycles from $24 load \n\
+ \n\
+ stq $6,48($17) \n\
+ xor $22,$24,$24 \n\
+ ldq $31,256($21) \n\
+ xor $24,$25,$25 # 8 cycles from $25 load \n\
+ \n\
+ stq $25,56($17) \n\
+ subq $16,1,$16 \n\
+ addq $21,64,$21 \n\
+ addq $20,64,$20 \n\
+ \n\
+ addq $19,64,$19 \n\
+ addq $18,64,$18 \n\
+ addq $17,64,$17 \n\
+ bgt $16,5b \n\
+ \n\
+ ret \n\
+ .end xor_alpha_prefetch_5 \n\
+");
+
+struct xor_block_template xor_block_alpha = {
+ .name = "alpha",
+ .do_2 = xor_alpha_2,
+ .do_3 = xor_alpha_3,
+ .do_4 = xor_alpha_4,
+ .do_5 = xor_alpha_5,
+};
+
+struct xor_block_template xor_block_alpha_prefetch = {
+ .name = "alpha prefetch",
+ .do_2 = xor_alpha_prefetch_2,
+ .do_3 = xor_alpha_prefetch_3,
+ .do_4 = xor_alpha_prefetch_4,
+ .do_5 = xor_alpha_prefetch_5,
+};
--
2.47.3
^ permalink raw reply related
* [PATCH 10/28] xor: move generic implementations out of asm-generic/xor.h
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Move the generic implementations from asm-generic/xor.h to
per-implementaion .c files in lib/raid. This will build them
unconditionally even when an architecture forces a specific
implementation, but as we'll need at least one generic version
for the static_call optimization later on we'll pay that price.
Note that this would cause the second xor_block_8regs instance created by
arch/arm/lib/xor-neon.c to be generated instead of discarded as dead
code, so add a NO_TEMPLATE symbol to disable it for this case.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
arch/arm/lib/xor-neon.c | 4 +-
include/asm-generic/xor.h | 727 +----------------------------
lib/raid/xor/Makefile | 4 +
lib/raid/xor/xor-32regs-prefetch.c | 268 +++++++++++
lib/raid/xor/xor-32regs.c | 219 +++++++++
lib/raid/xor/xor-8regs-prefetch.c | 146 ++++++
lib/raid/xor/xor-8regs.c | 105 +++++
7 files changed, 748 insertions(+), 725 deletions(-)
create mode 100644 lib/raid/xor/xor-32regs-prefetch.c
create mode 100644 lib/raid/xor/xor-32regs.c
create mode 100644 lib/raid/xor/xor-8regs-prefetch.c
create mode 100644 lib/raid/xor/xor-8regs.c
diff --git a/arch/arm/lib/xor-neon.c b/arch/arm/lib/xor-neon.c
index 282980b9bf2a..b5be50567991 100644
--- a/arch/arm/lib/xor-neon.c
+++ b/arch/arm/lib/xor-neon.c
@@ -26,8 +26,8 @@ MODULE_LICENSE("GPL");
#pragma GCC optimize "tree-vectorize"
#endif
-#pragma GCC diagnostic ignored "-Wunused-variable"
-#include <asm-generic/xor.h>
+#define NO_TEMPLATE
+#include "../../../lib/raid/xor/xor-8regs.c"
struct xor_block_template const xor_block_neon_inner = {
.name = "__inner_neon__",
diff --git a/include/asm-generic/xor.h b/include/asm-generic/xor.h
index 79c0096aa9d9..fc151fdc45ab 100644
--- a/include/asm-generic/xor.h
+++ b/include/asm-generic/xor.h
@@ -5,726 +5,7 @@
* Generic optimized RAID-5 checksumming functions.
*/
-#include <linux/prefetch.h>
-
-static void
-xor_8regs_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- long lines = bytes / (sizeof (long)) / 8;
-
- do {
- p1[0] ^= p2[0];
- p1[1] ^= p2[1];
- p1[2] ^= p2[2];
- p1[3] ^= p2[3];
- p1[4] ^= p2[4];
- p1[5] ^= p2[5];
- p1[6] ^= p2[6];
- p1[7] ^= p2[7];
- p1 += 8;
- p2 += 8;
- } while (--lines > 0);
-}
-
-static void
-xor_8regs_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- long lines = bytes / (sizeof (long)) / 8;
-
- do {
- p1[0] ^= p2[0] ^ p3[0];
- p1[1] ^= p2[1] ^ p3[1];
- p1[2] ^= p2[2] ^ p3[2];
- p1[3] ^= p2[3] ^ p3[3];
- p1[4] ^= p2[4] ^ p3[4];
- p1[5] ^= p2[5] ^ p3[5];
- p1[6] ^= p2[6] ^ p3[6];
- p1[7] ^= p2[7] ^ p3[7];
- p1 += 8;
- p2 += 8;
- p3 += 8;
- } while (--lines > 0);
-}
-
-static void
-xor_8regs_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- long lines = bytes / (sizeof (long)) / 8;
-
- do {
- p1[0] ^= p2[0] ^ p3[0] ^ p4[0];
- p1[1] ^= p2[1] ^ p3[1] ^ p4[1];
- p1[2] ^= p2[2] ^ p3[2] ^ p4[2];
- p1[3] ^= p2[3] ^ p3[3] ^ p4[3];
- p1[4] ^= p2[4] ^ p3[4] ^ p4[4];
- p1[5] ^= p2[5] ^ p3[5] ^ p4[5];
- p1[6] ^= p2[6] ^ p3[6] ^ p4[6];
- p1[7] ^= p2[7] ^ p3[7] ^ p4[7];
- p1 += 8;
- p2 += 8;
- p3 += 8;
- p4 += 8;
- } while (--lines > 0);
-}
-
-static void
-xor_8regs_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- long lines = bytes / (sizeof (long)) / 8;
-
- do {
- p1[0] ^= p2[0] ^ p3[0] ^ p4[0] ^ p5[0];
- p1[1] ^= p2[1] ^ p3[1] ^ p4[1] ^ p5[1];
- p1[2] ^= p2[2] ^ p3[2] ^ p4[2] ^ p5[2];
- p1[3] ^= p2[3] ^ p3[3] ^ p4[3] ^ p5[3];
- p1[4] ^= p2[4] ^ p3[4] ^ p4[4] ^ p5[4];
- p1[5] ^= p2[5] ^ p3[5] ^ p4[5] ^ p5[5];
- p1[6] ^= p2[6] ^ p3[6] ^ p4[6] ^ p5[6];
- p1[7] ^= p2[7] ^ p3[7] ^ p4[7] ^ p5[7];
- p1 += 8;
- p2 += 8;
- p3 += 8;
- p4 += 8;
- p5 += 8;
- } while (--lines > 0);
-}
-
-static void
-xor_32regs_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- long lines = bytes / (sizeof (long)) / 8;
-
- do {
- register long d0, d1, d2, d3, d4, d5, d6, d7;
- d0 = p1[0]; /* Pull the stuff into registers */
- d1 = p1[1]; /* ... in bursts, if possible. */
- d2 = p1[2];
- d3 = p1[3];
- d4 = p1[4];
- d5 = p1[5];
- d6 = p1[6];
- d7 = p1[7];
- d0 ^= p2[0];
- d1 ^= p2[1];
- d2 ^= p2[2];
- d3 ^= p2[3];
- d4 ^= p2[4];
- d5 ^= p2[5];
- d6 ^= p2[6];
- d7 ^= p2[7];
- p1[0] = d0; /* Store the result (in bursts) */
- p1[1] = d1;
- p1[2] = d2;
- p1[3] = d3;
- p1[4] = d4;
- p1[5] = d5;
- p1[6] = d6;
- p1[7] = d7;
- p1 += 8;
- p2 += 8;
- } while (--lines > 0);
-}
-
-static void
-xor_32regs_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- long lines = bytes / (sizeof (long)) / 8;
-
- do {
- register long d0, d1, d2, d3, d4, d5, d6, d7;
- d0 = p1[0]; /* Pull the stuff into registers */
- d1 = p1[1]; /* ... in bursts, if possible. */
- d2 = p1[2];
- d3 = p1[3];
- d4 = p1[4];
- d5 = p1[5];
- d6 = p1[6];
- d7 = p1[7];
- d0 ^= p2[0];
- d1 ^= p2[1];
- d2 ^= p2[2];
- d3 ^= p2[3];
- d4 ^= p2[4];
- d5 ^= p2[5];
- d6 ^= p2[6];
- d7 ^= p2[7];
- d0 ^= p3[0];
- d1 ^= p3[1];
- d2 ^= p3[2];
- d3 ^= p3[3];
- d4 ^= p3[4];
- d5 ^= p3[5];
- d6 ^= p3[6];
- d7 ^= p3[7];
- p1[0] = d0; /* Store the result (in bursts) */
- p1[1] = d1;
- p1[2] = d2;
- p1[3] = d3;
- p1[4] = d4;
- p1[5] = d5;
- p1[6] = d6;
- p1[7] = d7;
- p1 += 8;
- p2 += 8;
- p3 += 8;
- } while (--lines > 0);
-}
-
-static void
-xor_32regs_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- long lines = bytes / (sizeof (long)) / 8;
-
- do {
- register long d0, d1, d2, d3, d4, d5, d6, d7;
- d0 = p1[0]; /* Pull the stuff into registers */
- d1 = p1[1]; /* ... in bursts, if possible. */
- d2 = p1[2];
- d3 = p1[3];
- d4 = p1[4];
- d5 = p1[5];
- d6 = p1[6];
- d7 = p1[7];
- d0 ^= p2[0];
- d1 ^= p2[1];
- d2 ^= p2[2];
- d3 ^= p2[3];
- d4 ^= p2[4];
- d5 ^= p2[5];
- d6 ^= p2[6];
- d7 ^= p2[7];
- d0 ^= p3[0];
- d1 ^= p3[1];
- d2 ^= p3[2];
- d3 ^= p3[3];
- d4 ^= p3[4];
- d5 ^= p3[5];
- d6 ^= p3[6];
- d7 ^= p3[7];
- d0 ^= p4[0];
- d1 ^= p4[1];
- d2 ^= p4[2];
- d3 ^= p4[3];
- d4 ^= p4[4];
- d5 ^= p4[5];
- d6 ^= p4[6];
- d7 ^= p4[7];
- p1[0] = d0; /* Store the result (in bursts) */
- p1[1] = d1;
- p1[2] = d2;
- p1[3] = d3;
- p1[4] = d4;
- p1[5] = d5;
- p1[6] = d6;
- p1[7] = d7;
- p1 += 8;
- p2 += 8;
- p3 += 8;
- p4 += 8;
- } while (--lines > 0);
-}
-
-static void
-xor_32regs_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- long lines = bytes / (sizeof (long)) / 8;
-
- do {
- register long d0, d1, d2, d3, d4, d5, d6, d7;
- d0 = p1[0]; /* Pull the stuff into registers */
- d1 = p1[1]; /* ... in bursts, if possible. */
- d2 = p1[2];
- d3 = p1[3];
- d4 = p1[4];
- d5 = p1[5];
- d6 = p1[6];
- d7 = p1[7];
- d0 ^= p2[0];
- d1 ^= p2[1];
- d2 ^= p2[2];
- d3 ^= p2[3];
- d4 ^= p2[4];
- d5 ^= p2[5];
- d6 ^= p2[6];
- d7 ^= p2[7];
- d0 ^= p3[0];
- d1 ^= p3[1];
- d2 ^= p3[2];
- d3 ^= p3[3];
- d4 ^= p3[4];
- d5 ^= p3[5];
- d6 ^= p3[6];
- d7 ^= p3[7];
- d0 ^= p4[0];
- d1 ^= p4[1];
- d2 ^= p4[2];
- d3 ^= p4[3];
- d4 ^= p4[4];
- d5 ^= p4[5];
- d6 ^= p4[6];
- d7 ^= p4[7];
- d0 ^= p5[0];
- d1 ^= p5[1];
- d2 ^= p5[2];
- d3 ^= p5[3];
- d4 ^= p5[4];
- d5 ^= p5[5];
- d6 ^= p5[6];
- d7 ^= p5[7];
- p1[0] = d0; /* Store the result (in bursts) */
- p1[1] = d1;
- p1[2] = d2;
- p1[3] = d3;
- p1[4] = d4;
- p1[5] = d5;
- p1[6] = d6;
- p1[7] = d7;
- p1 += 8;
- p2 += 8;
- p3 += 8;
- p4 += 8;
- p5 += 8;
- } while (--lines > 0);
-}
-
-static void
-xor_8regs_p_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- long lines = bytes / (sizeof (long)) / 8 - 1;
- prefetchw(p1);
- prefetch(p2);
-
- do {
- prefetchw(p1+8);
- prefetch(p2+8);
- once_more:
- p1[0] ^= p2[0];
- p1[1] ^= p2[1];
- p1[2] ^= p2[2];
- p1[3] ^= p2[3];
- p1[4] ^= p2[4];
- p1[5] ^= p2[5];
- p1[6] ^= p2[6];
- p1[7] ^= p2[7];
- p1 += 8;
- p2 += 8;
- } while (--lines > 0);
- if (lines == 0)
- goto once_more;
-}
-
-static void
-xor_8regs_p_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- long lines = bytes / (sizeof (long)) / 8 - 1;
- prefetchw(p1);
- prefetch(p2);
- prefetch(p3);
-
- do {
- prefetchw(p1+8);
- prefetch(p2+8);
- prefetch(p3+8);
- once_more:
- p1[0] ^= p2[0] ^ p3[0];
- p1[1] ^= p2[1] ^ p3[1];
- p1[2] ^= p2[2] ^ p3[2];
- p1[3] ^= p2[3] ^ p3[3];
- p1[4] ^= p2[4] ^ p3[4];
- p1[5] ^= p2[5] ^ p3[5];
- p1[6] ^= p2[6] ^ p3[6];
- p1[7] ^= p2[7] ^ p3[7];
- p1 += 8;
- p2 += 8;
- p3 += 8;
- } while (--lines > 0);
- if (lines == 0)
- goto once_more;
-}
-
-static void
-xor_8regs_p_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- long lines = bytes / (sizeof (long)) / 8 - 1;
-
- prefetchw(p1);
- prefetch(p2);
- prefetch(p3);
- prefetch(p4);
-
- do {
- prefetchw(p1+8);
- prefetch(p2+8);
- prefetch(p3+8);
- prefetch(p4+8);
- once_more:
- p1[0] ^= p2[0] ^ p3[0] ^ p4[0];
- p1[1] ^= p2[1] ^ p3[1] ^ p4[1];
- p1[2] ^= p2[2] ^ p3[2] ^ p4[2];
- p1[3] ^= p2[3] ^ p3[3] ^ p4[3];
- p1[4] ^= p2[4] ^ p3[4] ^ p4[4];
- p1[5] ^= p2[5] ^ p3[5] ^ p4[5];
- p1[6] ^= p2[6] ^ p3[6] ^ p4[6];
- p1[7] ^= p2[7] ^ p3[7] ^ p4[7];
- p1 += 8;
- p2 += 8;
- p3 += 8;
- p4 += 8;
- } while (--lines > 0);
- if (lines == 0)
- goto once_more;
-}
-
-static void
-xor_8regs_p_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- long lines = bytes / (sizeof (long)) / 8 - 1;
-
- prefetchw(p1);
- prefetch(p2);
- prefetch(p3);
- prefetch(p4);
- prefetch(p5);
-
- do {
- prefetchw(p1+8);
- prefetch(p2+8);
- prefetch(p3+8);
- prefetch(p4+8);
- prefetch(p5+8);
- once_more:
- p1[0] ^= p2[0] ^ p3[0] ^ p4[0] ^ p5[0];
- p1[1] ^= p2[1] ^ p3[1] ^ p4[1] ^ p5[1];
- p1[2] ^= p2[2] ^ p3[2] ^ p4[2] ^ p5[2];
- p1[3] ^= p2[3] ^ p3[3] ^ p4[3] ^ p5[3];
- p1[4] ^= p2[4] ^ p3[4] ^ p4[4] ^ p5[4];
- p1[5] ^= p2[5] ^ p3[5] ^ p4[5] ^ p5[5];
- p1[6] ^= p2[6] ^ p3[6] ^ p4[6] ^ p5[6];
- p1[7] ^= p2[7] ^ p3[7] ^ p4[7] ^ p5[7];
- p1 += 8;
- p2 += 8;
- p3 += 8;
- p4 += 8;
- p5 += 8;
- } while (--lines > 0);
- if (lines == 0)
- goto once_more;
-}
-
-static void
-xor_32regs_p_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- long lines = bytes / (sizeof (long)) / 8 - 1;
-
- prefetchw(p1);
- prefetch(p2);
-
- do {
- register long d0, d1, d2, d3, d4, d5, d6, d7;
-
- prefetchw(p1+8);
- prefetch(p2+8);
- once_more:
- d0 = p1[0]; /* Pull the stuff into registers */
- d1 = p1[1]; /* ... in bursts, if possible. */
- d2 = p1[2];
- d3 = p1[3];
- d4 = p1[4];
- d5 = p1[5];
- d6 = p1[6];
- d7 = p1[7];
- d0 ^= p2[0];
- d1 ^= p2[1];
- d2 ^= p2[2];
- d3 ^= p2[3];
- d4 ^= p2[4];
- d5 ^= p2[5];
- d6 ^= p2[6];
- d7 ^= p2[7];
- p1[0] = d0; /* Store the result (in bursts) */
- p1[1] = d1;
- p1[2] = d2;
- p1[3] = d3;
- p1[4] = d4;
- p1[5] = d5;
- p1[6] = d6;
- p1[7] = d7;
- p1 += 8;
- p2 += 8;
- } while (--lines > 0);
- if (lines == 0)
- goto once_more;
-}
-
-static void
-xor_32regs_p_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- long lines = bytes / (sizeof (long)) / 8 - 1;
-
- prefetchw(p1);
- prefetch(p2);
- prefetch(p3);
-
- do {
- register long d0, d1, d2, d3, d4, d5, d6, d7;
-
- prefetchw(p1+8);
- prefetch(p2+8);
- prefetch(p3+8);
- once_more:
- d0 = p1[0]; /* Pull the stuff into registers */
- d1 = p1[1]; /* ... in bursts, if possible. */
- d2 = p1[2];
- d3 = p1[3];
- d4 = p1[4];
- d5 = p1[5];
- d6 = p1[6];
- d7 = p1[7];
- d0 ^= p2[0];
- d1 ^= p2[1];
- d2 ^= p2[2];
- d3 ^= p2[3];
- d4 ^= p2[4];
- d5 ^= p2[5];
- d6 ^= p2[6];
- d7 ^= p2[7];
- d0 ^= p3[0];
- d1 ^= p3[1];
- d2 ^= p3[2];
- d3 ^= p3[3];
- d4 ^= p3[4];
- d5 ^= p3[5];
- d6 ^= p3[6];
- d7 ^= p3[7];
- p1[0] = d0; /* Store the result (in bursts) */
- p1[1] = d1;
- p1[2] = d2;
- p1[3] = d3;
- p1[4] = d4;
- p1[5] = d5;
- p1[6] = d6;
- p1[7] = d7;
- p1 += 8;
- p2 += 8;
- p3 += 8;
- } while (--lines > 0);
- if (lines == 0)
- goto once_more;
-}
-
-static void
-xor_32regs_p_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- long lines = bytes / (sizeof (long)) / 8 - 1;
-
- prefetchw(p1);
- prefetch(p2);
- prefetch(p3);
- prefetch(p4);
-
- do {
- register long d0, d1, d2, d3, d4, d5, d6, d7;
-
- prefetchw(p1+8);
- prefetch(p2+8);
- prefetch(p3+8);
- prefetch(p4+8);
- once_more:
- d0 = p1[0]; /* Pull the stuff into registers */
- d1 = p1[1]; /* ... in bursts, if possible. */
- d2 = p1[2];
- d3 = p1[3];
- d4 = p1[4];
- d5 = p1[5];
- d6 = p1[6];
- d7 = p1[7];
- d0 ^= p2[0];
- d1 ^= p2[1];
- d2 ^= p2[2];
- d3 ^= p2[3];
- d4 ^= p2[4];
- d5 ^= p2[5];
- d6 ^= p2[6];
- d7 ^= p2[7];
- d0 ^= p3[0];
- d1 ^= p3[1];
- d2 ^= p3[2];
- d3 ^= p3[3];
- d4 ^= p3[4];
- d5 ^= p3[5];
- d6 ^= p3[6];
- d7 ^= p3[7];
- d0 ^= p4[0];
- d1 ^= p4[1];
- d2 ^= p4[2];
- d3 ^= p4[3];
- d4 ^= p4[4];
- d5 ^= p4[5];
- d6 ^= p4[6];
- d7 ^= p4[7];
- p1[0] = d0; /* Store the result (in bursts) */
- p1[1] = d1;
- p1[2] = d2;
- p1[3] = d3;
- p1[4] = d4;
- p1[5] = d5;
- p1[6] = d6;
- p1[7] = d7;
- p1 += 8;
- p2 += 8;
- p3 += 8;
- p4 += 8;
- } while (--lines > 0);
- if (lines == 0)
- goto once_more;
-}
-
-static void
-xor_32regs_p_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- long lines = bytes / (sizeof (long)) / 8 - 1;
-
- prefetchw(p1);
- prefetch(p2);
- prefetch(p3);
- prefetch(p4);
- prefetch(p5);
-
- do {
- register long d0, d1, d2, d3, d4, d5, d6, d7;
-
- prefetchw(p1+8);
- prefetch(p2+8);
- prefetch(p3+8);
- prefetch(p4+8);
- prefetch(p5+8);
- once_more:
- d0 = p1[0]; /* Pull the stuff into registers */
- d1 = p1[1]; /* ... in bursts, if possible. */
- d2 = p1[2];
- d3 = p1[3];
- d4 = p1[4];
- d5 = p1[5];
- d6 = p1[6];
- d7 = p1[7];
- d0 ^= p2[0];
- d1 ^= p2[1];
- d2 ^= p2[2];
- d3 ^= p2[3];
- d4 ^= p2[4];
- d5 ^= p2[5];
- d6 ^= p2[6];
- d7 ^= p2[7];
- d0 ^= p3[0];
- d1 ^= p3[1];
- d2 ^= p3[2];
- d3 ^= p3[3];
- d4 ^= p3[4];
- d5 ^= p3[5];
- d6 ^= p3[6];
- d7 ^= p3[7];
- d0 ^= p4[0];
- d1 ^= p4[1];
- d2 ^= p4[2];
- d3 ^= p4[3];
- d4 ^= p4[4];
- d5 ^= p4[5];
- d6 ^= p4[6];
- d7 ^= p4[7];
- d0 ^= p5[0];
- d1 ^= p5[1];
- d2 ^= p5[2];
- d3 ^= p5[3];
- d4 ^= p5[4];
- d5 ^= p5[5];
- d6 ^= p5[6];
- d7 ^= p5[7];
- p1[0] = d0; /* Store the result (in bursts) */
- p1[1] = d1;
- p1[2] = d2;
- p1[3] = d3;
- p1[4] = d4;
- p1[5] = d5;
- p1[6] = d6;
- p1[7] = d7;
- p1 += 8;
- p2 += 8;
- p3 += 8;
- p4 += 8;
- p5 += 8;
- } while (--lines > 0);
- if (lines == 0)
- goto once_more;
-}
-
-static struct xor_block_template xor_block_8regs = {
- .name = "8regs",
- .do_2 = xor_8regs_2,
- .do_3 = xor_8regs_3,
- .do_4 = xor_8regs_4,
- .do_5 = xor_8regs_5,
-};
-
-static struct xor_block_template xor_block_32regs = {
- .name = "32regs",
- .do_2 = xor_32regs_2,
- .do_3 = xor_32regs_3,
- .do_4 = xor_32regs_4,
- .do_5 = xor_32regs_5,
-};
-
-static struct xor_block_template xor_block_8regs_p __maybe_unused = {
- .name = "8regs_prefetch",
- .do_2 = xor_8regs_p_2,
- .do_3 = xor_8regs_p_3,
- .do_4 = xor_8regs_p_4,
- .do_5 = xor_8regs_p_5,
-};
-
-static struct xor_block_template xor_block_32regs_p __maybe_unused = {
- .name = "32regs_prefetch",
- .do_2 = xor_32regs_p_2,
- .do_3 = xor_32regs_p_3,
- .do_4 = xor_32regs_p_4,
- .do_5 = xor_32regs_p_5,
-};
+extern struct xor_block_template xor_block_8regs;
+extern struct xor_block_template xor_block_32regs;
+extern struct xor_block_template xor_block_8regs_p;
+extern struct xor_block_template xor_block_32regs_p;
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index 7bca0ce8e90a..89a944c9f990 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -3,3 +3,7 @@
obj-$(CONFIG_XOR_BLOCKS) += xor.o
xor-y += xor-core.o
+xor-y += xor-8regs.o
+xor-y += xor-32regs.o
+xor-y += xor-8regs-prefetch.o
+xor-y += xor-32regs-prefetch.o
diff --git a/lib/raid/xor/xor-32regs-prefetch.c b/lib/raid/xor/xor-32regs-prefetch.c
new file mode 100644
index 000000000000..8666c287f777
--- /dev/null
+++ b/lib/raid/xor/xor-32regs-prefetch.c
@@ -0,0 +1,268 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+#include <linux/prefetch.h>
+#include <linux/raid/xor_impl.h>
+#include <asm-generic/xor.h>
+
+static void
+xor_32regs_p_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2)
+{
+ long lines = bytes / (sizeof (long)) / 8 - 1;
+
+ prefetchw(p1);
+ prefetch(p2);
+
+ do {
+ register long d0, d1, d2, d3, d4, d5, d6, d7;
+
+ prefetchw(p1+8);
+ prefetch(p2+8);
+ once_more:
+ d0 = p1[0]; /* Pull the stuff into registers */
+ d1 = p1[1]; /* ... in bursts, if possible. */
+ d2 = p1[2];
+ d3 = p1[3];
+ d4 = p1[4];
+ d5 = p1[5];
+ d6 = p1[6];
+ d7 = p1[7];
+ d0 ^= p2[0];
+ d1 ^= p2[1];
+ d2 ^= p2[2];
+ d3 ^= p2[3];
+ d4 ^= p2[4];
+ d5 ^= p2[5];
+ d6 ^= p2[6];
+ d7 ^= p2[7];
+ p1[0] = d0; /* Store the result (in bursts) */
+ p1[1] = d1;
+ p1[2] = d2;
+ p1[3] = d3;
+ p1[4] = d4;
+ p1[5] = d5;
+ p1[6] = d6;
+ p1[7] = d7;
+ p1 += 8;
+ p2 += 8;
+ } while (--lines > 0);
+ if (lines == 0)
+ goto once_more;
+}
+
+static void
+xor_32regs_p_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ long lines = bytes / (sizeof (long)) / 8 - 1;
+
+ prefetchw(p1);
+ prefetch(p2);
+ prefetch(p3);
+
+ do {
+ register long d0, d1, d2, d3, d4, d5, d6, d7;
+
+ prefetchw(p1+8);
+ prefetch(p2+8);
+ prefetch(p3+8);
+ once_more:
+ d0 = p1[0]; /* Pull the stuff into registers */
+ d1 = p1[1]; /* ... in bursts, if possible. */
+ d2 = p1[2];
+ d3 = p1[3];
+ d4 = p1[4];
+ d5 = p1[5];
+ d6 = p1[6];
+ d7 = p1[7];
+ d0 ^= p2[0];
+ d1 ^= p2[1];
+ d2 ^= p2[2];
+ d3 ^= p2[3];
+ d4 ^= p2[4];
+ d5 ^= p2[5];
+ d6 ^= p2[6];
+ d7 ^= p2[7];
+ d0 ^= p3[0];
+ d1 ^= p3[1];
+ d2 ^= p3[2];
+ d3 ^= p3[3];
+ d4 ^= p3[4];
+ d5 ^= p3[5];
+ d6 ^= p3[6];
+ d7 ^= p3[7];
+ p1[0] = d0; /* Store the result (in bursts) */
+ p1[1] = d1;
+ p1[2] = d2;
+ p1[3] = d3;
+ p1[4] = d4;
+ p1[5] = d5;
+ p1[6] = d6;
+ p1[7] = d7;
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ } while (--lines > 0);
+ if (lines == 0)
+ goto once_more;
+}
+
+static void
+xor_32regs_p_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ long lines = bytes / (sizeof (long)) / 8 - 1;
+
+ prefetchw(p1);
+ prefetch(p2);
+ prefetch(p3);
+ prefetch(p4);
+
+ do {
+ register long d0, d1, d2, d3, d4, d5, d6, d7;
+
+ prefetchw(p1+8);
+ prefetch(p2+8);
+ prefetch(p3+8);
+ prefetch(p4+8);
+ once_more:
+ d0 = p1[0]; /* Pull the stuff into registers */
+ d1 = p1[1]; /* ... in bursts, if possible. */
+ d2 = p1[2];
+ d3 = p1[3];
+ d4 = p1[4];
+ d5 = p1[5];
+ d6 = p1[6];
+ d7 = p1[7];
+ d0 ^= p2[0];
+ d1 ^= p2[1];
+ d2 ^= p2[2];
+ d3 ^= p2[3];
+ d4 ^= p2[4];
+ d5 ^= p2[5];
+ d6 ^= p2[6];
+ d7 ^= p2[7];
+ d0 ^= p3[0];
+ d1 ^= p3[1];
+ d2 ^= p3[2];
+ d3 ^= p3[3];
+ d4 ^= p3[4];
+ d5 ^= p3[5];
+ d6 ^= p3[6];
+ d7 ^= p3[7];
+ d0 ^= p4[0];
+ d1 ^= p4[1];
+ d2 ^= p4[2];
+ d3 ^= p4[3];
+ d4 ^= p4[4];
+ d5 ^= p4[5];
+ d6 ^= p4[6];
+ d7 ^= p4[7];
+ p1[0] = d0; /* Store the result (in bursts) */
+ p1[1] = d1;
+ p1[2] = d2;
+ p1[3] = d3;
+ p1[4] = d4;
+ p1[5] = d5;
+ p1[6] = d6;
+ p1[7] = d7;
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ p4 += 8;
+ } while (--lines > 0);
+ if (lines == 0)
+ goto once_more;
+}
+
+static void
+xor_32regs_p_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ long lines = bytes / (sizeof (long)) / 8 - 1;
+
+ prefetchw(p1);
+ prefetch(p2);
+ prefetch(p3);
+ prefetch(p4);
+ prefetch(p5);
+
+ do {
+ register long d0, d1, d2, d3, d4, d5, d6, d7;
+
+ prefetchw(p1+8);
+ prefetch(p2+8);
+ prefetch(p3+8);
+ prefetch(p4+8);
+ prefetch(p5+8);
+ once_more:
+ d0 = p1[0]; /* Pull the stuff into registers */
+ d1 = p1[1]; /* ... in bursts, if possible. */
+ d2 = p1[2];
+ d3 = p1[3];
+ d4 = p1[4];
+ d5 = p1[5];
+ d6 = p1[6];
+ d7 = p1[7];
+ d0 ^= p2[0];
+ d1 ^= p2[1];
+ d2 ^= p2[2];
+ d3 ^= p2[3];
+ d4 ^= p2[4];
+ d5 ^= p2[5];
+ d6 ^= p2[6];
+ d7 ^= p2[7];
+ d0 ^= p3[0];
+ d1 ^= p3[1];
+ d2 ^= p3[2];
+ d3 ^= p3[3];
+ d4 ^= p3[4];
+ d5 ^= p3[5];
+ d6 ^= p3[6];
+ d7 ^= p3[7];
+ d0 ^= p4[0];
+ d1 ^= p4[1];
+ d2 ^= p4[2];
+ d3 ^= p4[3];
+ d4 ^= p4[4];
+ d5 ^= p4[5];
+ d6 ^= p4[6];
+ d7 ^= p4[7];
+ d0 ^= p5[0];
+ d1 ^= p5[1];
+ d2 ^= p5[2];
+ d3 ^= p5[3];
+ d4 ^= p5[4];
+ d5 ^= p5[5];
+ d6 ^= p5[6];
+ d7 ^= p5[7];
+ p1[0] = d0; /* Store the result (in bursts) */
+ p1[1] = d1;
+ p1[2] = d2;
+ p1[3] = d3;
+ p1[4] = d4;
+ p1[5] = d5;
+ p1[6] = d6;
+ p1[7] = d7;
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ p4 += 8;
+ p5 += 8;
+ } while (--lines > 0);
+ if (lines == 0)
+ goto once_more;
+}
+
+struct xor_block_template xor_block_32regs_p = {
+ .name = "32regs_prefetch",
+ .do_2 = xor_32regs_p_2,
+ .do_3 = xor_32regs_p_3,
+ .do_4 = xor_32regs_p_4,
+ .do_5 = xor_32regs_p_5,
+};
diff --git a/lib/raid/xor/xor-32regs.c b/lib/raid/xor/xor-32regs.c
new file mode 100644
index 000000000000..58d4fac43eb4
--- /dev/null
+++ b/lib/raid/xor/xor-32regs.c
@@ -0,0 +1,219 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+#include <linux/raid/xor_impl.h>
+#include <asm-generic/xor.h>
+
+static void
+xor_32regs_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2)
+{
+ long lines = bytes / (sizeof (long)) / 8;
+
+ do {
+ register long d0, d1, d2, d3, d4, d5, d6, d7;
+ d0 = p1[0]; /* Pull the stuff into registers */
+ d1 = p1[1]; /* ... in bursts, if possible. */
+ d2 = p1[2];
+ d3 = p1[3];
+ d4 = p1[4];
+ d5 = p1[5];
+ d6 = p1[6];
+ d7 = p1[7];
+ d0 ^= p2[0];
+ d1 ^= p2[1];
+ d2 ^= p2[2];
+ d3 ^= p2[3];
+ d4 ^= p2[4];
+ d5 ^= p2[5];
+ d6 ^= p2[6];
+ d7 ^= p2[7];
+ p1[0] = d0; /* Store the result (in bursts) */
+ p1[1] = d1;
+ p1[2] = d2;
+ p1[3] = d3;
+ p1[4] = d4;
+ p1[5] = d5;
+ p1[6] = d6;
+ p1[7] = d7;
+ p1 += 8;
+ p2 += 8;
+ } while (--lines > 0);
+}
+
+static void
+xor_32regs_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ long lines = bytes / (sizeof (long)) / 8;
+
+ do {
+ register long d0, d1, d2, d3, d4, d5, d6, d7;
+ d0 = p1[0]; /* Pull the stuff into registers */
+ d1 = p1[1]; /* ... in bursts, if possible. */
+ d2 = p1[2];
+ d3 = p1[3];
+ d4 = p1[4];
+ d5 = p1[5];
+ d6 = p1[6];
+ d7 = p1[7];
+ d0 ^= p2[0];
+ d1 ^= p2[1];
+ d2 ^= p2[2];
+ d3 ^= p2[3];
+ d4 ^= p2[4];
+ d5 ^= p2[5];
+ d6 ^= p2[6];
+ d7 ^= p2[7];
+ d0 ^= p3[0];
+ d1 ^= p3[1];
+ d2 ^= p3[2];
+ d3 ^= p3[3];
+ d4 ^= p3[4];
+ d5 ^= p3[5];
+ d6 ^= p3[6];
+ d7 ^= p3[7];
+ p1[0] = d0; /* Store the result (in bursts) */
+ p1[1] = d1;
+ p1[2] = d2;
+ p1[3] = d3;
+ p1[4] = d4;
+ p1[5] = d5;
+ p1[6] = d6;
+ p1[7] = d7;
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ } while (--lines > 0);
+}
+
+static void
+xor_32regs_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ long lines = bytes / (sizeof (long)) / 8;
+
+ do {
+ register long d0, d1, d2, d3, d4, d5, d6, d7;
+ d0 = p1[0]; /* Pull the stuff into registers */
+ d1 = p1[1]; /* ... in bursts, if possible. */
+ d2 = p1[2];
+ d3 = p1[3];
+ d4 = p1[4];
+ d5 = p1[5];
+ d6 = p1[6];
+ d7 = p1[7];
+ d0 ^= p2[0];
+ d1 ^= p2[1];
+ d2 ^= p2[2];
+ d3 ^= p2[3];
+ d4 ^= p2[4];
+ d5 ^= p2[5];
+ d6 ^= p2[6];
+ d7 ^= p2[7];
+ d0 ^= p3[0];
+ d1 ^= p3[1];
+ d2 ^= p3[2];
+ d3 ^= p3[3];
+ d4 ^= p3[4];
+ d5 ^= p3[5];
+ d6 ^= p3[6];
+ d7 ^= p3[7];
+ d0 ^= p4[0];
+ d1 ^= p4[1];
+ d2 ^= p4[2];
+ d3 ^= p4[3];
+ d4 ^= p4[4];
+ d5 ^= p4[5];
+ d6 ^= p4[6];
+ d7 ^= p4[7];
+ p1[0] = d0; /* Store the result (in bursts) */
+ p1[1] = d1;
+ p1[2] = d2;
+ p1[3] = d3;
+ p1[4] = d4;
+ p1[5] = d5;
+ p1[6] = d6;
+ p1[7] = d7;
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ p4 += 8;
+ } while (--lines > 0);
+}
+
+static void
+xor_32regs_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ long lines = bytes / (sizeof (long)) / 8;
+
+ do {
+ register long d0, d1, d2, d3, d4, d5, d6, d7;
+ d0 = p1[0]; /* Pull the stuff into registers */
+ d1 = p1[1]; /* ... in bursts, if possible. */
+ d2 = p1[2];
+ d3 = p1[3];
+ d4 = p1[4];
+ d5 = p1[5];
+ d6 = p1[6];
+ d7 = p1[7];
+ d0 ^= p2[0];
+ d1 ^= p2[1];
+ d2 ^= p2[2];
+ d3 ^= p2[3];
+ d4 ^= p2[4];
+ d5 ^= p2[5];
+ d6 ^= p2[6];
+ d7 ^= p2[7];
+ d0 ^= p3[0];
+ d1 ^= p3[1];
+ d2 ^= p3[2];
+ d3 ^= p3[3];
+ d4 ^= p3[4];
+ d5 ^= p3[5];
+ d6 ^= p3[6];
+ d7 ^= p3[7];
+ d0 ^= p4[0];
+ d1 ^= p4[1];
+ d2 ^= p4[2];
+ d3 ^= p4[3];
+ d4 ^= p4[4];
+ d5 ^= p4[5];
+ d6 ^= p4[6];
+ d7 ^= p4[7];
+ d0 ^= p5[0];
+ d1 ^= p5[1];
+ d2 ^= p5[2];
+ d3 ^= p5[3];
+ d4 ^= p5[4];
+ d5 ^= p5[5];
+ d6 ^= p5[6];
+ d7 ^= p5[7];
+ p1[0] = d0; /* Store the result (in bursts) */
+ p1[1] = d1;
+ p1[2] = d2;
+ p1[3] = d3;
+ p1[4] = d4;
+ p1[5] = d5;
+ p1[6] = d6;
+ p1[7] = d7;
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ p4 += 8;
+ p5 += 8;
+ } while (--lines > 0);
+}
+
+struct xor_block_template xor_block_32regs = {
+ .name = "32regs",
+ .do_2 = xor_32regs_2,
+ .do_3 = xor_32regs_3,
+ .do_4 = xor_32regs_4,
+ .do_5 = xor_32regs_5,
+};
diff --git a/lib/raid/xor/xor-8regs-prefetch.c b/lib/raid/xor/xor-8regs-prefetch.c
new file mode 100644
index 000000000000..67061e35a0a6
--- /dev/null
+++ b/lib/raid/xor/xor-8regs-prefetch.c
@@ -0,0 +1,146 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+#include <linux/prefetch.h>
+#include <linux/raid/xor_impl.h>
+#include <asm-generic/xor.h>
+
+static void
+xor_8regs_p_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2)
+{
+ long lines = bytes / (sizeof (long)) / 8 - 1;
+ prefetchw(p1);
+ prefetch(p2);
+
+ do {
+ prefetchw(p1+8);
+ prefetch(p2+8);
+ once_more:
+ p1[0] ^= p2[0];
+ p1[1] ^= p2[1];
+ p1[2] ^= p2[2];
+ p1[3] ^= p2[3];
+ p1[4] ^= p2[4];
+ p1[5] ^= p2[5];
+ p1[6] ^= p2[6];
+ p1[7] ^= p2[7];
+ p1 += 8;
+ p2 += 8;
+ } while (--lines > 0);
+ if (lines == 0)
+ goto once_more;
+}
+
+static void
+xor_8regs_p_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ long lines = bytes / (sizeof (long)) / 8 - 1;
+ prefetchw(p1);
+ prefetch(p2);
+ prefetch(p3);
+
+ do {
+ prefetchw(p1+8);
+ prefetch(p2+8);
+ prefetch(p3+8);
+ once_more:
+ p1[0] ^= p2[0] ^ p3[0];
+ p1[1] ^= p2[1] ^ p3[1];
+ p1[2] ^= p2[2] ^ p3[2];
+ p1[3] ^= p2[3] ^ p3[3];
+ p1[4] ^= p2[4] ^ p3[4];
+ p1[5] ^= p2[5] ^ p3[5];
+ p1[6] ^= p2[6] ^ p3[6];
+ p1[7] ^= p2[7] ^ p3[7];
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ } while (--lines > 0);
+ if (lines == 0)
+ goto once_more;
+}
+
+static void
+xor_8regs_p_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ long lines = bytes / (sizeof (long)) / 8 - 1;
+
+ prefetchw(p1);
+ prefetch(p2);
+ prefetch(p3);
+ prefetch(p4);
+
+ do {
+ prefetchw(p1+8);
+ prefetch(p2+8);
+ prefetch(p3+8);
+ prefetch(p4+8);
+ once_more:
+ p1[0] ^= p2[0] ^ p3[0] ^ p4[0];
+ p1[1] ^= p2[1] ^ p3[1] ^ p4[1];
+ p1[2] ^= p2[2] ^ p3[2] ^ p4[2];
+ p1[3] ^= p2[3] ^ p3[3] ^ p4[3];
+ p1[4] ^= p2[4] ^ p3[4] ^ p4[4];
+ p1[5] ^= p2[5] ^ p3[5] ^ p4[5];
+ p1[6] ^= p2[6] ^ p3[6] ^ p4[6];
+ p1[7] ^= p2[7] ^ p3[7] ^ p4[7];
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ p4 += 8;
+ } while (--lines > 0);
+ if (lines == 0)
+ goto once_more;
+}
+
+static void
+xor_8regs_p_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ long lines = bytes / (sizeof (long)) / 8 - 1;
+
+ prefetchw(p1);
+ prefetch(p2);
+ prefetch(p3);
+ prefetch(p4);
+ prefetch(p5);
+
+ do {
+ prefetchw(p1+8);
+ prefetch(p2+8);
+ prefetch(p3+8);
+ prefetch(p4+8);
+ prefetch(p5+8);
+ once_more:
+ p1[0] ^= p2[0] ^ p3[0] ^ p4[0] ^ p5[0];
+ p1[1] ^= p2[1] ^ p3[1] ^ p4[1] ^ p5[1];
+ p1[2] ^= p2[2] ^ p3[2] ^ p4[2] ^ p5[2];
+ p1[3] ^= p2[3] ^ p3[3] ^ p4[3] ^ p5[3];
+ p1[4] ^= p2[4] ^ p3[4] ^ p4[4] ^ p5[4];
+ p1[5] ^= p2[5] ^ p3[5] ^ p4[5] ^ p5[5];
+ p1[6] ^= p2[6] ^ p3[6] ^ p4[6] ^ p5[6];
+ p1[7] ^= p2[7] ^ p3[7] ^ p4[7] ^ p5[7];
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ p4 += 8;
+ p5 += 8;
+ } while (--lines > 0);
+ if (lines == 0)
+ goto once_more;
+}
+
+struct xor_block_template xor_block_8regs_p = {
+ .name = "8regs_prefetch",
+ .do_2 = xor_8regs_p_2,
+ .do_3 = xor_8regs_p_3,
+ .do_4 = xor_8regs_p_4,
+ .do_5 = xor_8regs_p_5,
+};
diff --git a/lib/raid/xor/xor-8regs.c b/lib/raid/xor/xor-8regs.c
new file mode 100644
index 000000000000..769f796ab2cf
--- /dev/null
+++ b/lib/raid/xor/xor-8regs.c
@@ -0,0 +1,105 @@
+// SPDX-License-Identifier: GPL-2.0-or-later
+#include <linux/raid/xor_impl.h>
+#include <asm-generic/xor.h>
+
+static void
+xor_8regs_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2)
+{
+ long lines = bytes / (sizeof (long)) / 8;
+
+ do {
+ p1[0] ^= p2[0];
+ p1[1] ^= p2[1];
+ p1[2] ^= p2[2];
+ p1[3] ^= p2[3];
+ p1[4] ^= p2[4];
+ p1[5] ^= p2[5];
+ p1[6] ^= p2[6];
+ p1[7] ^= p2[7];
+ p1 += 8;
+ p2 += 8;
+ } while (--lines > 0);
+}
+
+static void
+xor_8regs_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ long lines = bytes / (sizeof (long)) / 8;
+
+ do {
+ p1[0] ^= p2[0] ^ p3[0];
+ p1[1] ^= p2[1] ^ p3[1];
+ p1[2] ^= p2[2] ^ p3[2];
+ p1[3] ^= p2[3] ^ p3[3];
+ p1[4] ^= p2[4] ^ p3[4];
+ p1[5] ^= p2[5] ^ p3[5];
+ p1[6] ^= p2[6] ^ p3[6];
+ p1[7] ^= p2[7] ^ p3[7];
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ } while (--lines > 0);
+}
+
+static void
+xor_8regs_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ long lines = bytes / (sizeof (long)) / 8;
+
+ do {
+ p1[0] ^= p2[0] ^ p3[0] ^ p4[0];
+ p1[1] ^= p2[1] ^ p3[1] ^ p4[1];
+ p1[2] ^= p2[2] ^ p3[2] ^ p4[2];
+ p1[3] ^= p2[3] ^ p3[3] ^ p4[3];
+ p1[4] ^= p2[4] ^ p3[4] ^ p4[4];
+ p1[5] ^= p2[5] ^ p3[5] ^ p4[5];
+ p1[6] ^= p2[6] ^ p3[6] ^ p4[6];
+ p1[7] ^= p2[7] ^ p3[7] ^ p4[7];
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ p4 += 8;
+ } while (--lines > 0);
+}
+
+static void
+xor_8regs_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ long lines = bytes / (sizeof (long)) / 8;
+
+ do {
+ p1[0] ^= p2[0] ^ p3[0] ^ p4[0] ^ p5[0];
+ p1[1] ^= p2[1] ^ p3[1] ^ p4[1] ^ p5[1];
+ p1[2] ^= p2[2] ^ p3[2] ^ p4[2] ^ p5[2];
+ p1[3] ^= p2[3] ^ p3[3] ^ p4[3] ^ p5[3];
+ p1[4] ^= p2[4] ^ p3[4] ^ p4[4] ^ p5[4];
+ p1[5] ^= p2[5] ^ p3[5] ^ p4[5] ^ p5[5];
+ p1[6] ^= p2[6] ^ p3[6] ^ p4[6] ^ p5[6];
+ p1[7] ^= p2[7] ^ p3[7] ^ p4[7] ^ p5[7];
+ p1 += 8;
+ p2 += 8;
+ p3 += 8;
+ p4 += 8;
+ p5 += 8;
+ } while (--lines > 0);
+}
+
+#ifndef NO_TEMPLATE
+struct xor_block_template xor_block_8regs = {
+ .name = "8regs",
+ .do_2 = xor_8regs_2,
+ .do_3 = xor_8regs_3,
+ .do_4 = xor_8regs_4,
+ .do_5 = xor_8regs_5,
+};
+#endif /* NO_TEMPLATE */
--
2.47.3
^ permalink raw reply related
* [PATCH 09/28] xor: remove macro abuse for XOR implementation registrations
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Drop the pretty confusing historic XOR_TRY_TEMPLATES and
XOR_SELECT_TEMPLATE, and instead let the architectures provide a
arch_xor_init that calls either xor_register to register candidates
or xor_force to force a specific implementation.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
arch/alpha/include/asm/xor.h | 29 ++++++++++++----------
arch/arm/include/asm/xor.h | 25 +++++++++----------
arch/arm64/include/asm/xor.h | 18 +++++++-------
arch/loongarch/include/asm/xor.h | 42 ++++++++++++--------------------
arch/powerpc/include/asm/xor.h | 31 ++++++++++-------------
arch/riscv/include/asm/xor.h | 19 ++++++++-------
arch/s390/include/asm/xor.h | 12 ++++-----
arch/sparc/include/asm/xor_32.h | 14 +++++------
arch/sparc/include/asm/xor_64.h | 31 +++++++++++------------
arch/x86/include/asm/xor.h | 3 ---
arch/x86/include/asm/xor_32.h | 36 ++++++++++++++-------------
arch/x86/include/asm/xor_64.h | 18 ++++++++------
arch/x86/include/asm/xor_avx.h | 9 -------
include/asm-generic/xor.h | 8 ------
include/linux/raid/xor_impl.h | 5 ++++
lib/raid/xor/xor-core.c | 41 +++++++++++++++++++++++--------
16 files changed, 168 insertions(+), 173 deletions(-)
diff --git a/arch/alpha/include/asm/xor.h b/arch/alpha/include/asm/xor.h
index e0de0c233ab9..4c8085711df1 100644
--- a/arch/alpha/include/asm/xor.h
+++ b/arch/alpha/include/asm/xor.h
@@ -851,16 +851,19 @@ static struct xor_block_template xor_block_alpha_prefetch = {
/* For grins, also test the generic routines. */
#include <asm-generic/xor.h>
-#undef XOR_TRY_TEMPLATES
-#define XOR_TRY_TEMPLATES \
- do { \
- xor_speed(&xor_block_8regs); \
- xor_speed(&xor_block_32regs); \
- xor_speed(&xor_block_alpha); \
- xor_speed(&xor_block_alpha_prefetch); \
- } while (0)
-
-/* Force the use of alpha_prefetch if EV6, as it is significantly
- faster in the cold cache case. */
-#define XOR_SELECT_TEMPLATE(FASTEST) \
- (implver() == IMPLVER_EV6 ? &xor_block_alpha_prefetch : FASTEST)
+/*
+ * Force the use of alpha_prefetch if EV6, as it is significantly faster in the
+ * cold cache case.
+ */
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ if (implver() == IMPLVER_EV6) {
+ xor_force(&xor_block_alpha_prefetch);
+ } else {
+ xor_register(&xor_block_8regs);
+ xor_register(&xor_block_32regs);
+ xor_register(&xor_block_alpha);
+ xor_register(&xor_block_alpha_prefetch);
+ }
+}
diff --git a/arch/arm/include/asm/xor.h b/arch/arm/include/asm/xor.h
index bca2a6514746..b2dcd49186e2 100644
--- a/arch/arm/include/asm/xor.h
+++ b/arch/arm/include/asm/xor.h
@@ -138,15 +138,6 @@ static struct xor_block_template xor_block_arm4regs = {
.do_5 = xor_arm4regs_5,
};
-#undef XOR_TRY_TEMPLATES
-#define XOR_TRY_TEMPLATES \
- do { \
- xor_speed(&xor_block_arm4regs); \
- xor_speed(&xor_block_8regs); \
- xor_speed(&xor_block_32regs); \
- NEON_TEMPLATES; \
- } while (0)
-
#ifdef CONFIG_KERNEL_MODE_NEON
extern struct xor_block_template const xor_block_neon_inner;
@@ -201,8 +192,16 @@ static struct xor_block_template xor_block_neon = {
.do_5 = xor_neon_5
};
-#define NEON_TEMPLATES \
- do { if (cpu_has_neon()) xor_speed(&xor_block_neon); } while (0)
-#else
-#define NEON_TEMPLATES
+#endif /* CONFIG_KERNEL_MODE_NEON */
+
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ xor_register(&xor_block_arm4regs);
+ xor_register(&xor_block_8regs);
+ xor_register(&xor_block_32regs);
+#ifdef CONFIG_KERNEL_MODE_NEON
+ if (cpu_has_neon())
+ xor_register(&xor_block_neon);
#endif
+}
diff --git a/arch/arm64/include/asm/xor.h b/arch/arm64/include/asm/xor.h
index bb7428d4ebc6..3cee1eb86371 100644
--- a/arch/arm64/include/asm/xor.h
+++ b/arch/arm64/include/asm/xor.h
@@ -60,14 +60,14 @@ static struct xor_block_template xor_block_arm64 = {
.do_4 = xor_neon_4,
.do_5 = xor_neon_5
};
-#undef XOR_TRY_TEMPLATES
-#define XOR_TRY_TEMPLATES \
- do { \
- xor_speed(&xor_block_8regs); \
- xor_speed(&xor_block_32regs); \
- if (cpu_has_neon()) { \
- xor_speed(&xor_block_arm64);\
- } \
- } while (0)
+
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ xor_register(&xor_block_8regs);
+ xor_register(&xor_block_32regs);
+ if (cpu_has_neon())
+ xor_register(&xor_block_arm64);
+}
#endif /* ! CONFIG_KERNEL_MODE_NEON */
diff --git a/arch/loongarch/include/asm/xor.h b/arch/loongarch/include/asm/xor.h
index 12467fffee46..d17c0e3b047f 100644
--- a/arch/loongarch/include/asm/xor.h
+++ b/arch/loongarch/include/asm/xor.h
@@ -16,14 +16,6 @@ static struct xor_block_template xor_block_lsx = {
.do_4 = xor_lsx_4,
.do_5 = xor_lsx_5,
};
-
-#define XOR_SPEED_LSX() \
- do { \
- if (cpu_has_lsx) \
- xor_speed(&xor_block_lsx); \
- } while (0)
-#else /* CONFIG_CPU_HAS_LSX */
-#define XOR_SPEED_LSX()
#endif /* CONFIG_CPU_HAS_LSX */
#ifdef CONFIG_CPU_HAS_LASX
@@ -34,14 +26,6 @@ static struct xor_block_template xor_block_lasx = {
.do_4 = xor_lasx_4,
.do_5 = xor_lasx_5,
};
-
-#define XOR_SPEED_LASX() \
- do { \
- if (cpu_has_lasx) \
- xor_speed(&xor_block_lasx); \
- } while (0)
-#else /* CONFIG_CPU_HAS_LASX */
-#define XOR_SPEED_LASX()
#endif /* CONFIG_CPU_HAS_LASX */
/*
@@ -54,15 +38,21 @@ static struct xor_block_template xor_block_lasx = {
*/
#include <asm-generic/xor.h>
-#undef XOR_TRY_TEMPLATES
-#define XOR_TRY_TEMPLATES \
-do { \
- xor_speed(&xor_block_8regs); \
- xor_speed(&xor_block_8regs_p); \
- xor_speed(&xor_block_32regs); \
- xor_speed(&xor_block_32regs_p); \
- XOR_SPEED_LSX(); \
- XOR_SPEED_LASX(); \
-} while (0)
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ xor_register(&xor_block_8regs);
+ xor_register(&xor_block_8regs_p);
+ xor_register(&xor_block_32regs);
+ xor_register(&xor_block_32regs_p);
+#ifdef CONFIG_CPU_HAS_LSX
+ if (cpu_has_lsx)
+ xor_register(&xor_block_lsx);
+#endif
+#ifdef CONFIG_CPU_HAS_LASX
+ if (cpu_has_lasx)
+ xor_register(&xor_block_lasx);
+#endif
+}
#endif /* _ASM_LOONGARCH_XOR_H */
diff --git a/arch/powerpc/include/asm/xor.h b/arch/powerpc/include/asm/xor.h
index 37d05c11d09c..30224c5279c4 100644
--- a/arch/powerpc/include/asm/xor.h
+++ b/arch/powerpc/include/asm/xor.h
@@ -21,27 +21,22 @@ static struct xor_block_template xor_block_altivec = {
.do_4 = xor_altivec_4,
.do_5 = xor_altivec_5,
};
-
-#define XOR_SPEED_ALTIVEC() \
- do { \
- if (cpu_has_feature(CPU_FTR_ALTIVEC)) \
- xor_speed(&xor_block_altivec); \
- } while (0)
-#else
-#define XOR_SPEED_ALTIVEC()
-#endif
+#endif /* CONFIG_ALTIVEC */
/* Also try the generic routines. */
#include <asm-generic/xor.h>
-#undef XOR_TRY_TEMPLATES
-#define XOR_TRY_TEMPLATES \
-do { \
- xor_speed(&xor_block_8regs); \
- xor_speed(&xor_block_8regs_p); \
- xor_speed(&xor_block_32regs); \
- xor_speed(&xor_block_32regs_p); \
- XOR_SPEED_ALTIVEC(); \
-} while (0)
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ xor_register(&xor_block_8regs);
+ xor_register(&xor_block_8regs_p);
+ xor_register(&xor_block_32regs);
+ xor_register(&xor_block_32regs_p);
+#ifdef CONFIG_ALTIVEC
+ if (cpu_has_feature(CPU_FTR_ALTIVEC))
+ xor_register(&xor_block_altivec);
+#endif
+}
#endif /* _ASM_POWERPC_XOR_H */
diff --git a/arch/riscv/include/asm/xor.h b/arch/riscv/include/asm/xor.h
index 96011861e46b..ed5f27903efc 100644
--- a/arch/riscv/include/asm/xor.h
+++ b/arch/riscv/include/asm/xor.h
@@ -55,14 +55,15 @@ static struct xor_block_template xor_block_rvv = {
.do_4 = xor_vector_4,
.do_5 = xor_vector_5
};
+#endif /* CONFIG_RISCV_ISA_V */
-#undef XOR_TRY_TEMPLATES
-#define XOR_TRY_TEMPLATES \
- do { \
- xor_speed(&xor_block_8regs); \
- xor_speed(&xor_block_32regs); \
- if (has_vector()) { \
- xor_speed(&xor_block_rvv);\
- } \
- } while (0)
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ xor_register(&xor_block_8regs);
+ xor_register(&xor_block_32regs);
+#ifdef CONFIG_RISCV_ISA_V
+ if (has_vector())
+ xor_register(&xor_block_rvv);
#endif
+}
diff --git a/arch/s390/include/asm/xor.h b/arch/s390/include/asm/xor.h
index 857d6759b67f..4e2233f64da9 100644
--- a/arch/s390/include/asm/xor.h
+++ b/arch/s390/include/asm/xor.h
@@ -10,12 +10,10 @@
extern struct xor_block_template xor_block_xc;
-#undef XOR_TRY_TEMPLATES
-#define XOR_TRY_TEMPLATES \
-do { \
- xor_speed(&xor_block_xc); \
-} while (0)
-
-#define XOR_SELECT_TEMPLATE(FASTEST) (&xor_block_xc)
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ xor_force(&xor_block_xc);
+}
#endif /* _ASM_S390_XOR_H */
diff --git a/arch/sparc/include/asm/xor_32.h b/arch/sparc/include/asm/xor_32.h
index 0351813cf3af..8fbf0c07ec28 100644
--- a/arch/sparc/include/asm/xor_32.h
+++ b/arch/sparc/include/asm/xor_32.h
@@ -259,10 +259,10 @@ static struct xor_block_template xor_block_SPARC = {
/* For grins, also test the generic routines. */
#include <asm-generic/xor.h>
-#undef XOR_TRY_TEMPLATES
-#define XOR_TRY_TEMPLATES \
- do { \
- xor_speed(&xor_block_8regs); \
- xor_speed(&xor_block_32regs); \
- xor_speed(&xor_block_SPARC); \
- } while (0)
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ xor_register(&xor_block_8regs);
+ xor_register(&xor_block_32regs);
+ xor_register(&xor_block_SPARC);
+}
diff --git a/arch/sparc/include/asm/xor_64.h b/arch/sparc/include/asm/xor_64.h
index caaddea8ad79..e0482ecc0a68 100644
--- a/arch/sparc/include/asm/xor_64.h
+++ b/arch/sparc/include/asm/xor_64.h
@@ -60,20 +60,17 @@ static struct xor_block_template xor_block_niagara = {
.do_5 = xor_niagara_5,
};
-#undef XOR_TRY_TEMPLATES
-#define XOR_TRY_TEMPLATES \
- do { \
- xor_speed(&xor_block_VIS); \
- xor_speed(&xor_block_niagara); \
- } while (0)
-
-/* For VIS for everything except Niagara. */
-#define XOR_SELECT_TEMPLATE(FASTEST) \
- ((tlb_type == hypervisor && \
- (sun4v_chip_type == SUN4V_CHIP_NIAGARA1 || \
- sun4v_chip_type == SUN4V_CHIP_NIAGARA2 || \
- sun4v_chip_type == SUN4V_CHIP_NIAGARA3 || \
- sun4v_chip_type == SUN4V_CHIP_NIAGARA4 || \
- sun4v_chip_type == SUN4V_CHIP_NIAGARA5)) ? \
- &xor_block_niagara : \
- &xor_block_VIS)
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ /* Force VIS for everything except Niagara. */
+ if (tlb_type == hypervisor &&
+ (sun4v_chip_type == SUN4V_CHIP_NIAGARA1 ||
+ sun4v_chip_type == SUN4V_CHIP_NIAGARA2 ||
+ sun4v_chip_type == SUN4V_CHIP_NIAGARA3 ||
+ sun4v_chip_type == SUN4V_CHIP_NIAGARA4 ||
+ sun4v_chip_type == SUN4V_CHIP_NIAGARA5))
+ xor_force(&xor_block_niagara);
+ else
+ xor_force(&xor_block_VIS);
+}
diff --git a/arch/x86/include/asm/xor.h b/arch/x86/include/asm/xor.h
index 7b0307acc410..33f5620d8d69 100644
--- a/arch/x86/include/asm/xor.h
+++ b/arch/x86/include/asm/xor.h
@@ -496,7 +496,4 @@ static struct xor_block_template xor_block_sse_pf64 = {
# include <asm/xor_64.h>
#endif
-#define XOR_SELECT_TEMPLATE(FASTEST) \
- AVX_SELECT(FASTEST)
-
#endif /* _ASM_X86_XOR_H */
diff --git a/arch/x86/include/asm/xor_32.h b/arch/x86/include/asm/xor_32.h
index 7a6b9474591e..ee32d08c27bc 100644
--- a/arch/x86/include/asm/xor_32.h
+++ b/arch/x86/include/asm/xor_32.h
@@ -552,22 +552,24 @@ static struct xor_block_template xor_block_pIII_sse = {
/* We force the use of the SSE xor block because it can write around L2.
We may also be able to load into the L1 only depending on how the cpu
deals with a load to a line that is being prefetched. */
-#undef XOR_TRY_TEMPLATES
-#define XOR_TRY_TEMPLATES \
-do { \
- AVX_XOR_SPEED; \
- if (boot_cpu_has(X86_FEATURE_XMM)) { \
- xor_speed(&xor_block_pIII_sse); \
- xor_speed(&xor_block_sse_pf64); \
- } else if (boot_cpu_has(X86_FEATURE_MMX)) { \
- xor_speed(&xor_block_pII_mmx); \
- xor_speed(&xor_block_p5_mmx); \
- } else { \
- xor_speed(&xor_block_8regs); \
- xor_speed(&xor_block_8regs_p); \
- xor_speed(&xor_block_32regs); \
- xor_speed(&xor_block_32regs_p); \
- } \
-} while (0)
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ if (boot_cpu_has(X86_FEATURE_AVX) &&
+ boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ xor_force(&xor_block_avx);
+ } else if (boot_cpu_has(X86_FEATURE_XMM)) {
+ xor_register(&xor_block_pIII_sse);
+ xor_register(&xor_block_sse_pf64);
+ } else if (boot_cpu_has(X86_FEATURE_MMX)) {
+ xor_register(&xor_block_pII_mmx);
+ xor_register(&xor_block_p5_mmx);
+ } else {
+ xor_register(&xor_block_8regs);
+ xor_register(&xor_block_8regs_p);
+ xor_register(&xor_block_32regs);
+ xor_register(&xor_block_32regs_p);
+ }
+}
#endif /* _ASM_X86_XOR_32_H */
diff --git a/arch/x86/include/asm/xor_64.h b/arch/x86/include/asm/xor_64.h
index 0307e4ec5044..2d2ceb241866 100644
--- a/arch/x86/include/asm/xor_64.h
+++ b/arch/x86/include/asm/xor_64.h
@@ -17,12 +17,16 @@ static struct xor_block_template xor_block_sse = {
/* We force the use of the SSE xor block because it can write around L2.
We may also be able to load into the L1 only depending on how the cpu
deals with a load to a line that is being prefetched. */
-#undef XOR_TRY_TEMPLATES
-#define XOR_TRY_TEMPLATES \
-do { \
- AVX_XOR_SPEED; \
- xor_speed(&xor_block_sse_pf64); \
- xor_speed(&xor_block_sse); \
-} while (0)
+#define arch_xor_init arch_xor_init
+static __always_inline void __init arch_xor_init(void)
+{
+ if (boot_cpu_has(X86_FEATURE_AVX) &&
+ boot_cpu_has(X86_FEATURE_OSXSAVE)) {
+ xor_force(&xor_block_avx);
+ } else {
+ xor_register(&xor_block_sse_pf64);
+ xor_register(&xor_block_sse);
+ }
+}
#endif /* _ASM_X86_XOR_64_H */
diff --git a/arch/x86/include/asm/xor_avx.h b/arch/x86/include/asm/xor_avx.h
index 7f81dd5897f4..c600888436bb 100644
--- a/arch/x86/include/asm/xor_avx.h
+++ b/arch/x86/include/asm/xor_avx.h
@@ -166,13 +166,4 @@ static struct xor_block_template xor_block_avx = {
.do_5 = xor_avx_5,
};
-#define AVX_XOR_SPEED \
-do { \
- if (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_OSXSAVE)) \
- xor_speed(&xor_block_avx); \
-} while (0)
-
-#define AVX_SELECT(FASTEST) \
- (boot_cpu_has(X86_FEATURE_AVX) && boot_cpu_has(X86_FEATURE_OSXSAVE) ? &xor_block_avx : FASTEST)
-
#endif
diff --git a/include/asm-generic/xor.h b/include/asm-generic/xor.h
index 44509d48fca2..79c0096aa9d9 100644
--- a/include/asm-generic/xor.h
+++ b/include/asm-generic/xor.h
@@ -728,11 +728,3 @@ static struct xor_block_template xor_block_32regs_p __maybe_unused = {
.do_4 = xor_32regs_p_4,
.do_5 = xor_32regs_p_5,
};
-
-#define XOR_TRY_TEMPLATES \
- do { \
- xor_speed(&xor_block_8regs); \
- xor_speed(&xor_block_8regs_p); \
- xor_speed(&xor_block_32regs); \
- xor_speed(&xor_block_32regs_p); \
- } while (0)
diff --git a/include/linux/raid/xor_impl.h b/include/linux/raid/xor_impl.h
index a1890cd66812..6ed4c445ab24 100644
--- a/include/linux/raid/xor_impl.h
+++ b/include/linux/raid/xor_impl.h
@@ -2,6 +2,8 @@
#ifndef _XOR_IMPL_H
#define _XOR_IMPL_H
+#include <linux/init.h>
+
struct xor_block_template {
struct xor_block_template *next;
const char *name;
@@ -22,4 +24,7 @@ struct xor_block_template {
const unsigned long * __restrict);
};
+void __init xor_register(struct xor_block_template *tmpl);
+void __init xor_force(struct xor_block_template *tmpl);
+
#endif /* _XOR_IMPL_H */
diff --git a/lib/raid/xor/xor-core.c b/lib/raid/xor/xor-core.c
index db1824011a12..93608b5fece9 100644
--- a/lib/raid/xor/xor-core.c
+++ b/lib/raid/xor/xor-core.c
@@ -14,10 +14,6 @@
#include <linux/preempt.h>
#include <asm/xor.h>
-#ifndef XOR_SELECT_TEMPLATE
-#define XOR_SELECT_TEMPLATE(x) (x)
-#endif
-
/* The xor routines to use. */
static struct xor_block_template *active_template;
@@ -55,12 +51,33 @@ EXPORT_SYMBOL(xor_blocks);
static struct xor_block_template *__initdata template_list;
static bool __initdata xor_forced = false;
-static void __init do_xor_register(struct xor_block_template *tmpl)
+/**
+ * xor_register - register a XOR template
+ * @tmpl: template to register
+ *
+ * Register a XOR implementation with the core. Registered implementations
+ * will be measured by a trivial benchmark, and the fastest one is chosen
+ * unless an implementation is forced using xor_force().
+ */
+void __init xor_register(struct xor_block_template *tmpl)
{
tmpl->next = template_list;
template_list = tmpl;
}
+/**
+ * xor_force - force use of a XOR template
+ * @tmpl: template to register
+ *
+ * Register a XOR implementation with the core and force using it. Forcing
+ * an implementation will make the core ignore any template registered using
+ * xor_register(), or any previous implementation forced using xor_force().
+ */
+void __init xor_force(struct xor_block_template *tmpl)
+{
+ active_template = tmpl;
+}
+
#define BENCH_SIZE 4096
#define REPS 800U
@@ -126,11 +143,19 @@ static int __init calibrate_xor_blocks(void)
static int __init xor_init(void)
{
+#ifdef arch_xor_init
+ arch_xor_init();
+#else
+ xor_register(&xor_block_8regs);
+ xor_register(&xor_block_8regs_p);
+ xor_register(&xor_block_32regs);
+ xor_register(&xor_block_32regs_p);
+#endif
+
/*
* If this arch/cpu has a short-circuited selection, don't loop through
* all the possible functions, just use the best one.
*/
- active_template = XOR_SELECT_TEMPLATE(NULL);
if (active_template) {
pr_info("xor: automatically using best checksumming function %-10s\n",
active_template->name);
@@ -138,10 +163,6 @@ static int __init xor_init(void)
return 0;
}
-#define xor_speed do_xor_register
- XOR_TRY_TEMPLATES;
-#undef xor_speed
-
#ifdef MODULE
return calibrate_xor_blocks();
#else
--
2.47.3
^ permalink raw reply related
* [PATCH 08/28] xor: split xor.h
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Keep xor.h for the public API, and split the struct xor_block_template
definition that is only needed by the xor.ko core and
architecture-specific optimizations into a separate xor_impl.h header.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
arch/arm/lib/xor-neon.c | 1 +
arch/arm64/lib/xor-neon.c | 1 +
arch/s390/lib/xor.c | 2 +-
include/linux/raid/xor.h | 22 +---------------------
include/linux/raid/xor_impl.h | 25 +++++++++++++++++++++++++
lib/raid/xor/xor-core.c | 1 +
6 files changed, 30 insertions(+), 22 deletions(-)
create mode 100644 include/linux/raid/xor_impl.h
diff --git a/arch/arm/lib/xor-neon.c b/arch/arm/lib/xor-neon.c
index cf57fca97908..282980b9bf2a 100644
--- a/arch/arm/lib/xor-neon.c
+++ b/arch/arm/lib/xor-neon.c
@@ -6,6 +6,7 @@
*/
#include <linux/raid/xor.h>
+#include <linux/raid/xor_impl.h>
#include <linux/module.h>
MODULE_DESCRIPTION("NEON accelerated XOR implementation");
diff --git a/arch/arm64/lib/xor-neon.c b/arch/arm64/lib/xor-neon.c
index 8fffebfa17b2..351aba92d932 100644
--- a/arch/arm64/lib/xor-neon.c
+++ b/arch/arm64/lib/xor-neon.c
@@ -7,6 +7,7 @@
*/
#include <linux/raid/xor.h>
+#include <linux/raid/xor_impl.h>
#include <linux/module.h>
#include <asm/neon-intrinsics.h>
diff --git a/arch/s390/lib/xor.c b/arch/s390/lib/xor.c
index 5363e4c2462d..3bbe21b40e66 100644
--- a/arch/s390/lib/xor.c
+++ b/arch/s390/lib/xor.c
@@ -8,7 +8,7 @@
#include <linux/types.h>
#include <linux/export.h>
-#include <linux/raid/xor.h>
+#include <linux/raid/xor_impl.h>
#include <asm/xor.h>
static void xor_xc_2(unsigned long bytes, unsigned long * __restrict p1,
diff --git a/include/linux/raid/xor.h b/include/linux/raid/xor.h
index 51b811b62322..02bda8d99534 100644
--- a/include/linux/raid/xor.h
+++ b/include/linux/raid/xor.h
@@ -7,24 +7,4 @@
extern void xor_blocks(unsigned int count, unsigned int bytes,
void *dest, void **srcs);
-struct xor_block_template {
- struct xor_block_template *next;
- const char *name;
- int speed;
- void (*do_2)(unsigned long, unsigned long * __restrict,
- const unsigned long * __restrict);
- void (*do_3)(unsigned long, unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict);
- void (*do_4)(unsigned long, unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict);
- void (*do_5)(unsigned long, unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict);
-};
-
-#endif
+#endif /* _XOR_H */
diff --git a/include/linux/raid/xor_impl.h b/include/linux/raid/xor_impl.h
new file mode 100644
index 000000000000..a1890cd66812
--- /dev/null
+++ b/include/linux/raid/xor_impl.h
@@ -0,0 +1,25 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _XOR_IMPL_H
+#define _XOR_IMPL_H
+
+struct xor_block_template {
+ struct xor_block_template *next;
+ const char *name;
+ int speed;
+ void (*do_2)(unsigned long, unsigned long * __restrict,
+ const unsigned long * __restrict);
+ void (*do_3)(unsigned long, unsigned long * __restrict,
+ const unsigned long * __restrict,
+ const unsigned long * __restrict);
+ void (*do_4)(unsigned long, unsigned long * __restrict,
+ const unsigned long * __restrict,
+ const unsigned long * __restrict,
+ const unsigned long * __restrict);
+ void (*do_5)(unsigned long, unsigned long * __restrict,
+ const unsigned long * __restrict,
+ const unsigned long * __restrict,
+ const unsigned long * __restrict,
+ const unsigned long * __restrict);
+};
+
+#endif /* _XOR_IMPL_H */
diff --git a/lib/raid/xor/xor-core.c b/lib/raid/xor/xor-core.c
index e6e593e404fb..db1824011a12 100644
--- a/lib/raid/xor/xor-core.c
+++ b/lib/raid/xor/xor-core.c
@@ -9,6 +9,7 @@
#include <linux/module.h>
#include <linux/gfp.h>
#include <linux/raid/xor.h>
+#include <linux/raid/xor_impl.h>
#include <linux/jiffies.h>
#include <linux/preempt.h>
#include <asm/xor.h>
--
2.47.3
^ permalink raw reply related
* [PATCH 07/28] xor: cleanup registration and probing
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Originally, the XOR code benchmarked all algorithms at load time, but
it has since then been hacked multiple times to allow forcing an
algorithm, and then commit 524ccdbdfb52 ("crypto: xor - defer load time
benchmark to a later time") changed the logic to a two-step process
or registration and benchmarking, but only when built-in.
Rework this, so that the XOR_TRY_TEMPLATES macro magic now always just
deals with adding the templates to the list, and benchmarking is always
done in a second pass; for modular builds from module_init, and for the
built-in case using a separate init call level.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
lib/raid/xor/xor-core.c | 98 ++++++++++++++++++++---------------------
1 file changed, 48 insertions(+), 50 deletions(-)
diff --git a/lib/raid/xor/xor-core.c b/lib/raid/xor/xor-core.c
index c54f48405c40..e6e593e404fb 100644
--- a/lib/raid/xor/xor-core.c
+++ b/lib/raid/xor/xor-core.c
@@ -52,29 +52,14 @@ EXPORT_SYMBOL(xor_blocks);
/* Set of all registered templates. */
static struct xor_block_template *__initdata template_list;
+static bool __initdata xor_forced = false;
-#ifndef MODULE
static void __init do_xor_register(struct xor_block_template *tmpl)
{
tmpl->next = template_list;
template_list = tmpl;
}
-static int __init register_xor_blocks(void)
-{
- active_template = XOR_SELECT_TEMPLATE(NULL);
-
- if (!active_template) {
-#define xor_speed do_xor_register
- // register all the templates and pick the first as the default
- XOR_TRY_TEMPLATES;
-#undef xor_speed
- active_template = template_list;
- }
- return 0;
-}
-#endif
-
#define BENCH_SIZE 4096
#define REPS 800U
@@ -85,9 +70,6 @@ do_xor_speed(struct xor_block_template *tmpl, void *b1, void *b2)
unsigned long reps;
ktime_t min, start, t0;
- tmpl->next = template_list;
- template_list = tmpl;
-
preempt_disable();
reps = 0;
@@ -111,63 +93,79 @@ do_xor_speed(struct xor_block_template *tmpl, void *b1, void *b2)
pr_info(" %-16s: %5d MB/sec\n", tmpl->name, speed);
}
-static int __init
-calibrate_xor_blocks(void)
+static int __init calibrate_xor_blocks(void)
{
void *b1, *b2;
struct xor_block_template *f, *fastest;
- fastest = XOR_SELECT_TEMPLATE(NULL);
-
- if (fastest) {
- printk(KERN_INFO "xor: automatically using best "
- "checksumming function %-10s\n",
- fastest->name);
- goto out;
- }
+ if (xor_forced)
+ return 0;
b1 = (void *) __get_free_pages(GFP_KERNEL, 2);
if (!b1) {
- printk(KERN_WARNING "xor: Yikes! No memory available.\n");
+ pr_warn("xor: Yikes! No memory available.\n");
return -ENOMEM;
}
b2 = b1 + 2*PAGE_SIZE + BENCH_SIZE;
- /*
- * If this arch/cpu has a short-circuited selection, don't loop through
- * all the possible functions, just test the best one
- */
-
-#define xor_speed(templ) do_xor_speed((templ), b1, b2)
-
- printk(KERN_INFO "xor: measuring software checksum speed\n");
- template_list = NULL;
- XOR_TRY_TEMPLATES;
+ pr_info("xor: measuring software checksum speed\n");
fastest = template_list;
- for (f = fastest; f; f = f->next)
+ for (f = template_list; f; f = f->next) {
+ do_xor_speed(f, b1, b2);
if (f->speed > fastest->speed)
fastest = f;
-
+ }
+ active_template = fastest;
pr_info("xor: using function: %s (%d MB/sec)\n",
fastest->name, fastest->speed);
+ free_pages((unsigned long)b1, 2);
+ return 0;
+}
+
+static int __init xor_init(void)
+{
+ /*
+ * If this arch/cpu has a short-circuited selection, don't loop through
+ * all the possible functions, just use the best one.
+ */
+ active_template = XOR_SELECT_TEMPLATE(NULL);
+ if (active_template) {
+ pr_info("xor: automatically using best checksumming function %-10s\n",
+ active_template->name);
+ xor_forced = true;
+ return 0;
+ }
+
+#define xor_speed do_xor_register
+ XOR_TRY_TEMPLATES;
#undef xor_speed
- free_pages((unsigned long)b1, 2);
-out:
- active_template = fastest;
+#ifdef MODULE
+ return calibrate_xor_blocks();
+#else
+ /*
+ * Pick the first template as the temporary default until calibration
+ * happens.
+ */
+ active_template = template_list;
return 0;
+#endif
}
-static __exit void xor_exit(void) { }
+static __exit void xor_exit(void)
+{
+}
MODULE_DESCRIPTION("RAID-5 checksumming functions");
MODULE_LICENSE("GPL");
+/*
+ * When built-in we must register the default template before md, but we don't
+ * want calibration to run that early as that would delay the boot process.
+ */
#ifndef MODULE
-/* when built-in xor.o must initialize before drivers/md/md.o */
-core_initcall(register_xor_blocks);
+__initcall(calibrate_xor_blocks);
#endif
-
-module_init(calibrate_xor_blocks);
+core_initcall(xor_init);
module_exit(xor_exit);
--
2.47.3
^ permalink raw reply related
* [PATCH 06/28] xor: small cleanups
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Update the to of file comment to be correct and non-redundant, and drop
the unused BH_TRACE define.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
lib/raid/xor/xor-core.c | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/lib/raid/xor/xor-core.c b/lib/raid/xor/xor-core.c
index df530ddc9f06..c54f48405c40 100644
--- a/lib/raid/xor/xor-core.c
+++ b/lib/raid/xor/xor-core.c
@@ -1,14 +1,11 @@
// SPDX-License-Identifier: GPL-2.0-or-later
/*
- * xor.c : Multiple Devices driver for Linux
- *
* Copyright (C) 1996, 1997, 1998, 1999, 2000,
* Ingo Molnar, Matti Aarnio, Jakub Jelinek, Richard Henderson.
*
- * Dispatch optimized RAID-5 checksumming functions.
+ * Dispatch optimized XOR parity functions.
*/
-#define BH_TRACE 0
#include <linux/module.h>
#include <linux/gfp.h>
#include <linux/raid/xor.h>
--
2.47.3
^ permalink raw reply related
* [PATCH 05/28] xor: move to lib/raid/
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Move the RAID XOR code to lib/raid/ as it has nothing to do with the
crypto API.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
crypto/Kconfig | 2 --
crypto/Makefile | 1 -
lib/Kconfig | 1 +
lib/Makefile | 2 +-
lib/raid/Kconfig | 4 ++++
lib/raid/Makefile | 3 +++
lib/raid/xor/Makefile | 5 +++++
crypto/xor.c => lib/raid/xor/xor-core.c | 0
8 files changed, 14 insertions(+), 4 deletions(-)
create mode 100644 lib/raid/Kconfig
create mode 100644 lib/raid/Makefile
create mode 100644 lib/raid/xor/Makefile
rename crypto/xor.c => lib/raid/xor/xor-core.c (100%)
diff --git a/crypto/Kconfig b/crypto/Kconfig
index b4bb85e8e226..97d73136a0b4 100644
--- a/crypto/Kconfig
+++ b/crypto/Kconfig
@@ -2,8 +2,6 @@
#
# Generic algorithms support
#
-config XOR_BLOCKS
- tristate
#
# async_tx api: hardware offloaded memory transfer/transform support
diff --git a/crypto/Makefile b/crypto/Makefile
index 04e269117589..795c2eea51fe 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -196,7 +196,6 @@ obj-$(CONFIG_CRYPTO_ECRDSA) += ecrdsa_generic.o
#
# generic algorithms and the async_tx api
#
-obj-$(CONFIG_XOR_BLOCKS) += xor.o
obj-$(CONFIG_ASYNC_CORE) += async_tx/
obj-$(CONFIG_ASYMMETRIC_KEY_TYPE) += asymmetric_keys/
crypto_simd-y := simd.o
diff --git a/lib/Kconfig b/lib/Kconfig
index 0f2fb9610647..5be57adcd454 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -138,6 +138,7 @@ config TRACE_MMIO_ACCESS
source "lib/crc/Kconfig"
source "lib/crypto/Kconfig"
+source "lib/raid/Kconfig"
config XXHASH
tristate
diff --git a/lib/Makefile b/lib/Makefile
index 1b9ee167517f..84da412a044f 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -120,7 +120,7 @@ endif
obj-$(CONFIG_DEBUG_INFO_REDUCED) += debug_info.o
CFLAGS_debug_info.o += $(call cc-option, -femit-struct-debug-detailed=any)
-obj-y += math/ crc/ crypto/ tests/ vdso/
+obj-y += math/ crc/ crypto/ tests/ vdso/ raid/
obj-$(CONFIG_GENERIC_IOMAP) += iomap.o
obj-$(CONFIG_HAS_IOMEM) += iomap_copy.o devres.o
diff --git a/lib/raid/Kconfig b/lib/raid/Kconfig
new file mode 100644
index 000000000000..01b73a1c303f
--- /dev/null
+++ b/lib/raid/Kconfig
@@ -0,0 +1,4 @@
+# SPDX-License-Identifier: GPL-2.0
+
+config XOR_BLOCKS
+ tristate
diff --git a/lib/raid/Makefile b/lib/raid/Makefile
new file mode 100644
index 000000000000..3540fe846dc4
--- /dev/null
+++ b/lib/raid/Makefile
@@ -0,0 +1,3 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-y += xor/
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
new file mode 100644
index 000000000000..7bca0ce8e90a
--- /dev/null
+++ b/lib/raid/xor/Makefile
@@ -0,0 +1,5 @@
+# SPDX-License-Identifier: GPL-2.0
+
+obj-$(CONFIG_XOR_BLOCKS) += xor.o
+
+xor-y += xor-core.o
diff --git a/crypto/xor.c b/lib/raid/xor/xor-core.c
similarity index 100%
rename from crypto/xor.c
rename to lib/raid/xor/xor-core.c
--
2.47.3
^ permalink raw reply related
* [PATCH 04/28] um/xor: cleanup xor.h
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Since commit c055e3eae0f1 ("crypto: xor - use ktime for template
benchmarking") the benchmarking works just fine even for TT_MODE_INFCPU,
so drop the workarounds. Note that for CPUs supporting AVX2, which
includes almost everything built in the last 10 years, the AVX2
implementation is forced anyway.
CONFIG_X86_32 is always correctly set for UM in arch/x86/um/Kconfig,
so don't override it either.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Richard Weinberger <richard@nod.at>
---
arch/um/include/asm/xor.h | 16 ----------------
1 file changed, 16 deletions(-)
diff --git a/arch/um/include/asm/xor.h b/arch/um/include/asm/xor.h
index 647fae200c5d..99e5c7e1f475 100644
--- a/arch/um/include/asm/xor.h
+++ b/arch/um/include/asm/xor.h
@@ -2,23 +2,7 @@
#ifndef _ASM_UM_XOR_H
#define _ASM_UM_XOR_H
-#ifdef CONFIG_64BIT
-#undef CONFIG_X86_32
-#define TT_CPU_INF_XOR_DEFAULT (AVX_SELECT(&xor_block_sse_pf64))
-#else
-#define CONFIG_X86_32 1
-#define TT_CPU_INF_XOR_DEFAULT (AVX_SELECT(&xor_block_8regs))
-#endif
-
#include <asm/cpufeature.h>
#include <../../x86/include/asm/xor.h>
-#include <linux/time-internal.h>
-
-#ifdef CONFIG_UML_TIME_TRAVEL_SUPPORT
-#undef XOR_SELECT_TEMPLATE
-/* pick an arbitrary one - measuring isn't possible with inf-cpu */
-#define XOR_SELECT_TEMPLATE(x) \
- (time_travel_mode == TT_MODE_INFCPU ? TT_CPU_INF_XOR_DEFAULT : x)
-#endif
#endif
--
2.47.3
^ permalink raw reply related
* [PATCH 03/28] arm64/xor: fix conflicting attributes for xor_block_template
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Commit 2c54b423cf85 ("arm64/xor: use EOR3 instructions when available")
changes the definition to __ro_after_init instead of const, but failed to
update the external declaration in xor.h. This was not found because
xor-neon.c doesn't include <asm/xor.h>, and can't easily do that due to
current architecture of the XOR code.
Fixes: 2c54b423cf85 ("arm64/xor: use EOR3 instructions when available")
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
arch/arm64/include/asm/xor.h | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/arch/arm64/include/asm/xor.h b/arch/arm64/include/asm/xor.h
index c38e3d017a79..bb7428d4ebc6 100644
--- a/arch/arm64/include/asm/xor.h
+++ b/arch/arm64/include/asm/xor.h
@@ -13,7 +13,7 @@
#ifdef CONFIG_KERNEL_MODE_NEON
-extern struct xor_block_template const xor_block_inner_neon;
+extern struct xor_block_template xor_block_inner_neon __ro_after_init;
static void
xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
--
2.47.3
^ permalink raw reply related
* [PATCH 02/28] arm/xor: remove in_interrupt() handling
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
xor_blocks can't be called from interrupt context, so remove the
handling for that.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
arch/arm/include/asm/xor.h | 41 +++++++++++---------------------------
1 file changed, 12 insertions(+), 29 deletions(-)
diff --git a/arch/arm/include/asm/xor.h b/arch/arm/include/asm/xor.h
index 934b549905f5..bca2a6514746 100644
--- a/arch/arm/include/asm/xor.h
+++ b/arch/arm/include/asm/xor.h
@@ -4,7 +4,6 @@
*
* Copyright (C) 2001 Russell King
*/
-#include <linux/hardirq.h>
#include <asm-generic/xor.h>
#include <asm/hwcap.h>
#include <asm/neon.h>
@@ -156,13 +155,9 @@ static void
xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p2)
{
- if (in_interrupt()) {
- xor_arm4regs_2(bytes, p1, p2);
- } else {
- kernel_neon_begin();
- xor_block_neon_inner.do_2(bytes, p1, p2);
- kernel_neon_end();
- }
+ kernel_neon_begin();
+ xor_block_neon_inner.do_2(bytes, p1, p2);
+ kernel_neon_end();
}
static void
@@ -170,13 +165,9 @@ xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p2,
const unsigned long * __restrict p3)
{
- if (in_interrupt()) {
- xor_arm4regs_3(bytes, p1, p2, p3);
- } else {
- kernel_neon_begin();
- xor_block_neon_inner.do_3(bytes, p1, p2, p3);
- kernel_neon_end();
- }
+ kernel_neon_begin();
+ xor_block_neon_inner.do_3(bytes, p1, p2, p3);
+ kernel_neon_end();
}
static void
@@ -185,13 +176,9 @@ xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p3,
const unsigned long * __restrict p4)
{
- if (in_interrupt()) {
- xor_arm4regs_4(bytes, p1, p2, p3, p4);
- } else {
- kernel_neon_begin();
- xor_block_neon_inner.do_4(bytes, p1, p2, p3, p4);
- kernel_neon_end();
- }
+ kernel_neon_begin();
+ xor_block_neon_inner.do_4(bytes, p1, p2, p3, p4);
+ kernel_neon_end();
}
static void
@@ -201,13 +188,9 @@ xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p4,
const unsigned long * __restrict p5)
{
- if (in_interrupt()) {
- xor_arm4regs_5(bytes, p1, p2, p3, p4, p5);
- } else {
- kernel_neon_begin();
- xor_block_neon_inner.do_5(bytes, p1, p2, p3, p4, p5);
- kernel_neon_end();
- }
+ kernel_neon_begin();
+ xor_block_neon_inner.do_5(bytes, p1, p2, p3, p4, p5);
+ kernel_neon_end();
}
static struct xor_block_template xor_block_neon = {
--
2.47.3
^ permalink raw reply related
* [PATCH 01/28] xor: assert that xor_blocks is not call from interrupt context
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260327061704.3707577-1-hch@lst.de>
Most of the optimized xor_blocks versions require FPU/vector registers,
which generally are not supported in interrupt context.
Both callers already are in user context, so enforce this at the highest
level.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
crypto/xor.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/crypto/xor.c b/crypto/xor.c
index f39621a57bb3..df530ddc9f06 100644
--- a/crypto/xor.c
+++ b/crypto/xor.c
@@ -28,6 +28,8 @@ xor_blocks(unsigned int src_count, unsigned int bytes, void *dest, void **srcs)
{
unsigned long *p1, *p2, *p3, *p4;
+ WARN_ON_ONCE(!in_task() || irqs_disabled() || softirq_count());
+
p1 = (unsigned long *) srcs[0];
if (src_count == 1) {
active_template->do_2(bytes, dest, p1);
--
2.47.3
^ permalink raw reply related
* cleanup the RAID5 XOR library v4
From: Christoph Hellwig @ 2026-03-27 6:16 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
Hi all,
the XOR library used for the RAID5 parity is a bit of a mess right now.
The main file sits in crypto/ despite not being cryptography and not
using the crypto API, with the generic implementations sitting in
include/asm-generic and the arch implementations sitting in an asm/
header in theory. The latter doesn't work for many cases, so
architectures often build the code directly into the core kernel, or
create another module for the architecture code.
Changes this to a single module in lib/ that also contains the
architecture optimizations, similar to the library work Eric Biggers
has done for the CRC and crypto libraries later. After that it changes
to better calling conventions that allow for smarter architecture
implementations (although none is contained here yet), and uses
static_call to avoid indirection function call overhead.
A git tree is also available here:
git://git.infradead.org/users/hch/misc.git xor-improvements
Gitweb:
https://git.infradead.org/?p=users/hch/misc.git;a=shortlog;h=refs/heads/xor-improvements
Changes since v3:
- switch away from lockdep_assert_preemption_enabled() again
- fix a @ reference in a kerneldoc comment.
- build the arm4regs implementation also without kernel-mode neon
support
- fix a pre-existing issue about mismatched attributes on arm64's
xor_block_inner_neon
- reject 0-sized xor request and adjust the kunit test case to not
generate them
Changes since v2:
- drop use of CONFIG_KERNEL_MODE_NEON for arm64
- drop the new __limit_random_u32_below for the unit test
- require 64-bit alignment because sparc64 requires it
- use DEFINE_STATIC_CALL_NULL to avoid exposing a specific xor_gen
routine
- keep CONFIG_XOR_BLOCKS_ARCH self-contained in lib/raid/
- don't select library option from kunit test and add a .kunitconfig
instead
- fix the module description for the kunit test
- add a case where buffers are at the end of the allocation in the kunit test
- use separate src/dst alignment in the kunit test
- fix and improve the kunit assert message
Changes since v1:
- use lockdep_assert_preemption_enabled()
- improve the commit message for the initial um xor.h cleanup
- further clean up the um arch specific header
- add SPDX identifier to the new build system files
- use bool for xor_forced
- fix an incorrect printk level conversion from warn to info
- include xor_impl.h in xor-neon.c
- remove unused exports for riscv
- simply move the sparc code instead of splititng it
- simplify the makefile for the x86-specific implementations
- remove stray references to xor_blocks in crypto/async_tx
- rework __DO_XOR_BLOCKS to avoid (theoretical) out of bounds references
- improve the kerneldoc API documentration for xor_gen()
- spell the name of the srcs argument to xor_gen correctly in xor.h
- add a kunit test, and a new random helper for it.
Diffstat:
arch/arm64/include/asm/xor.h | 73 --
arch/loongarch/include/asm/xor.h | 68 --
arch/loongarch/include/asm/xor_simd.h | 34 -
arch/loongarch/lib/xor_simd_glue.c | 72 --
arch/powerpc/include/asm/xor.h | 47 -
arch/powerpc/include/asm/xor_altivec.h | 22
arch/powerpc/lib/xor_vmx.h | 22
arch/powerpc/lib/xor_vmx_glue.c | 63 --
arch/riscv/include/asm/xor.h | 68 --
arch/s390/include/asm/xor.h | 21
arch/sparc/include/asm/xor.h | 9
arch/um/include/asm/xor.h | 24
arch/x86/include/asm/xor_64.h | 28 -
b/arch/arm/lib/Makefile | 5
b/arch/arm64/lib/Makefile | 6
b/arch/loongarch/lib/Makefile | 2
b/arch/powerpc/lib/Makefile | 5
b/arch/riscv/lib/Makefile | 1
b/arch/s390/lib/Makefile | 2
b/arch/sparc/include/asm/asm-prototypes.h | 1
b/arch/sparc/lib/Makefile | 2
b/crypto/Kconfig | 2
b/crypto/Makefile | 1
b/crypto/async_tx/async_xor.c | 34 -
b/fs/btrfs/raid56.c | 27 -
b/include/asm-generic/Kbuild | 1
b/include/linux/raid/xor.h | 27 -
b/lib/Kconfig | 1
b/lib/Makefile | 2
b/lib/raid/.kunitconfig | 3
b/lib/raid/Kconfig | 30 +
b/lib/raid/Makefile | 3
b/lib/raid/xor/Makefile | 42 +
b/lib/raid/xor/alpha/xor.c | 46 -
b/lib/raid/xor/alpha/xor_arch.h | 22
b/lib/raid/xor/arm/xor-neon-glue.c | 19
b/lib/raid/xor/arm/xor-neon.c | 22
b/lib/raid/xor/arm/xor.c | 105 ----
b/lib/raid/xor/arm/xor_arch.h | 22
b/lib/raid/xor/arm64/xor-neon-glue.c | 26 +
b/lib/raid/xor/arm64/xor-neon.c | 94 +--
b/lib/raid/xor/arm64/xor-neon.h | 6
b/lib/raid/xor/arm64/xor_arch.h | 21
b/lib/raid/xor/loongarch/xor_arch.h | 33 +
b/lib/raid/xor/loongarch/xor_simd_glue.c | 37 +
b/lib/raid/xor/powerpc/xor_arch.h | 22
b/lib/raid/xor/powerpc/xor_vmx.c | 40 -
b/lib/raid/xor/powerpc/xor_vmx.h | 10
b/lib/raid/xor/powerpc/xor_vmx_glue.c | 28 +
b/lib/raid/xor/riscv/xor-glue.c | 25 +
b/lib/raid/xor/riscv/xor.S | 4
b/lib/raid/xor/riscv/xor_arch.h | 17
b/lib/raid/xor/s390/xor.c | 15
b/lib/raid/xor/s390/xor_arch.h | 13
b/lib/raid/xor/sparc/xor-sparc32.c | 32 -
b/lib/raid/xor/sparc/xor-sparc64-glue.c | 48 -
b/lib/raid/xor/sparc/xor-sparc64.S | 10
b/lib/raid/xor/sparc/xor_arch.h | 35 +
b/lib/raid/xor/tests/Makefile | 3
b/lib/raid/xor/tests/xor_kunit.c | 187 +++++++
b/lib/raid/xor/um/xor_arch.h | 2
b/lib/raid/xor/x86/xor-avx.c | 52 --
b/lib/raid/xor/x86/xor-mmx.c | 120 +---
b/lib/raid/xor/x86/xor-sse.c | 105 +---
b/lib/raid/xor/x86/xor_arch.h | 36 +
b/lib/raid/xor/xor-32regs-prefetch.c | 267 ++++++++++
b/lib/raid/xor/xor-32regs.c | 217 ++++++++
b/lib/raid/xor/xor-8regs-prefetch.c | 146 +++++
b/lib/raid/xor/xor-8regs.c | 103 ++++
b/lib/raid/xor/xor-core.c | 193 +++++++
b/lib/raid/xor/xor_impl.h | 56 ++
crypto/xor.c | 174 -------
include/asm-generic/xor.h | 738 ------------------------------
73 files changed, 1832 insertions(+), 2067 deletions(-)
^ permalink raw reply
* Re: [PATCH 02/17] raid6: remove __KERNEL__ ifdefs
From: Christoph Hellwig @ 2026-03-26 5:25 UTC (permalink / raw)
To: Eric Biggers
Cc: H. Peter Anvin, Christoph Hellwig, Andrew Morton, Catalin Marinas,
Will Deacon, Ard Biesheuvel, Huacai Chen, WANG Xuerui,
Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy (CS GROUP), Paul Walmsley, Palmer Dabbelt,
Albert Ou, Alexandre Ghiti, Heiko Carstens, Vasily Gorbik,
Alexander Gordeev, Christian Borntraeger, Sven Schnelle,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-kernel,
linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv,
linux-s390, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260325195821.GD2305@quark>
On Wed, Mar 25, 2026 at 12:58:21PM -0700, Eric Biggers wrote:
> While I do like developing code in userspace as well, I've been doing it
> less over time as the kernel's tests and benchmarks have been improved.
> Running the KUnit tests is pretty straightforward and fast.
Yes. I would have totally subscribed to hpa's position when he initially
wrote the code, but 20+ years later things look different. In fact these
days I often write code intended for userspace in the kernel first to
benefit from lockdep primarily, but also other checkers that are in theory
available in userspace but as easy to use. Now of course lockdep doesn't
really matter for the algorithms here, but the rest still stands.
I also find the point of developing new code for new platforms
interesting: in this decade we had two new platforms added: loongarch
and riscv and all other changes were to the wiring up and not the
algorithms. And of those riscv only had the compile in userspace
support added 8 month after the algorithm, so it doesn't really look
like development was aided by it. We also plan to add new optimized
code, and getting the library in shape and dropping the hard to
maintain userspace code is actually prep work for making that not
painful.
>
> The issues with providing userspace build support in the tree are that:
>
> - It has to be maintained.
> - It's fundamentally a bit of a hack that is used only by developers
> (who always have the option of doing something locally that is
> tailored to the specific function they're working on)
> - It diverts effort from the kernel's actual test and benchmark.
>
> So while the faster iteration speed that userspace development enables
> is definitely nice, I do think we should be cautious with committing to
> maintain it in the kernel tree. If it's causing problems for the
> ongoing refactoring, dropping it for now seems reasonable to me.
>
> I would suggest adding a benchmark to the KUnit test similar to the
> crypto and CRC ones, though.
The code already has a benchmark used for runtime selection, although
that could be improved on and run for bigger data sets from kunit.
^ permalink raw reply
* Re: cleanup the RAID5 XOR library v3
From: Christoph Hellwig @ 2026-03-26 5:18 UTC (permalink / raw)
To: Eric Biggers
Cc: Christoph Hellwig, Andrew Morton, Richard Henderson, Matt Turner,
Magnus Lindholm, Russell King, Catalin Marinas, Will Deacon,
Ard Biesheuvel, Huacai Chen, WANG Xuerui, Madhavan Srinivasan,
Michael Ellerman, Nicholas Piggin, Christophe Leroy (CS GROUP),
Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, David S. Miller,
Andreas Larsson, Richard Weinberger, Anton Ivanov, Johannes Berg,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Herbert Xu, Dan Williams, Chris Mason,
David Sterba, Arnd Bergmann, Song Liu, Yu Kuai, Li Nan,
Theodore Ts'o, Jason A. Donenfeld, linux-alpha, linux-kernel,
linux-arm-kernel, loongarch, linuxppc-dev, linux-riscv,
linux-s390, sparclinux, linux-um, linux-crypto, linux-btrfs,
linux-arch, linux-raid
In-Reply-To: <20260325193954.GC2305@quark>
On Wed, Mar 25, 2026 at 12:39:54PM -0700, Eric Biggers wrote:
> This generally looks good, but yes, please check the comments from
> https://sashiko.dev/#/patchset/20260324062211.3216301-1-hch@lst.de, as
> Andrew mentioned.
Yes, I've looked into them and fixed the, the current version in the
git branch already has the changes.
> looks real as well, though I haven't tested it. If preemption is indeed
> not the right thing to check, then I guess (following up from
> https://lore.kernel.org/linux-crypto/20260303195517.GC2846@sol/) it
> would need to be something like:
>
> WARN_ON_ONCE(!in_task() || irqs_disabled() || softirq_count() != 0);
>
> Ugly, but we're running out of options.
So far I've just reverted back to the in_interrupted() check we had
before. I can switch to the above, though.
> (This sort of thing is why the functions in lib/crypto/ and lib/crc/ are
> just supported in all contexts instead. If FPU/vector/SIMD registers
> cannot be used in the current context, then a scalar fallback is used.)
We could do this fairly easily, but I'm not sure it is a good idea.
The callers of these routines are extremely limited, so we'd have to
add code for this which will then only be used by the new extensive
test code we'd have to add for it.
^ permalink raw reply
* Re: [PATCH 02/17] raid6: remove __KERNEL__ ifdefs
From: Eric Biggers @ 2026-03-25 19:58 UTC (permalink / raw)
To: H. Peter Anvin
Cc: Christoph Hellwig, Andrew Morton, Catalin Marinas, Will Deacon,
Ard Biesheuvel, Huacai Chen, WANG Xuerui, Madhavan Srinivasan,
Michael Ellerman, Nicholas Piggin, Christophe Leroy (CS GROUP),
Paul Walmsley, Palmer Dabbelt, Albert Ou, Alexandre Ghiti,
Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, Herbert Xu,
Dan Williams, Chris Mason, David Sterba, Arnd Bergmann, Song Liu,
Yu Kuai, Li Nan, linux-kernel, linux-arm-kernel, loongarch,
linuxppc-dev, linux-riscv, linux-s390, linux-crypto, linux-btrfs,
linux-arch, linux-raid
In-Reply-To: <fa11e4d3-1c70-4fa7-9f03-8772c002be6a@zytor.com>
On Wed, Mar 25, 2026 at 09:13:16AM -0700, H. Peter Anvin wrote:
> On 2026-03-25 08:13, H. Peter Anvin wrote:
> > On 2026-03-23 23:40, Christoph Hellwig wrote:
> >> With the test code ported to kernel space, none of this is required.
> >
> > I really *really* don't like this.
> >
> > The ability of running in user space is really useful when it comes to
> > developing new code for new platforms, which happens often enough for this code.
> >
>
> That being said -- and I do say this as the original author of this code --
> this should be reduced to the maximum extent possible to a (minimal) set of
> #ifndef __KERNEL__, which should be localized as much as possible. The actual
> user space components, even such a thing as a simple #include, should be moved
> to a separate user space header.
>
> But pretty please do leave the ability to debug the algorithms in user space.
> This is hard code to write and debug; it is not just about regression testing.
While I do like developing code in userspace as well, I've been doing it
less over time as the kernel's tests and benchmarks have been improved.
Running the KUnit tests is pretty straightforward and fast.
The issues with providing userspace build support in the tree are that:
- It has to be maintained.
- It's fundamentally a bit of a hack that is used only by developers
(who always have the option of doing something locally that is
tailored to the specific function they're working on)
- It diverts effort from the kernel's actual test and benchmark.
So while the faster iteration speed that userspace development enables
is definitely nice, I do think we should be cautious with committing to
maintain it in the kernel tree. If it's causing problems for the
ongoing refactoring, dropping it for now seems reasonable to me.
I would suggest adding a benchmark to the KUnit test similar to the
crypto and CRC ones, though.
- Eric
^ permalink raw reply
* Re: cleanup the RAID5 XOR library v3
From: Eric Biggers @ 2026-03-25 19:39 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Ard Biesheuvel,
Huacai Chen, WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260324062211.3216301-1-hch@lst.de>
On Tue, Mar 24, 2026 at 07:21:36AM +0100, Christoph Hellwig wrote:
> Hi all,
>
> the XOR library used for the RAID5 parity is a bit of a mess right now.
> The main file sits in crypto/ despite not being cryptography and not
> using the crypto API, with the generic implementations sitting in
> include/asm-generic and the arch implementations sitting in an asm/
> header in theory. The latter doesn't work for many cases, so
> architectures often build the code directly into the core kernel, or
> create another module for the architecture code.
>
> Changes this to a single module in lib/ that also contains the
> architecture optimizations, similar to the library work Eric Biggers
> has done for the CRC and crypto libraries later. After that it changes
> to better calling conventions that allow for smarter architecture
> implementations (although none is contained here yet), and uses
> static_call to avoid indirection function call overhead.
>
> A git tree is also available here:
>
> git://git.infradead.org/users/hch/misc.git xor-improvements
>
> Gitweb:
>
> https://git.infradead.org/?p=users/hch/misc.git;a=shortlog;h=refs/heads/xor-improvements
This generally looks good, but yes, please check the comments from
https://sashiko.dev/#/patchset/20260324062211.3216301-1-hch@lst.de, as
Andrew mentioned.
The bug where the test generates length 0 is definitely real. I
verified it causes the test to crash on some platforms.
raid_run_ops() calling xor_gen() (indirectly) with preemption disabled
looks real as well, though I haven't tested it. If preemption is indeed
not the right thing to check, then I guess (following up from
https://lore.kernel.org/linux-crypto/20260303195517.GC2846@sol/) it
would need to be something like:
WARN_ON_ONCE(!in_task() || irqs_disabled() || softirq_count() != 0);
Ugly, but we're running out of options.
(This sort of thing is why the functions in lib/crypto/ and lib/crc/ are
just supported in all contexts instead. If FPU/vector/SIMD registers
cannot be used in the current context, then a scalar fallback is used.)
- Eric
^ permalink raw reply
* Re: [PATCH 02/17] raid6: remove __KERNEL__ ifdefs
From: H. Peter Anvin @ 2026-03-25 16:13 UTC (permalink / raw)
To: Christoph Hellwig, Andrew Morton
Cc: Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, Herbert Xu, Dan Williams, Chris Mason,
David Sterba, Arnd Bergmann, Song Liu, Yu Kuai, Li Nan,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, linux-crypto, linux-btrfs, linux-arch,
linux-raid
In-Reply-To: <59d1d178-c141-4229-81e9-a6c23fa81f2f@zytor.com>
On 2026-03-25 08:13, H. Peter Anvin wrote:
> On 2026-03-23 23:40, Christoph Hellwig wrote:
>> With the test code ported to kernel space, none of this is required.
>
> I really *really* don't like this.
>
> The ability of running in user space is really useful when it comes to
> developing new code for new platforms, which happens often enough for this code.
>
That being said -- and I do say this as the original author of this code --
this should be reduced to the maximum extent possible to a (minimal) set of
#ifndef __KERNEL__, which should be localized as much as possible. The actual
user space components, even such a thing as a simple #include, should be moved
to a separate user space header.
But pretty please do leave the ability to debug the algorithms in user space.
This is hard code to write and debug; it is not just about regression testing.
-hpa
^ permalink raw reply
* Re: [PATCH 02/17] raid6: remove __KERNEL__ ifdefs
From: H. Peter Anvin @ 2026-03-25 15:13 UTC (permalink / raw)
To: Christoph Hellwig, Andrew Morton
Cc: Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, Herbert Xu, Dan Williams, Chris Mason,
David Sterba, Arnd Bergmann, Song Liu, Yu Kuai, Li Nan,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, linux-crypto, linux-btrfs, linux-arch,
linux-raid
In-Reply-To: <20260324064115.3217136-3-hch@lst.de>
On 2026-03-23 23:40, Christoph Hellwig wrote:
> With the test code ported to kernel space, none of this is required.
I really *really* don't like this.
The ability of running in user space is really useful when it comes to
developing new code for new platforms, which happens often enough for this code.
-hpa
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox