* [PATCH 25/25] xor: use static_call for xor_gen
From: Christoph Hellwig @ 2026-02-26 15:10 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Huacai Chen, WANG Xuerui,
Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy (CS GROUP), Paul Walmsley, Palmer Dabbelt,
Albert Ou, Alexandre Ghiti, Heiko Carstens, Vasily Gorbik,
Alexander Gordeev, Christian Borntraeger, Sven Schnelle,
David S. Miller, Andreas Larsson, Richard Weinberger,
Anton Ivanov, Johannes Berg, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin, Herbert Xu,
Dan Williams, Chris Mason, David Sterba, Arnd Bergmann, Song Liu,
Yu Kuai, Li Nan, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-1-hch@lst.de>
Avoid the indirect call for xor_generation by using a static_call.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
lib/raid/xor/xor-32regs.c | 2 +-
lib/raid/xor/xor-core.c | 29 ++++++++++++++---------------
lib/raid/xor/xor_impl.h | 4 ++++
3 files changed, 19 insertions(+), 16 deletions(-)
diff --git a/lib/raid/xor/xor-32regs.c b/lib/raid/xor/xor-32regs.c
index acb4a10d1e95..759a31f76414 100644
--- a/lib/raid/xor/xor-32regs.c
+++ b/lib/raid/xor/xor-32regs.c
@@ -209,7 +209,7 @@ xor_32regs_5(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
-DO_XOR_BLOCKS(32regs, xor_32regs_2, xor_32regs_3, xor_32regs_4, xor_32regs_5);
+__DO_XOR_BLOCKS(32regs, xor_32regs_2, xor_32regs_3, xor_32regs_4, xor_32regs_5);
struct xor_block_template xor_block_32regs = {
.name = "32regs",
diff --git a/lib/raid/xor/xor-core.c b/lib/raid/xor/xor-core.c
index f18dcc57004b..2ab03dd294bf 100644
--- a/lib/raid/xor/xor-core.c
+++ b/lib/raid/xor/xor-core.c
@@ -11,10 +11,14 @@
#include <linux/raid/xor.h>
#include <linux/jiffies.h>
#include <linux/preempt.h>
+#include <linux/static_call.h>
#include "xor_impl.h"
-/* The xor routine to use. */
-static struct xor_block_template *active_template;
+/*
+ * Provide a temporary default until the fastest or forced implementation is
+ * picked.
+ */
+DEFINE_STATIC_CALL(xor_gen_impl, xor_gen_32regs);
/**
* xor_gen - generate RAID-style XOR information
@@ -32,13 +36,13 @@ static struct xor_block_template *active_template;
void xor_gen(void *dest, void **srcs, unsigned int src_cnt, unsigned int bytes)
{
WARN_ON_ONCE(in_interrupt());
- active_template->xor_gen(dest, srcs, src_cnt, bytes);
+ static_call(xor_gen_impl)(dest, srcs, src_cnt, bytes);
}
EXPORT_SYMBOL(xor_gen);
/* Set of all registered templates. */
static struct xor_block_template *__initdata template_list;
-static int __initdata xor_forced = false;
+static struct xor_block_template *forced_template;
/**
* xor_register - register a XOR template
@@ -64,7 +68,7 @@ void __init xor_register(struct xor_block_template *tmpl)
*/
void __init xor_force(struct xor_block_template *tmpl)
{
- active_template = tmpl;
+ forced_template = tmpl;
}
#define BENCH_SIZE 4096
@@ -106,7 +110,7 @@ static int __init calibrate_xor_blocks(void)
void *b1, *b2;
struct xor_block_template *f, *fastest;
- if (xor_forced)
+ if (forced_template)
return 0;
b1 = (void *) __get_free_pages(GFP_KERNEL, 2);
@@ -123,7 +127,7 @@ static int __init calibrate_xor_blocks(void)
if (f->speed > fastest->speed)
fastest = f;
}
- active_template = fastest;
+ static_call_update(xor_gen_impl, fastest->xor_gen);
pr_info("xor: using function: %s (%d MB/sec)\n",
fastest->name, fastest->speed);
@@ -151,21 +155,16 @@ static int __init xor_init(void)
* If this arch/cpu has a short-circuited selection, don't loop through
* all the possible functions, just use the best one.
*/
- if (active_template) {
+ if (forced_template) {
pr_info("xor: automatically using best checksumming function %-10s\n",
- active_template->name);
- xor_forced = true;
+ forced_template->name);
+ static_call_update(xor_gen_impl, forced_template->xor_gen);
return 0;
}
#ifdef MODULE
return calibrate_xor_blocks();
#else
- /*
- * Pick the first template as the temporary default until calibration
- * happens.
- */
- active_template = template_list;
return 0;
#endif
}
diff --git a/lib/raid/xor/xor_impl.h b/lib/raid/xor/xor_impl.h
index 968dd07df627..f11910162b08 100644
--- a/lib/raid/xor/xor_impl.h
+++ b/lib/raid/xor/xor_impl.h
@@ -50,6 +50,10 @@ extern struct xor_block_template xor_block_32regs;
extern struct xor_block_template xor_block_8regs_p;
extern struct xor_block_template xor_block_32regs_p;
+/* default call until updated */
+void xor_gen_32regs(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes);
+
void __init xor_register(struct xor_block_template *tmpl);
void __init xor_force(struct xor_block_template *tmpl);
--
2.47.3
^ permalink raw reply related
* [PATCH 24/25] xor: pass the entire operation to the low-level ops
From: Christoph Hellwig @ 2026-02-26 15:10 UTC (permalink / raw)
To: Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Huacai Chen, WANG Xuerui,
Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy (CS GROUP), Paul Walmsley, Palmer Dabbelt,
Albert Ou, Alexandre Ghiti, Heiko Carstens, Vasily Gorbik,
Alexander Gordeev, Christian Borntraeger, Sven Schnelle,
David S. Miller, Andreas Larsson, Richard Weinberger,
Anton Ivanov, Johannes Berg, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin, Herbert Xu,
Dan Williams, Chris Mason, David Sterba, Arnd Bergmann, Song Liu,
Yu Kuai, Li Nan, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-1-hch@lst.de>
Currently the high-level xor code chunks up all operations into small
units for only up to 1 + 4 vectors, and passes it to four different
methods. This means the FPU/vector context is entered and left a lot
for wide stripes, and a lot of indirect expensive indirect calls are
performed. Switch to passing the entire gen_xor request to the
low-level ops, and provide a macro to dispatch it to the existing
helper.
This reduce the number of indirect calls and FPU/vector context switches
by a factor approaching nr_stripes / 4, and also reduces source and
binary code size.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
include/linux/raid/xor.h | 5 --
lib/raid/xor/alpha/xor.c | 19 ++++----
lib/raid/xor/arm/xor-neon-glue.c | 49 ++------------------
lib/raid/xor/arm/xor-neon.c | 9 +---
lib/raid/xor/arm/xor.c | 10 ++--
lib/raid/xor/arm/xor_arch.h | 3 ++
lib/raid/xor/arm64/xor-neon-glue.c | 44 ++----------------
lib/raid/xor/arm64/xor-neon.c | 20 +++++---
lib/raid/xor/arm64/xor-neon.h | 32 ++-----------
lib/raid/xor/loongarch/xor_simd_glue.c | 62 +++++--------------------
lib/raid/xor/powerpc/xor_vmx.c | 40 ++++++++--------
lib/raid/xor/powerpc/xor_vmx.h | 16 +------
lib/raid/xor/powerpc/xor_vmx_glue.c | 49 ++------------------
lib/raid/xor/riscv/xor-glue.c | 43 +++--------------
lib/raid/xor/s390/xor.c | 9 ++--
lib/raid/xor/sparc/xor-niagara-glue.c | 10 ++--
lib/raid/xor/sparc/xor-sparc32.c | 9 ++--
lib/raid/xor/sparc/xor-vis-glue.c | 9 ++--
lib/raid/xor/x86/xor-avx.c | 29 ++++--------
lib/raid/xor/x86/xor-mmx.c | 64 ++++++++++----------------
lib/raid/xor/x86/xor-sse.c | 63 +++++++++----------------
lib/raid/xor/xor-32regs-prefetch.c | 10 ++--
lib/raid/xor/xor-32regs.c | 9 ++--
lib/raid/xor/xor-8regs-prefetch.c | 11 +++--
lib/raid/xor/xor-8regs.c | 9 ++--
lib/raid/xor/xor-core.c | 47 ++-----------------
lib/raid/xor/xor_impl.h | 48 +++++++++++++------
27 files changed, 224 insertions(+), 504 deletions(-)
diff --git a/include/linux/raid/xor.h b/include/linux/raid/xor.h
index 4735a4e960f9..11620d5f5b93 100644
--- a/include/linux/raid/xor.h
+++ b/include/linux/raid/xor.h
@@ -2,11 +2,6 @@
#ifndef _XOR_H
#define _XOR_H
-#define MAX_XOR_BLOCKS 4
-
-extern void xor_blocks(unsigned int count, unsigned int bytes,
- void *dest, void **srcs);
-
void xor_gen(void *dest, void **srcss, unsigned int src_cnt,
unsigned int bytes);
diff --git a/lib/raid/xor/alpha/xor.c b/lib/raid/xor/alpha/xor.c
index 90694cc47395..a8f72f2dd3a5 100644
--- a/lib/raid/xor/alpha/xor.c
+++ b/lib/raid/xor/alpha/xor.c
@@ -832,18 +832,17 @@ xor_alpha_prefetch_5: \n\
.end xor_alpha_prefetch_5 \n\
");
+DO_XOR_BLOCKS(alpha, xor_alpha_2, xor_alpha_3, xor_alpha_4, xor_alpha_5);
+
struct xor_block_template xor_block_alpha = {
- .name = "alpha",
- .do_2 = xor_alpha_2,
- .do_3 = xor_alpha_3,
- .do_4 = xor_alpha_4,
- .do_5 = xor_alpha_5,
+ .name = "alpha",
+ .xor_gen = xor_gen_alpha,
};
+DO_XOR_BLOCKS(alpha_prefetch, xor_alpha_prefetch_2, xor_alpha_prefetch_3,
+ xor_alpha_prefetch_4, xor_alpha_prefetch_5);
+
struct xor_block_template xor_block_alpha_prefetch = {
- .name = "alpha prefetch",
- .do_2 = xor_alpha_prefetch_2,
- .do_3 = xor_alpha_prefetch_3,
- .do_4 = xor_alpha_prefetch_4,
- .do_5 = xor_alpha_prefetch_5,
+ .name = "alpha prefetch",
+ .xor_gen = xor_gen_alpha_prefetch,
};
diff --git a/lib/raid/xor/arm/xor-neon-glue.c b/lib/raid/xor/arm/xor-neon-glue.c
index 7afd6294464b..cea39e019904 100644
--- a/lib/raid/xor/arm/xor-neon-glue.c
+++ b/lib/raid/xor/arm/xor-neon-glue.c
@@ -5,54 +5,15 @@
#include "xor_impl.h"
#include "xor_arch.h"
-extern struct xor_block_template const xor_block_neon_inner;
-
-static void
-xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- kernel_neon_begin();
- xor_block_neon_inner.do_2(bytes, p1, p2);
- kernel_neon_end();
-}
-
-static void
-xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- kernel_neon_begin();
- xor_block_neon_inner.do_3(bytes, p1, p2, p3);
- kernel_neon_end();
-}
-
-static void
-xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- kernel_neon_begin();
- xor_block_neon_inner.do_4(bytes, p1, p2, p3, p4);
- kernel_neon_end();
-}
-
-static void
-xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
+static void xor_gen_neon(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes)
{
kernel_neon_begin();
- xor_block_neon_inner.do_5(bytes, p1, p2, p3, p4, p5);
+ xor_gen_neon_inner(dest, srcs, src_cnt, bytes);
kernel_neon_end();
}
struct xor_block_template xor_block_neon = {
- .name = "neon",
- .do_2 = xor_neon_2,
- .do_3 = xor_neon_3,
- .do_4 = xor_neon_4,
- .do_5 = xor_neon_5
+ .name = "neon",
+ .xor_gen = xor_gen_neon,
};
diff --git a/lib/raid/xor/arm/xor-neon.c b/lib/raid/xor/arm/xor-neon.c
index 806a42c5952c..23147e3a7904 100644
--- a/lib/raid/xor/arm/xor-neon.c
+++ b/lib/raid/xor/arm/xor-neon.c
@@ -4,6 +4,7 @@
*/
#include "xor_impl.h"
+#include "xor_arch.h"
#ifndef __ARM_NEON__
#error You should compile this file with '-march=armv7-a -mfloat-abi=softfp -mfpu=neon'
@@ -22,10 +23,4 @@
#define NO_TEMPLATE
#include "../xor-8regs.c"
-struct xor_block_template const xor_block_neon_inner = {
- .name = "__inner_neon__",
- .do_2 = xor_8regs_2,
- .do_3 = xor_8regs_3,
- .do_4 = xor_8regs_4,
- .do_5 = xor_8regs_5,
-};
+__DO_XOR_BLOCKS(neon_inner, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);
diff --git a/lib/raid/xor/arm/xor.c b/lib/raid/xor/arm/xor.c
index 5bd5f048bbe9..45139b6c55ea 100644
--- a/lib/raid/xor/arm/xor.c
+++ b/lib/raid/xor/arm/xor.c
@@ -127,10 +127,10 @@ xor_arm4regs_5(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines);
}
+DO_XOR_BLOCKS(arm4regs, xor_arm4regs_2, xor_arm4regs_3, xor_arm4regs_4,
+ xor_arm4regs_5);
+
struct xor_block_template xor_block_arm4regs = {
- .name = "arm4regs",
- .do_2 = xor_arm4regs_2,
- .do_3 = xor_arm4regs_3,
- .do_4 = xor_arm4regs_4,
- .do_5 = xor_arm4regs_5,
+ .name = "arm4regs",
+ .xor_gen = xor_gen_arm4regs,
};
diff --git a/lib/raid/xor/arm/xor_arch.h b/lib/raid/xor/arm/xor_arch.h
index 5a7eedb48fbb..775ff835df65 100644
--- a/lib/raid/xor/arm/xor_arch.h
+++ b/lib/raid/xor/arm/xor_arch.h
@@ -7,6 +7,9 @@
extern struct xor_block_template xor_block_arm4regs;
extern struct xor_block_template xor_block_neon;
+void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes);
+
static __always_inline void __init arch_xor_init(void)
{
xor_register(&xor_block_arm4regs);
diff --git a/lib/raid/xor/arm64/xor-neon-glue.c b/lib/raid/xor/arm64/xor-neon-glue.c
index 3db0a318cf5b..f0284f86feb4 100644
--- a/lib/raid/xor/arm64/xor-neon-glue.c
+++ b/lib/raid/xor/arm64/xor-neon-glue.c
@@ -10,50 +10,16 @@
#include "xor-neon.h"
#define XOR_TEMPLATE(_name) \
-static void \
-xor_##_name##_2(unsigned long bytes, unsigned long * __restrict p1, \
- const unsigned long * __restrict p2) \
+static void xor_gen_##_name(void *dest, void **srcs, unsigned int src_cnt, \
+ unsigned int bytes) \
{ \
scoped_ksimd() \
- __xor_##_name##_2(bytes, p1, p2); \
-} \
- \
-static void \
-xor_##_name##_3(unsigned long bytes, unsigned long * __restrict p1, \
- const unsigned long * __restrict p2, \
- const unsigned long * __restrict p3) \
-{ \
- scoped_ksimd() \
- __xor_##_name##_3(bytes, p1, p2, p3); \
-} \
- \
-static void \
-xor_##_name##_4(unsigned long bytes, unsigned long * __restrict p1, \
- const unsigned long * __restrict p2, \
- const unsigned long * __restrict p3, \
- const unsigned long * __restrict p4) \
-{ \
- scoped_ksimd() \
- __xor_##_name##_4(bytes, p1, p2, p3, p4); \
-} \
- \
-static void \
-xor_##_name##_5(unsigned long bytes, unsigned long * __restrict p1, \
- const unsigned long * __restrict p2, \
- const unsigned long * __restrict p3, \
- const unsigned long * __restrict p4, \
- const unsigned long * __restrict p5) \
-{ \
- scoped_ksimd() \
- __xor_##_name##_5(bytes, p1, p2, p3, p4, p5); \
+ xor_gen_##_name##_inner(dest, srcs, src_cnt, bytes); \
} \
\
struct xor_block_template xor_block_##_name = { \
- .name = __stringify(_name), \
- .do_2 = xor_##_name##_2, \
- .do_3 = xor_##_name##_3, \
- .do_4 = xor_##_name##_4, \
- .do_5 = xor_##_name##_5 \
+ .name = __stringify(_name), \
+ .xor_gen = xor_gen_##_name, \
};
XOR_TEMPLATE(neon);
diff --git a/lib/raid/xor/arm64/xor-neon.c b/lib/raid/xor/arm64/xor-neon.c
index 61f00c4fee49..97ef3cb92496 100644
--- a/lib/raid/xor/arm64/xor-neon.c
+++ b/lib/raid/xor/arm64/xor-neon.c
@@ -10,7 +10,7 @@
#include "xor_arch.h"
#include "xor-neon.h"
-void __xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
+static void __xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p2)
{
uint64_t *dp1 = (uint64_t *)p1;
@@ -37,7 +37,7 @@ void __xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
-void __xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
+static void __xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p2,
const unsigned long * __restrict p3)
{
@@ -73,7 +73,7 @@ void __xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
-void __xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
+static void __xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p2,
const unsigned long * __restrict p3,
const unsigned long * __restrict p4)
@@ -118,7 +118,7 @@ void __xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
-void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
+static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p2,
const unsigned long * __restrict p3,
const unsigned long * __restrict p4,
@@ -172,6 +172,9 @@ void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
+__DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4,
+ __xor_neon_5);
+
static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r)
{
uint64x2_t res;
@@ -182,7 +185,7 @@ static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r)
return res;
}
-void __xor_eor3_3(unsigned long bytes, unsigned long * __restrict p1,
+static void __xor_eor3_3(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p2,
const unsigned long * __restrict p3)
{
@@ -216,7 +219,7 @@ void __xor_eor3_3(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
-void __xor_eor3_4(unsigned long bytes, unsigned long * __restrict p1,
+static void __xor_eor3_4(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p2,
const unsigned long * __restrict p3,
const unsigned long * __restrict p4)
@@ -259,7 +262,7 @@ void __xor_eor3_4(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
-void __xor_eor3_5(unsigned long bytes, unsigned long * __restrict p1,
+static void __xor_eor3_5(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p2,
const unsigned long * __restrict p3,
const unsigned long * __restrict p4,
@@ -304,3 +307,6 @@ void __xor_eor3_5(unsigned long bytes, unsigned long * __restrict p1,
dp5 += 8;
} while (--lines > 0);
}
+
+__DO_XOR_BLOCKS(eor3_inner, __xor_neon_2, __xor_eor3_3, __xor_eor3_4,
+ __xor_eor3_5);
diff --git a/lib/raid/xor/arm64/xor-neon.h b/lib/raid/xor/arm64/xor-neon.h
index cec0ac846fea..514699ba8f5f 100644
--- a/lib/raid/xor/arm64/xor-neon.h
+++ b/lib/raid/xor/arm64/xor-neon.h
@@ -1,30 +1,6 @@
/* SPDX-License-Identifier: GPL-2.0-only */
-void __xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2);
-void __xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3);
-void __xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4);
-void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5);
-
-#define __xor_eor3_2 __xor_neon_2
-void __xor_eor3_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3);
-void __xor_eor3_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4);
-void __xor_eor3_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5);
+void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes);
+void xor_gen_eor3_inner(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes);
diff --git a/lib/raid/xor/loongarch/xor_simd_glue.c b/lib/raid/xor/loongarch/xor_simd_glue.c
index b387aa0213b4..7f324d924f87 100644
--- a/lib/raid/xor/loongarch/xor_simd_glue.c
+++ b/lib/raid/xor/loongarch/xor_simd_glue.c
@@ -11,63 +11,23 @@
#include "xor_arch.h"
#include "xor_simd.h"
-#define MAKE_XOR_GLUE_2(flavor) \
-static void xor_##flavor##_2(unsigned long bytes, unsigned long * __restrict p1,\
- const unsigned long * __restrict p2) \
+#define MAKE_XOR_GLUES(flavor) \
+DO_XOR_BLOCKS(flavor##_inner, __xor_##flavor##_2, __xor_##flavor##_3, \
+ __xor_##flavor##_4, __xor_##flavor##_5); \
+ \
+static void xor_gen_##flavor(void *dest, void **srcs, unsigned int src_cnt, \
+ unsigned int bytes) \
{ \
kernel_fpu_begin(); \
- __xor_##flavor##_2(bytes, p1, p2); \
+ xor_gen_##flavor##_inner(dest, srcs, src_cnt, bytes); \
kernel_fpu_end(); \
} \
-
-#define MAKE_XOR_GLUE_3(flavor) \
-static void xor_##flavor##_3(unsigned long bytes, unsigned long * __restrict p1,\
- const unsigned long * __restrict p2, \
- const unsigned long * __restrict p3) \
-{ \
- kernel_fpu_begin(); \
- __xor_##flavor##_3(bytes, p1, p2, p3); \
- kernel_fpu_end(); \
-} \
-
-#define MAKE_XOR_GLUE_4(flavor) \
-static void xor_##flavor##_4(unsigned long bytes, unsigned long * __restrict p1,\
- const unsigned long * __restrict p2, \
- const unsigned long * __restrict p3, \
- const unsigned long * __restrict p4) \
-{ \
- kernel_fpu_begin(); \
- __xor_##flavor##_4(bytes, p1, p2, p3, p4); \
- kernel_fpu_end(); \
-} \
-
-#define MAKE_XOR_GLUE_5(flavor) \
-static void xor_##flavor##_5(unsigned long bytes, unsigned long * __restrict p1,\
- const unsigned long * __restrict p2, \
- const unsigned long * __restrict p3, \
- const unsigned long * __restrict p4, \
- const unsigned long * __restrict p5) \
-{ \
- kernel_fpu_begin(); \
- __xor_##flavor##_5(bytes, p1, p2, p3, p4, p5); \
- kernel_fpu_end(); \
-} \
-
-#define MAKE_XOR_GLUES(flavor) \
- MAKE_XOR_GLUE_2(flavor); \
- MAKE_XOR_GLUE_3(flavor); \
- MAKE_XOR_GLUE_4(flavor); \
- MAKE_XOR_GLUE_5(flavor); \
- \
-struct xor_block_template xor_block_##flavor = { \
- .name = __stringify(flavor), \
- .do_2 = xor_##flavor##_2, \
- .do_3 = xor_##flavor##_3, \
- .do_4 = xor_##flavor##_4, \
- .do_5 = xor_##flavor##_5, \
+ \
+struct xor_block_template xor_block_##flavor = { \
+ .name = __stringify(flavor), \
+ .xor_gen = xor_gen_##flavor \
}
-
#ifdef CONFIG_CPU_HAS_LSX
MAKE_XOR_GLUES(lsx);
#endif /* CONFIG_CPU_HAS_LSX */
diff --git a/lib/raid/xor/powerpc/xor_vmx.c b/lib/raid/xor/powerpc/xor_vmx.c
index aab49d056d18..09bed98c1bc7 100644
--- a/lib/raid/xor/powerpc/xor_vmx.c
+++ b/lib/raid/xor/powerpc/xor_vmx.c
@@ -10,6 +10,7 @@
* Sparse (as at v0.5.0) gets very, very confused by this file.
* Make it a bit simpler for it.
*/
+#include "xor_impl.h"
#if !defined(__CHECKER__)
#include <altivec.h>
#else
@@ -49,9 +50,9 @@ typedef vector signed char unative_t;
V1##_3 = vec_xor(V1##_3, V2##_3); \
} while (0)
-void __xor_altivec_2(unsigned long bytes,
- unsigned long * __restrict v1_in,
- const unsigned long * __restrict v2_in)
+static void __xor_altivec_2(unsigned long bytes,
+ unsigned long * __restrict v1_in,
+ const unsigned long * __restrict v2_in)
{
DEFINE(v1);
DEFINE(v2);
@@ -68,10 +69,10 @@ void __xor_altivec_2(unsigned long bytes,
} while (--lines > 0);
}
-void __xor_altivec_3(unsigned long bytes,
- unsigned long * __restrict v1_in,
- const unsigned long * __restrict v2_in,
- const unsigned long * __restrict v3_in)
+static void __xor_altivec_3(unsigned long bytes,
+ unsigned long * __restrict v1_in,
+ const unsigned long * __restrict v2_in,
+ const unsigned long * __restrict v3_in)
{
DEFINE(v1);
DEFINE(v2);
@@ -92,11 +93,11 @@ void __xor_altivec_3(unsigned long bytes,
} while (--lines > 0);
}
-void __xor_altivec_4(unsigned long bytes,
- unsigned long * __restrict v1_in,
- const unsigned long * __restrict v2_in,
- const unsigned long * __restrict v3_in,
- const unsigned long * __restrict v4_in)
+static void __xor_altivec_4(unsigned long bytes,
+ unsigned long * __restrict v1_in,
+ const unsigned long * __restrict v2_in,
+ const unsigned long * __restrict v3_in,
+ const unsigned long * __restrict v4_in)
{
DEFINE(v1);
DEFINE(v2);
@@ -121,12 +122,12 @@ void __xor_altivec_4(unsigned long bytes,
} while (--lines > 0);
}
-void __xor_altivec_5(unsigned long bytes,
- unsigned long * __restrict v1_in,
- const unsigned long * __restrict v2_in,
- const unsigned long * __restrict v3_in,
- const unsigned long * __restrict v4_in,
- const unsigned long * __restrict v5_in)
+static void __xor_altivec_5(unsigned long bytes,
+ unsigned long * __restrict v1_in,
+ const unsigned long * __restrict v2_in,
+ const unsigned long * __restrict v3_in,
+ const unsigned long * __restrict v4_in,
+ const unsigned long * __restrict v5_in)
{
DEFINE(v1);
DEFINE(v2);
@@ -154,3 +155,6 @@ void __xor_altivec_5(unsigned long bytes,
v5 += 4;
} while (--lines > 0);
}
+
+__DO_XOR_BLOCKS(altivec_inner, __xor_altivec_2, __xor_altivec_3,
+ __xor_altivec_4, __xor_altivec_5);
diff --git a/lib/raid/xor/powerpc/xor_vmx.h b/lib/raid/xor/powerpc/xor_vmx.h
index 573c41d90dac..1d26c1133a86 100644
--- a/lib/raid/xor/powerpc/xor_vmx.h
+++ b/lib/raid/xor/powerpc/xor_vmx.h
@@ -6,17 +6,5 @@
* outside of the enable/disable altivec block.
*/
-void __xor_altivec_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2);
-void __xor_altivec_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3);
-void __xor_altivec_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4);
-void __xor_altivec_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5);
+void xor_gen_altivec_inner(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes);
diff --git a/lib/raid/xor/powerpc/xor_vmx_glue.c b/lib/raid/xor/powerpc/xor_vmx_glue.c
index 56e99ddfb64f..dbfbb5cadc36 100644
--- a/lib/raid/xor/powerpc/xor_vmx_glue.c
+++ b/lib/raid/xor/powerpc/xor_vmx_glue.c
@@ -12,56 +12,17 @@
#include "xor_arch.h"
#include "xor_vmx.h"
-static void xor_altivec_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
+static void xor_gen_altivec(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes)
{
preempt_disable();
enable_kernel_altivec();
- __xor_altivec_2(bytes, p1, p2);
- disable_kernel_altivec();
- preempt_enable();
-}
-
-static void xor_altivec_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- preempt_disable();
- enable_kernel_altivec();
- __xor_altivec_3(bytes, p1, p2, p3);
- disable_kernel_altivec();
- preempt_enable();
-}
-
-static void xor_altivec_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- preempt_disable();
- enable_kernel_altivec();
- __xor_altivec_4(bytes, p1, p2, p3, p4);
- disable_kernel_altivec();
- preempt_enable();
-}
-
-static void xor_altivec_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- preempt_disable();
- enable_kernel_altivec();
- __xor_altivec_5(bytes, p1, p2, p3, p4, p5);
+ xor_gen_altivec_inner(dest, srcs, src_cnt, bytes);
disable_kernel_altivec();
preempt_enable();
}
struct xor_block_template xor_block_altivec = {
- .name = "altivec",
- .do_2 = xor_altivec_2,
- .do_3 = xor_altivec_3,
- .do_4 = xor_altivec_4,
- .do_5 = xor_altivec_5,
+ .name = "altivec",
+ .xor_gen = xor_gen_altivec,
};
diff --git a/lib/raid/xor/riscv/xor-glue.c b/lib/raid/xor/riscv/xor-glue.c
index 060e5f22ebcc..2e4c1b05d998 100644
--- a/lib/raid/xor/riscv/xor-glue.c
+++ b/lib/raid/xor/riscv/xor-glue.c
@@ -9,48 +9,17 @@
#include "xor_impl.h"
#include "xor_arch.h"
-static void xor_vector_2(unsigned long bytes, unsigned long *__restrict p1,
- const unsigned long *__restrict p2)
-{
- kernel_vector_begin();
- xor_regs_2_(bytes, p1, p2);
- kernel_vector_end();
-}
-
-static void xor_vector_3(unsigned long bytes, unsigned long *__restrict p1,
- const unsigned long *__restrict p2,
- const unsigned long *__restrict p3)
-{
- kernel_vector_begin();
- xor_regs_3_(bytes, p1, p2, p3);
- kernel_vector_end();
-}
-
-static void xor_vector_4(unsigned long bytes, unsigned long *__restrict p1,
- const unsigned long *__restrict p2,
- const unsigned long *__restrict p3,
- const unsigned long *__restrict p4)
-{
- kernel_vector_begin();
- xor_regs_4_(bytes, p1, p2, p3, p4);
- kernel_vector_end();
-}
+DO_XOR_BLOCKS(vector_inner, xor_regs_2_, xor_regs_3_, xor_regs_4_, xor_regs_5_);
-static void xor_vector_5(unsigned long bytes, unsigned long *__restrict p1,
- const unsigned long *__restrict p2,
- const unsigned long *__restrict p3,
- const unsigned long *__restrict p4,
- const unsigned long *__restrict p5)
+static void xor_gen_vector(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes)
{
kernel_vector_begin();
- xor_regs_5_(bytes, p1, p2, p3, p4, p5);
+ xor_gen_vector_inner(dest, srcs, src_cnt, bytes);
kernel_vector_end();
}
struct xor_block_template xor_block_rvv = {
- .name = "rvv",
- .do_2 = xor_vector_2,
- .do_3 = xor_vector_3,
- .do_4 = xor_vector_4,
- .do_5 = xor_vector_5
+ .name = "rvv",
+ .xor_gen = xor_gen_vector,
};
diff --git a/lib/raid/xor/s390/xor.c b/lib/raid/xor/s390/xor.c
index 48b8cdc684a3..d8a62a70db6c 100644
--- a/lib/raid/xor/s390/xor.c
+++ b/lib/raid/xor/s390/xor.c
@@ -126,10 +126,9 @@ static void xor_xc_5(unsigned long bytes, unsigned long * __restrict p1,
: : "0", "cc", "memory");
}
+DO_XOR_BLOCKS(xc, xor_xc_2, xor_xc_3, xor_xc_4, xor_xc_5);
+
struct xor_block_template xor_block_xc = {
- .name = "xc",
- .do_2 = xor_xc_2,
- .do_3 = xor_xc_3,
- .do_4 = xor_xc_4,
- .do_5 = xor_xc_5,
+ .name = "xc",
+ .xor_gen = xor_gen_xc,
};
diff --git a/lib/raid/xor/sparc/xor-niagara-glue.c b/lib/raid/xor/sparc/xor-niagara-glue.c
index 92d4712c65e1..a4adb088e7d3 100644
--- a/lib/raid/xor/sparc/xor-niagara-glue.c
+++ b/lib/raid/xor/sparc/xor-niagara-glue.c
@@ -24,10 +24,10 @@ void xor_niagara_5(unsigned long bytes, unsigned long * __restrict p1,
const unsigned long * __restrict p4,
const unsigned long * __restrict p5);
+DO_XOR_BLOCKS(niagara, xor_niagara_2, xor_niagara_3, xor_niagara_4,
+ xor_niagara_5);
+
struct xor_block_template xor_block_niagara = {
- .name = "Niagara",
- .do_2 = xor_niagara_2,
- .do_3 = xor_niagara_3,
- .do_4 = xor_niagara_4,
- .do_5 = xor_niagara_5,
+ .name = "Niagara",
+ .xor_gen = xor_gen_niagara,
};
diff --git a/lib/raid/xor/sparc/xor-sparc32.c b/lib/raid/xor/sparc/xor-sparc32.c
index 307c4a84f535..fb37631e90e6 100644
--- a/lib/raid/xor/sparc/xor-sparc32.c
+++ b/lib/raid/xor/sparc/xor-sparc32.c
@@ -244,10 +244,9 @@ sparc_5(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
+DO_XOR_BLOCKS(sparc32, sparc_2, sparc_3, sparc_4, sparc_5);
+
struct xor_block_template xor_block_SPARC = {
- .name = "SPARC",
- .do_2 = sparc_2,
- .do_3 = sparc_3,
- .do_4 = sparc_4,
- .do_5 = sparc_5,
+ .name = "SPARC",
+ .xor_gen = xor_gen_sparc32,
};
diff --git a/lib/raid/xor/sparc/xor-vis-glue.c b/lib/raid/xor/sparc/xor-vis-glue.c
index 1c0977e85f53..ef39d6c8b9bb 100644
--- a/lib/raid/xor/sparc/xor-vis-glue.c
+++ b/lib/raid/xor/sparc/xor-vis-glue.c
@@ -26,10 +26,9 @@ void xor_vis_5(unsigned long bytes, unsigned long * __restrict p1,
/* XXX Ugh, write cheetah versions... -DaveM */
+DO_XOR_BLOCKS(vis, xor_vis_2, xor_vis_3, xor_vis_4, xor_vis_5);
+
struct xor_block_template xor_block_VIS = {
- .name = "VIS",
- .do_2 = xor_vis_2,
- .do_3 = xor_vis_3,
- .do_4 = xor_vis_4,
- .do_5 = xor_vis_5,
+ .name = "VIS",
+ .xor_gen = xor_gen_vis,
};
diff --git a/lib/raid/xor/x86/xor-avx.c b/lib/raid/xor/x86/xor-avx.c
index d411efa1ff43..f7777d7aa269 100644
--- a/lib/raid/xor/x86/xor-avx.c
+++ b/lib/raid/xor/x86/xor-avx.c
@@ -29,8 +29,6 @@ static void xor_avx_2(unsigned long bytes, unsigned long * __restrict p0,
{
unsigned long lines = bytes >> 9;
- kernel_fpu_begin();
-
while (lines--) {
#undef BLOCK
#define BLOCK(i, reg) \
@@ -47,8 +45,6 @@ do { \
p0 = (unsigned long *)((uintptr_t)p0 + 512);
p1 = (unsigned long *)((uintptr_t)p1 + 512);
}
-
- kernel_fpu_end();
}
static void xor_avx_3(unsigned long bytes, unsigned long * __restrict p0,
@@ -57,8 +53,6 @@ static void xor_avx_3(unsigned long bytes, unsigned long * __restrict p0,
{
unsigned long lines = bytes >> 9;
- kernel_fpu_begin();
-
while (lines--) {
#undef BLOCK
#define BLOCK(i, reg) \
@@ -78,8 +72,6 @@ do { \
p1 = (unsigned long *)((uintptr_t)p1 + 512);
p2 = (unsigned long *)((uintptr_t)p2 + 512);
}
-
- kernel_fpu_end();
}
static void xor_avx_4(unsigned long bytes, unsigned long * __restrict p0,
@@ -89,8 +81,6 @@ static void xor_avx_4(unsigned long bytes, unsigned long * __restrict p0,
{
unsigned long lines = bytes >> 9;
- kernel_fpu_begin();
-
while (lines--) {
#undef BLOCK
#define BLOCK(i, reg) \
@@ -113,8 +103,6 @@ do { \
p2 = (unsigned long *)((uintptr_t)p2 + 512);
p3 = (unsigned long *)((uintptr_t)p3 + 512);
}
-
- kernel_fpu_end();
}
static void xor_avx_5(unsigned long bytes, unsigned long * __restrict p0,
@@ -125,8 +113,6 @@ static void xor_avx_5(unsigned long bytes, unsigned long * __restrict p0,
{
unsigned long lines = bytes >> 9;
- kernel_fpu_begin();
-
while (lines--) {
#undef BLOCK
#define BLOCK(i, reg) \
@@ -152,14 +138,19 @@ do { \
p3 = (unsigned long *)((uintptr_t)p3 + 512);
p4 = (unsigned long *)((uintptr_t)p4 + 512);
}
+}
+
+DO_XOR_BLOCKS(avx_inner, xor_avx_2, xor_avx_3, xor_avx_4, xor_avx_5);
+static void xor_gen_avx(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes)
+{
+ kernel_fpu_begin();
+ xor_gen_avx_inner(dest, srcs, src_cnt, bytes);
kernel_fpu_end();
}
struct xor_block_template xor_block_avx = {
- .name = "avx",
- .do_2 = xor_avx_2,
- .do_3 = xor_avx_3,
- .do_4 = xor_avx_4,
- .do_5 = xor_avx_5,
+ .name = "avx",
+ .xor_gen = xor_gen_avx,
};
diff --git a/lib/raid/xor/x86/xor-mmx.c b/lib/raid/xor/x86/xor-mmx.c
index e48c58f92874..63a8b0444fce 100644
--- a/lib/raid/xor/x86/xor-mmx.c
+++ b/lib/raid/xor/x86/xor-mmx.c
@@ -21,8 +21,6 @@ xor_pII_mmx_2(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 7;
- kernel_fpu_begin();
-
asm volatile(
#undef BLOCK
#define BLOCK(i) \
@@ -55,8 +53,6 @@ xor_pII_mmx_2(unsigned long bytes, unsigned long * __restrict p1,
"+r" (p1), "+r" (p2)
:
: "memory");
-
- kernel_fpu_end();
}
static void
@@ -66,8 +62,6 @@ xor_pII_mmx_3(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 7;
- kernel_fpu_begin();
-
asm volatile(
#undef BLOCK
#define BLOCK(i) \
@@ -105,8 +99,6 @@ xor_pII_mmx_3(unsigned long bytes, unsigned long * __restrict p1,
"+r" (p1), "+r" (p2), "+r" (p3)
:
: "memory");
-
- kernel_fpu_end();
}
static void
@@ -117,8 +109,6 @@ xor_pII_mmx_4(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 7;
- kernel_fpu_begin();
-
asm volatile(
#undef BLOCK
#define BLOCK(i) \
@@ -161,8 +151,6 @@ xor_pII_mmx_4(unsigned long bytes, unsigned long * __restrict p1,
"+r" (p1), "+r" (p2), "+r" (p3), "+r" (p4)
:
: "memory");
-
- kernel_fpu_end();
}
@@ -175,8 +163,6 @@ xor_pII_mmx_5(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 7;
- kernel_fpu_begin();
-
/* Make sure GCC forgets anything it knows about p4 or p5,
such that it won't pass to the asm volatile below a
register that is shared with any other variable. That's
@@ -237,8 +223,6 @@ xor_pII_mmx_5(unsigned long bytes, unsigned long * __restrict p1,
Clobber them just to be sure nobody does something stupid
like assuming they have some legal value. */
asm("" : "=r" (p4), "=r" (p5));
-
- kernel_fpu_end();
}
#undef LD
@@ -255,8 +239,6 @@ xor_p5_mmx_2(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 6;
- kernel_fpu_begin();
-
asm volatile(
" .align 32 ;\n"
" 1: ;\n"
@@ -293,8 +275,6 @@ xor_p5_mmx_2(unsigned long bytes, unsigned long * __restrict p1,
"+r" (p1), "+r" (p2)
:
: "memory");
-
- kernel_fpu_end();
}
static void
@@ -304,8 +284,6 @@ xor_p5_mmx_3(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 6;
- kernel_fpu_begin();
-
asm volatile(
" .align 32,0x90 ;\n"
" 1: ;\n"
@@ -351,8 +329,6 @@ xor_p5_mmx_3(unsigned long bytes, unsigned long * __restrict p1,
"+r" (p1), "+r" (p2), "+r" (p3)
:
: "memory" );
-
- kernel_fpu_end();
}
static void
@@ -363,8 +339,6 @@ xor_p5_mmx_4(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 6;
- kernel_fpu_begin();
-
asm volatile(
" .align 32,0x90 ;\n"
" 1: ;\n"
@@ -419,8 +393,6 @@ xor_p5_mmx_4(unsigned long bytes, unsigned long * __restrict p1,
"+r" (p1), "+r" (p2), "+r" (p3), "+r" (p4)
:
: "memory");
-
- kernel_fpu_end();
}
static void
@@ -432,8 +404,6 @@ xor_p5_mmx_5(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 6;
- kernel_fpu_begin();
-
/* Make sure GCC forgets anything it knows about p4 or p5,
such that it won't pass to the asm volatile below a
register that is shared with any other variable. That's
@@ -510,22 +480,36 @@ xor_p5_mmx_5(unsigned long bytes, unsigned long * __restrict p1,
Clobber them just to be sure nobody does something stupid
like assuming they have some legal value. */
asm("" : "=r" (p4), "=r" (p5));
+}
+
+DO_XOR_BLOCKS(pII_mmx_inner, xor_pII_mmx_2, xor_pII_mmx_3, xor_pII_mmx_4,
+ xor_pII_mmx_5);
+static void xor_gen_pII_mmx(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes)
+{
+ kernel_fpu_begin();
+ xor_gen_pII_mmx_inner(dest, srcs, src_cnt, bytes);
kernel_fpu_end();
}
struct xor_block_template xor_block_pII_mmx = {
- .name = "pII_mmx",
- .do_2 = xor_pII_mmx_2,
- .do_3 = xor_pII_mmx_3,
- .do_4 = xor_pII_mmx_4,
- .do_5 = xor_pII_mmx_5,
+ .name = "pII_mmx",
+ .xor_gen = xor_gen_pII_mmx,
};
+DO_XOR_BLOCKS(p5_mmx_inner, xor_p5_mmx_2, xor_p5_mmx_3, xor_p5_mmx_4,
+ xor_p5_mmx_5);
+
+static void xor_gen_p5_mmx(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes)
+{
+ kernel_fpu_begin();
+ xor_gen_p5_mmx_inner(dest, srcs, src_cnt, bytes);
+ kernel_fpu_end();
+}
+
struct xor_block_template xor_block_p5_mmx = {
- .name = "p5_mmx",
- .do_2 = xor_p5_mmx_2,
- .do_3 = xor_p5_mmx_3,
- .do_4 = xor_p5_mmx_4,
- .do_5 = xor_p5_mmx_5,
+ .name = "p5_mmx",
+ .xor_gen = xor_gen_p5_mmx,
};
diff --git a/lib/raid/xor/x86/xor-sse.c b/lib/raid/xor/x86/xor-sse.c
index 5993ed688c15..c6626ecae6ba 100644
--- a/lib/raid/xor/x86/xor-sse.c
+++ b/lib/raid/xor/x86/xor-sse.c
@@ -51,8 +51,6 @@ xor_sse_2(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 8;
- kernel_fpu_begin();
-
asm volatile(
#undef BLOCK
#define BLOCK(i) \
@@ -93,8 +91,6 @@ xor_sse_2(unsigned long bytes, unsigned long * __restrict p1,
[p1] "+r" (p1), [p2] "+r" (p2)
: [inc] XOR_CONSTANT_CONSTRAINT (256UL)
: "memory");
-
- kernel_fpu_end();
}
static void
@@ -103,8 +99,6 @@ xor_sse_2_pf64(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 8;
- kernel_fpu_begin();
-
asm volatile(
#undef BLOCK
#define BLOCK(i) \
@@ -128,8 +122,6 @@ xor_sse_2_pf64(unsigned long bytes, unsigned long * __restrict p1,
[p1] "+r" (p1), [p2] "+r" (p2)
: [inc] XOR_CONSTANT_CONSTRAINT (256UL)
: "memory");
-
- kernel_fpu_end();
}
static void
@@ -139,8 +131,6 @@ xor_sse_3(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 8;
- kernel_fpu_begin();
-
asm volatile(
#undef BLOCK
#define BLOCK(i) \
@@ -188,8 +178,6 @@ xor_sse_3(unsigned long bytes, unsigned long * __restrict p1,
[p1] "+r" (p1), [p2] "+r" (p2), [p3] "+r" (p3)
: [inc] XOR_CONSTANT_CONSTRAINT (256UL)
: "memory");
-
- kernel_fpu_end();
}
static void
@@ -199,8 +187,6 @@ xor_sse_3_pf64(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 8;
- kernel_fpu_begin();
-
asm volatile(
#undef BLOCK
#define BLOCK(i) \
@@ -226,8 +212,6 @@ xor_sse_3_pf64(unsigned long bytes, unsigned long * __restrict p1,
[p1] "+r" (p1), [p2] "+r" (p2), [p3] "+r" (p3)
: [inc] XOR_CONSTANT_CONSTRAINT (256UL)
: "memory");
-
- kernel_fpu_end();
}
static void
@@ -238,8 +222,6 @@ xor_sse_4(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 8;
- kernel_fpu_begin();
-
asm volatile(
#undef BLOCK
#define BLOCK(i) \
@@ -294,8 +276,6 @@ xor_sse_4(unsigned long bytes, unsigned long * __restrict p1,
[p2] "+r" (p2), [p3] "+r" (p3), [p4] "+r" (p4)
: [inc] XOR_CONSTANT_CONSTRAINT (256UL)
: "memory");
-
- kernel_fpu_end();
}
static void
@@ -306,8 +286,6 @@ xor_sse_4_pf64(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 8;
- kernel_fpu_begin();
-
asm volatile(
#undef BLOCK
#define BLOCK(i) \
@@ -335,8 +313,6 @@ xor_sse_4_pf64(unsigned long bytes, unsigned long * __restrict p1,
[p2] "+r" (p2), [p3] "+r" (p3), [p4] "+r" (p4)
: [inc] XOR_CONSTANT_CONSTRAINT (256UL)
: "memory");
-
- kernel_fpu_end();
}
static void
@@ -348,8 +324,6 @@ xor_sse_5(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 8;
- kernel_fpu_begin();
-
asm volatile(
#undef BLOCK
#define BLOCK(i) \
@@ -411,8 +385,6 @@ xor_sse_5(unsigned long bytes, unsigned long * __restrict p1,
[p3] "+r" (p3), [p4] "+r" (p4), [p5] "+r" (p5)
: [inc] XOR_CONSTANT_CONSTRAINT (256UL)
: "memory");
-
- kernel_fpu_end();
}
static void
@@ -424,8 +396,6 @@ xor_sse_5_pf64(unsigned long bytes, unsigned long * __restrict p1,
{
unsigned long lines = bytes >> 8;
- kernel_fpu_begin();
-
asm volatile(
#undef BLOCK
#define BLOCK(i) \
@@ -455,22 +425,35 @@ xor_sse_5_pf64(unsigned long bytes, unsigned long * __restrict p1,
[p3] "+r" (p3), [p4] "+r" (p4), [p5] "+r" (p5)
: [inc] XOR_CONSTANT_CONSTRAINT (256UL)
: "memory");
+}
+
+DO_XOR_BLOCKS(sse_inner, xor_sse_2, xor_sse_3, xor_sse_4, xor_sse_5);
+static void xor_gen_sse(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes)
+{
+ kernel_fpu_begin();
+ xor_gen_sse_inner(dest, srcs, src_cnt, bytes);
kernel_fpu_end();
}
struct xor_block_template xor_block_sse = {
- .name = "sse",
- .do_2 = xor_sse_2,
- .do_3 = xor_sse_3,
- .do_4 = xor_sse_4,
- .do_5 = xor_sse_5,
+ .name = "sse",
+ .xor_gen = xor_gen_sse,
};
+DO_XOR_BLOCKS(sse_pf64_inner, xor_sse_2_pf64, xor_sse_3_pf64, xor_sse_4_pf64,
+ xor_sse_5_pf64);
+
+static void xor_gen_sse_pf64(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes)
+{
+ kernel_fpu_begin();
+ xor_gen_sse_pf64_inner(dest, srcs, src_cnt, bytes);
+ kernel_fpu_end();
+}
+
struct xor_block_template xor_block_sse_pf64 = {
- .name = "prefetch64-sse",
- .do_2 = xor_sse_2_pf64,
- .do_3 = xor_sse_3_pf64,
- .do_4 = xor_sse_4_pf64,
- .do_5 = xor_sse_5_pf64,
+ .name = "prefetch64-sse",
+ .xor_gen = xor_gen_sse_pf64,
};
diff --git a/lib/raid/xor/xor-32regs-prefetch.c b/lib/raid/xor/xor-32regs-prefetch.c
index 2856a8e50cb8..ade2a7d8cbe2 100644
--- a/lib/raid/xor/xor-32regs-prefetch.c
+++ b/lib/raid/xor/xor-32regs-prefetch.c
@@ -258,10 +258,10 @@ xor_32regs_p_5(unsigned long bytes, unsigned long * __restrict p1,
goto once_more;
}
+DO_XOR_BLOCKS(32regs_p, xor_32regs_p_2, xor_32regs_p_3, xor_32regs_p_4,
+ xor_32regs_p_5);
+
struct xor_block_template xor_block_32regs_p = {
- .name = "32regs_prefetch",
- .do_2 = xor_32regs_p_2,
- .do_3 = xor_32regs_p_3,
- .do_4 = xor_32regs_p_4,
- .do_5 = xor_32regs_p_5,
+ .name = "32regs_prefetch",
+ .xor_gen = xor_gen_32regs_p,
};
diff --git a/lib/raid/xor/xor-32regs.c b/lib/raid/xor/xor-32regs.c
index cc44d64032fa..acb4a10d1e95 100644
--- a/lib/raid/xor/xor-32regs.c
+++ b/lib/raid/xor/xor-32regs.c
@@ -209,10 +209,9 @@ xor_32regs_5(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
+DO_XOR_BLOCKS(32regs, xor_32regs_2, xor_32regs_3, xor_32regs_4, xor_32regs_5);
+
struct xor_block_template xor_block_32regs = {
- .name = "32regs",
- .do_2 = xor_32regs_2,
- .do_3 = xor_32regs_3,
- .do_4 = xor_32regs_4,
- .do_5 = xor_32regs_5,
+ .name = "32regs",
+ .xor_gen = xor_gen_32regs,
};
diff --git a/lib/raid/xor/xor-8regs-prefetch.c b/lib/raid/xor/xor-8regs-prefetch.c
index 1d53aec50d27..451527a951b1 100644
--- a/lib/raid/xor/xor-8regs-prefetch.c
+++ b/lib/raid/xor/xor-8regs-prefetch.c
@@ -136,10 +136,11 @@ xor_8regs_p_5(unsigned long bytes, unsigned long * __restrict p1,
goto once_more;
}
+
+DO_XOR_BLOCKS(8regs_p, xor_8regs_p_2, xor_8regs_p_3, xor_8regs_p_4,
+ xor_8regs_p_5);
+
struct xor_block_template xor_block_8regs_p = {
- .name = "8regs_prefetch",
- .do_2 = xor_8regs_p_2,
- .do_3 = xor_8regs_p_3,
- .do_4 = xor_8regs_p_4,
- .do_5 = xor_8regs_p_5,
+ .name = "8regs_prefetch",
+ .xor_gen = xor_gen_8regs_p,
};
diff --git a/lib/raid/xor/xor-8regs.c b/lib/raid/xor/xor-8regs.c
index 72a44e898c55..1edaed8acffe 100644
--- a/lib/raid/xor/xor-8regs.c
+++ b/lib/raid/xor/xor-8regs.c
@@ -94,11 +94,10 @@ xor_8regs_5(unsigned long bytes, unsigned long * __restrict p1,
}
#ifndef NO_TEMPLATE
+DO_XOR_BLOCKS(8regs, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);
+
struct xor_block_template xor_block_8regs = {
- .name = "8regs",
- .do_2 = xor_8regs_2,
- .do_3 = xor_8regs_3,
- .do_4 = xor_8regs_4,
- .do_5 = xor_8regs_5,
+ .name = "8regs",
+ .xor_gen = xor_gen_8regs,
};
#endif /* NO_TEMPLATE */
diff --git a/lib/raid/xor/xor-core.c b/lib/raid/xor/xor-core.c
index b7c29ca931ec..f18dcc57004b 100644
--- a/lib/raid/xor/xor-core.c
+++ b/lib/raid/xor/xor-core.c
@@ -13,39 +13,9 @@
#include <linux/preempt.h>
#include "xor_impl.h"
-/* The xor routines to use. */
+/* The xor routine to use. */
static struct xor_block_template *active_template;
-void
-xor_blocks(unsigned int src_count, unsigned int bytes, void *dest, void **srcs)
-{
- unsigned long *p1, *p2, *p3, *p4;
-
- WARN_ON_ONCE(in_interrupt());
-
- p1 = (unsigned long *) srcs[0];
- if (src_count == 1) {
- active_template->do_2(bytes, dest, p1);
- return;
- }
-
- p2 = (unsigned long *) srcs[1];
- if (src_count == 2) {
- active_template->do_3(bytes, dest, p1, p2);
- return;
- }
-
- p3 = (unsigned long *) srcs[2];
- if (src_count == 3) {
- active_template->do_4(bytes, dest, p1, p2, p3);
- return;
- }
-
- p4 = (unsigned long *) srcs[3];
- active_template->do_5(bytes, dest, p1, p2, p3, p4);
-}
-EXPORT_SYMBOL(xor_blocks);
-
/**
* xor_gen - generate RAID-style XOR information
* @dest: destination vector
@@ -61,16 +31,8 @@ EXPORT_SYMBOL(xor_blocks);
*/
void xor_gen(void *dest, void **srcs, unsigned int src_cnt, unsigned int bytes)
{
- unsigned int src_off = 0;
-
- while (src_cnt > 0) {
- unsigned int this_cnt = min(src_cnt, MAX_XOR_BLOCKS);
-
- xor_blocks(this_cnt, bytes, dest, srcs + src_off);
-
- src_cnt -= this_cnt;
- src_off += this_cnt;
- }
+ WARN_ON_ONCE(in_interrupt());
+ active_template->xor_gen(dest, srcs, src_cnt, bytes);
}
EXPORT_SYMBOL(xor_gen);
@@ -114,6 +76,7 @@ do_xor_speed(struct xor_block_template *tmpl, void *b1, void *b2)
int speed;
unsigned long reps;
ktime_t min, start, t0;
+ void *srcs[1] = { b2 };
preempt_disable();
@@ -124,7 +87,7 @@ do_xor_speed(struct xor_block_template *tmpl, void *b1, void *b2)
cpu_relax();
do {
mb(); /* prevent loop optimization */
- tmpl->do_2(BENCH_SIZE, b1, b2);
+ tmpl->xor_gen(b1, srcs, 1, BENCH_SIZE);
mb();
} while (reps++ < REPS || (t0 = ktime_get()) == start);
min = ktime_sub(t0, start);
diff --git a/lib/raid/xor/xor_impl.h b/lib/raid/xor/xor_impl.h
index 44b6c99e2093..968dd07df627 100644
--- a/lib/raid/xor/xor_impl.h
+++ b/lib/raid/xor/xor_impl.h
@@ -3,27 +3,47 @@
#define _XOR_IMPL_H
#include <linux/init.h>
+#include <linux/minmax.h>
struct xor_block_template {
struct xor_block_template *next;
const char *name;
int speed;
- void (*do_2)(unsigned long, unsigned long * __restrict,
- const unsigned long * __restrict);
- void (*do_3)(unsigned long, unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict);
- void (*do_4)(unsigned long, unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict);
- void (*do_5)(unsigned long, unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict,
- const unsigned long * __restrict);
+ void (*xor_gen)(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes);
};
+#define __DO_XOR_BLOCKS(_name, _handle1, _handle2, _handle3, _handle4) \
+void \
+xor_gen_##_name(void *dest, void **srcs, unsigned int src_cnt, \
+ unsigned int bytes) \
+{ \
+ unsigned int src_off = 0; \
+ \
+ while (src_cnt > 0) { \
+ unsigned int this_cnt = min(src_cnt, 4); \
+ unsigned long *p1 = (unsigned long *)srcs[src_off]; \
+ unsigned long *p2 = (unsigned long *)srcs[src_off + 1]; \
+ unsigned long *p3 = (unsigned long *)srcs[src_off + 2]; \
+ unsigned long *p4 = (unsigned long *)srcs[src_off + 3]; \
+ \
+ if (this_cnt == 1) \
+ _handle1(bytes, dest, p1); \
+ else if (this_cnt == 2) \
+ _handle2(bytes, dest, p1, p2); \
+ else if (this_cnt == 3) \
+ _handle3(bytes, dest, p1, p2, p3); \
+ else \
+ _handle4(bytes, dest, p1, p2, p3, p4); \
+ \
+ src_cnt -= this_cnt; \
+ src_off += this_cnt; \
+ } \
+}
+
+#define DO_XOR_BLOCKS(_name, _handle1, _handle2, _handle3, _handle4) \
+ static __DO_XOR_BLOCKS(_name, _handle1, _handle2, _handle3, _handle4)
+
/* generic implementations */
extern struct xor_block_template xor_block_8regs;
extern struct xor_block_template xor_block_32regs;
--
2.47.3
^ permalink raw reply related
* Re: [PATCH 09/25] xor: move generic implementations out of asm-generic/xor.h
From: Arnd Bergmann @ 2026-02-26 15:40 UTC (permalink / raw)
To: Christoph Hellwig, Andrew Morton
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Huacai Chen, WANG Xuerui,
Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy, Paul Walmsley, Palmer Dabbelt, Albert Ou,
Alexandre Ghiti, Heiko Carstens, Vasily Gorbik, Alexander Gordeev,
Christian Borntraeger, Sven Schnelle, David S . Miller,
Andreas Larsson, Richard Weinberger, Anton Ivanov, Johannes Berg,
Thomas Gleixner, Ingo Molnar, Borislav Petkov, Dave Hansen, x86,
H. Peter Anvin, Herbert Xu, Dan Williams, Chris Mason,
David Sterba, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, Linux-Arch, linux-raid
In-Reply-To: <20260226151106.144735-10-hch@lst.de>
On Thu, Feb 26, 2026, at 16:10, Christoph Hellwig wrote:
> Move the generic implementations from asm-generic/xor.h to
> per-implementaion .c files in lib/raid.
>
> Note that this would cause the second xor_block_8regs instance created by
> arch/arm/lib/xor-neon.c to be generated instead of discarded as dead
> code, so add a NO_TEMPLATE symbol to disable it for this case.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
Acked-by: Arnd Bergmann <arnd@arndb.de> # for asm-generic
>
> -#pragma GCC diagnostic ignored "-Wunused-variable"
> -#include <asm-generic/xor.h>
> +#define NO_TEMPLATE
> +#include "../../../lib/raid/xor/xor-8regs.c"
The #include is slightly ugly, but I see it gets better in a later patch,
and is clearly worth it either way.
The rest of the series looks good to me as well. I had a brief
look at each patch, but nothing to complain about.
Arnd
^ permalink raw reply
* Re: cleanup the RAID5 XOR library
From: Andrew Morton @ 2026-02-26 18:20 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Huacai Chen, WANG Xuerui,
Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy (CS GROUP), Paul Walmsley, Palmer Dabbelt,
Albert Ou, Alexandre Ghiti, Heiko Carstens, Vasily Gorbik,
Alexander Gordeev, Christian Borntraeger, Sven Schnelle,
David S. Miller, Andreas Larsson, Richard Weinberger,
Anton Ivanov, Johannes Berg, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin, Herbert Xu,
Dan Williams, Chris Mason, David Sterba, Arnd Bergmann, Song Liu,
Yu Kuai, Li Nan, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-1-hch@lst.de>
On Thu, 26 Feb 2026 07:10:12 -0800 Christoph Hellwig <hch@lst.de> wrote:
> the XOR library used for the RAID5 parity is a bit of a mess right now.
> The main file sits in crypto/ despite not being cryptography and not
> using the crypto API, with the generic implementations sitting in
> include/asm-generic and the arch implementations sitting in an asm/
> header in theory. The latter doesn't work for many cases, so
> architectures often build the code directly into the core kernel, or
> create another module for the architecture code.
>
> Changes this to a single module in lib/ that also contains the
> architecture optimizations, similar to the library work Eric Biggers
> has done for the CRC and crypto libraries later. After that it changes
> to better calling conventions that allow for smarter architecture
> implementations (although none is contained here yet), and uses
> static_call to avoid indirection function call overhead.
Thanks, I'll add this to mm.git's mm-nonmm-unstable tree for some
testing in linux-next.
^ permalink raw reply
* Re: [PATCH 03/25] um/xor: don't override XOR_SELECT_TEMPLATE
From: Richard Weinberger @ 2026-02-26 21:45 UTC (permalink / raw)
To: hch
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, will, Huacai Chen, WANG Xuerui,
Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy (CS GROUP), Paul Walmsley, Palmer Dabbelt,
Albert Ou, Alexandre Ghiti, Heiko Carstens, Vasily Gorbik,
Alexander Gordeev, Christian Borntraeger, Sven Schnelle, davem,
Andreas Larsson, anton ivanov, Johannes Berg, Thomas Gleixner,
mingo, bp, dave hansen, x86, hpa, Herbert Xu, dan j williams,
Chris Mason, David Sterba, Arnd Bergmann, Song Liu, Yu Kuai,
Li Nan, linux-alpha, linux-kernel, linux-arm-kernel, loongarch,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, linux-um,
Linux Crypto Mailing List, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-4-hch@lst.de>
----- Ursprüngliche Mail -----
> Von: "hch" <hch@lst.de>
> XOR_SELECT_TEMPLATE is only ever called with a NULL argument, so all the
> ifdef'ery doesn't do anything. With our without this, the time travel
> mode should work fine on CPUs that support AVX2, as the AVX2
> implementation is forced in this case, and won't work otherwise.
IIRC Johannes added XOR_SELECT_TEMPLATE() here to skip
the template selection logic because it didn't work with time travel mode.
Johannes, can you please test whether this change does not break
time travel mode?
Thanks,
//richard
^ permalink raw reply
* Re: [PATCH 03/25] um/xor: don't override XOR_SELECT_TEMPLATE
From: hch @ 2026-02-26 22:00 UTC (permalink / raw)
To: Richard Weinberger
Cc: hch, Andrew Morton, Richard Henderson, Matt Turner,
Magnus Lindholm, Russell King, Catalin Marinas, will, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, davem, Andreas Larsson, anton ivanov,
Johannes Berg, Thomas Gleixner, mingo, bp, dave hansen, x86, hpa,
Herbert Xu, dan j williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um,
Linux Crypto Mailing List, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <445921547.2198.1772142326749.JavaMail.zimbra@nod.at>
On Thu, Feb 26, 2026 at 10:45:26PM +0100, Richard Weinberger wrote:
> ----- Ursprüngliche Mail -----
> > Von: "hch" <hch@lst.de>
> > XOR_SELECT_TEMPLATE is only ever called with a NULL argument, so all the
> > ifdef'ery doesn't do anything. With our without this, the time travel
> > mode should work fine on CPUs that support AVX2, as the AVX2
> > implementation is forced in this case, and won't work otherwise.
>
> IIRC Johannes added XOR_SELECT_TEMPLATE() here to skip
> the template selection logic because it didn't work with time travel mode.
>
> Johannes, can you please test whether this change does not break
> time travel mode?
I'm pretty sure that was the intent, but as I wrote above it worked
and still works on AVX-supporting CPUs by chance, and already doesn't
on older CPUs, and unless my git blaming went wrong someewhere already
didn't when this was originally added.
^ permalink raw reply
* Re: [PATCH 03/25] um/xor: don't override XOR_SELECT_TEMPLATE
From: Johannes Berg @ 2026-02-27 7:39 UTC (permalink / raw)
To: Richard Weinberger, hch
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, will, Huacai Chen, WANG Xuerui,
Madhavan Srinivasan, Michael Ellerman, Nicholas Piggin,
Christophe Leroy (CS GROUP), Paul Walmsley, Palmer Dabbelt,
Albert Ou, Alexandre Ghiti, Heiko Carstens, Vasily Gorbik,
Alexander Gordeev, Christian Borntraeger, Sven Schnelle, davem,
Andreas Larsson, anton ivanov, Thomas Gleixner, mingo, bp,
dave hansen, x86, hpa, Herbert Xu, dan j williams, Chris Mason,
David Sterba, Arnd Bergmann, Song Liu, Yu Kuai, Li Nan,
linux-alpha, linux-kernel, linux-arm-kernel, loongarch,
linuxppc-dev, linux-riscv, linux-s390, sparclinux, linux-um,
Linux Crypto Mailing List, linux-btrfs, linux-arch, linux-raid,
Ard Biesheuvel
In-Reply-To: <445921547.2198.1772142326749.JavaMail.zimbra@nod.at>
On Thu, 2026-02-26 at 22:45 +0100, Richard Weinberger wrote:
> ----- Ursprüngliche Mail -----
> > Von: "hch" <hch@lst.de>
> > XOR_SELECT_TEMPLATE is only ever called with a NULL argument, so all the
> > ifdef'ery doesn't do anything. With our without this, the time travel
> > mode should work fine on CPUs that support AVX2, as the AVX2
> > implementation is forced in this case, and won't work otherwise.
>
> IIRC Johannes added XOR_SELECT_TEMPLATE() here to skip
> the template selection logic because it didn't work with time travel mode.
>
> Johannes, can you please test whether this change does not break
> time travel mode?
It does work, even if it reports nonsense (as you'd expect):
xor: measuring software checksum speed
prefetch64-sse : 12816000 MB/sec
sse : 12816000 MB/sec
xor: using function: prefetch64-sse (12816000 MB/sec)
I think it works now because the loop is using ktime and is bounded by
REPS, since commit c055e3eae0f1 ("crypto: xor - use ktime for template
benchmarking").
The RAID speed select still hangs, but we've gotten that removed via
Kconfig, so that's already handled. Perhaps raid6_choose_gen() should
use a similar algorithm? But for UML it doesn't really matter since
CONFIG_RAID6_PQ_BENCHMARK exists.
As far as AVX2 is concerned, yeah, I guess that was a bug, but evidently
nobody (who configured time-travel) ever cared - what _did_ matter
though in practice is that the boot not get stuck entirely... Two
completely separate issues.
johannes
^ permalink raw reply
* Re: [PATCH 17/25] s390: move the XOR code to lib/raid/
From: Heiko Carstens @ 2026-02-27 9:09 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Vasily Gorbik,
Alexander Gordeev, Christian Borntraeger, Sven Schnelle,
David S. Miller, Andreas Larsson, Richard Weinberger,
Anton Ivanov, Johannes Berg, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin, Herbert Xu,
Dan Williams, Chris Mason, David Sterba, Arnd Bergmann, Song Liu,
Yu Kuai, Li Nan, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-18-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:29AM -0800, Christoph Hellwig wrote:
> Move the optimized XOR into lib/raid and include it it in xor.ko
> instead of unconditionally building it into the main kernel image.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> arch/s390/lib/Makefile | 2 +-
> lib/raid/xor/Makefile | 1 +
> {arch/s390/lib => lib/raid/xor/s390}/xor.c | 2 --
> 3 files changed, 2 insertions(+), 3 deletions(-)
> rename {arch/s390/lib => lib/raid/xor/s390}/xor.c (98%)
FWIW:
Acked-by: Heiko Carstens <hca@linux.ibm.com>
However, I just had a look at the s390 implementation and just saw that the
inline assembly constraints for xor_xc_2() are incorrect. "bytes", "p1",
and "p2" are input operands, while all three of them are modified within
the inline assembly. Given that the function consists only of this inline
assembly I doubt that this causes any harm, however I still want to fix
this now; but your patch should apply fine with or without this fixed.
^ permalink raw reply
* Re: [PATCH 17/25] s390: move the XOR code to lib/raid/
From: Christoph Hellwig @ 2026-02-27 14:13 UTC (permalink / raw)
To: Heiko Carstens
Cc: Christoph Hellwig, Andrew Morton, Richard Henderson, Matt Turner,
Magnus Lindholm, Russell King, Catalin Marinas, Will Deacon,
Huacai Chen, WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Vasily Gorbik,
Alexander Gordeev, Christian Borntraeger, Sven Schnelle,
David S. Miller, Andreas Larsson, Richard Weinberger,
Anton Ivanov, Johannes Berg, Thomas Gleixner, Ingo Molnar,
Borislav Petkov, Dave Hansen, x86, H. Peter Anvin, Herbert Xu,
Dan Williams, Chris Mason, David Sterba, Arnd Bergmann, Song Liu,
Yu Kuai, Li Nan, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260227090959.10882Af7-hca@linux.ibm.com>
On Fri, Feb 27, 2026 at 10:09:59AM +0100, Heiko Carstens wrote:
> However, I just had a look at the s390 implementation and just saw that the
> inline assembly constraints for xor_xc_2() are incorrect. "bytes", "p1",
> and "p2" are input operands, while all three of them are modified within
> the inline assembly. Given that the function consists only of this inline
> assembly I doubt that this causes any harm, however I still want to fix
> this now; but your patch should apply fine with or without this fixed.
Two comments on that: I thin kin the long run simply moving the
implementation to a pure assembly file might be easier to maintain.
Also with this series you can now optimize for more than 5 stripes,
which should be the mormal case. I'll try to make sure we'll get
units tests to help with that.
^ permalink raw reply
* Re: [PATCH 01/25] xor: assert that xor_blocks is not called from interrupt context
From: Peter Zijlstra @ 2026-02-27 14:24 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-2-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:13AM -0800, Christoph Hellwig wrote:
> Most of the optimized xor_blocks versions require FPU/vector registers,
> which generally are not supported in interrupt context.
>
> Both callers already are in user context, so enforce this at the highest
> level.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> crypto/xor.c | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/crypto/xor.c b/crypto/xor.c
> index f39621a57bb3..864f3604e867 100644
> --- a/crypto/xor.c
> +++ b/crypto/xor.c
> @@ -28,6 +28,8 @@ xor_blocks(unsigned int src_count, unsigned int bytes, void *dest, void **srcs)
> {
> unsigned long *p1, *p2, *p3, *p4;
>
> + WARN_ON_ONCE(in_interrupt());
Your changelog makes it sound like you want:
WARN_ON_ONCE(!in_task());
But perhaps something like so:
lockdep_assert_preempt_enabled();
Would do? That ensures we are in preemptible context, which is much the
same. That also ensures the cost of this assertion is only paid on debug
kernels.
> p1 = (unsigned long *) srcs[0];
> if (src_count == 1) {
> active_template->do_2(bytes, dest, p1);
> --
> 2.47.3
>
^ permalink raw reply
* Re: [PATCH 18/25] x86: move the XOR code to lib/raid/
From: Peter Zijlstra @ 2026-02-27 14:30 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-19-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:30AM -0800, Christoph Hellwig wrote:
> Move the optimized XOR code out of line into lib/raid.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> arch/x86/include/asm/xor.h | 518 ++----------------
> arch/x86/include/asm/xor_64.h | 32 --
> lib/raid/xor/Makefile | 8 +
> .../xor_avx.h => lib/raid/xor/x86/xor-avx.c | 14 +-
> .../xor_32.h => lib/raid/xor/x86/xor-mmx.c | 60 +-
> lib/raid/xor/x86/xor-sse.c | 476 ++++++++++++++++
I gotta ask, why lib/raid/xor/$arch/ instead of something like
arch/$arch/lib/xor ?
^ permalink raw reply
* Re: [PATCH 25/25] xor: use static_call for xor_gen
From: Peter Zijlstra @ 2026-02-27 14:36 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-26-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:37AM -0800, Christoph Hellwig wrote:
> Avoid the indirect call for xor_generation by using a static_call.
Nice!
^ permalink raw reply
* Re: [PATCH 18/25] x86: move the XOR code to lib/raid/
From: Eric Biggers @ 2026-02-27 23:55 UTC (permalink / raw)
To: Peter Zijlstra
Cc: Christoph Hellwig, Andrew Morton, Richard Henderson, Matt Turner,
Magnus Lindholm, Russell King, Catalin Marinas, Will Deacon,
Huacai Chen, WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260227143016.GH1282955@noisy.programming.kicks-ass.net>
On Fri, Feb 27, 2026 at 03:30:16PM +0100, Peter Zijlstra wrote:
> On Thu, Feb 26, 2026 at 07:10:30AM -0800, Christoph Hellwig wrote:
> > Move the optimized XOR code out of line into lib/raid.
> >
> > Signed-off-by: Christoph Hellwig <hch@lst.de>
> > ---
> > arch/x86/include/asm/xor.h | 518 ++----------------
> > arch/x86/include/asm/xor_64.h | 32 --
> > lib/raid/xor/Makefile | 8 +
> > .../xor_avx.h => lib/raid/xor/x86/xor-avx.c | 14 +-
> > .../xor_32.h => lib/raid/xor/x86/xor-mmx.c | 60 +-
> > lib/raid/xor/x86/xor-sse.c | 476 ++++++++++++++++
>
> I gotta ask, why lib/raid/xor/$arch/ instead of something like
> arch/$arch/lib/xor ?
Similar to lib/crypto/ and lib/crc/, it allows the translation units
(either .c or .S files) containing architecture-optimized XOR code to be
included directly in the xor.ko module, where they should be.
Previously, these were always built into the core kernel even if
XOR_BLOCKS was 'n' or 'm', or they were built into a separate module
xor-neon.ko which xor.ko depended on. So either the code was included
unnecessarily, or there was an extra module.
Technically we could instead have the lib makefile compile stuff in
arch/, but that would be unusual. It's much cleaner to have the
directory structure match the build system.
If we made this code always built-in, like memcpy(), then we could put
it anywhere. But (like many of the crypto and CRC algorithms) many
kernels don't need this code, and even if they do it may be needed only
by 'm' code. So it makes sense to support tristate.
- Eric
^ permalink raw reply
* Re: [PATCH 03/25] um/xor: don't override XOR_SELECT_TEMPLATE
From: Eric Biggers @ 2026-02-28 4:30 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-4-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:15AM -0800, Christoph Hellwig wrote:
> XOR_SELECT_TEMPLATE is only ever called with a NULL argument, so all the
> ifdef'ery doesn't do anything. With our without this, the time travel
> mode should work fine on CPUs that support AVX2, as the AVX2
> implementation is forced in this case, and won't work otherwise.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> arch/um/include/asm/xor.h | 10 ----------
> 1 file changed, 10 deletions(-)
>
> diff --git a/arch/um/include/asm/xor.h b/arch/um/include/asm/xor.h
> index 647fae200c5d..c9ddedc19301 100644
> --- a/arch/um/include/asm/xor.h
> +++ b/arch/um/include/asm/xor.h
> @@ -4,21 +4,11 @@
>
> #ifdef CONFIG_64BIT
> #undef CONFIG_X86_32
> -#define TT_CPU_INF_XOR_DEFAULT (AVX_SELECT(&xor_block_sse_pf64))
> #else
> #define CONFIG_X86_32 1
> -#define TT_CPU_INF_XOR_DEFAULT (AVX_SELECT(&xor_block_8regs))
> #endif
>
> #include <asm/cpufeature.h>
> #include <../../x86/include/asm/xor.h>
> -#include <linux/time-internal.h>
> -
> -#ifdef CONFIG_UML_TIME_TRAVEL_SUPPORT
> -#undef XOR_SELECT_TEMPLATE
> -/* pick an arbitrary one - measuring isn't possible with inf-cpu */
> -#define XOR_SELECT_TEMPLATE(x) \
> - (time_travel_mode == TT_MODE_INFCPU ? TT_CPU_INF_XOR_DEFAULT : x)
> -#endif
I'm not following this change. Previously, in TT_MODE_INFCPU mode,
XOR_SELECT_TEMPLATE(NULL) returned &xor_block_avx, &xor_block_sse_pf64,
or &xor_block_8regs, causing the benchmark to be skipped. After this
change, the benchmark starts being done on CPUs that don't support AVX.
- Eric
^ permalink raw reply
* Re: [PATCH 04/25] xor: move to lib/raid/
From: Eric Biggers @ 2026-02-28 4:35 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-5-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:16AM -0800, Christoph Hellwig wrote:
> diff --git a/lib/Kconfig b/lib/Kconfig
> index 0f2fb9610647..5be57adcd454 100644
> --- a/lib/Kconfig
> +++ b/lib/Kconfig
> @@ -138,6 +138,7 @@ config TRACE_MMIO_ACCESS
>
> source "lib/crc/Kconfig"
> source "lib/crypto/Kconfig"
> +source "lib/raid/Kconfig"
This adds lib/raid/ alongside the existing lib/raid6/ directory. Is
that the intended final state, or is the intent for the code in
lib/raid6/ to eventually be moved to a subdirectory of lib/raid/
(alongside the "xor" subdirectory)?
> diff --git a/lib/raid/Kconfig b/lib/raid/Kconfig
> new file mode 100644
> index 000000000000..4b720f3454a2
> --- /dev/null
> +++ b/lib/raid/Kconfig
> @@ -0,0 +1,3 @@
> +
> +config XOR_BLOCKS
> + tristate
> diff --git a/lib/raid/Makefile b/lib/raid/Makefile
> new file mode 100644
> index 000000000000..382f2d1694bd
> --- /dev/null
> +++ b/lib/raid/Makefile
> @@ -0,0 +1,2 @@
> +
> +obj-y += xor/
Probably should add an SPDX-License-Identifier to these new files.
- Eric
^ permalink raw reply
* Re: [PATCH 06/25] xor: cleanup registration and probing
From: Eric Biggers @ 2026-02-28 4:41 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-7-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:18AM -0800, Christoph Hellwig wrote:
> /* Set of all registered templates. */
> static struct xor_block_template *__initdata template_list;
> +static int __initdata xor_forced = false;
bool instead of int
> b1 = (void *) __get_free_pages(GFP_KERNEL, 2);
> if (!b1) {
> - printk(KERN_WARNING "xor: Yikes! No memory available.\n");
> + pr_info("xor: Yikes! No memory available.\n");
pr_warn() instead of pr_info()
- Eric
^ permalink raw reply
* Re: [PATCH 07/25] xor: split xor.h
From: Eric Biggers @ 2026-02-28 4:43 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-8-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:19AM -0800, Christoph Hellwig wrote:
> Keep xor.h for the public API, and split the struct xor_block_template
> definition that is only needed by the xor.ko core and
> architecture-specific optimizations into a separate xor_impl.h header.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> arch/arm/lib/xor-neon.c | 1 +
> arch/s390/lib/xor.c | 2 +-
> include/linux/raid/xor.h | 22 +---------------------
> include/linux/raid/xor_impl.h | 25 +++++++++++++++++++++++++
> lib/raid/xor/xor-core.c | 1 +
> 5 files changed, 29 insertions(+), 22 deletions(-)
> create mode 100644 include/linux/raid/xor_impl.h
arch/arm64/lib/xor-neon.c needs to be updated to include xor_impl.h.
- Eric
^ permalink raw reply
* Re: [PATCH 15/25] riscv: move the XOR code to lib/raid/
From: Eric Biggers @ 2026-02-28 5:37 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-16-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:27AM -0800, Christoph Hellwig wrote:
> Move the optimized XOR into lib/raid and include it it in xor.ko
> instead of always building it into the main kernel image.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> arch/riscv/include/asm/xor.h | 54 +------------------
> arch/riscv/lib/Makefile | 1 -
> lib/raid/xor/Makefile | 1 +
> lib/raid/xor/riscv/xor-glue.c | 56 ++++++++++++++++++++
> {arch/riscv/lib => lib/raid/xor/riscv}/xor.S | 0
The EXPORT_SYMBOL() statements in xor.S can be removed, since the
functions are now located in the same module as their callers.
- Eric
^ permalink raw reply
* Re: [PATCH 16/25] sparc: move the XOR code to lib/raid/
From: Eric Biggers @ 2026-02-28 5:47 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-17-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:28AM -0800, Christoph Hellwig wrote:
> diff --git a/arch/sparc/lib/xor.S b/lib/raid/xor/sparc/xor-niagara.S
> similarity index 53%
> rename from arch/sparc/lib/xor.S
> rename to lib/raid/xor/sparc/xor-niagara.S
> index 35461e3b2a9b..f8749a212eb3 100644
> --- a/arch/sparc/lib/xor.S
> +++ b/lib/raid/xor/sparc/xor-niagara.S
> @@ -1,11 +1,8 @@
> /* SPDX-License-Identifier: GPL-2.0 */
> /*
> - * arch/sparc64/lib/xor.S
> - *
> * High speed xor_block operation for RAID4/5 utilizing the
> - * UltraSparc Visual Instruction Set and Niagara store-init/twin-load.
> + * Niagara store-init/twin-load.
> *
> - * Copyright (C) 1997, 1999 Jakub Jelinek (jj@ultra.linux.cz)
> * Copyright (C) 2006 David S. Miller <davem@davemloft.net>
> */
>
> @@ -16,343 +13,6 @@
> #include <asm/dcu.h>
> #include <asm/spitfire.h>
>
<linux/export.h> can be removed from the two assembly files, since all
the invocations of EXPORT_SYMBOL() in them were removed.
Also, xor-niagara.S ended up without a .text directive at the beginning.
Probably it was unnecessary anyway. However, this seems unintentional,
given that xor-vis.S still has it.
- Eric
^ permalink raw reply
* Re: [PATCH 20/25] xor: make xor.ko self-contained in lib/raid/
From: Eric Biggers @ 2026-02-28 6:42 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-21-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:32AM -0800, Christoph Hellwig wrote:
> diff --git a/arch/um/include/asm/xor.h b/lib/raid/xor/um/xor_arch.h
> similarity index 61%
> rename from arch/um/include/asm/xor.h
> rename to lib/raid/xor/um/xor_arch.h
> index c9ddedc19301..c75cd9caf792 100644
> --- a/arch/um/include/asm/xor.h
> +++ b/lib/raid/xor/um/xor_arch.h
> @@ -1,7 +1,4 @@
> /* SPDX-License-Identifier: GPL-2.0 */
> -#ifndef _ASM_UM_XOR_H
> -#define _ASM_UM_XOR_H
> -
> #ifdef CONFIG_64BIT
> #undef CONFIG_X86_32
> #else
> #define CONFIG_X86_32 1
> #endif
Due to this change, the above code that sets CONFIG_X86_32 to the
opposite of CONFIG_64BIT is no longer included in xor-sse.c, which uses
CONFIG_X86_32. So if the above code actually did anything, this change
would have broken it for xor-sse.c. However, based on
arch/x86/um/Kconfig, CONFIG_X86_32 is always the opposite of
CONFIG_64BIT, so the above code actually has no effect. Does that sound
right?
- Eric
^ permalink raw reply
* Re: [PATCH 21/25] xor: add a better public API
From: Eric Biggers @ 2026-02-28 6:50 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-22-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:33AM -0800, Christoph Hellwig wrote:
> xor_blocks is very annoying to use, because it is limited to 4 + 1
> sources / destinations, has an odd argument order and is completely
> undocumented.
>
> Lift the code that loops around it from btrfs and async_tx/async_xor into
> common code under the name xor_gen and properly document it.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> include/linux/raid/xor.h | 3 +++
> lib/raid/xor/xor-core.c | 28 ++++++++++++++++++++++++++++
> 2 files changed, 31 insertions(+)
>
> diff --git a/include/linux/raid/xor.h b/include/linux/raid/xor.h
> index 02bda8d99534..4735a4e960f9 100644
> --- a/include/linux/raid/xor.h
> +++ b/include/linux/raid/xor.h
> @@ -7,4 +7,7 @@
> extern void xor_blocks(unsigned int count, unsigned int bytes,
> void *dest, void **srcs);
>
> +void xor_gen(void *dest, void **srcss, unsigned int src_cnt,
> + unsigned int bytes);
srcss => srcs
Ideally the source vectors would be 'const' as well.
> +/**
> + * xor_gen - generate RAID-style XOR information
> + * @dest: destination vector
> + * @srcs: source vectors
> + * @src_cnt: number of source vectors
> + * @bytes: length in bytes of each vector
> + *
> + * Performs bit-wise XOR operation into @dest for each of the @src_cnt vectors
> + * in @srcs for a length of @bytes bytes.
> + *
> + * Note: for typical RAID uses, @dest either needs to be zeroed, or filled with
> + * the first disk, which then needs to be removed from @srcs.
> + */
> +void xor_gen(void *dest, void **srcs, unsigned int src_cnt, unsigned int bytes)
> +{
> + unsigned int src_off = 0;
> +
> + while (src_cnt > 0) {
> + unsigned int this_cnt = min(src_cnt, MAX_XOR_BLOCKS);
> +
> + xor_blocks(this_cnt, bytes, dest, srcs + src_off);
> +
> + src_cnt -= this_cnt;
> + src_off += this_cnt;
> + }
> +}
> +EXPORT_SYMBOL(xor_gen);
The alignment requirements on the vectors should be documented, as
should which values of bytes are accepted. It looks like, at the very
least, the vectors have to be 32-byte aligned and the length has to be a
nonzero multiple of 512 bytes. But I didn't check every implementation.
Also, the requirement on the calling context (e.g. !is_interrupt())
should be documented as well.
- Eric
^ permalink raw reply
* Re: [PATCH 22/25] async_xor: use xor_gen
From: Eric Biggers @ 2026-02-28 6:55 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-23-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:34AM -0800, Christoph Hellwig wrote:
> Replace use of the loop around xor_blocks with the easier to use xor_gen
> API.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
> crypto/async_tx/async_xor.c | 16 ++--------------
> 1 file changed, 2 insertions(+), 14 deletions(-)
There are still comments in this file that refer to xor_blocks.
- Eric
^ permalink raw reply
* Re: [PATCH 24/25] xor: pass the entire operation to the low-level ops
From: Eric Biggers @ 2026-02-28 6:58 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-25-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:36AM -0800, Christoph Hellwig wrote:
> +#define __DO_XOR_BLOCKS(_name, _handle1, _handle2, _handle3, _handle4) \
> +void \
> +xor_gen_##_name(void *dest, void **srcs, unsigned int src_cnt, \
> + unsigned int bytes) \
> +{ \
> + unsigned int src_off = 0; \
> + \
> + while (src_cnt > 0) { \
> + unsigned int this_cnt = min(src_cnt, 4); \
> + unsigned long *p1 = (unsigned long *)srcs[src_off]; \
> + unsigned long *p2 = (unsigned long *)srcs[src_off + 1]; \
> + unsigned long *p3 = (unsigned long *)srcs[src_off + 2]; \
> + unsigned long *p4 = (unsigned long *)srcs[src_off + 3]; \
This reads out of bounds if src_cnt isn't a multiple of 4.
- Eric
^ permalink raw reply
* Re: [PATCH 09/25] xor: move generic implementations out of asm-generic/xor.h
From: Eric Biggers @ 2026-02-28 7:15 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-10-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:21AM -0800, Christoph Hellwig wrote:
> Move the generic implementations from asm-generic/xor.h to
> per-implementaion .c files in lib/raid.
>
> Note that this would cause the second xor_block_8regs instance created by
> arch/arm/lib/xor-neon.c to be generated instead of discarded as dead
> code, so add a NO_TEMPLATE symbol to disable it for this case.
>
> Signed-off-by: Christoph Hellwig <hch@lst.de>
This makes the generic code always be included in xor.ko, even when the
architecture doesn't need it. For example, x86_64 doesn't need it,
since it always selects either the AVX or SSE code.
Have you considered putting the generic code in xor-core.c (or in
headers included by it) before xor_arch.h is included, and putting
__maybe_unused on the xor_block_template structs? Then they'll still be
available for arch_xor_init() to use, but any of them that aren't used
in a particular build will be optimized out as dead code by the
compiler.
lib/crc/ and lib/crypto/ take a similar approach for most algorithms.
- Eric
^ permalink raw reply
* Re: cleanup the RAID5 XOR library
From: Eric Biggers @ 2026-02-28 7:35 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Andrew Morton, Richard Henderson, Matt Turner, Magnus Lindholm,
Russell King, Catalin Marinas, Will Deacon, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, linux-alpha,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, sparclinux, linux-um, linux-crypto,
linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260226151106.144735-1-hch@lst.de>
On Thu, Feb 26, 2026 at 07:10:12AM -0800, Christoph Hellwig wrote:
> Hi all,
>
> the XOR library used for the RAID5 parity is a bit of a mess right now.
> The main file sits in crypto/ despite not being cryptography and not
> using the crypto API, with the generic implementations sitting in
> include/asm-generic and the arch implementations sitting in an asm/
> header in theory. The latter doesn't work for many cases, so
> architectures often build the code directly into the core kernel, or
> create another module for the architecture code.
>
> Changes this to a single module in lib/ that also contains the
> architecture optimizations, similar to the library work Eric Biggers
> has done for the CRC and crypto libraries later. After that it changes
> to better calling conventions that allow for smarter architecture
> implementations (although none is contained here yet), and uses
> static_call to avoid indirection function call overhead.
>
> A git tree is also available here:
>
> git://git.infradead.org/users/hch/misc.git xor-improvements
>
> Gitweb:
>
> https://git.infradead.org/?p=users/hch/misc.git;a=shortlog;h=refs/heads/xor-improvements
Overall this looks great. xor_gen() really needs a KUnit test, though.
Without that, how was this tested?
Later we should remove some of the obsolete implementations, such as the
alpha or x86 MMX ones. Those platforms have no optimized code in
lib/crc/ or lib/crypto/, and I doubt anyone cares. But that should be a
separate series later: porting them over unchanged is the right call for
now so that this series doesn't get blocked on debates about removals...
Also, I notice that no one has optimized this for the latest x86_64 CPUs
by using the vpternlogd instruction to do 3-input XORs. That would be
another good future project.
- Eric
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox