The Linux Kernel Mailing List
 help / color / mirror / Atom feed
* [PATCH 0/3] raid6: x86: asm constraints, sfence and prefetch in xor_syndrome
@ 2026-11-22  3:29 Miroslav Cimerman
  2026-11-22  3:29 ` [PATCH 1/3] raid6: declare the AVX-512 P/Q stores as asm outputs Miroslav Cimerman
                   ` (2 more replies)
  0 siblings, 3 replies; 4+ messages in thread
From: Miroslav Cimerman @ 2026-11-22  3:29 UTC (permalink / raw)
  To: Song Liu, Yu Kuai, Andrew Morton
  Cc: Li Nan, Xiao Ni, Eric Biggers, Christoph Hellwig, David Laight,
	Markus Stockhausen, linux-raid, linux-kernel

This follows up on the review of "raid6: use normal stores in the x86 x4
xor_syndrome functions" [1]. With that patch, none of the x86
xor_syndrome() functions uses non-temporal stores, so their sfence
orders nothing. Christoph suggested removing it [2]. Eric pointed out
that the AVX-512 functions list P and Q as asm inputs only and rely on
the "memory" clobber of that sfence, and suggested prefetcht0 for Q [3].
1/3 fixes the constraints, 2/3 removes the sfence, and 3/3 changes the
prefetch of Q.

Based on mm-nonmm-unstable with [1] applied.

Testing:
- Builds without warnings with W=1, gcc 15.2 and clang 21.1.
- With 1/3, clang generates the same code as before. gcc reloads the
  data pointers inside the loop of the AVX-512 functions (up to two
  more instructions per function). With 2/3, the sfence instructions
  are gone, and apart from alignment padding the code is the same.
  3/3 only changes the six prefetch instructions.
- The raid6 KUnit test passes in QEMU with -cpu max under KVM on a
  Haswell host (avx2x1..x4) and with -cpu Westmere (sse2x1..x4). All
  8 test cases pass in each run.
- In userspace, gen_syndrome() and xor_syndrome() of all nine sse2,
  avx2 and avx512 variants match a scalar reference (gen_syndrome for
  2-24 data disks, xor_syndrome for every start/stop): sse2 natively on
  a Core i5-2520M, avx2 natively on a Xeon E5-2699 v3, all variants
  natively on a Xeon Gold 6130, and avx512 also with Intel SDE (-skx).
  The AVX-512 functions were not run in a kernel. I have no AVX-512
  machine with root access.

The patches and the tests were prepared with the help of an LLM. I
reviewed everything.

[1] https://lore.kernel.org/linux-raid/20261109030805.25709-1-mc@raidlab.org/
[2] https://lore.kernel.org/linux-raid/avqegZd_W0kce0UA@infradead.org/
[3] https://lore.kernel.org/linux-raid/20261116210440.GE229817@google.com/

Miroslav Cimerman (3):
  raid6: declare the AVX-512 P/Q stores as asm outputs
  raid6: remove the sfence from the x86 xor_syndrome functions
  raid6: prefetch Q with prefetcht0 in the x86 x4 xor_syndrome functions

 lib/raid/raid6/x86/avx2.c   |  7 ++-----
 lib/raid/raid6/x86/avx512.c | 37 ++++++++++++++-----------------------
 lib/raid/raid6/x86/sse2.c   |  7 ++-----
 3 files changed, 18 insertions(+), 33 deletions(-)


base-commit: 609cd6a5e6138620c16d8818bb585e8766064c23
prerequisite-patch-id: 0242f328222b644b2b329d90c8d50aad322b7655
prerequisite-message-id: <20261109030805.25709-1-mc@raidlab.org>
-- 
2.53.0


^ permalink raw reply	[flat|nested] 4+ messages in thread

* [PATCH 1/3] raid6: declare the AVX-512 P/Q stores as asm outputs
  2026-11-22  3:29 [PATCH 0/3] raid6: x86: asm constraints, sfence and prefetch in xor_syndrome Miroslav Cimerman
@ 2026-11-22  3:29 ` Miroslav Cimerman
  2026-11-22  3:29 ` [PATCH 2/3] raid6: remove the sfence from the x86 xor_syndrome functions Miroslav Cimerman
  2026-11-22  3:29 ` [PATCH 3/3] raid6: prefetch Q with prefetcht0 in the x86 x4 " Miroslav Cimerman
  2 siblings, 0 replies; 4+ messages in thread
From: Miroslav Cimerman @ 2026-11-22  3:29 UTC (permalink / raw)
  To: Song Liu, Yu Kuai, Andrew Morton
  Cc: Li Nan, Xiao Ni, Eric Biggers, Christoph Hellwig, David Laight,
	Markus Stockhausen, linux-raid, linux-kernel

The AVX-512 gen_syndrome() and xor_syndrome() functions store P and Q in
inline asm that lists these memory operands as inputs only. The compiler
may then assume that the asm does not change P and Q. Each function ends
with an sfence whose asm has a "memory" clobber, and the constraints
technically rely on it.

Declare the stored P and Q as outputs: "=m" where the asm only writes
them, and "+m" for Q in xor_syndrome(), which the same asm reads and
writes back. The SSE2 and AVX2 functions already declare their stores as
outputs.

With clang the generated code does not change: it already loads the data
pointers from ptrs[] inside the loop over the block. gcc 15.2 now does
the same, because a byte store may alias the pointer array. Five of the
six functions get one or two more instructions.

Suggested-by: Eric Biggers <ebiggers@kernel.org>
Link: https://lore.kernel.org/linux-raid/20261116210440.GE229817@google.com/
Signed-off-by: Miroslav Cimerman <mc@raidlab.org>
---
 lib/raid/raid6/x86/avx512.c | 30 ++++++++++++------------------
 1 file changed, 12 insertions(+), 18 deletions(-)

diff --git a/lib/raid/raid6/x86/avx512.c b/lib/raid/raid6/x86/avx512.c
index dac0edb9189e..7dc0a45cd2fb 100644
--- a/lib/raid/raid6/x86/avx512.c
+++ b/lib/raid/raid6/x86/avx512.c
@@ -73,8 +73,7 @@ static void raid6_avx5121_gen_syndrome(int disks, size_t bytes, void **ptrs)
 			     "vpxorq %%zmm2,%%zmm2,%%zmm2\n\t"
 			     "vmovntdq %%zmm4,%1\n\t"
 			     "vpxorq %%zmm4,%%zmm4,%%zmm4"
-			     :
-			     : "m" (p[d]), "m" (q[d]));
+			     : "=m" (p[d]), "=m" (q[d]));
 	}
 
 	asm volatile("sfence" : : : "memory");
@@ -133,8 +132,7 @@ static void raid6_avx5121_xor_syndrome(int disks, int start, int stop,
 		/* Don't use movntdq for r/w memory area < cache line */
 			     "vmovdqa64 %%zmm4,%0\n\t"
 			     "vmovdqa64 %%zmm2,%1"
-			     :
-			     : "m" (q[d]), "m" (p[d]));
+			     : "+m" (q[d]), "=m" (p[d]));
 	}
 
 	asm volatile("sfence" : : : "memory");
@@ -204,9 +202,8 @@ static void raid6_avx5122_gen_syndrome(int disks, size_t bytes, void **ptrs)
 			     "vmovntdq %%zmm3,%1\n\t"
 			     "vmovntdq %%zmm4,%2\n\t"
 			     "vmovntdq %%zmm6,%3"
-			     :
-			     : "m" (p[d]), "m" (p[d+64]), "m" (q[d]),
-			       "m" (q[d+64]));
+			     : "=m" (p[d]), "=m" (p[d+64]), "=m" (q[d]),
+			       "=m" (q[d+64]));
 	}
 
 	asm volatile("sfence" : : : "memory");
@@ -289,9 +286,8 @@ static void raid6_avx5122_xor_syndrome(int disks, int start, int stop,
 			     "vmovdqa64 %%zmm6,%1\n\t"
 			     "vmovdqa64 %%zmm2,%2\n\t"
 			     "vmovdqa64 %%zmm3,%3"
-			     :
-			     : "m" (q[d]), "m" (q[d+64]), "m" (p[d]),
-			       "m" (p[d+64]));
+			     : "+m" (q[d]), "+m" (q[d+64]), "=m" (p[d]),
+			       "=m" (p[d+64]));
 	}
 
 	asm volatile("sfence" : : : "memory");
@@ -393,10 +389,9 @@ static void raid6_avx5124_gen_syndrome(int disks, size_t bytes, void **ptrs)
 			     "vpxorq %%zmm12,%%zmm12,%%zmm12\n\t"
 			     "vmovntdq %%zmm14,%7\n\t"
 			     "vpxorq %%zmm14,%%zmm14,%%zmm14"
-			     :
-			     : "m" (p[d]), "m" (p[d+64]), "m" (p[d+128]),
-			       "m" (p[d+192]), "m" (q[d]), "m" (q[d+64]),
-			       "m" (q[d+128]), "m" (q[d+192]));
+			     : "=m" (p[d]), "=m" (p[d+64]), "=m" (p[d+128]),
+			       "=m" (p[d+192]), "=m" (q[d]), "=m" (q[d+64]),
+			       "=m" (q[d+128]), "=m" (q[d+192]));
 	}
 
 	asm volatile("sfence" : : : "memory");
@@ -528,10 +523,9 @@ static void raid6_avx5124_xor_syndrome(int disks, int start, int stop,
 			     "vmovdqa64 %%zmm6,%5\n\t"
 			     "vmovdqa64 %%zmm12,%6\n\t"
 			     "vmovdqa64 %%zmm14,%7"
-			     :
-			     : "m" (p[d]),  "m" (p[d+64]), "m" (p[d+128]),
-			       "m" (p[d+192]), "m" (q[d]),  "m" (q[d+64]),
-			       "m" (q[d+128]), "m" (q[d+192]));
+			     : "=m" (p[d]), "=m" (p[d+64]), "=m" (p[d+128]),
+			       "=m" (p[d+192]), "+m" (q[d]), "+m" (q[d+64]),
+			       "+m" (q[d+128]), "+m" (q[d+192]));
 	}
 	asm volatile("sfence" : : : "memory");
 	asm volatile("vzeroupper");
-- 
2.53.0


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* [PATCH 2/3] raid6: remove the sfence from the x86 xor_syndrome functions
  2026-11-22  3:29 [PATCH 0/3] raid6: x86: asm constraints, sfence and prefetch in xor_syndrome Miroslav Cimerman
  2026-11-22  3:29 ` [PATCH 1/3] raid6: declare the AVX-512 P/Q stores as asm outputs Miroslav Cimerman
@ 2026-11-22  3:29 ` Miroslav Cimerman
  2026-11-22  3:29 ` [PATCH 3/3] raid6: prefetch Q with prefetcht0 in the x86 x4 " Miroslav Cimerman
  2 siblings, 0 replies; 4+ messages in thread
From: Miroslav Cimerman @ 2026-11-22  3:29 UTC (permalink / raw)
  To: Song Liu, Yu Kuai, Andrew Morton
  Cc: Li Nan, Xiao Ni, Eric Biggers, Christoph Hellwig, David Laight,
	Markus Stockhausen, linux-raid, linux-kernel

The SSE2, AVX2 and AVX-512 xor_syndrome() functions end with an sfence.
It orders non-temporal stores, but none of these functions uses them
anymore. The x4 versions stopped with "raid6: use normal stores in the
x86 x4 xor_syndrome functions", and the x1 and x2 versions never used
them.

Remove the sfence from all nine functions. The gen_syndrome() functions
still write P and Q with non-temporal stores and keep their sfence.

The asm of the AVX-512 functions relied on the "memory" clobber of the
sfence. "raid6: declare the AVX-512 P/Q stores as asm outputs" declares
their stores as outputs instead.

The memory operands are still declared as single bytes while the asm
accesses 16 to 64 bytes. This is unchanged. No C code in these functions
accesses the buffers.

Suggested-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/linux-raid/avqegZd_W0kce0UA@infradead.org/
Signed-off-by: Miroslav Cimerman <mc@raidlab.org>
---
 lib/raid/raid6/x86/avx2.c   | 3 ---
 lib/raid/raid6/x86/avx512.c | 3 ---
 lib/raid/raid6/x86/sse2.c   | 3 ---
 3 files changed, 9 deletions(-)

diff --git a/lib/raid/raid6/x86/avx2.c b/lib/raid/raid6/x86/avx2.c
index a9e709ce2104..d3f488a9c1a5 100644
--- a/lib/raid/raid6/x86/avx2.c
+++ b/lib/raid/raid6/x86/avx2.c
@@ -115,7 +115,6 @@ static void raid6_avx21_xor_syndrome(int disks, int start, int stop,
 		asm volatile("vmovdqa %%ymm2,%0" : "=m" (p[d]));
 	}
 
-	asm volatile("sfence" : : : "memory");
 	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
@@ -245,7 +244,6 @@ static void raid6_avx22_xor_syndrome(int disks, int start, int stop,
 		asm volatile("vmovdqa %%ymm3,%0" : "=m" (p[d+32]));
 	}
 
-	asm volatile("sfence" : : : "memory");
 	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
@@ -448,7 +446,6 @@ static void raid6_avx24_xor_syndrome(int disks, int start, int stop,
 		asm volatile("vmovdqa %%ymm12,%0" : "=m" (q[d+64]));
 		asm volatile("vmovdqa %%ymm14,%0" : "=m" (q[d+96]));
 	}
-	asm volatile("sfence" : : : "memory");
 	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
diff --git a/lib/raid/raid6/x86/avx512.c b/lib/raid/raid6/x86/avx512.c
index 7dc0a45cd2fb..33d7639da5b3 100644
--- a/lib/raid/raid6/x86/avx512.c
+++ b/lib/raid/raid6/x86/avx512.c
@@ -135,7 +135,6 @@ static void raid6_avx5121_xor_syndrome(int disks, int start, int stop,
 			     : "+m" (q[d]), "=m" (p[d]));
 	}
 
-	asm volatile("sfence" : : : "memory");
 	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
@@ -290,7 +289,6 @@ static void raid6_avx5122_xor_syndrome(int disks, int start, int stop,
 			       "=m" (p[d+64]));
 	}
 
-	asm volatile("sfence" : : : "memory");
 	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
@@ -527,7 +525,6 @@ static void raid6_avx5124_xor_syndrome(int disks, int start, int stop,
 			       "=m" (p[d+192]), "+m" (q[d]), "+m" (q[d+64]),
 			       "+m" (q[d+128]), "+m" (q[d+192]));
 	}
-	asm volatile("sfence" : : : "memory");
 	asm volatile("vzeroupper");
 	kernel_fpu_end();
 }
diff --git a/lib/raid/raid6/x86/sse2.c b/lib/raid/raid6/x86/sse2.c
index 9e8bece51690..a0709916289c 100644
--- a/lib/raid/raid6/x86/sse2.c
+++ b/lib/raid/raid6/x86/sse2.c
@@ -113,7 +113,6 @@ static void raid6_sse21_xor_syndrome(int disks, int start, int stop,
 		asm volatile("movdqa %%xmm2,%0" : "=m" (p[d]));
 	}
 
-	asm volatile("sfence" : : : "memory");
 	kernel_fpu_end();
 }
 
@@ -241,7 +240,6 @@ static void raid6_sse22_xor_syndrome(int disks, int start, int stop,
 		asm volatile("movdqa %%xmm3,%0" : "=m" (p[d+16]));
 	}
 
-	asm volatile("sfence" : : : "memory");
 	kernel_fpu_end();
 }
 
@@ -445,7 +443,6 @@ static void raid6_sse24_xor_syndrome(int disks, int start, int stop,
 		asm volatile("movdqa %%xmm12,%0" : "=m" (q[d+32]));
 		asm volatile("movdqa %%xmm14,%0" : "=m" (q[d+48]));
 	}
-	asm volatile("sfence" : : : "memory");
 	kernel_fpu_end();
 }
 
-- 
2.53.0


^ permalink raw reply related	[flat|nested] 4+ messages in thread

* [PATCH 3/3] raid6: prefetch Q with prefetcht0 in the x86 x4 xor_syndrome functions
  2026-11-22  3:29 [PATCH 0/3] raid6: x86: asm constraints, sfence and prefetch in xor_syndrome Miroslav Cimerman
  2026-11-22  3:29 ` [PATCH 1/3] raid6: declare the AVX-512 P/Q stores as asm outputs Miroslav Cimerman
  2026-11-22  3:29 ` [PATCH 2/3] raid6: remove the sfence from the x86 xor_syndrome functions Miroslav Cimerman
@ 2026-11-22  3:29 ` Miroslav Cimerman
  2 siblings, 0 replies; 4+ messages in thread
From: Miroslav Cimerman @ 2026-11-22  3:29 UTC (permalink / raw)
  To: Song Liu, Yu Kuai, Andrew Morton
  Cc: Li Nan, Xiao Ni, Eric Biggers, Christoph Hellwig, David Laight,
	Markus Stockhausen, linux-raid, linux-kernel

The x4 xor_syndrome() functions for SSE2, AVX2 and AVX-512 prefetch Q
with prefetchnta. That hint is meant for data that is used once. Q is
read, updated and written back with normal stores, and md calls
xor_syndrome() twice on the same P/Q page for each write.

Prefetch Q with prefetcht0, which keeps it in all cache levels. The data
blocks are read once and keep prefetchnta.

Time per write measured in a userspace harness in md's read-modify-write
pattern (two xor_syndrome() calls on the same P/Q page), avx512x4 on a
Xeon Gold 6130, 2-24 data disks, write to the first, middle and last
data disk, median of 10 passes:

  P and Q in DRAM:      -2.2% (geometric mean), up to -5.7%
  P and Q in L3:        +0.2%
  P and Q in L1/L2:     +0.3%

The gain comes with 2-8 data disks, where Q is a larger part of the
work. The SSE2 and AVX2 versions were not measured.

Suggested-by: Eric Biggers <ebiggers@kernel.org>
Link: https://lore.kernel.org/linux-raid/20261116210440.GE229817@google.com/
Signed-off-by: Miroslav Cimerman <mc@raidlab.org>
---
 lib/raid/raid6/x86/avx2.c   | 4 ++--
 lib/raid/raid6/x86/avx512.c | 4 ++--
 lib/raid/raid6/x86/sse2.c   | 4 ++--
 3 files changed, 6 insertions(+), 6 deletions(-)

diff --git a/lib/raid/raid6/x86/avx2.c b/lib/raid/raid6/x86/avx2.c
index d3f488a9c1a5..5a49079ace68 100644
--- a/lib/raid/raid6/x86/avx2.c
+++ b/lib/raid/raid6/x86/avx2.c
@@ -408,8 +408,8 @@ static void raid6_avx24_xor_syndrome(int disks, int start, int stop,
 			asm volatile("vpxor %ymm13,%ymm12,%ymm12");
 			asm volatile("vpxor %ymm15,%ymm14,%ymm14");
 		}
-		asm volatile("prefetchnta %0" :: "m" (q[d]));
-		asm volatile("prefetchnta %0" :: "m" (q[d+64]));
+		asm volatile("prefetcht0 %0" :: "m" (q[d]));
+		asm volatile("prefetcht0 %0" :: "m" (q[d+64]));
 		/* P/Q left side optimization */
 		for (z = start-1 ; z >= 0 ; z--) {
 			asm volatile("vpxor %ymm5,%ymm5,%ymm5");
diff --git a/lib/raid/raid6/x86/avx512.c b/lib/raid/raid6/x86/avx512.c
index 33d7639da5b3..38077109ab0f 100644
--- a/lib/raid/raid6/x86/avx512.c
+++ b/lib/raid/raid6/x86/avx512.c
@@ -476,8 +476,8 @@ static void raid6_avx5124_xor_syndrome(int disks, int start, int stop,
 				       "m" (dptr[z][d+128]),
 				       "m" (dptr[z][d+192]));
 		}
-		asm volatile("prefetchnta %0\n\t"
-			     "prefetchnta %1\n\t"
+		asm volatile("prefetcht0 %0\n\t"
+			     "prefetcht0 %1\n\t"
 			     :
 			     : "m" (q[d]), "m" (q[d+128]));
 		/* P/Q left side optimization */
diff --git a/lib/raid/raid6/x86/sse2.c b/lib/raid/raid6/x86/sse2.c
index a0709916289c..c27a4c8254ba 100644
--- a/lib/raid/raid6/x86/sse2.c
+++ b/lib/raid/raid6/x86/sse2.c
@@ -405,8 +405,8 @@ static void raid6_sse24_xor_syndrome(int disks, int start, int stop,
 			asm volatile("pxor %xmm13,%xmm12");
 			asm volatile("pxor %xmm15,%xmm14");
 		}
-		asm volatile("prefetchnta %0" :: "m" (q[d]));
-		asm volatile("prefetchnta %0" :: "m" (q[d+32]));
+		asm volatile("prefetcht0 %0" :: "m" (q[d]));
+		asm volatile("prefetcht0 %0" :: "m" (q[d+32]));
 		/* P/Q left side optimization */
 		for ( z = start-1 ; z >= 0 ; z-- ) {
 			asm volatile("pxor %xmm5,%xmm5");
-- 
2.53.0


^ permalink raw reply related	[flat|nested] 4+ messages in thread

end of thread, other threads:[~2026-11-22  3:29 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2026-11-22  3:29 [PATCH 0/3] raid6: x86: asm constraints, sfence and prefetch in xor_syndrome Miroslav Cimerman
2026-11-22  3:29 ` [PATCH 1/3] raid6: declare the AVX-512 P/Q stores as asm outputs Miroslav Cimerman
2026-11-22  3:29 ` [PATCH 2/3] raid6: remove the sfence from the x86 xor_syndrome functions Miroslav Cimerman
2026-11-22  3:29 ` [PATCH 3/3] raid6: prefetch Q with prefetcht0 in the x86 x4 " Miroslav Cimerman

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox