* [PATCH 1/3] raid6: declare the AVX-512 P/Q stores as asm outputs
2026-11-22 3:29 [PATCH 0/3] raid6: x86: asm constraints, sfence and prefetch in xor_syndrome Miroslav Cimerman
@ 2026-11-22 3:29 ` Miroslav Cimerman
2026-11-22 3:29 ` [PATCH 2/3] raid6: remove the sfence from the x86 xor_syndrome functions Miroslav Cimerman
2026-11-22 3:29 ` [PATCH 3/3] raid6: prefetch Q with prefetcht0 in the x86 x4 " Miroslav Cimerman
2 siblings, 0 replies; 4+ messages in thread
From: Miroslav Cimerman @ 2026-11-22 3:29 UTC (permalink / raw)
To: Song Liu, Yu Kuai, Andrew Morton
Cc: Li Nan, Xiao Ni, Eric Biggers, Christoph Hellwig, David Laight,
Markus Stockhausen, linux-raid, linux-kernel
The AVX-512 gen_syndrome() and xor_syndrome() functions store P and Q in
inline asm that lists these memory operands as inputs only. The compiler
may then assume that the asm does not change P and Q. Each function ends
with an sfence whose asm has a "memory" clobber, and the constraints
technically rely on it.
Declare the stored P and Q as outputs: "=m" where the asm only writes
them, and "+m" for Q in xor_syndrome(), which the same asm reads and
writes back. The SSE2 and AVX2 functions already declare their stores as
outputs.
With clang the generated code does not change: it already loads the data
pointers from ptrs[] inside the loop over the block. gcc 15.2 now does
the same, because a byte store may alias the pointer array. Five of the
six functions get one or two more instructions.
Suggested-by: Eric Biggers <ebiggers@kernel.org>
Link: https://lore.kernel.org/linux-raid/20261116210440.GE229817@google.com/
Signed-off-by: Miroslav Cimerman <mc@raidlab.org>
---
lib/raid/raid6/x86/avx512.c | 30 ++++++++++++------------------
1 file changed, 12 insertions(+), 18 deletions(-)
diff --git a/lib/raid/raid6/x86/avx512.c b/lib/raid/raid6/x86/avx512.c
index dac0edb9189e..7dc0a45cd2fb 100644
--- a/lib/raid/raid6/x86/avx512.c
+++ b/lib/raid/raid6/x86/avx512.c
@@ -73,8 +73,7 @@ static void raid6_avx5121_gen_syndrome(int disks, size_t bytes, void **ptrs)
"vpxorq %%zmm2,%%zmm2,%%zmm2\n\t"
"vmovntdq %%zmm4,%1\n\t"
"vpxorq %%zmm4,%%zmm4,%%zmm4"
- :
- : "m" (p[d]), "m" (q[d]));
+ : "=m" (p[d]), "=m" (q[d]));
}
asm volatile("sfence" : : : "memory");
@@ -133,8 +132,7 @@ static void raid6_avx5121_xor_syndrome(int disks, int start, int stop,
/* Don't use movntdq for r/w memory area < cache line */
"vmovdqa64 %%zmm4,%0\n\t"
"vmovdqa64 %%zmm2,%1"
- :
- : "m" (q[d]), "m" (p[d]));
+ : "+m" (q[d]), "=m" (p[d]));
}
asm volatile("sfence" : : : "memory");
@@ -204,9 +202,8 @@ static void raid6_avx5122_gen_syndrome(int disks, size_t bytes, void **ptrs)
"vmovntdq %%zmm3,%1\n\t"
"vmovntdq %%zmm4,%2\n\t"
"vmovntdq %%zmm6,%3"
- :
- : "m" (p[d]), "m" (p[d+64]), "m" (q[d]),
- "m" (q[d+64]));
+ : "=m" (p[d]), "=m" (p[d+64]), "=m" (q[d]),
+ "=m" (q[d+64]));
}
asm volatile("sfence" : : : "memory");
@@ -289,9 +286,8 @@ static void raid6_avx5122_xor_syndrome(int disks, int start, int stop,
"vmovdqa64 %%zmm6,%1\n\t"
"vmovdqa64 %%zmm2,%2\n\t"
"vmovdqa64 %%zmm3,%3"
- :
- : "m" (q[d]), "m" (q[d+64]), "m" (p[d]),
- "m" (p[d+64]));
+ : "+m" (q[d]), "+m" (q[d+64]), "=m" (p[d]),
+ "=m" (p[d+64]));
}
asm volatile("sfence" : : : "memory");
@@ -393,10 +389,9 @@ static void raid6_avx5124_gen_syndrome(int disks, size_t bytes, void **ptrs)
"vpxorq %%zmm12,%%zmm12,%%zmm12\n\t"
"vmovntdq %%zmm14,%7\n\t"
"vpxorq %%zmm14,%%zmm14,%%zmm14"
- :
- : "m" (p[d]), "m" (p[d+64]), "m" (p[d+128]),
- "m" (p[d+192]), "m" (q[d]), "m" (q[d+64]),
- "m" (q[d+128]), "m" (q[d+192]));
+ : "=m" (p[d]), "=m" (p[d+64]), "=m" (p[d+128]),
+ "=m" (p[d+192]), "=m" (q[d]), "=m" (q[d+64]),
+ "=m" (q[d+128]), "=m" (q[d+192]));
}
asm volatile("sfence" : : : "memory");
@@ -528,10 +523,9 @@ static void raid6_avx5124_xor_syndrome(int disks, int start, int stop,
"vmovdqa64 %%zmm6,%5\n\t"
"vmovdqa64 %%zmm12,%6\n\t"
"vmovdqa64 %%zmm14,%7"
- :
- : "m" (p[d]), "m" (p[d+64]), "m" (p[d+128]),
- "m" (p[d+192]), "m" (q[d]), "m" (q[d+64]),
- "m" (q[d+128]), "m" (q[d+192]));
+ : "=m" (p[d]), "=m" (p[d+64]), "=m" (p[d+128]),
+ "=m" (p[d+192]), "+m" (q[d]), "+m" (q[d+64]),
+ "+m" (q[d+128]), "+m" (q[d+192]));
}
asm volatile("sfence" : : : "memory");
asm volatile("vzeroupper");
--
2.53.0
^ permalink raw reply related [flat|nested] 4+ messages in thread* [PATCH 2/3] raid6: remove the sfence from the x86 xor_syndrome functions
2026-11-22 3:29 [PATCH 0/3] raid6: x86: asm constraints, sfence and prefetch in xor_syndrome Miroslav Cimerman
2026-11-22 3:29 ` [PATCH 1/3] raid6: declare the AVX-512 P/Q stores as asm outputs Miroslav Cimerman
@ 2026-11-22 3:29 ` Miroslav Cimerman
2026-11-22 3:29 ` [PATCH 3/3] raid6: prefetch Q with prefetcht0 in the x86 x4 " Miroslav Cimerman
2 siblings, 0 replies; 4+ messages in thread
From: Miroslav Cimerman @ 2026-11-22 3:29 UTC (permalink / raw)
To: Song Liu, Yu Kuai, Andrew Morton
Cc: Li Nan, Xiao Ni, Eric Biggers, Christoph Hellwig, David Laight,
Markus Stockhausen, linux-raid, linux-kernel
The SSE2, AVX2 and AVX-512 xor_syndrome() functions end with an sfence.
It orders non-temporal stores, but none of these functions uses them
anymore. The x4 versions stopped with "raid6: use normal stores in the
x86 x4 xor_syndrome functions", and the x1 and x2 versions never used
them.
Remove the sfence from all nine functions. The gen_syndrome() functions
still write P and Q with non-temporal stores and keep their sfence.
The asm of the AVX-512 functions relied on the "memory" clobber of the
sfence. "raid6: declare the AVX-512 P/Q stores as asm outputs" declares
their stores as outputs instead.
The memory operands are still declared as single bytes while the asm
accesses 16 to 64 bytes. This is unchanged. No C code in these functions
accesses the buffers.
Suggested-by: Christoph Hellwig <hch@lst.de>
Link: https://lore.kernel.org/linux-raid/avqegZd_W0kce0UA@infradead.org/
Signed-off-by: Miroslav Cimerman <mc@raidlab.org>
---
lib/raid/raid6/x86/avx2.c | 3 ---
lib/raid/raid6/x86/avx512.c | 3 ---
lib/raid/raid6/x86/sse2.c | 3 ---
3 files changed, 9 deletions(-)
diff --git a/lib/raid/raid6/x86/avx2.c b/lib/raid/raid6/x86/avx2.c
index a9e709ce2104..d3f488a9c1a5 100644
--- a/lib/raid/raid6/x86/avx2.c
+++ b/lib/raid/raid6/x86/avx2.c
@@ -115,7 +115,6 @@ static void raid6_avx21_xor_syndrome(int disks, int start, int stop,
asm volatile("vmovdqa %%ymm2,%0" : "=m" (p[d]));
}
- asm volatile("sfence" : : : "memory");
asm volatile("vzeroupper");
kernel_fpu_end();
}
@@ -245,7 +244,6 @@ static void raid6_avx22_xor_syndrome(int disks, int start, int stop,
asm volatile("vmovdqa %%ymm3,%0" : "=m" (p[d+32]));
}
- asm volatile("sfence" : : : "memory");
asm volatile("vzeroupper");
kernel_fpu_end();
}
@@ -448,7 +446,6 @@ static void raid6_avx24_xor_syndrome(int disks, int start, int stop,
asm volatile("vmovdqa %%ymm12,%0" : "=m" (q[d+64]));
asm volatile("vmovdqa %%ymm14,%0" : "=m" (q[d+96]));
}
- asm volatile("sfence" : : : "memory");
asm volatile("vzeroupper");
kernel_fpu_end();
}
diff --git a/lib/raid/raid6/x86/avx512.c b/lib/raid/raid6/x86/avx512.c
index 7dc0a45cd2fb..33d7639da5b3 100644
--- a/lib/raid/raid6/x86/avx512.c
+++ b/lib/raid/raid6/x86/avx512.c
@@ -135,7 +135,6 @@ static void raid6_avx5121_xor_syndrome(int disks, int start, int stop,
: "+m" (q[d]), "=m" (p[d]));
}
- asm volatile("sfence" : : : "memory");
asm volatile("vzeroupper");
kernel_fpu_end();
}
@@ -290,7 +289,6 @@ static void raid6_avx5122_xor_syndrome(int disks, int start, int stop,
"=m" (p[d+64]));
}
- asm volatile("sfence" : : : "memory");
asm volatile("vzeroupper");
kernel_fpu_end();
}
@@ -527,7 +525,6 @@ static void raid6_avx5124_xor_syndrome(int disks, int start, int stop,
"=m" (p[d+192]), "+m" (q[d]), "+m" (q[d+64]),
"+m" (q[d+128]), "+m" (q[d+192]));
}
- asm volatile("sfence" : : : "memory");
asm volatile("vzeroupper");
kernel_fpu_end();
}
diff --git a/lib/raid/raid6/x86/sse2.c b/lib/raid/raid6/x86/sse2.c
index 9e8bece51690..a0709916289c 100644
--- a/lib/raid/raid6/x86/sse2.c
+++ b/lib/raid/raid6/x86/sse2.c
@@ -113,7 +113,6 @@ static void raid6_sse21_xor_syndrome(int disks, int start, int stop,
asm volatile("movdqa %%xmm2,%0" : "=m" (p[d]));
}
- asm volatile("sfence" : : : "memory");
kernel_fpu_end();
}
@@ -241,7 +240,6 @@ static void raid6_sse22_xor_syndrome(int disks, int start, int stop,
asm volatile("movdqa %%xmm3,%0" : "=m" (p[d+16]));
}
- asm volatile("sfence" : : : "memory");
kernel_fpu_end();
}
@@ -445,7 +443,6 @@ static void raid6_sse24_xor_syndrome(int disks, int start, int stop,
asm volatile("movdqa %%xmm12,%0" : "=m" (q[d+32]));
asm volatile("movdqa %%xmm14,%0" : "=m" (q[d+48]));
}
- asm volatile("sfence" : : : "memory");
kernel_fpu_end();
}
--
2.53.0
^ permalink raw reply related [flat|nested] 4+ messages in thread* [PATCH 3/3] raid6: prefetch Q with prefetcht0 in the x86 x4 xor_syndrome functions
2026-11-22 3:29 [PATCH 0/3] raid6: x86: asm constraints, sfence and prefetch in xor_syndrome Miroslav Cimerman
2026-11-22 3:29 ` [PATCH 1/3] raid6: declare the AVX-512 P/Q stores as asm outputs Miroslav Cimerman
2026-11-22 3:29 ` [PATCH 2/3] raid6: remove the sfence from the x86 xor_syndrome functions Miroslav Cimerman
@ 2026-11-22 3:29 ` Miroslav Cimerman
2 siblings, 0 replies; 4+ messages in thread
From: Miroslav Cimerman @ 2026-11-22 3:29 UTC (permalink / raw)
To: Song Liu, Yu Kuai, Andrew Morton
Cc: Li Nan, Xiao Ni, Eric Biggers, Christoph Hellwig, David Laight,
Markus Stockhausen, linux-raid, linux-kernel
The x4 xor_syndrome() functions for SSE2, AVX2 and AVX-512 prefetch Q
with prefetchnta. That hint is meant for data that is used once. Q is
read, updated and written back with normal stores, and md calls
xor_syndrome() twice on the same P/Q page for each write.
Prefetch Q with prefetcht0, which keeps it in all cache levels. The data
blocks are read once and keep prefetchnta.
Time per write measured in a userspace harness in md's read-modify-write
pattern (two xor_syndrome() calls on the same P/Q page), avx512x4 on a
Xeon Gold 6130, 2-24 data disks, write to the first, middle and last
data disk, median of 10 passes:
P and Q in DRAM: -2.2% (geometric mean), up to -5.7%
P and Q in L3: +0.2%
P and Q in L1/L2: +0.3%
The gain comes with 2-8 data disks, where Q is a larger part of the
work. The SSE2 and AVX2 versions were not measured.
Suggested-by: Eric Biggers <ebiggers@kernel.org>
Link: https://lore.kernel.org/linux-raid/20261116210440.GE229817@google.com/
Signed-off-by: Miroslav Cimerman <mc@raidlab.org>
---
lib/raid/raid6/x86/avx2.c | 4 ++--
lib/raid/raid6/x86/avx512.c | 4 ++--
lib/raid/raid6/x86/sse2.c | 4 ++--
3 files changed, 6 insertions(+), 6 deletions(-)
diff --git a/lib/raid/raid6/x86/avx2.c b/lib/raid/raid6/x86/avx2.c
index d3f488a9c1a5..5a49079ace68 100644
--- a/lib/raid/raid6/x86/avx2.c
+++ b/lib/raid/raid6/x86/avx2.c
@@ -408,8 +408,8 @@ static void raid6_avx24_xor_syndrome(int disks, int start, int stop,
asm volatile("vpxor %ymm13,%ymm12,%ymm12");
asm volatile("vpxor %ymm15,%ymm14,%ymm14");
}
- asm volatile("prefetchnta %0" :: "m" (q[d]));
- asm volatile("prefetchnta %0" :: "m" (q[d+64]));
+ asm volatile("prefetcht0 %0" :: "m" (q[d]));
+ asm volatile("prefetcht0 %0" :: "m" (q[d+64]));
/* P/Q left side optimization */
for (z = start-1 ; z >= 0 ; z--) {
asm volatile("vpxor %ymm5,%ymm5,%ymm5");
diff --git a/lib/raid/raid6/x86/avx512.c b/lib/raid/raid6/x86/avx512.c
index 33d7639da5b3..38077109ab0f 100644
--- a/lib/raid/raid6/x86/avx512.c
+++ b/lib/raid/raid6/x86/avx512.c
@@ -476,8 +476,8 @@ static void raid6_avx5124_xor_syndrome(int disks, int start, int stop,
"m" (dptr[z][d+128]),
"m" (dptr[z][d+192]));
}
- asm volatile("prefetchnta %0\n\t"
- "prefetchnta %1\n\t"
+ asm volatile("prefetcht0 %0\n\t"
+ "prefetcht0 %1\n\t"
:
: "m" (q[d]), "m" (q[d+128]));
/* P/Q left side optimization */
diff --git a/lib/raid/raid6/x86/sse2.c b/lib/raid/raid6/x86/sse2.c
index a0709916289c..c27a4c8254ba 100644
--- a/lib/raid/raid6/x86/sse2.c
+++ b/lib/raid/raid6/x86/sse2.c
@@ -405,8 +405,8 @@ static void raid6_sse24_xor_syndrome(int disks, int start, int stop,
asm volatile("pxor %xmm13,%xmm12");
asm volatile("pxor %xmm15,%xmm14");
}
- asm volatile("prefetchnta %0" :: "m" (q[d]));
- asm volatile("prefetchnta %0" :: "m" (q[d+32]));
+ asm volatile("prefetcht0 %0" :: "m" (q[d]));
+ asm volatile("prefetcht0 %0" :: "m" (q[d+32]));
/* P/Q left side optimization */
for ( z = start-1 ; z >= 0 ; z-- ) {
asm volatile("pxor %xmm5,%xmm5");
--
2.53.0
^ permalink raw reply related [flat|nested] 4+ messages in thread