* Re: [PATCH 02/17] raid6: remove __KERNEL__ ifdefs
From: H. Peter Anvin @ 2026-03-25 15:13 UTC (permalink / raw)
To: Christoph Hellwig, Andrew Morton
Cc: Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, Herbert Xu, Dan Williams, Chris Mason,
David Sterba, Arnd Bergmann, Song Liu, Yu Kuai, Li Nan,
linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, linux-crypto, linux-btrfs, linux-arch,
linux-raid
In-Reply-To: <20260324064115.3217136-3-hch@lst.de>
On 2026-03-23 23:40, Christoph Hellwig wrote:
> With the test code ported to kernel space, none of this is required.
I really *really* don't like this.
The ability of running in user space is really useful when it comes to
developing new code for new platforms, which happens often enough for this code.
-hpa
^ permalink raw reply
* Current GPU and Server Component Inventory Update
From: Computer Gaming pc @ 2026-03-25 12:23 UTC (permalink / raw)
To: linux-raid
Dear Partner,
We're pleased to share a current snapshot of our available inventory.
* RTX 5090 Founders Edition 32GB (90 units) - $2,000
* RTX Asus/Gigabyte 4090 24GB (45 units) - $1,600
* RTX 4090 Founders Edition 24GB (90 units) - $1,800
* RTX A6000 48GB Ada (54 units) - $1,100
* RTX 6000 48GB Ada (51 units) - $1,200
* Nvidia L40s GPU (12 units) - $3,000
* Nvidia H100 80GB PCIe (9 units) - $20,000
Samsung PM9A3 2.5" SSD PCIe 4.0 7.68TB (115 units) - $250
7.68TB SAS SSD 2.5" 12G Server Drive (140 units) - $250
* 32GB 2Rx4 PC4-2400T (Qty: 1,500) - USD 14
* 32GB 2Rx4 PC4-3200AA (Qty: 1,500) - USD 15
* 16GB 2Rx8 PC4-3200AA (Qty: 1,500) - USD 10
* 128GB 4DRx4 DDR4 PC4-2933Y REG ECC LRDIMM (Qty: 980) - USD 70
- DDR5 64GB 4800Mhz ECC RDimm 2Rx4 P43331-B21 (Qty: 500) - USD 550
- DDR5 64GB 5600Mhz ECC RDimm 2Rx4 P64707-B21 (Qty: 500) - USD 550
- DDR4 64GB 3200Mhz ECC RDimm 2Rx4 SK Hynix (Qty: 500) - USD 450
- DDR5 64GB 4800Mhz ECC RDimm 2Rx4 SK Hynix (Qty: 500) - USD 450
- DDR5 64GB 5600Mhz ECC RDimm 2Rx4 Samsung (Qty: 500) - USD 550
For quotes, availability, or inspection scheduling, please reach outdirectly:
sales@computergamingpc.com
Call or WhatsApp: +1(774) 559-1248 | +1 (641)232-4364
Thank you for your continued partnership.
Ann Kamila
Computer Gaming PC Trading
2500 US-6, Iowa City, IA 52240, USA
To unsubscribe from future updates, reply with "REMOVE" in the subject line.
^ permalink raw reply
* Re: [PATCH RESEND v3 2/3] lib/raid6: Optimizing the raid6_select_algo time through asynchronous processing
From: Christoph Hellwig @ 2026-03-25 5:54 UTC (permalink / raw)
To: sunliming
Cc: Christoph Hellwig, song, yukuai, akpm, linux-raid, linux-kernel,
sunliming
In-Reply-To: <51f57b049c46e9d22d69d2cb1589526814b9bf65@linux.dev>
On Wed, Mar 25, 2026 at 02:25:16AM +0000, sunliming@linux.dev wrote:
> > which could affect different test algorithms differently, so this is not
> > a good idea.
> >
> Yes，but this code can be compiled as a module，Could the same problem exist when loading the module?
It could. But at least it is under user control, while offloading
during normal boot almost guarantees a spiky load because that is
how the boot process works.
^ permalink raw reply
* Re: [PATCH 2/2] nvme-multipath: enable PCI P2PDMA for multipath devices
From: Chaitanya Kulkarni @ 2026-03-25 3:50 UTC (permalink / raw)
To: Keith Busch, Christoph Hellwig
Cc: Chaitanya Kulkarni, song@kernel.org, yukuai@fnnas.com,
linan122@huawei.com, axboe@kernel.dk, sagi@grimberg.me,
linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org,
Kiran Modukuri
In-Reply-To: <acKibPVk578SwyFR@kbusch-mbp>
On 3/24/26 07:40, Keith Busch wrote:
> On Tue, Mar 24, 2026 at 07:49:37AM +0100, Christoph Hellwig wrote:
>> On Mon, Mar 23, 2026 at 04:44:16PM -0700, Chaitanya Kulkarni wrote:
>>> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
>>> index ba00f0b72b85..c49fca43ef19 100644
>>> --- a/drivers/nvme/host/multipath.c
>>> +++ b/drivers/nvme/host/multipath.c
>>> @@ -737,6 +737,9 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
>>> BLK_FEAT_POLL | BLK_FEAT_ATOMIC_WRITES;
>>> if (head->ids.csi == NVME_CSI_ZNS)
>>> lim.features |= BLK_FEAT_ZONED;
>>> + if (ctrl->ops && ctrl->ops->supports_pci_p2pdma &&
>>> + ctrl->ops->supports_pci_p2pdma(ctrl))
>>> + lim.features |= BLK_FEAT_PCI_P2PDMA;
>> This assumes all controllers support P2P, but we allow matching
>> over different transports. So you'll need to do the same scheme
>> as for MD RAID that checks that every member supports P2P.
> If that is a possible setup, then you could add a path that is non-P2P
> capable sometime after the MD volume was setup with P2P supported, so
> that case might need special handling to notify the stacking device of
> the new limits.
Thanks for review working on V2 and testscripts.
-ck
^ permalink raw reply
* Re: [PATCH 1/2] md: Add PCI_P2PDMA support for MD RAID volumes
From: Keith Busch @ 2026-03-24 22:08 UTC (permalink / raw)
To: Kiran Modukuri
Cc: Christoph Hellwig, Chaitanya Kulkarni, song@kernel.org,
yukuai@fnnas.com, linan122@huawei.com, axboe@kernel.dk,
sagi@grimberg.me, linux-raid@vger.kernel.org,
linux-nvme@lists.infradead.org
In-Reply-To: <DM4PR12MB84736C00E876FD160464EB35D548A@DM4PR12MB8473.namprd12.prod.outlook.com>
On Tue, Mar 24, 2026 at 09:43:15PM +0000, Kiran Modukuri wrote:
> Hi Keith,
>
> So do you suggest we leave the current patch as is without restricting the P2PDMA support for RAID4/5 or restrict the support only for the RAID0, RAID1 and RAID10.
I think you currently have to remove it. If you want to do parity
against P2P memory (which sounds like like a nice feature to me), then
you'd have introduce a prep patch to detect that an xor dma offload
exists for the raid volume to use, and do something to ensure it will
always get offloaded for P2P memory instead of falling back to the CPU
driven synchronous implementation.
Unrelated suggestion, you need to change your email client settings to
plain text in order for the mailing list to accept the message.
> From: Keith Busch <kbusch@kernel.org>
> Date: Tuesday, March 24, 2026 at 2:29 PM
> To: Kiran Modukuri <kmodukuri@nvidia.com>
> Cc: Christoph Hellwig <hch@lst.de>, Chaitanya Kulkarni <kch@nvidia.com>, song@kernel.org <song@kernel.org>, yukuai@fnnas.com <yukuai@fnnas.com>, linan122@huawei.com <linan122@huawei.com>, axboe@kernel.dk <axboe@kernel.dk>, sagi@grimberg.me <sagi@grimberg.me>, linux-raid@vger.kernel.org <linux-raid@vger.kernel.org>, linux-nvme@lists.infradead.org <linux-nvme@lists.infradead.org>
> Subject: Re: [PATCH 1/2] md: Add PCI_P2PDMA support for MD RAID volumes
>
> On Tue, Mar 24, 2026 at 09:13:55PM +0000, Kiran Modukuri wrote:
> > Hi Christoph,
> >
> > We tested with RAID0 , RAID1 and RAID10 only. You're right that parity RAID personalities
> > need CPU access to data pages for XOR/parity computation, which won't
> > work with P2P mappings.
> >
> >
> > We'll send a v2 that moves BLK_FEAT_PCI_P2PDMA out of
> > md_init_stacking_limits() and instead has raid0, raid1 and raid10
> > opt in individually during their queue limits setup. raid4/5/6 will
> > not set the flag.
>
> I think the parity could work with P2P memory if the calculation is
> offloaded to a dma_aysnc_tx. It doesn't look like we necessarily know if
> any particular xor is going to get offloaded, though.
^ permalink raw reply
* Re: [PATCH 1/2] md: Add PCI_P2PDMA support for MD RAID volumes
From: Keith Busch @ 2026-03-24 21:29 UTC (permalink / raw)
To: Kiran Modukuri
Cc: Christoph Hellwig, Chaitanya Kulkarni, song@kernel.org,
yukuai@fnnas.com, linan122@huawei.com, axboe@kernel.dk,
sagi@grimberg.me, linux-raid@vger.kernel.org,
linux-nvme@lists.infradead.org
In-Reply-To: <DM4PR12MB8473B3907CAF51AEBAE573E5D548A@DM4PR12MB8473.namprd12.prod.outlook.com>
On Tue, Mar 24, 2026 at 09:13:55PM +0000, Kiran Modukuri wrote:
> Hi Christoph,
>
> We tested with RAID0 , RAID1 and RAID10 only. You're right that parity RAID personalities
> need CPU access to data pages for XOR/parity computation, which won't
> work with P2P mappings.
>
>
> We'll send a v2 that moves BLK_FEAT_PCI_P2PDMA out of
> md_init_stacking_limits() and instead has raid0, raid1 and raid10
> opt in individually during their queue limits setup. raid4/5/6 will
> not set the flag.
I think the parity could work with P2P memory if the calculation is
offloaded to a dma_aysnc_tx. It doesn't look like we necessarily know if
any particular xor is going to get offloaded, though.
^ permalink raw reply
* Re: cleanup the RAID5 XOR library v3
From: Andrew Morton @ 2026-03-24 16:42 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260324062211.3216301-1-hch@lst.de>
On Tue, 24 Mar 2026 07:21:36 +0100 Christoph Hellwig <hch@lst.de> wrote:
> the XOR library used for the RAID5 parity is a bit of a mess right now.
> The main file sits in crypto/ despite not being cryptography and not
> using the crypto API, with the generic implementations sitting in
> include/asm-generic and the arch implementations sitting in an asm/
> header in theory. The latter doesn't work for many cases, so
> architectures often build the code directly into the core kernel, or
> create another module for the architecture code.
>
> Changes this to a single module in lib/ that also contains the
> architecture optimizations, similar to the library work Eric Biggers
> has done for the CRC and crypto libraries later. After that it changes
> to better calling conventions that allow for smarter architecture
> implementations (although none is contained here yet), and uses
> static_call to avoid indirection function call overhead.
Thanks, I've updated mm.git's mm-nonmm-unstable branch to this version
of the patchset.
^ permalink raw reply
* Re: [PATCH 2/2] nvme-multipath: enable PCI P2PDMA for multipath devices
From: Keith Busch @ 2026-03-24 14:40 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Chaitanya Kulkarni, song, yukuai, linan122, axboe, sagi,
linux-raid, linux-nvme, Kiran Kumar Modukuri
In-Reply-To: <20260324064937.GB1409@lst.de>
On Tue, Mar 24, 2026 at 07:49:37AM +0100, Christoph Hellwig wrote:
> On Mon, Mar 23, 2026 at 04:44:16PM -0700, Chaitanya Kulkarni wrote:
> > diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
> > index ba00f0b72b85..c49fca43ef19 100644
> > --- a/drivers/nvme/host/multipath.c
> > +++ b/drivers/nvme/host/multipath.c
> > @@ -737,6 +737,9 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
> > BLK_FEAT_POLL | BLK_FEAT_ATOMIC_WRITES;
> > if (head->ids.csi == NVME_CSI_ZNS)
> > lim.features |= BLK_FEAT_ZONED;
> > + if (ctrl->ops && ctrl->ops->supports_pci_p2pdma &&
> > + ctrl->ops->supports_pci_p2pdma(ctrl))
> > + lim.features |= BLK_FEAT_PCI_P2PDMA;
>
> This assumes all controllers support P2P, but we allow matching
> over different transports. So you'll need to do the same scheme
> as for MD RAID that checks that every member supports P2P.
If that is a possible setup, then you could add a path that is non-P2P
capable sometime after the MD volume was setup with P2P supported, so
that case might need special handling to notify the stacking device of
the new limits.
^ permalink raw reply
* Re: cleanup the RAID5 XOR library v3
From: Andrew Morton @ 2026-03-24 12:59 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Richard Henderson, Matt Turner, Magnus Lindholm, Russell King,
Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, David S. Miller, Andreas Larsson,
Richard Weinberger, Anton Ivanov, Johannes Berg, Thomas Gleixner,
Ingo Molnar, Borislav Petkov, Dave Hansen, x86, H. Peter Anvin,
Herbert Xu, Dan Williams, Chris Mason, David Sterba,
Arnd Bergmann, Song Liu, Yu Kuai, Li Nan, Theodore Ts'o,
Jason A. Donenfeld, linux-alpha, linux-kernel, linux-arm-kernel,
loongarch, linuxppc-dev, linux-riscv, linux-s390, sparclinux,
linux-um, linux-crypto, linux-btrfs, linux-arch, linux-raid
In-Reply-To: <20260324062211.3216301-1-hch@lst.de>
On Tue, 24 Mar 2026 07:21:36 +0100 Christoph Hellwig <hch@lst.de> wrote:
> the XOR library used for the RAID5 parity is a bit of a mess right now.
> The main file sits in crypto/ despite not being cryptography and not
> using the crypto API, with the generic implementations sitting in
> include/asm-generic and the arch implementations sitting in an asm/
> header in theory. The latter doesn't work for many cases, so
> architectures often build the code directly into the core kernel, or
> create another module for the architecture code.
>
> Changes this to a single module in lib/ that also contains the
> architecture optimizations, similar to the library work Eric Biggers
> has done for the CRC and crypto libraries later. After that it changes
> to better calling conventions that allow for smarter architecture
> implementations (although none is contained here yet), and uses
> static_call to avoid indirection function call overhead.
Thanks, I'll update mm.git to this version.
It looks like AI review has found a few things:
https://sashiko.dev/#/patchset/20260324062211.3216301-1-hch@lst.de
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Mark Rutland @ 2026-03-24 10:04 UTC (permalink / raw)
To: Ard Biesheuvel, Christoph Hellwig, Demian Shulhan
Cc: Song Liu, Yu Kuai, Will Deacon, Catalin Marinas, broonie,
linux-arm-kernel, robin.murphy, Li Nan, linux-raid, linux-kernel
In-Reply-To: <f9bc0534-4b7a-4b14-974b-4d7811ccd899@app.fastmail.com>
On Tue, Mar 24, 2026 at 09:00:16AM +0100, Ard Biesheuvel wrote:
> On Wed, 18 Mar 2026, at 16:02, Demian Shulhan wrote:
> > Implement Scalable Vector Extension (SVE) optimized routines for RAID6
> > syndrome generation and recovery on ARM64.
> >
> > The SVE instruction set allows for variable vector lengths (from 128 to
> > 2048 bits), scaling automatically with the hardware capabilities. This
> > implementation handles arbitrary SVE vector lengths using the `cntb`
> > instruction to determine the runtime vector length.
> >
> > The implementation introduces `svex1`, `svex2`, and `svex4` algorithms.
> > The `svex4` algorithm utilizes loop unrolling by 4 blocks per iteration
> > and manual software pipelining (interleaving memory loads with XORs)
> > to minimize instruction dependency stalls and maximize CPU pipeline
> > utilization and memory bandwidth.
> >
> > Performance was tested on an AWS Graviton3 (Neoverse-V1) instance which
> > features 256-bit SVE vector length. The `svex4` implementation outperforms
> > the existing 128-bit `neonx4` baseline for syndrome generation:
> >
> > raid6: svex4 gen() 19688 MB/s
> ...
> > raid6: neonx4 gen() 19612 MB/s
>
> You're being very generous characterising a 0.3% speedup as 'outperforms'
>
> But the real problem here is that the kernel-mode SIMD API only
> supports NEON and not SVE, and preserves/restores only the 128-bit
> view on the NEON/SVE register file. So any context switch or softirq
> that uses kernel-mode SIMD too, and your SVE register values will get
> truncated.
Just to be a bit more explicit, since only the NEON register file is
saved:
* The vector registers will be truncated to 128-bit across
preemption or softirq.
* The predicates won't be saved/restored and will change arbitrarily
across preemption.
* The VL won't be saved/restored, and might change arbitrarily across
preemption.
* The VL to use hasn't been programmed, so performance might vary
arbitrarily even in the absence of preemption.
... so this isn't even safe on machines with (only) a 128-bit VL, and
there are big open design questions for the infrastructure we'd need.
> Once we encounter a good use case for SVE in the kernel, we might
> reconsider this, but as it stands, this patch should not be applied.
I agree.
Christoph, please do not pick this or any other in-kernel SVE patches.
They cannot function correctly without additional infrastructure.
Demian, for patches that use NEON/SVE/SME/etc, please Cc LAKML
(linux-arm-kernel@lists.infradead.org), so that folk familiar with ARM
see the patches.
Mark
> (leaving the reply untrimmed for the benefit of the cc'ees I added)
>
> > raid6: neonx2 gen() 16248 MB/s
> > raid6: neonx1 gen() 13591 MB/s
> > raid6: using algorithm svex4 gen() 19688 MB/s
> > raid6: .... xor() 11212 MB/s, rmw enabled
> > raid6: using neon recovery algorithm
> >
> > Note that for the recovery path (`xor_syndrome`), NEON may still be
> > selected dynamically by the algorithm benchmark, as the recovery
> > workload is heavily memory-bound.
> >
> > Signed-off-by: Demian Shulhan <demyansh@gmail.com>
> > Reported-by: kernel test robot <lkp@intel.com>
> > Closes:
> > https://lore.kernel.org/oe-kbuild-all/202603181940.cFwYmYoi-lkp@intel.com/
> > ---
> > include/linux/raid/pq.h | 3 +
> > lib/raid6/Makefile | 5 +
> > lib/raid6/algos.c | 5 +
> > lib/raid6/sve.c | 675 ++++++++++++++++++++++++++++++++++++++++
> > 4 files changed, 688 insertions(+)
> > create mode 100644 lib/raid6/sve.c
> >
> > diff --git a/include/linux/raid/pq.h b/include/linux/raid/pq.h
> > index 2467b3be15c9..787cc57aea9d 100644
> > --- a/include/linux/raid/pq.h
> > +++ b/include/linux/raid/pq.h
> > @@ -140,6 +140,9 @@ extern const struct raid6_calls raid6_neonx1;
> > extern const struct raid6_calls raid6_neonx2;
> > extern const struct raid6_calls raid6_neonx4;
> > extern const struct raid6_calls raid6_neonx8;
> > +extern const struct raid6_calls raid6_svex1;
> > +extern const struct raid6_calls raid6_svex2;
> > +extern const struct raid6_calls raid6_svex4;
> >
> > /* Algorithm list */
> > extern const struct raid6_calls * const raid6_algos[];
> > diff --git a/lib/raid6/Makefile b/lib/raid6/Makefile
> > index 5be0a4e60ab1..6cdaa6f206fb 100644
> > --- a/lib/raid6/Makefile
> > +++ b/lib/raid6/Makefile
> > @@ -8,6 +8,7 @@ raid6_pq-$(CONFIG_X86) += recov_ssse3.o recov_avx2.o
> > mmx.o sse1.o sse2.o avx2.o
> > raid6_pq-$(CONFIG_ALTIVEC) += altivec1.o altivec2.o altivec4.o
> > altivec8.o \
> > vpermxor1.o vpermxor2.o vpermxor4.o
> > vpermxor8.o
> > raid6_pq-$(CONFIG_KERNEL_MODE_NEON) += neon.o neon1.o neon2.o neon4.o
> > neon8.o recov_neon.o recov_neon_inner.o
> > +raid6_pq-$(CONFIG_ARM64_SVE) += sve.o
> > raid6_pq-$(CONFIG_S390) += s390vx8.o recov_s390xc.o
> > raid6_pq-$(CONFIG_LOONGARCH) += loongarch_simd.o recov_loongarch_simd.o
> > raid6_pq-$(CONFIG_RISCV_ISA_V) += rvv.o recov_rvv.o
> > @@ -67,6 +68,10 @@ CFLAGS_REMOVE_neon2.o += $(CC_FLAGS_NO_FPU)
> > CFLAGS_REMOVE_neon4.o += $(CC_FLAGS_NO_FPU)
> > CFLAGS_REMOVE_neon8.o += $(CC_FLAGS_NO_FPU)
> > CFLAGS_REMOVE_recov_neon_inner.o += $(CC_FLAGS_NO_FPU)
> > +
> > +CFLAGS_sve.o += $(CC_FLAGS_FPU)
> > +CFLAGS_REMOVE_sve.o += $(CC_FLAGS_NO_FPU)
> > +
> > targets += neon1.c neon2.c neon4.c neon8.c
> > $(obj)/neon%.c: $(src)/neon.uc $(src)/unroll.awk FORCE
> > $(call if_changed,unroll)
> > diff --git a/lib/raid6/algos.c b/lib/raid6/algos.c
> > index 799e0e5eac26..0ae73c3a4be3 100644
> > --- a/lib/raid6/algos.c
> > +++ b/lib/raid6/algos.c
> > @@ -66,6 +66,11 @@ const struct raid6_calls * const raid6_algos[] = {
> > &raid6_neonx2,
> > &raid6_neonx1,
> > #endif
> > +#ifdef CONFIG_ARM64_SVE
> > + &raid6_svex4,
> > + &raid6_svex2,
> > + &raid6_svex1,
> > +#endif
> > #ifdef CONFIG_LOONGARCH
> > #ifdef CONFIG_CPU_HAS_LASX
> > &raid6_lasx,
> > diff --git a/lib/raid6/sve.c b/lib/raid6/sve.c
> > new file mode 100644
> > index 000000000000..d52937f806d4
> > --- /dev/null
> > +++ b/lib/raid6/sve.c
> > @@ -0,0 +1,675 @@
> > +// SPDX-License-Identifier: GPL-2.0-or-later
> > +/*
> > + * RAID-6 syndrome calculation using ARM SVE instructions
> > + */
> > +
> > +#include <linux/raid/pq.h>
> > +
> > +#ifdef __KERNEL__
> > +#include <asm/simd.h>
> > +#include <linux/cpufeature.h>
> > +#else
> > +#define scoped_ksimd()
> > +#define system_supports_sve() (1)
> > +#endif
> > +
> > +static void raid6_sve1_gen_syndrome_real(int disks, unsigned long
> > bytes, void **ptrs)
> > +{
> > + u8 **dptr = (u8 **)ptrs;
> > + u8 *p, *q;
> > + long z0 = disks - 3;
> > +
> > + p = dptr[z0 + 1];
> > + q = dptr[z0 + 2];
> > +
> > + asm volatile(
> > + ".arch armv8.2-a+sve\n"
> > + "ptrue p0.b\n"
> > + "cntb x3\n"
> > + "mov w4, #0x1d\n"
> > + "dup z4.b, w4\n"
> > + "mov x5, #0\n"
> > +
> > + "0:\n"
> > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > + "ld1b z0.b, p0/z, [x6, x5]\n"
> > + "mov z1.d, z0.d\n"
> > +
> > + "mov w7, %w[z0]\n"
> > + "sub w7, w7, #1\n"
> > +
> > + "1:\n"
> > + "cmp w7, #0\n"
> > + "blt 2f\n"
> > +
> > + "mov z3.d, z1.d\n"
> > + "asr z3.b, p0/m, z3.b, #7\n"
> > + "lsl z1.b, p0/m, z1.b, #1\n"
> > +
> > + "and z3.d, z3.d, z4.d\n"
> > + "eor z1.d, z1.d, z3.d\n"
> > +
> > + "sxtw x8, w7\n"
> > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > +
> > + "eor z1.d, z1.d, z2.d\n"
> > + "eor z0.d, z0.d, z2.d\n"
> > +
> > + "sub w7, w7, #1\n"
> > + "b 1b\n"
> > + "2:\n"
> > +
> > + "st1b z0.b, p0, [%[p], x5]\n"
> > + "st1b z1.b, p0, [%[q], x5]\n"
> > +
> > + "add x5, x5, x3\n"
> > + "cmp x5, %[bytes]\n"
> > + "blt 0b\n"
> > + :
> > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > + [p] "r" (p), [q] "r" (q)
> > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > + "z0", "z1", "z2", "z3", "z4"
> > + );
> > +}
> > +
> > +static void raid6_sve1_xor_syndrome_real(int disks, int start, int
> > stop,
> > + unsigned long bytes, void **ptrs)
> > +{
> > + u8 **dptr = (u8 **)ptrs;
> > + u8 *p, *q;
> > + long z0 = stop;
> > +
> > + p = dptr[disks - 2];
> > + q = dptr[disks - 1];
> > +
> > + asm volatile(
> > + ".arch armv8.2-a+sve\n"
> > + "ptrue p0.b\n"
> > + "cntb x3\n"
> > + "mov w4, #0x1d\n"
> > + "dup z4.b, w4\n"
> > + "mov x5, #0\n"
> > +
> > + "0:\n"
> > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > + "ld1b z1.b, p0/z, [x6, x5]\n"
> > + "ld1b z0.b, p0/z, [%[p], x5]\n"
> > + "eor z0.d, z0.d, z1.d\n"
> > +
> > + "mov w7, %w[z0]\n"
> > + "sub w7, w7, #1\n"
> > +
> > + "1:\n"
> > + "cmp w7, %w[start]\n"
> > + "blt 2f\n"
> > +
> > + "mov z3.d, z1.d\n"
> > + "asr z3.b, p0/m, z3.b, #7\n"
> > + "lsl z1.b, p0/m, z1.b, #1\n"
> > + "and z3.d, z3.d, z4.d\n"
> > + "eor z1.d, z1.d, z3.d\n"
> > +
> > + "sxtw x8, w7\n"
> > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > +
> > + "eor z1.d, z1.d, z2.d\n"
> > + "eor z0.d, z0.d, z2.d\n"
> > +
> > + "sub w7, w7, #1\n"
> > + "b 1b\n"
> > + "2:\n"
> > +
> > + "mov w7, %w[start]\n"
> > + "sub w7, w7, #1\n"
> > + "3:\n"
> > + "cmp w7, #0\n"
> > + "blt 4f\n"
> > +
> > + "mov z3.d, z1.d\n"
> > + "asr z3.b, p0/m, z3.b, #7\n"
> > + "lsl z1.b, p0/m, z1.b, #1\n"
> > + "and z3.d, z3.d, z4.d\n"
> > + "eor z1.d, z1.d, z3.d\n"
> > +
> > + "sub w7, w7, #1\n"
> > + "b 3b\n"
> > + "4:\n"
> > +
> > + "ld1b z2.b, p0/z, [%[q], x5]\n"
> > + "eor z1.d, z1.d, z2.d\n"
> > +
> > + "st1b z0.b, p0, [%[p], x5]\n"
> > + "st1b z1.b, p0, [%[q], x5]\n"
> > +
> > + "add x5, x5, x3\n"
> > + "cmp x5, %[bytes]\n"
> > + "blt 0b\n"
> > + :
> > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > + [p] "r" (p), [q] "r" (q), [start] "r" (start)
> > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > + "z0", "z1", "z2", "z3", "z4"
> > + );
> > +}
> > +
> > +static void raid6_sve2_gen_syndrome_real(int disks, unsigned long
> > bytes, void **ptrs)
> > +{
> > + u8 **dptr = (u8 **)ptrs;
> > + u8 *p, *q;
> > + long z0 = disks - 3;
> > +
> > + p = dptr[z0 + 1];
> > + q = dptr[z0 + 2];
> > +
> > + asm volatile(
> > + ".arch armv8.2-a+sve\n"
> > + "ptrue p0.b\n"
> > + "cntb x3\n"
> > + "mov w4, #0x1d\n"
> > + "dup z4.b, w4\n"
> > + "mov x5, #0\n"
> > +
> > + "0:\n"
> > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > + "ld1b z0.b, p0/z, [x6, x5]\n"
> > + "add x8, x5, x3\n"
> > + "ld1b z5.b, p0/z, [x6, x8]\n"
> > + "mov z1.d, z0.d\n"
> > + "mov z6.d, z5.d\n"
> > +
> > + "mov w7, %w[z0]\n"
> > + "sub w7, w7, #1\n"
> > +
> > + "1:\n"
> > + "cmp w7, #0\n"
> > + "blt 2f\n"
> > +
> > + "mov z3.d, z1.d\n"
> > + "asr z3.b, p0/m, z3.b, #7\n"
> > + "lsl z1.b, p0/m, z1.b, #1\n"
> > + "and z3.d, z3.d, z4.d\n"
> > + "eor z1.d, z1.d, z3.d\n"
> > +
> > + "mov z8.d, z6.d\n"
> > + "asr z8.b, p0/m, z8.b, #7\n"
> > + "lsl z6.b, p0/m, z6.b, #1\n"
> > + "and z8.d, z8.d, z4.d\n"
> > + "eor z6.d, z6.d, z8.d\n"
> > +
> > + "sxtw x8, w7\n"
> > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > + "add x8, x5, x3\n"
> > + "ld1b z7.b, p0/z, [x6, x8]\n"
> > +
> > + "eor z1.d, z1.d, z2.d\n"
> > + "eor z0.d, z0.d, z2.d\n"
> > +
> > + "eor z6.d, z6.d, z7.d\n"
> > + "eor z5.d, z5.d, z7.d\n"
> > +
> > + "sub w7, w7, #1\n"
> > + "b 1b\n"
> > + "2:\n"
> > +
> > + "st1b z0.b, p0, [%[p], x5]\n"
> > + "st1b z1.b, p0, [%[q], x5]\n"
> > + "add x8, x5, x3\n"
> > + "st1b z5.b, p0, [%[p], x8]\n"
> > + "st1b z6.b, p0, [%[q], x8]\n"
> > +
> > + "add x5, x5, x3\n"
> > + "add x5, x5, x3\n"
> > + "cmp x5, %[bytes]\n"
> > + "blt 0b\n"
> > + :
> > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > + [p] "r" (p), [q] "r" (q)
> > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > + "z0", "z1", "z2", "z3", "z4",
> > + "z5", "z6", "z7", "z8"
> > + );
> > +}
> > +
> > +static void raid6_sve2_xor_syndrome_real(int disks, int start, int
> > stop,
> > + unsigned long bytes, void **ptrs)
> > +{
> > + u8 **dptr = (u8 **)ptrs;
> > + u8 *p, *q;
> > + long z0 = stop;
> > +
> > + p = dptr[disks - 2];
> > + q = dptr[disks - 1];
> > +
> > + asm volatile(
> > + ".arch armv8.2-a+sve\n"
> > + "ptrue p0.b\n"
> > + "cntb x3\n"
> > + "mov w4, #0x1d\n"
> > + "dup z4.b, w4\n"
> > + "mov x5, #0\n"
> > +
> > + "0:\n"
> > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > + "ld1b z1.b, p0/z, [x6, x5]\n"
> > + "add x8, x5, x3\n"
> > + "ld1b z6.b, p0/z, [x6, x8]\n"
> > +
> > + "ld1b z0.b, p0/z, [%[p], x5]\n"
> > + "ld1b z5.b, p0/z, [%[p], x8]\n"
> > +
> > + "eor z0.d, z0.d, z1.d\n"
> > + "eor z5.d, z5.d, z6.d\n"
> > +
> > + "mov w7, %w[z0]\n"
> > + "sub w7, w7, #1\n"
> > +
> > + "1:\n"
> > + "cmp w7, %w[start]\n"
> > + "blt 2f\n"
> > +
> > + "mov z3.d, z1.d\n"
> > + "asr z3.b, p0/m, z3.b, #7\n"
> > + "lsl z1.b, p0/m, z1.b, #1\n"
> > + "and z3.d, z3.d, z4.d\n"
> > + "eor z1.d, z1.d, z3.d\n"
> > +
> > + "mov z8.d, z6.d\n"
> > + "asr z8.b, p0/m, z8.b, #7\n"
> > + "lsl z6.b, p0/m, z6.b, #1\n"
> > + "and z8.d, z8.d, z4.d\n"
> > + "eor z6.d, z6.d, z8.d\n"
> > +
> > + "sxtw x8, w7\n"
> > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > + "add x8, x5, x3\n"
> > + "ld1b z7.b, p0/z, [x6, x8]\n"
> > +
> > + "eor z1.d, z1.d, z2.d\n"
> > + "eor z0.d, z0.d, z2.d\n"
> > +
> > + "eor z6.d, z6.d, z7.d\n"
> > + "eor z5.d, z5.d, z7.d\n"
> > +
> > + "sub w7, w7, #1\n"
> > + "b 1b\n"
> > + "2:\n"
> > +
> > + "mov w7, %w[start]\n"
> > + "sub w7, w7, #1\n"
> > + "3:\n"
> > + "cmp w7, #0\n"
> > + "blt 4f\n"
> > +
> > + "mov z3.d, z1.d\n"
> > + "asr z3.b, p0/m, z3.b, #7\n"
> > + "lsl z1.b, p0/m, z1.b, #1\n"
> > + "and z3.d, z3.d, z4.d\n"
> > + "eor z1.d, z1.d, z3.d\n"
> > +
> > + "mov z8.d, z6.d\n"
> > + "asr z8.b, p0/m, z8.b, #7\n"
> > + "lsl z6.b, p0/m, z6.b, #1\n"
> > + "and z8.d, z8.d, z4.d\n"
> > + "eor z6.d, z6.d, z8.d\n"
> > +
> > + "sub w7, w7, #1\n"
> > + "b 3b\n"
> > + "4:\n"
> > +
> > + "ld1b z2.b, p0/z, [%[q], x5]\n"
> > + "eor z1.d, z1.d, z2.d\n"
> > + "st1b z0.b, p0, [%[p], x5]\n"
> > + "st1b z1.b, p0, [%[q], x5]\n"
> > +
> > + "add x8, x5, x3\n"
> > + "ld1b z7.b, p0/z, [%[q], x8]\n"
> > + "eor z6.d, z6.d, z7.d\n"
> > + "st1b z5.b, p0, [%[p], x8]\n"
> > + "st1b z6.b, p0, [%[q], x8]\n"
> > +
> > + "add x5, x5, x3\n"
> > + "add x5, x5, x3\n"
> > + "cmp x5, %[bytes]\n"
> > + "blt 0b\n"
> > + :
> > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > + [p] "r" (p), [q] "r" (q), [start] "r" (start)
> > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > + "z0", "z1", "z2", "z3", "z4",
> > + "z5", "z6", "z7", "z8"
> > + );
> > +}
> > +
> > +static void raid6_sve4_gen_syndrome_real(int disks, unsigned long
> > bytes, void **ptrs)
> > +{
> > + u8 **dptr = (u8 **)ptrs;
> > + u8 *p, *q;
> > + long z0 = disks - 3;
> > +
> > + p = dptr[z0 + 1];
> > + q = dptr[z0 + 2];
> > +
> > + asm volatile(
> > + ".arch armv8.2-a+sve\n"
> > + "ptrue p0.b\n"
> > + "cntb x3\n"
> > + "mov w4, #0x1d\n"
> > + "dup z4.b, w4\n"
> > + "mov x5, #0\n"
> > +
> > + "0:\n"
> > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > + "ld1b z0.b, p0/z, [x6, x5]\n"
> > + "add x8, x5, x3\n"
> > + "ld1b z5.b, p0/z, [x6, x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z10.b, p0/z, [x6, x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z15.b, p0/z, [x6, x8]\n"
> > +
> > + "mov z1.d, z0.d\n"
> > + "mov z6.d, z5.d\n"
> > + "mov z11.d, z10.d\n"
> > + "mov z16.d, z15.d\n"
> > +
> > + "mov w7, %w[z0]\n"
> > + "sub w7, w7, #1\n"
> > +
> > + "1:\n"
> > + "cmp w7, #0\n"
> > + "blt 2f\n"
> > +
> > + // software pipelining: load data early
> > + "sxtw x8, w7\n"
> > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > + "add x8, x5, x3\n"
> > + "ld1b z7.b, p0/z, [x6, x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z12.b, p0/z, [x6, x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z17.b, p0/z, [x6, x8]\n"
> > +
> > + // math block 1
> > + "mov z3.d, z1.d\n"
> > + "asr z3.b, p0/m, z3.b, #7\n"
> > + "lsl z1.b, p0/m, z1.b, #1\n"
> > + "and z3.d, z3.d, z4.d\n"
> > + "eor z1.d, z1.d, z3.d\n"
> > + "eor z1.d, z1.d, z2.d\n"
> > + "eor z0.d, z0.d, z2.d\n"
> > +
> > + // math block 2
> > + "mov z8.d, z6.d\n"
> > + "asr z8.b, p0/m, z8.b, #7\n"
> > + "lsl z6.b, p0/m, z6.b, #1\n"
> > + "and z8.d, z8.d, z4.d\n"
> > + "eor z6.d, z6.d, z8.d\n"
> > + "eor z6.d, z6.d, z7.d\n"
> > + "eor z5.d, z5.d, z7.d\n"
> > +
> > + // math block 3
> > + "mov z13.d, z11.d\n"
> > + "asr z13.b, p0/m, z13.b, #7\n"
> > + "lsl z11.b, p0/m, z11.b, #1\n"
> > + "and z13.d, z13.d, z4.d\n"
> > + "eor z11.d, z11.d, z13.d\n"
> > + "eor z11.d, z11.d, z12.d\n"
> > + "eor z10.d, z10.d, z12.d\n"
> > +
> > + // math block 4
> > + "mov z18.d, z16.d\n"
> > + "asr z18.b, p0/m, z18.b, #7\n"
> > + "lsl z16.b, p0/m, z16.b, #1\n"
> > + "and z18.d, z18.d, z4.d\n"
> > + "eor z16.d, z16.d, z18.d\n"
> > + "eor z16.d, z16.d, z17.d\n"
> > + "eor z15.d, z15.d, z17.d\n"
> > +
> > + "sub w7, w7, #1\n"
> > + "b 1b\n"
> > + "2:\n"
> > +
> > + "st1b z0.b, p0, [%[p], x5]\n"
> > + "st1b z1.b, p0, [%[q], x5]\n"
> > + "add x8, x5, x3\n"
> > + "st1b z5.b, p0, [%[p], x8]\n"
> > + "st1b z6.b, p0, [%[q], x8]\n"
> > + "add x8, x8, x3\n"
> > + "st1b z10.b, p0, [%[p], x8]\n"
> > + "st1b z11.b, p0, [%[q], x8]\n"
> > + "add x8, x8, x3\n"
> > + "st1b z15.b, p0, [%[p], x8]\n"
> > + "st1b z16.b, p0, [%[q], x8]\n"
> > +
> > + "add x8, x3, x3\n"
> > + "add x5, x5, x8, lsl #1\n"
> > + "cmp x5, %[bytes]\n"
> > + "blt 0b\n"
> > + :
> > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > + [p] "r" (p), [q] "r" (q)
> > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > + "z0", "z1", "z2", "z3", "z4",
> > + "z5", "z6", "z7", "z8",
> > + "z10", "z11", "z12", "z13",
> > + "z15", "z16", "z17", "z18"
> > + );
> > +}
> > +
> > +static void raid6_sve4_xor_syndrome_real(int disks, int start, int
> > stop,
> > + unsigned long bytes, void **ptrs)
> > +{
> > + u8 **dptr = (u8 **)ptrs;
> > + u8 *p, *q;
> > + long z0 = stop;
> > +
> > + p = dptr[disks - 2];
> > + q = dptr[disks - 1];
> > +
> > + asm volatile(
> > + ".arch armv8.2-a+sve\n"
> > + "ptrue p0.b\n"
> > + "cntb x3\n"
> > + "mov w4, #0x1d\n"
> > + "dup z4.b, w4\n"
> > + "mov x5, #0\n"
> > +
> > + "0:\n"
> > + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> > + "ld1b z1.b, p0/z, [x6, x5]\n"
> > + "add x8, x5, x3\n"
> > + "ld1b z6.b, p0/z, [x6, x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z11.b, p0/z, [x6, x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z16.b, p0/z, [x6, x8]\n"
> > +
> > + "ld1b z0.b, p0/z, [%[p], x5]\n"
> > + "add x8, x5, x3\n"
> > + "ld1b z5.b, p0/z, [%[p], x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z10.b, p0/z, [%[p], x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z15.b, p0/z, [%[p], x8]\n"
> > +
> > + "eor z0.d, z0.d, z1.d\n"
> > + "eor z5.d, z5.d, z6.d\n"
> > + "eor z10.d, z10.d, z11.d\n"
> > + "eor z15.d, z15.d, z16.d\n"
> > +
> > + "mov w7, %w[z0]\n"
> > + "sub w7, w7, #1\n"
> > +
> > + "1:\n"
> > + "cmp w7, %w[start]\n"
> > + "blt 2f\n"
> > +
> > + // software pipelining: load data early
> > + "sxtw x8, w7\n"
> > + "ldr x6, [%[dptr], x8, lsl #3]\n"
> > + "ld1b z2.b, p0/z, [x6, x5]\n"
> > + "add x8, x5, x3\n"
> > + "ld1b z7.b, p0/z, [x6, x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z12.b, p0/z, [x6, x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z17.b, p0/z, [x6, x8]\n"
> > +
> > + // math block 1
> > + "mov z3.d, z1.d\n"
> > + "asr z3.b, p0/m, z3.b, #7\n"
> > + "lsl z1.b, p0/m, z1.b, #1\n"
> > + "and z3.d, z3.d, z4.d\n"
> > + "eor z1.d, z1.d, z3.d\n"
> > + "eor z1.d, z1.d, z2.d\n"
> > + "eor z0.d, z0.d, z2.d\n"
> > +
> > + // math block 2
> > + "mov z8.d, z6.d\n"
> > + "asr z8.b, p0/m, z8.b, #7\n"
> > + "lsl z6.b, p0/m, z6.b, #1\n"
> > + "and z8.d, z8.d, z4.d\n"
> > + "eor z6.d, z6.d, z8.d\n"
> > + "eor z6.d, z6.d, z7.d\n"
> > + "eor z5.d, z5.d, z7.d\n"
> > +
> > + // math block 3
> > + "mov z13.d, z11.d\n"
> > + "asr z13.b, p0/m, z13.b, #7\n"
> > + "lsl z11.b, p0/m, z11.b, #1\n"
> > + "and z13.d, z13.d, z4.d\n"
> > + "eor z11.d, z11.d, z13.d\n"
> > + "eor z11.d, z11.d, z12.d\n"
> > + "eor z10.d, z10.d, z12.d\n"
> > +
> > + // math block 4
> > + "mov z18.d, z16.d\n"
> > + "asr z18.b, p0/m, z18.b, #7\n"
> > + "lsl z16.b, p0/m, z16.b, #1\n"
> > + "and z18.d, z18.d, z4.d\n"
> > + "eor z16.d, z16.d, z18.d\n"
> > + "eor z16.d, z16.d, z17.d\n"
> > + "eor z15.d, z15.d, z17.d\n"
> > +
> > + "sub w7, w7, #1\n"
> > + "b 1b\n"
> > + "2:\n"
> > +
> > + "mov w7, %w[start]\n"
> > + "sub w7, w7, #1\n"
> > + "3:\n"
> > + "cmp w7, #0\n"
> > + "blt 4f\n"
> > +
> > + // math block 1
> > + "mov z3.d, z1.d\n"
> > + "asr z3.b, p0/m, z3.b, #7\n"
> > + "lsl z1.b, p0/m, z1.b, #1\n"
> > + "and z3.d, z3.d, z4.d\n"
> > + "eor z1.d, z1.d, z3.d\n"
> > +
> > + // math block 2
> > + "mov z8.d, z6.d\n"
> > + "asr z8.b, p0/m, z8.b, #7\n"
> > + "lsl z6.b, p0/m, z6.b, #1\n"
> > + "and z8.d, z8.d, z4.d\n"
> > + "eor z6.d, z6.d, z8.d\n"
> > +
> > + // math block 3
> > + "mov z13.d, z11.d\n"
> > + "asr z13.b, p0/m, z13.b, #7\n"
> > + "lsl z11.b, p0/m, z11.b, #1\n"
> > + "and z13.d, z13.d, z4.d\n"
> > + "eor z11.d, z11.d, z13.d\n"
> > +
> > + // math block 4
> > + "mov z18.d, z16.d\n"
> > + "asr z18.b, p0/m, z18.b, #7\n"
> > + "lsl z16.b, p0/m, z16.b, #1\n"
> > + "and z18.d, z18.d, z4.d\n"
> > + "eor z16.d, z16.d, z18.d\n"
> > +
> > + "sub w7, w7, #1\n"
> > + "b 3b\n"
> > + "4:\n"
> > +
> > + // Load q and XOR
> > + "ld1b z2.b, p0/z, [%[q], x5]\n"
> > + "add x8, x5, x3\n"
> > + "ld1b z7.b, p0/z, [%[q], x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z12.b, p0/z, [%[q], x8]\n"
> > + "add x8, x8, x3\n"
> > + "ld1b z17.b, p0/z, [%[q], x8]\n"
> > +
> > + "eor z1.d, z1.d, z2.d\n"
> > + "eor z6.d, z6.d, z7.d\n"
> > + "eor z11.d, z11.d, z12.d\n"
> > + "eor z16.d, z16.d, z17.d\n"
> > +
> > + // Store results
> > + "st1b z0.b, p0, [%[p], x5]\n"
> > + "st1b z1.b, p0, [%[q], x5]\n"
> > + "add x8, x5, x3\n"
> > + "st1b z5.b, p0, [%[p], x8]\n"
> > + "st1b z6.b, p0, [%[q], x8]\n"
> > + "add x8, x8, x3\n"
> > + "st1b z10.b, p0, [%[p], x8]\n"
> > + "st1b z11.b, p0, [%[q], x8]\n"
> > + "add x8, x8, x3\n"
> > + "st1b z15.b, p0, [%[p], x8]\n"
> > + "st1b z16.b, p0, [%[q], x8]\n"
> > +
> > + "add x8, x3, x3\n"
> > + "add x5, x5, x8, lsl #1\n"
> > + "cmp x5, %[bytes]\n"
> > + "blt 0b\n"
> > + :
> > + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> > + [p] "r" (p), [q] "r" (q), [start] "r" (start)
> > + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> > + "z0", "z1", "z2", "z3", "z4",
> > + "z5", "z6", "z7", "z8",
> > + "z10", "z11", "z12", "z13",
> > + "z15", "z16", "z17", "z18"
> > + );
> > +}
> > +
> > +#define RAID6_SVE_WRAPPER(_n) \
> > + static void raid6_sve ## _n ## _gen_syndrome(int disks, \
> > + size_t bytes, void **ptrs) \
> > + { \
> > + scoped_ksimd() \
> > + raid6_sve ## _n ## _gen_syndrome_real(disks, \
> > + (unsigned long)bytes, ptrs); \
> > + } \
> > + static void raid6_sve ## _n ## _xor_syndrome(int disks, \
> > + int start, int stop, \
> > + size_t bytes, void **ptrs) \
> > + { \
> > + scoped_ksimd() \
> > + raid6_sve ## _n ## _xor_syndrome_real(disks, \
> > + start, stop, (unsigned long)bytes, ptrs);\
> > + } \
> > + struct raid6_calls const raid6_svex ## _n = { \
> > + raid6_sve ## _n ## _gen_syndrome, \
> > + raid6_sve ## _n ## _xor_syndrome, \
> > + raid6_have_sve, \
> > + "svex" #_n, \
> > + 0 \
> > + }
> > +
> > +static int raid6_have_sve(void)
> > +{
> > + return system_supports_sve();
> > +}
> > +
> > +RAID6_SVE_WRAPPER(1);
> > +RAID6_SVE_WRAPPER(2);
> > +RAID6_SVE_WRAPPER(4);
> > --
> > 2.43.0
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Ard Biesheuvel @ 2026-03-24 8:00 UTC (permalink / raw)
To: Demian Shulhan, Song Liu, Yu Kuai, Will Deacon, Catalin Marinas,
Mark Rutland, broonie, linux-arm-kernel, robin.murphy,
Christoph Hellwig
Cc: Li Nan, linux-raid, linux-kernel
In-Reply-To: <20260318150245.3080719-1-demyansh@gmail.com>
Hi Damian,
On Wed, 18 Mar 2026, at 16:02, Demian Shulhan wrote:
> Implement Scalable Vector Extension (SVE) optimized routines for RAID6
> syndrome generation and recovery on ARM64.
>
> The SVE instruction set allows for variable vector lengths (from 128 to
> 2048 bits), scaling automatically with the hardware capabilities. This
> implementation handles arbitrary SVE vector lengths using the `cntb`
> instruction to determine the runtime vector length.
>
> The implementation introduces `svex1`, `svex2`, and `svex4` algorithms.
> The `svex4` algorithm utilizes loop unrolling by 4 blocks per iteration
> and manual software pipelining (interleaving memory loads with XORs)
> to minimize instruction dependency stalls and maximize CPU pipeline
> utilization and memory bandwidth.
>
> Performance was tested on an AWS Graviton3 (Neoverse-V1) instance which
> features 256-bit SVE vector length. The `svex4` implementation outperforms
> the existing 128-bit `neonx4` baseline for syndrome generation:
>
> raid6: svex4 gen() 19688 MB/s
...
> raid6: neonx4 gen() 19612 MB/s
You're being very generous characterising a 0.3% speedup as 'outperforms'
But the real problem here is that the kernel-mode SIMD API only supports NEON and not SVE, and preserves/restores only the 128-bit view on the NEON/SVE register file. So any context switch or softirq that uses kernel-mode SIMD too, and your SVE register values will get truncated.
Once we encounter a good use case for SVE in the kernel, we might reconsider this, but as it stands, this patch should not be applied.
(leaving the reply untrimmed for the benefit of the cc'ees I added)
> raid6: neonx2 gen() 16248 MB/s
> raid6: neonx1 gen() 13591 MB/s
> raid6: using algorithm svex4 gen() 19688 MB/s
> raid6: .... xor() 11212 MB/s, rmw enabled
> raid6: using neon recovery algorithm
>
> Note that for the recovery path (`xor_syndrome`), NEON may still be
> selected dynamically by the algorithm benchmark, as the recovery
> workload is heavily memory-bound.
>
> Signed-off-by: Demian Shulhan <demyansh@gmail.com>
> Reported-by: kernel test robot <lkp@intel.com>
> Closes:
> https://lore.kernel.org/oe-kbuild-all/202603181940.cFwYmYoi-lkp@intel.com/
> ---
> include/linux/raid/pq.h | 3 +
> lib/raid6/Makefile | 5 +
> lib/raid6/algos.c | 5 +
> lib/raid6/sve.c | 675 ++++++++++++++++++++++++++++++++++++++++
> 4 files changed, 688 insertions(+)
> create mode 100644 lib/raid6/sve.c
>
> diff --git a/include/linux/raid/pq.h b/include/linux/raid/pq.h
> index 2467b3be15c9..787cc57aea9d 100644
> --- a/include/linux/raid/pq.h
> +++ b/include/linux/raid/pq.h
> @@ -140,6 +140,9 @@ extern const struct raid6_calls raid6_neonx1;
> extern const struct raid6_calls raid6_neonx2;
> extern const struct raid6_calls raid6_neonx4;
> extern const struct raid6_calls raid6_neonx8;
> +extern const struct raid6_calls raid6_svex1;
> +extern const struct raid6_calls raid6_svex2;
> +extern const struct raid6_calls raid6_svex4;
>
> /* Algorithm list */
> extern const struct raid6_calls * const raid6_algos[];
> diff --git a/lib/raid6/Makefile b/lib/raid6/Makefile
> index 5be0a4e60ab1..6cdaa6f206fb 100644
> --- a/lib/raid6/Makefile
> +++ b/lib/raid6/Makefile
> @@ -8,6 +8,7 @@ raid6_pq-$(CONFIG_X86) += recov_ssse3.o recov_avx2.o
> mmx.o sse1.o sse2.o avx2.o
> raid6_pq-$(CONFIG_ALTIVEC) += altivec1.o altivec2.o altivec4.o
> altivec8.o \
> vpermxor1.o vpermxor2.o vpermxor4.o
> vpermxor8.o
> raid6_pq-$(CONFIG_KERNEL_MODE_NEON) += neon.o neon1.o neon2.o neon4.o
> neon8.o recov_neon.o recov_neon_inner.o
> +raid6_pq-$(CONFIG_ARM64_SVE) += sve.o
> raid6_pq-$(CONFIG_S390) += s390vx8.o recov_s390xc.o
> raid6_pq-$(CONFIG_LOONGARCH) += loongarch_simd.o recov_loongarch_simd.o
> raid6_pq-$(CONFIG_RISCV_ISA_V) += rvv.o recov_rvv.o
> @@ -67,6 +68,10 @@ CFLAGS_REMOVE_neon2.o += $(CC_FLAGS_NO_FPU)
> CFLAGS_REMOVE_neon4.o += $(CC_FLAGS_NO_FPU)
> CFLAGS_REMOVE_neon8.o += $(CC_FLAGS_NO_FPU)
> CFLAGS_REMOVE_recov_neon_inner.o += $(CC_FLAGS_NO_FPU)
> +
> +CFLAGS_sve.o += $(CC_FLAGS_FPU)
> +CFLAGS_REMOVE_sve.o += $(CC_FLAGS_NO_FPU)
> +
> targets += neon1.c neon2.c neon4.c neon8.c
> $(obj)/neon%.c: $(src)/neon.uc $(src)/unroll.awk FORCE
> $(call if_changed,unroll)
> diff --git a/lib/raid6/algos.c b/lib/raid6/algos.c
> index 799e0e5eac26..0ae73c3a4be3 100644
> --- a/lib/raid6/algos.c
> +++ b/lib/raid6/algos.c
> @@ -66,6 +66,11 @@ const struct raid6_calls * const raid6_algos[] = {
> &raid6_neonx2,
> &raid6_neonx1,
> #endif
> +#ifdef CONFIG_ARM64_SVE
> + &raid6_svex4,
> + &raid6_svex2,
> + &raid6_svex1,
> +#endif
> #ifdef CONFIG_LOONGARCH
> #ifdef CONFIG_CPU_HAS_LASX
> &raid6_lasx,
> diff --git a/lib/raid6/sve.c b/lib/raid6/sve.c
> new file mode 100644
> index 000000000000..d52937f806d4
> --- /dev/null
> +++ b/lib/raid6/sve.c
> @@ -0,0 +1,675 @@
> +// SPDX-License-Identifier: GPL-2.0-or-later
> +/*
> + * RAID-6 syndrome calculation using ARM SVE instructions
> + */
> +
> +#include <linux/raid/pq.h>
> +
> +#ifdef __KERNEL__
> +#include <asm/simd.h>
> +#include <linux/cpufeature.h>
> +#else
> +#define scoped_ksimd()
> +#define system_supports_sve() (1)
> +#endif
> +
> +static void raid6_sve1_gen_syndrome_real(int disks, unsigned long
> bytes, void **ptrs)
> +{
> + u8 **dptr = (u8 **)ptrs;
> + u8 *p, *q;
> + long z0 = disks - 3;
> +
> + p = dptr[z0 + 1];
> + q = dptr[z0 + 2];
> +
> + asm volatile(
> + ".arch armv8.2-a+sve\n"
> + "ptrue p0.b\n"
> + "cntb x3\n"
> + "mov w4, #0x1d\n"
> + "dup z4.b, w4\n"
> + "mov x5, #0\n"
> +
> + "0:\n"
> + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> + "ld1b z0.b, p0/z, [x6, x5]\n"
> + "mov z1.d, z0.d\n"
> +
> + "mov w7, %w[z0]\n"
> + "sub w7, w7, #1\n"
> +
> + "1:\n"
> + "cmp w7, #0\n"
> + "blt 2f\n"
> +
> + "mov z3.d, z1.d\n"
> + "asr z3.b, p0/m, z3.b, #7\n"
> + "lsl z1.b, p0/m, z1.b, #1\n"
> +
> + "and z3.d, z3.d, z4.d\n"
> + "eor z1.d, z1.d, z3.d\n"
> +
> + "sxtw x8, w7\n"
> + "ldr x6, [%[dptr], x8, lsl #3]\n"
> + "ld1b z2.b, p0/z, [x6, x5]\n"
> +
> + "eor z1.d, z1.d, z2.d\n"
> + "eor z0.d, z0.d, z2.d\n"
> +
> + "sub w7, w7, #1\n"
> + "b 1b\n"
> + "2:\n"
> +
> + "st1b z0.b, p0, [%[p], x5]\n"
> + "st1b z1.b, p0, [%[q], x5]\n"
> +
> + "add x5, x5, x3\n"
> + "cmp x5, %[bytes]\n"
> + "blt 0b\n"
> + :
> + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> + [p] "r" (p), [q] "r" (q)
> + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> + "z0", "z1", "z2", "z3", "z4"
> + );
> +}
> +
> +static void raid6_sve1_xor_syndrome_real(int disks, int start, int
> stop,
> + unsigned long bytes, void **ptrs)
> +{
> + u8 **dptr = (u8 **)ptrs;
> + u8 *p, *q;
> + long z0 = stop;
> +
> + p = dptr[disks - 2];
> + q = dptr[disks - 1];
> +
> + asm volatile(
> + ".arch armv8.2-a+sve\n"
> + "ptrue p0.b\n"
> + "cntb x3\n"
> + "mov w4, #0x1d\n"
> + "dup z4.b, w4\n"
> + "mov x5, #0\n"
> +
> + "0:\n"
> + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> + "ld1b z1.b, p0/z, [x6, x5]\n"
> + "ld1b z0.b, p0/z, [%[p], x5]\n"
> + "eor z0.d, z0.d, z1.d\n"
> +
> + "mov w7, %w[z0]\n"
> + "sub w7, w7, #1\n"
> +
> + "1:\n"
> + "cmp w7, %w[start]\n"
> + "blt 2f\n"
> +
> + "mov z3.d, z1.d\n"
> + "asr z3.b, p0/m, z3.b, #7\n"
> + "lsl z1.b, p0/m, z1.b, #1\n"
> + "and z3.d, z3.d, z4.d\n"
> + "eor z1.d, z1.d, z3.d\n"
> +
> + "sxtw x8, w7\n"
> + "ldr x6, [%[dptr], x8, lsl #3]\n"
> + "ld1b z2.b, p0/z, [x6, x5]\n"
> +
> + "eor z1.d, z1.d, z2.d\n"
> + "eor z0.d, z0.d, z2.d\n"
> +
> + "sub w7, w7, #1\n"
> + "b 1b\n"
> + "2:\n"
> +
> + "mov w7, %w[start]\n"
> + "sub w7, w7, #1\n"
> + "3:\n"
> + "cmp w7, #0\n"
> + "blt 4f\n"
> +
> + "mov z3.d, z1.d\n"
> + "asr z3.b, p0/m, z3.b, #7\n"
> + "lsl z1.b, p0/m, z1.b, #1\n"
> + "and z3.d, z3.d, z4.d\n"
> + "eor z1.d, z1.d, z3.d\n"
> +
> + "sub w7, w7, #1\n"
> + "b 3b\n"
> + "4:\n"
> +
> + "ld1b z2.b, p0/z, [%[q], x5]\n"
> + "eor z1.d, z1.d, z2.d\n"
> +
> + "st1b z0.b, p0, [%[p], x5]\n"
> + "st1b z1.b, p0, [%[q], x5]\n"
> +
> + "add x5, x5, x3\n"
> + "cmp x5, %[bytes]\n"
> + "blt 0b\n"
> + :
> + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> + [p] "r" (p), [q] "r" (q), [start] "r" (start)
> + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> + "z0", "z1", "z2", "z3", "z4"
> + );
> +}
> +
> +static void raid6_sve2_gen_syndrome_real(int disks, unsigned long
> bytes, void **ptrs)
> +{
> + u8 **dptr = (u8 **)ptrs;
> + u8 *p, *q;
> + long z0 = disks - 3;
> +
> + p = dptr[z0 + 1];
> + q = dptr[z0 + 2];
> +
> + asm volatile(
> + ".arch armv8.2-a+sve\n"
> + "ptrue p0.b\n"
> + "cntb x3\n"
> + "mov w4, #0x1d\n"
> + "dup z4.b, w4\n"
> + "mov x5, #0\n"
> +
> + "0:\n"
> + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> + "ld1b z0.b, p0/z, [x6, x5]\n"
> + "add x8, x5, x3\n"
> + "ld1b z5.b, p0/z, [x6, x8]\n"
> + "mov z1.d, z0.d\n"
> + "mov z6.d, z5.d\n"
> +
> + "mov w7, %w[z0]\n"
> + "sub w7, w7, #1\n"
> +
> + "1:\n"
> + "cmp w7, #0\n"
> + "blt 2f\n"
> +
> + "mov z3.d, z1.d\n"
> + "asr z3.b, p0/m, z3.b, #7\n"
> + "lsl z1.b, p0/m, z1.b, #1\n"
> + "and z3.d, z3.d, z4.d\n"
> + "eor z1.d, z1.d, z3.d\n"
> +
> + "mov z8.d, z6.d\n"
> + "asr z8.b, p0/m, z8.b, #7\n"
> + "lsl z6.b, p0/m, z6.b, #1\n"
> + "and z8.d, z8.d, z4.d\n"
> + "eor z6.d, z6.d, z8.d\n"
> +
> + "sxtw x8, w7\n"
> + "ldr x6, [%[dptr], x8, lsl #3]\n"
> + "ld1b z2.b, p0/z, [x6, x5]\n"
> + "add x8, x5, x3\n"
> + "ld1b z7.b, p0/z, [x6, x8]\n"
> +
> + "eor z1.d, z1.d, z2.d\n"
> + "eor z0.d, z0.d, z2.d\n"
> +
> + "eor z6.d, z6.d, z7.d\n"
> + "eor z5.d, z5.d, z7.d\n"
> +
> + "sub w7, w7, #1\n"
> + "b 1b\n"
> + "2:\n"
> +
> + "st1b z0.b, p0, [%[p], x5]\n"
> + "st1b z1.b, p0, [%[q], x5]\n"
> + "add x8, x5, x3\n"
> + "st1b z5.b, p0, [%[p], x8]\n"
> + "st1b z6.b, p0, [%[q], x8]\n"
> +
> + "add x5, x5, x3\n"
> + "add x5, x5, x3\n"
> + "cmp x5, %[bytes]\n"
> + "blt 0b\n"
> + :
> + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> + [p] "r" (p), [q] "r" (q)
> + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> + "z0", "z1", "z2", "z3", "z4",
> + "z5", "z6", "z7", "z8"
> + );
> +}
> +
> +static void raid6_sve2_xor_syndrome_real(int disks, int start, int
> stop,
> + unsigned long bytes, void **ptrs)
> +{
> + u8 **dptr = (u8 **)ptrs;
> + u8 *p, *q;
> + long z0 = stop;
> +
> + p = dptr[disks - 2];
> + q = dptr[disks - 1];
> +
> + asm volatile(
> + ".arch armv8.2-a+sve\n"
> + "ptrue p0.b\n"
> + "cntb x3\n"
> + "mov w4, #0x1d\n"
> + "dup z4.b, w4\n"
> + "mov x5, #0\n"
> +
> + "0:\n"
> + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> + "ld1b z1.b, p0/z, [x6, x5]\n"
> + "add x8, x5, x3\n"
> + "ld1b z6.b, p0/z, [x6, x8]\n"
> +
> + "ld1b z0.b, p0/z, [%[p], x5]\n"
> + "ld1b z5.b, p0/z, [%[p], x8]\n"
> +
> + "eor z0.d, z0.d, z1.d\n"
> + "eor z5.d, z5.d, z6.d\n"
> +
> + "mov w7, %w[z0]\n"
> + "sub w7, w7, #1\n"
> +
> + "1:\n"
> + "cmp w7, %w[start]\n"
> + "blt 2f\n"
> +
> + "mov z3.d, z1.d\n"
> + "asr z3.b, p0/m, z3.b, #7\n"
> + "lsl z1.b, p0/m, z1.b, #1\n"
> + "and z3.d, z3.d, z4.d\n"
> + "eor z1.d, z1.d, z3.d\n"
> +
> + "mov z8.d, z6.d\n"
> + "asr z8.b, p0/m, z8.b, #7\n"
> + "lsl z6.b, p0/m, z6.b, #1\n"
> + "and z8.d, z8.d, z4.d\n"
> + "eor z6.d, z6.d, z8.d\n"
> +
> + "sxtw x8, w7\n"
> + "ldr x6, [%[dptr], x8, lsl #3]\n"
> + "ld1b z2.b, p0/z, [x6, x5]\n"
> + "add x8, x5, x3\n"
> + "ld1b z7.b, p0/z, [x6, x8]\n"
> +
> + "eor z1.d, z1.d, z2.d\n"
> + "eor z0.d, z0.d, z2.d\n"
> +
> + "eor z6.d, z6.d, z7.d\n"
> + "eor z5.d, z5.d, z7.d\n"
> +
> + "sub w7, w7, #1\n"
> + "b 1b\n"
> + "2:\n"
> +
> + "mov w7, %w[start]\n"
> + "sub w7, w7, #1\n"
> + "3:\n"
> + "cmp w7, #0\n"
> + "blt 4f\n"
> +
> + "mov z3.d, z1.d\n"
> + "asr z3.b, p0/m, z3.b, #7\n"
> + "lsl z1.b, p0/m, z1.b, #1\n"
> + "and z3.d, z3.d, z4.d\n"
> + "eor z1.d, z1.d, z3.d\n"
> +
> + "mov z8.d, z6.d\n"
> + "asr z8.b, p0/m, z8.b, #7\n"
> + "lsl z6.b, p0/m, z6.b, #1\n"
> + "and z8.d, z8.d, z4.d\n"
> + "eor z6.d, z6.d, z8.d\n"
> +
> + "sub w7, w7, #1\n"
> + "b 3b\n"
> + "4:\n"
> +
> + "ld1b z2.b, p0/z, [%[q], x5]\n"
> + "eor z1.d, z1.d, z2.d\n"
> + "st1b z0.b, p0, [%[p], x5]\n"
> + "st1b z1.b, p0, [%[q], x5]\n"
> +
> + "add x8, x5, x3\n"
> + "ld1b z7.b, p0/z, [%[q], x8]\n"
> + "eor z6.d, z6.d, z7.d\n"
> + "st1b z5.b, p0, [%[p], x8]\n"
> + "st1b z6.b, p0, [%[q], x8]\n"
> +
> + "add x5, x5, x3\n"
> + "add x5, x5, x3\n"
> + "cmp x5, %[bytes]\n"
> + "blt 0b\n"
> + :
> + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> + [p] "r" (p), [q] "r" (q), [start] "r" (start)
> + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> + "z0", "z1", "z2", "z3", "z4",
> + "z5", "z6", "z7", "z8"
> + );
> +}
> +
> +static void raid6_sve4_gen_syndrome_real(int disks, unsigned long
> bytes, void **ptrs)
> +{
> + u8 **dptr = (u8 **)ptrs;
> + u8 *p, *q;
> + long z0 = disks - 3;
> +
> + p = dptr[z0 + 1];
> + q = dptr[z0 + 2];
> +
> + asm volatile(
> + ".arch armv8.2-a+sve\n"
> + "ptrue p0.b\n"
> + "cntb x3\n"
> + "mov w4, #0x1d\n"
> + "dup z4.b, w4\n"
> + "mov x5, #0\n"
> +
> + "0:\n"
> + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> + "ld1b z0.b, p0/z, [x6, x5]\n"
> + "add x8, x5, x3\n"
> + "ld1b z5.b, p0/z, [x6, x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z10.b, p0/z, [x6, x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z15.b, p0/z, [x6, x8]\n"
> +
> + "mov z1.d, z0.d\n"
> + "mov z6.d, z5.d\n"
> + "mov z11.d, z10.d\n"
> + "mov z16.d, z15.d\n"
> +
> + "mov w7, %w[z0]\n"
> + "sub w7, w7, #1\n"
> +
> + "1:\n"
> + "cmp w7, #0\n"
> + "blt 2f\n"
> +
> + // software pipelining: load data early
> + "sxtw x8, w7\n"
> + "ldr x6, [%[dptr], x8, lsl #3]\n"
> + "ld1b z2.b, p0/z, [x6, x5]\n"
> + "add x8, x5, x3\n"
> + "ld1b z7.b, p0/z, [x6, x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z12.b, p0/z, [x6, x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z17.b, p0/z, [x6, x8]\n"
> +
> + // math block 1
> + "mov z3.d, z1.d\n"
> + "asr z3.b, p0/m, z3.b, #7\n"
> + "lsl z1.b, p0/m, z1.b, #1\n"
> + "and z3.d, z3.d, z4.d\n"
> + "eor z1.d, z1.d, z3.d\n"
> + "eor z1.d, z1.d, z2.d\n"
> + "eor z0.d, z0.d, z2.d\n"
> +
> + // math block 2
> + "mov z8.d, z6.d\n"
> + "asr z8.b, p0/m, z8.b, #7\n"
> + "lsl z6.b, p0/m, z6.b, #1\n"
> + "and z8.d, z8.d, z4.d\n"
> + "eor z6.d, z6.d, z8.d\n"
> + "eor z6.d, z6.d, z7.d\n"
> + "eor z5.d, z5.d, z7.d\n"
> +
> + // math block 3
> + "mov z13.d, z11.d\n"
> + "asr z13.b, p0/m, z13.b, #7\n"
> + "lsl z11.b, p0/m, z11.b, #1\n"
> + "and z13.d, z13.d, z4.d\n"
> + "eor z11.d, z11.d, z13.d\n"
> + "eor z11.d, z11.d, z12.d\n"
> + "eor z10.d, z10.d, z12.d\n"
> +
> + // math block 4
> + "mov z18.d, z16.d\n"
> + "asr z18.b, p0/m, z18.b, #7\n"
> + "lsl z16.b, p0/m, z16.b, #1\n"
> + "and z18.d, z18.d, z4.d\n"
> + "eor z16.d, z16.d, z18.d\n"
> + "eor z16.d, z16.d, z17.d\n"
> + "eor z15.d, z15.d, z17.d\n"
> +
> + "sub w7, w7, #1\n"
> + "b 1b\n"
> + "2:\n"
> +
> + "st1b z0.b, p0, [%[p], x5]\n"
> + "st1b z1.b, p0, [%[q], x5]\n"
> + "add x8, x5, x3\n"
> + "st1b z5.b, p0, [%[p], x8]\n"
> + "st1b z6.b, p0, [%[q], x8]\n"
> + "add x8, x8, x3\n"
> + "st1b z10.b, p0, [%[p], x8]\n"
> + "st1b z11.b, p0, [%[q], x8]\n"
> + "add x8, x8, x3\n"
> + "st1b z15.b, p0, [%[p], x8]\n"
> + "st1b z16.b, p0, [%[q], x8]\n"
> +
> + "add x8, x3, x3\n"
> + "add x5, x5, x8, lsl #1\n"
> + "cmp x5, %[bytes]\n"
> + "blt 0b\n"
> + :
> + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> + [p] "r" (p), [q] "r" (q)
> + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> + "z0", "z1", "z2", "z3", "z4",
> + "z5", "z6", "z7", "z8",
> + "z10", "z11", "z12", "z13",
> + "z15", "z16", "z17", "z18"
> + );
> +}
> +
> +static void raid6_sve4_xor_syndrome_real(int disks, int start, int
> stop,
> + unsigned long bytes, void **ptrs)
> +{
> + u8 **dptr = (u8 **)ptrs;
> + u8 *p, *q;
> + long z0 = stop;
> +
> + p = dptr[disks - 2];
> + q = dptr[disks - 1];
> +
> + asm volatile(
> + ".arch armv8.2-a+sve\n"
> + "ptrue p0.b\n"
> + "cntb x3\n"
> + "mov w4, #0x1d\n"
> + "dup z4.b, w4\n"
> + "mov x5, #0\n"
> +
> + "0:\n"
> + "ldr x6, [%[dptr], %[z0], lsl #3]\n"
> + "ld1b z1.b, p0/z, [x6, x5]\n"
> + "add x8, x5, x3\n"
> + "ld1b z6.b, p0/z, [x6, x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z11.b, p0/z, [x6, x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z16.b, p0/z, [x6, x8]\n"
> +
> + "ld1b z0.b, p0/z, [%[p], x5]\n"
> + "add x8, x5, x3\n"
> + "ld1b z5.b, p0/z, [%[p], x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z10.b, p0/z, [%[p], x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z15.b, p0/z, [%[p], x8]\n"
> +
> + "eor z0.d, z0.d, z1.d\n"
> + "eor z5.d, z5.d, z6.d\n"
> + "eor z10.d, z10.d, z11.d\n"
> + "eor z15.d, z15.d, z16.d\n"
> +
> + "mov w7, %w[z0]\n"
> + "sub w7, w7, #1\n"
> +
> + "1:\n"
> + "cmp w7, %w[start]\n"
> + "blt 2f\n"
> +
> + // software pipelining: load data early
> + "sxtw x8, w7\n"
> + "ldr x6, [%[dptr], x8, lsl #3]\n"
> + "ld1b z2.b, p0/z, [x6, x5]\n"
> + "add x8, x5, x3\n"
> + "ld1b z7.b, p0/z, [x6, x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z12.b, p0/z, [x6, x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z17.b, p0/z, [x6, x8]\n"
> +
> + // math block 1
> + "mov z3.d, z1.d\n"
> + "asr z3.b, p0/m, z3.b, #7\n"
> + "lsl z1.b, p0/m, z1.b, #1\n"
> + "and z3.d, z3.d, z4.d\n"
> + "eor z1.d, z1.d, z3.d\n"
> + "eor z1.d, z1.d, z2.d\n"
> + "eor z0.d, z0.d, z2.d\n"
> +
> + // math block 2
> + "mov z8.d, z6.d\n"
> + "asr z8.b, p0/m, z8.b, #7\n"
> + "lsl z6.b, p0/m, z6.b, #1\n"
> + "and z8.d, z8.d, z4.d\n"
> + "eor z6.d, z6.d, z8.d\n"
> + "eor z6.d, z6.d, z7.d\n"
> + "eor z5.d, z5.d, z7.d\n"
> +
> + // math block 3
> + "mov z13.d, z11.d\n"
> + "asr z13.b, p0/m, z13.b, #7\n"
> + "lsl z11.b, p0/m, z11.b, #1\n"
> + "and z13.d, z13.d, z4.d\n"
> + "eor z11.d, z11.d, z13.d\n"
> + "eor z11.d, z11.d, z12.d\n"
> + "eor z10.d, z10.d, z12.d\n"
> +
> + // math block 4
> + "mov z18.d, z16.d\n"
> + "asr z18.b, p0/m, z18.b, #7\n"
> + "lsl z16.b, p0/m, z16.b, #1\n"
> + "and z18.d, z18.d, z4.d\n"
> + "eor z16.d, z16.d, z18.d\n"
> + "eor z16.d, z16.d, z17.d\n"
> + "eor z15.d, z15.d, z17.d\n"
> +
> + "sub w7, w7, #1\n"
> + "b 1b\n"
> + "2:\n"
> +
> + "mov w7, %w[start]\n"
> + "sub w7, w7, #1\n"
> + "3:\n"
> + "cmp w7, #0\n"
> + "blt 4f\n"
> +
> + // math block 1
> + "mov z3.d, z1.d\n"
> + "asr z3.b, p0/m, z3.b, #7\n"
> + "lsl z1.b, p0/m, z1.b, #1\n"
> + "and z3.d, z3.d, z4.d\n"
> + "eor z1.d, z1.d, z3.d\n"
> +
> + // math block 2
> + "mov z8.d, z6.d\n"
> + "asr z8.b, p0/m, z8.b, #7\n"
> + "lsl z6.b, p0/m, z6.b, #1\n"
> + "and z8.d, z8.d, z4.d\n"
> + "eor z6.d, z6.d, z8.d\n"
> +
> + // math block 3
> + "mov z13.d, z11.d\n"
> + "asr z13.b, p0/m, z13.b, #7\n"
> + "lsl z11.b, p0/m, z11.b, #1\n"
> + "and z13.d, z13.d, z4.d\n"
> + "eor z11.d, z11.d, z13.d\n"
> +
> + // math block 4
> + "mov z18.d, z16.d\n"
> + "asr z18.b, p0/m, z18.b, #7\n"
> + "lsl z16.b, p0/m, z16.b, #1\n"
> + "and z18.d, z18.d, z4.d\n"
> + "eor z16.d, z16.d, z18.d\n"
> +
> + "sub w7, w7, #1\n"
> + "b 3b\n"
> + "4:\n"
> +
> + // Load q and XOR
> + "ld1b z2.b, p0/z, [%[q], x5]\n"
> + "add x8, x5, x3\n"
> + "ld1b z7.b, p0/z, [%[q], x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z12.b, p0/z, [%[q], x8]\n"
> + "add x8, x8, x3\n"
> + "ld1b z17.b, p0/z, [%[q], x8]\n"
> +
> + "eor z1.d, z1.d, z2.d\n"
> + "eor z6.d, z6.d, z7.d\n"
> + "eor z11.d, z11.d, z12.d\n"
> + "eor z16.d, z16.d, z17.d\n"
> +
> + // Store results
> + "st1b z0.b, p0, [%[p], x5]\n"
> + "st1b z1.b, p0, [%[q], x5]\n"
> + "add x8, x5, x3\n"
> + "st1b z5.b, p0, [%[p], x8]\n"
> + "st1b z6.b, p0, [%[q], x8]\n"
> + "add x8, x8, x3\n"
> + "st1b z10.b, p0, [%[p], x8]\n"
> + "st1b z11.b, p0, [%[q], x8]\n"
> + "add x8, x8, x3\n"
> + "st1b z15.b, p0, [%[p], x8]\n"
> + "st1b z16.b, p0, [%[q], x8]\n"
> +
> + "add x8, x3, x3\n"
> + "add x5, x5, x8, lsl #1\n"
> + "cmp x5, %[bytes]\n"
> + "blt 0b\n"
> + :
> + : [dptr] "r" (dptr), [z0] "r" (z0), [bytes] "r" (bytes),
> + [p] "r" (p), [q] "r" (q), [start] "r" (start)
> + : "memory", "p0", "x3", "x4", "x5", "x6", "x7", "x8",
> + "z0", "z1", "z2", "z3", "z4",
> + "z5", "z6", "z7", "z8",
> + "z10", "z11", "z12", "z13",
> + "z15", "z16", "z17", "z18"
> + );
> +}
> +
> +#define RAID6_SVE_WRAPPER(_n) \
> + static void raid6_sve ## _n ## _gen_syndrome(int disks, \
> + size_t bytes, void **ptrs) \
> + { \
> + scoped_ksimd() \
> + raid6_sve ## _n ## _gen_syndrome_real(disks, \
> + (unsigned long)bytes, ptrs); \
> + } \
> + static void raid6_sve ## _n ## _xor_syndrome(int disks, \
> + int start, int stop, \
> + size_t bytes, void **ptrs) \
> + { \
> + scoped_ksimd() \
> + raid6_sve ## _n ## _xor_syndrome_real(disks, \
> + start, stop, (unsigned long)bytes, ptrs);\
> + } \
> + struct raid6_calls const raid6_svex ## _n = { \
> + raid6_sve ## _n ## _gen_syndrome, \
> + raid6_sve ## _n ## _xor_syndrome, \
> + raid6_have_sve, \
> + "svex" #_n, \
> + 0 \
> + }
> +
> +static int raid6_have_sve(void)
> +{
> + return system_supports_sve();
> +}
> +
> +RAID6_SVE_WRAPPER(1);
> +RAID6_SVE_WRAPPER(2);
> +RAID6_SVE_WRAPPER(4);
> --
> 2.43.0
^ permalink raw reply
* Re: [PATCH RESEND v3 2/3] lib/raid6: Optimizing the raid6_select_algo time through asynchronous processing
From: Christoph Hellwig @ 2026-03-24 7:56 UTC (permalink / raw)
To: sunliming; +Cc: song, yukuai, akpm, linux-raid, linux-kernel, sunliming
In-Reply-To: <20260320060750.31334-3-sunliming@linux.dev>
On Fri, Mar 20, 2026 at 02:07:49PM +0800, sunliming@linux.dev wrote:
> From: sunliming <sunliming@kylinos.cn>
>
> Optimizing the raid6_select_algo time. In raid6_select_algo(), an raid6 gen
> algorithm is first selected quickly through synchronous processing, while
> the time-consuming process of selecting the optimal algorithm via benchmarking
> is handled asynchronously. This approach speeds up the overall startup time
> and ultimately ensures the selection of an optimal algorithm.
Offloading the selection means it run in parallem with other code,
which could affect different test algorithms differently, so this is not
a good idea.
^ permalink raw reply
* Re: [PATCH RESEND v3 1/3] lib/raid6: Divide the raid6 algorithm selection process into two parts
From: Christoph Hellwig @ 2026-03-24 7:55 UTC (permalink / raw)
To: sunliming; +Cc: song, yukuai, akpm, linux-raid, linux-kernel, sunliming
In-Reply-To: <20260320060750.31334-2-sunliming@linux.dev>
I've fixed this slightly different in my "cleanup the RAID6 P/Q library"
series by simply not considering non-priority algorithms at all.
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Christoph Hellwig @ 2026-03-24 7:45 UTC (permalink / raw)
To: Demian Shulhan
Cc: Song Liu, Yu Kuai, Li Nan, linux-raid, linux-kernel,
kernel test robot, Catalin Marinas, Ard Biesheuvel, Will Deacon,
linux-arm-kernel
In-Reply-To: <20260318150245.3080719-1-demyansh@gmail.com>
Hi Damian,
thanks for looking into this.
I've added the arm64 maintainers and arm list as that's your best bet
for someone actually understanding the low-level assembly code.
On Wed, Mar 18, 2026 at 03:02:45PM +0000, Demian Shulhan wrote:
> Note that for the recovery path (`xor_syndrome`), NEON may still be
> selected dynamically by the algorithm benchmark, as the recovery
> workload is heavily memory-bound.
The recovery side has no benchmarking, you need to manually select
a priority.
Note that I just sent out a "cleanup the RAID6 P/Q library" series that
make this more explicit. It also make it clear by prioritizing
implementations using better instructions available we can short-cut
the generation side probing path a lot, which might be worth looking
into for this.
I'm also curious if you looked why the 4x unroll is slower than
the lesser unroll, and if that is inherent in the implementation. Or
just an effect of the small number of disks in that we don't actually
have 4 disks to unroll for every other iteration. I.e. what would the
numbers be if RAID6_TEST_DISKS was increased to 10 or 18?
I plan into potentially select the unrolling variants based on the
number of "disks" to calculate over as a follow-on.
We'll have to wait for review on my series, but I'd love to just rebase
this ontop if possible. I can offer to do the work, but I'd need to
run it past you for testing and final review.
> +static void raid6_sve1_gen_syndrome_real(int disks, unsigned long bytes, void **ptrs)
Overly long line.
> +{
> + u8 **dptr = (u8 **)ptrs;
> + u8 *p, *q;
> + long z0 = disks - 3;
> +
> + p = dptr[z0 + 1];
> + q = dptr[z0 + 2];
I know all this is derived from existing code, but as I started to hate
that I'll add my cosmetic comments:
This would read nicer by initializing at declaration time:
u8 **dptr = (u8 **)ptrs;
long z0 = disks - 3;
u8 *p = dptr[z0 + 1];
u8 *q = dptr[z0 + 2];
Also z0 might better be named z_last or last_disk, or stop as in the
_xor variant routines.
> + asm volatile(
But I wonder if just implementing the entire routine as assembly in a
.S file would make more sense than this anyway?
> +static void raid6_sve1_xor_syndrome_real(int disks, int start, int stop,
> + unsigned long bytes, void **ptrs)
> +{
> + u8 **dptr = (u8 **)ptrs;
> + u8 *p, *q;
> + long z0 = stop;
> +
> + p = dptr[disks - 2];
> + q = dptr[disks - 1];
> +
> + asm volatile(
Same comments here, plus just dropping z0 vs using stop directly.
> +#define RAID6_SVE_WRAPPER(_n) \
> + static void raid6_sve ## _n ## _gen_syndrome(int disks, \
> + size_t bytes, void **ptrs) \
> + { \
> + scoped_ksimd() \
> + raid6_sve ## _n ## _gen_syndrome_real(disks, \
> + (unsigned long)bytes, ptrs); \
Missing indentation after the scoped_ksimd(). A lot of other code uses
separate compilation units for the SIMD code, which seems pretty useful
to avoid mixing SIMD with non-SIMD code. That would also combine nicely
with the suggestion above to implement the low-level routines entirely
in assembly.
I'll leave comments on the actual assembly details to people that
actually know ARM64 assembly.
^ permalink raw reply
* [PATCH v3 1/1] md/raid1: serialize overlap io for writemostly disk
From: Xiao Ni @ 2026-03-24 7:24 UTC (permalink / raw)
To: yukuai; +Cc: ncroxon, linux-raid
Previously, using wait_event() would wake up all waiters simultaneously,
and they would compete for the tree lock. The bio which gets the lock
first will be handled, so the write sequence cannot be guaranteed.
For example:
bio1(100,200)
bio2(150,200)
bio3(150,300)
The write sequence of fast device is bio1,bio2,bio3. But the write sequence
of slow device could be bio1,bio3,bio2 due to lock competition. This causes
data corruption.
Replace waitqueue with a fifo list to guarantee the write sequence. And it
also needs to iterate the list when removing one entry. If not, it may miss
the opportunity to wake up the waiting io.
For example:
bio1(1,3), bio2(2,4)
bio3(5,7), bio4(6,8)
These four bios are in the same bucket. bio1 and bio3 are inserted into
the rbtree. bio2 and bio4 are added to the waiting list and bio2 is the
first one. bio3 returns from slow disk and tries to wake up the waiting
bios. bio2 is removed from the list and will be handled. But bio1 hasn't
finished. So bio2 will be added into waiting list again. Then bio1 returns
from slow disk and wakes up waiting bios. bio4 is removed from the list
and will be handled. Now bio1, bio3 and bio4 all finish and bio2 is left
on the waiting list. So it needs to iterate the waiting list to wake up
the right bio.
Signed-off-by: Xiao Ni <xni@redhat.com>
---
v2: use prepare_to_wait_exclusive
v3: return back to self managed fifo list to find the right waiting node
drivers/md/md.c | 1 -
drivers/md/md.h | 5 ++++-
drivers/md/raid1.c | 45 ++++++++++++++++++++++++++++++++++-----------
3 files changed, 38 insertions(+), 13 deletions(-)
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 521d9b34cd9e..3348224e36f8 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -188,7 +188,6 @@ static int rdev_init_serial(struct md_rdev *rdev)
spin_lock_init(&serial_tmp->serial_lock);
serial_tmp->serial_rb = RB_ROOT_CACHED;
- init_waitqueue_head(&serial_tmp->serial_io_wait);
}
rdev->serial = serial;
diff --git a/drivers/md/md.h b/drivers/md/md.h
index ac84289664cd..2208da7915e6 100644
--- a/drivers/md/md.h
+++ b/drivers/md/md.h
@@ -126,7 +126,6 @@ enum sync_action {
struct serial_in_rdev {
struct rb_root_cached serial_rb;
spinlock_t serial_lock;
- wait_queue_head_t serial_io_wait;
};
/*
@@ -381,7 +380,11 @@ struct serial_info {
struct rb_node node;
sector_t start; /* start sector of rb node */
sector_t last; /* end sector of rb node */
+ sector_t wnode_start; /* address of waiting nodes on the same list */
sector_t _subtree_last; /* highest sector in subtree of rb node */
+ struct list_head list_node;
+ struct list_head waiters;
+ struct completion ready;
};
/*
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index 16f671ab12c0..1a8f876765c2 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -57,21 +57,29 @@ INTERVAL_TREE_DEFINE(struct serial_info, node, sector_t, _subtree_last,
START, LAST, static inline, raid1_rb);
static int check_and_add_serial(struct md_rdev *rdev, struct r1bio *r1_bio,
- struct serial_info *si, int idx)
+ struct serial_info *si)
{
unsigned long flags;
int ret = 0;
sector_t lo = r1_bio->sector;
sector_t hi = lo + r1_bio->sectors - 1;
+ int idx = sector_to_idx(r1_bio->sector);
struct serial_in_rdev *serial = &rdev->serial[idx];
+ struct serial_info *head_si;
spin_lock_irqsave(&serial->serial_lock, flags);
/* collision happened */
- if (raid1_rb_iter_first(&serial->serial_rb, lo, hi))
+ head_si = raid1_rb_iter_first(&serial->serial_rb, lo, hi);
+ if (head_si && head_si != si) {
+ si->start = lo;
+ si->last = hi;
+ si->wnode_start = head_si->wnode_start;
+ list_add_tail(&si->list_node, &head_si->waiters);
ret = -EBUSY;
- else {
+ } else if (!head_si) {
si->start = lo;
si->last = hi;
+ si->wnode_start = si->start;
raid1_rb_insert(si, &serial->serial_rb);
}
spin_unlock_irqrestore(&serial->serial_lock, flags);
@@ -83,19 +91,22 @@ static void wait_for_serialization(struct md_rdev *rdev, struct r1bio *r1_bio)
{
struct mddev *mddev = rdev->mddev;
struct serial_info *si;
- int idx = sector_to_idx(r1_bio->sector);
- struct serial_in_rdev *serial = &rdev->serial[idx];
if (WARN_ON(!mddev->serial_info_pool))
return;
si = mempool_alloc(mddev->serial_info_pool, GFP_NOIO);
- wait_event(serial->serial_io_wait,
- check_and_add_serial(rdev, r1_bio, si, idx) == 0);
+ INIT_LIST_HEAD(&si->waiters);
+ INIT_LIST_HEAD(&si->list_node);
+ init_completion(&si->ready);
+ while (check_and_add_serial(rdev, r1_bio, si)) {
+ wait_for_completion(&si->ready);
+ reinit_completion(&si->ready);
+ }
}
static void remove_serial(struct md_rdev *rdev, sector_t lo, sector_t hi)
{
- struct serial_info *si;
+ struct serial_info *si, *iter_si;
unsigned long flags;
int found = 0;
struct mddev *mddev = rdev->mddev;
@@ -106,16 +117,28 @@ static void remove_serial(struct md_rdev *rdev, sector_t lo, sector_t hi)
for (si = raid1_rb_iter_first(&serial->serial_rb, lo, hi);
si; si = raid1_rb_iter_next(si, lo, hi)) {
if (si->start == lo && si->last == hi) {
- raid1_rb_remove(si, &serial->serial_rb);
- mempool_free(si, mddev->serial_info_pool);
found = 1;
break;
}
}
if (!found)
WARN(1, "The write IO is not recorded for serialization\n");
+ else {
+ raid1_rb_remove(si, &serial->serial_rb);
+ if (!list_empty(&si->waiters)) {
+ list_for_each_entry(iter_si, &si->waiters, list_node) {
+ if (iter_si->wnode_start == si->wnode_start) {
+ list_del_init(&iter_si->list_node);
+ list_splice_init(&si->waiters, &iter_si->waiters);
+ raid1_rb_insert(iter_si, &serial->serial_rb);
+ complete(&iter_si->ready);
+ break;
+ }
+ }
+ }
+ mempool_free(si, mddev->serial_info_pool);
+ }
spin_unlock_irqrestore(&serial->serial_lock, flags);
- wake_up(&serial->serial_io_wait);
}
/*
--
2.50.1 (Apple Git-155)
^ permalink raw reply related
* Re: [PATCH 2/2] nvme-multipath: enable PCI P2PDMA for multipath devices
From: Christoph Hellwig @ 2026-03-24 6:49 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: song, yukuai, linan122, kbusch, axboe, hch, sagi, linux-raid,
linux-nvme, Kiran Kumar Modukuri
In-Reply-To: <20260323234416.46944-3-kch@nvidia.com>
On Mon, Mar 23, 2026 at 04:44:16PM -0700, Chaitanya Kulkarni wrote:
> NVMe multipath does not expose BLK_FEAT_PCI_P2PDMA on the head disk
> even when the underlying controller supports it.
>
> Set BLK_FEAT_PCI_P2PDMA in nvme_mpath_alloc_disk() when the controller
> advertises P2PDMA support via ctrl->ops->supports_pci_p2pdma.
>
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
> Signed-off-by: Kiran Kumar Modukuri <kmodukuri@nvidia.com>
This signoff chain is wrong - you are both author and patch submitter,
so other signoffs should not appear.
Otherwise looks good:
> diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
> index ba00f0b72b85..c49fca43ef19 100644
> --- a/drivers/nvme/host/multipath.c
> +++ b/drivers/nvme/host/multipath.c
> @@ -737,6 +737,9 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
> BLK_FEAT_POLL | BLK_FEAT_ATOMIC_WRITES;
> if (head->ids.csi == NVME_CSI_ZNS)
> lim.features |= BLK_FEAT_ZONED;
> + if (ctrl->ops && ctrl->ops->supports_pci_p2pdma &&
> + ctrl->ops->supports_pci_p2pdma(ctrl))
> + lim.features |= BLK_FEAT_PCI_P2PDMA;
This assumes all controllers support P2P, but we allow matching
over different transports. So you'll need to do the same scheme
as for MD RAID that checks that every member supports P2P.
^ permalink raw reply
* Re: [PATCH 1/2] md: Add PCI_P2PDMA support for MD RAID volumes
From: Christoph Hellwig @ 2026-03-24 6:48 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: song, yukuai, linan122, kbusch, axboe, hch, sagi, linux-raid,
linux-nvme, Kiran Kumar Modukuri
In-Reply-To: <20260323234416.46944-2-kch@nvidia.com>
On Mon, Mar 23, 2026 at 04:44:15PM -0700, Chaitanya Kulkarni wrote:
> MD RAID does not propagate BLK_FEAT_PCI_P2PDMA from member devices to
> the RAID device, preventing peer-to-peer DMA through the RAID layer even
> when all underlying devices support it.
>
> Enable BLK_FEAT_PCI_P2PDMA by default in md_init_stacking_limits() and
> clear it in mddev_stack_rdev_limits() during array init and
> mddev_stack_new_rdev() during hot-add if any member device lacks support.
>
> Tested with RAID arrays containing multiple NVMe devices with P2PDMA
> support, confirming that peer-to-peer transfers work correctly through
> the RAID layer.
Which personalities did you test this with? Parity RAID needs to
copy from the data payload, which is not going to work very well
with P2P mappings.
^ permalink raw reply
* [PATCH 17/17] raid6_kunit: randomize buffer alignment
From: Christoph Hellwig @ 2026-03-24 6:40 UTC (permalink / raw)
To: Andrew Morton
Cc: Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, H. Peter Anvin, Herbert Xu, Dan Williams,
Chris Mason, David Sterba, Arnd Bergmann, Song Liu, Yu Kuai,
Li Nan, linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, linux-crypto, linux-btrfs, linux-arch,
linux-raid
In-Reply-To: <20260324064115.3217136-1-hch@lst.de>
Add code to add random alignment to the buffers to test the case where
they are not page aligned, and to move the buffers to the end of the
allocation so that they are next to the vmalloc guard page.
This does not include the recovery buffers as the recovery requires
page alignment.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
lib/raid/raid6/tests/raid6_kunit.c | 41 +++++++++++++++++++++++++-----
1 file changed, 35 insertions(+), 6 deletions(-)
diff --git a/lib/raid/raid6/tests/raid6_kunit.c b/lib/raid/raid6/tests/raid6_kunit.c
index a0a473643e91..d2fd4a9b74d4 100644
--- a/lib/raid/raid6/tests/raid6_kunit.c
+++ b/lib/raid/raid6/tests/raid6_kunit.c
@@ -20,6 +20,7 @@ MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
static struct rnd_state rng;
static void *test_buffers[RAID6_KUNIT_MAX_BUFFERS];
+static void *aligned_buffers[RAID6_KUNIT_MAX_BUFFERS];
static void *test_recov_buffers[RAID6_KUNIT_MAX_FAILURES];
static size_t test_buflen;
@@ -43,6 +44,14 @@ static unsigned int random_length(unsigned int max_length)
return (rand32() % (max_length + 1)) & ~511;
}
+/* Generate a random alignment that is a multiple of 64. */
+static unsigned int random_alignment(unsigned int max_alignment)
+{
+ if (max_alignment == 0)
+ return 0;
+ return (rand32() % (max_alignment + 1)) & ~63;
+}
+
static void makedata(int start, int stop)
{
int i;
@@ -73,7 +82,7 @@ static void test_recover_one(struct kunit *test, unsigned int nr_buffers,
for (i = 0; i < RAID6_KUNIT_MAX_FAILURES; i++)
memset(test_recov_buffers[i], 0xf0, test_buflen);
- memcpy(dataptrs, test_buffers, sizeof(dataptrs));
+ memcpy(dataptrs, aligned_buffers, sizeof(dataptrs));
dataptrs[faila] = test_recov_buffers[0];
dataptrs[failb] = test_recov_buffers[1];
@@ -95,13 +104,13 @@ static void test_recover_one(struct kunit *test, unsigned int nr_buffers,
ta->recov->data2(nr_buffers, len, faila, failb, dataptrs);
}
- KUNIT_EXPECT_MEMEQ_MSG(test, test_buffers[faila], test_recov_buffers[0],
+ KUNIT_EXPECT_MEMEQ_MSG(test, aligned_buffers[faila], dataptrs[faila],
len,
"faila miscompared: %3d[%c] buffers %u len %u (failb=%3d[%c])\n",
faila, member_type(nr_buffers, faila),
nr_buffers, len,
failb, member_type(nr_buffers, failb));
- KUNIT_EXPECT_MEMEQ_MSG(test, test_buffers[failb], test_recov_buffers[1],
+ KUNIT_EXPECT_MEMEQ_MSG(test, aligned_buffers[failb], dataptrs[failb],
len,
"failb miscompared: %3d[%c] buffers %u len %u (faila=%3d[%c])\n",
failb, member_type(nr_buffers, failb),
@@ -140,9 +149,9 @@ static void test_rmw_one(struct kunit *test, unsigned int nr_buffers,
{
const struct test_args *ta = test->param_value;
- ta->gen->xor_syndrome(nr_buffers, p1, p2, len, test_buffers);
+ ta->gen->xor_syndrome(nr_buffers, p1, p2, len, aligned_buffers);
makedata(p1, p2);
- ta->gen->xor_syndrome(nr_buffers, p1, p2, len, test_buffers);
+ ta->gen->xor_syndrome(nr_buffers, p1, p2, len, aligned_buffers);
test_recover(test, nr_buffers, len);
}
@@ -168,13 +177,33 @@ static void raid6_test_one(struct kunit *test)
unsigned int nr_buffers =
(rand32() % (RAID6_KUNIT_MAX_BUFFERS - 2)) + 3;
unsigned int len = random_length(RAID6_KUNIT_MAX_BYTES);
+ unsigned int max_alignment;
+ int i;
/* Nuke syndromes */
memset(test_buffers[nr_buffers - 2], 0xee, test_buflen);
memset(test_buffers[nr_buffers - 1], 0xee, test_buflen);
+ /*
+ * If we're not using the entire buffer size, inject randomize alignment
+ * into the buffer.
+ */
+ max_alignment = RAID6_KUNIT_MAX_BYTES - len;
+ if (rand32() % 2 == 0) {
+ /* Use random alignments mod 64 */
+ for (i = 0; i < nr_buffers; i++)
+ aligned_buffers[i] = test_buffers[i] +
+ random_alignment(max_alignment);
+ } else {
+ /* Go up to the guard page, to catch buffer overreads */
+ unsigned int align = test_buflen - len;
+
+ for (i = 0; i < nr_buffers; i++)
+ aligned_buffers[i] = test_buffers[i] + align;
+ }
+
/* Generate assumed good syndrome */
- ta->gen->gen_syndrome(nr_buffers, len, test_buffers);
+ ta->gen->gen_syndrome(nr_buffers, len, aligned_buffers);
test_recover(test, nr_buffers, len);
--
2.47.3
^ permalink raw reply related
* [PATCH 16/17] raid6_kunit: randomize parameters and increase limits
From: Christoph Hellwig @ 2026-03-24 6:40 UTC (permalink / raw)
To: Andrew Morton
Cc: Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, H. Peter Anvin, Herbert Xu, Dan Williams,
Chris Mason, David Sterba, Arnd Bergmann, Song Liu, Yu Kuai,
Li Nan, linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, linux-crypto, linux-btrfs, linux-arch,
linux-raid
In-Reply-To: <20260324064115.3217136-1-hch@lst.de>
The current test has double-quadratic behavior in the selection for
the updated ("XORed") disks, and in the selection of updated pointers,
which makes scaling it to more tests difficult. At the same time it
only ever tests with the maximum number of disks, which leaves a
coverage hole for smaller ones.
Fix this by randomizing the total number, failed disks and regions
to update, and increasing the upper number of tests disks.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
lib/raid/raid6/tests/raid6_kunit.c | 188 +++++++++++++++++++----------
1 file changed, 124 insertions(+), 64 deletions(-)
diff --git a/lib/raid/raid6/tests/raid6_kunit.c b/lib/raid/raid6/tests/raid6_kunit.c
index ba6cfabc67a4..a0a473643e91 100644
--- a/lib/raid/raid6/tests/raid6_kunit.c
+++ b/lib/raid/raid6/tests/raid6_kunit.c
@@ -13,13 +13,15 @@
MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
#define RAID6_KUNIT_SEED 42
+#define RAID6_KUNIT_NUM_TEST_ITERS 10
+#define RAID6_KUNIT_MAX_BUFFERS 64 /* Including P and Q */
#define RAID6_KUNIT_MAX_FAILURES 2
-
-#define NDISKS 16 /* Including P and Q */
+#define RAID6_KUNIT_MAX_BYTES PAGE_SIZE
static struct rnd_state rng;
-static void *test_buffers[NDISKS];
+static void *test_buffers[RAID6_KUNIT_MAX_BUFFERS];
static void *test_recov_buffers[RAID6_KUNIT_MAX_FAILURES];
+static size_t test_buflen;
struct test_args {
unsigned int recov_idx;
@@ -30,105 +32,162 @@ struct test_args {
static struct test_args args;
+static u32 rand32(void)
+{
+ return prandom_u32_state(&rng);
+}
+
+/* Generate a random length that is a multiple of 512. */
+static unsigned int random_length(unsigned int max_length)
+{
+ return (rand32() % (max_length + 1)) & ~511;
+}
+
static void makedata(int start, int stop)
{
int i;
for (i = start; i <= stop; i++)
- prandom_bytes_state(&rng, test_buffers[i], PAGE_SIZE);
+ prandom_bytes_state(&rng, test_buffers[i], test_buflen);
}
-static char member_type(int d)
+static char member_type(unsigned int nr_buffers, int d)
{
- switch (d) {
- case NDISKS-2:
+ if (d == nr_buffers - 2)
return 'P';
- case NDISKS-1:
+ if (d == nr_buffers - 1)
return 'Q';
- default:
- return 'D';
- }
+ return 'D';
}
-static void test_recover(struct kunit *test, int faila, int failb)
+static void test_recover_one(struct kunit *test, unsigned int nr_buffers,
+ unsigned int len, int faila, int failb)
{
const struct test_args *ta = test->param_value;
- void *dataptrs[NDISKS];
+ void *dataptrs[RAID6_KUNIT_MAX_BUFFERS];
int i;
+ if (faila > failb)
+ swap(faila, failb);
+
for (i = 0; i < RAID6_KUNIT_MAX_FAILURES; i++)
- memset(test_recov_buffers[i], 0xf0, PAGE_SIZE);
+ memset(test_recov_buffers[i], 0xf0, test_buflen);
memcpy(dataptrs, test_buffers, sizeof(dataptrs));
dataptrs[faila] = test_recov_buffers[0];
dataptrs[failb] = test_recov_buffers[1];
- if (faila > failb)
- swap(faila, failb);
-
- if (failb == NDISKS - 1) {
+ if (failb == nr_buffers - 1) {
/*
* We don't implement the data+Q failure scenario, since it
* is equivalent to a RAID-5 failure (XOR, then recompute Q).
*/
- if (faila != NDISKS - 2)
+ if (WARN_ON_ONCE(faila != nr_buffers - 2))
return;
/* P+Q failure. Just rebuild the syndrome. */
- ta->gen->gen_syndrome(NDISKS, PAGE_SIZE, dataptrs);
- } else if (failb == NDISKS - 2) {
+ ta->gen->gen_syndrome(nr_buffers, len, dataptrs);
+ } else if (failb == nr_buffers - 2) {
/* data+P failure. */
- ta->recov->datap(NDISKS, PAGE_SIZE, faila, dataptrs);
+ ta->recov->datap(nr_buffers, len, faila, dataptrs);
} else {
/* data+data failure. */
- ta->recov->data2(NDISKS, PAGE_SIZE, faila, failb, dataptrs);
+ ta->recov->data2(nr_buffers, len, faila, failb, dataptrs);
}
KUNIT_EXPECT_MEMEQ_MSG(test, test_buffers[faila], test_recov_buffers[0],
- PAGE_SIZE,
- "faila miscompared: %3d[%c] (failb=%3d[%c])\n",
- faila, member_type(faila),
- failb, member_type(failb));
+ len,
+ "faila miscompared: %3d[%c] buffers %u len %u (failb=%3d[%c])\n",
+ faila, member_type(nr_buffers, faila),
+ nr_buffers, len,
+ failb, member_type(nr_buffers, failb));
KUNIT_EXPECT_MEMEQ_MSG(test, test_buffers[failb], test_recov_buffers[1],
- PAGE_SIZE,
- "failb miscompared: %3d[%c] (faila=%3d[%c])\n",
- failb, member_type(failb),
- faila, member_type(faila));
+ len,
+ "failb miscompared: %3d[%c] buffers %u len %u (faila=%3d[%c])\n",
+ failb, member_type(nr_buffers, failb),
+ nr_buffers, len,
+ faila, member_type(nr_buffers, faila));
}
-static void raid6_test(struct kunit *test)
+static void test_recover(struct kunit *test, unsigned int nr_buffers,
+ unsigned int len)
+{
+ int iterations, i;
+
+ /* Test P+Q recovery */
+ test_recover_one(test, nr_buffers, len, nr_buffers - 2, nr_buffers - 1);
+
+ /* Test data+P recovery */
+ for (i = 0; i < nr_buffers - 2; i++)
+ test_recover_one(test, nr_buffers, len, i, nr_buffers - 2);
+
+ /* Test data+data recovery using random sampling */
+ iterations = nr_buffers * 2; /* should provide good enough coverage */
+ for (i = 0; i < iterations; i++) {
+ int faila = rand32() % (nr_buffers - 2), failb;
+
+ do {
+ failb = rand32() % (nr_buffers - 2);
+ } while (failb == faila);
+
+ test_recover_one(test, nr_buffers, len, faila, failb);
+ }
+}
+
+/* Simulate rmw run */
+static void test_rmw_one(struct kunit *test, unsigned int nr_buffers,
+ unsigned int len, int p1, int p2)
+{
+ const struct test_args *ta = test->param_value;
+
+ ta->gen->xor_syndrome(nr_buffers, p1, p2, len, test_buffers);
+ makedata(p1, p2);
+ ta->gen->xor_syndrome(nr_buffers, p1, p2, len, test_buffers);
+ test_recover(test, nr_buffers, len);
+}
+
+static void test_rmw(struct kunit *test, unsigned int nr_buffers,
+ unsigned int len)
+{
+ int iterations = nr_buffers / 2, i;
+
+ for (i = 0; i < iterations; i++) {
+ int p1 = rand32() % (nr_buffers - 2);
+ int p2 = rand32() % (nr_buffers - 2);
+
+ if (p2 < p1)
+ swap(p1, p2);
+ test_rmw_one(test, nr_buffers, len, p1, p2);
+ }
+}
+
+static void raid6_test_one(struct kunit *test)
{
const struct test_args *ta = test->param_value;
- int i, j, p1, p2;
+ /* including P/Q we need at least three buffers */
+ unsigned int nr_buffers =
+ (rand32() % (RAID6_KUNIT_MAX_BUFFERS - 2)) + 3;
+ unsigned int len = random_length(RAID6_KUNIT_MAX_BYTES);
/* Nuke syndromes */
- memset(test_buffers[NDISKS - 2], 0xee, PAGE_SIZE);
- memset(test_buffers[NDISKS - 1], 0xee, PAGE_SIZE);
+ memset(test_buffers[nr_buffers - 2], 0xee, test_buflen);
+ memset(test_buffers[nr_buffers - 1], 0xee, test_buflen);
/* Generate assumed good syndrome */
- ta->gen->gen_syndrome(NDISKS, PAGE_SIZE, test_buffers);
-
- for (i = 0; i < NDISKS - 1; i++)
- for (j = i + 1; j < NDISKS; j++)
- test_recover(test, i, j);
-
- if (!ta->gen->xor_syndrome)
- return;
-
- for (p1 = 0; p1 < NDISKS - 2; p1++) {
- for (p2 = p1; p2 < NDISKS - 2; p2++) {
- /* Simulate rmw run */
- ta->gen->xor_syndrome(NDISKS, p1, p2, PAGE_SIZE,
- test_buffers);
- makedata(p1, p2);
- ta->gen->xor_syndrome(NDISKS, p1, p2, PAGE_SIZE,
- test_buffers);
-
- for (i = 0; i < NDISKS - 1; i++)
- for (j = i + 1; j < NDISKS; j++)
- test_recover(test, i, j);
- }
- }
+ ta->gen->gen_syndrome(nr_buffers, len, test_buffers);
+
+ test_recover(test, nr_buffers, len);
+
+ if (ta->gen->xor_syndrome)
+ test_rmw(test, nr_buffers, len);
+}
+
+static void raid6_test(struct kunit *test)
+{
+ int i;
+
+ for (i = 0; i < RAID6_KUNIT_NUM_TEST_ITERS; i++)
+ raid6_test_one(test);
}
static const void *raid6_gen_params(struct kunit *test, const void *prev,
@@ -172,23 +231,24 @@ static int raid6_suite_init(struct kunit_suite *suite)
* so that it is immediately followed by a guard page. This allows
* buffer overreads to be detected, even in assembly code.
*/
+ test_buflen = round_up(RAID6_KUNIT_MAX_BYTES, PAGE_SIZE);
for (i = 0; i < RAID6_KUNIT_MAX_FAILURES; i++) {
- test_recov_buffers[i] = vmalloc(PAGE_SIZE);
+ test_recov_buffers[i] = vmalloc(test_buflen);
if (!test_recov_buffers[i])
goto out_free_recov_buffers;
}
- for (i = 0; i < NDISKS; i++) {
- test_buffers[i] = vmalloc(PAGE_SIZE);
+ for (i = 0; i < RAID6_KUNIT_MAX_BUFFERS; i++) {
+ test_buffers[i] = vmalloc(test_buflen);
if (!test_buffers[i])
goto out_free_buffers;
}
- makedata(0, NDISKS - 1);
+ makedata(0, RAID6_KUNIT_MAX_BUFFERS - 1);
return 0;
out_free_buffers:
- for (i = 0; i < NDISKS; i++)
+ for (i = 0; i < RAID6_KUNIT_MAX_BUFFERS; i++)
vfree(test_buffers[i]);
out_free_recov_buffers:
for (i = 0; i < RAID6_KUNIT_MAX_FAILURES; i++)
@@ -200,7 +260,7 @@ static void raid6_suite_exit(struct kunit_suite *suite)
{
int i;
- for (i = 0; i < NDISKS; i++)
+ for (i = 0; i < RAID6_KUNIT_MAX_BUFFERS; i++)
vfree(test_buffers[i]);
for (i = 0; i < RAID6_KUNIT_MAX_FAILURES; i++)
vfree(test_recov_buffers[i]);
--
2.47.3
^ permalink raw reply related
* [PATCH 15/17] raid6_kunit: cleanup dataptr handling
From: Christoph Hellwig @ 2026-03-24 6:40 UTC (permalink / raw)
To: Andrew Morton
Cc: Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, H. Peter Anvin, Herbert Xu, Dan Williams,
Chris Mason, David Sterba, Arnd Bergmann, Song Liu, Yu Kuai,
Li Nan, linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, linux-crypto, linux-btrfs, linux-arch,
linux-raid
In-Reply-To: <20260324064115.3217136-1-hch@lst.de>
Move the global dataptr array into test_recover() as all sites that fill
data or parity can use test_buffers directly, and this localized the
override for the failed slots to the recovery testing routine.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
lib/raid/raid6/tests/raid6_kunit.c | 19 +++++++------------
1 file changed, 7 insertions(+), 12 deletions(-)
diff --git a/lib/raid/raid6/tests/raid6_kunit.c b/lib/raid/raid6/tests/raid6_kunit.c
index 1793b952a595..ba6cfabc67a4 100644
--- a/lib/raid/raid6/tests/raid6_kunit.c
+++ b/lib/raid/raid6/tests/raid6_kunit.c
@@ -18,7 +18,6 @@ MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
#define NDISKS 16 /* Including P and Q */
static struct rnd_state rng;
-static void *dataptrs[NDISKS];
static void *test_buffers[NDISKS];
static void *test_recov_buffers[RAID6_KUNIT_MAX_FAILURES];
@@ -35,10 +34,8 @@ static void makedata(int start, int stop)
{
int i;
- for (i = start; i <= stop; i++) {
+ for (i = start; i <= stop; i++)
prandom_bytes_state(&rng, test_buffers[i], PAGE_SIZE);
- dataptrs[i] = test_buffers[i];
- }
}
static char member_type(int d)
@@ -56,11 +53,13 @@ static char member_type(int d)
static void test_recover(struct kunit *test, int faila, int failb)
{
const struct test_args *ta = test->param_value;
+ void *dataptrs[NDISKS];
int i;
for (i = 0; i < RAID6_KUNIT_MAX_FAILURES; i++)
memset(test_recov_buffers[i], 0xf0, PAGE_SIZE);
+ memcpy(dataptrs, test_buffers, sizeof(dataptrs));
dataptrs[faila] = test_recov_buffers[0];
dataptrs[failb] = test_recov_buffers[1];
@@ -73,7 +72,7 @@ static void test_recover(struct kunit *test, int faila, int failb)
* is equivalent to a RAID-5 failure (XOR, then recompute Q).
*/
if (faila != NDISKS - 2)
- goto skip;
+ return;
/* P+Q failure. Just rebuild the syndrome. */
ta->gen->gen_syndrome(NDISKS, PAGE_SIZE, dataptrs);
@@ -95,10 +94,6 @@ static void test_recover(struct kunit *test, int faila, int failb)
"failb miscompared: %3d[%c] (faila=%3d[%c])\n",
failb, member_type(failb),
faila, member_type(faila));
-
-skip:
- dataptrs[faila] = test_buffers[faila];
- dataptrs[failb] = test_buffers[failb];
}
static void raid6_test(struct kunit *test)
@@ -111,7 +106,7 @@ static void raid6_test(struct kunit *test)
memset(test_buffers[NDISKS - 1], 0xee, PAGE_SIZE);
/* Generate assumed good syndrome */
- ta->gen->gen_syndrome(NDISKS, PAGE_SIZE, (void **)&dataptrs);
+ ta->gen->gen_syndrome(NDISKS, PAGE_SIZE, test_buffers);
for (i = 0; i < NDISKS - 1; i++)
for (j = i + 1; j < NDISKS; j++)
@@ -124,10 +119,10 @@ static void raid6_test(struct kunit *test)
for (p2 = p1; p2 < NDISKS - 2; p2++) {
/* Simulate rmw run */
ta->gen->xor_syndrome(NDISKS, p1, p2, PAGE_SIZE,
- (void **)&dataptrs);
+ test_buffers);
makedata(p1, p2);
ta->gen->xor_syndrome(NDISKS, p1, p2, PAGE_SIZE,
- (void **)&dataptrs);
+ test_buffers);
for (i = 0; i < NDISKS - 1; i++)
for (j = i + 1; j < NDISKS; j++)
--
2.47.3
^ permalink raw reply related
* [PATCH 14/17] raid6_kunit: dynamically allocate data buffers using vmalloc
From: Christoph Hellwig @ 2026-03-24 6:40 UTC (permalink / raw)
To: Andrew Morton
Cc: Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, H. Peter Anvin, Herbert Xu, Dan Williams,
Chris Mason, David Sterba, Arnd Bergmann, Song Liu, Yu Kuai,
Li Nan, linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, linux-crypto, linux-btrfs, linux-arch,
linux-raid
In-Reply-To: <20260324064115.3217136-1-hch@lst.de>
Use vmalloc for the data buffers instead of using static .data allocations.
This provides for better out of bounds checking and avoids wasting kernel
memory after the test has run. vmalloc is used instead of kmalloc to
provide for better out of bounds access checking as in other kunit tests.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
lib/raid/raid6/tests/raid6_kunit.c | 75 +++++++++++++++++++++++-------
1 file changed, 59 insertions(+), 16 deletions(-)
diff --git a/lib/raid/raid6/tests/raid6_kunit.c b/lib/raid/raid6/tests/raid6_kunit.c
index 7095952ad8e3..1793b952a595 100644
--- a/lib/raid/raid6/tests/raid6_kunit.c
+++ b/lib/raid/raid6/tests/raid6_kunit.c
@@ -7,19 +7,20 @@
#include <kunit/test.h>
#include <linux/prandom.h>
+#include <linux/vmalloc.h>
#include "../algos.h"
MODULE_IMPORT_NS("EXPORTED_FOR_KUNIT_TESTING");
#define RAID6_KUNIT_SEED 42
+#define RAID6_KUNIT_MAX_FAILURES 2
#define NDISKS 16 /* Including P and Q */
static struct rnd_state rng;
static void *dataptrs[NDISKS];
-static char data[NDISKS][PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
-static char recovi[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
-static char recovj[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
+static void *test_buffers[NDISKS];
+static void *test_recov_buffers[RAID6_KUNIT_MAX_FAILURES];
struct test_args {
unsigned int recov_idx;
@@ -35,8 +36,8 @@ static void makedata(int start, int stop)
int i;
for (i = start; i <= stop; i++) {
- prandom_bytes_state(&rng, data[i], PAGE_SIZE);
- dataptrs[i] = data[i];
+ prandom_bytes_state(&rng, test_buffers[i], PAGE_SIZE);
+ dataptrs[i] = test_buffers[i];
}
}
@@ -55,12 +56,13 @@ static char member_type(int d)
static void test_recover(struct kunit *test, int faila, int failb)
{
const struct test_args *ta = test->param_value;
+ int i;
- memset(recovi, 0xf0, PAGE_SIZE);
- memset(recovj, 0xba, PAGE_SIZE);
+ for (i = 0; i < RAID6_KUNIT_MAX_FAILURES; i++)
+ memset(test_recov_buffers[i], 0xf0, PAGE_SIZE);
- dataptrs[faila] = recovi;
- dataptrs[failb] = recovj;
+ dataptrs[faila] = test_recov_buffers[0];
+ dataptrs[failb] = test_recov_buffers[1];
if (faila > failb)
swap(faila, failb);
@@ -83,18 +85,20 @@ static void test_recover(struct kunit *test, int faila, int failb)
ta->recov->data2(NDISKS, PAGE_SIZE, faila, failb, dataptrs);
}
- KUNIT_EXPECT_MEMEQ_MSG(test, data[faila], recovi, PAGE_SIZE,
+ KUNIT_EXPECT_MEMEQ_MSG(test, test_buffers[faila], test_recov_buffers[0],
+ PAGE_SIZE,
"faila miscompared: %3d[%c] (failb=%3d[%c])\n",
faila, member_type(faila),
failb, member_type(failb));
- KUNIT_EXPECT_MEMEQ_MSG(test, data[failb], recovj, PAGE_SIZE,
+ KUNIT_EXPECT_MEMEQ_MSG(test, test_buffers[failb], test_recov_buffers[1],
+ PAGE_SIZE,
"failb miscompared: %3d[%c] (faila=%3d[%c])\n",
failb, member_type(failb),
faila, member_type(faila));
skip:
- dataptrs[faila] = data[faila];
- dataptrs[failb] = data[failb];
+ dataptrs[faila] = test_buffers[faila];
+ dataptrs[failb] = test_buffers[failb];
}
static void raid6_test(struct kunit *test)
@@ -103,8 +107,8 @@ static void raid6_test(struct kunit *test)
int i, j, p1, p2;
/* Nuke syndromes */
- memset(data[NDISKS - 2], 0xee, PAGE_SIZE);
- memset(data[NDISKS - 1], 0xee, PAGE_SIZE);
+ memset(test_buffers[NDISKS - 2], 0xee, PAGE_SIZE);
+ memset(test_buffers[NDISKS - 1], 0xee, PAGE_SIZE);
/* Generate assumed good syndrome */
ta->gen->gen_syndrome(NDISKS, PAGE_SIZE, (void **)&dataptrs);
@@ -163,16 +167,55 @@ static struct kunit_case raid6_test_cases[] = {
static int raid6_suite_init(struct kunit_suite *suite)
{
+ int i;
+
prandom_seed_state(&rng, RAID6_KUNIT_SEED);
- makedata(0, NDISKS - 1);
memset(&args, 0, sizeof(args));
+
+ /*
+ * Allocate the test buffer using vmalloc() with a page-aligned length
+ * so that it is immediately followed by a guard page. This allows
+ * buffer overreads to be detected, even in assembly code.
+ */
+ for (i = 0; i < RAID6_KUNIT_MAX_FAILURES; i++) {
+ test_recov_buffers[i] = vmalloc(PAGE_SIZE);
+ if (!test_recov_buffers[i])
+ goto out_free_recov_buffers;
+ }
+ for (i = 0; i < NDISKS; i++) {
+ test_buffers[i] = vmalloc(PAGE_SIZE);
+ if (!test_buffers[i])
+ goto out_free_buffers;
+ }
+
+ makedata(0, NDISKS - 1);
+
return 0;
+
+out_free_buffers:
+ for (i = 0; i < NDISKS; i++)
+ vfree(test_buffers[i]);
+out_free_recov_buffers:
+ for (i = 0; i < RAID6_KUNIT_MAX_FAILURES; i++)
+ vfree(test_recov_buffers[i]);
+ return -ENOMEM;
+}
+
+static void raid6_suite_exit(struct kunit_suite *suite)
+{
+ int i;
+
+ for (i = 0; i < NDISKS; i++)
+ vfree(test_buffers[i]);
+ for (i = 0; i < RAID6_KUNIT_MAX_FAILURES; i++)
+ vfree(test_recov_buffers[i]);
}
static struct kunit_suite raid6_test_suite = {
.name = "raid6",
.test_cases = raid6_test_cases,
.suite_init = raid6_suite_init,
+ .suite_exit = raid6_suite_exit,
};
kunit_test_suite(raid6_test_suite);
--
2.47.3
^ permalink raw reply related
* [PATCH 13/17] raid6_kunit: use KUNIT_CASE_PARAM
From: Christoph Hellwig @ 2026-03-24 6:40 UTC (permalink / raw)
To: Andrew Morton
Cc: Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, H. Peter Anvin, Herbert Xu, Dan Williams,
Chris Mason, David Sterba, Arnd Bergmann, Song Liu, Yu Kuai,
Li Nan, linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, linux-crypto, linux-btrfs, linux-arch,
linux-raid
In-Reply-To: <20260324064115.3217136-1-hch@lst.de>
The raid6 test combines various generation and recovery algorithms. Use
KUNIT_CASE_PARAM and provide a generator that iterates over the possible
combinations instead of looping inside a single test instance.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
lib/raid/raid6/tests/raid6_kunit.c | 120 ++++++++++++++++-------------
1 file changed, 68 insertions(+), 52 deletions(-)
diff --git a/lib/raid/raid6/tests/raid6_kunit.c b/lib/raid/raid6/tests/raid6_kunit.c
index abc854cf4a98..7095952ad8e3 100644
--- a/lib/raid/raid6/tests/raid6_kunit.c
+++ b/lib/raid/raid6/tests/raid6_kunit.c
@@ -21,6 +21,15 @@ static char data[NDISKS][PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
static char recovi[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
static char recovj[PAGE_SIZE] __attribute__((aligned(PAGE_SIZE)));
+struct test_args {
+ unsigned int recov_idx;
+ const struct raid6_recov_calls *recov;
+ unsigned int gen_idx;
+ const struct raid6_calls *gen;
+};
+
+static struct test_args args;
+
static void makedata(int start, int stop)
{
int i;
@@ -43,9 +52,10 @@ static char member_type(int d)
}
}
-static void test_disks(struct kunit *test, const struct raid6_calls *calls,
- const struct raid6_recov_calls *ra, int faila, int failb)
+static void test_recover(struct kunit *test, int faila, int failb)
{
+ const struct test_args *ta = test->param_value;
+
memset(recovi, 0xf0, PAGE_SIZE);
memset(recovj, 0xba, PAGE_SIZE);
@@ -64,25 +74,23 @@ static void test_disks(struct kunit *test, const struct raid6_calls *calls,
goto skip;
/* P+Q failure. Just rebuild the syndrome. */
- calls->gen_syndrome(NDISKS, PAGE_SIZE, dataptrs);
+ ta->gen->gen_syndrome(NDISKS, PAGE_SIZE, dataptrs);
} else if (failb == NDISKS - 2) {
/* data+P failure. */
- ra->datap(NDISKS, PAGE_SIZE, faila, dataptrs);
+ ta->recov->datap(NDISKS, PAGE_SIZE, faila, dataptrs);
} else {
/* data+data failure. */
- ra->data2(NDISKS, PAGE_SIZE, faila, failb, dataptrs);
+ ta->recov->data2(NDISKS, PAGE_SIZE, faila, failb, dataptrs);
}
KUNIT_EXPECT_MEMEQ_MSG(test, data[faila], recovi, PAGE_SIZE,
- "algo=%-8s/%-8s faila miscompared: %3d[%c] (failb=%3d[%c])\n",
- calls->name, ra->name,
- faila, member_type(faila),
- failb, member_type(failb));
+ "faila miscompared: %3d[%c] (failb=%3d[%c])\n",
+ faila, member_type(faila),
+ failb, member_type(failb));
KUNIT_EXPECT_MEMEQ_MSG(test, data[failb], recovj, PAGE_SIZE,
- "algo=%-8s/%-8s failb miscompared: %3d[%c] (faila=%3d[%c])\n",
- calls->name, ra->name,
- failb, member_type(failb),
- faila, member_type(faila));
+ "failb miscompared: %3d[%c] (faila=%3d[%c])\n",
+ failb, member_type(failb),
+ faila, member_type(faila));
skip:
dataptrs[faila] = data[faila];
@@ -91,58 +99,65 @@ static void test_disks(struct kunit *test, const struct raid6_calls *calls,
static void raid6_test(struct kunit *test)
{
+ const struct test_args *ta = test->param_value;
int i, j, p1, p2;
- unsigned int r, g;
-
- for (r = 0; ; r++) {
- const struct raid6_recov_calls *ra = raid6_recov_algo_find(r);
-
- if (!ra)
- break;
-
- for (g = 0; ; g++) {
- const struct raid6_calls *calls = raid6_algo_find(g);
-
- if (!calls)
- break;
- /* Nuke syndromes */
- memset(data[NDISKS - 2], 0xee, PAGE_SIZE);
- memset(data[NDISKS - 1], 0xee, PAGE_SIZE);
+ /* Nuke syndromes */
+ memset(data[NDISKS - 2], 0xee, PAGE_SIZE);
+ memset(data[NDISKS - 1], 0xee, PAGE_SIZE);
- /* Generate assumed good syndrome */
- calls->gen_syndrome(NDISKS, PAGE_SIZE,
- (void **)&dataptrs);
+ /* Generate assumed good syndrome */
+ ta->gen->gen_syndrome(NDISKS, PAGE_SIZE, (void **)&dataptrs);
- for (i = 0; i < NDISKS-1; i++)
- for (j = i+1; j < NDISKS; j++)
- test_disks(test, calls, ra, i, j);
+ for (i = 0; i < NDISKS - 1; i++)
+ for (j = i + 1; j < NDISKS; j++)
+ test_recover(test, i, j);
- if (!calls->xor_syndrome)
- continue;
+ if (!ta->gen->xor_syndrome)
+ return;
- for (p1 = 0; p1 < NDISKS-2; p1++)
- for (p2 = p1; p2 < NDISKS-2; p2++) {
+ for (p1 = 0; p1 < NDISKS - 2; p1++) {
+ for (p2 = p1; p2 < NDISKS - 2; p2++) {
+ /* Simulate rmw run */
+ ta->gen->xor_syndrome(NDISKS, p1, p2, PAGE_SIZE,
+ (void **)&dataptrs);
+ makedata(p1, p2);
+ ta->gen->xor_syndrome(NDISKS, p1, p2, PAGE_SIZE,
+ (void **)&dataptrs);
- /* Simulate rmw run */
- calls->xor_syndrome(NDISKS, p1, p2, PAGE_SIZE,
- (void **)&dataptrs);
- makedata(p1, p2);
- calls->xor_syndrome(NDISKS, p1, p2, PAGE_SIZE,
- (void **)&dataptrs);
+ for (i = 0; i < NDISKS - 1; i++)
+ for (j = i + 1; j < NDISKS; j++)
+ test_recover(test, i, j);
+ }
+ }
+}
- for (i = 0; i < NDISKS-1; i++)
- for (j = i+1; j < NDISKS; j++)
- test_disks(test, calls,
- ra, i, j);
- }
+static const void *raid6_gen_params(struct kunit *test, const void *prev,
+ char *desc)
+{
+ if (!args.gen) {
+next_algo:
+ args.recov_idx = 0;
+ args.gen = raid6_algo_find(args.gen_idx);
+ if (!args.gen)
+ return NULL;
+ }
- }
+ if (args.recov)
+ args.recov_idx++;
+ args.recov = raid6_recov_algo_find(args.recov_idx);
+ if (!args.recov) {
+ args.gen_idx++;
+ goto next_algo;
}
+
+ snprintf(desc, KUNIT_PARAM_DESC_SIZE, "gen=%s recov=%s",
+ args.gen->name, args.recov->name);
+ return &args;
}
static struct kunit_case raid6_test_cases[] = {
- KUNIT_CASE(raid6_test),
+ KUNIT_CASE_PARAM(raid6_test, raid6_gen_params),
{},
};
@@ -150,6 +165,7 @@ static int raid6_suite_init(struct kunit_suite *suite)
{
prandom_seed_state(&rng, RAID6_KUNIT_SEED);
makedata(0, NDISKS - 1);
+ memset(&args, 0, sizeof(args));
return 0;
}
--
2.47.3
^ permalink raw reply related
* [PATCH 12/17] raid6: update top of file comments
From: Christoph Hellwig @ 2026-03-24 6:40 UTC (permalink / raw)
To: Andrew Morton
Cc: Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, H. Peter Anvin, Herbert Xu, Dan Williams,
Chris Mason, David Sterba, Arnd Bergmann, Song Liu, Yu Kuai,
Li Nan, linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, linux-crypto, linux-btrfs, linux-arch,
linux-raid
In-Reply-To: <20260324064115.3217136-1-hch@lst.de>
Drop the pointless mention of the file name, and use standard formatting
for the top of file comments.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
lib/raid/raid6/algos.c | 8 +-------
lib/raid/raid6/arm/neon.c | 2 +-
lib/raid/raid6/mktables.c | 12 +++---------
lib/raid/raid6/recov.c | 14 ++++----------
lib/raid/raid6/riscv/rvv.h | 2 --
lib/raid/raid6/x86/avx2.c | 13 ++++---------
lib/raid/raid6/x86/avx512.c | 18 ++++++------------
lib/raid/raid6/x86/mmx.c | 10 ++--------
lib/raid/raid6/x86/sse1.c | 18 ++++++------------
lib/raid/raid6/x86/sse2.c | 9 +--------
10 files changed, 28 insertions(+), 78 deletions(-)
diff --git a/lib/raid/raid6/algos.c b/lib/raid/raid6/algos.c
index 201443f6cac2..5fdf6d362f58 100644
--- a/lib/raid/raid6/algos.c
+++ b/lib/raid/raid6/algos.c
@@ -1,12 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-or-later
-/* -*- linux-c -*- ------------------------------------------------------- *
- *
- * Copyright 2002 H. Peter Anvin - All Rights Reserved
- *
- * ----------------------------------------------------------------------- */
-
/*
- * raid6/algos.c
+ * Copyright 2002 H. Peter Anvin - All Rights Reserved
*
* Algorithm list and algorithm selection for RAID-6
*/
diff --git a/lib/raid/raid6/arm/neon.c b/lib/raid/raid6/arm/neon.c
index 341c61af675e..af90869aaffc 100644
--- a/lib/raid/raid6/arm/neon.c
+++ b/lib/raid/raid6/arm/neon.c
@@ -1,6 +1,6 @@
// SPDX-License-Identifier: GPL-2.0-only
/*
- * linux/lib/raid6/neon.c - RAID6 syndrome calculation using ARM NEON intrinsics
+ * RAID6 syndrome calculation using ARM NEON intrinsics
*
* Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
*/
diff --git a/lib/raid/raid6/mktables.c b/lib/raid/raid6/mktables.c
index 97a17493bbd8..b6327b562fdb 100644
--- a/lib/raid/raid6/mktables.c
+++ b/lib/raid/raid6/mktables.c
@@ -1,15 +1,9 @@
// SPDX-License-Identifier: GPL-2.0-or-later
-/* -*- linux-c -*- ------------------------------------------------------- *
- *
- * Copyright 2002-2007 H. Peter Anvin - All Rights Reserved
- *
- * ----------------------------------------------------------------------- */
-
/*
- * mktables.c
+ * Copyright 2002-2007 H. Peter Anvin - All Rights Reserved
*
- * Make RAID-6 tables. This is a host user space program to be run at
- * compile time.
+ * Make RAID-6 tables. This is a host user space program to be run at compile
+ * time.
*/
#include <stdio.h>
diff --git a/lib/raid/raid6/recov.c b/lib/raid/raid6/recov.c
index 76eb2aef3667..3fa53bc3fde4 100644
--- a/lib/raid/raid6/recov.c
+++ b/lib/raid/raid6/recov.c
@@ -1,16 +1,10 @@
// SPDX-License-Identifier: GPL-2.0-or-later
-/* -*- linux-c -*- ------------------------------------------------------- *
- *
- * Copyright 2002 H. Peter Anvin - All Rights Reserved
- *
- * ----------------------------------------------------------------------- */
-
/*
- * raid6/recov.c
+ * Copyright 2002 H. Peter Anvin - All Rights Reserved
*
- * RAID-6 data recovery in dual failure mode. In single failure mode,
- * use the RAID-5 algorithm (or, in the case of Q failure, just reconstruct
- * the syndrome.)
+ * RAID-6 data recovery in dual failure mode. In single failure mode, use the
+ * RAID-5 algorithm (or, in the case of Q failure, just reconstruct the
+ * syndrome.)
*/
#include <linux/mm.h>
diff --git a/lib/raid/raid6/riscv/rvv.h b/lib/raid/raid6/riscv/rvv.h
index 3a7c2468b1ea..df0e3637cae8 100644
--- a/lib/raid/raid6/riscv/rvv.h
+++ b/lib/raid/raid6/riscv/rvv.h
@@ -2,8 +2,6 @@
/*
* Copyright 2024 Institute of Software, CAS.
*
- * raid6/rvv.h
- *
* Definitions for RISC-V RAID-6 code
*/
diff --git a/lib/raid/raid6/x86/avx2.c b/lib/raid/raid6/x86/avx2.c
index 7efd94e6a87a..7d829c669ea7 100644
--- a/lib/raid/raid6/x86/avx2.c
+++ b/lib/raid/raid6/x86/avx2.c
@@ -1,16 +1,11 @@
// SPDX-License-Identifier: GPL-2.0-or-later
-/* -*- linux-c -*- ------------------------------------------------------- *
- *
- * Copyright (C) 2012 Intel Corporation
- * Author: Yuanhan Liu <yuanhan.liu@linux.intel.com>
+/*
+ * Copyright (C) 2012 Intel Corporation
+ * Author: Yuanhan Liu <yuanhan.liu@linux.intel.com>
*
- * Based on sse2.c: Copyright 2002 H. Peter Anvin - All Rights Reserved
+ * Based on sse2.c: Copyright 2002 H. Peter Anvin - All Rights Reserved
*
- * ----------------------------------------------------------------------- */
-
-/*
* AVX2 implementation of RAID-6 syndrome functions
- *
*/
#include <asm/cpufeature.h>
diff --git a/lib/raid/raid6/x86/avx512.c b/lib/raid/raid6/x86/avx512.c
index 0772e798b742..e671eb5bde63 100644
--- a/lib/raid/raid6/x86/avx512.c
+++ b/lib/raid/raid6/x86/avx512.c
@@ -1,20 +1,14 @@
// SPDX-License-Identifier: GPL-2.0-or-later
-/* -*- linux-c -*- --------------------------------------------------------
- *
- * Copyright (C) 2016 Intel Corporation
+/*
+ * Copyright (C) 2016 Intel Corporation
*
- * Author: Gayatri Kammela <gayatri.kammela@intel.com>
- * Author: Megha Dey <megha.dey@linux.intel.com>
+ * Author: Gayatri Kammela <gayatri.kammela@intel.com>
+ * Author: Megha Dey <megha.dey@linux.intel.com>
*
- * Based on avx2.c: Copyright 2012 Yuanhan Liu All Rights Reserved
- * Based on sse2.c: Copyright 2002 H. Peter Anvin - All Rights Reserved
+ * Based on avx2.c: Copyright 2012 Yuanhan Liu All Rights Reserved
+ * Based on sse2.c: Copyright 2002 H. Peter Anvin - All Rights Reserved
*
- * -----------------------------------------------------------------------
- */
-
-/*
* AVX512 implementation of RAID-6 syndrome functions
- *
*/
#include <asm/cpufeature.h>
diff --git a/lib/raid/raid6/x86/mmx.c b/lib/raid/raid6/x86/mmx.c
index 3228c335965a..afa82536142d 100644
--- a/lib/raid/raid6/x86/mmx.c
+++ b/lib/raid/raid6/x86/mmx.c
@@ -1,14 +1,8 @@
// SPDX-License-Identifier: GPL-2.0-or-later
-/* -*- linux-c -*- ------------------------------------------------------- *
- *
- * Copyright 2002 H. Peter Anvin - All Rights Reserved
- *
- * ----------------------------------------------------------------------- */
-
/*
- * raid6/mmx.c
+ * Copyright 2002 H. Peter Anvin - All Rights Reserved
*
- * MMX implementation of RAID-6 syndrome functions
+ * MMX implementation of RAID-6 syndrome functions.
*/
#include <asm/cpufeature.h>
diff --git a/lib/raid/raid6/x86/sse1.c b/lib/raid/raid6/x86/sse1.c
index 6ebdcf824e00..f4b260df522a 100644
--- a/lib/raid/raid6/x86/sse1.c
+++ b/lib/raid/raid6/x86/sse1.c
@@ -1,19 +1,13 @@
// SPDX-License-Identifier: GPL-2.0-or-later
-/* -*- linux-c -*- ------------------------------------------------------- *
- *
- * Copyright 2002 H. Peter Anvin - All Rights Reserved
- *
- * ----------------------------------------------------------------------- */
-
/*
- * raid6/sse1.c
+ * Copyright 2002 H. Peter Anvin - All Rights Reserved
*
- * SSE-1/MMXEXT implementation of RAID-6 syndrome functions
+ * SSE-1/MMXEXT implementation of RAID-6 syndrome functions.
*
- * This is really an MMX implementation, but it requires SSE-1 or
- * AMD MMXEXT for prefetch support and a few other features. The
- * support for nontemporal memory accesses is enough to make this
- * worthwhile as a separate implementation.
+ * This is really an MMX implementation, but it requires SSE-1 or AMD MMXEXT for
+ * prefetch support and a few other features. The support for nontemporal
+ * memory accesses is enough to make this worthwhile as a separate
+ * implementation.
*/
#include <asm/cpufeature.h>
diff --git a/lib/raid/raid6/x86/sse2.c b/lib/raid/raid6/x86/sse2.c
index 7049c8512f35..43b09ce58270 100644
--- a/lib/raid/raid6/x86/sse2.c
+++ b/lib/raid/raid6/x86/sse2.c
@@ -1,15 +1,8 @@
// SPDX-License-Identifier: GPL-2.0-or-later
-/* -*- linux-c -*- ------------------------------------------------------- *
- *
- * Copyright 2002 H. Peter Anvin - All Rights Reserved
- *
- * ----------------------------------------------------------------------- */
-
/*
- * raid6/sse2.c
+ * Copyright 2002 H. Peter Anvin - All Rights Reserved
*
* SSE-2 implementation of RAID-6 syndrome functions
- *
*/
#include <asm/cpufeature.h>
--
2.47.3
^ permalink raw reply related
* [PATCH 11/17] raid6: use static_call for raid6_recov_2data and raid6_recov_datap
From: Christoph Hellwig @ 2026-03-24 6:40 UTC (permalink / raw)
To: Andrew Morton
Cc: Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, H. Peter Anvin, Herbert Xu, Dan Williams,
Chris Mason, David Sterba, Arnd Bergmann, Song Liu, Yu Kuai,
Li Nan, linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, linux-crypto, linux-btrfs, linux-arch,
linux-raid
In-Reply-To: <20260324064115.3217136-1-hch@lst.de>
Avoid expensive indirect calls for the recovery routines as well.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
lib/raid/raid6/algos.c | 8 ++++++--
1 file changed, 6 insertions(+), 2 deletions(-)
diff --git a/lib/raid/raid6/algos.c b/lib/raid/raid6/algos.c
index b81c7594f6c4..201443f6cac2 100644
--- a/lib/raid/raid6/algos.c
+++ b/lib/raid/raid6/algos.c
@@ -26,6 +26,8 @@ static const struct raid6_recov_calls *raid6_recov_algo;
/* Selected algorithm */
DEFINE_STATIC_CALL_NULL(raid6_gen_syndrome_impl, *raid6_intx1.gen_syndrome);
DEFINE_STATIC_CALL_NULL(raid6_xor_syndrome_impl, *raid6_intx1.xor_syndrome);
+DEFINE_STATIC_CALL_NULL(raid6_recov_2data_impl, *raid6_recov_intx1.data2);
+DEFINE_STATIC_CALL_NULL(raid6_recov_datap_impl, *raid6_recov_intx1.datap);
/**
* raid6_gen_syndrome - generate RAID6 P/Q parity
@@ -124,7 +126,7 @@ void raid6_recov_2data(int disks, size_t bytes, int faila, int failb,
WARN_ON_ONCE(bytes > PAGE_SIZE);
WARN_ON_ONCE(failb <= faila);
- raid6_recov_algo->data2(disks, bytes, faila, failb, ptrs);
+ static_call(raid6_recov_2data_impl)(disks, bytes, faila, failb, ptrs);
}
EXPORT_SYMBOL_GPL(raid6_recov_2data);
@@ -149,7 +151,7 @@ void raid6_recov_datap(int disks, size_t bytes, int faila, void **ptrs)
WARN_ON_ONCE(bytes & 511);
WARN_ON_ONCE(bytes > PAGE_SIZE);
- raid6_recov_algo->datap(disks, bytes, faila, ptrs);
+ static_call(raid6_recov_datap_impl)(disks, bytes, faila, ptrs);
}
EXPORT_SYMBOL_GPL(raid6_recov_datap);
@@ -322,6 +324,8 @@ static int __init raid6_init(void)
*/
if (!raid6_recov_algo)
raid6_recov_algo = &raid6_recov_intx1;
+ static_call_update(raid6_recov_2data_impl, raid6_recov_algo->data2);
+ static_call_update(raid6_recov_datap_impl, raid6_recov_algo->datap);
pr_info("raid6: using %s recovery algorithm\n", raid6_recov_algo->name);
#ifdef MODULE
--
2.47.3
^ permalink raw reply related
* [PATCH 10/17] raid6: use static_call for gen_syndrom and xor_syndrom
From: Christoph Hellwig @ 2026-03-24 6:40 UTC (permalink / raw)
To: Andrew Morton
Cc: Catalin Marinas, Will Deacon, Ard Biesheuvel, Huacai Chen,
WANG Xuerui, Madhavan Srinivasan, Michael Ellerman,
Nicholas Piggin, Christophe Leroy (CS GROUP), Paul Walmsley,
Palmer Dabbelt, Albert Ou, Alexandre Ghiti, Heiko Carstens,
Vasily Gorbik, Alexander Gordeev, Christian Borntraeger,
Sven Schnelle, Thomas Gleixner, Ingo Molnar, Borislav Petkov,
Dave Hansen, x86, H. Peter Anvin, Herbert Xu, Dan Williams,
Chris Mason, David Sterba, Arnd Bergmann, Song Liu, Yu Kuai,
Li Nan, linux-kernel, linux-arm-kernel, loongarch, linuxppc-dev,
linux-riscv, linux-s390, linux-crypto, linux-btrfs, linux-arch,
linux-raid
In-Reply-To: <20260324064115.3217136-1-hch@lst.de>
Avoid indirect calls for P/Q parity generation.
Signed-off-by: Christoph Hellwig <hch@lst.de>
---
lib/raid/raid6/algos.c | 18 ++++++++++++------
1 file changed, 12 insertions(+), 6 deletions(-)
diff --git a/lib/raid/raid6/algos.c b/lib/raid/raid6/algos.c
index f9e8a8752e2d..b81c7594f6c4 100644
--- a/lib/raid/raid6/algos.c
+++ b/lib/raid/raid6/algos.c
@@ -14,6 +14,7 @@
#include <linux/module.h>
#include <linux/gfp.h>
#include <linux/raid/pq.h>
+#include <linux/static_call.h>
#include <kunit/visibility.h>
#include "algos.h"
@@ -23,7 +24,8 @@ static unsigned int raid6_nr_algos;
static const struct raid6_recov_calls *raid6_recov_algo;
/* Selected algorithm */
-static struct raid6_calls raid6_call;
+DEFINE_STATIC_CALL_NULL(raid6_gen_syndrome_impl, *raid6_intx1.gen_syndrome);
+DEFINE_STATIC_CALL_NULL(raid6_xor_syndrome_impl, *raid6_intx1.xor_syndrome);
/**
* raid6_gen_syndrome - generate RAID6 P/Q parity
@@ -47,7 +49,7 @@ void raid6_gen_syndrome(int disks, size_t bytes, void **ptrs)
lockdep_assert_preemption_enabled();
WARN_ON_ONCE(bytes & 511);
- raid6_call.gen_syndrome(disks, bytes, ptrs);
+ static_call(raid6_gen_syndrome_impl)(disks, bytes, ptrs);
}
EXPORT_SYMBOL_GPL(raid6_gen_syndrome);
@@ -83,7 +85,7 @@ void raid6_xor_syndrome(int disks, int start, int stop, size_t bytes,
WARN_ON_ONCE(bytes & 511);
WARN_ON_ONCE(stop < start);
- raid6_call.xor_syndrome(disks, start, stop, bytes, ptrs);
+ static_call(raid6_xor_syndrome_impl)(disks, start, stop, bytes, ptrs);
}
EXPORT_SYMBOL_GPL(raid6_xor_syndrome);
@@ -94,7 +96,7 @@ EXPORT_SYMBOL_GPL(raid6_xor_syndrome);
*/
bool raid6_can_xor_syndrome(void)
{
- return !!raid6_call.xor_syndrome;
+ return !!static_call_query(raid6_xor_syndrome_impl);
}
EXPORT_SYMBOL_GPL(raid6_can_xor_syndrome);
@@ -193,7 +195,8 @@ static int raid6_choose_gen(void *(*const dptrs)[RAID6_TEST_DISKS],
return -EINVAL;
}
- raid6_call = *best;
+ static_call_update(raid6_gen_syndrome_impl, best->gen_syndrome);
+ static_call_update(raid6_xor_syndrome_impl, best->xor_syndrome);
pr_info("raid6: using algorithm %s gen() %ld MB/s\n",
best->name,
@@ -238,7 +241,10 @@ static int __init raid6_select_algo(void)
if (!IS_ENABLED(CONFIG_RAID6_PQ_BENCHMARK) || raid6_nr_algos == 1) {
pr_info("raid6: skipped pq benchmark and selected %s\n",
raid6_algos[0]->name);
- raid6_call = *raid6_algos[0];
+ static_call_update(raid6_gen_syndrome_impl,
+ raid6_algos[0]->gen_syndrome);
+ static_call_update(raid6_xor_syndrome_impl,
+ raid6_algos[0]->xor_syndrome);
return 0;
}
--
2.47.3
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox