* Re: [PATCH V4 0/3] md/nvme: Enable PCI P2PDMA support for RAID0 and NVMe Multipath
From: Jens Axboe @ 2026-05-26 21:51 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: song@kernel.org, yukuai@fnnas.com, Christoph Hellwig,
linan122@huawei.com, kbusch@kernel.org, sagi@grimberg.me,
linux-block@vger.kernel.org, linux-raid@vger.kernel.org,
linux-nvme@lists.infradead.org, Kiran Modukuri
In-Reply-To: <053b99b2-c994-42ff-af63-6e63ab468557@nvidia.com>
On 5/26/26 11:09 AM, Chaitanya Kulkarni wrote:
> Jens,
>
> On 5/19/26 17:11, Chaitanya Kulkarni wrote:
>> Jens,
>>
>>
>> On 5/14/26 9:35 PM, Christoph Hellwig wrote:
>>> Still looks good to me as per the reviews.
>>>
>> If there no objection, can we merge this ?
>>
>> -Chaitanya
>>
>>
> There is outstanding work I want to send out based on this one.
Out standing, outstanding, or both? :-)
> May I please request you to merge this patch series ?
Was waiting on the md parts to get reviewed, by I missed that Xiao Ni
already did. I'll queue it up.
--
Jens Axboe
^ permalink raw reply
* Re: [PATCH v2 2/2] lib/raid6: use kvmalloc() in raid6_select_algo()
From: Andrew Morton @ 2026-05-26 17:34 UTC (permalink / raw)
To: Mike Rapoport (Microsoft)
Cc: Song Liu, Yu Kuai, Li Nan, Xiao Ni, David Laight, linux-kernel,
linux-mm, linux-raid, Christoph Hellwig
In-Reply-To: <20260526-lib-v2-2-ca3f0fc24b14@kernel.org>
On Tue, 26 May 2026 15:50:39 +0300 "Mike Rapoport (Microsoft)" <rppt@kernel.org> wrote:
> raid6_select_algo() allocates an order 3 (8 pages) buffer that is used
> as a scratch area for selection of the best algorithm.
>
> This buffer does not need to be physically contiguous and can be
> allocated with kvmalloc().
>
> Replace __get_free_pages() call with kvmalloc().
This one needed some massaging due to hch's "cleanup the RAID6 P/Q
library" in mm-nonmm-unstable
(https://lore.kernel.org/20260518051804.462141-1-hch@lst.de)
From: "Mike Rapoport (Microsoft)" <rppt@kernel.org>
Subject: lib/raid6: use kvmalloc() in raid6_select_algo()
Date: Tue, 26 May 2026 15:50:39 +0300
raid6_select_algo() allocates an order 3 (8 pages) buffer that is used
as a scratch area for selection of the best algorithm.
This buffer does not need to be physically contiguous and can be
allocated with kvmalloc().
Replace __get_free_pages() call with kvmalloc().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Link: https://lore.kernel.org/20260526-lib-v2-2-ca3f0fc24b14@kernel.org
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
lib/raid/raid6/algos.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
--- a/lib/raid/raid6/algos.c~lib-raid6-use-kvmalloc-in-raid6_select_algo
+++ a/lib/raid/raid6/algos.c
@@ -7,6 +7,7 @@
#include <linux/module.h>
#include <linux/gfp.h>
+#include <linux/slab.h>
#include <linux/raid/pq.h>
#include <linux/static_call.h>
#include <kunit/visibility.h>
@@ -153,7 +154,6 @@ EXPORT_SYMBOL_GPL(raid6_recov_datap);
#define RAID6_TIME_JIFFIES_LG2 4
#define RAID6_TEST_DISKS 8
-#define RAID6_TEST_DISKS_ORDER 3
static int raid6_choose_gen(void *(*const dptrs)[RAID6_TEST_DISKS],
const int disks)
@@ -247,7 +247,7 @@ static int __init raid6_select_algo(void
}
/* prepare the buffer and fill it circularly with gfmul table */
- disk_ptr = (char *)__get_free_pages(GFP_KERNEL, RAID6_TEST_DISKS_ORDER);
+ disk_ptr = kvmalloc(PAGE_SIZE * RAID6_TEST_DISKS, GFP_KERNEL);
if (!disk_ptr) {
pr_err("raid6: Yikes! No memory available.\n");
return -ENOMEM;
@@ -269,7 +269,7 @@ static int __init raid6_select_algo(void
/* select raid gen_syndrome function */
error = raid6_choose_gen(&dptrs, disks);
- free_pages((unsigned long)disk_ptr, RAID6_TEST_DISKS_ORDER);
+ kvfree(disk_ptr);
return error;
}
_
^ permalink raw reply
* Re: [PATCH V4 0/3] md/nvme: Enable PCI P2PDMA support for RAID0 and NVMe Multipath
From: Chaitanya Kulkarni @ 2026-05-26 17:09 UTC (permalink / raw)
To: axboe@kernel.dk
Cc: song@kernel.org, yukuai@fnnas.com, Christoph Hellwig,
linan122@huawei.com, kbusch@kernel.org, sagi@grimberg.me,
linux-block@vger.kernel.org, linux-raid@vger.kernel.org,
linux-nvme@lists.infradead.org, Kiran Modukuri
In-Reply-To: <4ed83782-04cf-45b5-93a0-05a08e61b82e@nvidia.com>
Jens,
On 5/19/26 17:11, Chaitanya Kulkarni wrote:
> Jens,
>
>
> On 5/14/26 9:35 PM, Christoph Hellwig wrote:
>> Still looks good to me as per the reviews.
>>
> If there no objection, can we merge this ?
>
> -Chaitanya
>
>
There is outstanding work I want to send out based on this one.
May I please request you to merge this patch series ?
-ck
^ permalink raw reply
* Re: [PATCH 2/2] lib/raid6: use kmalloc() in raid6_select_algo()
From: Mike Rapoport @ 2026-05-26 14:38 UTC (permalink / raw)
To: David Laight
Cc: Song Liu, Yu Kuai, Li Nan, Xiao Ni, linux-kernel, linux-mm,
linux-raid
In-Reply-To: <20260520140657.2b5b5f3b@pumpkin>
On Wed, May 20, 2026 at 02:06:57PM +0100, David Laight wrote:
> On Wed, 20 May 2026 11:17:52 +0300
> "Mike Rapoport (Microsoft)" <rppt@kernel.org> wrote:
>
> > raid6_select_algo() allocates an order 3 (8 pages) buffer that is used
> > as a scratch area for selection of the best algorithm.
>
> Should this code really be using a 4k buffer rather than a PAGE_SIZE one?
Why?
> -- David
>
> >
> > For such large allocations kmalloc() would fall back to alloc_pages() but
> > still kmalloc() is a better API as it does not require unnecessary
> > castings and may provide more debugging possibilities.
> >
> > Replace __get_free_pages() call with kmalloc().
> >
> > Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
> > Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
> > ---
> > lib/raid6/algos.c | 6 +++---
> > 1 file changed, 3 insertions(+), 3 deletions(-)
> >
> > diff --git a/lib/raid6/algos.c b/lib/raid6/algos.c
> > index 799e0e5eac26..89e627c62e30 100644
> > --- a/lib/raid6/algos.c
> > +++ b/lib/raid6/algos.c
> > @@ -12,6 +12,7 @@
> > */
> >
> > #include <linux/raid/pq.h>
> > +#include <linux/slab.h>
> > #ifndef __KERNEL__
> > #include <sys/mman.h>
> > #include <stdio.h>
> > @@ -129,7 +130,6 @@ const struct raid6_recov_calls *const raid6_recov_algos[] = {
> > #endif
> >
> > #define RAID6_TEST_DISKS 8
> > -#define RAID6_TEST_DISKS_ORDER 3
> >
> > static inline const struct raid6_recov_calls *raid6_choose_recov(void)
> > {
> > @@ -250,7 +250,7 @@ int __init raid6_select_algo(void)
> > int i, cycle;
> >
> > /* prepare the buffer and fill it circularly with gfmul table */
> > - disk_ptr = (char *)__get_free_pages(GFP_KERNEL, RAID6_TEST_DISKS_ORDER);
> > + disk_ptr = kmalloc(PAGE_SIZE * RAID6_TEST_DISKS, GFP_KERNEL);
> > if (!disk_ptr) {
> > pr_err("raid6: Yikes! No memory available.\n");
> > return -ENOMEM;
> > @@ -275,7 +275,7 @@ int __init raid6_select_algo(void)
> > /* select raid recover functions */
> > rec_best = raid6_choose_recov();
> >
> > - free_pages((unsigned long)disk_ptr, RAID6_TEST_DISKS_ORDER);
> > + kfree(disk_ptr);
> >
> > return gen_best && rec_best ? 0 : -EINVAL;
> > }
> >
>
>
--
Sincerely yours,
Mike.
^ permalink raw reply
* [PATCH v2 2/2] lib/raid6: use kvmalloc() in raid6_select_algo()
From: Mike Rapoport (Microsoft) @ 2026-05-26 12:50 UTC (permalink / raw)
To: Andrew Morton, Song Liu, Yu Kuai, Li Nan, Xiao Ni
Cc: David Laight, Mike Rapoport, linux-kernel, linux-mm, linux-raid
In-Reply-To: <20260526-lib-v2-0-ca3f0fc24b14@kernel.org>
raid6_select_algo() allocates an order 3 (8 pages) buffer that is used
as a scratch area for selection of the best algorithm.
This buffer does not need to be physically contiguous and can be
allocated with kvmalloc().
Replace __get_free_pages() call with kvmalloc().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
---
lib/raid6/algos.c | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/lib/raid6/algos.c b/lib/raid6/algos.c
index 799e0e5eac26..1f565c3b63ce 100644
--- a/lib/raid6/algos.c
+++ b/lib/raid6/algos.c
@@ -12,6 +12,7 @@
*/
#include <linux/raid/pq.h>
+#include <linux/slab.h>
#ifndef __KERNEL__
#include <sys/mman.h>
#include <stdio.h>
@@ -129,7 +130,6 @@ const struct raid6_recov_calls *const raid6_recov_algos[] = {
#endif
#define RAID6_TEST_DISKS 8
-#define RAID6_TEST_DISKS_ORDER 3
static inline const struct raid6_recov_calls *raid6_choose_recov(void)
{
@@ -250,7 +250,7 @@ int __init raid6_select_algo(void)
int i, cycle;
/* prepare the buffer and fill it circularly with gfmul table */
- disk_ptr = (char *)__get_free_pages(GFP_KERNEL, RAID6_TEST_DISKS_ORDER);
+ disk_ptr = kvmalloc(PAGE_SIZE * RAID6_TEST_DISKS, GFP_KERNEL);
if (!disk_ptr) {
pr_err("raid6: Yikes! No memory available.\n");
return -ENOMEM;
@@ -275,7 +275,7 @@ int __init raid6_select_algo(void)
/* select raid recover functions */
rec_best = raid6_choose_recov();
- free_pages((unsigned long)disk_ptr, RAID6_TEST_DISKS_ORDER);
+ kvfree(disk_ptr);
return gen_best && rec_best ? 0 : -EINVAL;
}
--
2.53.0
^ permalink raw reply related
* [PATCH v2 1/2] lib/raid: use kvmalloc() in calibrate_xor_blocks()
From: Mike Rapoport (Microsoft) @ 2026-05-26 12:50 UTC (permalink / raw)
To: Andrew Morton, Song Liu, Yu Kuai, Li Nan, Xiao Ni
Cc: David Laight, Mike Rapoport, linux-kernel, linux-mm, linux-raid
In-Reply-To: <20260526-lib-v2-0-ca3f0fc24b14@kernel.org>
The xor benchmark allocates an order 2 (4 pages) scratch buffer that is
used purely as a CPU-only XOR working area.
This buffer does not need to be physically contiguous and can be
allocated with kvmalloc().
Replace __get_free_pages() call with kvmalloc().
Link: https://lore.kernel.org/all/635405e4-9423-4a25-a6e7-e03c8ea0bcbe@redhat.com
Signed-off-by: Mike Rapoport (Microsoft) <rppt@kernel.org>
---
lib/raid/xor/xor-core.c | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/lib/raid/xor/xor-core.c b/lib/raid/xor/xor-core.c
index bd4e6e434418..9f90620617cd 100644
--- a/lib/raid/xor/xor-core.c
+++ b/lib/raid/xor/xor-core.c
@@ -8,6 +8,7 @@
#include <linux/module.h>
#include <linux/gfp.h>
+#include <linux/slab.h>
#include <linux/raid/xor.h>
#include <linux/jiffies.h>
#include <linux/preempt.h>
@@ -114,7 +115,7 @@ static int __init calibrate_xor_blocks(void)
if (forced_template)
return 0;
- b1 = (void *) __get_free_pages(GFP_KERNEL, 2);
+ b1 = kvmalloc(PAGE_SIZE * 4, GFP_KERNEL);
if (!b1) {
pr_warn("xor: Yikes! No memory available.\n");
return -ENOMEM;
@@ -132,7 +133,7 @@ static int __init calibrate_xor_blocks(void)
pr_info("xor: using function: %s (%d MB/sec)\n",
fastest->name, fastest->speed);
- free_pages((unsigned long)b1, 2);
+ kvfree(b1);
return 0;
}
--
2.53.0
^ permalink raw reply related
* [PATCH v2 0/2] lib/raid: replace __get_free_pages() call with kvmalloc()
From: Mike Rapoport (Microsoft) @ 2026-05-26 12:50 UTC (permalink / raw)
To: Andrew Morton, Song Liu, Yu Kuai, Li Nan, Xiao Ni
Cc: David Laight, Mike Rapoport, linux-kernel, linux-mm, linux-raid
This is a (tiny) part of larger work of replacing page allocator calls
with k*malloc:
Also in git:
https://git.kernel.org/pub/scm/linux/kernel/git/rppt/linux.git gfp-to-kmalloc/lib
---
v2 changes:
* replace kmalloc() with kvmalloc()
v1: https://patch.msgid.link/20260520-lib-v1-0-cb3045bef2d8@kernel.org
---
Mike Rapoport (Microsoft) (2):
lib/raid: use kvmalloc() in calibrate_xor_blocks()
lib/raid6: use kvmalloc() in raid6_select_algo()
lib/raid/xor/xor-core.c | 5 +++--
lib/raid6/algos.c | 6 +++---
2 files changed, 6 insertions(+), 5 deletions(-)
---
base-commit: 5d6919055dec134de3c40167a490f33c74c12581
change-id: 20260520-lib-8afb92134307
Best regards,
--
Sincerely yours,
Mike.
^ permalink raw reply
* Re: [PATCH 3/3] btrfs: disguise single-data-RAID56 as RAID1/RAID1C3
From: Christoph Hellwig @ 2026-05-26 6:46 UTC (permalink / raw)
To: Qu Wenruo; +Cc: dsterba, Christoph Hellwig, linux-btrfs, linux-raid
In-Reply-To: <e5eafd64-2704-42de-ad7b-e4f170c451e4@suse.com>
On Mon, May 25, 2026 at 08:09:54PM +0930, Qu Wenruo wrote:
>> It's a lazy hack at best and fix on absolutely wrong layer. The library
>> should provide the support for the edge case. But we disagree on that.
>
> I strongly disagree.
>
> Firstly on the layer to fix, let me be clear, if something writes like
> RAID1, reads like RAID1, repairs like RAID1, then it's RAID1.
*nod*
> With that mindset, changing chunk type at read/add time is exactly the
> correct layer to fix.
It also is amazingly simple!
^ permalink raw reply
* Re: [PATCH 3/3] btrfs: disguise single-data-RAID56 as RAID1/RAID1C3
From: Christoph Hellwig @ 2026-05-26 6:45 UTC (permalink / raw)
To: David Sterba; +Cc: Christoph Hellwig, Qu Wenruo, linux-btrfs, linux-raid
In-Reply-To: <20260525101707.GV12792@suse.cz>
On Mon, May 25, 2026 at 12:17:07PM +0200, David Sterba wrote:
> > It absolutely is. Adding fast path workarounds for this is completely
> > stupid when it can be trivially handled on the mount side.
>
> It's a lazy hack at best and fix on absolutely wrong layer. The library
> should provide the support for the edge case. But we disagree on that.
No, the library should not provide hacks for something it never intended
to support, and which is a bad idea. It's also something the
implementation that the library was factored out from for btrfs use
never supported.
So adding the workaround to btrfs, which caused the problem is a
requirement. Doing it to only create mount time overhead instead of
burdening the I/O path is a very smart way to solve the problem,
while adding hacks to the I/O fastpath is everything but smart.
But in the end this is for the btrfs maintainers to decide.
^ permalink raw reply
* [PATCH 30/34] md-bitmap: Convert read_file_page and write_file_page to bh_submit()
From: Matthew Wilcox (Oracle) @ 2026-05-25 17:19 UTC (permalink / raw)
To: Jan Kara
Cc: Matthew Wilcox (Oracle), Christian Brauner, Christoph Hellwig,
linux-fsdevel, linux-raid
In-Reply-To: <20260525171931.4144395-1-willy@infradead.org>
Avoid an extra indirect function call by using bh_submit() instead of
submit_bh().
Signed-off-by: Matthew Wilcox (Oracle) <willy@infradead.org>
Cc: linux-raid@vger.kernel.org
---
drivers/md/md-bitmap.c | 27 ++++++++++++++-------------
1 file changed, 14 insertions(+), 13 deletions(-)
diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c
index 028b9ca8ce52..c0a4559186e2 100644
--- a/drivers/md/md-bitmap.c
+++ b/drivers/md/md-bitmap.c
@@ -502,6 +502,18 @@ static void write_sb_page(struct bitmap *bitmap, unsigned long pg_index,
static void md_bitmap_file_kick(struct bitmap *bitmap);
#ifdef CONFIG_MD_BITMAP_FILE
+static void end_bitmap_write(struct bio *bio)
+{
+ bool uptodate = bio->bi_status == BLK_STS_OK;
+ struct buffer_head *bh = bio_endio_bh(bio);
+ struct bitmap *bitmap = bh->b_private;
+
+ if (!uptodate)
+ set_bit(BITMAP_WRITE_ERROR, &bitmap->flags);
+ if (atomic_dec_and_test(&bitmap->pending_writes))
+ wake_up(&bitmap->write_wait);
+}
+
static void write_file_page(struct bitmap *bitmap, struct page *page, int wait)
{
struct buffer_head *bh = page_buffers(page);
@@ -510,7 +522,7 @@ static void write_file_page(struct bitmap *bitmap, struct page *page, int wait)
atomic_inc(&bitmap->pending_writes);
set_buffer_locked(bh);
set_buffer_mapped(bh);
- submit_bh(REQ_OP_WRITE | REQ_SYNC, bh);
+ bh_submit(bh, REQ_OP_WRITE | REQ_SYNC, end_bitmap_write);
bh = bh->b_this_page;
}
@@ -519,16 +531,6 @@ static void write_file_page(struct bitmap *bitmap, struct page *page, int wait)
atomic_read(&bitmap->pending_writes) == 0);
}
-static void end_bitmap_write(struct buffer_head *bh, int uptodate)
-{
- struct bitmap *bitmap = bh->b_private;
-
- if (!uptodate)
- set_bit(BITMAP_WRITE_ERROR, &bitmap->flags);
- if (atomic_dec_and_test(&bitmap->pending_writes))
- wake_up(&bitmap->write_wait);
-}
-
static void free_buffers(struct page *page)
{
struct buffer_head *bh;
@@ -592,12 +594,11 @@ static int read_file_page(struct file *file, unsigned long index,
else
count -= blocksize;
- bh->b_end_io = end_bitmap_write;
bh->b_private = bitmap;
atomic_inc(&bitmap->pending_writes);
set_buffer_locked(bh);
set_buffer_mapped(bh);
- submit_bh(REQ_OP_READ, bh);
+ bh_submit(bh, REQ_OP_READ, end_bitmap_write);
}
blk_cur++;
bh = bh->b_this_page;
--
2.47.3
^ permalink raw reply related
* Re: --bitmap=lockless across server reboot
From: Anton Gavriliuk @ 2026-05-25 17:11 UTC (permalink / raw)
To: yukuai; +Cc: linux-raid
In-Reply-To: <CAAiJnjoh9rTHh4gRu2=7A4C7EFP4yfBED94qmeS_ATDyB0yevQ@mail.gmail.com>
I remembered how to reproduce.
The raid is gone after I performed precondition for performance tests
and reboot.
After reboot there are no issues in dmesg and messages.
[root@qdevice ~]# fio --name=raid5_final --rw=write --bs=2048k
--filename=/dev/md0 --direct=1 --numjobs=1 --iodepth=64
--group_reporting --ioengine=io_uring --time_based --runtime=1800
raid5_final: (g=0): rw=write, bs=(R) 2048KiB-2048KiB, (W)
2048KiB-2048KiB, (T) 2048KiB-2048KiB, ioengine=io_uring, iodepth=64
fio-3.42-29-gd569
Starting 1 process
Jobs: 1 (f=1): [W(1)][100.0%][w=8316MiB/s][w=4158 IOPS][eta 00m:00s]
raid5_final: (groupid=0, jobs=1): err= 0: pid=5070: Mon May 25 19:56:09 2026
write: IOPS=4088, BW=8176MiB/s (8574MB/s)(14.0TiB/1800013msec)
slat (usec): min=15, max=1287, avg=36.23, stdev=11.79
clat (usec): min=2072, max=43284, avg=15618.22, stdev=3312.23
lat (usec): min=2094, max=43313, avg=15654.45, stdev=3312.52
clat percentiles (usec):
| 1.00th=[ 8356], 5.00th=[10421], 10.00th=[11469], 20.00th=[12911],
| 30.00th=[13829], 40.00th=[14615], 50.00th=[15533], 60.00th=[16319],
| 70.00th=[17171], 80.00th=[18220], 90.00th=[19792], 95.00th=[21103],
| 99.00th=[23987], 99.50th=[25297], 99.90th=[28181], 99.95th=[29492],
| 99.99th=[32375]
bw ( MiB/s): min= 7736, max= 9132, per=100.00%, avg=8178.90,
stdev=166.52, samples=3599
iops : min= 3868, max= 4566, avg=4089.42, stdev=83.27, samples=3599
lat (msec) : 4=0.01%, 10=3.68%, 20=87.01%, 50=9.32%
cpu : usr=10.21%, sys=5.19%, ctx=5637748, majf=0, minf=754452
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
issued rwts: total=0,7358832,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0.00ns, window=0.00ns, percentile=100.00%, depth=64
Run status group 0 (all jobs):
WRITE: bw=8176MiB/s (8574MB/s), 8176MiB/s-8176MiB/s
(8574MB/s-8574MB/s), io=14.0TiB (15.4TB), run=1800013-1800013msec
Disk stats (read/write):
md0: ios=20/14716476, sectors=2560/30139342848, merge=0/0,
ticks=0/125528858, in_queue=125528858, util=100.00%,
aggrios=68/7358896, aggsectors=3046/7535444480, aggrmerge=0/0,
aggrticks=6/413853, aggrin_queue=413859, aggrutil=21.24%
nvme0n1: ios=48/7358896, sectors=1680/7535444480, merge=0/0,
ticks=6/408945, in_queue=408951, util=20.67%
nvme3n1: ios=67/7358896, sectors=3992/7535444480, merge=0/0,
ticks=4/433875, in_queue=433879, util=21.24%
nvme2n1: ios=118/7358896, sectors=2960/7535444480, merge=0/0,
ticks=7/405682, in_queue=405689, util=20.45%
nvme5n1: ios=75/7358896, sectors=5136/7535444480, merge=0/0,
ticks=9/407693, in_queue=407702, util=20.42%
nvme1n1: ios=51/7358896, sectors=2064/7535444480, merge=0/0,
ticks=7/409760, in_queue=409767, util=20.59%
nvme4n1: ios=54/7358896, sectors=2448/7535444480, merge=0/0,
ticks=5/417163, in_queue=417168, util=20.58%
[root@qdevice ~]#
[root@qdevice ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme6n1 259:1 0 447.1G 0 disk
├─nvme6n1p1 259:2 0 1G 0 part /boot/efi
├─nvme6n1p2 259:3 0 1G 0 part /boot
├─nvme6n1p3 259:4 0 220G 0 part /
├─nvme6n1p4 259:5 0 1G 0 part
└─nvme6n1p5 259:6 0 220G 0 part
nvme0n1 259:9 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme1n1 259:10 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme4n1 259:15 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme2n1 259:16 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme5n1 259:17 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme3n1 259:18 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
[root@qdevice ~]#
[root@qdevice ~]# reboot
login as: root
root@10.72.14.186's password:
Web console: https://qdevice:9090/ or https://10.72.14.186:9090/
Last login: Mon May 25 19:25:38 2026 from 10.72.12.178
[root@qdevice ~]# uptime
20:01:59 up 0 min, 1 user, load average: 0.87, 0.26, 0.09
[root@qdevice ~]#
[root@qdevice ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme5n1 259:1 0 447.1G 0 disk
├─nvme5n1p1 259:2 0 1G 0 part /boot/efi
├─nvme5n1p2 259:3 0 1G 0 part /boot
├─nvme5n1p3 259:4 0 220G 0 part /
├─nvme5n1p4 259:5 0 1G 0 part
└─nvme5n1p5 259:6 0 220G 0 part
nvme1n1 259:8 0 2.9T 0 disk
nvme3n1 259:10 0 2.9T 0 disk
nvme4n1 259:13 0 2.9T 0 disk
nvme2n1 259:14 0 2.9T 0 disk
nvme0n1 259:17 0 2.9T 0 disk
nvme6n1 259:18 0 2.9T 0 disk
[root@qdevice ~]#
[root@qdevice ~]# uptime
20:02:40 up 1 min, 1 user, load average: 0.46, 0.24, 0.09
[root@qdevice ~]#
Anton
пн, 25 мая 2026 г. в 18:08, Anton Gavriliuk <antosha20xx@gmail.com>:
>
> Hhmm.... I just re-checked it with latest kernel (7.1-rc5) on Fedora
> Server 44, and now it works, log below. I don't remember on which
> kernel I saw that issue (7.1-rc3 or 7.1-rc4)
>
> BTW, earlier you mentioned that you are working on raid5 performance
> improvements,
>
> "“BTW we're working on performance
> improvement for large raid5 arrays for at most 64 disks, results looks great.
> However, it might take sometime before we push our work to upstream.”
>
> Are there any progress ?, I'm very interested to test on PCIe 5.0 NVMe drives.
>
> This is Fedora Server 44 (up to date) with compiled kernel 7.1-rc5
>
> [root@qdevice ~]# cat /etc/*release
> Fedora release 44 (Forty Four)
> NAME="Fedora Linux"
> VERSION="44 (Server Edition)"
> RELEASE_TYPE=stable
> ID=fedora
> VERSION_ID=44
> VERSION_CODENAME=""
> PRETTY_NAME="Fedora Linux 44 (Server Edition)"
> ANSI_COLOR="0;38;2;60;110;180"
> LOGO=fedora-logo-icon
> CPE_NAME="cpe:/o:fedoraproject:fedora:44"
> HOME_URL="https://fedoraproject.org/"
> DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f44/"
> SUPPORT_URL="https://ask.fedoraproject.org/"
> BUG_REPORT_URL="https://bugzilla.redhat.com/"
> REDHAT_BUGZILLA_PRODUCT="Fedora"
> REDHAT_BUGZILLA_PRODUCT_VERSION=44
> REDHAT_SUPPORT_PRODUCT="Fedora"
> REDHAT_SUPPORT_PRODUCT_VERSION=44
> SUPPORT_END=2027-05-19
> VARIANT="Server Edition"
> VARIANT_ID=server
> Fedora release 44 (Forty Four)
> Fedora release 44 (Forty Four)
> [root@qdevice ~]#
> [root@qdevice ~]# uname -a
> Linux qdevice 7.1.0-rc5 #1 SMP PREEMPT_DYNAMIC Mon May 25 15:02:20
> EEST 2026 x86_64 GNU/Linux
> [root@qdevice ~]#
> [root@qdevice mdadm]# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
> nvme6n1 259:1 0 447.1G 0 disk
> ├─nvme6n1p1 259:2 0 1G 0 part /boot/efi
> ├─nvme6n1p2 259:3 0 1G 0 part /boot
> ├─nvme6n1p3 259:4 0 220G 0 part /
> ├─nvme6n1p4 259:5 0 1G 0 part
> └─nvme6n1p5 259:6 0 220G 0 part
> nvme3n1 259:10 0 2.9T 0 disk
> nvme1n1 259:11 0 2.9T 0 disk
> nvme0n1 259:12 0 2.9T 0 disk
> nvme4n1 259:16 0 2.9T 0 disk
> nvme2n1 259:17 0 2.9T 0 disk
> nvme5n1 259:18 0 2.9T 0 disk
> [root@qdevice mdadm]#
> [root@qdevice mdadm]# mdadm --create --verbose /dev/md0 --level=6
> --bitmap=lockless --raid-devices=6 /dev/nvme0n1 /dev/nvme1n1
> /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1
> mdadm: Experimental lockless bitmap, use at your own risk!
> mdadm: layout defaults to left-symmetric
> mdadm: layout defaults to left-symmetric
> mdadm: chunk size defaults to 512K
> mdadm: /dev/nvme0n1 appears to be part of a raid array:
> level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
> mdadm: /dev/nvme1n1 appears to be part of a raid array:
> level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
> mdadm: /dev/nvme2n1 appears to be part of a raid array:
> level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
> mdadm: /dev/nvme3n1 appears to be part of a raid array:
> level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
> mdadm: /dev/nvme4n1 appears to be part of a raid array:
> level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
> mdadm: /dev/nvme5n1 appears to be part of a raid array:
> level=raid6 devices=6 ctime=Wed May 20 11:45:38 2026
> mdadm: size set to 3125484544K
> Continue creating array [y/N]? y
> mdadm: Defaulting to version 1.2 metadata
> mdadm: array /dev/md0 started.
> [root@qdevice mdadm]#
> [root@qdevice mdadm]# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
> nvme6n1 259:1 0 447.1G 0 disk
> ├─nvme6n1p1 259:2 0 1G 0 part /boot/efi
> ├─nvme6n1p2 259:3 0 1G 0 part /boot
> ├─nvme6n1p3 259:4 0 220G 0 part /
> ├─nvme6n1p4 259:5 0 1G 0 part
> └─nvme6n1p5 259:6 0 220G 0 part
> nvme3n1 259:10 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme1n1 259:11 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme0n1 259:12 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme4n1 259:16 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme2n1 259:17 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme5n1 259:18 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> [root@qdevice mdadm]#
> [root@qdevice mdadm]# reboot
> [root@qdevice mdadm]#
>
> After reboot,
>
> [root@qdevice ~]# lsblk
> NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
> nvme6n1 259:1 0 447.1G 0 disk
> ├─nvme6n1p1 259:2 0 1G 0 part /boot/efi
> ├─nvme6n1p2 259:3 0 1G 0 part /boot
> ├─nvme6n1p3 259:4 0 220G 0 part /
> ├─nvme6n1p4 259:5 0 1G 0 part
> └─nvme6n1p5 259:6 0 220G 0 part
> nvme2n1 259:9 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme1n1 259:10 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme4n1 259:15 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme0n1 259:16 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme3n1 259:17 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> nvme5n1 259:18 0 2.9T 0 disk
> └─md0 9:0 0 11.6T 0 raid6
> [root@qdevice ~]#
>
>
> Anton
>
> пн, 25 мая 2026 г. в 08:32, Yu Kuai <yukuai@fnnas.com>:
> >
> > Hi,
> >
> > 在 2026/5/20 1:28, Anton Gavriliuk 写道:
> > > Hi
> > >
> > > Does md-raid lockless bitmap support server's reboot ?
> >
> > Yes, of course it should support reboot.
> >
> > >
> > > I don't see md-raid6 created with --bitmap=lockless after reboot.
> >
> > Do you check kernel log if there is any failure log? And do you try
> > mdadm -a manually?
> >
> > BTW, pPlease report with your environment and test script in details,
> > especially kernel version and mdadm version.
> >
> > >
> > > Anton
> > >
> > --
> > Thansk,
> > Kuai
^ permalink raw reply
* Re: --bitmap=lockless across server reboot
From: Anton Gavriliuk @ 2026-05-25 15:08 UTC (permalink / raw)
To: yukuai; +Cc: linux-raid
In-Reply-To: <5d4b72d7-7e03-463c-82a1-63adcae20344@fygo.io>
Hhmm.... I just re-checked it with latest kernel (7.1-rc5) on Fedora
Server 44, and now it works, log below. I don't remember on which
kernel I saw that issue (7.1-rc3 or 7.1-rc4)
BTW, earlier you mentioned that you are working on raid5 performance
improvements,
"“BTW we're working on performance
improvement for large raid5 arrays for at most 64 disks, results looks great.
However, it might take sometime before we push our work to upstream.”
Are there any progress ?, I'm very interested to test on PCIe 5.0 NVMe drives.
This is Fedora Server 44 (up to date) with compiled kernel 7.1-rc5
[root@qdevice ~]# cat /etc/*release
Fedora release 44 (Forty Four)
NAME="Fedora Linux"
VERSION="44 (Server Edition)"
RELEASE_TYPE=stable
ID=fedora
VERSION_ID=44
VERSION_CODENAME=""
PRETTY_NAME="Fedora Linux 44 (Server Edition)"
ANSI_COLOR="0;38;2;60;110;180"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:44"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f44/"
SUPPORT_URL="https://ask.fedoraproject.org/"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=44
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=44
SUPPORT_END=2027-05-19
VARIANT="Server Edition"
VARIANT_ID=server
Fedora release 44 (Forty Four)
Fedora release 44 (Forty Four)
[root@qdevice ~]#
[root@qdevice ~]# uname -a
Linux qdevice 7.1.0-rc5 #1 SMP PREEMPT_DYNAMIC Mon May 25 15:02:20
EEST 2026 x86_64 GNU/Linux
[root@qdevice ~]#
[root@qdevice mdadm]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme6n1 259:1 0 447.1G 0 disk
├─nvme6n1p1 259:2 0 1G 0 part /boot/efi
├─nvme6n1p2 259:3 0 1G 0 part /boot
├─nvme6n1p3 259:4 0 220G 0 part /
├─nvme6n1p4 259:5 0 1G 0 part
└─nvme6n1p5 259:6 0 220G 0 part
nvme3n1 259:10 0 2.9T 0 disk
nvme1n1 259:11 0 2.9T 0 disk
nvme0n1 259:12 0 2.9T 0 disk
nvme4n1 259:16 0 2.9T 0 disk
nvme2n1 259:17 0 2.9T 0 disk
nvme5n1 259:18 0 2.9T 0 disk
[root@qdevice mdadm]#
[root@qdevice mdadm]# mdadm --create --verbose /dev/md0 --level=6
--bitmap=lockless --raid-devices=6 /dev/nvme0n1 /dev/nvme1n1
/dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1
mdadm: Experimental lockless bitmap, use at your own risk!
mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: /dev/nvme0n1 appears to be part of a raid array:
level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
mdadm: /dev/nvme1n1 appears to be part of a raid array:
level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
mdadm: /dev/nvme2n1 appears to be part of a raid array:
level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
mdadm: /dev/nvme3n1 appears to be part of a raid array:
level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
mdadm: /dev/nvme4n1 appears to be part of a raid array:
level=raid6 devices=5 ctime=Wed May 20 12:45:26 2026
mdadm: /dev/nvme5n1 appears to be part of a raid array:
level=raid6 devices=6 ctime=Wed May 20 11:45:38 2026
mdadm: size set to 3125484544K
Continue creating array [y/N]? y
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md0 started.
[root@qdevice mdadm]#
[root@qdevice mdadm]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme6n1 259:1 0 447.1G 0 disk
├─nvme6n1p1 259:2 0 1G 0 part /boot/efi
├─nvme6n1p2 259:3 0 1G 0 part /boot
├─nvme6n1p3 259:4 0 220G 0 part /
├─nvme6n1p4 259:5 0 1G 0 part
└─nvme6n1p5 259:6 0 220G 0 part
nvme3n1 259:10 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme1n1 259:11 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme0n1 259:12 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme4n1 259:16 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme2n1 259:17 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme5n1 259:18 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
[root@qdevice mdadm]#
[root@qdevice mdadm]# reboot
[root@qdevice mdadm]#
After reboot,
[root@qdevice ~]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
nvme6n1 259:1 0 447.1G 0 disk
├─nvme6n1p1 259:2 0 1G 0 part /boot/efi
├─nvme6n1p2 259:3 0 1G 0 part /boot
├─nvme6n1p3 259:4 0 220G 0 part /
├─nvme6n1p4 259:5 0 1G 0 part
└─nvme6n1p5 259:6 0 220G 0 part
nvme2n1 259:9 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme1n1 259:10 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme4n1 259:15 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme0n1 259:16 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme3n1 259:17 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
nvme5n1 259:18 0 2.9T 0 disk
└─md0 9:0 0 11.6T 0 raid6
[root@qdevice ~]#
Anton
пн, 25 мая 2026 г. в 08:32, Yu Kuai <yukuai@fnnas.com>:
>
> Hi,
>
> 在 2026/5/20 1:28, Anton Gavriliuk 写道:
> > Hi
> >
> > Does md-raid lockless bitmap support server's reboot ?
>
> Yes, of course it should support reboot.
>
> >
> > I don't see md-raid6 created with --bitmap=lockless after reboot.
>
> Do you check kernel log if there is any failure log? And do you try
> mdadm -a manually?
>
> BTW, pPlease report with your environment and test script in details,
> especially kernel version and mdadm version.
>
> >
> > Anton
> >
> --
> Thansk,
> Kuai
^ permalink raw reply
* Re: [PATCH 3/3] btrfs: disguise single-data-RAID56 as RAID1/RAID1C3
From: Qu Wenruo @ 2026-05-25 10:39 UTC (permalink / raw)
To: dsterba, Christoph Hellwig; +Cc: linux-btrfs, linux-raid
In-Reply-To: <20260525101707.GV12792@suse.cz>
在 2026/5/25 19:47, David Sterba 写道:
> On Mon, May 25, 2026 at 08:00:30AM +0200, Christoph Hellwig wrote:
>> On Sat, May 23, 2026 at 04:23:44PM +0200, David Sterba wrote:
>>> On Fri, May 22, 2026 at 06:53:53PM +0930, Qu Wenruo wrote:
>>>> Recently kernel RAID56 lib is trying to remove the unexpected
>>>> single-data-RAID56 (2 disks RAID5 or 3 disk RAID5) support, meanwhile
>>>> btrfs still supports such setup, which means in the long run btrfs has
>>>> to handle such corner case by ourselves.
>>>>
>>>> Thankfully single-data-RAID56 is really RAID1/RAID1C3, since data and
>>>> P/Q stripes all match each other, rotation also makes no difference.
>>>>
>>>> This patch will disguise those single-data-RAID56 chunks as
>>>> RAID1/RAID1C3 chunks.
>>>
>>> I don't think this is the right way to fix it. Calculations are an
>>> implementation detail and should be done at the time of the xor_gen or
>>> raid6_call, not touching the upper level structures related to format.
>>
>> It absolutely is. Adding fast path workarounds for this is completely
>> stupid when it can be trivially handled on the mount side.
>
> It's a lazy hack at best and fix on absolutely wrong layer. The library
> should provide the support for the edge case. But we disagree on that.
I strongly disagree.
Firstly on the layer to fix, let me be clear, if something writes like
RAID1, reads like RAID1, repairs like RAID1, then it's RAID1.
With that mindset, changing chunk type at read/add time is exactly the
correct layer to fix.
Or let me change the title of the patch, ENHANCE single-data-RAID56
handling to get rid all the corner cases.
Do you still think that's the wrong correct layer to fix?
So the wrong layer argument doesn't stand. If you're so focused on
fixing the problem in the library layer, then all you can see is to put
some adhoc fixes to emulate the library fix.
Furthermore, we are already doing all the warning about 2-disk raid5,
now I'm providing a solution that is as mature as RAID1, now you're
complaining about the simplicity but wants all the complex striping and
reduced read performance?
I don't see any point in this.
^ permalink raw reply
* Re: [PATCH 3/3] btrfs: disguise single-data-RAID56 as RAID1/RAID1C3
From: Qu Wenruo @ 2026-05-25 10:25 UTC (permalink / raw)
To: dsterba; +Cc: linux-btrfs, hch, linux-raid
In-Reply-To: <20260525101236.GU12792@suse.cz>
在 2026/5/25 19:42, David Sterba 写道:
> On Sun, May 24, 2026 at 01:43:54PM +0930, Qu Wenruo wrote:
>> I tried the idea, and it doesn't look elegant at all, we need a lot of
>> if () checks around every xor_gen()/raid6_call.
>>
>> Sure the end result is not that huge (and I have only compile tested
>> it), but if we had some parameters passed wrongly, or some corner case
>> missed, it can be very hard to notice.
>>
>> Just paste the adhoc fix for reference:
>
> I ended up with something similar, but replaced the open coded ifs with
> a helper so there's not that much churn at the call sites.
>
>> diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c
>> index 08ee8f316d96..50fefe33bbe3 100644
>> --- a/fs/btrfs/raid56.c
>> +++ b/fs/btrfs/raid56.c
>> @@ -1410,11 +1410,17 @@ static void generate_pq_vertical_step(struct
>> btrfs_raid_bio *rbio, unsigned int
>> rbio_qstripe_paddr(rbio, sector_nr, step_nr));
>>
>> assert_rbio(rbio);
>> - raid6_call.gen_syndrome(rbio->real_stripes, step, pointers);
>> + if (rbio->nr_data > 1) {
>> + raid6_call.gen_syndrome(rbio->real_stripes, step, pointers);
>> + } else {
>> + memcpy(pointers[1], pointers[0], step);
>> + memcpy(pointers[2], pointers[0], step);
>> + }
>> } else {
>> /* raid5 */
>> memcpy(pointers[rbio->nr_data], pointers[0], step);
>> - xor_gen(pointers[rbio->nr_data], pointers + 1, rbio->nr_data - 1,
>> + if (rbio->nr_data > 1)
>> + xor_gen(pointers[rbio->nr_data], pointers + 1, rbio->nr_data - 1,
>> step);
>> }
>> for (stripe = stripe - 1; stripe >= 0; stripe--)
>> @@ -1987,11 +1993,17 @@ static void recover_vertical_step(struct
>> btrfs_raid_bio *rbio,
>> }
>>
>> if (failb == rbio->real_stripes - 2) {
>> - raid6_datap_recov(rbio->real_stripes, step,
>> - faila, pointers);
>> + if (unlikely(rbio->nr_data == 1))
>> + memcpy(pointers[0], pointers[rbio->real_stripes - 1], step);
>> + else
>> + raid6_datap_recov(rbio->real_stripes, step,
>> + faila, pointers);
>> } else {
>> - raid6_2data_recov(rbio->real_stripes, step,
>> - faila, failb, pointers);
>> + if (unlikely(rbio->nr_data == 1))
>> + memcpy(pointers[0], pointers[rbio->real_stripes - 2], step);
>
> The 2data recovery needs to do 2 memcpy
Nope. The recovery is only to repair the data stripe.
We do not need to bother the failed P or Q stripe here.
And I think this explains exactly why doing adhoc fix is a stupid idea
in the first place.
We're wasting time on useless corner cases, and corner cases inside
corner cases.
It makes no sense to waste all the time reviewing those useless details,
when we have a much better solution to avoid all those hassles.
^ permalink raw reply
* Re: [PATCH 3/3] btrfs: disguise single-data-RAID56 as RAID1/RAID1C3
From: David Sterba @ 2026-05-25 10:17 UTC (permalink / raw)
To: Christoph Hellwig; +Cc: Qu Wenruo, linux-btrfs, linux-raid
In-Reply-To: <20260525060030.GB3370@lst.de>
On Mon, May 25, 2026 at 08:00:30AM +0200, Christoph Hellwig wrote:
> On Sat, May 23, 2026 at 04:23:44PM +0200, David Sterba wrote:
> > On Fri, May 22, 2026 at 06:53:53PM +0930, Qu Wenruo wrote:
> > > Recently kernel RAID56 lib is trying to remove the unexpected
> > > single-data-RAID56 (2 disks RAID5 or 3 disk RAID5) support, meanwhile
> > > btrfs still supports such setup, which means in the long run btrfs has
> > > to handle such corner case by ourselves.
> > >
> > > Thankfully single-data-RAID56 is really RAID1/RAID1C3, since data and
> > > P/Q stripes all match each other, rotation also makes no difference.
> > >
> > > This patch will disguise those single-data-RAID56 chunks as
> > > RAID1/RAID1C3 chunks.
> >
> > I don't think this is the right way to fix it. Calculations are an
> > implementation detail and should be done at the time of the xor_gen or
> > raid6_call, not touching the upper level structures related to format.
>
> It absolutely is. Adding fast path workarounds for this is completely
> stupid when it can be trivially handled on the mount side.
It's a lazy hack at best and fix on absolutely wrong layer. The library
should provide the support for the edge case. But we disagree on that.
^ permalink raw reply
* Re: [PATCH 3/3] btrfs: disguise single-data-RAID56 as RAID1/RAID1C3
From: David Sterba @ 2026-05-25 10:12 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs, hch, linux-raid
In-Reply-To: <1b5c39fe-df28-4a21-bace-fc742498945e@suse.com>
On Sun, May 24, 2026 at 01:43:54PM +0930, Qu Wenruo wrote:
> I tried the idea, and it doesn't look elegant at all, we need a lot of
> if () checks around every xor_gen()/raid6_call.
>
> Sure the end result is not that huge (and I have only compile tested
> it), but if we had some parameters passed wrongly, or some corner case
> missed, it can be very hard to notice.
>
> Just paste the adhoc fix for reference:
I ended up with something similar, but replaced the open coded ifs with
a helper so there's not that much churn at the call sites.
> diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c
> index 08ee8f316d96..50fefe33bbe3 100644
> --- a/fs/btrfs/raid56.c
> +++ b/fs/btrfs/raid56.c
> @@ -1410,11 +1410,17 @@ static void generate_pq_vertical_step(struct
> btrfs_raid_bio *rbio, unsigned int
> rbio_qstripe_paddr(rbio, sector_nr, step_nr));
>
> assert_rbio(rbio);
> - raid6_call.gen_syndrome(rbio->real_stripes, step, pointers);
> + if (rbio->nr_data > 1) {
> + raid6_call.gen_syndrome(rbio->real_stripes, step, pointers);
> + } else {
> + memcpy(pointers[1], pointers[0], step);
> + memcpy(pointers[2], pointers[0], step);
> + }
> } else {
> /* raid5 */
> memcpy(pointers[rbio->nr_data], pointers[0], step);
> - xor_gen(pointers[rbio->nr_data], pointers + 1, rbio->nr_data - 1,
> + if (rbio->nr_data > 1)
> + xor_gen(pointers[rbio->nr_data], pointers + 1, rbio->nr_data - 1,
> step);
> }
> for (stripe = stripe - 1; stripe >= 0; stripe--)
> @@ -1987,11 +1993,17 @@ static void recover_vertical_step(struct
> btrfs_raid_bio *rbio,
> }
>
> if (failb == rbio->real_stripes - 2) {
> - raid6_datap_recov(rbio->real_stripes, step,
> - faila, pointers);
> + if (unlikely(rbio->nr_data == 1))
> + memcpy(pointers[0], pointers[rbio->real_stripes - 1], step);
> + else
> + raid6_datap_recov(rbio->real_stripes, step,
> + faila, pointers);
> } else {
> - raid6_2data_recov(rbio->real_stripes, step,
> - faila, failb, pointers);
> + if (unlikely(rbio->nr_data == 1))
> + memcpy(pointers[0], pointers[rbio->real_stripes - 2], step);
The 2data recovery needs to do 2 memcpy
> + else
> + raid6_2data_recov(rbio->real_stripes, step,
> + faila, failb, pointers);
> }
> } else {
> void *p;
> @@ -2002,6 +2014,9 @@ static void recover_vertical_step(struct
> btrfs_raid_bio *rbio,
> /* Copy parity block into failed block to start with */
> memcpy(pointers[faila], pointers[rbio->nr_data], step);
>
> + if (rbio->nr_data == 1)
> + goto cleanup;
> +
> /* Rearrange the pointer array */
> p = pointers[faila];
> for (stripe_nr = faila; stripe_nr < rbio->nr_data - 1;
> @@ -2644,11 +2659,17 @@ static bool verify_one_parity_step(struct
> btrfs_raid_bio *rbio,
> if (has_qstripe) {
> assert_rbio(rbio);
> /* RAID6, call the library function to fill in our P/Q. */
> - raid6_call.gen_syndrome(rbio->real_stripes, step, pointers);
> + if (rbio->nr_data > 1) {
> + raid6_call.gen_syndrome(rbio->real_stripes, step, pointers);
> + } else {
> + memcpy(pointers[1], pointers[0], step);
> + memcpy(pointers[2], pointers[0], step);
> + }
> } else {
> /* RAID5. */
> memcpy(pointers[nr_data], pointers[0], step);
> - xor_gen(pointers[nr_data], pointers + 1, nr_data - 1, step);
> + if (rbio->nr_data > 1)
> + xor_gen(pointers[nr_data], pointers + 1, nr_data - 1, step);
> }
>
> /* Check scrubbing parity and repair it. */
> --
> 2.54.0
^ permalink raw reply
* Re: [PATCH 3/3] btrfs: disguise single-data-RAID56 as RAID1/RAID1C3
From: Christoph Hellwig @ 2026-05-25 6:00 UTC (permalink / raw)
To: David Sterba; +Cc: Qu Wenruo, linux-btrfs, hch, linux-raid
In-Reply-To: <20260523142344.GC183694@twin.jikos.cz>
On Sat, May 23, 2026 at 04:23:44PM +0200, David Sterba wrote:
> On Fri, May 22, 2026 at 06:53:53PM +0930, Qu Wenruo wrote:
> > Recently kernel RAID56 lib is trying to remove the unexpected
> > single-data-RAID56 (2 disks RAID5 or 3 disk RAID5) support, meanwhile
> > btrfs still supports such setup, which means in the long run btrfs has
> > to handle such corner case by ourselves.
> >
> > Thankfully single-data-RAID56 is really RAID1/RAID1C3, since data and
> > P/Q stripes all match each other, rotation also makes no difference.
> >
> > This patch will disguise those single-data-RAID56 chunks as
> > RAID1/RAID1C3 chunks.
>
> I don't think this is the right way to fix it. Calculations are an
> implementation detail and should be done at the time of the xor_gen or
> raid6_call, not touching the upper level structures related to format.
It absolutely is. Adding fast path workarounds for this is completely
stupid when it can be trivially handled on the mount side.
^ permalink raw reply
* Re: --bitmap=lockless across server reboot
From: Yu Kuai @ 2026-05-25 5:32 UTC (permalink / raw)
To: Anton Gavriliuk, linux-raid, yukuai
In-Reply-To: <CAAiJnjpPAur30W4FO2sTZcYWc24yFafKWGR=BZWZsh2JnbrRpQ@mail.gmail.com>
Hi,
在 2026/5/20 1:28, Anton Gavriliuk 写道:
> Hi
>
> Does md-raid lockless bitmap support server's reboot ?
Yes, of course it should support reboot.
>
> I don't see md-raid6 created with --bitmap=lockless after reboot.
Do you check kernel log if there is any failure log? And do you try
mdadm -a manually?
BTW, pPlease report with your environment and test script in details,
especially kernel version and mdadm version.
>
> Anton
>
--
Thansk,
Kuai
^ permalink raw reply
* [PATCH v3 2/2] md/raid10: bound reused r10bio devs[] walks by used_nr_devs
From: Chen Cheng @ 2026-05-25 1:55 UTC (permalink / raw)
To: Yu Kuai, linux-raid; +Cc: Chen Cheng, linux-kernel
In-Reply-To: <20260525015520.2565423-1-chencheng@fnnas.com>
From: Chen Cheng <chencheng@fnnas.com>
After reshape changes raid_disks, an in-flight r10bio from the old geometry
can still be completed or freed later. In that case, using the current
geometry to walk r10_bio->devs[] is unsafe. A failure was reproduced with a
simple write workload while reshaping a raid10 array from 4 disks to 5 disks.
e.g.:
mdadm -C /dev/md777 -l10 -n4 /dev/sda /dev/sdb /dev/sdc /dev/sdd
mkfs.ext4 /dev/md777
mount /dev/md777 /mnt/test
fsstress -d /mnt/test -n 24000 -p 8 -l 24 &
mdadm /dev/md777 --add /dev/sde
mdadm --grow /dev/md777 --raid-devices=5 \
--backup-file=/tmp/md-reshape-backup
the sequence above can trigger:
BUG: KASAN: slab-out-of-bounds in free_r10bio+0x1c4/0x260 [raid10]
Read of size 8 at addr ffff00008c2dfac8 by task ksoftirqd/0/15
free_r10bio
raid_end_bio_io
one_write_done
raid10_end_write_request
The buggy object was 200 bytes long, which matches an r10bio with space for
only four devs[] entries. However, put_all_bios() and find_bio_disk() walk
r10_bio->devs[] using the current conf->geo.raid_disks value. Once reshape
switches conf->geo.raid_disks from 4 to 5, an old 4-slot r10bio can be
completed or freed as if it had 5 slots, and the walk overruns devs[4]. The
same stale-width mismatch can also surface during a 5-disk to 4-disk reshape.
Track the number of valid devs[] entries in each reused r10bio with
used_nr_devs. Initialize it whenever an r10bio is prepared for regular I/O,
discard, or resync/recovery/reshape work, and use it to bound devs[] walks
in put_all_bios() and find_bio_disk().
Signed-off-by: Chen Cheng <chencheng@fnnas.com>
---
drivers/md/raid10.c | 8 ++++++--
drivers/md/raid10.h | 2 ++
2 files changed, 8 insertions(+), 2 deletions(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 64677dbe5152..d4695fa9c076 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -273,11 +273,11 @@ static void r10buf_pool_free(void *__r10_bio, void *data)
static void put_all_bios(struct r10conf *conf, struct r10bio *r10_bio)
{
int i;
- for (i = 0; i < conf->geo.raid_disks; i++) {
+ for (i = 0; i < r10_bio->used_nr_devs; i++) {
struct bio **bio = & r10_bio->devs[i].bio;
if (!BIO_SPECIAL(*bio))
bio_put(*bio);
*bio = NULL;
bio = &r10_bio->devs[i].repl_bio;
@@ -370,11 +370,11 @@ static int find_bio_disk(struct r10conf *conf, struct r10bio *r10_bio,
struct bio *bio, int *slotp, int *replp)
{
int slot;
int repl = 0;
- for (slot = 0; slot < conf->geo.raid_disks; slot++) {
+ for (slot = 0; slot < r10_bio->used_nr_devs; slot++) {
if (r10_bio->devs[slot].bio == bio)
break;
if (r10_bio->devs[slot].repl_bio == bio) {
repl = 1;
break;
@@ -1553,10 +1553,11 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
r10_bio->mddev = mddev;
r10_bio->sector = bio->bi_iter.bi_sector;
r10_bio->state = 0;
r10_bio->read_slot = -1;
+ r10_bio->used_nr_devs = conf->geo.raid_disks;
memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) *
conf->geo.raid_disks);
if (bio_data_dir(bio) == READ)
raid10_read_request(mddev, bio, r10_bio, true);
@@ -1740,10 +1741,11 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
retry_discard:
r10_bio = mempool_alloc(conf->r10bio_pool, GFP_NOIO);
r10_bio->mddev = mddev;
r10_bio->state = 0;
r10_bio->sectors = 0;
+ r10_bio->used_nr_devs = geo->raid_disks;
memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) * geo->raid_disks);
wait_blocked_dev(mddev, r10_bio);
/*
* For far layout it needs more than one r10bio to cover all regions.
@@ -3074,10 +3076,12 @@ static struct r10bio *raid10_alloc_init_r10buf(struct r10conf *conf)
test_bit(MD_RECOVERY_RESHAPE, &conf->mddev->recovery))
nalloc = conf->copies; /* resync */
else
nalloc = 2; /* recovery */
+ r10bio->used_nr_devs = nalloc;
+
for (i = 0; i < nalloc; i++) {
bio = r10bio->devs[i].bio;
rp = bio->bi_private;
bio_reset(bio, NULL, 0);
bio->bi_private = rp;
diff --git a/drivers/md/raid10.h b/drivers/md/raid10.h
index b711626a5db7..4751119f9770 100644
--- a/drivers/md/raid10.h
+++ b/drivers/md/raid10.h
@@ -125,10 +125,12 @@ struct r10bio {
struct bio *master_bio;
/*
* if the IO is in READ direction, then this is where we read
*/
int read_slot;
+ /* Used to bound devs[] walks when the object is reused. */
+ unsigned int used_nr_devs;
struct list_head retry_list;
/*
* if the IO is in WRITE direction, then multiple bios are used,
* one for each copy.
--
2.54.0
^ permalink raw reply related
* [PATCH v3 1/2] md/raid10: make r10bio_pool use fixed-size objects
From: Chen Cheng @ 2026-05-25 1:55 UTC (permalink / raw)
To: Yu Kuai, linux-raid; +Cc: Chen Cheng, linux-kernel
In-Reply-To: <20260525015520.2565423-1-chencheng@fnnas.com>
From: Chen Cheng <chencheng@fnnas.com>
raid10 currently allocates r10bio_pool objects with conf->geo.raid_disks,
which makes regular r10bio objects geometry-dependent.
That model breaks down across reshape. mempool objects are preallocated and
reused, so a reshape that changes the number of raid disks can leave old
r10bio objects in the regular I/O pool with a devs[] array sized for the
previous geometry. After the geometry switch, those stale objects may be
reused or later freed under the new layout, creating a width mismatch
between the reused r10bio and the current array geometry.
For example, during a 4-disk to 5-disk reshape, an r10bio allocated before
the geometry switch has room for only 4 devs[] entries. After reshape
updates conf->geo.raid_disks to 5, that stale object can be reused under
the new geometry. Code such as __make_request(), put_all_bios(), and
find_bio_disk() may then access devs[] using the new geometry and step
past the end of the old 4-slot object, leading to slab out-of-bounds
accesses.
The root problem is that regular r10bio pool objects are geometry-dependent,
while mempool elements are preallocated and reused across requests.
Switch r10bio_pool to a fixed-size kmalloc mempool so regular I/O objects no
longer carry an allocation width tied to the current geometry. Use the same
fixed-size allocation rule for the standalone r10bio allocated from
r10buf_pool_alloc().
Signed-off-by: Chen Cheng <chencheng@fnnas.com>
---
drivers/md/raid10.c | 57 ++++++++++++++++++++++++++++++++++-----------
drivers/md/raid10.h | 2 +-
2 files changed, 45 insertions(+), 14 deletions(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 39085e7dd6d2..64677dbe5152 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -101,17 +101,32 @@ static void end_reshape(struct r10conf *conf);
static inline struct r10bio *get_resync_r10bio(struct bio *bio)
{
return get_resync_pages(bio)->raid_bio;
}
-static void * r10bio_pool_alloc(gfp_t gfp_flags, void *data)
+static inline unsigned int calc_r10bio_pool_disks(struct mddev *mddev)
{
- struct r10conf *conf = data;
- int size = offsetof(struct r10bio, devs[conf->geo.raid_disks]);
+ /* If delta_disks < 0, use bigger r10bio->devs[] is ok. */
+ return mddev->raid_disks + max(0, mddev->delta_disks);
+}
+
+static inline int calc_r10bio_size(struct mddev *mddev)
+{
+ return offsetof(struct r10bio, devs[calc_r10bio_pool_disks(mddev)]);
+}
+
+static mempool_t *create_r10bio_pool(struct mddev *mddev)
+{
+ int size = calc_r10bio_size(mddev);
+
+ return mempool_create_kmalloc_pool(NR_RAID_BIOS, size);
+}
+
+static struct r10bio *alloc_r10bio(struct mddev *mddev, gfp_t gfp_flags)
+{
+ int size = calc_r10bio_size(mddev);
- /* allocate a r10bio with room for raid_disks entries in the
- * bios array */
return kzalloc(size, gfp_flags);
}
#define RESYNC_SECTORS (RESYNC_BLOCK_SIZE >> 9)
/* amount of memory to reserve for resync requests */
@@ -135,11 +150,11 @@ static void * r10buf_pool_alloc(gfp_t gfp_flags, void *data)
struct bio *bio;
int j;
int nalloc, nalloc_rp;
struct resync_pages *rps;
- r10_bio = r10bio_pool_alloc(gfp_flags, conf);
+ r10_bio = alloc_r10bio(conf->mddev, gfp_flags);
if (!r10_bio)
return NULL;
if (test_bit(MD_RECOVERY_SYNC, &conf->mddev->recovery) ||
test_bit(MD_RECOVERY_RESHAPE, &conf->mddev->recovery))
@@ -275,11 +290,11 @@ static void put_all_bios(struct r10conf *conf, struct r10bio *r10_bio)
static void free_r10bio(struct r10bio *r10_bio)
{
struct r10conf *conf = r10_bio->mddev->private;
put_all_bios(conf, r10_bio);
- mempool_free(r10_bio, &conf->r10bio_pool);
+ mempool_free(r10_bio, conf->r10bio_pool);
}
static void put_buf(struct r10bio *r10_bio)
{
struct r10conf *conf = r10_bio->mddev->private;
@@ -1529,11 +1544,11 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio,
static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
{
struct r10conf *conf = mddev->private;
struct r10bio *r10_bio;
- r10_bio = mempool_alloc(&conf->r10bio_pool, GFP_NOIO);
+ r10_bio = mempool_alloc(conf->r10bio_pool, GFP_NOIO);
r10_bio->master_bio = bio;
r10_bio->sectors = sectors;
r10_bio->mddev = mddev;
@@ -1721,11 +1736,11 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
last_stripe_index *= geo->far_copies;
end_disk_offset = (bio_end & geo->chunk_mask) +
(last_stripe_index << geo->chunk_shift);
retry_discard:
- r10_bio = mempool_alloc(&conf->r10bio_pool, GFP_NOIO);
+ r10_bio = mempool_alloc(conf->r10bio_pool, GFP_NOIO);
r10_bio->mddev = mddev;
r10_bio->state = 0;
r10_bio->sectors = 0;
memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) * geo->raid_disks);
wait_blocked_dev(mddev, r10_bio);
@@ -3821,11 +3836,11 @@ static int setup_geo(struct geom *geo, struct mddev *mddev, enum geo_type new)
static void raid10_free_conf(struct r10conf *conf)
{
if (!conf)
return;
- mempool_exit(&conf->r10bio_pool);
+ mempool_destroy(conf->r10bio_pool);
kfree(conf->mirrors);
kfree(conf->mirrors_old);
kfree(conf->mirrors_new);
safe_put_page(conf->tmppage);
bioset_exit(&conf->bio_split);
@@ -3868,13 +3883,12 @@ static struct r10conf *setup_conf(struct mddev *mddev)
if (!conf->tmppage)
goto out;
conf->geo = geo;
conf->copies = copies;
- err = mempool_init(&conf->r10bio_pool, NR_RAID_BIOS, r10bio_pool_alloc,
- rbio_pool_free, conf);
- if (err)
+ conf->r10bio_pool = create_r10bio_pool(mddev);
+ if (!conf->r10bio_pool)
goto out;
err = bioset_init(&conf->bio_split, BIO_POOL_SIZE, 0, 0);
if (err)
goto out;
@@ -4363,10 +4377,11 @@ static int raid10_start_reshape(struct mddev *mddev)
struct geom new;
struct r10conf *conf = mddev->private;
struct md_rdev *rdev;
int spares = 0;
int ret;
+ mempool_t *new_pool;
if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery))
return -EBUSY;
if (setup_geo(&new, mddev, geo_start) != conf->copies)
@@ -4398,11 +4413,23 @@ static int raid10_start_reshape(struct mddev *mddev)
return -EINVAL;
if (spares < mddev->delta_disks)
return -EINVAL;
+ mddev_unlock(mddev);
+ mddev_suspend_and_lock_nointr(mddev);
conf->offset_diff = min_offset_diff;
+ if (mddev->delta_disks > 0) {
+ new_pool = create_r10bio_pool(mddev);
+ if (!new_pool) {
+ mddev_unlock_and_resume(mddev);
+ mddev_lock_nointr(mddev);
+ return -ENOMEM;
+ }
+ mempool_destroy(conf->r10bio_pool);
+ conf->r10bio_pool = new_pool;
+ }
spin_lock_irq(&conf->device_lock);
if (conf->mirrors_new) {
memcpy(conf->mirrors_new, conf->mirrors,
sizeof(struct raid10_info)*conf->prev.raid_disks);
smp_mb();
@@ -4415,20 +4442,24 @@ static int raid10_start_reshape(struct mddev *mddev)
smp_mb();
if (mddev->reshape_backwards) {
sector_t size = raid10_size(mddev, 0, 0);
if (size < mddev->array_sectors) {
spin_unlock_irq(&conf->device_lock);
+ mddev_unlock_and_resume(mddev);
+ mddev_lock_nointr(mddev);
pr_warn("md/raid10:%s: array size must be reduce before number of disks\n",
mdname(mddev));
return -EINVAL;
}
mddev->resync_max_sectors = size;
conf->reshape_progress = size;
} else
conf->reshape_progress = 0;
conf->reshape_safe = conf->reshape_progress;
spin_unlock_irq(&conf->device_lock);
+ mddev_unlock_and_resume(mddev);
+ mddev_lock_nointr(mddev);
if (mddev->delta_disks && mddev->bitmap) {
struct mdp_superblock_1 *sb = NULL;
sector_t oldsize, newsize;
diff --git a/drivers/md/raid10.h b/drivers/md/raid10.h
index ec79d87fb92f..b711626a5db7 100644
--- a/drivers/md/raid10.h
+++ b/drivers/md/raid10.h
@@ -85,11 +85,11 @@ struct r10conf {
int have_replacement; /* There is at least one
* replacement device.
*/
wait_queue_head_t wait_barrier;
- mempool_t r10bio_pool;
+ mempool_t *r10bio_pool;
mempool_t r10buf_pool;
struct page *tmppage;
struct bio_set bio_split;
/* When taking over an array from a different personality, we store
--
2.54.0
^ permalink raw reply related
* [PATCH v3 0/2] md/raid10: fix r10bio width mismatches across reshape
From: Chen Cheng @ 2026-05-25 1:55 UTC (permalink / raw)
To: Yu Kuai, linux-raid; +Cc: Chen Cheng, linux-kernel
From: Chen Cheng <chencheng@fnnas.com>
Hi,
This series fixes slab out-of-bounds accesses in raid10 when reshape changes
the number of raid disks while regular I/O is still reusing r10bio objects
allocated under the previous geometry.
The bug is reproducible with a simple 4-disk to 5-disk reshape under write
load, for example:
mdadm -C /dev/md777 -l10 -n4 /dev/sda /dev/sdb /dev/sdc /dev/sdd
mkfs.ext4 /dev/md777
mount /dev/md777 /mnt/test
fsstress -d /mnt/test -n 24000 -p 8 -l 24 &
mdadm /dev/md777 --add /dev/sde
mdadm --grow /dev/md777 --raid-devices=5 \
--backup-file=/tmp/md-reshape-backup
Without these changes, an r10bio allocated under the old geometry can later be
reused, initialized, or freed after conf->geo.raid_disks has switched to the
new geometry. This creates width mismatches between the object and the current
devs[] walk/initialization width, which can trigger KASAN reports such as
slab-out-of-bounds in __make_request(), put_all_bios(), or find_bio_disk().
This series addresses the problem in two steps:
1. make the regular r10bio pool fixed-size across reshape transitions, and
move the pool rebuild into the freeze window before the live geometry
switch;
2. track the number of valid devs[] entries in each reused r10bio and use
that recorded width when walking devs[] after reshape.
Changes in v3:
- Replace freeze_array()/unfreeze_array() in raid10_start_reshape() with
mddev_suspend_and_lock_nointr()/mddev_unlock_and_resume(). freeze_array()
returns when nr_pending == nr_queued, which still allows retry-list items
to hold pool objects; mddev_suspend() provides the correct upper-layer
quiesce interface. (Suggested by Yu Kuai)
Changes in v2:
- add this cover letter
- convert r10bio_pool to a fixed-size kmalloc mempool
- rebuild r10bio_pool inside the freeze window before switching live reshape
geometry
- switch raid10_quiesce() to freeze_array()/unfreeze_array()
Testing:
- reproduced the original KASAN slab-out-of-bounds on 4-disk -> 5-disk
raid10 reshape with fsstress
- verified that this series fixes that reproducer
- exercised the 5-disk -> 4-disk reshape direction as well
Thanks,
Chen Cheng
Chen Cheng (2):
md/raid10: make r10bio_pool use fixed-size objects
md/raid10: bound reused r10bio devs[] walks by used_nr_devs
drivers/md/raid10.c | 65 ++++++++++++++++++++++++++++++++++-----------
drivers/md/raid10.h | 4 ++-
2 files changed, 53 insertions(+), 16 deletions(-)
--
2.54.0
^ permalink raw reply
* LBS setup
From: Piergiorgio Sartor @ 2026-05-24 10:27 UTC (permalink / raw)
To: linux-raid
Hi all,
I've some questions about some recent
report from kernel.
I assume from 7.x.y, from what I understood.
Not sure about the <= 6.18 reported.
In the boot log there is:
md0: echo current LBS to md/logical_block_size to prevent data loss issues from LBS changes.
Note: After setting, array will not be assembled in old kernels (<= 6.18)
I found the docs stating:
logical_block_size
... Write the value before starting array. ...
Now, should we really copy the value from:
/sys/class/block/md0/queue/logical_block_size
to:
/sys/class/block/md0/md/logical_block_size
???
Note, the first in my system is 512, the
second is 0.
Second, the docs says "before starting array".
In case this is due (copy the value), how to do
for a system array?
I mean, the array where the whole OS lies.
If needed, is there any way to do this automatically?
Using mdadm or something?
Thanks,
bye,
--
piergiorgio
^ permalink raw reply
* Re: [PATCH 3/3] btrfs: disguise single-data-RAID56 as RAID1/RAID1C3
From: Qu Wenruo @ 2026-05-24 4:13 UTC (permalink / raw)
To: dsterba; +Cc: linux-btrfs, hch, linux-raid
In-Reply-To: <e1961351-d563-4d73-b9f3-30063bbe7c4d@suse.com>
在 2026/5/24 08:16, Qu Wenruo 写道:
>
>
> 在 2026/5/23 23:53, David Sterba 写道:
>> On Fri, May 22, 2026 at 06:53:53PM +0930, Qu Wenruo wrote:
>>> Recently kernel RAID56 lib is trying to remove the unexpected
>>> single-data-RAID56 (2 disks RAID5 or 3 disk RAID5) support, meanwhile
>>> btrfs still supports such setup, which means in the long run btrfs has
>>> to handle such corner case by ourselves.
>>>
>>> Thankfully single-data-RAID56 is really RAID1/RAID1C3, since data and
>>> P/Q stripes all match each other, rotation also makes no difference.
>>>
>>> This patch will disguise those single-data-RAID56 chunks as
>>> RAID1/RAID1C3 chunks.
>>
>> I don't think this is the right way to fix it. Calculations are an
>> implementation detail and should be done at the time of the xor_gen or
>> raid6_call, not touching the upper level structures related to format.
>>
>> Instead, please add helpers that take the same parameters and do the
>> switch if there's the edge case of raid5 or raid6, using memcpy. Thanks.
>
> I'd say going dedicated memcpy() is an adhoc fix, and not any easier to
> maintain.
>
> Now you have two paths, one from the raid56 lib, one from the btrfs
> specific handling for the single-data specific corner case, and that no
> one else is going to review/utilize that corner case.
>
> From this point of view, reusing the existing RAID1/RAID1C3 is way more
> reliable and sane.
I tried the idea, and it doesn't look elegant at all, we need a lot of
if () checks around every xor_gen()/raid6_call.
Sure the end result is not that huge (and I have only compile tested
it), but if we had some parameters passed wrongly, or some corner case
missed, it can be very hard to notice.
Just paste the adhoc fix for reference:
diff --git a/fs/btrfs/raid56.c b/fs/btrfs/raid56.c
index 08ee8f316d96..50fefe33bbe3 100644
--- a/fs/btrfs/raid56.c
+++ b/fs/btrfs/raid56.c
@@ -1410,11 +1410,17 @@ static void generate_pq_vertical_step(struct
btrfs_raid_bio *rbio, unsigned int
rbio_qstripe_paddr(rbio, sector_nr, step_nr));
assert_rbio(rbio);
- raid6_call.gen_syndrome(rbio->real_stripes, step, pointers);
+ if (rbio->nr_data > 1) {
+ raid6_call.gen_syndrome(rbio->real_stripes, step, pointers);
+ } else {
+ memcpy(pointers[1], pointers[0], step);
+ memcpy(pointers[2], pointers[0], step);
+ }
} else {
/* raid5 */
memcpy(pointers[rbio->nr_data], pointers[0], step);
- xor_gen(pointers[rbio->nr_data], pointers + 1, rbio->nr_data - 1,
+ if (rbio->nr_data > 1)
+ xor_gen(pointers[rbio->nr_data], pointers + 1, rbio->nr_data - 1,
step);
}
for (stripe = stripe - 1; stripe >= 0; stripe--)
@@ -1987,11 +1993,17 @@ static void recover_vertical_step(struct
btrfs_raid_bio *rbio,
}
if (failb == rbio->real_stripes - 2) {
- raid6_datap_recov(rbio->real_stripes, step,
- faila, pointers);
+ if (unlikely(rbio->nr_data == 1))
+ memcpy(pointers[0], pointers[rbio->real_stripes - 1], step);
+ else
+ raid6_datap_recov(rbio->real_stripes, step,
+ faila, pointers);
} else {
- raid6_2data_recov(rbio->real_stripes, step,
- faila, failb, pointers);
+ if (unlikely(rbio->nr_data == 1))
+ memcpy(pointers[0], pointers[rbio->real_stripes - 2], step);
+ else
+ raid6_2data_recov(rbio->real_stripes, step,
+ faila, failb, pointers);
}
} else {
void *p;
@@ -2002,6 +2014,9 @@ static void recover_vertical_step(struct
btrfs_raid_bio *rbio,
/* Copy parity block into failed block to start with */
memcpy(pointers[faila], pointers[rbio->nr_data], step);
+ if (rbio->nr_data == 1)
+ goto cleanup;
+
/* Rearrange the pointer array */
p = pointers[faila];
for (stripe_nr = faila; stripe_nr < rbio->nr_data - 1;
@@ -2644,11 +2659,17 @@ static bool verify_one_parity_step(struct
btrfs_raid_bio *rbio,
if (has_qstripe) {
assert_rbio(rbio);
/* RAID6, call the library function to fill in our P/Q. */
- raid6_call.gen_syndrome(rbio->real_stripes, step, pointers);
+ if (rbio->nr_data > 1) {
+ raid6_call.gen_syndrome(rbio->real_stripes, step, pointers);
+ } else {
+ memcpy(pointers[1], pointers[0], step);
+ memcpy(pointers[2], pointers[0], step);
+ }
} else {
/* RAID5. */
memcpy(pointers[nr_data], pointers[0], step);
- xor_gen(pointers[nr_data], pointers + 1, nr_data - 1, step);
+ if (rbio->nr_data > 1)
+ xor_gen(pointers[nr_data], pointers + 1, nr_data - 1, step);
}
/* Check scrubbing parity and repair it. */
--
2.54.0
^ permalink raw reply related
* Re: [PATCH 3/3] btrfs: disguise single-data-RAID56 as RAID1/RAID1C3
From: Qu Wenruo @ 2026-05-23 22:46 UTC (permalink / raw)
To: dsterba; +Cc: linux-btrfs, hch, linux-raid
In-Reply-To: <20260523142344.GC183694@twin.jikos.cz>
在 2026/5/23 23:53, David Sterba 写道:
> On Fri, May 22, 2026 at 06:53:53PM +0930, Qu Wenruo wrote:
>> Recently kernel RAID56 lib is trying to remove the unexpected
>> single-data-RAID56 (2 disks RAID5 or 3 disk RAID5) support, meanwhile
>> btrfs still supports such setup, which means in the long run btrfs has
>> to handle such corner case by ourselves.
>>
>> Thankfully single-data-RAID56 is really RAID1/RAID1C3, since data and
>> P/Q stripes all match each other, rotation also makes no difference.
>>
>> This patch will disguise those single-data-RAID56 chunks as
>> RAID1/RAID1C3 chunks.
>
> I don't think this is the right way to fix it. Calculations are an
> implementation detail and should be done at the time of the xor_gen or
> raid6_call, not touching the upper level structures related to format.
>
> Instead, please add helpers that take the same parameters and do the
> switch if there's the edge case of raid5 or raid6, using memcpy. Thanks.
I'd say going dedicated memcpy() is an adhoc fix, and not any easier to
maintain.
Now you have two paths, one from the raid56 lib, one from the btrfs
specific handling for the single-data specific corner case, and that no
one else is going to review/utilize that corner case.
From this point of view, reusing the existing RAID1/RAID1C3 is way more
reliable and sane.
^ permalink raw reply
* Re: [PATCH 3/3] btrfs: disguise single-data-RAID56 as RAID1/RAID1C3
From: David Sterba @ 2026-05-23 14:23 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs, hch, linux-raid
In-Reply-To: <ce02ebf8a9a6f5e2357b838df9d7ecce1f0cfd4b.1779412152.git.wqu@suse.com>
On Fri, May 22, 2026 at 06:53:53PM +0930, Qu Wenruo wrote:
> Recently kernel RAID56 lib is trying to remove the unexpected
> single-data-RAID56 (2 disks RAID5 or 3 disk RAID5) support, meanwhile
> btrfs still supports such setup, which means in the long run btrfs has
> to handle such corner case by ourselves.
>
> Thankfully single-data-RAID56 is really RAID1/RAID1C3, since data and
> P/Q stripes all match each other, rotation also makes no difference.
>
> This patch will disguise those single-data-RAID56 chunks as
> RAID1/RAID1C3 chunks.
I don't think this is the right way to fix it. Calculations are an
implementation detail and should be done at the time of the xor_gen or
raid6_call, not touching the upper level structures related to format.
Instead, please add helpers that take the same parameters and do the
switch if there's the edge case of raid5 or raid6, using memcpy. Thanks.
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox