* Re: [PATCH V3 3/3] nvme-multipath: enable PCI P2PDMA for multipath devices
From: Nitesh Shetty @ 2026-04-17 10:42 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: song, yukuai, linan122, kbusch, axboe, hch, sagi, linux-raid,
linux-nvme, kmodukuri
In-Reply-To: <20260416212633.72650-4-kch@nvidia.com>
[-- Attachment #1: Type: text/plain, Size: 1498 bytes --]
On 16/04/26 02:26PM, Chaitanya Kulkarni wrote:
>From: Kiran Kumar Modukuri <kmodukuri@nvidia.com>
>
>NVMe multipath does not expose BLK_FEAT_PCI_P2PDMA on the head disk
>even when all underlying controllers support it.
>
>Set BLK_FEAT_PCI_P2PDMA unconditionally in nvme_mpath_alloc_disk()
>alongside the other features. nvme_update_ns_info_block() already
>calls queue_limits_stack_bdev() to stack each path's limits onto the
>head disk, which routes through blk_stack_limits(). The core now
>clears BLK_FEAT_PCI_P2PDMA automatically if any path (e.g., FC) does
>not support it, consistent with how BLK_FEAT_NOWAIT and BLK_FEAT_POLL
>are handled.
>
>Signed-off-by: Kiran Kumar Modukuri <kmodukuri@nvidia.com>
>Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
>---
> drivers/nvme/host/multipath.c | 2 +-
> 1 file changed, 1 insertion(+), 1 deletion(-)
>
>diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
>index ba00f0b72b85..957e39c4795d 100644
>--- a/drivers/nvme/host/multipath.c
>+++ b/drivers/nvme/host/multipath.c
>@@ -734,7 +734,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
> blk_set_stacking_limits(&lim);
> lim.dma_alignment = 3;
> lim.features |= BLK_FEAT_IO_STAT | BLK_FEAT_NOWAIT |
>- BLK_FEAT_POLL | BLK_FEAT_ATOMIC_WRITES;
>+ BLK_FEAT_POLL | BLK_FEAT_ATOMIC_WRITES | BLK_FEAT_PCI_P2PDMA;
> if (head->ids.csi == NVME_CSI_ZNS)
> lim.features |= BLK_FEAT_ZONED;
Reviewed-by: Nitesh Shetty <nj.shetty@samsung.com>
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply
* Re: [PATCH V3 1/3] block: clear BLK_FEAT_PCI_P2PDMA in blk_stack_limits() for non-supporting devices
From: Nitesh Shetty @ 2026-04-17 10:11 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: song, yukuai, linan122, kbusch, axboe, hch, sagi, linux-raid,
linux-nvme, kmodukuri
In-Reply-To: <20260416212633.72650-2-kch@nvidia.com>
[-- Attachment #1: Type: text/plain, Size: 1635 bytes --]
On 16/04/26 02:26PM, Chaitanya Kulkarni wrote:
>BLK_FEAT_NOWAIT and BLK_FEAT_POLL are cleared in blk_stack_limits()
>when an underlying device does not support them. Apply the same
>treatment to BLK_FEAT_PCI_P2PDMA: stacking drivers set it
>unconditionally and rely on the core to clear it whenever a
>non-supporting member device is stacked.
>
>Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
>---
> block/blk-settings.c | 2 ++
> 1 file changed, 2 insertions(+)
>
>diff --git a/block/blk-settings.c b/block/blk-settings.c
>index 78c83817b9d3..8274631290db 100644
>--- a/block/blk-settings.c
>+++ b/block/blk-settings.c
>@@ -795,6 +795,8 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b,
> t->features &= ~BLK_FEAT_NOWAIT;
> if (!(b->features & BLK_FEAT_POLL))
> t->features &= ~BLK_FEAT_POLL;
>+ if (!(b->features & BLK_FEAT_PCI_P2PDMA))
>+ t->features &= ~BLK_FEAT_PCI_P2PDMA;
>
> t->flags |= (b->flags & BLK_FLAG_MISALIGNED);
I think you need below patch[1] as well to unset this here.
Also I feel better to include Mike,Mikulas and dm-devel mailing list as well.
Thanks,
Nitesh
[1]
diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
index dc2eff6b739d..0442c1f4c686 100644
--- a/drivers/md/dm-table.c
+++ b/drivers/md/dm-table.c
@@ -590,7 +590,8 @@ int dm_split_args(int *argc, char ***argvp, char *input)
static void dm_set_stacking_limits(struct queue_limits *limits)
{
blk_set_stacking_limits(limits);
- limits->features |= BLK_FEAT_IO_STAT | BLK_FEAT_NOWAIT | BLK_FEAT_POLL;
+ limits->features |= BLK_FEAT_IO_STAT | BLK_FEAT_NOWAIT | BLK_FEAT_POLL \
+ BLK_FEAT_PCI_P2PDMA;
}
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply related
* Re: [PATCH] md/raid1,raid10: don't fail devices for invalid IO errors
From: Tomáš Trnka @ 2026-04-17 8:01 UTC (permalink / raw)
To: linux-raid, song, yukuai, Keith Busch; +Cc: linan122, axboe, Keith Busch
In-Reply-To: <20260416140345.3872265-1-kbusch@meta.com>
On Thursday, 16 April 2026 16:03:45, CEST Keith Busch wrote:
> From: Keith Busch <kbusch@kernel.org>
>
> BLK_STS_INVAL indicates the IO request itself was invalid, not that the
> device has failed. When raid1 treats this as a device error, it retries
> on alternate mirrors which fail the same way, eventually exceeding the
> read error threshold and removing the device from the array.
>
> This happens when stacking configurations bypass bio_split_to_limits()
> in the IO path: dm-raid calls md_handle_request() directly without going
> through md_submit_bio(), skipping the alignment validation that would
> otherwise reject invalid bios early. The invalid bio reaches the
> lower block layers, which fail the bio with BLK_STS_INVAL, and raid1
> wrongly interprets this as a device failure.
>
> Add BLK_STS_INVAL to raid1_should_handle_error() so that invalid IO
> errors are propagated back to the caller rather than triggering device
> removal. This is consistent with the previous kernel behavior when
> alignment checks were done earlier in the direct-io path.
>
> Fixes: 5ff3f74e145adc7 ("block: simplify direct io validity check")
> Link: https://lore.kernel.org/linux-block/2982107.4sosBPzcNG@electra/
> Reported-by: Tomáš Trnka <trnka@scm.com>
> Signed-off-by: Keith Busch <kbusch@kernel.org>
Tested-by: Tomáš Trnka <trnka@scm.com>
> ---
> drivers/md/raid1-10.c | 7 ++++++-
> 1 file changed, 6 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c
> index c33099925f230..56a56a4da4f83 100644
> --- a/drivers/md/raid1-10.c
> +++ b/drivers/md/raid1-10.c
> @@ -293,8 +293,13 @@ static inline bool raid1_should_read_first(struct mddev
> *mddev, * bio with REQ_RAHEAD or REQ_NOWAIT can fail at anytime, before
> such IO is * submitted to the underlying disks, hence don't record
> badblocks or retry * in this case.
> + *
> + * BLK_STS_INVAL means the bio was not valid for the underlying device.
> This + * is a user error, not a device failure, so retrying or recording
> bad blocks + * would be wrong.
> */
> static inline bool raid1_should_handle_error(struct bio *bio)
> {
> - return !(bio->bi_opf & (REQ_RAHEAD | REQ_NOWAIT));
> + return !(bio->bi_opf & (REQ_RAHEAD | REQ_NOWAIT)) &&
> + bio->bi_status != BLK_STS_INVAL;
> }
^ permalink raw reply
* Re: [PATCH V3 3/3] nvme-multipath: enable PCI P2PDMA for multipath devices
From: Christoph Hellwig @ 2026-04-17 7:53 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: song, yukuai, linan122, kbusch, axboe, hch, sagi, linux-raid,
linux-nvme, kmodukuri
In-Reply-To: <20260416212633.72650-4-kch@nvidia.com>
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: [PATCH V3 2/3] md: propagate BLK_FEAT_PCI_P2PDMA from member devices to RAID device
From: Christoph Hellwig @ 2026-04-17 7:53 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: song, yukuai, linan122, kbusch, axboe, hch, sagi, linux-raid,
linux-nvme, kmodukuri
In-Reply-To: <20260416212633.72650-3-kch@nvidia.com>
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: [PATCH V3 1/3] block: clear BLK_FEAT_PCI_P2PDMA in blk_stack_limits() for non-supporting devices
From: Christoph Hellwig @ 2026-04-17 7:52 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: song, yukuai, linan122, kbusch, axboe, hch, sagi, linux-raid,
linux-nvme, kmodukuri
In-Reply-To: <20260416212633.72650-2-kch@nvidia.com>
On Thu, Apr 16, 2026 at 02:26:31PM -0700, Chaitanya Kulkarni wrote:
> BLK_FEAT_NOWAIT and BLK_FEAT_POLL are cleared in blk_stack_limits()
> when an underlying device does not support them. Apply the same
> treatment to BLK_FEAT_PCI_P2PDMA: stacking drivers set it
> unconditionally and rely on the core to clear it whenever a
> non-supporting member device is stacked.
>
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
Looks good:
Reviewed-by: Christoph Hellwig <hch@lst.de>
^ permalink raw reply
* Re: [PATCH v2] md: use md_free_cloned_bio() in md_end_clone_io()
From: Li Nan @ 2026-04-17 1:29 UTC (permalink / raw)
To: Abd-Alrhman Masalkhi, Li Nan, song, yukuai; +Cc: linux-raid, linux-kernel
In-Reply-To: <m27bq72w2q.fsf@gmail.com>
在 2026/4/16 15:39, Abd-Alrhman Masalkhi 写道:
> Hi Nan,
>
> thank you for the feedback.
>
> On Thu, Apr 16, 2026 at 14:47 +0800, Li Nan wrote:
>> 在 2026/4/15 16:19, Abd-Alrhman Masalkhi 写道:
>>> md_end_clone_io() and md_free_cloned_bio() share identical teardown
>>> logic. Use md_free_cloned_bio() in md_end_clone_io() for cleanup and
>>> call bio_endio() afterwards.
>>>
>>> Signed-off-by: Abd-Alrhman Masalkhi <abd.masalkhi@gmail.com>
>>> ---
>>> Changes in v2:
>>> - Reuse md_free_cloned_bio() instead of introducing a new helper
>>> - Link to v1: https://lore.kernel.org/linux-raid/20260414103813.307601-1-abd.masalkhi@gmail.com
>>> ---
>>> drivers/md/md.c | 13 +------------
>>> 1 file changed, 1 insertion(+), 12 deletions(-)
>>>
>>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>>> index ac71640ff3a8..8565566a447b 100644
>>> --- a/drivers/md/md.c
>>> +++ b/drivers/md/md.c
>>> @@ -9212,20 +9212,9 @@ static void md_end_clone_io(struct bio *bio)
>>> {
>>> struct md_io_clone *md_io_clone = bio->bi_private;
>>> struct bio *orig_bio = md_io_clone->orig_bio;
>>> - struct mddev *mddev = md_io_clone->mddev;
>>> -
>>> - if (bio_data_dir(orig_bio) == WRITE && md_bitmap_enabled(mddev, false))
>>> - md_bitmap_end(mddev, md_io_clone);
>>> -
>>> - if (bio->bi_status && !orig_bio->bi_status)
>>> - orig_bio->bi_status = bio->bi_status;
>>> -
>>> - if (md_io_clone->start_time)
>>> - bio_end_io_acct(orig_bio, md_io_clone->start_time);
>>>
>>> - bio_put(bio);
>>> + md_free_cloned_bio(bio);
>>> bio_endio(orig_bio);
>>
>> Is it safe to end orig_bio after putting active_io?
>
> As I understand it, active_io is primarily used to account for
> in-flight bios, allowing the array to be safely suspended once no
> bios are outstanding.
>
> I have made a diagram, while i am reviewing the md driver:
>
> mddev->active_io
>
> md driver:
> md_handle_request()
> mddev->active_io +1
> pers->make_request()
> md_account_bio() +1
> if (err)
> md_free_cloned_bio -1
> mddev->active_io -1
>
> lower driver:
> bio_endio()
> md_end_clone_io -1
>
> As shown, it incremented in two places, the first is before passing
> orig_bio to pers->make_request(), and decremented after it returns
> (regardless of what make_request() does with it). and the second
> is when cloning the original bio, and it should be decremented after
> the cloned bio is released.
>
> In this context, it should not matter whether bio_endio(orig_bio) is
> called before or after decrementing active_io, as long as the cloned
> bio teardown is completed correctly.
>
> Am I missing anything that would make this unsafe?
>
Thanks for your reply, LGTM.
Reviewed-by: Li Nan <linan122@huawei.com>
--
Thanks,
Nan
^ permalink raw reply
* [PATCH V3 3/3] nvme-multipath: enable PCI P2PDMA for multipath devices
From: Chaitanya Kulkarni @ 2026-04-16 21:26 UTC (permalink / raw)
To: song, yukuai, linan122, kbusch, axboe, hch, sagi
Cc: linux-raid, linux-nvme, kmodukuri, Chaitanya Kulkarni
In-Reply-To: <20260416212633.72650-1-kch@nvidia.com>
From: Kiran Kumar Modukuri <kmodukuri@nvidia.com>
NVMe multipath does not expose BLK_FEAT_PCI_P2PDMA on the head disk
even when all underlying controllers support it.
Set BLK_FEAT_PCI_P2PDMA unconditionally in nvme_mpath_alloc_disk()
alongside the other features. nvme_update_ns_info_block() already
calls queue_limits_stack_bdev() to stack each path's limits onto the
head disk, which routes through blk_stack_limits(). The core now
clears BLK_FEAT_PCI_P2PDMA automatically if any path (e.g., FC) does
not support it, consistent with how BLK_FEAT_NOWAIT and BLK_FEAT_POLL
are handled.
Signed-off-by: Kiran Kumar Modukuri <kmodukuri@nvidia.com>
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/nvme/host/multipath.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index ba00f0b72b85..957e39c4795d 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -734,7 +734,7 @@ int nvme_mpath_alloc_disk(struct nvme_ctrl *ctrl, struct nvme_ns_head *head)
blk_set_stacking_limits(&lim);
lim.dma_alignment = 3;
lim.features |= BLK_FEAT_IO_STAT | BLK_FEAT_NOWAIT |
- BLK_FEAT_POLL | BLK_FEAT_ATOMIC_WRITES;
+ BLK_FEAT_POLL | BLK_FEAT_ATOMIC_WRITES | BLK_FEAT_PCI_P2PDMA;
if (head->ids.csi == NVME_CSI_ZNS)
lim.features |= BLK_FEAT_ZONED;
--
2.39.5
^ permalink raw reply related
* [PATCH V3 2/3] md: propagate BLK_FEAT_PCI_P2PDMA from member devices to RAID device
From: Chaitanya Kulkarni @ 2026-04-16 21:26 UTC (permalink / raw)
To: song, yukuai, linan122, kbusch, axboe, hch, sagi
Cc: linux-raid, linux-nvme, kmodukuri, Chaitanya Kulkarni
In-Reply-To: <20260416212633.72650-1-kch@nvidia.com>
From: Kiran Kumar Modukuri <kmodukuri@nvidia.com>
MD RAID does not propagate BLK_FEAT_PCI_P2PDMA from member devices to
the RAID device, preventing peer-to-peer DMA through the RAID layer even
when all underlying devices support it.
Enable BLK_FEAT_PCI_P2PDMA unconditionally in raid0, raid1 and raid10
personalities during queue limits setup. blk_stack_limits() clears it
automatically if any member device lacks support, consistent with how
BLK_FEAT_NOWAIT and BLK_FEAT_POLL are handled in the block core.
Parity RAID personalities (raid4/5/6) are excluded because they require
CPU access to data pages for parity computation, which is incompatible
with P2P mappings.
Tested with RAID0/1/10 arrays containing multiple NVMe devices with
P2PDMA support, confirming that peer-to-peer transfers work correctly
through the RAID layer.
Signed-off-by: Kiran Kumar Modukuri <kmodukuri@nvidia.com>
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
drivers/md/raid0.c | 1 +
drivers/md/raid1.c | 1 +
drivers/md/raid10.c | 1 +
3 files changed, 3 insertions(+)
diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
index 5e38a51e349a..2cdaf7495d92 100644
--- a/drivers/md/raid0.c
+++ b/drivers/md/raid0.c
@@ -392,6 +392,7 @@ static int raid0_set_limits(struct mddev *mddev)
lim.io_opt = lim.io_min * mddev->raid_disks;
lim.chunk_sectors = mddev->chunk_sectors;
lim.features |= BLK_FEAT_ATOMIC_WRITES;
+ lim.features |= BLK_FEAT_PCI_P2PDMA;
err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY);
if (err)
return err;
diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
index ba91f7e61920..422ad4786569 100644
--- a/drivers/md/raid1.c
+++ b/drivers/md/raid1.c
@@ -3215,6 +3215,7 @@ static int raid1_set_limits(struct mddev *mddev)
lim.max_hw_wzeroes_unmap_sectors = 0;
lim.logical_block_size = mddev->logical_block_size;
lim.features |= BLK_FEAT_ATOMIC_WRITES;
+ lim.features |= BLK_FEAT_PCI_P2PDMA;
err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY);
if (err)
return err;
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 4901ebe45c87..07a5b734c8f3 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -3939,6 +3939,7 @@ static int raid10_set_queue_limits(struct mddev *mddev)
lim.chunk_sectors = mddev->chunk_sectors;
lim.io_opt = lim.io_min * raid10_nr_stripes(conf);
lim.features |= BLK_FEAT_ATOMIC_WRITES;
+ lim.features |= BLK_FEAT_PCI_P2PDMA;
err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY);
if (err)
return err;
--
2.39.5
^ permalink raw reply related
* [PATCH V3 1/3] block: clear BLK_FEAT_PCI_P2PDMA in blk_stack_limits() for non-supporting devices
From: Chaitanya Kulkarni @ 2026-04-16 21:26 UTC (permalink / raw)
To: song, yukuai, linan122, kbusch, axboe, hch, sagi
Cc: linux-raid, linux-nvme, kmodukuri, Chaitanya Kulkarni
In-Reply-To: <20260416212633.72650-1-kch@nvidia.com>
BLK_FEAT_NOWAIT and BLK_FEAT_POLL are cleared in blk_stack_limits()
when an underlying device does not support them. Apply the same
treatment to BLK_FEAT_PCI_P2PDMA: stacking drivers set it
unconditionally and rely on the core to clear it whenever a
non-supporting member device is stacked.
Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
---
block/blk-settings.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/block/blk-settings.c b/block/blk-settings.c
index 78c83817b9d3..8274631290db 100644
--- a/block/blk-settings.c
+++ b/block/blk-settings.c
@@ -795,6 +795,8 @@ int blk_stack_limits(struct queue_limits *t, struct queue_limits *b,
t->features &= ~BLK_FEAT_NOWAIT;
if (!(b->features & BLK_FEAT_POLL))
t->features &= ~BLK_FEAT_POLL;
+ if (!(b->features & BLK_FEAT_PCI_P2PDMA))
+ t->features &= ~BLK_FEAT_PCI_P2PDMA;
t->flags |= (b->flags & BLK_FLAG_MISALIGNED);
--
2.39.5
^ permalink raw reply related
* [PATCH V3 0/2] md/nvme: Enable PCI P2PDMA support for RAID0 and NVMe Multipath
From: Chaitanya Kulkarni @ 2026-04-16 21:26 UTC (permalink / raw)
To: song, yukuai, linan122, kbusch, axboe, hch, sagi
Cc: linux-raid, linux-nvme, kmodukuri, Chaitanya Kulkarni
Hi,
This patch series extends PCI peer-to-peer DMA (P2PDMA) support to enable
direct data transfers between PCIe devices through RAID and NVMe multipath
block layers.
Current Linux kernel P2PDMA infrastructure supports direct peer-to-peer
transfers, but this support is not propagated through certain storage
stacks like MD RAID and NVMe multipath. This adds two patches for
MD RAID 0/1/10 and NVMe to propogate P2PDMA support through the
storage stack.
All four test scenarios demonstrate that P2PDMA capabilities are correctly
propagated through both the MD RAID layer (patch 1/2) and NVMe multipath
layer (patch 2/2). Direct peer-to-peer transfers complete successfully with
full data integrity verification, confirming that:
1. RAID devices properly inherit P2PDMA capability from member devices
2. NVMe multipath devices correctly expose P2PDMA support
3. P2P memory buffers can be used for transfers involving both types
4. Data integrity is maintained across all transfer combinations
I've added the patch specific tests and blktest log as well at the end.
Repo:-
git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git
Branch HEAD:-
commit 88a57e15861997dd6fa98154ad087f7831bbead1 (origin/for-next)
Merge: 81a0a2e4e535 36446de0c30c
Author: Jens Axboe <axboe@kernel.dk>
Date: Fri Apr 10 07:02:42 2026 -0600
Merge branch 'for-7.1/block' into for-next
* for-7.1/block:
ublk: fix tautological comparison warning in ublk_ctrl_reg_buf
-ck
Changes from V2:-
1. Unconditionally set the BLK_FEAT_PCI_P2PDMA for md and nvme multipath.
(Christoph)
2. Add a prep patch to diable BLK_FEAT_PCI_P2PDMA in the blk_stack_limit().
(christoph)
Changes from V1:-
- Update patch 1 to explicitly support MD RAID 0/1/10.
- Fix signoff chain order for patch 2.
- Clear BLK_FEAT_PCI_P2PDMA in nvme_mpath_add_disk() when a newly
added path does not support it, to handle multipath across different
transports.
- Add nvme multipath test log for mixed transport TCP and PCIe.
Chaitanya Kulkarni (1):
block: clear BLK_FEAT_PCI_P2PDMA in blk_stack_limits() for
non-supporting devices
Kiran Kumar Modukuri (2):
md: propagate BLK_FEAT_PCI_P2PDMA from member devices to RAID device
nvme-multipath: enable PCI P2PDMA for multipath devices
block/blk-settings.c | 2 ++
drivers/md/raid0.c | 1 +
drivers/md/raid1.c | 1 +
drivers/md/raid10.c | 1 +
drivers/nvme/host/multipath.c | 2 +-
5 files changed, 6 insertions(+), 1 deletion(-)
========================================================
* MD RAID Personalities and NVMe testing :-
========================================================
================================================================================
P2PDMA Comprehensive Test Report
================================================================================
Date: Thu Apr 16, 2026 19:00:32 UTC
Patch Series Under Test:
1/3 blk: clear BLK_FEAT_PCI_P2PDMA in blk_stack_limits
2/3 nvme-multipath: expose BLK_FEAT_PCI_P2PDMA on head disk
3/3 md: raid0/1/10: expose BLK_FEAT_PCI_P2PDMA on array disk
================================================================================
1. System Information
================================================================================
Kernel:
Linux vm70 7.0.0-rc2-p2pdma-v2+ #19 SMP PREEMPT Thu Apr 16 18:01:55 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
Kernel patches (git log above baseline):
5bf19d9 md: raid0/1/10: expose BLK_FEAT_PCI_P2PDMA on array disk
02dc9a6 nvme-multipath: expose BLK_FEAT_PCI_P2PDMA on head disk
ba22b62 blk: clear BLK_FEAT_PCI_P2PDMA in blk_stack_limits
NVMe Modules:
nvme_fabrics 24576 0
MD Modules loaded:
(none yet -- will be loaded on demand)
--------------------------------------------------------------------------------
1.1 NVMe Device Inventory (nvme list -v)
--------------------------------------------------------------------------------
Subsystem Subsystem-NQN Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys1 nqn.2019-08.org.qemu:nqn.2019-08.org.qemu:shared-ns nvme0, nvme1
nvme-subsys2 nqn.2019-08.org.qemu:nvme3 nvme2
nvme-subsys3 nqn.2019-08.org.qemu:nvme4 nvme3
nvme-subsys4 nqn.2019-08.org.qemu:nvme5 nvme4
nvme-subsys5 nqn.2019-08.org.qemu:nvme6 nvme5
Device SN MN FR TxPort Address Slot Subsystem Namespaces
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ----------------
nvme0 shared2 QEMU NVMe Ctrl 1.0 pcie 0000:0a:00.0 nvme-subsys1 nvme1n1
nvme1 shared2 QEMU NVMe Ctrl 1.0 pcie 0000:0b:00.0 nvme-subsys1 nvme1n1
nvme2 nvme3 QEMU NVMe Ctrl 1.0 pcie 0000:0c:00.0 nvme-subsys2 nvme2n1
nvme3 nvme4 QEMU NVMe Ctrl 1.0 pcie 0000:0d:00.0 nvme-subsys3 nvme3n1
nvme4 nvme5 QEMU NVMe Ctrl 1.0 pcie 0000:0e:00.0 nvme-subsys4 nvme4n1
nvme5 nvme6 QEMU NVMe Ctrl 1.0 pcie 0000:10:00.0 nvme-subsys5 nvme5n1
Device Generic NSID Usage Format Controllers
------------ ------------ ---------- -------------------------- ---------------- ----------------
/dev/nvme1n1 /dev/ng1n1 0x1 10.74 GB / 10.74 GB 512 B + 0 B nvme0, nvme1
/dev/nvme2n1 /dev/ng2n1 0x1 10.74 GB / 10.74 GB 512 B + 0 B nvme2
/dev/nvme3n1 /dev/ng3n1 0x1 10.74 GB / 10.74 GB 512 B + 0 B nvme3
/dev/nvme4n1 /dev/ng4n1 0x1 10.74 GB / 10.74 GB 512 B + 0 B nvme4
/dev/nvme5n1 /dev/ng5n1 0x1 10.74 GB / 10.74 GB 512 B + 0 B nvme5
--------------------------------------------------------------------------------
1.2 Shared Namespace Configuration (nvme-subsys1)
--------------------------------------------------------------------------------
nvme-subsys1 - NQN=nqn.2019-08.org.qemu:nqn.2019-08.org.qemu:shared-ns
hostnqn=nqn.2014-08.org.nvmexpress:uuid:148a9e69-3f22-420f-afea-bfd5d4b77f36
iopolicy=numa
\
+- nvme0 pcie 0000:0a:00.0 live optimized
+- nvme1 pcie 0000:0b:00.0 live optimized
--------------------------------------------------------------------------------
1.3 PCI P2PDMA / CMB Configuration
--------------------------------------------------------------------------------
CMB-enabled NVMe controllers:
0000:0a:00.0 (nvme0) - CMB 64 MB
0000:0b:00.0 (nvme1) - CMB 64 MB
0000:0c:00.0 (nvme2) - CMB 64 MB
dmesg (P2P):
[ 7.283954] nvme 0000:0a:00.0: added peer-to-peer DMA memory 0x1808000000-0x180bffffff
[ 7.288711] nvme 0000:0c:00.0: added peer-to-peer DMA memory 0x1800000000-0x1803ffffff
[ 7.293117] nvme 0000:0b:00.0: added peer-to-peer DMA memory 0x1804000000-0x1807ffffff
--------------------------------------------------------------------------------
1.4 Standalone NVMe Devices (for RAID tests)
--------------------------------------------------------------------------------
/dev/nvme2n1 ( 10G)
/dev/nvme3n1 ( 10G)
/dev/nvme4n1 ( 10G)
/dev/nvme5n1 ( 10G)
P2PMEM for multipath tests: /sys/bus/pci/devices/0000:0a:00.0/p2pmem/allocate
P2PMEM for RAID tests: /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate
================================================================================
2. Test 1: NVMe Multipath P2PDMA (Patch 2/3)
================================================================================
Objective:
Verify BLK_FEAT_PCI_P2PDMA is set on the multipath head when all paths
support P2PDMA (PCIe-only), and cleared when a non-P2P path (TCP) is added.
Clearing is handled by blk_stack_limits() in the block core (Patch 1/3).
Test tool: /home/lab/p2pmem-test/p2pmem-test
P2PMEM buffer: /sys/bus/pci/devices/0000:0a:00.0/p2pmem/allocate
Target device: /dev/nvme1n1 (multipath head, nvme-subsys1)
--------------------------------------------------------------------------------
2.1 Test 1a: P2PDMA with PCIe-only Multipath Paths (Expect PASS)
--------------------------------------------------------------------------------
Paths before test:
\
+- nvme0 pcie 0000:0a:00.0 live optimized
+- nvme1 pcie 0000:0b:00.0 live optimized
All paths are PCIe with CMB -> P2PDMA supported.
Patch 2/3 sets BLK_FEAT_PCI_P2PDMA unconditionally in nvme_mpath_alloc_disk().
Command:
/home/lab/p2pmem-test/p2pmem-test /dev/nvme1n1 /dev/nvme1n1 /sys/bus/pci/devices/0000:0a:00.0/p2pmem/allocate -c 1 -s 4k --check
Output:
Running p2pmem-test: reading /dev/nvme1n1 (10.74GB): writing /dev/nvme1n1 (10.74GB): p2pmem buffer /sys/bus/pci/devices/0000:0a:00.0/p2pmem/allocate.
chunk size = 4096 : number of chunks = 1: total = 4.096kB : thread(s) = 1 : overlap = OFF.
skip-read = OFF : skip-write = OFF : duration = INF sec.
buffer = 0x7f30b32dc000 (p2pmem): mmap = 4.096kB
PAGE_SIZE = 4096B
checking data with seed = 1776366032
MATCH on data check, 0x23039cdb = 0x23039cdb.
Transfer:
4.10kB in 860.0 us 4.76MB/s
Exit code: 0
Result: PASS
P2PDMA transfer succeeded with data verification.
--------------------------------------------------------------------------------
2.2 Test 1b: Add NVMe-oF TCP Path, Then Test P2PDMA (Expect FAIL)
--------------------------------------------------------------------------------
Setting up NVMe-oF TCP target (nvmet) on loopback...
Subsystem NQN: nqn.2019-08.org.qemu:nqn.2019-08.org.qemu:shared-ns
Namespace 1: backed by /dev/nvme1n1
Device UUID: 00000000-0000-0000-0000-000000000000 (matches QEMU quirk)
Transport: TCP, 127.0.0.1:4420
CNTLID min: 10
Paths after TCP connection:
\
+- nvme0 pcie 0000:0a:00.0 live optimized
+- nvme1 pcie 0000:0b:00.0 live optimized
+- nvme6 tcp traddr=127.0.0.1,trsvcid=4420,src_addr=127.0.0.1 live optimized
NVMe device inventory (nvme list -v) confirming TCP path in shared namespace:
Subsystem Subsystem-NQN Controllers
---------------- ------------------------------------------------------------------------------------------------ ----------------
nvme-subsys1 nqn.2019-08.org.qemu:nqn.2019-08.org.qemu:shared-ns nvme0, nvme1, nvme6
nvme-subsys2 nqn.2019-08.org.qemu:nvme3 nvme2
nvme-subsys3 nqn.2019-08.org.qemu:nvme4 nvme3
nvme-subsys4 nqn.2019-08.org.qemu:nvme5 nvme4
nvme-subsys5 nqn.2019-08.org.qemu:nvme6 nvme5
Device SN MN FR TxPort Address Slot Subsystem Namespaces
-------- -------------------- ---------------------------------------- -------- ------ -------------- ------ ------------ ----------------
nvme0 shared2 QEMU NVMe Ctrl 7.0.0-rc pcie 0000:0a:00.0 nvme-subsys1 nvme1n1
nvme1 shared2 QEMU NVMe Ctrl 7.0.0-rc pcie 0000:0b:00.0 nvme-subsys1 nvme1n1
nvme2 nvme3 QEMU NVMe Ctrl 1.0 pcie 0000:0c:00.0 nvme-subsys2 nvme2n1
nvme3 nvme4 QEMU NVMe Ctrl 1.0 pcie 0000:0d:00.0 nvme-subsys3 nvme3n1
nvme4 nvme5 QEMU NVMe Ctrl 1.0 pcie 0000:0e:00.0 nvme-subsys4 nvme4n1
nvme5 nvme6 QEMU NVMe Ctrl 1.0 pcie 0000:10:00.0 nvme-subsys5 nvme5n1
nvme6 shared2 QEMU NVMe Ctrl 7.0.0-rc tcp traddr=127.0.0.1,trsvcid=4420,src_addr=127.0.0.1 nvme-subsys1 nvme1n1
Device Generic NSID Usage Format Controllers
------------ ------------ ---------- -------------------------- ---------------- ----------------
/dev/nvme1n1 /dev/ng1n1 0x1 10.74 GB / 10.74 GB 512 B + 0 B nvme0, nvme1, nvme6
/dev/nvme2n1 /dev/ng2n1 0x1 10.74 GB / 10.74 GB 512 B + 0 B nvme2
/dev/nvme3n1 /dev/ng3n1 0x1 10.74 GB / 10.74 GB 512 B + 0 B nvme3
/dev/nvme4n1 /dev/ng4n1 0x1 10.74 GB / 10.74 GB 512 B + 0 B nvme4
/dev/nvme5n1 /dev/ng5n1 0x1 10.74 GB / 10.74 GB 512 B + 0 B nvme5
TCP path lacks PCI P2PDMA. Patch 1/3 causes blk_stack_limits() to clear
BLK_FEAT_PCI_P2PDMA when the TCP path's limits are stacked onto the head.
Command:
/home/lab/p2pmem-test/p2pmem-test /dev/nvme1n1 /dev/nvme1n1 /sys/bus/pci/devices/0000:0a:00.0/p2pmem/allocate -c 1 -s 4k --check
Output:
pread: Remote I/O error
Running p2pmem-test: reading /dev/nvme1n1 (10.74GB): writing /dev/nvme1n1 (10.74GB): p2pmem buffer /sys/bus/pci/devices/0000:0a:00.0/p2pmem/allocate.
chunk size = 4096 : number of chunks = 1: total = 4.096kB : thread(s) = 1 : overlap = OFF.
skip-read = OFF : skip-write = OFF : duration = INF sec.
buffer = 0x7f9851d3d000 (p2pmem): mmap = 4.096kB
PAGE_SIZE = 4096B
checking data with seed = 1776366033
Exit code: 1
Result: PASS (expected failure)
P2PDMA transfer correctly rejected -- BLK_FEAT_PCI_P2PDMA was
cleared because a non-P2P-capable component is present.
Cleaning up TCP path...
TCP path removed.
================================================================================
3. Test 2: MD RAID0 P2PDMA (Patch 3/3)
================================================================================
Objective:
Verify BLK_FEAT_PCI_P2PDMA propagates through RAID0. Patch 3/3 sets
the flag in raid0_set_limits(); blk_stack_limits() preserves it when
all member devices support P2PDMA.
Members: /dev/nvme2n1 /dev/nvme3n1
--------------------------------------------------------------------------------
3.1 Test 2a: P2PDMA on RAID0 Array (Expect PASS)
--------------------------------------------------------------------------------
mdadm create output:
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/p2p-test started.
RAID0 device: /dev/md127
Array detail (mdadm --detail):
/dev/md127:
Version : 1.2
Creation Time : Thu Apr 16 19:00:35 2026
Raid Level : raid0
Array Size : 20953088 (19.98 GiB 21.46 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Thu Apr 16 19:00:35 2026
State : clean
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Layout : original
Chunk Size : 512K
Consistency Policy : none
Name : vm70:p2p-test (local to host vm70)
UUID : 84f669e0:6b24971e:680cdf64:ca4087e0
Events : 0
Number Major Minor RaidDevice State
0 259 4 0 active sync /dev/nvme2n1
1 259 5 1 active sync /dev/nvme3n1
/proc/mdstat:
Personalities : [raid0] [raid1] [raid4] [raid5] [raid6] [raid10]
md127 : active raid0 nvme3n1[1] nvme2n1[0]
20953088 blocks super 1.2 512k chunks
unused devices: <none>
P2PMEM buffer: /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate
Command:
/home/lab/p2pmem-test/p2pmem-test /dev/md127 /dev/md127 /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate -c 1 -s 4k --check
Output:
Running p2pmem-test: reading /dev/md127 (21.46GB): writing /dev/md127 (21.46GB): p2pmem buffer /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate.
chunk size = 4096 : number of chunks = 1: total = 4.096kB : thread(s) = 1 : overlap = OFF.
skip-read = OFF : skip-write = OFF : duration = INF sec.
buffer = 0x7f87e5cd4000 (p2pmem): mmap = 4.096kB
PAGE_SIZE = 4096B
checking data with seed = 1776366036
MATCH on data check, 0x3db0cfbb = 0x3db0cfbb.
Transfer:
4.10kB in 747.0 us 5.48MB/s
Exit code: 0
Result: PASS
P2PDMA transfer succeeded with data verification.
RAID0 array stopped.
================================================================================
4. Test 3: MD RAID1 P2PDMA (Patch 3/3)
================================================================================
Objective:
Verify BLK_FEAT_PCI_P2PDMA propagates through RAID1. Patch 3/3 sets
the flag in raid1_set_limits(); blk_stack_limits() preserves it when
all member devices support P2PDMA.
Members: /dev/nvme2n1 /dev/nvme3n1
--------------------------------------------------------------------------------
4.1 Test 3a: P2PDMA on RAID1 Array (Expect PASS)
--------------------------------------------------------------------------------
mdadm create output:
mdadm: Note: this array has metadata at the start and
may not be suitable as a boot device. If you plan to
store '/boot' on this device please ensure that
your boot-loader understands md/v1.x metadata, or use
--metadata=0.90
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/p2p-test started.
RAID1 device: /dev/md127
Array detail (mdadm --detail):
/dev/md127:
Version : 1.2
Creation Time : Thu Apr 16 19:00:38 2026
Raid Level : raid1
Array Size : 10476544 (9.99 GiB 10.73 GB)
Used Dev Size : 10476544 (9.99 GiB 10.73 GB)
Raid Devices : 2
Total Devices : 2
Persistence : Superblock is persistent
Update Time : Thu Apr 16 19:00:38 2026
State : clean, resyncing
Active Devices : 2
Working Devices : 2
Failed Devices : 0
Spare Devices : 0
Consistency Policy : resync
Resync Status : 2% complete
Name : vm70:p2p-test (local to host vm70)
UUID : 7917cc10:d47660cb:46454ecd:ccc8f946
Events : 0
Number Major Minor RaidDevice State
0 259 4 0 active sync /dev/nvme2n1
1 259 5 1 active sync /dev/nvme3n1
/proc/mdstat:
Personalities : [raid0] [raid1] [raid4] [raid5] [raid6] [raid10]
md127 : active raid1 nvme3n1[1] nvme2n1[0]
10476544 blocks super 1.2 [2/2] [UU]
[>....................] resync = 2.0% (215936/10476544) finish=0.7min speed=215936K/sec
unused devices: <none>
P2PMEM buffer: /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate
Command:
/home/lab/p2pmem-test/p2pmem-test /dev/md127 /dev/md127 /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate -c 1 -s 4k --check
Output:
Running p2pmem-test: reading /dev/md127 (10.73GB): writing /dev/md127 (10.73GB): p2pmem buffer /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate.
chunk size = 4096 : number of chunks = 1: total = 4.096kB : thread(s) = 1 : overlap = OFF.
skip-read = OFF : skip-write = OFF : duration = INF sec.
buffer = 0x7f0947c68000 (p2pmem): mmap = 4.096kB
PAGE_SIZE = 4096B
checking data with seed = 1776366040
MATCH on data check, 0x1b7cad6e = 0x1b7cad6e.
Transfer:
4.10kB in 2.8 ms 1.44MB/s
Exit code: 0
Result: PASS
P2PDMA transfer succeeded with data verification.
RAID1 array stopped.
================================================================================
5. Test 4: MD RAID10 P2PDMA (Patch 3/3)
================================================================================
Objective:
Verify BLK_FEAT_PCI_P2PDMA propagates through RAID10. Patch 3/3 sets
the flag in raid10_set_limits(); blk_stack_limits() preserves it when
all member devices support P2PDMA.
Members: /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1 /dev/nvme5n1
--------------------------------------------------------------------------------
5.1 Test 4a: P2PDMA on RAID10 Array (Expect PASS)
--------------------------------------------------------------------------------
mdadm create output:
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/p2p-test started.
RAID10 device: /dev/md127
Array detail (mdadm --detail):
/dev/md127:
Version : 1.2
Creation Time : Thu Apr 16 19:00:43 2026
Raid Level : raid10
Array Size : 20953088 (19.98 GiB 21.46 GB)
Used Dev Size : 10476544 (9.99 GiB 10.73 GB)
Raid Devices : 4
Total Devices : 4
Persistence : Superblock is persistent
Update Time : Thu Apr 16 19:00:43 2026
State : clean, resyncing
Active Devices : 4
Working Devices : 4
Failed Devices : 0
Spare Devices : 0
Layout : near=2
Chunk Size : 512K
Consistency Policy : resync
Resync Status : 1% complete
Name : vm70:p2p-test (local to host vm70)
UUID : 0a6b9ded:3f38328a:61352e9c:6f5f7853
Events : 0
Number Major Minor RaidDevice State
0 259 4 0 active sync set-A /dev/nvme2n1
1 259 5 1 active sync set-B /dev/nvme3n1
2 259 3 2 active sync set-A /dev/nvme4n1
3 259 6 3 active sync set-B /dev/nvme5n1
/proc/mdstat:
Personalities : [raid0] [raid1] [raid4] [raid5] [raid6] [raid10]
md127 : active raid10 nvme5n1[3] nvme4n1[2] nvme3n1[1] nvme2n1[0]
20953088 blocks super 1.2 512K chunks 2 near-copies [4/4] [UUUU]
[>....................] resync = 1.8% (387200/20953088) finish=0.8min speed=387200K/sec
unused devices: <none>
P2PMEM buffer: /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate
Command:
/home/lab/p2pmem-test/p2pmem-test /dev/md127 /dev/md127 /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate -c 1 -s 4k --check
Output:
Running p2pmem-test: reading /dev/md127 (21.46GB): writing /dev/md127 (21.46GB): p2pmem buffer /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate.
chunk size = 4096 : number of chunks = 1: total = 4.096kB : thread(s) = 1 : overlap = OFF.
skip-read = OFF : skip-write = OFF : duration = INF sec.
buffer = 0x7f2061b9a000 (p2pmem): mmap = 4.096kB
PAGE_SIZE = 4096B
checking data with seed = 1776366044
MATCH on data check, 0x744f0811 = 0x744f0811.
Transfer:
4.10kB in 484.9 us 8.45MB/s
Exit code: 0
Result: PASS
P2PDMA transfer succeeded with data verification.
RAID10 array stopped.
================================================================================
6. Test 5: MD RAID4 P2PDMA -- Negative Test
================================================================================
Objective:
Verify that P2PDMA does NOT work on RAID4. Parity RAID levels (4/5/6)
require CPU access to data pages for XOR/parity computation, which is
incompatible with P2P mappings. Patch 3/3 intentionally does NOT add
BLK_FEAT_PCI_P2PDMA to raid456 personalities.
Members: /dev/nvme2n1 /dev/nvme3n1 /dev/nvme4n1
--------------------------------------------------------------------------------
6.1 Test 5a: P2PDMA on RAID4 Array (Expect FAIL)
--------------------------------------------------------------------------------
mdadm create output:
mdadm: Defaulting to version 1.2 metadata
mdadm: array /dev/md/p2p-test started.
RAID4 device: /dev/md127
Array detail (mdadm --detail):
/dev/md127:
Version : 1.2
Creation Time : Thu Apr 16 19:00:47 2026
Raid Level : raid4
Array Size : 20953088 (19.98 GiB 21.46 GB)
Used Dev Size : 10476544 (9.99 GiB 10.73 GB)
Raid Devices : 3
Total Devices : 3
Persistence : Superblock is persistent
Update Time : Thu Apr 16 19:00:47 2026
State : clean, resyncing
Active Devices : 3
Working Devices : 3
Failed Devices : 0
Spare Devices : 0
Chunk Size : 512K
Consistency Policy : resync
Resync Status : 1% complete
Name : vm70:p2p-test (local to host vm70)
UUID : b9fb0c5f:d6471fd5:4704f465:88bd6425
Events : 0
Number Major Minor RaidDevice State
0 259 4 0 active sync /dev/nvme2n1
1 259 5 1 active sync /dev/nvme3n1
2 259 3 2 active sync /dev/nvme4n1
/proc/mdstat:
Personalities : [raid0] [raid1] [raid4] [raid5] [raid6] [raid10]
md127 : active raid4 nvme4n1[2] nvme3n1[1] nvme2n1[0]
20953088 blocks super 1.2 level 4, 512k chunk, algorithm 0 [3/3] [UUU]
[>....................] resync = 1.6% (172572/10476544) finish=0.9min speed=172572K/sec
unused devices: <none>
P2PMEM buffer: /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate
Command:
/home/lab/p2pmem-test/p2pmem-test /dev/md127 /dev/md127 /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate -c 1 -s 4k --check
Output:
pread: Remote I/O error
Running p2pmem-test: reading /dev/md127 (21.46GB): writing /dev/md127 (21.46GB): p2pmem buffer /sys/bus/pci/devices/0000:0c:00.0/p2pmem/allocate.
chunk size = 4096 : number of chunks = 1: total = 4.096kB : thread(s) = 1 : overlap = OFF.
skip-read = OFF : skip-write = OFF : duration = INF sec.
buffer = 0x7f4f647b8000 (p2pmem): mmap = 4.096kB
PAGE_SIZE = 4096B
checking data with seed = 1776366048
Exit code: 1
Result: PASS (expected failure)
P2PDMA transfer correctly rejected -- BLK_FEAT_PCI_P2PDMA was
cleared because a non-P2P-capable component is present.
RAID4 array stopped.
================================================================================
7. Test Summary
================================================================================
Test Description Expected Actual Result
------ ----------------------------------------------- --------- ------- ------
1a NVMe multipath P2PDMA (PCIe-only paths) PASS PASS OK
1b NVMe multipath P2PDMA (PCIe + TCP paths) FAIL FAIL OK
2a MD RAID0 P2PDMA PASS PASS OK
3a MD RAID1 P2PDMA PASS PASS OK
4a MD RAID10 P2PDMA PASS PASS OK
5a MD RAID4 P2PDMA (negative test) FAIL FAIL OK
Totals: 6 tests, 6 passed, 0 failed
All tests PASSED.
================================================================================
========================================================
* BLKTEST Testing :- nvme, block, and md category
========================================================
blktests (master) # ./test-nvme.sh
++ for t in loop tcp
++ echo '################NVMET_TRTYPES=loop############'
################NVMET_TRTYPES=loop############
++ NVME_IMG_SIZE=1G
++ NVME_NUM_ITER=1
++ NVMET_TRTYPES=loop
++ ./check nvme
nvme/002 (tr=loop) (create many subsystems and test discovery) [passed]
runtime 36.834s ... 35.055s
nvme/003 (tr=loop) (test if we're sending keep-alives to a discovery controller) [passed]
runtime 10.240s ... 10.233s
nvme/004 (tr=loop) (test nvme and nvmet UUID NS descriptors) [passed]
runtime 0.684s ... 0.656s
nvme/005 (tr=loop) (reset local loopback target) [passed]
runtime 0.981s ... 0.970s
nvme/006 (tr=loop bd=device) (create an NVMeOF target) [passed]
runtime 0.091s ... 0.097s
nvme/006 (tr=loop bd=file) (create an NVMeOF target) [passed]
runtime 0.086s ... 0.083s
nvme/008 (tr=loop bd=device) (create an NVMeOF host) [passed]
runtime 0.645s ... 0.631s
nvme/008 (tr=loop bd=file) (create an NVMeOF host) [passed]
runtime 0.655s ... 0.632s
nvme/010 (tr=loop bd=device) (run data verification fio job) [passed]
runtime 9.238s ... 9.716s
nvme/010 (tr=loop bd=file) (run data verification fio job) [passed]
runtime 47.943s ... 39.701s
nvme/012 (tr=loop bd=device) (run mkfs and data verification fio) [passed]
runtime 52.603s ... 47.570s
nvme/012 (tr=loop bd=file) (run mkfs and data verification fio) [passed]
runtime 41.337s ... 38.148s
nvme/014 (tr=loop bd=device) (flush a command from host) [passed]
runtime 9.359s ... 9.667s
nvme/014 (tr=loop bd=file) (flush a command from host) [passed]
runtime 9.076s ... 8.428s
nvme/016 (tr=loop) (create/delete many NVMeOF block device-backed ns and test discovery) [passed]
runtime 0.141s ... 0.123s
nvme/017 (tr=loop) (create/delete many file-ns and test discovery) [passed]
runtime 0.146s ... 0.140s
nvme/018 (tr=loop) (unit test NVMe-oF out of range access on a file backend) [passed]
runtime 0.632s ... 0.623s
nvme/019 (tr=loop bd=device) (test NVMe DSM Discard command) [passed]
runtime 0.643s ... 0.626s
nvme/019 (tr=loop bd=file) (test NVMe DSM Discard command) [passed]
runtime 0.636s ... 0.623s
nvme/021 (tr=loop bd=device) (test NVMe list command) [passed]
runtime 0.662s ... 0.635s
nvme/021 (tr=loop bd=file) (test NVMe list command) [passed]
runtime 0.645s ... 0.635s
nvme/022 (tr=loop bd=device) (test NVMe reset command) [passed]
runtime 0.991s ... 0.993s
nvme/022 (tr=loop bd=file) (test NVMe reset command) [passed]
runtime 1.009s ... 0.997s
nvme/023 (tr=loop bd=device) (test NVMe smart-log command) [passed]
runtime 0.647s ... 0.620s
nvme/023 (tr=loop bd=file) (test NVMe smart-log command) [passed]
runtime 0.653s ... 0.618s
nvme/025 (tr=loop bd=device) (test NVMe effects-log) [passed]
runtime 0.649s ... 0.626s
nvme/025 (tr=loop bd=file) (test NVMe effects-log) [passed]
runtime 0.665s ... 0.627s
nvme/026 (tr=loop bd=device) (test NVMe ns-descs) [passed]
runtime 0.649s ... 0.639s
nvme/026 (tr=loop bd=file) (test NVMe ns-descs) [passed]
runtime 0.641s ... 0.620s
nvme/027 (tr=loop bd=device) (test NVMe ns-rescan command) [passed]
runtime 0.675s ... 0.641s
nvme/027 (tr=loop bd=file) (test NVMe ns-rescan command) [passed]
runtime 0.673s ... 0.637s
nvme/028 (tr=loop bd=device) (test NVMe list-subsys) [passed]
runtime 0.648s ... 0.626s
nvme/028 (tr=loop bd=file) (test NVMe list-subsys) [passed]
runtime 0.640s ... 0.614s
nvme/029 (tr=loop) (test userspace IO via nvme-cli read/write interface) [passed]
runtime 1.002s ... 0.925s
nvme/030 (tr=loop) (ensure the discovery generation counter is updated appropriately) [passed]
runtime 0.438s ... 0.420s
nvme/031 (tr=loop) (test deletion of NVMeOF controllers immediately after setup) [passed]
runtime 5.997s ... 5.825s
nvme/038 (tr=loop) (test deletion of NVMeOF subsystem without enabling) [passed]
runtime 0.034s ... 0.034s
nvme/040 (tr=loop) (test nvme fabrics controller reset/disconnect operation during I/O) [passed]
runtime 7.032s ... 7.002s
nvme/041 (tr=loop) (Create authenticated connections) [passed]
runtime 0.701s ... 0.681s
nvme/042 (tr=loop) (Test dhchap key types for authenticated connections) [passed]
runtime 3.906s ... 3.772s
nvme/043 (tr=loop) (Test hash and DH group variations for authenticated connections) [passed]
runtime 4.987s ... 4.815s
nvme/044 (tr=loop) (Test bi-directional authentication) [passed]
runtime 1.482s ... 1.259s
nvme/045 (tr=loop) (Test re-authentication) [passed]
runtime 1.643s ... 1.566s
nvme/047 (tr=loop) (test different queue types for fabric transports) [not run]
nvme_trtype=loop is not supported in this test
nvme/048 (tr=loop) (Test queue count changes on reconnect) [not run]
nvme_trtype=loop is not supported in this test
nvme/051 (tr=loop) (test nvmet concurrent ns enable/disable) [passed]
runtime 1.390s ... 1.330s
nvme/052 (tr=loop) (Test file-ns creation/deletion under one subsystem) [passed]
runtime 6.363s ... 6.276s
nvme/054 (tr=loop) (Test the NVMe reservation feature) [passed]
runtime 0.775s ... 0.742s
nvme/055 (tr=loop) (Test nvme write to a loop target ns just after ns is disabled) [not run]
kernel option DEBUG_ATOMIC_SLEEP has not been enabled
nvme/056 (tr=loop) (enable zero copy offload and run rw traffic) [not run]
Remote target required but NVME_TARGET_CONTROL is not set
nvme_trtype=loop is not supported in this test
kernel option ULP_DDP has not been enabled
module nvme_tcp does not have parameter ddp_offload
KERNELSRC not set
Kernel sources do not have tools/net/ynl/cli.py
NVME_IFACE not set
nvme/057 (tr=loop) (test nvme fabrics controller ANA failover during I/O) [passed]
runtime 27.176s ... 27.143s
nvme/058 (tr=loop) (test rapid namespace remapping) [passed]
runtime 4.517s ... 4.414s
nvme/060 (tr=loop) (test nvme fabrics target reset) [not run]
nvme_trtype=loop is not supported in this test
nvme/061 (tr=loop) (test fabric target teardown and setup during I/O) [not run]
nvme_trtype=loop is not supported in this test
nvme/062 (tr=loop) (Create TLS-encrypted connections) [not run]
nvme_trtype=loop is not supported in this test
nvme/063 (tr=loop) (Create authenticated TCP connections with secure concatenation) [not run]
nvme_trtype=loop is not supported in this test
nvme/065 (tr=loop) (test unmap write zeroes sysfs interface with nvmet devices) [passed]
runtime 2.356s ... 2.340s
++ for t in loop tcp
++ echo '################NVMET_TRTYPES=tcp############'
################NVMET_TRTYPES=tcp############
++ NVME_IMG_SIZE=1G
++ NVME_NUM_ITER=1
++ NVMET_TRTYPES=tcp
++ ./check nvme
nvme/002 (tr=tcp) (create many subsystems and test discovery) [not run]
nvme_trtype=tcp is not supported in this test
nvme/003 (tr=tcp) (test if we're sending keep-alives to a discovery controller) [passed]
runtime 10.245s ... 10.241s
nvme/004 (tr=tcp) (test nvme and nvmet UUID NS descriptors) [passed]
runtime 0.380s ... 0.362s
nvme/005 (tr=tcp) (reset local loopback target) [passed]
runtime 0.450s ... 0.431s
nvme/006 (tr=tcp bd=device) (create an NVMeOF target) [passed]
runtime 0.104s ... 0.098s
nvme/006 (tr=tcp bd=file) (create an NVMeOF target) [passed]
runtime 0.096s ... 0.093s
nvme/008 (tr=tcp bd=device) (create an NVMeOF host) [passed]
runtime 0.385s ... 0.361s
nvme/008 (tr=tcp bd=file) (create an NVMeOF host) [passed]
runtime 0.389s ... 0.353s
nvmenvme/010 (tr=tcp bd=device) (run data verification fio job) [passed]
runtime 75.462s ... 76.437s
nvme/010 (tr=tcp bd=file) (run data verification fio job) [passed]
runtime 126.440s ... 117.933s
nvme/012 (tr=tcp bd=device) (run mkfs and data verification fio) [passed]
runtime 88.197s ... 81.428s
nvme/012 (tr=tcp bd=file) (run mkfs and data verification fio) [passed]
runtime 120.398s ... 117.412s
nvme/014 (tr=tcp bd=device) (flush a command from host) [passed]
runtime 9.931s ... 10.182s
nvme/014 (tr=tcp bd=file) (flush a command from host) [passed]
runtime 9.745s ... 9.867s
nvme/016 (tr=tcp) (create/delete many NVMeOF block device-backed ns and test discovery) [not run]
nvme_trtype=tcp is not supported in this test
nvme/017 (tr=tcp) (create/delete many file-ns and test discovery) [not run]
nvme_trtype=tcp is not supported in this test
nvme/018 (tr=tcp) (unit test NVMe-oF out of range access on a file backend) [passed]
runtime 0.380s ... 0.351s
nvme/019 (tr=tcp bd=device) (test NVMe DSM Discard command) [passed]
runtime 0.385s ... 0.344s
nvme/019 (tr=tcp bd=file) (test NVMe DSM Discard command) [passed]
runtime 0.382s ... 0.339s
nvme/021 (tr=tcp bd=device) (test NVMe list command) [passed]
runtime 0.371s ... 0.368s
nvme/021 (tr=tcp bd=file) (test NVMe list command) [passed]
runtime 0.396s ... 0.359s
nvme/022 (tr=tcp bd=device) (test NVMe reset command) [passed]
runtime 0.482s ... 0.451s
nvme/022 (tr=tcp bd=file) (test NVMe reset command) [passed]
runtime 0.453s ... 0.443s
nvme/023 (tr=tcp bd=device) (test NVMe smart-log command) [passed]
runtime 0.379s ... 0.349s
nvme/023 (tr=tcp bd=file) (test NVMe smart-log command) [passed]
runtime 0.361s ... 0.339s
nvme/025 (tr=tcp bd=device) (test NVMe effects-log) [passed]
runtime 0.391s ... 0.371s
nvme/025 (tr=tcp bd=file) (test NVMe effects-log) [passed]
runtime 0.380s ... 0.380s
nvme/026 (tr=tcp bd=device) (test NVMe ns-descs) [passed]
runtime 0.373s ... 0.352s
nvme/026 (tr=tcp bd=file) (test NVMe ns-descs) [passed]
runtime 0.356s ... 0.351s
nvme/027 (tr=tcp bd=device) (test NVMe ns-rescan command) [passed]
runtime 0.417s ... 0.380s
nvme/027 (tr=tcp bd=file) (test NVMe ns-rescan command) [passed]
runtime 0.396s ... 0.383s
nvme/028 (tr=tcp bd=device) (test NVMe list-subsys) [passed]
runtime 0.353s ... 0.348s
nvme/028 (tr=tcp bd=file) (test NVMe list-subsys) [passed]
runtime 0.351s ... 0.341s
nvme/029 (tr=tcp) (test userspace IO via nvme-cli read/write interface) [passed]
runtime 0.762s ... 0.722s
nvme/030 (tr=tcp) (ensure the discovery generation counter is updated appropriately) [passed]
runtime 0.410s ... 0.377s
nvme/031 (tr=tcp) (test deletion of NVMeOF controllers immediately after setup) [passed]
runtime 3.055s ... 3.008s
nvme/038 (tr=tcp) (test deletion of NVMeOF subsystem without enabling) [passed]
runtime 0.044s ... 0.042s
nvme/040 (tr=tcp) (test nvme fabrics controller reset/disconnect operation during I/O) [passed]
runtime 6.492s ... 6.454s
nvme/041 (tr=tcp) (Create authenticated connections) [passed]
runtime 0.418s ... 0.388s
nvme/042 (tr=tcp) (Test dhchap key types for authenticated connections) [passed]
runtime 1.827s ... 1.780s
nvme/043 (tr=tcp) (Test hash and DH group variations for authenticated connections) [passed]
runtime 2.504s ... 2.343s
nvme/044 (tr=tcp) (Test bi-directional authentication) [passed]
runtime 0.776s ... 0.709s
nvme/045 (tr=tcp) (Test re-authentication) [passed]
runtime 1.338s ... 1.311s
nvme/047 (tr=tcp) (test different queue types for fabric transports) [passed]
runtime 1.887s ... 1.792s
nvme/048 (tr=tcp) (Test queue count changes on reconnect) [passed]
runtime 5.531s ... 4.498s
nvme/051 (tr=tcp) (test nvmet concurrent ns enable/disable) [passed]
runtime 1.344s ... 1.375s
nvme/052 (tr=tcp) (Test file-ns creation/deletion under one subsystem) [not run]
nvme_trtype=tcp is not supported in this test
nvme/054 (tr=tcp) (Test the NVMe reservation feature) [passed]
runtime 0.506s ... 0.459s
nvme/055 (tr=tcp) (Test nvme write to a loop target ns just after ns is disabled) [not run]
nvme_trtype=tcp is not supported in this test
kernel option DEBUG_ATOMIC_SLEEP has not been enabled
nvme/056 (tr=tcp) (enable zero copy offload and run rw traffic) [not run]
Remote target required but NVME_TARGET_CONTROL is not set
kernel option ULP_DDP has not been enabled
module nvme_tcp does not have parameter ddp_offload
KERNELSRC not set
Kernel sources do not have tools/net/ynl/cli.py
NVME_IFACE not set
nvme/057 (tr=tcp) (test nvme fabrics controller ANA failover during I/O) [passed]
runtime 25.972s ... 25.924s
nvme/058 (tr=tcp) (test rapid namespace remapping) [passed]
runtime 3.612s ... 2.850s
nvme/060 (tr=tcp) (test nvme fabrics target reset) [passed]
runtime 19.437s ... 19.330s
nvme/061 (tr=tcp) (test fabric target teardown and setup during I/O) [passed]
runtime 8.645s ... 8.580s
nvme/062 (tr=tcp) (Create TLS-encrypted connections) [failed]
runtime 5.242s ... 5.176s
--- tests/nvme/062.out 2026-01-28 12:04:48.888356244 -0800
+++ /mnt/sda/blktests/results/nodev_tr_tcp/nvme/062.out.bad 2026-04-16 10:33:14.946941197 -0700
@@ -2,9 +2,13 @@
Test unencrypted connection w/ tls not required
disconnected 1 controller(s)
Test encrypted connection w/ tls not required
-disconnected 1 controller(s)
+FAIL: nvme connect return error code
+WARNING: connection is not encrypted
+disconnected 0 controller(s)
...
(Run 'diff -u tests/nvme/062.out /mnt/sda/blktests/results/nodev_tr_tcp/nvme/062.out.bad' to see the entire diff)
nvme/063 (tr=tcp) (Create authenticated TCP connections with secure concatenation) [passed]
runtime 2.026s ... 1.919s
nvme/065 (tr=tcp) (test unmap write zeroes sysfs interface with nvmet devices) [passed]
runtime 1.767s ... 1.726s
++ ./manage-rdma-nvme.sh --cleanup
====== RDMA NVMe Cleanup ======
[INFO] Disconnecting NVMe RDMA controllers...
[INFO] No NVMe RDMA controllers to disconnect
[INFO] Removing RDMA links...
[INFO] No RDMA links to remove
[INFO] Unloading NVMe RDMA modules...
[INFO] Unloading module: nvmet
[ERROR] Failed to unload module nvmet after 10 attempts
[WARN] Failed to unload 1 NVMe module(s)
[WARN] Some NVMe modules could not be unloaded
[INFO] Unloading soft-RDMA modules...
[INFO] Soft-RDMA modules unloaded successfully
[INFO] Verifying cleanup...
[INFO] Verification passed
[INFO] RDMA cleanup completed successfully
====== RDMA Network Configuration Status ======
Loaded Modules:
nvmet 258048
RDMA Links:
None
Network Interfaces (RDMA-capable):
None
blktests Configuration:
Not configured (run --setup first)
NVMe RDMA Controllers:
None
=================================================
++ ./manage-rdma-nvme.sh --setup
====== RDMA NVMe Setup ======
RDMA Type: siw
Interface: auto-detect
[INFO] Checking prerequisites...
[INFO] Prerequisites check passed
[INFO] Loading RDMA module: siw
[INFO] Module siw loaded successfully
[INFO] Creating RDMA links...
[INFO] Creating RDMA link: ens5_siw
[INFO] Created RDMA link: ens5_siw -> ens5
++ ./manage-rdma-nvme.sh --status
====== RDMA Configuration Status ======
====== RDMA Network Configuration Status ======
Loaded Modules:
siw 217088
nvmet 258048
RDMA Links:
link ens5_siw/1 state ACTIVE physical_state LINK_UP netdev ens5
Network Interfaces (RDMA-capable):
Interface: ens5
IPv4: 192.168.0.46
IPv6: fe80::5054:98ff:fe76:5440%ens5
blktests Configuration:
Transport Address: 192.168.0.46:4420
Transport Type: rdma
Command: NVMET_TRTYPES=rdma ./check nvme/
NVMe RDMA Controllers:
None
=================================================
++ echo '################NVMET_TRTYPES=rdma############'
################NVMET_TRTYPES=rdma############
++ NVME_IMG_SIZE=1G
++ NVME_NUM_ITER=1
++ nvme_trtype=rdma
++ ./check nvme
nvme/002 (tr=rdma) (create many subsystems and test discovery) [not run]
nvme_trtype=rdma is not supported in this test
nvme/003 (tr=rdma) (test if we're sending keep-alives to a discovery controller) [passed]
runtime 10.343s ... 10.315s
nvme/004 (tr=rdma) (test nvme and nvmet UUID NS descriptors) [passed]
runtime 0.691s ... 0.695s
nvme/005 (tr=rdma) (reset local loopback target) [passed]
runtime 1.021s ... 0.974s
nvme/006 (tr=rdma bd=device) (create an NVMeOF target) [passed]
runtime 0.147s ... 0.136s
nvme/006 (tr=rdma bd=file) (create an NVMeOF target) [passed]
runtime 0.148s ... 0.127s
nvme/008 (tr=rdma bd=device) (create an NVMeOF host) [passed]
runtime 0.706s ... 0.684s
nvme/008 (tr=rdma bd=file) (create an NVMeOF host) [passed]
runtime 0.704s ... 0.672s
nvme/010 (tr=rdma bd=device) (run data verification fio job) [passed]
runtime 35.986s ... 35.255s
nvme/010 (tr=rdma bd=file) (run data verification fio job) [passed]
runtime 61.777s ... 68.570s
nvme/012 (tr=rdma bd=device) (run mkfs and data verification fio) [passed]
runtime 42.996s ... 47.482s
nvme/012 (tr=rdma bd=file) (run mkfs and data verification fio) [passed]
runtime 65.456s ... 61.407s
nvme/014 (tr=rdma bd=device) (flush a command from host) [passed]
runtime 9.546s ... 9.855s
nvme/014 (tr=rdma bd=file) (flush a command from host) [passed]
runtime 9.791s ... 9.919s
nvme/016 (tr=rdma) (create/delete many NVMeOF block device-backed ns and test discovery) [not run]
nvme_trtype=rdma is not supported in this test
nvme/017 (tr=rdma) (create/delete many file-ns and test discovery) [not run]
nvme_trtype=rdma is not supported in this test
nvme/018 (tr=rdma) (unit test NVMe-oF out of range access on a file backend) [passed]
runtime 0.710s ... 0.654s
nvme/019 (tr=rdma bd=device) (test NVMe DSM Discard command) [passed]
runtime 0.699s ... 0.664s
nvme/019 (tr=rdma bd=file) (test NVMe DSM Discard command) [passed]
runtime 0.686s ... 0.649s
nvme/021 (tr=rdma bd=device) (test NVMe list command) [passed]
runtime 0.725s ... 0.676s
nvme/021 (tr=rdma bd=file) (test NVMe list command) [passed]
runtime 0.703s ... 0.673s
nvme/022 (tr=rdma bd=device) (test NVMe reset command) [passed]
runtime 1.048s ... 1.021s
nvme/022 (tr=rdma bd=file) (test NVMe reset command) [passed]
runtime 1.032s ... 1.015s
nvme/023 (tr=rdma bd=device) (test NVMe smart-log command) [passed]
runtime 0.699s ... 0.633s
nvme/023 (tr=rdma bd=file) (test NVMe smart-log command) [passed]
runtime 0.687s ... 0.656s
nvme/025 (tr=rdma bd=device) (test NVMe effects-log) [passed]
runtime 0.701s ... 0.706s
nvme/025 (tr=rdma bd=file) (test NVMe effects-log) [passed]
runtime 0.696s ... 0.688s
nvme/026 (tr=rdma bd=device) (test NVMe ns-descs) [passed]
runtime 0.703s ... 0.686s
nvme/026 (tr=rdma bd=file) (test NVMe ns-descs) [passed]
runtime 0.684s ... 0.669s
nvme/027 (tr=rdma bd=device) (test NVMe ns-rescan command) [passed]
runtime 0.727s ... 0.703s
nvme/027 (tr=rdma bd=file) (test NVMe ns-rescan command) [passed]
runtime 0.715s ... 0.707s
nvme/028 (tr=rdma bd=device) (test NVMe list-subsys) [passed]
runtime 0.688s ... 0.680s
nvme/028 (tr=rdma bd=file) (test NVMe list-subsys) [passed]
runtime 0.672s ... 0.670s
nvme/029 (tr=rdma) (test userspace IO via nvme-cli read/write interface) [passed]
runtime 1.067s ... 1.077s
nvme/030 (tr=rdma) (ensure the discovery generation counter is updated appropriately) [passed]
runtime 0.541s ... 0.511s
nvme/031 (tr=rdma) (test deletion of NVMeOF controllers immediately after setup) [passed]
runtime 5.793s ... 5.869s
nvme/038 (tr=rdma) (test deletion of NVMeOF subsystem without enabling) [passed]
runtime 0.086s ... 0.084s
nvme/040 (tr=rdma) (test nvme fabrics controller reset/disconnect operation during I/O) [passed]
runtime 7.059s ... 7.024s
nvme/041 (tr=rdma) (Create authenticated connections) [passed]
runtime 0.742s ... 0.718s
nvme/042 (tr=rdma) (Test dhchap key types for authenticated connections) [passed]
runtime 3.781s ... 3.759s
nvme/043 (tr=rdma) (Test hash and DH group variations for authenticated connections) [passed]
runtime 4.797s ... 4.487s
nvme/044 (tr=rdma) (Test bi-directional authentication) [passed]
runtime 1.346s ... 1.279s
nvme/045 (tr=rdma) (Test re-authentication) [passed]
runtime 1.806s ... 1.837s
nvme/047 (tr=rdma) (test different queue types for fabric transports) [passed]
runtime 2.688s ... 2.642s
nvme/048 (tr=rdma) (Test queue count changes on reconnect) [passed]
runtime 6.846s ... 5.810s
nvme/051 (tr=rdma) (test nvmet concurrent ns enable/disable) [passed]
runtime 1.399s ... 1.479s
nvme/052 (tr=rdma) (Test file-ns creation/deletion under one subsystem) [not run]
nvme_trtype=rdma is not supported in this test
nvme/054 (tr=rdma) (Test the NVMe reservation feature) [passed]
runtime 0.831s ... 0.799s
nvme/055 (tr=rdma) (Test nvme write to a loop target ns just after ns is disabled) [not run]
nvme_trtype=rdma is not supported in this test
kernel option DEBUG_ATOMIC_SLEEP has not been enabled
nvme/056 (tr=rdma) (enable zero copy offload and run rw traffic) [not run]
Remote target required but NVME_TARGET_CONTROL is not set
nvme_trtype=rdma is not supported in this test
kernel option ULP_DDP has not been enabled
module nvme_tcp does not have parameter ddp_offload
KERNELSRC not set
Kernel sources do not have tools/net/ynl/cli.py
NVME_IFACE not set
nvme/057 (tr=rdma) (test nvme fabrics controller ANA failover during I/O) [passed]
runtime 27.023s ... 26.984s
nvme/058 (tr=rdma) (test rapid namespace remapping) [passed]
runtime 4.478s ... 4.444s
nvme/060 (tr=rdma) (test nvme fabrics target reset) [passed]
runtime 20.821s ... 20.696s
nvme/061 (tr=rdma) (test fabric target teardown and setup during I/O) [passed]
runtime 15.509s ... 15.375s
nvme/062 (tr=rdma) (Create TLS-encrypted connections) [not run]
nvme_trtype=rdma is not supported in this test
nvme/063 (tr=rdma) (Create authenticated TCP connections with secure concatenation) [not run]
nvme_trtype=rdma is not supported in this test
nvme/065 (tr=rdma) (test unmap write zeroes sysfs interface with nvmet devices) [passed]
runtime 2.331s ... 2.369s
++ ./manage-rdma-nvme.sh --cleanup
====== RDMA NVMe Cleanup ======
[INFO] Disconnecting NVMe RDMA controllers...
[INFO] No NVMe RDMA controllers to disconnect
[INFO] Removing RDMA links...
[INFO] No RDMA links to remove
[INFO] Unloading NVMe RDMA modules...
[INFO] Unloading module: nvme_rdma
[INFO] Module nvme_rdma unloaded
[INFO] Unloading module: nvmet_rdma
[INFO] Module nvmet_rdma unloaded
[INFO] Unloading module: nvmet
[ERROR] Failed to unload module nvmet after 10 attempts
[WARN] Failed to unload 1 NVMe module(s)
[WARN] Some NVMe modules could not be unloaded
[INFO] Unloading soft-RDMA modules...
[INFO] Unloading module: siw
[INFO] Module siw unloaded
[INFO] Soft-RDMA modules unloaded successfully
[INFO] Verifying cleanup...
[INFO] Verification passed
[INFO] RDMA cleanup completed successfully
====== RDMA Network Configuration Status ======
Loaded Modules:
nvmet 258048
RDMA Links:
None
Network Interfaces (RDMA-capable):
None
blktests Configuration:
Not configured (run --setup first)
NVMe RDMA Controllers:
None
=================================================
blktests (master) #
--
2.39.5
^ permalink raw reply
* Re: [RFC PATCH 1/2] kernel/notifier: replace single-linked list with double-linked list for reverse traversal
From: David Laight @ 2026-04-16 19:15 UTC (permalink / raw)
To: Petr Mladek
Cc: chensong_2000, rafael, lenb, mturquette, sboyd, viresh.kumar, agk,
snitzer, mpatocka, bmarzins, song, yukuai, linan122, jason.wessel,
danielt, dianders, horms, davem, edumazet, kuba, pabeni, paulmck,
frederic, mcgrof, petr.pavlu, da.gomez, samitolvanen, atomlin,
jpoimboe, jikos, mbenes, joe.lawrence, rostedt, mhiramat,
mark.rutland, mathieu.desnoyers, linux-modules, linux-kernel,
linux-trace-kernel, linux-acpi, linux-clk, linux-pm,
live-patching, dm-devel, linux-raid, kgdb-bugreport, netdev
In-Reply-To: <aeD4H8P1DiPQoM8V@pathway.suse.cz>
On Thu, 16 Apr 2026 16:54:23 +0200
Petr Mladek <pmladek@suse.com> wrote:
> On Thu 2026-04-16 13:30:04, David Laight wrote:
> > On Wed, 15 Apr 2026 15:01:37 +0800
> > chensong_2000@189.cn wrote:
> >
> > > From: Song Chen <chensong_2000@189.cn>
> > >
> > > The current notifier chain implementation uses a single-linked list
> > > (struct notifier_block *next), which only supports forward traversal
> > > in priority order. This makes it difficult to handle cleanup/teardown
> > > scenarios that require notifiers to be called in reverse priority order.
> >
> > If it is only cleanup/teardown then the list can be order-reversed
> > as part of that process at the same time as the list is deleted.
>
> Interesting idea. But it won't work in all situations.
It is useful for things like locklessy queuing a request to be processed later.
Items can be added with a cmpxchg and the list grabbed by xchg of NULL.
The only downside is that reversing a list isn't cache friendly.
Thinks... although that may not be any worse than accessing the current 'tail'
to add to the end of a doubly linked (or singly linked with a tail ptr) list.
David
>
> Note that the motivation for this update are the module loader
> notifiers which are called several times for each loaded/removed module.
>
> Best Regards,
> Petr
>
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Robin Murphy @ 2026-04-16 17:03 UTC (permalink / raw)
To: Mark Brown
Cc: Demian Shulhan, Ard Biesheuvel, Christoph Hellwig, Mark Rutland,
Song Liu, Yu Kuai, Will Deacon, Catalin Marinas, linux-arm-kernel,
Li Nan, linux-raid, linux-kernel
In-Reply-To: <acc15515-4c43-402c-9800-c9b5e3da9891@sirena.org.uk>
On 16/04/2026 5:47 pm, Mark Brown wrote:
> On Thu, Apr 16, 2026 at 05:26:08PM +0100, Robin Murphy wrote:
>
>> Unless you've got a CPU with truly big wide vector units that _can't_ be
>> fully utilised by ASMID ops, then SVE is only really offering whatever
>> incidental benefits fall out of smaller code size. However, if you do have
>> those wider vectors, then the cost of correctly saving/restoring the SVE
>> state - of which a userspace benchmark isn't likely to be very
>> representative - is also going to scale up significantly.
>
> The other case will be when there's some SVE only extension that
> accelerates something that's relevant for the algorithm. That's not
> really a thing at present but I imagine that we'll run into that at some
> point.
Indeed - I was implicitly thinking in terms of things that _are_ just
transliterated from NEON to SVE, where the primary gain is stuff like
predicate loops, but even that _could_ potentially be enough to justify
an argument in-kernel SVE (using a 128-bit VL to keep the additional
state/cost to a minimum).
Cheers,
Robin.
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Mark Brown @ 2026-04-16 16:47 UTC (permalink / raw)
To: Robin Murphy
Cc: Demian Shulhan, Ard Biesheuvel, Christoph Hellwig, Mark Rutland,
Song Liu, Yu Kuai, Will Deacon, Catalin Marinas, linux-arm-kernel,
Li Nan, linux-raid, linux-kernel
In-Reply-To: <8db4defe-8b5e-4cc3-880b-72d46510b034@arm.com>
[-- Attachment #1: Type: text/plain, Size: 703 bytes --]
On Thu, Apr 16, 2026 at 05:26:08PM +0100, Robin Murphy wrote:
> Unless you've got a CPU with truly big wide vector units that _can't_ be
> fully utilised by ASMID ops, then SVE is only really offering whatever
> incidental benefits fall out of smaller code size. However, if you do have
> those wider vectors, then the cost of correctly saving/restoring the SVE
> state - of which a userspace benchmark isn't likely to be very
> representative - is also going to scale up significantly.
The other case will be when there's some SVE only extension that
accelerates something that's relevant for the algorithm. That's not
really a thing at present but I imagine that we'll run into that at some
point.
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Robin Murphy @ 2026-04-16 16:26 UTC (permalink / raw)
To: Demian Shulhan, Ard Biesheuvel
Cc: Christoph Hellwig, Mark Rutland, Song Liu, Yu Kuai, Will Deacon,
Catalin Marinas, Mark Brown, linux-arm-kernel, Li Nan, linux-raid,
linux-kernel
In-Reply-To: <CAOLeWCtf2rZyPeJH-LuZ2A+c7mC9M2r-Ya0VjyOJFpun3TFMnw@mail.gmail.com>
On 16/04/2026 3:59 pm, Demian Shulhan wrote:
> Hi Ard!
>
>> So what exactly did you fix in your test case?
>
> I just added the missing memset. You're right, "aliasing" was the
> wrong term for PIP.
>
>> This is the result where all data buffer pointers point to the same
>> memory, right? I.e., the zero page? So this is an unrealistic use
>> case that we can disregard.
>
> Yes, that's right. It was a flaw in my previous test setup.
>
>> Sorry but your result that SVE is 2x faster does not remain fully intact,
>> right? Given that the speedup is now 5.5%?
>> Should we just disregard the above results (and explanations) and focus
>> on the stuff below?
>
> Yes, it's better to focus on the data from SnapRAID. It was made on
> larger blocks and a wider range of disks, providing more realistic
> metrics.
>
>> OK, so the takeaway here is that SVE is only worth the hassle if the vector
>> length is at least 256 bits. This is not entirely surprising, but given that
>> Graviton4 went back to 128 bit vectors from 256, I wonder what the future
>> expectation is here.
>
> I agree. The results from the SnapRAID tests are not as impressive as
> I hoped, and the fact that Neoverse-V2 went back to 128-bit is a red
> flag. It suggests that wide SVE registers might not be a priority in
> future architecture versions.
If you look at the Neoverse V1 software optimisation guide[1], the SVE
instructions generally have half the throughput of their ASIMD
equivalents (i.e. presumably the vector pipes are still only 128 bits
wide and SVE is just using them in pairs), so indeed the total
instruction count is largely meaningless - IPC might be somewhat more
relevant, but I'd say the only performance number that's really
meaningful is the end-to-end MB/s measure of how fast the function
implementation as a whole can process data.
Unless you've got a CPU with truly big wide vector units that _can't_ be
fully utilised by ASMID ops, then SVE is only really offering whatever
incidental benefits fall out of smaller code size. However, if you do
have those wider vectors, then the cost of correctly saving/restoring
the SVE state - of which a userspace benchmark isn't likely to be very
representative - is also going to scale up significantly.
>> These results seem very relevant - perhaps Christoph can give some guidance
>> on how we might use these to improve the built-in benchmarks to be more
>> accurate.
>
> This is the most important part of this report, I think. SVE looks
> good only like my first idea on paper but in the real scenario it
> brings more problems than benefits.
>
> I’m happy to drop the SVE implementation for now and instead focus on
> modernizing the built-in benchmarks to ensure the kernel chooses the
> best available NEON path for actual storage workloads.
It's probably also worth checking whether the current NEON routines
themselves are actually optimal for modern big CPUs - things have moved
on quite a bit since Cortex-A57 (whose ASIMD performance could also be
described as "esoteric" at the best of times...)
Thanks,
Robin.
[1] https://developer.arm.com/documentation/110659/
>
> If you give me the green flag for it, I can start working on improving
> these built-in tests.
>
> Best regards,
> Demian
>
>
> чт, 16 квіт. 2026 р. о 16:40 Ard Biesheuvel <ardb@kernel.org> пише:
>>
>> Hi Demian,
>>
>> On Thu, 16 Apr 2026, at 14:40, Demian Shulhan wrote:
>>> Hi all,
>>>
>>> Sorry for the delay. The tests became more complex than I initially
>>> thought, so I needed to gather more data and properly validate the
>>> results across different hardware configurations.
>>>
>>> Firstly, I want to clarify the results from my March 29 tests. I found
>>> a flaw in my initial custom benchmark. The massive 2x throughput gap on
>>> 24 disks wasn't solely due to SVE's superiority, but rather a severe L1
>>> D-Cache thrashing issue that disproportionately penalized NEON.
>>>
>>> My custom test lacked memset() initialization, causing all data buffers
>>> to map to the Linux Zero Page (Virtually Indexed, Physically Tagged
>>> cache aliasing).
>>
>> D-caches always behave as PIPT on arm64. This is complex stuff, so please
>> don't present conjecture as fact.
>>
>>> Furthermore, even with memset(), allocating contiguous
>>> page-aligned buffers can causes severe Cache Address Sharing (a known
>>> issue that Andrea Mazzoleni solved in SnapRAID 13 years ago using
>>> RAID_MALLOC_DISPLACEMENT).
>>>
>>> Because SVE (svex4) uses 256-bit registers on Neoverse-V1, it performs
>>> exactly half the number of memory load instructions compared to 128-bit
>>> NEON. This dramatically reduced the L1 cache alias thrashing, allowing
>>> SVE to survive the memory bottleneck while NEON choked:
>>>
>>
>> You are drawing some conclusions here without disclosing the actual
>> information that you based this on. D-caches are non-aliasing on arm64.
>>
>> So what exactly did you fix in your test case?
>>
>>> Custom test without memset (4kb block):
>>> | algo=neonx4 ndisks=24 iterations=1M time=11.014s MB/s=7802.57
>>> | algo=svex4 ndisks=24 iterations=1M time=5.719s MB/s=15026.92
>>>
>>
>> This is the result where all data buffer pointers point to the same
>> memory, right? I.e., the zero page? So this is an unrealistic use
>> case that we can disregard.
>>
>>> Custom test with memset (4kb block):
>>> | algo=neonx4 ndisks=24 iterations=1M time=6.165s MB/s=13939.08
>>> | algo=svex4 ndisks=24 iterations=1M time=5.839s MB/s=14718.23
>>>
>>> Even with the corrected memory setup, the throughput gap narrowed, but
>>> the fundamental CPU-efficiency result remained fully intact.
>>>
>>
>> Sorry but your result that SVE is 2x faster does not remain fully intact,
>> right? Given that the speedup is now 5.5%?
>>
>> Should we just disregard the above results (and explanations) and focus
>> on the stuff below?
>>
>>> To completely isolate these variables and provide accurate real-world
>>> data, the following test campaigns were done based on the SnapRAID
>>> project (https://github.com/amadvance/snapraid) using its
>>> perf_bench.c tool with proper memory displacement and a 256 KiB block
>>> size.
>>>
>>> Test configurations:
>>> - c7g.medium (AWS Graviton3, 1 vCPU): Neoverse-V1, 256-bit SVE
>>> - c7g.xlarge (AWS Graviton3, 4 vCPUs): Neoverse-V1, 256-bit SVE
>>> - c8g.xlarge (AWS Graviton4, 4 vCPUs): Neoverse-V2, 128-bit SVE
>>>
>>>
>>> =========================================================
>>> Section 1: SnapRAID Validation on Graviton3 / Neoverse-V1
>>> =========================================================
>>>
>> ...
>>>
>>> 1.3 Main Graviton3 Conclusions
>>> - On 256-bit SVE hardware, svex4 consistently retires about ~34% fewer
>>> instructions and ~10-15% fewer CPU cycles than neonx4.
>>>
>>> =========================================================
>>> Section 2: SnapRAID Validation on Graviton4 / Neoverse-V2
>>> =========================================================
>>>
>> ...
>>>
>>> 2.3 Main Graviton4 Conclusions
>>> - On Neoverse-V2, SVE vector length is 128-bit (same as NEON).
>>> - Without the 256-bit width, NEON outperforms SVE.
>>> - svex4 retires ~32% MORE instructions here and is consistently slower.
>>>
>>> =========================================================
>>> Section 3: Validation on c7g.medium (1 vCPU)
>>> =========================================================
>>>
>> ...
>>> 3.3 Main c7g.medium Conclusions
>>> - The instruction count reduction (~34%) perfectly matches the 4-vCPU
>>> instance.
>>> - The single vCPU is heavily memory-bandwidth constrained (cycle counts
>>> are much higher waiting for RAM).
>>>
>>
>> OK, so the takeaway here is that SVE is only worth the hassle if the vector
>> length is at least 256 bits. This is not entirely surprising, but given that
>> Graviton4 went back to 128 bit vectors from 256, I wonder what the future
>> expectation is here.
>>
>> But having these numbers is definitely a good first step. Now we need to
>> quantify the overhead associated with having kernel mode SVE state that
>> needs to be preserved/restored.
>>
>> However, 10%-15% speedup that can only be achieved on SVE implementations
>> with 256 bit vectors or more may not be that enticing in the end. (The
>> fact that you are retiring 34% instructions less does not really matter
>> here unless there is some meaningful SMT-like sharing of functional units
>> going on in the meantime, which seems unlikely on a CPU that is maxed out
>> on the data side)
>>
>>
>>> =========================================================
>>> Section 4: The Pitfalls of the Current Kernel Benchmark
>>> =========================================================
>>>
>>
>> These results seem very relevant - perhaps Christoph can give some guidance
>> on how we might use these to improve the built-in benchmarks to be more
>> accurate.
>>
>>
>> Thanks,
>>
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Demian Shulhan @ 2026-04-16 14:59 UTC (permalink / raw)
To: Ard Biesheuvel
Cc: Christoph Hellwig, Mark Rutland, Song Liu, Yu Kuai, Will Deacon,
Catalin Marinas, Mark Brown, linux-arm-kernel, Robin Murphy,
Li Nan, linux-raid, linux-kernel
In-Reply-To: <5158e4e0-3275-4c29-a8fc-2dfabc13a69d@app.fastmail.com>
Hi Ard!
> So what exactly did you fix in your test case?
I just added the missing memset. You're right, "aliasing" was the
wrong term for PIP.
> This is the result where all data buffer pointers point to the same
> memory, right? I.e., the zero page? So this is an unrealistic use
> case that we can disregard.
Yes, that's right. It was a flaw in my previous test setup.
> Sorry but your result that SVE is 2x faster does not remain fully intact,
> right? Given that the speedup is now 5.5%?
> Should we just disregard the above results (and explanations) and focus
> on the stuff below?
Yes, it's better to focus on the data from SnapRAID. It was made on
larger blocks and a wider range of disks, providing more realistic
metrics.
> OK, so the takeaway here is that SVE is only worth the hassle if the vector
> length is at least 256 bits. This is not entirely surprising, but given that
> Graviton4 went back to 128 bit vectors from 256, I wonder what the future
> expectation is here.
I agree. The results from the SnapRAID tests are not as impressive as
I hoped, and the fact that Neoverse-V2 went back to 128-bit is a red
flag. It suggests that wide SVE registers might not be a priority in
future architecture versions.
> These results seem very relevant - perhaps Christoph can give some guidance
> on how we might use these to improve the built-in benchmarks to be more
> accurate.
This is the most important part of this report, I think. SVE looks
good only like my first idea on paper but in the real scenario it
brings more problems than benefits.
I’m happy to drop the SVE implementation for now and instead focus on
modernizing the built-in benchmarks to ensure the kernel chooses the
best available NEON path for actual storage workloads.
If you give me the green flag for it, I can start working on improving
these built-in tests.
Best regards,
Demian
чт, 16 квіт. 2026 р. о 16:40 Ard Biesheuvel <ardb@kernel.org> пише:
>
> Hi Demian,
>
> On Thu, 16 Apr 2026, at 14:40, Demian Shulhan wrote:
> > Hi all,
> >
> > Sorry for the delay. The tests became more complex than I initially
> > thought, so I needed to gather more data and properly validate the
> > results across different hardware configurations.
> >
> > Firstly, I want to clarify the results from my March 29 tests. I found
> > a flaw in my initial custom benchmark. The massive 2x throughput gap on
> > 24 disks wasn't solely due to SVE's superiority, but rather a severe L1
> > D-Cache thrashing issue that disproportionately penalized NEON.
> >
> > My custom test lacked memset() initialization, causing all data buffers
> > to map to the Linux Zero Page (Virtually Indexed, Physically Tagged
> > cache aliasing).
>
> D-caches always behave as PIPT on arm64. This is complex stuff, so please
> don't present conjecture as fact.
>
> > Furthermore, even with memset(), allocating contiguous
> > page-aligned buffers can causes severe Cache Address Sharing (a known
> > issue that Andrea Mazzoleni solved in SnapRAID 13 years ago using
> > RAID_MALLOC_DISPLACEMENT).
> >
> > Because SVE (svex4) uses 256-bit registers on Neoverse-V1, it performs
> > exactly half the number of memory load instructions compared to 128-bit
> > NEON. This dramatically reduced the L1 cache alias thrashing, allowing
> > SVE to survive the memory bottleneck while NEON choked:
> >
>
> You are drawing some conclusions here without disclosing the actual
> information that you based this on. D-caches are non-aliasing on arm64.
>
> So what exactly did you fix in your test case?
>
> > Custom test without memset (4kb block):
> > | algo=neonx4 ndisks=24 iterations=1M time=11.014s MB/s=7802.57
> > | algo=svex4 ndisks=24 iterations=1M time=5.719s MB/s=15026.92
> >
>
> This is the result where all data buffer pointers point to the same
> memory, right? I.e., the zero page? So this is an unrealistic use
> case that we can disregard.
>
> > Custom test with memset (4kb block):
> > | algo=neonx4 ndisks=24 iterations=1M time=6.165s MB/s=13939.08
> > | algo=svex4 ndisks=24 iterations=1M time=5.839s MB/s=14718.23
> >
> > Even with the corrected memory setup, the throughput gap narrowed, but
> > the fundamental CPU-efficiency result remained fully intact.
> >
>
> Sorry but your result that SVE is 2x faster does not remain fully intact,
> right? Given that the speedup is now 5.5%?
>
> Should we just disregard the above results (and explanations) and focus
> on the stuff below?
>
> > To completely isolate these variables and provide accurate real-world
> > data, the following test campaigns were done based on the SnapRAID
> > project (https://github.com/amadvance/snapraid) using its
> > perf_bench.c tool with proper memory displacement and a 256 KiB block
> > size.
> >
> > Test configurations:
> > - c7g.medium (AWS Graviton3, 1 vCPU): Neoverse-V1, 256-bit SVE
> > - c7g.xlarge (AWS Graviton3, 4 vCPUs): Neoverse-V1, 256-bit SVE
> > - c8g.xlarge (AWS Graviton4, 4 vCPUs): Neoverse-V2, 128-bit SVE
> >
> >
> > =========================================================
> > Section 1: SnapRAID Validation on Graviton3 / Neoverse-V1
> > =========================================================
> >
> ...
> >
> > 1.3 Main Graviton3 Conclusions
> > - On 256-bit SVE hardware, svex4 consistently retires about ~34% fewer
> > instructions and ~10-15% fewer CPU cycles than neonx4.
> >
> > =========================================================
> > Section 2: SnapRAID Validation on Graviton4 / Neoverse-V2
> > =========================================================
> >
> ...
> >
> > 2.3 Main Graviton4 Conclusions
> > - On Neoverse-V2, SVE vector length is 128-bit (same as NEON).
> > - Without the 256-bit width, NEON outperforms SVE.
> > - svex4 retires ~32% MORE instructions here and is consistently slower.
> >
> > =========================================================
> > Section 3: Validation on c7g.medium (1 vCPU)
> > =========================================================
> >
> ...
> > 3.3 Main c7g.medium Conclusions
> > - The instruction count reduction (~34%) perfectly matches the 4-vCPU
> > instance.
> > - The single vCPU is heavily memory-bandwidth constrained (cycle counts
> > are much higher waiting for RAM).
> >
>
> OK, so the takeaway here is that SVE is only worth the hassle if the vector
> length is at least 256 bits. This is not entirely surprising, but given that
> Graviton4 went back to 128 bit vectors from 256, I wonder what the future
> expectation is here.
>
> But having these numbers is definitely a good first step. Now we need to
> quantify the overhead associated with having kernel mode SVE state that
> needs to be preserved/restored.
>
> However, 10%-15% speedup that can only be achieved on SVE implementations
> with 256 bit vectors or more may not be that enticing in the end. (The
> fact that you are retiring 34% instructions less does not really matter
> here unless there is some meaningful SMT-like sharing of functional units
> going on in the meantime, which seems unlikely on a CPU that is maxed out
> on the data side)
>
>
> > =========================================================
> > Section 4: The Pitfalls of the Current Kernel Benchmark
> > =========================================================
> >
>
> These results seem very relevant - perhaps Christoph can give some guidance
> on how we might use these to improve the built-in benchmarks to be more
> accurate.
>
>
> Thanks,
>
^ permalink raw reply
* Re: [RFC PATCH 1/2] kernel/notifier: replace single-linked list with double-linked list for reverse traversal
From: Petr Mladek @ 2026-04-16 14:54 UTC (permalink / raw)
To: David Laight
Cc: chensong_2000, rafael, lenb, mturquette, sboyd, viresh.kumar, agk,
snitzer, mpatocka, bmarzins, song, yukuai, linan122, jason.wessel,
danielt, dianders, horms, davem, edumazet, kuba, pabeni, paulmck,
frederic, mcgrof, petr.pavlu, da.gomez, samitolvanen, atomlin,
jpoimboe, jikos, mbenes, joe.lawrence, rostedt, mhiramat,
mark.rutland, mathieu.desnoyers, linux-modules, linux-kernel,
linux-trace-kernel, linux-acpi, linux-clk, linux-pm,
live-patching, dm-devel, linux-raid, kgdb-bugreport, netdev
In-Reply-To: <20260416133004.07bd2886@pumpkin>
On Thu 2026-04-16 13:30:04, David Laight wrote:
> On Wed, 15 Apr 2026 15:01:37 +0800
> chensong_2000@189.cn wrote:
>
> > From: Song Chen <chensong_2000@189.cn>
> >
> > The current notifier chain implementation uses a single-linked list
> > (struct notifier_block *next), which only supports forward traversal
> > in priority order. This makes it difficult to handle cleanup/teardown
> > scenarios that require notifiers to be called in reverse priority order.
>
> If it is only cleanup/teardown then the list can be order-reversed
> as part of that process at the same time as the list is deleted.
Interesting idea. But it won't work in all situations.
Note that the motivation for this update are the module loader
notifiers which are called several times for each loaded/removed module.
Best Regards,
Petr
^ permalink raw reply
* Re: [RFC PATCH 2/2] kernel/module: Decouple klp and ftrace from load_module
From: Petr Mladek @ 2026-04-16 14:49 UTC (permalink / raw)
To: Petr Pavlu
Cc: Song Chen, rafael, lenb, mturquette, sboyd, viresh.kumar, agk,
snitzer, mpatocka, bmarzins, song, yukuai, linan122, jason.wessel,
danielt, dianders, horms, davem, edumazet, kuba, pabeni, paulmck,
frederic, mcgrof, da.gomez, samitolvanen, atomlin, jpoimboe,
jikos, mbenes, joe.lawrence, rostedt, mhiramat, mark.rutland,
mathieu.desnoyers, linux-modules, linux-kernel,
linux-trace-kernel, linux-acpi, linux-clk, linux-pm,
live-patching, dm-devel, linux-raid, kgdb-bugreport, netdev
In-Reply-To: <1db425bf-58a9-4768-8c38-3ae25d7662a5@suse.com>
On Thu 2026-04-16 13:18:30, Petr Pavlu wrote:
> On 4/15/26 8:43 AM, Song Chen wrote:
> > On 4/14/26 22:33, Petr Pavlu wrote:
> >> On 4/13/26 10:07 AM, chensong_2000@189.cn wrote:
> >>> diff --git a/include/linux/module.h b/include/linux/module.h
> >>> index 14f391b186c6..0bdd56f9defd 100644
> >>> --- a/include/linux/module.h
> >>> +++ b/include/linux/module.h
> >>> @@ -308,6 +308,14 @@ enum module_state {
> >>> MODULE_STATE_COMING, /* Full formed, running module_init. */
> >>> MODULE_STATE_GOING, /* Going away. */
> >>> MODULE_STATE_UNFORMED, /* Still setting it up. */
> >>> + MODULE_STATE_FORMED,
> >>
> >> I don't see a reason to add a new module state. Why is it necessary and
> >> how does it fit with the existing states?
> >>
> > because once notifier fails in state MODULE_STATE_UNFORMED (now only ftrace has someting to do in this state), notifier chain will roll back by calling blocking_notifier_call_chain_robust, i'm afraid MODULE_STATE_GOING is going to jeopardise the notifers which don't handle it appropriately, like:
> >
> > case MODULE_STATE_COMING:
> > kmalloc();
> > case MODULE_STATE_GOING:
> > kfree();
>
> My understanding is that the current module "state machine" operates as
> follows. Transitions marked with an asterisk (*) are announced via the
> module notifier.
>
> ---> UNFORMED --*> COMING --*> LIVE --*> GOING -.
> ^ | ^ |
> | '---------------------* |
> '---------------------------------------'
>
> The new code aims to replace the current ftrace_module_init() call in
> load_module(). To achieve this, it adds a notification for the UNFORMED
> state (only when loading a module) and introduces a new FORMED state for
> rollback. FORMED is purely a fake state because it never appears in
> module::state. The new structure is as follows:
>
> ,--*> (FORMED)
> |
> --*> UNFORMED --*> COMING --*> LIVE --*> GOING -.
> ^ | ^ |
> | '---------------------* |
> '---------------------------------------'
>
> I'm afraid this is quite complex and inconsistent. Unless it can be kept
> simple, we would be just replacing one special handling with a different
> complexity, which is not worth it.
> >>
> >>> + if (err)
> >>> + goto ddebug_cleanup;
> >>> /* Finally it's fully formed, ready to start executing. */
> >>> err = complete_formation(mod, info);
> >>> - if (err)
> >>> + if (err) {
> >>> + blocking_notifier_call_chain_reverse(&module_notify_list,
> >>> + MODULE_STATE_FORMED, mod);
> >>> goto ddebug_cleanup;
> >>> + }
> >>> - err = prepare_coming_module(mod);
> >>> + err = prepare_module_state_transaction(mod,
> >>> + MODULE_STATE_COMING, MODULE_STATE_GOING);
> >>> if (err)
> >>> goto bug_cleanup;
> >>> @@ -3522,7 +3519,6 @@ static int load_module(struct load_info *info, const char __user *uargs,
> >>> destroy_params(mod->kp, mod->num_kp);
> >>> blocking_notifier_call_chain(&module_notify_list,
> >>> MODULE_STATE_GOING, mod);
> >>
> >> My understanding is that all notifier chains for MODULE_STATE_GOING
> >> should be reversed.
> > yes, all, from lowest priority notifier to highest.
> > I will resend patch 1 which was failed due to my proxy setting.
>
> What I meant here is that the call:
>
> blocking_notifier_call_chain(&module_notify_list, MODULE_STATE_GOING, mod);
>
> should be replaced with:
>
> blocking_notifier_call_chain_reverse(&module_notify_list, MODULE_STATE_GOING, mod);
>
> >
> >>
> >>> - klp_module_going(mod);
> >>> bug_cleanup:
> >>> mod->state = MODULE_STATE_GOING;
> >>> /* module_bug_cleanup needs module_mutex protection */
> >>
> >> The patch removes the klp_module_going() cleanup call in load_module().
> >> Similarly, the ftrace_release_mod() call under the ddebug_cleanup label
> >> should be removed and appropriately replaced with a cleanup via
> >> a notifier.
> >>
> > err = prepare_module_state_transaction(mod,
> > MODULE_STATE_UNFORMED, MODULE_STATE_FORMED);
> > if (err)
> > goto ddebug_cleanup;
> >
> > ftrace will be cleanup in blocking_notifier_call_chain_robust rolling back.
> >
> > err = prepare_module_state_transaction(mod,
> > MODULE_STATE_COMING, MODULE_STATE_GOING);
> >
> > each notifier including ftrace and klp will be cleanup in blocking_notifier_call_chain_robust rolling back.
> >
> > if all notifiers are successful in MODULE_STATE_COMING, they all will be clean up in
> > coming_cleanup:
> > mod->state = MODULE_STATE_GOING;
> > destroy_params(mod->kp, mod->num_kp);
> > blocking_notifier_call_chain(&module_notify_list,
> > MODULE_STATE_GOING, mod);
> >
> > if something wrong underneath.
>
> My point is that the patch leaves a call to ftrace_release_mod() in
> load_module(), which I expected to be handled via a notifier.
I think that I have got it. The ftrace code needs two notifiers when
the module is being loaded and two when it is going.
This is why Sond added the new state. But I think that we would
need two new states to call:
+ ftrace_module_init() in MODULE_STATE_UNFORMED
+ ftrace_module_enable() in MODULE_STATE_FORMED
and
+ ftrace_free_mem() in MODULE_STATE_PRE_GOING
+ ftrace_free_mem() in MODULE_STATE_GOING
By using the ascii art:
-*> UNFORMED -*> FORMED -> COMING -*> LIVE -*> PRE_GOING -*> GOING -.
| | | ^ ^ ^
| | '----------------' | |
| '--------------------------------------' |
'------------------------------------------------------'
But I think that this is not worth it.
Best Regards,
Petr
^ permalink raw reply
* Re: [PATCH v2 0/5] md: bitmap grow fixes
From: Su Yue @ 2026-04-16 14:10 UTC (permalink / raw)
To: Su Yue; +Cc: linux-raid, song, xni, linan122, yukuai, heming.zhao
In-Reply-To: <20260407102625.5686-1-glass.su@suse.com>
On Tue 07 Apr 2026 at 18:26, Su Yue <glass.su@suse.com> wrote:
> Hi, This v2 series is to fixes bugs exposed by
> mdadm/clustermd-autotest.
> The bugs are not cluster md only but also bitmap related.
>
> The series based on v7.0-rc7 passes tests with mdadm v4.6
> without regression.
>
> v2:
> Add a dummy bitmap operations per Kuai's suggestion.
>
> To Yu Kuai:
> A NULL group can't used for internal_bitmap_group since
> the entries from internal_bitmap_attrs should be under bitmap.
> So instead of sysfs_create_groups(), sysfs_create_group() and
> sysfs_merge_group()
> are still needed /sigh.
>
> Su Yue (5):
> md/md-bitmap: call md_bitmap_create,destroy in location_store
> md/md-bitmap: add an extra sysfs argument to md_bitmap_create
> and
> destroy
> md/md-bitmap: add dummy bitmap ops for none to fix wrong
> bitmap offset
> md: skip ID_BITMAP_NONE when show available bitmap types
> md/md-bitmap: remove member group from bitmap_operations
>
Missing part should be added in v3:
diff --git a/drivers/md/md.c b/drivers/md/md.c
index 20a953676319..453b89941db2 100644
--- a/drivers/md/md.c
+++ b/drivers/md/md.c
@@ -6450,9 +6450,6 @@ static int start_dirty_degraded;
int md_bitmap_create(struct mddev *mddev, bool create_sysfs)
{
- if (mddev->bitmap_id == ID_BITMAP_NONE)
- return -EINVAL;
-
}
> drivers/md/md-bitmap.c | 121
> ++++++++++++++++++++++++++++++++++++---
> drivers/md/md-bitmap.h | 3 +-
> drivers/md/md-llbitmap.c | 13 ++++-
> drivers/md/md.c | 55 +++++++++---------
> drivers/md/md.h | 2 +
> 5 files changed, 155 insertions(+), 39 deletions(-)
^ permalink raw reply related
* Re: [PATCH v2 1/5] md/md-bitmap: call md_bitmap_create,destroy in location_store
From: Su Yue @ 2026-04-16 14:08 UTC (permalink / raw)
To: Xiao Ni; +Cc: Su Yue, linux-raid, song, linan122, yukuai, heming.zhao
In-Reply-To: <CALTww28wN=9R2RMApc89M=cRtHmrFYgc8qgakR6LwcyXeQNdDw@mail.gmail.com>
On Wed 15 Apr 2026 at 18:34, Xiao Ni <xni@redhat.com> wrote:
> On Tue, Apr 7, 2026 at 6:26 PM Su Yue <glass.su@suse.com> wrote:
>>
>> If bitmap/location is present, mdadm will call
>> update_array_info()
>> while growing bitmap from none to internal via
>> location_store().
>> md_bitmap_create() is needed to set mddev->bitmap_ops otherwise
>> mddev->bitmap_ops->get_stats() in update_array_info() will
>> trigger
>> kernel NULL pointer dereference.
>
>
> Hi Su Yue
>
> How can bitmap/location be present when bitmap is none? Could
> you
> provide the test commands that reproduce this problem?
>
Sorry for the misleading commit message. It can only be reproduced
patch 3 is appiled.
I adjusted the sequence of this patch for easy review because
md_bitmap_create,destroy
are touched in patch1,2 and 3. Also if put the patch after 3rd
patch,
it will break ability to bisect.
# mdadm --create --assume-clean /dev/md0 -f --bitmap=internal
--raid-devices=2 --level=mirror --metadata=1.2 /dev/vdc /dev/vdd
# mdadm --grow /dev/md0 --bitmap=none
# mdadm --grow /dev/md0 --bitmap=internal # step 3
# mdadm --grow /dev/md0 --bitmap=none # step 4
[1] 2325 killed mdadm --grow /dev/md0 --bitmap=none
When step 3 is called,
md_bitmap_destroy() is called in update_array_info() to set NULL
mddev->bitmap_ops
then in step 4 kernel Oops is triggered.
I am willing to amend commit message or move it after patch 3 if
you would like.
--
Su
>
> mdadm -CR /dev/md0 -l1 -n2 /dev/loop0 /dev/loop1 --bitmap=none
> (There
> is not bitmap/location, because bitmap directory is not created)
> mdadm /dev/md0 --grow --bitmap=internal
> Grow.c md_set_array_info runs
> 451 array.state |= (1 << MD_SB_BITMAP_PRESENT);
> 452 rv = md_set_array_info(fd, &array);
> In kernel space, it runs
> 8125 rv = md_bitmap_create(mddev);
> 8126 if (!rv)
> 8127 rv = mddev->bitmap_ops->load(mddev);
>
> Best Regards
> Xiao
>
>>
>> Fixes: fb8cc3b0d9db ("md/md-bitmap: delay registration of
>> bitmap_ops until creating bitmap")
>> Signed-off-by: Su Yue <glass.su@suse.com>
>> ---
>> drivers/md/md-bitmap.c | 11 ++++++++---
>> drivers/md/md.c | 4 ++--
>> drivers/md/md.h | 2 ++
>> 3 files changed, 12 insertions(+), 5 deletions(-)
>>
>> diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c
>> index 83378c033c72..2f24aae05552 100644
>> --- a/drivers/md/md-bitmap.c
>> +++ b/drivers/md/md-bitmap.c
>> @@ -2618,7 +2618,7 @@ location_store(struct mddev *mddev, const
>> char *buf, size_t len)
>> goto out;
>> }
>>
>> - bitmap_destroy(mddev);
>> + md_bitmap_destroy(mddev);
>> mddev->bitmap_info.offset = 0;
>> if (mddev->bitmap_info.file) {
>> struct file *f =
>> mddev->bitmap_info.file;
>> @@ -2653,15 +2653,20 @@ location_store(struct mddev *mddev,
>> const char *buf, size_t len)
>> goto out;
>> }
>>
>> + /*
>> + * lockless bitmap shoudle have set
>> bitmap_id
>> + * using bitmap_type, so always
>> ID_BITMAP.
>> + */
>> + mddev->bitmap_id = ID_BITMAP;
>> mddev->bitmap_info.offset = offset;
>> - rv = bitmap_create(mddev);
>> + rv = md_bitmap_create(mddev);
>> if (rv)
>> goto out;
>>
>> rv = bitmap_load(mddev);
>> if (rv) {
>> mddev->bitmap_info.offset = 0;
>> - bitmap_destroy(mddev);
>> + md_bitmap_destroy(mddev);
>> goto out;
>> }
>> }
>> diff --git a/drivers/md/md.c b/drivers/md/md.c
>> index 3ce6f9e9d38e..8b1ecc370ad6 100644
>> --- a/drivers/md/md.c
>> +++ b/drivers/md/md.c
>> @@ -6447,7 +6447,7 @@ static void md_safemode_timeout(struct
>> timer_list *t)
>>
>> static int start_dirty_degraded;
>>
>> -static int md_bitmap_create(struct mddev *mddev)
>> +int md_bitmap_create(struct mddev *mddev)
>> {
>> if (mddev->bitmap_id == ID_BITMAP_NONE)
>> return -EINVAL;
>> @@ -6458,7 +6458,7 @@ static int md_bitmap_create(struct mddev
>> *mddev)
>> return mddev->bitmap_ops->create(mddev);
>> }
>>
>> -static void md_bitmap_destroy(struct mddev *mddev)
>> +void md_bitmap_destroy(struct mddev *mddev)
>> {
>> if (!md_bitmap_registered(mddev))
>> return;
>> diff --git a/drivers/md/md.h b/drivers/md/md.h
>> index ac84289664cd..ed69244af00d 100644
>> --- a/drivers/md/md.h
>> +++ b/drivers/md/md.h
>> @@ -895,6 +895,8 @@ static inline void safe_put_page(struct
>> page *p)
>>
>> int register_md_submodule(struct md_submodule_head *msh);
>> void unregister_md_submodule(struct md_submodule_head *msh);
>> +int md_bitmap_create(struct mddev *mddev);
>> +void md_bitmap_destroy(struct mddev *mddev);
>>
>> extern struct md_thread *md_register_thread(
>> void (*run)(struct md_thread *thread),
>> --
>> 2.53.0
>>
^ permalink raw reply
* [PATCH] md/raid1,raid10: don't fail devices for invalid IO errors
From: Keith Busch @ 2026-04-16 14:03 UTC (permalink / raw)
To: linux-raid, song, yukuai
Cc: linan122, axboe, Keith Busch, Tomáš Trnka
From: Keith Busch <kbusch@kernel.org>
BLK_STS_INVAL indicates the IO request itself was invalid, not that the
device has failed. When raid1 treats this as a device error, it retries
on alternate mirrors which fail the same way, eventually exceeding the
read error threshold and removing the device from the array.
This happens when stacking configurations bypass bio_split_to_limits()
in the IO path: dm-raid calls md_handle_request() directly without going
through md_submit_bio(), skipping the alignment validation that would
otherwise reject invalid bios early. The invalid bio reaches the
lower block layers, which fail the bio with BLK_STS_INVAL, and raid1
wrongly interprets this as a device failure.
Add BLK_STS_INVAL to raid1_should_handle_error() so that invalid IO
errors are propagated back to the caller rather than triggering device
removal. This is consistent with the previous kernel behavior when
alignment checks were done earlier in the direct-io path.
Fixes: 5ff3f74e145adc7 ("block: simplify direct io validity check")
Link: https://lore.kernel.org/linux-block/2982107.4sosBPzcNG@electra/
Reported-by: Tomáš Trnka <trnka@scm.com>
Signed-off-by: Keith Busch <kbusch@kernel.org>
---
drivers/md/raid1-10.c | 7 ++++++-
1 file changed, 6 insertions(+), 1 deletion(-)
diff --git a/drivers/md/raid1-10.c b/drivers/md/raid1-10.c
index c33099925f230..56a56a4da4f83 100644
--- a/drivers/md/raid1-10.c
+++ b/drivers/md/raid1-10.c
@@ -293,8 +293,13 @@ static inline bool raid1_should_read_first(struct mddev *mddev,
* bio with REQ_RAHEAD or REQ_NOWAIT can fail at anytime, before such IO is
* submitted to the underlying disks, hence don't record badblocks or retry
* in this case.
+ *
+ * BLK_STS_INVAL means the bio was not valid for the underlying device. This
+ * is a user error, not a device failure, so retrying or recording bad blocks
+ * would be wrong.
*/
static inline bool raid1_should_handle_error(struct bio *bio)
{
- return !(bio->bi_opf & (REQ_RAHEAD | REQ_NOWAIT));
+ return !(bio->bi_opf & (REQ_RAHEAD | REQ_NOWAIT)) &&
+ bio->bi_status != BLK_STS_INVAL;
}
--
2.52.0
^ permalink raw reply related
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Ard Biesheuvel @ 2026-04-16 13:39 UTC (permalink / raw)
To: Demian Shulhan, Christoph Hellwig
Cc: Mark Rutland, Song Liu, Yu Kuai, Will Deacon, Catalin Marinas,
Mark Brown, linux-arm-kernel, Robin Murphy, Li Nan, linux-raid,
linux-kernel
In-Reply-To: <CAOLeWCtxPk7q_bVvrcKaipSKr+_=57Auobcj0cFRvXxXMdH58g@mail.gmail.com>
Hi Demian,
On Thu, 16 Apr 2026, at 14:40, Demian Shulhan wrote:
> Hi all,
>
> Sorry for the delay. The tests became more complex than I initially
> thought, so I needed to gather more data and properly validate the
> results across different hardware configurations.
>
> Firstly, I want to clarify the results from my March 29 tests. I found
> a flaw in my initial custom benchmark. The massive 2x throughput gap on
> 24 disks wasn't solely due to SVE's superiority, but rather a severe L1
> D-Cache thrashing issue that disproportionately penalized NEON.
>
> My custom test lacked memset() initialization, causing all data buffers
> to map to the Linux Zero Page (Virtually Indexed, Physically Tagged
> cache aliasing).
D-caches always behave as PIPT on arm64. This is complex stuff, so please
don't present conjecture as fact.
> Furthermore, even with memset(), allocating contiguous
> page-aligned buffers can causes severe Cache Address Sharing (a known
> issue that Andrea Mazzoleni solved in SnapRAID 13 years ago using
> RAID_MALLOC_DISPLACEMENT).
>
> Because SVE (svex4) uses 256-bit registers on Neoverse-V1, it performs
> exactly half the number of memory load instructions compared to 128-bit
> NEON. This dramatically reduced the L1 cache alias thrashing, allowing
> SVE to survive the memory bottleneck while NEON choked:
>
You are drawing some conclusions here without disclosing the actual
information that you based this on. D-caches are non-aliasing on arm64.
So what exactly did you fix in your test case?
> Custom test without memset (4kb block):
> | algo=neonx4 ndisks=24 iterations=1M time=11.014s MB/s=7802.57
> | algo=svex4 ndisks=24 iterations=1M time=5.719s MB/s=15026.92
>
This is the result where all data buffer pointers point to the same
memory, right? I.e., the zero page? So this is an unrealistic use
case that we can disregard.
> Custom test with memset (4kb block):
> | algo=neonx4 ndisks=24 iterations=1M time=6.165s MB/s=13939.08
> | algo=svex4 ndisks=24 iterations=1M time=5.839s MB/s=14718.23
>
> Even with the corrected memory setup, the throughput gap narrowed, but
> the fundamental CPU-efficiency result remained fully intact.
>
Sorry but your result that SVE is 2x faster does not remain fully intact,
right? Given that the speedup is now 5.5%?
Should we just disregard the above results (and explanations) and focus
on the stuff below?
> To completely isolate these variables and provide accurate real-world
> data, the following test campaigns were done based on the SnapRAID
> project (https://github.com/amadvance/snapraid) using its
> perf_bench.c tool with proper memory displacement and a 256 KiB block
> size.
>
> Test configurations:
> - c7g.medium (AWS Graviton3, 1 vCPU): Neoverse-V1, 256-bit SVE
> - c7g.xlarge (AWS Graviton3, 4 vCPUs): Neoverse-V1, 256-bit SVE
> - c8g.xlarge (AWS Graviton4, 4 vCPUs): Neoverse-V2, 128-bit SVE
>
>
> =========================================================
> Section 1: SnapRAID Validation on Graviton3 / Neoverse-V1
> =========================================================
>
...
>
> 1.3 Main Graviton3 Conclusions
> - On 256-bit SVE hardware, svex4 consistently retires about ~34% fewer
> instructions and ~10-15% fewer CPU cycles than neonx4.
>
> =========================================================
> Section 2: SnapRAID Validation on Graviton4 / Neoverse-V2
> =========================================================
>
...
>
> 2.3 Main Graviton4 Conclusions
> - On Neoverse-V2, SVE vector length is 128-bit (same as NEON).
> - Without the 256-bit width, NEON outperforms SVE.
> - svex4 retires ~32% MORE instructions here and is consistently slower.
>
> =========================================================
> Section 3: Validation on c7g.medium (1 vCPU)
> =========================================================
>
...
> 3.3 Main c7g.medium Conclusions
> - The instruction count reduction (~34%) perfectly matches the 4-vCPU
> instance.
> - The single vCPU is heavily memory-bandwidth constrained (cycle counts
> are much higher waiting for RAM).
>
OK, so the takeaway here is that SVE is only worth the hassle if the vector
length is at least 256 bits. This is not entirely surprising, but given that
Graviton4 went back to 128 bit vectors from 256, I wonder what the future
expectation is here.
But having these numbers is definitely a good first step. Now we need to
quantify the overhead associated with having kernel mode SVE state that
needs to be preserved/restored.
However, 10%-15% speedup that can only be achieved on SVE implementations
with 256 bit vectors or more may not be that enticing in the end. (The
fact that you are retiring 34% instructions less does not really matter
here unless there is some meaningful SMT-like sharing of functional units
going on in the meantime, which seems unlikely on a CPU that is maxed out
on the data side)
> =========================================================
> Section 4: The Pitfalls of the Current Kernel Benchmark
> =========================================================
>
These results seem very relevant - perhaps Christoph can give some guidance
on how we might use these to improve the built-in benchmarks to be more
accurate.
Thanks,
^ permalink raw reply
* Re: [RFC PATCH 2/2] kernel/module: Decouple klp and ftrace from load_module
From: Petr Mladek @ 2026-04-16 13:09 UTC (permalink / raw)
To: Song Chen
Cc: Petr Pavlu, rafael, lenb, mturquette, sboyd, viresh.kumar, agk,
snitzer, mpatocka, bmarzins, song, yukuai, linan122, jason.wessel,
danielt, dianders, horms, davem, edumazet, kuba, pabeni, paulmck,
frederic, mcgrof, da.gomez, samitolvanen, atomlin, jpoimboe,
jikos, mbenes, joe.lawrence, rostedt, mhiramat, mark.rutland,
mathieu.desnoyers, linux-modules, linux-kernel,
linux-trace-kernel, linux-acpi, linux-clk, linux-pm,
live-patching, dm-devel, linux-raid, kgdb-bugreport, netdev
In-Reply-To: <a35f5f94-7d5a-4347-974b-b270c89ef241@189.cn>
On Wed 2026-04-15 14:43:53, Song Chen wrote:
> Hi,
>
> On 4/14/26 22:33, Petr Pavlu wrote:
> > On 4/13/26 10:07 AM, chensong_2000@189.cn wrote:
> > > From: Song Chen <chensong_2000@189.cn>
> > >
> > > ftrace and livepatch currently have their module load/unload callbacks
> > > hard-coded in the module loader as direct function calls to
> > > ftrace_module_enable(), klp_module_coming(), klp_module_going()
> > > and ftrace_release_mod(). This tight coupling was originally introduced
> > > to enforce strict call ordering that could not be guaranteed by the
> > > module notifier chain, which only supported forward traversal. Their
> > > notifiers were moved in and out back and forth. see [1] and [2].
> >
> > I'm unclear about what is meant by the notifiers being moved back and
> > forth. The links point to patches that converted ftrace+klp from using
> > module notifiers to explicit callbacks due to ordering issues, but this
> > switch occurred only once. Have there been other attempts to use
> > notifiers again?
> >
> > > diff --git a/include/linux/module.h b/include/linux/module.h
> > > index 14f391b186c6..0bdd56f9defd 100644
> > > --- a/include/linux/module.h
> > > +++ b/include/linux/module.h
> > > @@ -308,6 +308,14 @@ enum module_state {
> > > MODULE_STATE_COMING, /* Full formed, running module_init. */
> > > MODULE_STATE_GOING, /* Going away. */
> > > MODULE_STATE_UNFORMED, /* Still setting it up. */
> > > + MODULE_STATE_FORMED,
> >
> > I don't see a reason to add a new module state. Why is it necessary and
> > how does it fit with the existing states?
> >
> because once notifier fails in state MODULE_STATE_UNFORMED (now only ftrace
> has someting to do in this state), notifier chain will roll back by calling
> blocking_notifier_call_chain_robust, i'm afraid MODULE_STATE_GOING is going
> to jeopardise the notifers which don't handle it appropriately, like:
>
> case MODULE_STATE_COMING:
> kmalloc();
> case MODULE_STATE_GOING:
> kfree();
>
>
> > > +};
> > > +
> > > +enum module_notifier_prio {
> > > + MODULE_NOTIFIER_PRIO_LOW = INT_MIN, /* Low prioroty, coming last, going first */
> > > + MODULE_NOTIFIER_PRIO_MID = 0, /* Normal priority. */
> > > + MODULE_NOTIFIER_PRIO_SECOND_HIGH = INT_MAX - 1, /* Second high priorigy, coming second*/
> > > + MODULE_NOTIFIER_PRIO_HIGH = INT_MAX, /* High priorigy, coming first, going late. */
> >
> > I suggest being explicit about how the notifiers are ordered. For
> > example:
> >
> > enum module_notifier_prio {
> > MODULE_NOTIFIER_PRIO_NORMAL, /* Normal priority, coming last, going first. */
> > MODULE_NOTIFIER_PRIO_LIVEPATCH,
> > MODULE_NOTIFIER_PRIO_FTRACE, /* High priority, coming first, going late. */
> > };
> >
I like the explicit PRIO_LIVEPATCH/FTRACE names.
But I would keep the INT_MAX - 1 and INT_MAX priorities. I believe
that ftrace/livepatching will always be the first/last to call.
And INT_MAX would help to preserve kABI when PRIO_NORMAL is not
enough for the rest of notifiers.
That said, I am not sure whether this is worth the effort.
This patch tries to move the explicit callbacks in a generic
notifiers API. But it will still need to use some explictly
defined (reserved) priorities. And it will
not guarantee a misuse. Also it requires the double linked
list which complicates the notifiers code.
> > > };
> > > struct mod_tree_node {
> > > --- a/kernel/module/main.c
> > > +++ b/kernel/module/main.c
> > > @@ -3281,20 +3277,14 @@ static int complete_formation(struct module *mod, struct load_info *info)
> > > return err;
> > > }
> > > -static int prepare_coming_module(struct module *mod)
> > > +static int prepare_module_state_transaction(struct module *mod,
> > > + unsigned long val_up, unsigned long val_down)
> > > {
> > > int err;
> > > - ftrace_module_enable(mod);
> > > - err = klp_module_coming(mod);
> > > - if (err)
> > > - return err;
> > > -
> > > err = blocking_notifier_call_chain_robust(&module_notify_list,
> > > - MODULE_STATE_COMING, MODULE_STATE_GOING, mod);
> > > + val_up, val_down, mod);
> > > err = notifier_to_errno(err);
> > > - if (err)
> > > - klp_module_going(mod);
> > > return err;
> > > }
I personally find the name "prepare_module_state_transaction"
misleading. What is the "transaction" here? If this was a "preparation"
step then where is the transaction done/finished?
It might be better to just opencode the
blocking_notifier_call_chain_robust() instead.
> > > @@ -3468,14 +3458,21 @@ static int load_module(struct load_info *info, const char __user *uargs,
> > > init_build_id(mod, info);
> > > /* Ftrace init must be called in the MODULE_STATE_UNFORMED state */
> > > - ftrace_module_init(mod);
> > > + err = prepare_module_state_transaction(mod,
> > > + MODULE_STATE_UNFORMED, MODULE_STATE_FORMED);
> >
> > I believe val_down should be MODULE_STATE_GOING to reverse the
> > operation. Why is the new state MODULE_STATE_FORMED needed here?
> to avoid this:
>
> case MODULE_STATE_COMING:
> kmalloc();
> case MODULE_STATE_GOING:
> kfree();
Hmm, the module is in "FORMED" state here.
> > > + if (err)
> > > + goto ddebug_cleanup;
> > > /* Finally it's fully formed, ready to start executing. */
> > > err = complete_formation(mod, info);
And we call "complete_formation()" function. This sounds like
it was not really "FORMED" before. => It is confusing and nono.
Please, try to avoid the new state if possible. My experience
with reading the module loader code is that any new state
brings a lot of complexity. You need to take it into account
when checking correctness of other changes, features, ...
Something tells me that if the state was not needed before
then we could avoid it.
> > > - if (err)
> > > + if (err) {
> > > + blocking_notifier_call_chain_reverse(&module_notify_list,
> > > + MODULE_STATE_FORMED, mod);
> > > goto ddebug_cleanup;
> > > + }
> > > - err = prepare_coming_module(mod);
> > > + err = prepare_module_state_transaction(mod,
> > > + MODULE_STATE_COMING, MODULE_STATE_GOING);
> > > if (err)
> > > goto bug_cleanup;
> > > --- a/kernel/trace/ftrace.c
> > > +++ b/kernel/trace/ftrace.c
> > > @@ -5241,6 +5241,44 @@ static int __init ftrace_mod_cmd_init(void)
> > > }
> > > core_initcall(ftrace_mod_cmd_init);
> > > +static int ftrace_module_callback(struct notifier_block *nb, unsigned long op,
> > > + void *module)
> > > +{
> > > + struct module *mod = module;
> > > +
> > > + switch (op) {
> > > + case MODULE_STATE_UNFORMED:
> > > + ftrace_module_init(mod);
> > > + break;
> > > + case MODULE_STATE_COMING:
> > > + ftrace_module_enable(mod);
> > > + break;
> > > + case MODULE_STATE_LIVE:
> > > + ftrace_free_mem(mod, mod->mem[MOD_INIT_TEXT].base,
> > > + mod->mem[MOD_INIT_TEXT].base + mod->mem[MOD_INIT_TEXT].size);
> > > + break;
> > > + case MODULE_STATE_GOING:
> > > + case MODULE_STATE_FORMED:
> > > + ftrace_release_mod(mod);
This calls "release" in a "FORMED" state. It does not make any
sense. Something looks fishy, either the code or the naming.
> > > + break;
> > > + default:
> > > + break;
> > > + }
> >
I am sorry for being so picky about names. I believe that good names
help to prevent bugs and reduce headaches.
Best Regards,
Petr
^ permalink raw reply
* Re: [PATCH v2] raid6: arm64: add SVE optimized implementation for syndrome generation
From: Demian Shulhan @ 2026-04-16 12:40 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Ard Biesheuvel, Mark Rutland, Song Liu, Yu Kuai, Will Deacon,
Catalin Marinas, Mark Brown, linux-arm-kernel, robin.murphy,
Li Nan, linux-raid, linux-kernel
In-Reply-To: <CAOLeWCtnZz=kHGk4C2f9Mbfi3tEBE100iCkMBhcG47dmR9eDWw@mail.gmail.com>
Hi all,
Sorry for the delay. The tests became more complex than I initially
thought, so I needed to gather more data and properly validate the
results across different hardware configurations.
Firstly, I want to clarify the results from my March 29 tests. I found
a flaw in my initial custom benchmark. The massive 2x throughput gap on
24 disks wasn't solely due to SVE's superiority, but rather a severe L1
D-Cache thrashing issue that disproportionately penalized NEON.
My custom test lacked memset() initialization, causing all data buffers
to map to the Linux Zero Page (Virtually Indexed, Physically Tagged
cache aliasing). Furthermore, even with memset(), allocating contiguous
page-aligned buffers can causes severe Cache Address Sharing (a known
issue that Andrea Mazzoleni solved in SnapRAID 13 years ago using
RAID_MALLOC_DISPLACEMENT).
Because SVE (svex4) uses 256-bit registers on Neoverse-V1, it performs
exactly half the number of memory load instructions compared to 128-bit
NEON. This dramatically reduced the L1 cache alias thrashing, allowing
SVE to survive the memory bottleneck while NEON choked:
Custom test without memset (4kb block):
| algo=neonx4 ndisks=24 iterations=1M time=11.014s MB/s=7802.57
| algo=svex4 ndisks=24 iterations=1M time=5.719s MB/s=15026.92
Custom test with memset (4kb block):
| algo=neonx4 ndisks=24 iterations=1M time=6.165s MB/s=13939.08
| algo=svex4 ndisks=24 iterations=1M time=5.839s MB/s=14718.23
Even with the corrected memory setup, the throughput gap narrowed, but
the fundamental CPU-efficiency result remained fully intact.
To completely isolate these variables and provide accurate real-world
data, the following test campaigns were done based on the SnapRAID
project (https://github.com/amadvance/snapraid) using its
perf_bench.c tool with proper memory displacement and a 256 KiB block
size.
Test configurations:
- c7g.medium (AWS Graviton3, 1 vCPU): Neoverse-V1, 256-bit SVE
- c7g.xlarge (AWS Graviton3, 4 vCPUs): Neoverse-V1, 256-bit SVE
- c8g.xlarge (AWS Graviton4, 4 vCPUs): Neoverse-V2, 128-bit SVE
=========================================================
Section 1: SnapRAID Validation on Graviton3 / Neoverse-V1
=========================================================
These runs are the most representative userspace validation. The tests
were run with standard -O2 optimizations.
1.1 SnapRAID speedtest, O2, c7g.xlarge (Raw Throughput)
disks neonx4 neonx8 svex4 delta(nx4) delta(nx8)
----- ------ ------ ------ ---------- ----------
8 21394 21138 23601 +10.3% +11.6%
24 20368 19850 21009 +3.1% +5.8%
48 16727 19290 20222 +20.9% +4.8%
96 15562 18925 17549 +12.8% -7.3%
1.2 perf_bench, O2, c7g.xlarge (Hardware Efficiency)
disks neonx4 inst svex4 inst reduction | neonx4 cyc svex4 cyc | MB/s (N/S)
----- ----------- ---------- --------- | ---------- --------- | -----------
8 4.02 B 2.61 B -35.1% | 1.01 B 0.92 B | 20304/22346
24 12.16 B 8.00 B -34.2% | 3.20 B 3.11 B | 19354/19933
48 24.37 B 16.08 B -34.0% | 7.73 B 6.51 B | 16048/19047
96 48.80 B 32.24 B -33.9% | 16.94 B 15.11 B | 14638/16421
1.3 Main Graviton3 Conclusions
- On 256-bit SVE hardware, svex4 consistently retires about ~34% fewer
instructions and ~10-15% fewer CPU cycles than neonx4.
=========================================================
Section 2: SnapRAID Validation on Graviton4 / Neoverse-V2
=========================================================
2.1 SnapRAID speedtest, O2, c8g.xlarge (Raw Throughput)
disks neonx4 neonx8 svex4 delta(nx4) delta(nx8)
----- ------ ------ ------ ---------- ----------
8 24802 25409 20451 -17.5% -19.5%
24 22607 24026 18577 -17.8% -22.7%
48 20984 22171 18019 -14.1% -18.7%
96 21254 21690 17108 -19.5% -21.1%
2.2 perf_bench, O2, c8g.xlarge (Hardware Efficiency)
disks neonx4 inst svex4 inst overhead | neonx4 cyc svex4 cyc | MB/s (N/S)
----- ----------- ---------- --------- | ---------- --------- | -----------
8 4.02 B 5.22 B +29.9% | 0.95 B 1.14 B | 23529/19512
24 12.16 B 15.98 B +31.4% | 3.11 B 3.79 B | 21621/17777
48 24.37 B 32.12 B +31.8% | 6.70 B 7.81 B | 20000/17204
96 48.78 B 64.40 B +32.0% | 13.24 B 16.32 B | 20253/16410
2.3 Main Graviton4 Conclusions
- On Neoverse-V2, SVE vector length is 128-bit (same as NEON).
- Without the 256-bit width, NEON outperforms SVE.
- svex4 retires ~32% MORE instructions here and is consistently slower.
=========================================================
Section 3: Validation on c7g.medium (1 vCPU)
=========================================================
3.1 SnapRAID speedtest, O2, c7g.medium (Raw Throughput)
disks neonx4 neonx8 svex4 delta(nx4) delta(nx8)
----- ------ ------ ------ ---------- ----------
8 16768 17466 17310 +3.2% -0.9%
24 15843 16684 16205 +2.3% -2.9%
48 14032 14475 15389 +9.7% +6.3%
96 13404 13045 14677 +9.5% +12.5%
3.2 perf_bench, O2, c7g.medium (Hardware Efficiency)
disks neonx4 inst svex4 inst reduction | neonx4 cyc svex4 cyc | MB/s (N/S)
----- ----------- ---------- --------- | ---------- --------- | -----------
8 3.99 B 2.61 B -34.6% | 1.30 B 1.25 B | 16000/16666
24 12.13 B 8.00 B -34.0% | 4.08 B 4.02 B | 15189/15483
48 24.34 B 16.08 B -33.9% | 9.23 B 8.35 B | 13445/14860
96 48.76 B 32.24 B -33.9% | 19.34 B 17.92 B | 12834/13852
3.3 Main c7g.medium Conclusions
- The instruction count reduction (~34%) perfectly matches the 4-vCPU
instance.
- The single vCPU is heavily memory-bandwidth constrained (cycle counts
are much higher waiting for RAM).
=========================================================
Section 4: The Pitfalls of the Current Kernel Benchmark
=========================================================
As Christoph pointed out, the current in-kernel benchmark setup
(hardcoded to 8 disks and a PAGE_SIZE buffer) can be not representative
of real-life arrays.
Because 8 disks * 4 KiB = 32 KiB total data, the entire benchmark fits
into the 64 KiB L1 D-Cache of Neoverse-V1, masking memory bandwidth limits
and register spilling. This leads to objectively wrong selections.
---------------------------------------------------
Case 1: Wrong NEON unrolling selection (Graviton3)
--------------------------------------------------
The kernel benchmark tests 8 disks and locks in neonx4. However, on
real-world wide arrays (48-96 disks), neonx8 is significantly faster.
disks neonx4 MB/s neonx8 MB/s Actual Winner Kernel's Choice
-------- ------------- ------------- ------------- ---------------
8 (Boot) 21,394 21,138 neonx4 neonx4 (Locked)
48 16,727 19,290 neonx8 neonx4 (-15.3%)
96 15,562 18,925 neonx8 neonx4 (-21.6%)
Result: Users lose up to 21% NEON throughput because of the 8-disk test.
---------------------------------------------------
Case 2: Wrong SVE vs NEON selection (Graviton3)
--------------------------------------------------
If SVE is enabled, the 8-disk benchmark strongly prefers svex4. But on
extreme wide arrays (96 disks), the heavily unrolled neonx8 actually
overtakes SVE.
disks neonx8 MB/s svex4 MB/s Actual Winner Kernel's Choice
-------- ------------- ------------- ------------- ---------------
8 (Boot) 21,138 23,601 svex4 svex4 (Locked)
96 18,925 17,549 neonx8 svex4 (-7.8%)
Result: On extreme workloads, forcing svex4 loses ~7.8% throughput.
Conclusion: The kernel benchmark requires testing with larger buffers
(exceeding L1 capacity) or simulated wide arrays to guarantee the optimal
algorithm is chosen for actual storage workloads.
---------------------------------------------------
Case 3: Buffer size distortion (Graviton3, 8 disks)
---------------------------------------------------
Even on the exact same 8-disk array, testing with a 4 KiB buffer (which
fits entirely in the L1 cache) yields a completely different winner than
testing with 256 KiB buffer (which exercises L2/L3/RAM).
buffer neonx4 MB/s svex4 MB/s Actual Winner Kernel's Choice
---------- ------------- ------------- ------------- ---------------
4 KiB 20211 19818 neonx4 neonx4 (Locked)
256 KiB 21394 23601 svex4 neonx4 (-9.3%)
Result: By benchmarking exclusively in the L1 cache (4 KiB buffer), the
kernel incorrectly chooses neonx4, losing ~9.3% throughput for
larger I/O block sizes.
Thanks again for your time and review!
вт, 31 бер. 2026 р. о 16:18 Demian Shulhan <demyansh@gmail.com> пише:
>
> Hi all,
>
> Ard, your questions regarding real-world I/O bottlenecks and SVE power
> efficiency versus raw throughput are entirely valid. I agree that
> introducing SVE support requires solid real-world data to justify the
> added complexity.
>
> Due to my current workload, I won't be able to run the necessary
> hardware tests and prepare the benchmark code immediately. I will get
> back to the list in about 1 week with the requested source code,
> unmangled test results, and further analysis.
>
> Thanks!
>
>
> вт, 31 бер. 2026 р. о 09:37 Christoph Hellwig <hch@lst.de> пише:
> >
> > On Mon, Mar 30, 2026 at 06:39:49PM +0200, Ard Biesheuvel wrote:
> > > I think the results are impressive, but I'd like to better understand
> > > its implications on a real-world scenario. Is this code only a
> > > bottleneck when rebuilding an array?
> >
> > The syndrome generation is run every time you write data to a RAID6
> > array, and if you do partial stripe writes it (or rather the XOR
> > variant) is run twice. So this is the most performance critical
> > path for writing to RAID6.
> >
> > Rebuild usually runs totally different code, but can end up here as well
> > when both parity disks are lost.
> >
> > > > Furthermore, as Christoph suggested, I tested scalability on wider
> > > > arrays since the default kernel benchmark is hardcoded to 8 disks,
> > > > which doesn't give the unrolled SVE loop enough data to shine. On a
> > > > 16-disk array, svex4 hits 15.1 GB/s compared to 8.0 GB/s for neonx4.
> > > > On a 24-disk array, while neonx4 chokes and drops to 7.8 GB/s, svex4
> > > > maintains a stable 15.0 GB/s — effectively doubling the throughput.
> > >
> > > Does this mean the kernel benchmark is no longer fit for purpose? If
> > > it cannot distinguish between implementations that differ in performance
> > > by a factor of 2, I don't think we can rely on it to pick the optimal one.
> >
> > It is not good, and we should either fix it or run more than one.
> > The current setup is not really representative of real-life array.
> > It also leads to wrong selections on x86, but only at the which unroll
> > level to pick level, and only for minor differences so far. I plan
> > to add this to the next version of the raid6 lib patches.
> >
^ permalink raw reply
* Re: [RFC PATCH 1/2] kernel/notifier: replace single-linked list with double-linked list for reverse traversal
From: David Laight @ 2026-04-16 12:30 UTC (permalink / raw)
To: chensong_2000
Cc: rafael, lenb, mturquette, sboyd, viresh.kumar, agk, snitzer,
mpatocka, bmarzins, song, yukuai, linan122, jason.wessel, danielt,
dianders, horms, davem, edumazet, kuba, pabeni, paulmck, frederic,
mcgrof, petr.pavlu, da.gomez, samitolvanen, atomlin, jpoimboe,
jikos, mbenes, pmladek, joe.lawrence, rostedt, mhiramat,
mark.rutland, mathieu.desnoyers, linux-modules, linux-kernel,
linux-trace-kernel, linux-acpi, linux-clk, linux-pm,
live-patching, dm-devel, linux-raid, kgdb-bugreport, netdev
In-Reply-To: <20260415070137.17860-1-chensong_2000@189.cn>
On Wed, 15 Apr 2026 15:01:37 +0800
chensong_2000@189.cn wrote:
> From: Song Chen <chensong_2000@189.cn>
>
> The current notifier chain implementation uses a single-linked list
> (struct notifier_block *next), which only supports forward traversal
> in priority order. This makes it difficult to handle cleanup/teardown
> scenarios that require notifiers to be called in reverse priority order.
If it is only cleanup/teardown then the list can be order-reversed
as part of that process at the same time as the list is deleted.
David
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox