* Re: [PATCH] md/raid5: fix race between reshape and chunk-aligned read
From: FengWei Shih @ 2026-04-21 7:09 UTC (permalink / raw)
To: yukuai, song; +Cc: linan122, linux-raid, linux-kernel
In-Reply-To: <68b61dc3-f7fa-49ba-9f26-f07b7fa4768d@fnnas.com>
Hi Kuai,
Yu Kuai 於 2026/4/19 下午 01:14 寫道:
> Hi,
>
> 在 2026/4/9 13:17, FengWei Shih 写道:
>> raid5_make_request() checks mddev->reshape_position to decide whether
>> to allow chunk-aligned reads. However in raid5_start_reshape(), the
>> layout configuration (raid_disks, algorithm, etc.) is updated before
>> mddev->reshape_position is set:
>>
>> reshape (raid5_start_reshape) read (raid5_make_request)
>> ============================== ===========================
>> write_seqcount_begin
>> update raid_disks, algorithm...
>> set conf->reshape_progress
>> write_seqcount_end
>> check mddev->reshape_position
>> * still MaxSector, allow
>> raid5_read_one_chunk()
>> * use new layout
>> raid5_quiesce()
>> set mddev->reshape_position
> While we're here, I think it's pretty ugly to disable raid5_read_one_chunk
> when reshape is not fully done. So a better solution should be:
>
> - data behind reshape: read with new layout
> - data ahead of reshape: read with old layout, reshape will also need to wait
> for this IO to be done, before reshape can make progress.
> - data intersecting the active reshape window: wait for reshape to make progress.
Thanks for the feedback. I agree that using reshape_progress to distinguish
the three cases (ahead/behind/inside reshape) would be more refined.
But disabling chunk-aligned reads during reshape was already the
existing design.
In the original code, the check is at the caller level:
if (rw == READ && mddev->degraded == 0 &&
mddev->reshape_position == MaxSector) {
bi = chunk_aligned_read(mddev, bi);
}
My patch is focused on fixing the race condition in the existing
lockless check of whether
reshape is in progress. So just to confirm: are you suggesting we add a
mechanism to
allow chunk-aligned reads during reshape (based on reshape progress), rather
than simply disabling them?
>> Since reshape_position is not yet updated, raid5_make_request()
>> considers no reshape is in progress and proceeds with the
>> chunk-aligned path, but the layout has already changed, causing
>> raid5_compute_sector() to return an incorrect physical address.
>>
>> Fix this by reading conf->reshape_progress under gen_lock in
>> raid5_read_one_chunk() and falling back to the stripe path if a
>> reshape is in progress.
>>
>> Signed-off-by: FengWei Shih <dannyshih@synology.com>
>> ---
>> drivers/md/raid5.c | 8 ++++++++
>> 1 file changed, 8 insertions(+)
>>
>> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
>> index a8e8d431071b..bded2b86f0ef 100644
>> --- a/drivers/md/raid5.c
>> +++ b/drivers/md/raid5.c
>> @@ -5421,6 +5421,11 @@ static int raid5_read_one_chunk(struct mddev *mddev, struct bio *raid_bio)
>> sector_t sector, end_sector;
>> int dd_idx;
>> bool did_inc;
>> + int seq;
>> +
>> + seq = read_seqcount_begin(&conf->gen_lock);
>> + if (unlikely(conf->reshape_progress != MaxSector))
>> + return 0;
>>
>> if (!in_chunk_boundary(mddev, raid_bio)) {
>> pr_debug("%s: non aligned\n", __func__);
>> @@ -5431,6 +5436,9 @@ static int raid5_read_one_chunk(struct mddev *mddev, struct bio *raid_bio)
>> &dd_idx, NULL);
>> end_sector = sector + bio_sectors(raid_bio);
>>
>> + if (read_seqcount_retry(&conf->gen_lock, seq))
>> + return 0;
>> +
>> if (r5c_big_stripe_cached(conf, sector))
>> return 0;
>>
Thanks,
FengWei Shih
Disclaimer: The contents of this e-mail message and any attachments are confidential and are intended solely for addressee. The information may also be legally privileged. This transmission is sent in trust, for the sole purpose of delivery to the intended recipient. If you have received this transmission in error, any use, reproduction or dissemination of this transmission is strictly prohibited. If you are not the intended recipient, please immediately notify the sender by reply e-mail or phone and delete this message and its attachments, if any.
^ permalink raw reply
* Re: [PATCH v2 3/5] md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
From: Xiao Ni @ 2026-04-21 7:36 UTC (permalink / raw)
To: Su Yue; +Cc: Su Yue, linux-raid, song, linan122, yukuai, heming.zhao
In-Reply-To: <1pg982rx.fsf@damenly.org>
On Tue, Apr 21, 2026 at 10:29 AM Su Yue <l@damenly.org> wrote:
>
> On Mon 20 Apr 2026 at 15:05, Xiao Ni <xni@redhat.com> wrote:
>
> > On Tue, Apr 7, 2026 at 6:27 PM Su Yue <glass.su@suse.com> wrote:
> >>
> >> Before commit fb8cc3b0d9db ("md/md-bitmap: delay registration
> >> of bitmap_ops until creating bitmap")
> >> if CONFIG_MD_BITMAP is enabled, both bitmap none, internal and
> >> clustered have
> >> the sysfs file bitmap/location.
> >>
> >> After the commit, if bitmap is none, bitmap/location doesn't
> >> exist anymore.
> >> It breaks 'grow' behavior of a md array of madam with
> >> MD_FEATURE_BITMAP_OFFSET.
> >> Take level=mirror and metadata=1.2 as an example:
> >>
> >> $ mdadm --create /dev/md0 -f --bitmap=none --raid-devices=2
> >> --level=mirror \
> >> --metadata=1.2 /dev/vdd /dev/vde
> >> $ mdadm --grow /dev/md0 --bitmap=internal
> >> $ cat /sys/block/md0/md/bitmap/location
> >> Before:+8
> >> After: +2
> >>
> >> While growing bitmap from none to internal, clustered and
> >> llbitmap,
> >> mdadm/Grow.c:Grow_addbitmap() tries to detect bitmap/location
> >> first.
> >> 1)If bitmap/location exists, it sets bitmap/location after
> >> getinfo_super().
> >> 2)If bitmap/location doesn't exist, mdadm just calls
> >> md_set_array_info() then
> >> mddev->bitmap_info.default_offset will be used.
> >> Situation can be worse if growing none to clustered, bitmap
> >> offset of the node
> >> calling `madm --grow` will be changed but the other node are
> >> reading bitmap sb from
> >> the old location.
> >>
> >> Here introducing a dummy bitmap_operations for ID_BITMAP_NONE
> >> to restore sysfs
> >> file bitmap/location for ID_BITMAP_NONE and ID_BITMAP.
> >> location_store() now calls md_bitmap_(create|destroy) with
> >> false sysfs parameter then
> >> (add,remove)_internal_bitmap() will be called to manage sysfs
> >> entries.
> >>
> >> Fixes: fb8cc3b0d9db ("md/md-bitmap: delay registration of
> >> bitmap_ops until creating bitmap")
> >> Suggested-by: Yu Kuai <yukuai@fnnas.com>
> >> Signed-off-by: Su Yue <glass.su@suse.com>
> >> ---
> >> drivers/md/md-bitmap.c | 116
> >> ++++++++++++++++++++++++++++++++++++---
> >> drivers/md/md-bitmap.h | 3 +
> >> drivers/md/md-llbitmap.c | 12 ++++
> >> drivers/md/md.c | 16 +++---
> >> 4 files changed, 130 insertions(+), 17 deletions(-)
> >>
> >> diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c
> >> index ac06c9647bf0..a8a176428c61 100644
> >> --- a/drivers/md/md-bitmap.c
> >> +++ b/drivers/md/md-bitmap.c
> >> @@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data,
> >> bool flush)
> >> bitmap->storage.filemap != NULL;
> >> }
> >>
> >> +static bool dummy_bitmap_enabled(void *data, bool flush)
> >> +{
> >> + return false;
> >> +}
> >> +
> >> /*
> >> * check a page and, if necessary, allocate it (or hijack it
> >> if the alloc fails)
> >> *
> >> @@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev
> >> *mddev)
> >> return 0;
> >> }
> >>
> >> +static int dummy_bitmap_create(struct mddev *mddev)
> >> +{
> >> + return 0;
> >> +}
> >> +
> >> static int bitmap_load(struct mddev *mddev)
> >> {
> >> int err = 0;
> >> @@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev, char
> >> *page)
> >> return len;
> >> }
> >>
> >> +static int create_internal_group(struct mddev *mddev);
> >> +static void remove_internal_group(struct mddev *mddev);
> >> +
> >> static ssize_t
> >> location_store(struct mddev *mddev, const char *buf, size_t
> >> len)
> >> {
> >> @@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev, const
> >> char *buf, size_t len)
> >> goto out;
> >> }
> >>
> >> - md_bitmap_destroy(mddev, true);
> >> + md_bitmap_destroy(mddev, false);
> >> + remove_internal_group(mddev);
> >> mddev->bitmap_info.offset = 0;
> >> if (mddev->bitmap_info.file) {
> >> struct file *f =
> >> mddev->bitmap_info.file;
> >> @@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev,
> >> const char *buf, size_t len)
> >> */
> >> mddev->bitmap_id = ID_BITMAP;
> >> mddev->bitmap_info.offset = offset;
> >> - rv = md_bitmap_create(mddev, true);
> >> + rv = md_bitmap_create(mddev, false);
> >> if (rv)
> >> goto out;
> >>
> >> + rv = create_internal_group(mddev);
> >> + if (rv) {
> >> + mddev->bitmap_info.offset = 0;
> >> + bitmap_destroy(mddev);
> >> + goto out;
> >> + }
> >> +
> >> rv = bitmap_load(mddev);
> >> if (rv) {
> >> + remove_internal_group(mddev);
> >> mddev->bitmap_info.offset = 0;
> >> - md_bitmap_destroy(mddev, true);
> >> + md_bitmap_destroy(mddev,
> >> false);
> >> goto out;
> >> }
> >> }
> >> @@ -2960,8 +2982,7 @@ static struct md_sysfs_entry
> >> max_backlog_used =
> >> __ATTR(max_backlog_used, S_IRUGO | S_IWUSR,
> >> behind_writes_used_show, behind_writes_used_reset);
> >>
> >> -static struct attribute *md_bitmap_attrs[] = {
> >> - &bitmap_location.attr,
> >> +static struct attribute *internal_bitmap_attrs[] = {
> >> &bitmap_space.attr,
> >> &bitmap_timeout.attr,
> >> &bitmap_backlog.attr,
> >> @@ -2972,11 +2993,57 @@ static struct attribute
> >> *md_bitmap_attrs[] = {
> >> NULL
> >> };
> >>
> >> -static struct attribute_group md_bitmap_group = {
> >> +static struct attribute_group internal_bitmap_group = {
> >> .name = "bitmap",
> >> - .attrs = md_bitmap_attrs,
> >> + .attrs = internal_bitmap_attrs,
> >> };
> >>
> >> +/* Only necessary attrs for compatibility */
> >> +static struct attribute *common_bitmap_attrs[] = {
> >> + &bitmap_location.attr,
> >> + NULL
> >> +};
> >> +
> >> +static const struct attribute_group common_bitmap_group = {
> >> + .name = "bitmap",
> >> + .attrs = common_bitmap_attrs,
> >> +};
> >> +
> >> +static int create_internal_group(struct mddev *mddev)
> >> +{
> >> + /*
> >> + * md_bitmap_group and md_bitmap_common_group are using
> >> same name
> >> + * 'bitmap'.
> >> + */
> >> + return sysfs_merge_group(&mddev->kobj,
> >> &internal_bitmap_group);
> >> +}
> >> +
> >> +static void remove_internal_group(struct mddev *mddev)
> >> +{
> >> + sysfs_unmerge_group(&mddev->kobj,
> >> &internal_bitmap_group);
> >> +}
> >> +
> >> +static int bitmap_register_groups(struct mddev *mddev)
> >> +{
> >> + int ret;
> >> +
> >> + ret = sysfs_create_group(&mddev->kobj,
> >> &common_bitmap_group);
> >> +
> >> + if (ret)
> >> + return ret;
> >> +
> >> + ret = sysfs_merge_group(&mddev->kobj,
> >> &internal_bitmap_group);
> >> + if (ret)
> >> + sysfs_remove_group(&mddev->kobj,
> >> &common_bitmap_group);
> >> +
> >> + return ret;
> >> +}
> >> +
> >> +static void bitmap_unregister_groups(struct mddev *mddev)
> >> +{
> >> + sysfs_unmerge_group(&mddev->kobj,
> >> &internal_bitmap_group);
> >> +}
> >
> > Hi Su
> >
> > bitmap_unregister_groups should also remove common_bitmap_group,
> > right?
> >
>
> As you pasted before, mdadm:Grow.c:
>
> int Grow_addbitmap(char *devname, int fd, struct context *c,
> struct shape *s)
> {
> ...
> if (array.state & (1 << MD_SB_BITMAP_PRESENT)) {
> if (s->btype == BitmapNone) {
> array.state &= ~(1 << MD_SB_BITMAP_PRESENT);
> if (md_set_array_info(fd, &array) != 0) {
> if (array.state & (1 << MD_SB_CLUSTERED))
> pr_err("failed to remove clustered
> bitmap.\n");
> else
> pr_err("failed to remove internal bitmap.\n");
> return 1;
> }
> return 0;
> }
> pr_err("bitmap already present on %s\n", devname);
> return 1;
> }
> ...
> }
>
> In case of growing from internal to none,
> bitmap_unregister_groups() will
> be called, if common_bitmap_group is removed, bitmap/location
> won't exist.
> update_array_info() is a common function used by many call paths.
> Also
> llbitmap is involved here. I don't want to make situation and code
> more
> complicated like adding more codes in update_array_info().
The above mdadm codes use bitmap/location. In your patch, you already
pass false to md_bitmap_destroy in location_store. So removing
common_bitmap_group in bitmap_unregister_groups can't affect the above
case.
But after reading your below comments. It's good to me that
bitmap_unregister_groups doesn't remove common_bitmap_group.
>
>
> >> +
> >> static struct bitmap_operations bitmap_ops = {
> >
> > You already changed md_bitmap_group to internal_bitmap_group.
> > It's
> > better to change bitmap_ops to internal_bitmap_ops?
> >
> >> .head = {
> >> .type = MD_BITMAP,
> >> @@ -3018,21 +3085,52 @@ static struct bitmap_operations
> >> bitmap_ops = {
> >> .set_pages = bitmap_set_pages,
> >> .free = md_bitmap_free,
> >>
> >> - .group = &md_bitmap_group,
> >> + .register_groups = bitmap_register_groups,
> >> + .unregister_groups = bitmap_unregister_groups,
> >> +};
> >> +
> >> +static int none_bitmap_register_groups(struct mddev *mddev)
> >> +{
> >> + return sysfs_create_group(&mddev->kobj,
> >> &common_bitmap_group);
> >> +}
> >> +
> >> +static struct bitmap_operations none_bitmap_ops = {
> >> + .head = {
> >> + .type = MD_BITMAP,
> >> + .id = ID_BITMAP_NONE,
> >> + .name = "none",
> >> + },
> >> +
> >> + .enabled = dummy_bitmap_enabled,
> >> + .create = dummy_bitmap_create,
> >> + .destroy = bitmap_destroy,
> >> + .load = bitmap_load,
> >> + .get_stats = bitmap_get_stats,
> >> + .free = md_bitmap_free,
> >> +
> >> + .register_groups = none_bitmap_register_groups,
> >> + .unregister_groups = NULL,
> >
> > How does bitmap/location can be deleted if array is created with
> > no
> > bitmap? Can the `mdadm --stop` command get stuck?
> >
> No. It wont get stuck.
>
> I guess here your concern is the timing of removing
> common_bitmap_group.
> The life cycle is same as before fb8cc3b0d9db. At the time
> llbitmap is
> not introduced, because there is no need to switch bitmap ops, so
> all
> attrs of bitmap are removed in kobject_put() in
> mddev_delayed_delete()
> queued by mddev_put(). This is now where common_bitmap_group is
> being removed.
Thanks for the explanation.
Best Regards
Xiao
>
> --
> Su
> > Best Regards
> > Xiao
> >
> >> };
> >>
> >> int md_bitmap_init(void)
> >> {
> >> + int ret;
> >> +
> >> md_bitmap_wq = alloc_workqueue("md_bitmap",
> >> WQ_MEM_RECLAIM | WQ_UNBOUND,
> >> 0);
> >> if (!md_bitmap_wq)
> >> return -ENOMEM;
> >>
> >> - return register_md_submodule(&bitmap_ops.head);
> >> + ret = register_md_submodule(&bitmap_ops.head);
> >> + if (ret)
> >> + return ret;
> >> +
> >> + return register_md_submodule(&none_bitmap_ops.head);
> >> }
> >>
> >> void md_bitmap_exit(void)
> >> {
> >> destroy_workqueue(md_bitmap_wq);
> >> unregister_md_submodule(&bitmap_ops.head);
> >> + unregister_md_submodule(&none_bitmap_ops.head);
> >> }
> >> diff --git a/drivers/md/md-bitmap.h b/drivers/md/md-bitmap.h
> >> index b42a28fa83a0..10bc6854798c 100644
> >> --- a/drivers/md/md-bitmap.h
> >> +++ b/drivers/md/md-bitmap.h
> >> @@ -125,6 +125,9 @@ struct bitmap_operations {
> >> void (*set_pages)(void *data, unsigned long pages);
> >> void (*free)(void *data);
> >>
> >> + int (*register_groups)(struct mddev *mddev);
> >> + void (*unregister_groups)(struct mddev *mddev);
> >> +
> >> struct attribute_group *group;
> >> };
> >>
> >> diff --git a/drivers/md/md-llbitmap.c
> >> b/drivers/md/md-llbitmap.c
> >> index bf398d7476b3..9b3ea4f1d268 100644
> >> --- a/drivers/md/md-llbitmap.c
> >> +++ b/drivers/md/md-llbitmap.c
> >> @@ -1561,6 +1561,16 @@ static struct attribute_group
> >> md_llbitmap_group = {
> >> .attrs = md_llbitmap_attrs,
> >> };
> >>
> >> +static int llbitmap_register_groups(struct mddev *mddev)
> >> +{
> >> + return sysfs_create_group(&mddev->kobj,
> >> &md_llbitmap_group);
> >> +}
> >> +
> >> +static void llbitmap_unregister_groups(struct mddev *mddev)
> >> +{
> >> + sysfs_remove_group(&mddev->kobj, &md_llbitmap_group);
> >> +}
> >> +
> >> static struct bitmap_operations llbitmap_ops = {
> >> .head = {
> >> .type = MD_BITMAP,
> >> @@ -1597,6 +1607,8 @@ static struct bitmap_operations
> >> llbitmap_ops = {
> >> .dirty_bits = llbitmap_dirty_bits,
> >> .write_all = llbitmap_write_all,
> >>
> >> + .register_groups = llbitmap_register_groups,
> >> + .unregister_groups = llbitmap_unregister_groups,
> >> .group = &md_llbitmap_group,
> >> };
> >>
> >> diff --git a/drivers/md/md.c b/drivers/md/md.c
> >> index d3c8f77b4fe3..55a95b227b83 100644
> >> --- a/drivers/md/md.c
> >> +++ b/drivers/md/md.c
> >> @@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct
> >> mddev *mddev, bool create_sysfs)
> >> struct bitmap_operations *old = mddev->bitmap_ops;
> >> struct md_submodule_head *head;
> >>
> >> - if (mddev->bitmap_id == ID_BITMAP_NONE ||
> >> - (old && old->head.id == mddev->bitmap_id))
> >> + if (old && old->head.id == mddev->bitmap_id)
> >> return true;
> >>
> >> xa_lock(&md_submodule);
> >> @@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct
> >> mddev *mddev, bool create_sysfs)
> >> mddev->bitmap_ops = (void *)head;
> >> xa_unlock(&md_submodule);
> >>
> >> - if (create_sysfs && !mddev_is_dm(mddev) &&
> >> mddev->bitmap_ops->group) {
> >> - if (sysfs_create_group(&mddev->kobj,
> >> mddev->bitmap_ops->group))
> >> + if (create_sysfs && !mddev_is_dm(mddev) &&
> >> mddev->bitmap_ops->register_groups) {
> >> + if (mddev->bitmap_ops->register_groups(mddev))
> >> pr_warn("md: cannot register extra
> >> bitmap attributes for %s\n",
> >> mdname(mddev));
> >> else
> >> @@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct
> >> mddev *mddev, bool create_sysfs)
> >> static void mddev_clear_bitmap_ops(struct mddev *mddev, bool
> >> remove_sysfs)
> >> {
> >> if (remove_sysfs && !mddev_is_dm(mddev) &&
> >> mddev->bitmap_ops &&
> >> - mddev->bitmap_ops->group)
> >> - sysfs_remove_group(&mddev->kobj,
> >> mddev->bitmap_ops->group);
> >> + mddev->bitmap_ops->unregister_groups)
> >> + mddev->bitmap_ops->unregister_groups(mddev);
> >>
> >> mddev->bitmap_ops = NULL;
> >> }
> >> @@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev)
> >> (unsigned long long)pers->size(mddev,
> >> 0, 0) / 2);
> >> err = -EINVAL;
> >> }
> >> - if (err == 0 && pers->sync_request &&
> >> - (mddev->bitmap_info.file ||
> >> mddev->bitmap_info.offset)) {
> >> + if (err == 0 && pers->sync_request) {
> >> + if (mddev->bitmap_info.offset == 0)
> >> + mddev->bitmap_id = ID_BITMAP_NONE;
> >> err = md_bitmap_create(mddev, true);
> >> if (err)
> >> pr_warn("%s: failed to create bitmap
> >> (%d)\n",
> >> --
> >> 2.53.0
> >>
>
^ permalink raw reply
* Re: [RFC PATCH 1/2] kernel/notifier: replace single-linked list with double-linked list for reverse traversal
From: Petr Mladek @ 2026-04-21 9:05 UTC (permalink / raw)
To: Masami Hiramatsu
Cc: chensong_2000, rafael, lenb, mturquette, sboyd, viresh.kumar, agk,
snitzer, mpatocka, bmarzins, song, yukuai, linan122, jason.wessel,
danielt, dianders, horms, davem, edumazet, kuba, pabeni, paulmck,
frederic, mcgrof, petr.pavlu, da.gomez, samitolvanen, atomlin,
jpoimboe, jikos, mbenes, joe.lawrence, rostedt, mark.rutland,
mathieu.desnoyers, linux-modules, linux-kernel,
linux-trace-kernel, linux-acpi, linux-clk, linux-pm,
live-patching, dm-devel, linux-raid, kgdb-bugreport, netdev
In-Reply-To: <20260420144429.57b45f2beece690bceea96ec@kernel.org>
On Mon 2026-04-20 14:44:29, Masami Hiramatsu wrote:
> Hi Song,
>
> On Wed, 15 Apr 2026 15:01:37 +0800
> chensong_2000@189.cn wrote:
>
> > From: Song Chen <chensong_2000@189.cn>
> >
> > The current notifier chain implementation uses a single-linked list
> > (struct notifier_block *next), which only supports forward traversal
> > in priority order. This makes it difficult to handle cleanup/teardown
> > scenarios that require notifiers to be called in reverse priority order.
>
> What about introducing a new notification callback API that allows you
> to describe dependencies between callback functions?
>
> For example, when registering a callback, you could register a string
> as an ID and specify whether to call it before or after that ID,
> or you could register a comparison function that is called when adding
> to a list. (I prefer @name and @depends fields so that it can be easily
> maintained.)
This looks too complex. It would make sense only
when this API has more users.
Also this won't be enough for the ftrace/livepatch callbacks.
They need to be ordered against against each other. But they
also need to be called before/after all other callbacks.
For example, when the module is loaded:
+ 1st frace
+ 2nd livepatch
+ then other notifiers
See the commit c1bf08ac26e92122 ("ftrace: Be first to run code
modification on modules").
> This would allow for better dependency building when adding to the list.
> >
> > A concrete example is the ordering dependency between ftrace and
> > livepatch during module load/unload. see the detail here [1].
>
> If this only concerns notification callback issues with the ftrace
> and livepatch modules, it's far more robust to simply call the
> necessary processing directly when the modules load and unload,
> rather than registering notification callbacks externally.
>
> There are fprobe, kprobe and its trace-events, all of them are using
> ftrace as its fundation layer. In this case, I always needs to
> consider callback order when a module is unloaded.
>
> If ftrace is working as a part of module callbacks, it will conflict
> with fprobe/kprobe module callback. Of course we can reorder it with
> modifying its priority. But this is ugly, because when we introduce
> a new other feature which depends on another layer, we need to
> reorder the callback's priority number on the list.
>
> Based on the above, I don't think this can be resolved simply by
> changing the list of notification callbacks to a bidirectional list.
I agree. I would keep it as is (hardcoded).
Best Regards,
Petr
^ permalink raw reply
* Re: [PATCH V3 2/3] md: propagate BLK_FEAT_PCI_P2PDMA from member devices to RAID device
From: Xiao Ni @ 2026-04-21 9:18 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: song, yukuai, linan122, kbusch, axboe, hch, sagi, linux-raid,
linux-nvme, kmodukuri
In-Reply-To: <20260416212633.72650-3-kch@nvidia.com>
On Fri, Apr 17, 2026 at 5:27 AM Chaitanya Kulkarni <kch@nvidia.com> wrote:
>
> From: Kiran Kumar Modukuri <kmodukuri@nvidia.com>
>
> MD RAID does not propagate BLK_FEAT_PCI_P2PDMA from member devices to
> the RAID device, preventing peer-to-peer DMA through the RAID layer even
> when all underlying devices support it.
>
> Enable BLK_FEAT_PCI_P2PDMA unconditionally in raid0, raid1 and raid10
> personalities during queue limits setup. blk_stack_limits() clears it
> automatically if any member device lacks support, consistent with how
> BLK_FEAT_NOWAIT and BLK_FEAT_POLL are handled in the block core.
>
> Parity RAID personalities (raid4/5/6) are excluded because they require
> CPU access to data pages for parity computation, which is incompatible
> with P2P mappings.
>
> Tested with RAID0/1/10 arrays containing multiple NVMe devices with
> P2PDMA support, confirming that peer-to-peer transfers work correctly
> through the RAID layer.
>
> Signed-off-by: Kiran Kumar Modukuri <kmodukuri@nvidia.com>
> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
> ---
> drivers/md/raid0.c | 1 +
> drivers/md/raid1.c | 1 +
> drivers/md/raid10.c | 1 +
> 3 files changed, 3 insertions(+)
>
> diff --git a/drivers/md/raid0.c b/drivers/md/raid0.c
> index 5e38a51e349a..2cdaf7495d92 100644
> --- a/drivers/md/raid0.c
> +++ b/drivers/md/raid0.c
> @@ -392,6 +392,7 @@ static int raid0_set_limits(struct mddev *mddev)
> lim.io_opt = lim.io_min * mddev->raid_disks;
> lim.chunk_sectors = mddev->chunk_sectors;
> lim.features |= BLK_FEAT_ATOMIC_WRITES;
> + lim.features |= BLK_FEAT_PCI_P2PDMA;
> err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY);
> if (err)
> return err;
> diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c
> index ba91f7e61920..422ad4786569 100644
> --- a/drivers/md/raid1.c
> +++ b/drivers/md/raid1.c
> @@ -3215,6 +3215,7 @@ static int raid1_set_limits(struct mddev *mddev)
> lim.max_hw_wzeroes_unmap_sectors = 0;
> lim.logical_block_size = mddev->logical_block_size;
> lim.features |= BLK_FEAT_ATOMIC_WRITES;
> + lim.features |= BLK_FEAT_PCI_P2PDMA;
> err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY);
> if (err)
> return err;
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index 4901ebe45c87..07a5b734c8f3 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -3939,6 +3939,7 @@ static int raid10_set_queue_limits(struct mddev *mddev)
> lim.chunk_sectors = mddev->chunk_sectors;
> lim.io_opt = lim.io_min * raid10_nr_stripes(conf);
> lim.features |= BLK_FEAT_ATOMIC_WRITES;
> + lim.features |= BLK_FEAT_PCI_P2PDMA;
> err = mddev_stack_rdev_limits(mddev, &lim, MDDEV_STACK_INTEGRITY);
> if (err)
> return err;
> --
> 2.39.5
>
>
Thanks for enabling this feature. Looks good to me.
Reviewed-by: Xiao Ni <xni@redhat.com>
^ permalink raw reply
* Re: [PATCH v2 3/5] md/md-bitmap: add dummy bitmap ops for none to fix wrong bitmap offset
From: Su Yue @ 2026-04-21 9:21 UTC (permalink / raw)
To: Xiao Ni; +Cc: Su Yue, linux-raid, song, linan122, yukuai, heming.zhao
In-Reply-To: <CALTww29y_EJq3Qkimn67cSTnL6_+Yi6Lf1LoT=JJM6YqmFTjaw@mail.gmail.com>
On Tue 21 Apr 2026 at 15:36, Xiao Ni <xni@redhat.com> wrote:
> On Tue, Apr 21, 2026 at 10:29 AM Su Yue <l@damenly.org> wrote:
>>
>> On Mon 20 Apr 2026 at 15:05, Xiao Ni <xni@redhat.com> wrote:
>>
>> > On Tue, Apr 7, 2026 at 6:27 PM Su Yue <glass.su@suse.com>
>> > wrote:
>> >>
>> >> Before commit fb8cc3b0d9db ("md/md-bitmap: delay
>> >> registration
>> >> of bitmap_ops until creating bitmap")
>> >> if CONFIG_MD_BITMAP is enabled, both bitmap none, internal
>> >> and
>> >> clustered have
>> >> the sysfs file bitmap/location.
>> >>
>> >> After the commit, if bitmap is none, bitmap/location doesn't
>> >> exist anymore.
>> >> It breaks 'grow' behavior of a md array of madam with
>> >> MD_FEATURE_BITMAP_OFFSET.
>> >> Take level=mirror and metadata=1.2 as an example:
>> >>
>> >> $ mdadm --create /dev/md0 -f --bitmap=none
>> >> --raid-devices=2
>> >> --level=mirror \
>> >> --metadata=1.2 /dev/vdd /dev/vde
>> >> $ mdadm --grow /dev/md0 --bitmap=internal
>> >> $ cat /sys/block/md0/md/bitmap/location
>> >> Before:+8
>> >> After: +2
>> >>
>> >> While growing bitmap from none to internal, clustered and
>> >> llbitmap,
>> >> mdadm/Grow.c:Grow_addbitmap() tries to detect
>> >> bitmap/location
>> >> first.
>> >> 1)If bitmap/location exists, it sets bitmap/location after
>> >> getinfo_super().
>> >> 2)If bitmap/location doesn't exist, mdadm just calls
>> >> md_set_array_info() then
>> >> mddev->bitmap_info.default_offset will be used.
>> >> Situation can be worse if growing none to clustered, bitmap
>> >> offset of the node
>> >> calling `madm --grow` will be changed but the other node are
>> >> reading bitmap sb from
>> >> the old location.
>> >>
>> >> Here introducing a dummy bitmap_operations for
>> >> ID_BITMAP_NONE
>> >> to restore sysfs
>> >> file bitmap/location for ID_BITMAP_NONE and ID_BITMAP.
>> >> location_store() now calls md_bitmap_(create|destroy) with
>> >> false sysfs parameter then
>> >> (add,remove)_internal_bitmap() will be called to manage
>> >> sysfs
>> >> entries.
>> >>
>> >> Fixes: fb8cc3b0d9db ("md/md-bitmap: delay registration of
>> >> bitmap_ops until creating bitmap")
>> >> Suggested-by: Yu Kuai <yukuai@fnnas.com>
>> >> Signed-off-by: Su Yue <glass.su@suse.com>
>> >> ---
>> >> drivers/md/md-bitmap.c | 116
>> >> ++++++++++++++++++++++++++++++++++++---
>> >> drivers/md/md-bitmap.h | 3 +
>> >> drivers/md/md-llbitmap.c | 12 ++++
>> >> drivers/md/md.c | 16 +++---
>> >> 4 files changed, 130 insertions(+), 17 deletions(-)
>> >>
>> >> diff --git a/drivers/md/md-bitmap.c b/drivers/md/md-bitmap.c
>> >> index ac06c9647bf0..a8a176428c61 100644
>> >> --- a/drivers/md/md-bitmap.c
>> >> +++ b/drivers/md/md-bitmap.c
>> >> @@ -240,6 +240,11 @@ static bool bitmap_enabled(void *data,
>> >> bool flush)
>> >> bitmap->storage.filemap != NULL;
>> >> }
>> >>
>> >> +static bool dummy_bitmap_enabled(void *data, bool flush)
>> >> +{
>> >> + return false;
>> >> +}
>> >> +
>> >> /*
>> >> * check a page and, if necessary, allocate it (or hijack
>> >> it
>> >> if the alloc fails)
>> >> *
>> >> @@ -2201,6 +2206,11 @@ static int bitmap_create(struct mddev
>> >> *mddev)
>> >> return 0;
>> >> }
>> >>
>> >> +static int dummy_bitmap_create(struct mddev *mddev)
>> >> +{
>> >> + return 0;
>> >> +}
>> >> +
>> >> static int bitmap_load(struct mddev *mddev)
>> >> {
>> >> int err = 0;
>> >> @@ -2594,6 +2604,9 @@ location_show(struct mddev *mddev,
>> >> char
>> >> *page)
>> >> return len;
>> >> }
>> >>
>> >> +static int create_internal_group(struct mddev *mddev);
>> >> +static void remove_internal_group(struct mddev *mddev);
>> >> +
>> >> static ssize_t
>> >> location_store(struct mddev *mddev, const char *buf, size_t
>> >> len)
>> >> {
>> >> @@ -2618,7 +2631,8 @@ location_store(struct mddev *mddev,
>> >> const
>> >> char *buf, size_t len)
>> >> goto out;
>> >> }
>> >>
>> >> - md_bitmap_destroy(mddev, true);
>> >> + md_bitmap_destroy(mddev, false);
>> >> + remove_internal_group(mddev);
>> >> mddev->bitmap_info.offset = 0;
>> >> if (mddev->bitmap_info.file) {
>> >> struct file *f =
>> >> mddev->bitmap_info.file;
>> >> @@ -2659,14 +2673,22 @@ location_store(struct mddev *mddev,
>> >> const char *buf, size_t len)
>> >> */
>> >> mddev->bitmap_id = ID_BITMAP;
>> >> mddev->bitmap_info.offset = offset;
>> >> - rv = md_bitmap_create(mddev, true);
>> >> + rv = md_bitmap_create(mddev, false);
>> >> if (rv)
>> >> goto out;
>> >>
>> >> + rv = create_internal_group(mddev);
>> >> + if (rv) {
>> >> + mddev->bitmap_info.offset =
>> >> 0;
>> >> + bitmap_destroy(mddev);
>> >> + goto out;
>> >> + }
>> >> +
>> >> rv = bitmap_load(mddev);
>> >> if (rv) {
>> >> +
>> >> remove_internal_group(mddev);
>> >> mddev->bitmap_info.offset =
>> >> 0;
>> >> - md_bitmap_destroy(mddev,
>> >> true);
>> >> + md_bitmap_destroy(mddev,
>> >> false);
>> >> goto out;
>> >> }
>> >> }
>> >> @@ -2960,8 +2982,7 @@ static struct md_sysfs_entry
>> >> max_backlog_used =
>> >> __ATTR(max_backlog_used, S_IRUGO | S_IWUSR,
>> >> behind_writes_used_show, behind_writes_used_reset);
>> >>
>> >> -static struct attribute *md_bitmap_attrs[] = {
>> >> - &bitmap_location.attr,
>> >> +static struct attribute *internal_bitmap_attrs[] = {
>> >> &bitmap_space.attr,
>> >> &bitmap_timeout.attr,
>> >> &bitmap_backlog.attr,
>> >> @@ -2972,11 +2993,57 @@ static struct attribute
>> >> *md_bitmap_attrs[] = {
>> >> NULL
>> >> };
>> >>
>> >> -static struct attribute_group md_bitmap_group = {
>> >> +static struct attribute_group internal_bitmap_group = {
>> >> .name = "bitmap",
>> >> - .attrs = md_bitmap_attrs,
>> >> + .attrs = internal_bitmap_attrs,
>> >> };
>> >>
>> >> +/* Only necessary attrs for compatibility */
>> >> +static struct attribute *common_bitmap_attrs[] = {
>> >> + &bitmap_location.attr,
>> >> + NULL
>> >> +};
>> >> +
>> >> +static const struct attribute_group common_bitmap_group = {
>> >> + .name = "bitmap",
>> >> + .attrs = common_bitmap_attrs,
>> >> +};
>> >> +
>> >> +static int create_internal_group(struct mddev *mddev)
>> >> +{
>> >> + /*
>> >> + * md_bitmap_group and md_bitmap_common_group are
>> >> using
>> >> same name
>> >> + * 'bitmap'.
>> >> + */
>> >> + return sysfs_merge_group(&mddev->kobj,
>> >> &internal_bitmap_group);
>> >> +}
>> >> +
>> >> +static void remove_internal_group(struct mddev *mddev)
>> >> +{
>> >> + sysfs_unmerge_group(&mddev->kobj,
>> >> &internal_bitmap_group);
>> >> +}
>> >> +
>> >> +static int bitmap_register_groups(struct mddev *mddev)
>> >> +{
>> >> + int ret;
>> >> +
>> >> + ret = sysfs_create_group(&mddev->kobj,
>> >> &common_bitmap_group);
>> >> +
>> >> + if (ret)
>> >> + return ret;
>> >> +
>> >> + ret = sysfs_merge_group(&mddev->kobj,
>> >> &internal_bitmap_group);
>> >> + if (ret)
>> >> + sysfs_remove_group(&mddev->kobj,
>> >> &common_bitmap_group);
>> >> +
>> >> + return ret;
>> >> +}
>> >> +
>> >> +static void bitmap_unregister_groups(struct mddev *mddev)
>> >> +{
>> >> + sysfs_unmerge_group(&mddev->kobj,
>> >> &internal_bitmap_group);
>> >> +}
>> >
>> > Hi Su
>> >
>> > bitmap_unregister_groups should also remove
>> > common_bitmap_group,
>> > right?
>> >
>>
>> As you pasted before, mdadm:Grow.c:
>>
>> int Grow_addbitmap(char *devname, int fd, struct context *c,
>> struct shape *s)
>> {
>> ...
>> if (array.state & (1 << MD_SB_BITMAP_PRESENT)) {
>> if (s->btype == BitmapNone) {
>> array.state &= ~(1 <<
>> MD_SB_BITMAP_PRESENT);
>> if (md_set_array_info(fd, &array) != 0)
>> {
>> if (array.state & (1 <<
>> MD_SB_CLUSTERED))
>> pr_err("failed to
>> remove clustered
>> bitmap.\n");
>> else
>> pr_err("failed to
>> remove internal
>> bitmap.\n");
>> return 1;
>> }
>> return 0;
>> }
>> pr_err("bitmap already present on %s\n",
>> devname);
>> return 1;
>> }
>> ...
>> }
>>
>> In case of growing from internal to none,
>> bitmap_unregister_groups() will
>> be called, if common_bitmap_group is removed, bitmap/location
>> won't exist.
>> update_array_info() is a common function used by many call
>> paths.
>> Also
>> llbitmap is involved here. I don't want to make situation and
>> code
>> more
>> complicated like adding more codes in update_array_info().
>
> The above mdadm codes use bitmap/location. In your patch, you
> already
> pass false to md_bitmap_destroy in location_store. So removing
> common_bitmap_group in bitmap_unregister_groups can't affect the
> above
> case.
>
Right. Got confusion between md_set_array_info() and
update_array_info().
>
> But after reading your below comments. It's good to me that
> bitmap_unregister_groups doesn't remove common_bitmap_group.
>
Thanks.
--
Su
>>
>>
>> >> +
>> >> static struct bitmap_operations bitmap_ops = {
>> >
>> > You already changed md_bitmap_group to internal_bitmap_group.
>> > It's
>> > better to change bitmap_ops to internal_bitmap_ops?
>> >
>> >> .head = {
>> >> .type = MD_BITMAP,
>> >> @@ -3018,21 +3085,52 @@ static struct bitmap_operations
>> >> bitmap_ops = {
>> >> .set_pages = bitmap_set_pages,
>> >> .free = md_bitmap_free,
>> >>
>> >> - .group = &md_bitmap_group,
>> >> + .register_groups = bitmap_register_groups,
>> >> + .unregister_groups = bitmap_unregister_groups,
>> >> +};
>> >> +
>> >> +static int none_bitmap_register_groups(struct mddev *mddev)
>> >> +{
>> >> + return sysfs_create_group(&mddev->kobj,
>> >> &common_bitmap_group);
>> >> +}
>> >> +
>> >> +static struct bitmap_operations none_bitmap_ops = {
>> >> + .head = {
>> >> + .type = MD_BITMAP,
>> >> + .id = ID_BITMAP_NONE,
>> >> + .name = "none",
>> >> + },
>> >> +
>> >> + .enabled = dummy_bitmap_enabled,
>> >> + .create = dummy_bitmap_create,
>> >> + .destroy = bitmap_destroy,
>> >> + .load = bitmap_load,
>> >> + .get_stats = bitmap_get_stats,
>> >> + .free = md_bitmap_free,
>> >> +
>> >> + .register_groups =
>> >> none_bitmap_register_groups,
>> >> + .unregister_groups = NULL,
>> >
>> > How does bitmap/location can be deleted if array is created
>> > with
>> > no
>> > bitmap? Can the `mdadm --stop` command get stuck?
>> >
>> No. It wont get stuck.
>>
>> I guess here your concern is the timing of removing
>> common_bitmap_group.
>> The life cycle is same as before fb8cc3b0d9db. At the time
>> llbitmap is
>> not introduced, because there is no need to switch bitmap ops,
>> so
>> all
>> attrs of bitmap are removed in kobject_put() in
>> mddev_delayed_delete()
>> queued by mddev_put(). This is now where common_bitmap_group is
>> being removed.
>
> Thanks for the explanation.
>
> Best Regards
> Xiao
>>
>> --
>> Su
>> > Best Regards
>> > Xiao
>> >
>> >> };
>> >>
>> >> int md_bitmap_init(void)
>> >> {
>> >> + int ret;
>> >> +
>> >> md_bitmap_wq = alloc_workqueue("md_bitmap",
>> >> WQ_MEM_RECLAIM | WQ_UNBOUND,
>> >> 0);
>> >> if (!md_bitmap_wq)
>> >> return -ENOMEM;
>> >>
>> >> - return register_md_submodule(&bitmap_ops.head);
>> >> + ret = register_md_submodule(&bitmap_ops.head);
>> >> + if (ret)
>> >> + return ret;
>> >> +
>> >> + return register_md_submodule(&none_bitmap_ops.head);
>> >> }
>> >>
>> >> void md_bitmap_exit(void)
>> >> {
>> >> destroy_workqueue(md_bitmap_wq);
>> >> unregister_md_submodule(&bitmap_ops.head);
>> >> + unregister_md_submodule(&none_bitmap_ops.head);
>> >> }
>> >> diff --git a/drivers/md/md-bitmap.h b/drivers/md/md-bitmap.h
>> >> index b42a28fa83a0..10bc6854798c 100644
>> >> --- a/drivers/md/md-bitmap.h
>> >> +++ b/drivers/md/md-bitmap.h
>> >> @@ -125,6 +125,9 @@ struct bitmap_operations {
>> >> void (*set_pages)(void *data, unsigned long pages);
>> >> void (*free)(void *data);
>> >>
>> >> + int (*register_groups)(struct mddev *mddev);
>> >> + void (*unregister_groups)(struct mddev *mddev);
>> >> +
>> >> struct attribute_group *group;
>> >> };
>> >>
>> >> diff --git a/drivers/md/md-llbitmap.c
>> >> b/drivers/md/md-llbitmap.c
>> >> index bf398d7476b3..9b3ea4f1d268 100644
>> >> --- a/drivers/md/md-llbitmap.c
>> >> +++ b/drivers/md/md-llbitmap.c
>> >> @@ -1561,6 +1561,16 @@ static struct attribute_group
>> >> md_llbitmap_group = {
>> >> .attrs = md_llbitmap_attrs,
>> >> };
>> >>
>> >> +static int llbitmap_register_groups(struct mddev *mddev)
>> >> +{
>> >> + return sysfs_create_group(&mddev->kobj,
>> >> &md_llbitmap_group);
>> >> +}
>> >> +
>> >> +static void llbitmap_unregister_groups(struct mddev *mddev)
>> >> +{
>> >> + sysfs_remove_group(&mddev->kobj,
>> >> &md_llbitmap_group);
>> >> +}
>> >> +
>> >> static struct bitmap_operations llbitmap_ops = {
>> >> .head = {
>> >> .type = MD_BITMAP,
>> >> @@ -1597,6 +1607,8 @@ static struct bitmap_operations
>> >> llbitmap_ops = {
>> >> .dirty_bits = llbitmap_dirty_bits,
>> >> .write_all = llbitmap_write_all,
>> >>
>> >> + .register_groups = llbitmap_register_groups,
>> >> + .unregister_groups =
>> >> llbitmap_unregister_groups,
>> >> .group = &md_llbitmap_group,
>> >> };
>> >>
>> >> diff --git a/drivers/md/md.c b/drivers/md/md.c
>> >> index d3c8f77b4fe3..55a95b227b83 100644
>> >> --- a/drivers/md/md.c
>> >> +++ b/drivers/md/md.c
>> >> @@ -683,8 +683,7 @@ static bool mddev_set_bitmap_ops(struct
>> >> mddev *mddev, bool create_sysfs)
>> >> struct bitmap_operations *old = mddev->bitmap_ops;
>> >> struct md_submodule_head *head;
>> >>
>> >> - if (mddev->bitmap_id == ID_BITMAP_NONE ||
>> >> - (old && old->head.id == mddev->bitmap_id))
>> >> + if (old && old->head.id == mddev->bitmap_id)
>> >> return true;
>> >>
>> >> xa_lock(&md_submodule);
>> >> @@ -703,8 +702,8 @@ static bool mddev_set_bitmap_ops(struct
>> >> mddev *mddev, bool create_sysfs)
>> >> mddev->bitmap_ops = (void *)head;
>> >> xa_unlock(&md_submodule);
>> >>
>> >> - if (create_sysfs && !mddev_is_dm(mddev) &&
>> >> mddev->bitmap_ops->group) {
>> >> - if (sysfs_create_group(&mddev->kobj,
>> >> mddev->bitmap_ops->group))
>> >> + if (create_sysfs && !mddev_is_dm(mddev) &&
>> >> mddev->bitmap_ops->register_groups) {
>> >> + if
>> >> (mddev->bitmap_ops->register_groups(mddev))
>> >> pr_warn("md: cannot register extra
>> >> bitmap attributes for %s\n",
>> >> mdname(mddev));
>> >> else
>> >> @@ -724,8 +723,8 @@ static bool mddev_set_bitmap_ops(struct
>> >> mddev *mddev, bool create_sysfs)
>> >> static void mddev_clear_bitmap_ops(struct mddev *mddev,
>> >> bool
>> >> remove_sysfs)
>> >> {
>> >> if (remove_sysfs && !mddev_is_dm(mddev) &&
>> >> mddev->bitmap_ops &&
>> >> - mddev->bitmap_ops->group)
>> >> - sysfs_remove_group(&mddev->kobj,
>> >> mddev->bitmap_ops->group);
>> >> + mddev->bitmap_ops->unregister_groups)
>> >> + mddev->bitmap_ops->unregister_groups(mddev);
>> >>
>> >> mddev->bitmap_ops = NULL;
>> >> }
>> >> @@ -6610,8 +6609,9 @@ int md_run(struct mddev *mddev)
>> >> (unsigned long
>> >> long)pers->size(mddev,
>> >> 0, 0) / 2);
>> >> err = -EINVAL;
>> >> }
>> >> - if (err == 0 && pers->sync_request &&
>> >> - (mddev->bitmap_info.file ||
>> >> mddev->bitmap_info.offset)) {
>> >> + if (err == 0 && pers->sync_request) {
>> >> + if (mddev->bitmap_info.offset == 0)
>> >> + mddev->bitmap_id = ID_BITMAP_NONE;
>> >> err = md_bitmap_create(mddev, true);
>> >> if (err)
>> >> pr_warn("%s: failed to create bitmap
>> >> (%d)\n",
>> >> --
>> >> 2.53.0
>> >>
>>
^ permalink raw reply
* Re: [PATCH V3 1/3] block: clear BLK_FEAT_PCI_P2PDMA in blk_stack_limits() for non-supporting devices
From: Chaitanya Kulkarni @ 2026-04-21 22:30 UTC (permalink / raw)
To: Nitesh Shetty, Chaitanya Kulkarni
Cc: song@kernel.org, yukuai@fnnas.com, linan122@huawei.com,
kbusch@kernel.org, axboe@kernel.dk, hch@lst.de, sagi@grimberg.me,
linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org,
Kiran Modukuri
In-Reply-To: <20260417101127.uku7q65akj6gwwh4@green245.gost>
On 4/17/26 03:11, Nitesh Shetty wrote:
> On 16/04/26 02:26PM, Chaitanya Kulkarni wrote:
>> BLK_FEAT_NOWAIT and BLK_FEAT_POLL are cleared in blk_stack_limits()
>> when an underlying device does not support them. Apply the same
>> treatment to BLK_FEAT_PCI_P2PDMA: stacking drivers set it
>> unconditionally and rely on the core to clear it whenever a
>> non-supporting member device is stacked.
>>
>> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
>> ---
>> block/blk-settings.c | 2 ++
>> 1 file changed, 2 insertions(+)
>>
>> diff --git a/block/blk-settings.c b/block/blk-settings.c
>> index 78c83817b9d3..8274631290db 100644
>> --- a/block/blk-settings.c
>> +++ b/block/blk-settings.c
>> @@ -795,6 +795,8 @@ int blk_stack_limits(struct queue_limits *t,
>> struct queue_limits *b,
>> t->features &= ~BLK_FEAT_NOWAIT;
>> if (!(b->features & BLK_FEAT_POLL))
>> t->features &= ~BLK_FEAT_POLL;
>> + if (!(b->features & BLK_FEAT_PCI_P2PDMA))
>> + t->features &= ~BLK_FEAT_PCI_P2PDMA;
>>
>> t->flags |= (b->flags & BLK_FLAG_MISALIGNED);
> I think you need below patch[1] as well to unset this here.
> Also I feel better to include Mike,Mikulas and dm-devel mailing list
> as well.
>
get_maintainers.pl didn't list Mike/Mikulas and dm-devel.
> Thanks,
> Nitesh
>
> [1]
> diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c
> index dc2eff6b739d..0442c1f4c686 100644
> --- a/drivers/md/dm-table.c
> +++ b/drivers/md/dm-table.c
> @@ -590,7 +590,8 @@ int dm_split_args(int *argc, char ***argvp, char
> *input)
> static void dm_set_stacking_limits(struct queue_limits *limits)
> {
> blk_set_stacking_limits(limits);
> - limits->features |= BLK_FEAT_IO_STAT | BLK_FEAT_NOWAIT |
> BLK_FEAT_POLL;
> + limits->features |= BLK_FEAT_IO_STAT | BLK_FEAT_NOWAIT |
> BLK_FEAT_POLL \
> + BLK_FEAT_PCI_P2PDMA;
> }
>
DM P2PDMA support needs a separate DM patch series with explicit target
capability plumbing and enablement only for vetted targets.
Enabling it blindly for all the targets may not be accepted.
This series is limited to md RAID0/1/10 and native NVMe multipath.
I’ll send an incremental DM series separately and will CC the DM
maintainers and dm-devel, thanks for looking into it.
-ck
^ permalink raw reply
* Re: [PATCH V3 0/2] md/nvme: Enable PCI P2PDMA support for RAID0 and NVMe Multipath
From: Chaitanya Kulkarni @ 2026-04-21 22:32 UTC (permalink / raw)
To: song@kernel.org, yukuai@fnnas.com, linan122@huawei.com,
kbusch@kernel.org, axboe@kernel.dk, hch@lst.de
Cc: Chaitanya Kulkarni, linux-raid@vger.kernel.org,
linux-nvme@lists.infradead.org, Kiran Modukuri, sagi@grimberg.me
In-Reply-To: <20260416212633.72650-1-kch@nvidia.com>
On 4/16/26 14:26, Chaitanya Kulkarni wrote:
> Hi,
>
> This patch series extends PCI peer-to-peer DMA (P2PDMA) support to enable
> direct data transfers between PCIe devices through RAID and NVMe multipath
> block layers.
>
> Current Linux kernel P2PDMA infrastructure supports direct peer-to-peer
> transfers, but this support is not propagated through certain storage
> stacks like MD RAID and NVMe multipath. This adds two patches for
> MD RAID 0/1/10 and NVMe to propogate P2PDMA support through the
> storage stack.
>
> All four test scenarios demonstrate that P2PDMA capabilities are correctly
> propagated through both the MD RAID layer (patch 1/2) and NVMe multipath
> layer (patch 2/2). Direct peer-to-peer transfers complete successfully with
> full data integrity verification, confirming that:
>
> 1. RAID devices properly inherit P2PDMA capability from member devices
> 2. NVMe multipath devices correctly expose P2PDMA support
> 3. P2P memory buffers can be used for transfers involving both types
> 4. Data integrity is maintained across all transfer combinations
>
> I've added the patch specific tests and blktest log as well at the end.
>
> Repo:-
>
> git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git
>
> Branch HEAD:-
>
> commit 88a57e15861997dd6fa98154ad087f7831bbead1 (origin/for-next)
> Merge: 81a0a2e4e535 36446de0c30c
> Author: Jens Axboe<axboe@kernel.dk>
> Date: Fri Apr 10 07:02:42 2026 -0600
>
> Merge branch 'for-7.1/block' into for-next
>
> * for-7.1/block:
> ublk: fix tautological comparison warning in ublk_ctrl_reg_buf
> -ck
>
> Changes from V2:-
>
> 1. Unconditionally set the BLK_FEAT_PCI_P2PDMA for md and nvme multipath.
> (Christoph)
> 2. Add a prep patch to diable BLK_FEAT_PCI_P2PDMA in the blk_stack_limit().
> (christoph)
>
> Changes from V1:-
> - Update patch 1 to explicitly support MD RAID 0/1/10.
> - Fix signoff chain order for patch 2.
> - Clear BLK_FEAT_PCI_P2PDMA in nvme_mpath_add_disk() when a newly
> added path does not support it, to handle multipath across different
> transports.
> - Add nvme multipath test log for mixed transport TCP and PCIe.
>
> Chaitanya Kulkarni (1):
> block: clear BLK_FEAT_PCI_P2PDMA in blk_stack_limits() for
> non-supporting devices
>
> Kiran Kumar Modukuri (2):
> md: propagate BLK_FEAT_PCI_P2PDMA from member devices to RAID device
> nvme-multipath: enable PCI P2PDMA for multipath devices
>
> block/blk-settings.c | 2 ++
> drivers/md/raid0.c | 1 +
> drivers/md/raid1.c | 1 +
> drivers/md/raid10.c | 1 +
> drivers/nvme/host/multipath.c | 2 +-
> 5 files changed, 6 insertions(+), 1 deletion(-)
If there is no objection is it possible to merge this via
linux-block/for-next ?
-ck
^ permalink raw reply
* [PATCH 2/4] md/raid10: prepare r10bio allocation width tracking
From: Chen Cheng @ 2026-04-22 2:33 UTC (permalink / raw)
To: linux-raid, yukuai; +Cc: chencheng, chenchneg33
In-Reply-To: <20260422023317.796326-1-chencheng@fnnas.com>
From: Chen Cheng <chencheng@fnnas.com>
Record how many devs[] slots each r10bio was allocated with.
Keep the active r10bio pool in a separate object that carries its width.
This keeps the allocation width separate from the per-request width stored
in used_nr_devs and prepares the pool for replacement during reshape.
---
drivers/md/raid10.c | 40 +++++++++++++++++++++++++++-------------
drivers/md/raid10.h | 8 +++++++-
2 files changed, 34 insertions(+), 14 deletions(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index e93933632893..b447903fbdc6 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -103,14 +103,22 @@ static inline struct r10bio *get_resync_r10bio(struct bio *bio)
return get_resync_pages(bio)->raid_bio;
}
-static void * r10bio_pool_alloc(gfp_t gfp_flags, void *data)
+static void *r10bio_pool_alloc(gfp_t gfp_flags, void *data)
{
- struct r10conf *conf = data;
- int size = offsetof(struct r10bio, devs[conf->geo.raid_disks]);
+ struct r10bio_pool *pool = data;
+ int size = offsetof(struct r10bio, devs[pool->nr_devs]);
+ struct r10bio *r10_bio = kzalloc(size, gfp_flags);
+
+ if (r10_bio)
+ r10_bio->alloc_nr_devs = pool->nr_devs;
+ return r10_bio;
+}
- /* allocate a r10bio with room for raid_disks entries in the
- * bios array */
- return kzalloc(size, gfp_flags);
+static int init_r10bio_pool(struct r10bio_pool *pool, unsigned int nr_devs)
+{
+ pool->nr_devs = nr_devs;
+ return mempool_init(&pool->pool, NR_RAID_BIOS, r10bio_pool_alloc,
+ rbio_pool_free, pool);
}
#define RESYNC_SECTORS (RESYNC_BLOCK_SIZE >> 9)
@@ -137,7 +145,7 @@ static void * r10buf_pool_alloc(gfp_t gfp_flags, void *data)
int nalloc, nalloc_rp;
struct resync_pages *rps;
- r10_bio = r10bio_pool_alloc(gfp_flags, conf);
+ r10_bio = r10bio_pool_alloc(gfp_flags, conf->r10bio_pool);
if (!r10_bio)
return NULL;
@@ -277,7 +285,7 @@ static void free_r10bio(struct r10bio *r10_bio)
struct r10conf *conf = r10_bio->mddev->private;
put_all_bios(conf, r10_bio);
- mempool_free(r10_bio, &conf->r10bio_pool);
+ mempool_free(r10_bio, &conf->r10bio_pool->pool);
}
static void put_buf(struct r10bio *r10_bio)
@@ -1531,7 +1539,7 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
struct r10conf *conf = mddev->private;
struct r10bio *r10_bio;
- r10_bio = mempool_alloc(&conf->r10bio_pool, GFP_NOIO);
+ r10_bio = mempool_alloc(&conf->r10bio_pool->pool, GFP_NOIO);
r10_bio->master_bio = bio;
r10_bio->sectors = sectors;
@@ -1724,7 +1732,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
(last_stripe_index << geo->chunk_shift);
retry_discard:
- r10_bio = mempool_alloc(&conf->r10bio_pool, GFP_NOIO);
+ r10_bio = mempool_alloc(&conf->r10bio_pool->pool, GFP_NOIO);
r10_bio->mddev = mddev;
r10_bio->state = 0;
r10_bio->sectors = 0;
@@ -3825,7 +3833,10 @@ static void raid10_free_conf(struct r10conf *conf)
if (!conf)
return;
- mempool_exit(&conf->r10bio_pool);
+ if (conf->r10bio_pool) {
+ mempool_exit(&conf->r10bio_pool->pool);
+ kfree(conf->r10bio_pool);
+ }
kfree(conf->mirrors);
kfree(conf->mirrors_old);
kfree(conf->mirrors_new);
@@ -3870,10 +3881,13 @@ static struct r10conf *setup_conf(struct mddev *mddev)
if (!conf->tmppage)
goto out;
+ conf->r10bio_pool = kzalloc_obj(struct r10bio_pool);
+ if (!conf->r10bio_pool)
+ goto out;
+
conf->geo = geo;
conf->copies = copies;
- err = mempool_init(&conf->r10bio_pool, NR_RAID_BIOS, r10bio_pool_alloc,
- rbio_pool_free, conf);
+ err = init_r10bio_pool(conf->r10bio_pool, conf->geo.raid_disks);
if (err)
goto out;
diff --git a/drivers/md/raid10.h b/drivers/md/raid10.h
index 92e8743023e6..8fa4e54c444c 100644
--- a/drivers/md/raid10.h
+++ b/drivers/md/raid10.h
@@ -20,6 +20,11 @@ struct raid10_info {
sector_t head_position;
};
+struct r10bio_pool {
+ mempool_t pool;
+ unsigned int nr_devs;
+};
+
struct r10conf {
struct mddev *mddev;
struct raid10_info *mirrors;
@@ -87,7 +92,7 @@ struct r10conf {
*/
wait_queue_head_t wait_barrier;
- mempool_t r10bio_pool;
+ struct r10bio_pool *r10bio_pool;
mempool_t r10buf_pool;
struct page *tmppage;
struct bio_set bio_split;
@@ -128,6 +133,7 @@ struct r10bio {
*/
int read_slot;
unsigned int used_nr_devs;
+ unsigned int alloc_nr_devs;
struct list_head retry_list;
/*
--
2.53.0
^ permalink raw reply related
* [PATCH 1/4] md/raid10: prepare per-r10bio dev slot tracking
From: Chen Cheng @ 2026-04-22 2:33 UTC (permalink / raw)
To: linux-raid, yukuai; +Cc: chencheng, chenchneg33
From: Chen Cheng <chencheng@fnnas.com>
raid10 reuses r10bio objects from both r10bio_pool and r10buf_pool. Track
the number of devs[] slots used by each request in the r10bio itself and
initialize it whenever one of these objects is reused.
No functional change yet. A later patch will use this width when reshape
changes conf->geo.raid_disks.
---
drivers/md/raid10.c | 4 ++++
drivers/md/raid10.h | 1 +
2 files changed, 5 insertions(+)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 0653b5d8545a..e93933632893 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1540,6 +1540,7 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
r10_bio->sector = bio->bi_iter.bi_sector;
r10_bio->state = 0;
r10_bio->read_slot = -1;
+ r10_bio->used_nr_devs = conf->geo.raid_disks;
memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) *
conf->geo.raid_disks);
@@ -1727,6 +1728,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
r10_bio->mddev = mddev;
r10_bio->state = 0;
r10_bio->sectors = 0;
+ r10_bio->used_nr_devs = geo->raid_disks;
memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) * geo->raid_disks);
wait_blocked_dev(mddev, r10_bio);
@@ -3061,6 +3063,8 @@ static struct r10bio *raid10_alloc_init_r10buf(struct r10conf *conf)
else
nalloc = 2; /* recovery */
+ r10bio->used_nr_devs = nalloc;
+
for (i = 0; i < nalloc; i++) {
bio = r10bio->devs[i].bio;
rp = bio->bi_private;
diff --git a/drivers/md/raid10.h b/drivers/md/raid10.h
index ec79d87fb92f..92e8743023e6 100644
--- a/drivers/md/raid10.h
+++ b/drivers/md/raid10.h
@@ -127,6 +127,7 @@ struct r10bio {
* if the IO is in READ direction, then this is where we read
*/
int read_slot;
+ unsigned int used_nr_devs;
struct list_head retry_list;
/*
--
2.53.0
^ permalink raw reply related
* [PATCH 3/4] md/raid10: fix r10bio devs overflow across reshape
From: Chen Cheng @ 2026-04-22 2:33 UTC (permalink / raw)
To: linux-raid, yukuai; +Cc: chencheng, chenchneg33
In-Reply-To: <20260422023317.796326-1-chencheng@fnnas.com>
From: Chen Cheng <chencheng@fnnas.com>
A 4-disk to 5-disk raid10 reshape can complete or free an r10bio that was
allocated before the geometry switch.
The failure was reproduced with a simple write workload while reshaping a
raid10 array from 4 disks to 5 disks, e.g.:
mdadm -C /dev/md777 -l10 -n4 /dev/sda /dev/sdb /dev/sdc /dev/sdd
mkfs.ext4 /dev/md777
mount /dev/md777 /mnt/test
fsstress -d /mnt/test -n 24000 -p 8 -l 24 &
mdadm /dev/md777 --add /dev/sde
mdadm --grow /dev/md777 --raid-devices=5 \
--backup-file=/tmp/md-reshape-backup
Without this patch, the sequence above can trigger:
BUG: KASAN: slab-out-of-bounds in free_r10bio+0x1c4/0x260 [raid10]
Read of size 8 at addr ffff00008c2dfac8 by task ksoftirqd/0/15
free_r10bio
raid_end_bio_io
one_write_done
raid10_end_write_request
The buggy object was 200 bytes long, which matches an r10bio with space for
only four devs[] entries. However, put_all_bios() and find_bio_disk() walk
r10_bio->devs[] using the current conf->geo.raid_disks value. Once reshape
switches conf->geo.raid_disks from 4 to 5, an old 4-slot r10bio can be
completed or freed as if it had 5 slots, and the walk overruns devs[4].
The same stale-width mismatch can also surface during a 5-disk to 4-disk
reshape.
The same transition also leaves stale-width objects in the active r10bio
pool, so new requests can reuse a 4-slot object after reshape starts unless
the pool is replaced for the new geometry.
Fix this by recording the actual devs[] slot count in each r10bio and using
that count when scanning or freeing the object. Also replace the active
r10bio pool with one sized for the new geometry before reshape switches
layouts. Old-width r10bio objects are freed directly instead of being
returned to a pool that now expects a different width.
A/B validation:
- Without this patch, the 4-disk to 5-disk reshape test triggered the
KASAN report.
- With this patch, neither the 4-disk to 5-disk nor the 5-disk to 4-disk
reshape test triggers KASAN.
---
drivers/md/raid10.c | 43 +++++++++++++++++++++++++++++++++++++++++--
1 file changed, 41 insertions(+), 2 deletions(-)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index b447903fbdc6..3edde440623a 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -234,6 +234,30 @@ static void * r10buf_pool_alloc(gfp_t gfp_flags, void *data)
return NULL;
}
+static int reinit_r10bio_pool(struct r10conf *conf, unsigned int nr_devs)
+{
+ struct r10bio_pool *new_pool, *old_pool = conf->r10bio_pool;
+ int ret;
+
+ if (old_pool->nr_devs == nr_devs)
+ return 0;
+
+ new_pool = kzalloc_obj(struct r10bio_pool);
+ if (!new_pool)
+ return -ENOMEM;
+
+ ret = init_r10bio_pool(new_pool, nr_devs);
+ if (ret) {
+ kfree(new_pool);
+ return ret;
+ }
+
+ conf->r10bio_pool = new_pool;
+ mempool_exit(&old_pool->pool);
+ kfree(old_pool);
+ return 0;
+}
+
static void r10buf_pool_free(void *__r10_bio, void *data)
{
struct r10conf *conf = data;
@@ -268,7 +292,7 @@ static void put_all_bios(struct r10conf *conf, struct r10bio *r10_bio)
{
int i;
- for (i = 0; i < conf->geo.raid_disks; i++) {
+ for (i = 0; i < r10_bio->used_nr_devs; i++) {
struct bio **bio = & r10_bio->devs[i].bio;
if (!BIO_SPECIAL(*bio))
bio_put(*bio);
@@ -285,6 +309,10 @@ static void free_r10bio(struct r10bio *r10_bio)
struct r10conf *conf = r10_bio->mddev->private;
put_all_bios(conf, r10_bio);
+ if (r10_bio->alloc_nr_devs != conf->r10bio_pool->nr_devs) {
+ rbio_pool_free(r10_bio, conf);
+ return;
+ }
mempool_free(r10_bio, &conf->r10bio_pool->pool);
}
@@ -365,7 +393,7 @@ static int find_bio_disk(struct r10conf *conf, struct r10bio *r10_bio,
int slot;
int repl = 0;
- for (slot = 0; slot < conf->geo.raid_disks; slot++) {
+ for (slot = 0; slot < r10_bio->used_nr_devs; slot++) {
if (r10_bio->devs[slot].bio == bio)
break;
if (r10_bio->devs[slot].repl_bio == bio) {
@@ -4416,6 +4444,11 @@ static int raid10_start_reshape(struct mddev *mddev)
if (spares < mddev->delta_disks)
return -EINVAL;
+ raise_barrier(conf, 0);
+ ret = reinit_r10bio_pool(conf, new.raid_disks);
+ if (ret)
+ goto out_lower_barrier;
+
conf->offset_diff = min_offset_diff;
spin_lock_irq(&conf->device_lock);
if (conf->mirrors_new) {
@@ -4433,6 +4466,7 @@ static int raid10_start_reshape(struct mddev *mddev)
sector_t size = raid10_size(mddev, 0, 0);
if (size < mddev->array_sectors) {
spin_unlock_irq(&conf->device_lock);
+ lower_barrier(conf);
pr_warn("md/raid10:%s: array size must be reduce before number of disks\n",
mdname(mddev));
return -EINVAL;
@@ -4443,6 +4477,7 @@ static int raid10_start_reshape(struct mddev *mddev)
conf->reshape_progress = 0;
conf->reshape_safe = conf->reshape_progress;
spin_unlock_irq(&conf->device_lock);
+ lower_barrier(conf);
if (mddev->delta_disks && mddev->bitmap) {
struct mdp_superblock_1 *sb = NULL;
@@ -4527,6 +4562,10 @@ static int raid10_start_reshape(struct mddev *mddev)
md_new_event();
return 0;
+out_lower_barrier:
+ lower_barrier(conf);
+ return ret;
+
abort:
mddev->recovery = 0;
spin_lock_irq(&conf->device_lock);
--
2.53.0
^ permalink raw reply related
* [PATCH 4/4] md/raid10: reset read_slot when reusing r10bio for discard
From: Chen Cheng @ 2026-04-22 2:33 UTC (permalink / raw)
To: linux-raid, yukuai; +Cc: chencheng, chenchneg33
In-Reply-To: <20260422023317.796326-1-chencheng@fnnas.com>
From: Chen Cheng <chencheng@fnnas.com>
raid10_handle_discard() reuses r10bio objects from r10bio_pool.
put_all_bios() always drops devs[i].bio, but it only drops
devs[i].repl_bio when r10_bio->read_slot < 0. If discard reuses an
r10bio that was previously used for a read, read_slot can still be
non-negative, and discard cleanup can skip bio_put() on repl_bio.
Reset read_slot to -1 when preparing an r10bio for discard so the
replacement bio is always released correctly.
---
drivers/md/raid10.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
index 3edde440623a..19d7f6f62beb 100644
--- a/drivers/md/raid10.c
+++ b/drivers/md/raid10.c
@@ -1764,6 +1764,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
r10_bio->mddev = mddev;
r10_bio->state = 0;
r10_bio->sectors = 0;
+ r10_bio->read_slot = -1;
r10_bio->used_nr_devs = geo->raid_disks;
memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) * geo->raid_disks);
wait_blocked_dev(mddev, r10_bio);
--
2.53.0
^ permalink raw reply related
* Re: [PATCH V3 1/3] block: clear BLK_FEAT_PCI_P2PDMA in blk_stack_limits() for non-supporting devices
From: Nitesh Shetty @ 2026-04-22 5:30 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: song@kernel.org, yukuai@fnnas.com, linan122@huawei.com,
kbusch@kernel.org, axboe@kernel.dk, hch@lst.de, sagi@grimberg.me,
linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org,
Kiran Modukuri
In-Reply-To: <a2a14d08-cb31-453a-9ddd-71f7dd4d6992@nvidia.com>
[-- Attachment #1: Type: text/plain, Size: 569 bytes --]
On 21/04/26 10:30PM, Chaitanya Kulkarni wrote:
>On 4/17/26 03:11, Nitesh Shetty wrote:
>> On 16/04/26 02:26PM, Chaitanya Kulkarni wrote:
>>> BLK_FEAT_NOWAIT and BLK_FEAT_POLL are cleared in blk_stack_limits()
>>> when an underlying device does not support them. Apply the same
>>> treatment to BLK_FEAT_PCI_P2PDMA: stacking drivers set it
>>> unconditionally and rely on the core to clear it whenever a
>>> non-supporting member device is stacked.
>>>
>>> Signed-off-by: Chaitanya Kulkarni <kch@nvidia.com>
>>> ---
Reviewed-by: Nitesh Shetty <nj.shetty@samsung.com>
[-- Attachment #2: Type: text/plain, Size: 0 bytes --]
^ permalink raw reply
* Re: [PATCH V3 1/3] block: clear BLK_FEAT_PCI_P2PDMA in blk_stack_limits() for non-supporting devices
From: Christoph Hellwig @ 2026-04-22 6:20 UTC (permalink / raw)
To: Nitesh Shetty
Cc: Chaitanya Kulkarni, song, yukuai, linan122, kbusch, axboe, hch,
sagi, linux-raid, linux-nvme, kmodukuri
In-Reply-To: <20260417101127.uku7q65akj6gwwh4@green245.gost>
On Fri, Apr 17, 2026 at 03:41:27PM +0530, Nitesh Shetty wrote:
>>
>> t->flags |= (b->flags & BLK_FLAG_MISALIGNED);
> I think you need below patch[1] as well to unset this here.
> Also I feel better to include Mike,Mikulas and dm-devel mailing list as well.
Why do we need that? dm support is a different project and will require
additional verification. Especially as the multiple tables per devices
concept makes it a lot more complicated.
^ permalink raw reply
* Re: [PATCH V3 0/2] md/nvme: Enable PCI P2PDMA support for RAID0 and NVMe Multipath
From: hch @ 2026-04-22 6:22 UTC (permalink / raw)
To: Chaitanya Kulkarni
Cc: song@kernel.org, yukuai@fnnas.com, linan122@huawei.com,
kbusch@kernel.org, axboe@kernel.dk, hch@lst.de,
linux-raid@vger.kernel.org, linux-nvme@lists.infradead.org,
Kiran Modukuri, sagi@grimberg.me
In-Reply-To: <3c4d2714-cbd4-40ce-b845-be9a02561d71@nvidia.com>
On Tue, Apr 21, 2026 at 10:32:14PM +0000, Chaitanya Kulkarni wrote:
[full quote deleted]
> If there is no objection is it possible to merge this via
> linux-block/for-next ?
We're too late for 7.1, so let's wait for the block tree to open.
^ permalink raw reply
* Re: [PATCH 1/4] md/raid10: prepare per-r10bio dev slot tracking
From: Paul Menzel @ 2026-04-22 6:40 UTC (permalink / raw)
To: Chen Cheng; +Cc: linux-raid, yukuai, chenchneg33
In-Reply-To: <20260422023317.796326-1-chencheng@fnnas.com>
Dear Cheng,
Am 22.04.26 um 04:33 schrieb Chen Cheng:
> From: Chen Cheng <chencheng@fnnas.com>
>
> raid10 reuses r10bio objects from both r10bio_pool and r10buf_pool. Track
> the number of devs[] slots used by each request in the r10bio itself and
> initialize it whenever one of these objects is reused.
>
> No functional change yet. A later patch will use this width when reshape
> changes conf->geo.raid_disks.
Your Signed-off-by: line is missing.
> ---
> drivers/md/raid10.c | 4 ++++
> drivers/md/raid10.h | 1 +
> 2 files changed, 5 insertions(+)
>
> diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c
> index 0653b5d8545a..e93933632893 100644
> --- a/drivers/md/raid10.c
> +++ b/drivers/md/raid10.c
> @@ -1540,6 +1540,7 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors)
> r10_bio->sector = bio->bi_iter.bi_sector;
> r10_bio->state = 0;
> r10_bio->read_slot = -1;
> + r10_bio->used_nr_devs = conf->geo.raid_disks;
> memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) *
> conf->geo.raid_disks);
>
> @@ -1727,6 +1728,7 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio)
> r10_bio->mddev = mddev;
> r10_bio->state = 0;
> r10_bio->sectors = 0;
> + r10_bio->used_nr_devs = geo->raid_disks;
> memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) * geo->raid_disks);
> wait_blocked_dev(mddev, r10_bio);
>
> @@ -3061,6 +3063,8 @@ static struct r10bio *raid10_alloc_init_r10buf(struct r10conf *conf)
> else
> nalloc = 2; /* recovery */
>
> + r10bio->used_nr_devs = nalloc;
> +
> for (i = 0; i < nalloc; i++) {
> bio = r10bio->devs[i].bio;
> rp = bio->bi_private;
> diff --git a/drivers/md/raid10.h b/drivers/md/raid10.h
> index ec79d87fb92f..92e8743023e6 100644
> --- a/drivers/md/raid10.h
> +++ b/drivers/md/raid10.h
> @@ -127,6 +127,7 @@ struct r10bio {
> * if the IO is in READ direction, then this is where we read
> */
> int read_slot;
> + unsigned int used_nr_devs;
Most entries have a comment describing the use. Maybe add one too, or at
least a blank line, so it’s clear that the existing comment is just for
`read_slot`?
>
> struct list_head retry_list;
> /*
From a performance and resource usage point of view, will increasing
the struct have a negative impact?
The diff looks good.
Reviewed-by: Paul Menzel <pmenzel@molgen.mpg.de>
Kind regards,
Paul
^ permalink raw reply
* Re: [RFC PATCH] dm-raid: only requeue bios when dm is suspending.
From: Xiao Ni @ 2026-04-22 9:58 UTC (permalink / raw)
To: Benjamin Marzinski
Cc: Yang Xiuwei, Yu Kuai, Li Nan, Song Liu, linux-raid, dm-devel,
Nigel Croxon
In-Reply-To: <20260414190350.1946406-1-bmarzins@redhat.com>
On Wed, Apr 15, 2026 at 3:03 AM Benjamin Marzinski <bmarzins@redhat.com> wrote:
>
> returning DM_MAPIO_REQUEUE from the target map() function only requeues
> the bio during noflush suspends. During regular operations or during
> flushing suspends, it fails the bio. Failing the bio during flushing
> suspends is the correct behavior here. We cannot handle the bio, and we
> cannot suspends while it is outstanding. But during normal operations,
> we should not push the bio back to do. Instead, wait for the reshape
> to be resumed.
>
> Signed-off-by: Benjamin Marzinski <bmarzins@redhat.com>
> ---
>
> Yang Xiuwei, if you are still able to see I/O errors during LVM testing,
> does this patch fix them?
>
> drivers/md/dm-raid.c | 7 +++++++
> drivers/md/md.h | 1 +
> drivers/md/raid5.c | 6 ++++--
> 3 files changed, 12 insertions(+), 2 deletions(-)
>
> diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c
> index 4bacdc499984..cac61d57e7e2 100644
> --- a/drivers/md/dm-raid.c
> +++ b/drivers/md/dm-raid.c
> @@ -3831,6 +3831,7 @@ static void raid_presuspend(struct dm_target *ti)
> * resume, raid_postsuspend() is too late.
> */
> set_bit(RT_FLAG_RS_FROZEN, &rs->runtime_flags);
> + WRITE_ONCE(mddev->dm_suspending, 1);
>
> if (!reshape_interrupted(mddev))
> return;
> @@ -3847,6 +3848,9 @@ static void raid_presuspend(struct dm_target *ti)
> static void raid_presuspend_undo(struct dm_target *ti)
> {
> struct raid_set *rs = ti->private;
> + struct mddev *mddev = &rs->md;
> +
> + WRITE_ONCE(mddev->dm_suspending, 0);
>
> clear_bit(RT_FLAG_RS_FROZEN, &rs->runtime_flags);
> }
> @@ -3854,6 +3858,7 @@ static void raid_presuspend_undo(struct dm_target *ti)
> static void raid_postsuspend(struct dm_target *ti)
> {
> struct raid_set *rs = ti->private;
> + struct mddev *mddev = &rs->md;
>
> if (!test_and_set_bit(RT_FLAG_RS_SUSPENDED, &rs->runtime_flags)) {
> /*
> @@ -3864,6 +3869,8 @@ static void raid_postsuspend(struct dm_target *ti)
> mddev_suspend(&rs->md, false);
> rs->md.ro = MD_RDONLY;
> }
> + WRITE_ONCE(mddev->dm_suspending, 0);
> +
> }
>
> static void attempt_restore_of_faulty_devices(struct raid_set *rs)
> diff --git a/drivers/md/md.h b/drivers/md/md.h
> index ac84289664cd..e8d7332c5cb9 100644
> --- a/drivers/md/md.h
> +++ b/drivers/md/md.h
> @@ -463,6 +463,7 @@ struct mddev {
> int delta_disks, new_level, new_layout;
> int new_chunk_sectors;
> int reshape_backwards;
> + int dm_suspending;
>
> struct md_thread __rcu *thread; /* management thread */
> struct md_thread __rcu *sync_thread; /* doing resync or reconstruct */
> diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c
> index 8854e024f311..d528263f92a3 100644
> --- a/drivers/md/raid5.c
> +++ b/drivers/md/raid5.c
> @@ -6042,8 +6042,10 @@ static enum stripe_result make_stripe_request(struct mddev *mddev,
> raid5_release_stripe(sh);
> out:
> if (ret == STRIPE_SCHEDULE_AND_RETRY && reshape_interrupted(mddev)) {
> - bi->bi_status = BLK_STS_RESOURCE;
> - ret = STRIPE_WAIT_RESHAPE;
> + if (!mddev_is_dm(mddev) || READ_ONCE(mddev->dm_suspending)) {
> + bi->bi_status = BLK_STS_RESOURCE;
> + ret = STRIPE_WAIT_RESHAPE;
> + }
> pr_err_ratelimited("dm-raid456: io across reshape position while reshape can't make progress");
> }
> return ret;
> --
> 2.50.1
>
Looks good to me.
Reviewed-by: Xiao Ni <xni@redhat.com>
^ permalink raw reply
* [PATCH 0/8] ARM crc64 and XOR using NEON intrinsics
From: Ard Biesheuvel @ 2026-04-22 17:16 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
From: Ard Biesheuvel <ardb@kernel.org>
This is a follow-up to both [0] and [1], both of which included patch #1
of this series, which introduces the asm/neon-intrinsics.h header on
32-bit ARM. The remaining changes rely on this.
The purpose of this series is to streamline / clean up the use of NEON
intrinsics on 32-bit ARM, by sharing more code, clean up Make rules and
finally, getting rid of the hacked up types.h header, which does some
nasty things that are only needed when building NEON intrinsics code.
Patches #2 and #3 replace the ARM autovectorized XOR implementation with
the NEON intrinsics version used by arm64.
Patches #4 and #5 enable the arm64 NEON intrinsics implementation of
crc64 on 32-bit ARM.
Patches #6 and #7 drop the direct includes of <arm_neon.h> and perform
some additional cleanup to reduce the delta between ARM and arm64 code
and Make rules.
It would probably be easiest to take all these changes through a single
tree, and the CRC tree seems like a suitable candidate, if Eric agrees.
Cc: Christoph Hellwig <hch@lst.de>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Eric Biggers <ebiggers@kernel.org>
[0] https://lore.kernel.org/all/20260331074940.55502-7-ardb+git@google.com/
[1] https://lore.kernel.org/all/20260330144630.33026-7-ardb@kernel.org/
Ard Biesheuvel (8):
ARM: Add a neon-intrinsics.h header like on arm64
xor/arm: Replace vectorized implementation with arm64's intrinsics
xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
lib/crc: Turn NEON intrinsics crc64 implementation into common code
lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64
crypto: aegis128 - Use neon-intrinsics.h on ARM too
lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
ARM: Remove hacked-up asm/types.h header
Documentation/arch/arm/kernel_mode_neon.rst | 4 +-
arch/arm/include/asm/neon-intrinsics.h | 60 ++++++++
arch/arm/include/uapi/asm/types.h | 41 ------
crypto/Makefile | 10 +-
crypto/aegis128-neon-inner.c | 4 +-
lib/crc/Kconfig | 1 +
lib/crc/Makefile | 9 +-
lib/crc/arm/crc64-neon.h | 34 +++++
lib/crc/arm/crc64.h | 36 +++++
lib/crc/arm64/crc64-neon.h | 21 +++
lib/crc/arm64/crc64.h | 4 +-
lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} | 26 +---
lib/raid/xor/Makefile | 13 +-
lib/raid/xor/arm/xor-neon.c | 26 ----
lib/raid/xor/arm/xor-neon.h | 7 +
lib/raid/xor/arm/xor_arch.h | 7 +-
lib/raid/xor/arm64/xor-eor3.c | 146 ++++++++++++++++++++
lib/raid/xor/xor-8regs.c | 2 -
lib/raid/xor/{arm64 => }/xor-neon.c | 143 +------------------
lib/raid6/neon.uc | 2 +-
lib/raid6/recov_neon_inner.c | 2 +-
21 files changed, 340 insertions(+), 258 deletions(-)
create mode 100644 arch/arm/include/asm/neon-intrinsics.h
delete mode 100644 arch/arm/include/uapi/asm/types.h
create mode 100644 lib/crc/arm/crc64-neon.h
create mode 100644 lib/crc/arm/crc64.h
create mode 100644 lib/crc/arm64/crc64-neon.h
rename lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} (62%)
delete mode 100644 lib/raid/xor/arm/xor-neon.c
create mode 100644 lib/raid/xor/arm/xor-neon.h
create mode 100644 lib/raid/xor/arm64/xor-eor3.c
rename lib/raid/xor/{arm64 => }/xor-neon.c (56%)
base-commit: 6596a02b207886e9e00bb0161c7fd59fea53c081
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply
* [PATCH 1/8] ARM: Add a neon-intrinsics.h header like on arm64
From: Ard Biesheuvel @ 2026-04-22 17:16 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Add a header asm/neon-intrinsics.h similar to the one that arm64 has.
This makes it possible for NEON intrinsics code to be shared seamlessly
between ARM and arm64.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
Documentation/arch/arm/kernel_mode_neon.rst | 4 +-
arch/arm/include/asm/neon-intrinsics.h | 60 ++++++++++++++++++++
2 files changed, 63 insertions(+), 1 deletion(-)
diff --git a/Documentation/arch/arm/kernel_mode_neon.rst b/Documentation/arch/arm/kernel_mode_neon.rst
index 9bfb71a2a9b9..1efb6d35b7bd 100644
--- a/Documentation/arch/arm/kernel_mode_neon.rst
+++ b/Documentation/arch/arm/kernel_mode_neon.rst
@@ -121,4 +121,6 @@ observe the following in addition to the rules above:
* Compile the unit containing the NEON intrinsics with '-ffreestanding' so GCC
uses its builtin version of <stdint.h> (this is a C99 header which the kernel
does not supply);
-* Include <arm_neon.h> last, or at least after <linux/types.h>
+* Do not include <arm_neon.h> directly: instead, include <asm/neon-intrinsics.h>,
+ which tweaks some macro definitions so that system headers can be included
+ safely.
diff --git a/arch/arm/include/asm/neon-intrinsics.h b/arch/arm/include/asm/neon-intrinsics.h
new file mode 100644
index 000000000000..8b80c05ce1d7
--- /dev/null
+++ b/arch/arm/include/asm/neon-intrinsics.h
@@ -0,0 +1,60 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+#ifndef __ASM_NEON_INTRINSICS_H
+#define __ASM_NEON_INTRINSICS_H
+
+#ifndef __ARM_NEON__
+#error You should compile this file with '-march=armv7-a -mfloat-abi=softfp -mfpu=neon'
+#endif
+
+#include <asm-generic/int-ll64.h>
+
+/*
+ * The C99 types uintXX_t that are usually defined in 'stdint.h' are not as
+ * unambiguous on ARM as you would expect. For the types below, there is a
+ * difference on ARM between GCC built for bare metal ARM, GCC built for glibc
+ * and the kernel itself, which results in build errors if you try to build
+ * with -ffreestanding and include 'stdint.h' (such as when you include
+ * 'arm_neon.h' in order to use NEON intrinsics)
+ *
+ * As the typedefs for these types in 'stdint.h' are based on builtin defines
+ * supplied by GCC, we can tweak these to align with the kernel's idea of those
+ * types, so 'linux/types.h' and 'stdint.h' can be safely included from the
+ * same source file (provided that -ffreestanding is used).
+ *
+ * int32_t uint32_t intptr_t uintptr_t
+ * bare metal GCC long unsigned long int unsigned int
+ * glibc GCC int unsigned int int unsigned int
+ * kernel int unsigned int long unsigned long
+ */
+
+#ifdef __INT32_TYPE__
+#undef __INT32_TYPE__
+#define __INT32_TYPE__ int
+#endif
+
+#ifdef __UINT32_TYPE__
+#undef __UINT32_TYPE__
+#define __UINT32_TYPE__ unsigned int
+#endif
+
+#ifdef __INTPTR_TYPE__
+#undef __INTPTR_TYPE__
+#define __INTPTR_TYPE__ long
+#endif
+
+#ifdef __UINTPTR_TYPE__
+#undef __UINTPTR_TYPE__
+#define __UINTPTR_TYPE__ unsigned long
+#endif
+
+/*
+ * genksyms chokes on the ARM NEON instrinsics system header, but we
+ * don't export anything it defines anyway, so just disregard when
+ * genksyms execute.
+ */
+#ifndef __GENKSYMS__
+#include <arm_neon.h>
+#endif
+
+#endif /* __ASM_NEON_INTRINSICS_H */
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 2/8] xor/arm: Replace vectorized implementation with arm64's intrinsics
From: Ard Biesheuvel @ 2026-04-22 17:16 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Drop the XOR implementation generated by the vectorizer: this has always
been a bit of a hack, and now that arm64 has an intrinsics version that
works on ARM too, let's use that instead.
So copy the part of the arm64 code that can be shared (so not the EOR3
version). The arm64 code will be updated in a subsequent patch to share
this implementation.
Performance (QEMU mach-virt VM running on Synquacer [Cortex-A53 @ 1 GHz]
Before:
[ 3.519687] xor: measuring software checksum speed
[ 3.521725] neon : 1660 MB/sec
[ 3.524733] 32regs : 1105 MB/sec
[ 3.527751] 8regs : 1098 MB/sec
[ 3.529911] arm4regs : 1540 MB/sec
After:
[ 3.517654] xor: measuring software checksum speed
[ 3.519454] neon : 1896 MB/sec
[ 3.522499] 32regs : 1090 MB/sec
[ 3.525560] 8regs : 1083 MB/sec
[ 3.527700] arm4regs : 1556 MB/sec
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/raid/xor/Makefile | 6 +-
lib/raid/xor/arm/xor-neon.c | 26 ---
lib/raid/xor/arm/xor-neon.h | 7 +
lib/raid/xor/arm/xor_arch.h | 7 +-
lib/raid/xor/xor-8regs.c | 2 -
lib/raid/xor/xor-neon.c | 175 ++++++++++++++++++++
6 files changed, 186 insertions(+), 37 deletions(-)
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index 4d633dfd5b90..d78400f2427a 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -17,7 +17,7 @@ endif
xor-$(CONFIG_ALPHA) += alpha/xor.o
xor-$(CONFIG_ARM) += arm/xor.o
ifeq ($(CONFIG_ARM),y)
-xor-$(CONFIG_KERNEL_MODE_NEON) += arm/xor-neon.o arm/xor-neon-glue.o
+xor-$(CONFIG_KERNEL_MODE_NEON) += xor-neon.o arm/xor-neon-glue.o
endif
xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o
@@ -31,8 +31,8 @@ xor-$(CONFIG_X86_32) += x86/xor-avx.o x86/xor-sse.o x86/xor-mmx.o
xor-$(CONFIG_X86_64) += x86/xor-avx.o x86/xor-sse.o
obj-y += tests/
-CFLAGS_arm/xor-neon.o += $(CC_FLAGS_FPU)
-CFLAGS_REMOVE_arm/xor-neon.o += $(CC_FLAGS_NO_FPU)
+CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH)
+CFLAGS_REMOVE_xor-neon.o += $(CC_FLAGS_NO_FPU)
CFLAGS_arm64/xor-neon.o += $(CC_FLAGS_FPU)
CFLAGS_REMOVE_arm64/xor-neon.o += $(CC_FLAGS_NO_FPU)
diff --git a/lib/raid/xor/arm/xor-neon.c b/lib/raid/xor/arm/xor-neon.c
deleted file mode 100644
index 23147e3a7904..000000000000
--- a/lib/raid/xor/arm/xor-neon.c
+++ /dev/null
@@ -1,26 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Copyright (C) 2013 Linaro Ltd <ard.biesheuvel@linaro.org>
- */
-
-#include "xor_impl.h"
-#include "xor_arch.h"
-
-#ifndef __ARM_NEON__
-#error You should compile this file with '-march=armv7-a -mfloat-abi=softfp -mfpu=neon'
-#endif
-
-/*
- * Pull in the reference implementations while instructing GCC (through
- * -ftree-vectorize) to attempt to exploit implicit parallelism and emit
- * NEON instructions. Clang does this by default at O2 so no pragma is
- * needed.
- */
-#ifdef CONFIG_CC_IS_GCC
-#pragma GCC optimize "tree-vectorize"
-#endif
-
-#define NO_TEMPLATE
-#include "../xor-8regs.c"
-
-__DO_XOR_BLOCKS(neon_inner, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);
diff --git a/lib/raid/xor/arm/xor-neon.h b/lib/raid/xor/arm/xor-neon.h
new file mode 100644
index 000000000000..406e0356f05b
--- /dev/null
+++ b/lib/raid/xor/arm/xor-neon.h
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+
+extern struct xor_block_template xor_block_arm4regs;
+extern struct xor_block_template xor_block_neon;
+
+void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt,
+ unsigned int bytes);
diff --git a/lib/raid/xor/arm/xor_arch.h b/lib/raid/xor/arm/xor_arch.h
index 775ff835df65..f1ddb64fe62a 100644
--- a/lib/raid/xor/arm/xor_arch.h
+++ b/lib/raid/xor/arm/xor_arch.h
@@ -3,12 +3,7 @@
* Copyright (C) 2001 Russell King
*/
#include <asm/neon.h>
-
-extern struct xor_block_template xor_block_arm4regs;
-extern struct xor_block_template xor_block_neon;
-
-void xor_gen_neon_inner(void *dest, void **srcs, unsigned int src_cnt,
- unsigned int bytes);
+#include "xor-neon.h"
static __always_inline void __init arch_xor_init(void)
{
diff --git a/lib/raid/xor/xor-8regs.c b/lib/raid/xor/xor-8regs.c
index 1edaed8acffe..46b3c8bdc27f 100644
--- a/lib/raid/xor/xor-8regs.c
+++ b/lib/raid/xor/xor-8regs.c
@@ -93,11 +93,9 @@ xor_8regs_5(unsigned long bytes, unsigned long * __restrict p1,
} while (--lines > 0);
}
-#ifndef NO_TEMPLATE
DO_XOR_BLOCKS(8regs, xor_8regs_2, xor_8regs_3, xor_8regs_4, xor_8regs_5);
struct xor_block_template xor_block_8regs = {
.name = "8regs",
.xor_gen = xor_gen_8regs,
};
-#endif /* NO_TEMPLATE */
diff --git a/lib/raid/xor/xor-neon.c b/lib/raid/xor/xor-neon.c
new file mode 100644
index 000000000000..a3e2b4af8d36
--- /dev/null
+++ b/lib/raid/xor/xor-neon.c
@@ -0,0 +1,175 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Authors: Jackie Liu <liuyun01@kylinos.cn>
+ * Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd.
+ */
+
+#include "xor_impl.h"
+#include "xor-neon.h"
+
+#include <asm/neon-intrinsics.h>
+
+static void __xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* p1 ^= p3 */
+ v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+ uint64_t *dp4 = (uint64_t *)p4;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* p1 ^= p3 */
+ v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
+
+ /* p1 ^= p4 */
+ v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ dp4 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+ uint64_t *dp4 = (uint64_t *)p4;
+ uint64_t *dp5 = (uint64_t *)p5;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 */
+ v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
+ v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
+ v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
+ v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
+
+ /* p1 ^= p3 */
+ v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
+
+ /* p1 ^= p4 */
+ v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
+
+ /* p1 ^= p5 */
+ v0 = veorq_u64(v0, vld1q_u64(dp5 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp5 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp5 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp5 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ dp4 += 8;
+ dp5 += 8;
+ } while (--lines > 0);
+}
+
+__DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4,
+ __xor_neon_5);
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 3/8] xor/arm64: Use shared NEON intrinsics implementation from 32-bit ARM
From: Ard Biesheuvel @ 2026-04-22 17:16 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Tweak the arm64 code so that the pure NEON intrinsics implementation of
XOR is shared between arm64 and ARM. While at it, rename the arm64
specific piece xor-eor3.c to reflect that only the version based on the
EOR3 instruction is kept there.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/raid/xor/Makefile | 7 +-
lib/raid/xor/arm64/xor-eor3.c | 146 +++++++++
lib/raid/xor/arm64/xor-neon.c | 312 --------------------
lib/raid/xor/xor-neon.c | 4 +
4 files changed, 154 insertions(+), 315 deletions(-)
diff --git a/lib/raid/xor/Makefile b/lib/raid/xor/Makefile
index d78400f2427a..e8ecec3c09f9 100644
--- a/lib/raid/xor/Makefile
+++ b/lib/raid/xor/Makefile
@@ -19,7 +19,8 @@ xor-$(CONFIG_ARM) += arm/xor.o
ifeq ($(CONFIG_ARM),y)
xor-$(CONFIG_KERNEL_MODE_NEON) += xor-neon.o arm/xor-neon-glue.o
endif
-xor-$(CONFIG_ARM64) += arm64/xor-neon.o arm64/xor-neon-glue.o
+xor-$(CONFIG_ARM64) += xor-neon.o arm64/xor-eor3.o \
+ arm64/xor-neon-glue.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd.o
xor-$(CONFIG_CPU_HAS_LSX) += loongarch/xor_simd_glue.o
xor-$(CONFIG_ALTIVEC) += powerpc/xor_vmx.o powerpc/xor_vmx_glue.o
@@ -34,8 +35,8 @@ obj-y += tests/
CFLAGS_xor-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH)
CFLAGS_REMOVE_xor-neon.o += $(CC_FLAGS_NO_FPU)
-CFLAGS_arm64/xor-neon.o += $(CC_FLAGS_FPU)
-CFLAGS_REMOVE_arm64/xor-neon.o += $(CC_FLAGS_NO_FPU)
+CFLAGS_arm64/xor-eor3.o += $(CC_FLAGS_FPU)
+CFLAGS_REMOVE_arm64/xor-eor3.o += $(CC_FLAGS_NO_FPU)
CFLAGS_powerpc/xor_vmx.o += -mhard-float -maltivec \
$(call cc-option,-mabi=altivec) \
diff --git a/lib/raid/xor/arm64/xor-eor3.c b/lib/raid/xor/arm64/xor-eor3.c
new file mode 100644
index 000000000000..e44016c363f1
--- /dev/null
+++ b/lib/raid/xor/arm64/xor-eor3.c
@@ -0,0 +1,146 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+#include <linux/cache.h>
+#include <asm/neon-intrinsics.h>
+#include "xor_impl.h"
+#include "xor_arch.h"
+#include "xor-neon.h"
+
+extern void __xor_eor3_2(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2);
+
+static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r)
+{
+ uint64x2_t res;
+
+ asm(ARM64_ASM_PREAMBLE ".arch_extension sha3\n"
+ "eor3 %0.16b, %1.16b, %2.16b, %3.16b"
+ : "=w"(res) : "w"(p), "w"(q), "w"(r));
+ return res;
+}
+
+static void __xor_eor3_3(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 ^ p3 */
+ v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
+ vld1q_u64(dp3 + 0));
+ v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
+ vld1q_u64(dp3 + 2));
+ v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
+ vld1q_u64(dp3 + 4));
+ v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
+ vld1q_u64(dp3 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_eor3_4(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+ uint64_t *dp4 = (uint64_t *)p4;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 ^ p3 */
+ v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
+ vld1q_u64(dp3 + 0));
+ v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
+ vld1q_u64(dp3 + 2));
+ v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
+ vld1q_u64(dp3 + 4));
+ v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
+ vld1q_u64(dp3 + 6));
+
+ /* p1 ^= p4 */
+ v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
+ v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
+ v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
+ v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ dp4 += 8;
+ } while (--lines > 0);
+}
+
+static void __xor_eor3_5(unsigned long bytes, unsigned long * __restrict p1,
+ const unsigned long * __restrict p2,
+ const unsigned long * __restrict p3,
+ const unsigned long * __restrict p4,
+ const unsigned long * __restrict p5)
+{
+ uint64_t *dp1 = (uint64_t *)p1;
+ uint64_t *dp2 = (uint64_t *)p2;
+ uint64_t *dp3 = (uint64_t *)p3;
+ uint64_t *dp4 = (uint64_t *)p4;
+ uint64_t *dp5 = (uint64_t *)p5;
+
+ register uint64x2_t v0, v1, v2, v3;
+ long lines = bytes / (sizeof(uint64x2_t) * 4);
+
+ do {
+ /* p1 ^= p2 ^ p3 */
+ v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
+ vld1q_u64(dp3 + 0));
+ v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
+ vld1q_u64(dp3 + 2));
+ v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
+ vld1q_u64(dp3 + 4));
+ v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
+ vld1q_u64(dp3 + 6));
+
+ /* p1 ^= p4 ^ p5 */
+ v0 = eor3(v0, vld1q_u64(dp4 + 0), vld1q_u64(dp5 + 0));
+ v1 = eor3(v1, vld1q_u64(dp4 + 2), vld1q_u64(dp5 + 2));
+ v2 = eor3(v2, vld1q_u64(dp4 + 4), vld1q_u64(dp5 + 4));
+ v3 = eor3(v3, vld1q_u64(dp4 + 6), vld1q_u64(dp5 + 6));
+
+ /* store */
+ vst1q_u64(dp1 + 0, v0);
+ vst1q_u64(dp1 + 2, v1);
+ vst1q_u64(dp1 + 4, v2);
+ vst1q_u64(dp1 + 6, v3);
+
+ dp1 += 8;
+ dp2 += 8;
+ dp3 += 8;
+ dp4 += 8;
+ dp5 += 8;
+ } while (--lines > 0);
+}
+
+__DO_XOR_BLOCKS(eor3_inner, __xor_eor3_2, __xor_eor3_3, __xor_eor3_4,
+ __xor_eor3_5);
diff --git a/lib/raid/xor/arm64/xor-neon.c b/lib/raid/xor/arm64/xor-neon.c
deleted file mode 100644
index 97ef3cb92496..000000000000
--- a/lib/raid/xor/arm64/xor-neon.c
+++ /dev/null
@@ -1,312 +0,0 @@
-// SPDX-License-Identifier: GPL-2.0-only
-/*
- * Authors: Jackie Liu <liuyun01@kylinos.cn>
- * Copyright (C) 2018,Tianjin KYLIN Information Technology Co., Ltd.
- */
-
-#include <linux/cache.h>
-#include <asm/neon-intrinsics.h>
-#include "xor_impl.h"
-#include "xor_arch.h"
-#include "xor-neon.h"
-
-static void __xor_neon_2(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_neon_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* p1 ^= p3 */
- v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_neon_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
- uint64_t *dp4 = (uint64_t *)p4;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* p1 ^= p3 */
- v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
-
- /* p1 ^= p4 */
- v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- dp4 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
- uint64_t *dp4 = (uint64_t *)p4;
- uint64_t *dp5 = (uint64_t *)p5;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 */
- v0 = veorq_u64(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0));
- v1 = veorq_u64(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2));
- v2 = veorq_u64(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4));
- v3 = veorq_u64(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6));
-
- /* p1 ^= p3 */
- v0 = veorq_u64(v0, vld1q_u64(dp3 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp3 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp3 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp3 + 6));
-
- /* p1 ^= p4 */
- v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
-
- /* p1 ^= p5 */
- v0 = veorq_u64(v0, vld1q_u64(dp5 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp5 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp5 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp5 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- dp4 += 8;
- dp5 += 8;
- } while (--lines > 0);
-}
-
-__DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4,
- __xor_neon_5);
-
-static inline uint64x2_t eor3(uint64x2_t p, uint64x2_t q, uint64x2_t r)
-{
- uint64x2_t res;
-
- asm(ARM64_ASM_PREAMBLE ".arch_extension sha3\n"
- "eor3 %0.16b, %1.16b, %2.16b, %3.16b"
- : "=w"(res) : "w"(p), "w"(q), "w"(r));
- return res;
-}
-
-static void __xor_eor3_3(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 ^ p3 */
- v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
- vld1q_u64(dp3 + 0));
- v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
- vld1q_u64(dp3 + 2));
- v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
- vld1q_u64(dp3 + 4));
- v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
- vld1q_u64(dp3 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_eor3_4(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
- uint64_t *dp4 = (uint64_t *)p4;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 ^ p3 */
- v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
- vld1q_u64(dp3 + 0));
- v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
- vld1q_u64(dp3 + 2));
- v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
- vld1q_u64(dp3 + 4));
- v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
- vld1q_u64(dp3 + 6));
-
- /* p1 ^= p4 */
- v0 = veorq_u64(v0, vld1q_u64(dp4 + 0));
- v1 = veorq_u64(v1, vld1q_u64(dp4 + 2));
- v2 = veorq_u64(v2, vld1q_u64(dp4 + 4));
- v3 = veorq_u64(v3, vld1q_u64(dp4 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- dp4 += 8;
- } while (--lines > 0);
-}
-
-static void __xor_eor3_5(unsigned long bytes, unsigned long * __restrict p1,
- const unsigned long * __restrict p2,
- const unsigned long * __restrict p3,
- const unsigned long * __restrict p4,
- const unsigned long * __restrict p5)
-{
- uint64_t *dp1 = (uint64_t *)p1;
- uint64_t *dp2 = (uint64_t *)p2;
- uint64_t *dp3 = (uint64_t *)p3;
- uint64_t *dp4 = (uint64_t *)p4;
- uint64_t *dp5 = (uint64_t *)p5;
-
- register uint64x2_t v0, v1, v2, v3;
- long lines = bytes / (sizeof(uint64x2_t) * 4);
-
- do {
- /* p1 ^= p2 ^ p3 */
- v0 = eor3(vld1q_u64(dp1 + 0), vld1q_u64(dp2 + 0),
- vld1q_u64(dp3 + 0));
- v1 = eor3(vld1q_u64(dp1 + 2), vld1q_u64(dp2 + 2),
- vld1q_u64(dp3 + 2));
- v2 = eor3(vld1q_u64(dp1 + 4), vld1q_u64(dp2 + 4),
- vld1q_u64(dp3 + 4));
- v3 = eor3(vld1q_u64(dp1 + 6), vld1q_u64(dp2 + 6),
- vld1q_u64(dp3 + 6));
-
- /* p1 ^= p4 ^ p5 */
- v0 = eor3(v0, vld1q_u64(dp4 + 0), vld1q_u64(dp5 + 0));
- v1 = eor3(v1, vld1q_u64(dp4 + 2), vld1q_u64(dp5 + 2));
- v2 = eor3(v2, vld1q_u64(dp4 + 4), vld1q_u64(dp5 + 4));
- v3 = eor3(v3, vld1q_u64(dp4 + 6), vld1q_u64(dp5 + 6));
-
- /* store */
- vst1q_u64(dp1 + 0, v0);
- vst1q_u64(dp1 + 2, v1);
- vst1q_u64(dp1 + 4, v2);
- vst1q_u64(dp1 + 6, v3);
-
- dp1 += 8;
- dp2 += 8;
- dp3 += 8;
- dp4 += 8;
- dp5 += 8;
- } while (--lines > 0);
-}
-
-__DO_XOR_BLOCKS(eor3_inner, __xor_neon_2, __xor_eor3_3, __xor_eor3_4,
- __xor_eor3_5);
diff --git a/lib/raid/xor/xor-neon.c b/lib/raid/xor/xor-neon.c
index a3e2b4af8d36..c7c3cf634e23 100644
--- a/lib/raid/xor/xor-neon.c
+++ b/lib/raid/xor/xor-neon.c
@@ -173,3 +173,7 @@ static void __xor_neon_5(unsigned long bytes, unsigned long * __restrict p1,
__DO_XOR_BLOCKS(neon_inner, __xor_neon_2, __xor_neon_3, __xor_neon_4,
__xor_neon_5);
+
+#ifdef CONFIG_ARM64
+extern typeof(__xor_neon_2) __xor_eor3_2 __alias(__xor_neon_2);
+#endif
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 4/8] lib/crc: Turn NEON intrinsics crc64 implementation into common code
From: Ard Biesheuvel @ 2026-04-22 17:17 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Move and rename the CRC64 NEON intrinsics implementation source file and
rename the function name to reflect that it is NEON code that can be
shared. This will be wired up for 32-bit ARM in a subsequent patch.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/crc/Makefile | 6 ++---
lib/crc/arm64/crc64-neon.h | 21 ++++++++++++++++
lib/crc/arm64/crc64.h | 4 +--
lib/crc/{arm64/crc64-neon-inner.c => crc64-neon.c} | 26 +++-----------------
4 files changed, 30 insertions(+), 27 deletions(-)
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index ff213590e4e3..193257ae466f 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -39,9 +39,9 @@ crc64-y := crc64-main.o
ifeq ($(CONFIG_CRC64_ARCH),y)
CFLAGS_crc64-main.o += -I$(src)/$(SRCARCH)
-CFLAGS_REMOVE_arm64/crc64-neon-inner.o += $(CC_FLAGS_NO_FPU)
-CFLAGS_arm64/crc64-neon-inner.o += $(CC_FLAGS_FPU) -march=armv8-a+crypto
-crc64-$(CONFIG_ARM64) += arm64/crc64-neon-inner.o
+CFLAGS_REMOVE_crc64-neon.o += $(CC_FLAGS_NO_FPU)
+CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) -march=armv8-a+crypto
+crc64-$(CONFIG_ARM64) += crc64-neon.o
crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o
crc64-$(CONFIG_X86) += x86/crc64-pclmul.o
diff --git a/lib/crc/arm64/crc64-neon.h b/lib/crc/arm64/crc64-neon.h
new file mode 100644
index 000000000000..fcd5b1e6f812
--- /dev/null
+++ b/lib/crc/arm64/crc64-neon.h
@@ -0,0 +1,21 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+static inline uint64x2_t pmull64(uint64x2_t a, uint64x2_t b)
+{
+ return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 0),
+ vgetq_lane_u64(b, 0)));
+}
+
+static inline uint64x2_t pmull64_high(uint64x2_t a, uint64x2_t b)
+{
+ poly64x2_t l = vreinterpretq_p64_u64(a);
+ poly64x2_t m = vreinterpretq_p64_u64(b);
+
+ return vreinterpretq_u64_p128(vmull_high_p64(l, m));
+}
+
+static inline uint64x2_t pmull64_hi_lo(uint64x2_t a, uint64x2_t b)
+{
+ return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 1),
+ vgetq_lane_u64(b, 0)));
+}
diff --git a/lib/crc/arm64/crc64.h b/lib/crc/arm64/crc64.h
index 60151ec3035a..c7a69e1f3d8f 100644
--- a/lib/crc/arm64/crc64.h
+++ b/lib/crc/arm64/crc64.h
@@ -8,7 +8,7 @@
#include <linux/minmax.h>
#include <linux/sizes.h>
-u64 crc64_nvme_arm64_c(u64 crc, const u8 *p, size_t len);
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len);
#define crc64_be_arch crc64_be_generic
@@ -19,7 +19,7 @@ static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)
size_t chunk = len & ~15;
scoped_ksimd()
- crc = crc64_nvme_arm64_c(crc, p, chunk);
+ crc = crc64_nvme_neon(crc, p, chunk);
p += chunk;
len &= 15;
diff --git a/lib/crc/arm64/crc64-neon-inner.c b/lib/crc/crc64-neon.c
similarity index 62%
rename from lib/crc/arm64/crc64-neon-inner.c
rename to lib/crc/crc64-neon.c
index 28527e544ff6..4753fb94a4be 100644
--- a/lib/crc/arm64/crc64-neon-inner.c
+++ b/lib/crc/crc64-neon.c
@@ -6,7 +6,9 @@
#include <linux/types.h>
#include <asm/neon-intrinsics.h>
-u64 crc64_nvme_arm64_c(u64 crc, const u8 *p, size_t len);
+#include "crc64-neon.h"
+
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len);
/* x^191 mod G, x^127 mod G */
static const u64 fold_consts_val[2] = { 0xeadc41fd2ba3d420ULL,
@@ -15,27 +17,7 @@ static const u64 fold_consts_val[2] = { 0xeadc41fd2ba3d420ULL,
static const u64 bconsts_val[2] = { 0x27ecfa329aef9f77ULL,
0x34d926535897936aULL };
-static inline uint64x2_t pmull64(uint64x2_t a, uint64x2_t b)
-{
- return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 0),
- vgetq_lane_u64(b, 0)));
-}
-
-static inline uint64x2_t pmull64_high(uint64x2_t a, uint64x2_t b)
-{
- poly64x2_t l = vreinterpretq_p64_u64(a);
- poly64x2_t m = vreinterpretq_p64_u64(b);
-
- return vreinterpretq_u64_p128(vmull_high_p64(l, m));
-}
-
-static inline uint64x2_t pmull64_hi_lo(uint64x2_t a, uint64x2_t b)
-{
- return vreinterpretq_u64_p128(vmull_p64(vgetq_lane_u64(a, 1),
- vgetq_lane_u64(b, 0)));
-}
-
-u64 crc64_nvme_arm64_c(u64 crc, const u8 *p, size_t len)
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len)
{
uint64x2_t fold_consts = vld1q_u64(fold_consts_val);
uint64x2_t v0 = { crc, 0 };
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 5/8] lib/crc: arm: Enable arm64's NEON intrinsics implementation of crc64
From: Ard Biesheuvel @ 2026-04-22 17:17 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Tweak the NEON intrinsics crc64 code written for arm64 so it can be
built for 32-bit ARM as well. The only workaround needed is to provide
alternatives for vmull_p64() and vmull_high_p64() on Clang, which only
defines those when building for the AArch64 or arm64ec ISA. Use the same
helpers for GCC too, to avoid doubling the size of the test/validation
matrix.
KUnit benchmark results (Cortex-A53 @ 1 Ghz)
Before:
# crc64_nvme_benchmark: len=1: 35 MB/s
# crc64_nvme_benchmark: len=16: 78 MB/s
# crc64_nvme_benchmark: len=64: 87 MB/s
# crc64_nvme_benchmark: len=127: 88 MB/s
# crc64_nvme_benchmark: len=128: 88 MB/s
# crc64_nvme_benchmark: len=200: 89 MB/s
# crc64_nvme_benchmark: len=256: 89 MB/s
# crc64_nvme_benchmark: len=511: 89 MB/s
# crc64_nvme_benchmark: len=512: 89 MB/s
# crc64_nvme_benchmark: len=1024: 90 MB/s
# crc64_nvme_benchmark: len=3173: 90 MB/s
# crc64_nvme_benchmark: len=4096: 90 MB/s
# crc64_nvme_benchmark: len=16384: 90 MB/s
After:
# crc64_nvme_benchmark: len=1: 32 MB/s
# crc64_nvme_benchmark: len=16: 76 MB/s
# crc64_nvme_benchmark: len=64: 71 MB/s
# crc64_nvme_benchmark: len=127: 88 MB/s
# crc64_nvme_benchmark: len=128: 618 MB/s
# crc64_nvme_benchmark: len=200: 542 MB/s
# crc64_nvme_benchmark: len=256: 920 MB/s
# crc64_nvme_benchmark: len=511: 836 MB/s
# crc64_nvme_benchmark: len=512: 1261 MB/s
# crc64_nvme_benchmark: len=1024: 1531 MB/s
# crc64_nvme_benchmark: len=3173: 1731 MB/s
# crc64_nvme_benchmark: len=4096: 1851 MB/s
# crc64_nvme_benchmark: len=16384: 1858 MB/s
Don't bother with big-endian, as it doesn't work correctly on Clang, and
is barely used these days.
Note that ARM disables preemption and softirq processing when using
kernel mode SIMD, so take care not to hog the CPU for too long.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/crc/Kconfig | 1 +
lib/crc/Makefile | 5 ++-
lib/crc/arm/crc64-neon.h | 34 ++++++++++++++++++
lib/crc/arm/crc64.h | 36 ++++++++++++++++++++
4 files changed, 75 insertions(+), 1 deletion(-)
diff --git a/lib/crc/Kconfig b/lib/crc/Kconfig
index 31038c8d111a..86a0e4bfec77 100644
--- a/lib/crc/Kconfig
+++ b/lib/crc/Kconfig
@@ -82,6 +82,7 @@ config CRC64
config CRC64_ARCH
bool
depends on CRC64 && CRC_OPTIMIZATIONS
+ default y if ARM && KERNEL_MODE_NEON && !CPU_BIG_ENDIAN
default y if ARM64
default y if RISCV && RISCV_ISA_ZBC && 64BIT
default y if X86_64
diff --git a/lib/crc/Makefile b/lib/crc/Makefile
index 193257ae466f..386e9c175263 100644
--- a/lib/crc/Makefile
+++ b/lib/crc/Makefile
@@ -39,8 +39,11 @@ crc64-y := crc64-main.o
ifeq ($(CONFIG_CRC64_ARCH),y)
CFLAGS_crc64-main.o += -I$(src)/$(SRCARCH)
+crc64-cflags-$(CONFIG_ARM) += -march=armv8-a -mfpu=crypto-neon-fp-armv8
+crc64-cflags-$(CONFIG_ARM64) += -march=armv8-a+crypto
CFLAGS_REMOVE_crc64-neon.o += $(CC_FLAGS_NO_FPU)
-CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) -march=armv8-a+crypto
+CFLAGS_crc64-neon.o += $(CC_FLAGS_FPU) -I$(src)/$(SRCARCH) $(crc64-cflags-y)
+crc64-$(CONFIG_ARM) += crc64-neon.o
crc64-$(CONFIG_ARM64) += crc64-neon.o
crc64-$(CONFIG_RISCV) += riscv/crc64_lsb.o riscv/crc64_msb.o
diff --git a/lib/crc/arm/crc64-neon.h b/lib/crc/arm/crc64-neon.h
new file mode 100644
index 000000000000..645f553220ff
--- /dev/null
+++ b/lib/crc/arm/crc64-neon.h
@@ -0,0 +1,34 @@
+// SPDX-License-Identifier: GPL-2.0-only
+
+static inline uint64x2_t pmull64(uint64x2_t a, uint64x2_t b)
+{
+ uint64_t l = vgetq_lane_u64(a, 0);
+ uint64_t m = vgetq_lane_u64(b, 0);
+ uint64x2_t result;
+
+ asm("vmull.p64 %q0, %P1, %P2" : "=w"(result) : "w"(l), "w"(m));
+
+ return result;
+}
+
+static inline uint64x2_t pmull64_high(uint64x2_t a, uint64x2_t b)
+{
+ uint64_t l = vgetq_lane_u64(a, 1);
+ uint64_t m = vgetq_lane_u64(b, 1);
+ uint64x2_t result;
+
+ asm("vmull.p64 %q0, %P1, %P2" : "=w"(result) : "w"(l), "w"(m));
+
+ return result;
+}
+
+static inline uint64x2_t pmull64_hi_lo(uint64x2_t a, uint64x2_t b)
+{
+ uint64_t l = vgetq_lane_u64(a, 1);
+ uint64_t m = vgetq_lane_u64(b, 0);
+ uint64x2_t result;
+
+ asm("vmull.p64 %q0, %P1, %P2" : "=w"(result) : "w"(l), "w"(m));
+
+ return result;
+}
diff --git a/lib/crc/arm/crc64.h b/lib/crc/arm/crc64.h
new file mode 100644
index 000000000000..de274288af61
--- /dev/null
+++ b/lib/crc/arm/crc64.h
@@ -0,0 +1,36 @@
+/* SPDX-License-Identifier: GPL-2.0-only */
+/*
+ * CRC64 using ARM PMULL instructions
+ */
+
+#include <asm/simd.h>
+
+static __ro_after_init DEFINE_STATIC_KEY_FALSE(have_pmull);
+
+u64 crc64_nvme_neon(u64 crc, const u8 *p, size_t len);
+
+#define crc64_be_arch crc64_be_generic
+
+static inline u64 crc64_nvme_arch(u64 crc, const u8 *p, size_t len)
+{
+ if (len >= 128 && static_branch_likely(&have_pmull) &&
+ likely(may_use_simd())) {
+ do {
+ size_t chunk = min_t(size_t, len & ~15, SZ_4K);
+
+ scoped_ksimd()
+ crc = crc64_nvme_neon(crc, p, chunk);
+
+ p += chunk;
+ len -= chunk;
+ } while (len >= 128);
+ }
+ return crc64_nvme_generic(crc, p, len);
+}
+
+#define crc64_mod_init_arch crc64_mod_init_arch
+static void crc64_mod_init_arch(void)
+{
+ if (elf_hwcap2 & HWCAP2_PMULL)
+ static_branch_enable(&have_pmull);
+}
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 6/8] crypto: aegis128 - Use neon-intrinsics.h on ARM too
From: Ard Biesheuvel @ 2026-04-22 17:17 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
Use the asm/neon-intrinsics.h header on ARM as well as arm64, so that
the calling code does not have to know the difference.
Clean up the Makefile a bit while at it.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
crypto/Makefile | 10 ++++------
crypto/aegis128-neon-inner.c | 4 +---
2 files changed, 5 insertions(+), 9 deletions(-)
diff --git a/crypto/Makefile b/crypto/Makefile
index 162242593c7c..69d1a18e8519 100644
--- a/crypto/Makefile
+++ b/crypto/Makefile
@@ -103,13 +103,14 @@ obj-$(CONFIG_CRYPTO_CHACHA20POLY1305) += chacha20poly1305.o
obj-$(CONFIG_CRYPTO_AEGIS128) += aegis128.o
aegis128-y := aegis128-core.o
+CFLAGS_aegis128-neon-inner.o += $(CC_FLAGS_FPU)
+CFLAGS_REMOVE_aegis128-neon-inner.o += $(CC_FLAGS_NO_FPU)
ifeq ($(ARCH),arm)
-CFLAGS_aegis128-neon-inner.o += -ffreestanding -march=armv8-a -mfloat-abi=softfp
-CFLAGS_aegis128-neon-inner.o += -mfpu=crypto-neon-fp-armv8
+CFLAGS_aegis128-neon-inner.o += -march=armv8-a -mfpu=crypto-neon-fp-armv8
aegis128-$(CONFIG_CRYPTO_AEGIS128_SIMD) += aegis128-neon.o aegis128-neon-inner.o
endif
ifeq ($(ARCH),arm64)
-aegis128-cflags-y := -ffreestanding -mcpu=generic+crypto
+aegis128-cflags-y := -mcpu=generic+crypto
aegis128-cflags-$(CONFIG_CC_IS_GCC) += -ffixed-q16 -ffixed-q17 -ffixed-q18 \
-ffixed-q19 -ffixed-q20 -ffixed-q21 \
-ffixed-q22 -ffixed-q23 -ffixed-q24 \
@@ -117,11 +118,8 @@ aegis128-cflags-$(CONFIG_CC_IS_GCC) += -ffixed-q16 -ffixed-q17 -ffixed-q18 \
-ffixed-q28 -ffixed-q29 -ffixed-q30 \
-ffixed-q31
CFLAGS_aegis128-neon-inner.o += $(aegis128-cflags-y)
-CFLAGS_REMOVE_aegis128-neon-inner.o += -mgeneral-regs-only
aegis128-$(CONFIG_CRYPTO_AEGIS128_SIMD) += aegis128-neon.o aegis128-neon-inner.o
endif
-# Enable <arm_neon.h>
-CFLAGS_aegis128-neon-inner.o += -isystem $(shell $(CC) -print-file-name=include)
obj-$(CONFIG_CRYPTO_PCRYPT) += pcrypt.o
obj-$(CONFIG_CRYPTO_CRYPTD) += cryptd.o
diff --git a/crypto/aegis128-neon-inner.c b/crypto/aegis128-neon-inner.c
index b6a52a386b22..56b534eeb680 100644
--- a/crypto/aegis128-neon-inner.c
+++ b/crypto/aegis128-neon-inner.c
@@ -3,13 +3,11 @@
* Copyright (C) 2019 Linaro, Ltd. <ard.biesheuvel@linaro.org>
*/
-#ifdef CONFIG_ARM64
#include <asm/neon-intrinsics.h>
+#ifdef CONFIG_ARM64
#define AES_ROUND "aese %0.16b, %1.16b \n\t aesmc %0.16b, %0.16b"
#else
-#include <arm_neon.h>
-
#define AES_ROUND "aese.8 %q0, %q1 \n\t aesmc.8 %q0, %q0"
#endif
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 7/8] lib/raid6: Include asm/neon-intrinsics.h rather than arm_neon.h
From: Ard Biesheuvel @ 2026-04-22 17:17 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
arm_neon.h is a compiler header which needs some scaffolding to work
correctly in the linux context, and so it is better not to include it
directly. Both ARM and arm64 now provide asm/neon-intrinsics.h which
takes care of this.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
lib/raid6/neon.uc | 2 +-
lib/raid6/recov_neon_inner.c | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/lib/raid6/neon.uc b/lib/raid6/neon.uc
index 355270af0cd6..3dc20511103a 100644
--- a/lib/raid6/neon.uc
+++ b/lib/raid6/neon.uc
@@ -24,7 +24,7 @@
* This file is postprocessed using unroll.awk
*/
-#include <arm_neon.h>
+#include <asm/neon-intrinsics.h>
#include "neon.h"
typedef uint8x16_t unative_t;
diff --git a/lib/raid6/recov_neon_inner.c b/lib/raid6/recov_neon_inner.c
index f9e7e8f5a151..06b2967fb8b6 100644
--- a/lib/raid6/recov_neon_inner.c
+++ b/lib/raid6/recov_neon_inner.c
@@ -4,7 +4,7 @@
* Copyright (C) 2017 Linaro Ltd. <ard.biesheuvel@linaro.org>
*/
-#include <arm_neon.h>
+#include <asm/neon-intrinsics.h>
#include "neon.h"
#ifdef CONFIG_ARM
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
* [PATCH 8/8] ARM: Remove hacked-up asm/types.h header
From: Ard Biesheuvel @ 2026-04-22 17:17 UTC (permalink / raw)
To: linux-arm-kernel
Cc: linux-crypto, linux-raid, Ard Biesheuvel, Christoph Hellwig,
Russell King, Arnd Bergmann, Eric Biggers
In-Reply-To: <20260422171655.3437334-10-ardb+git@google.com>
From: Ard Biesheuvel <ardb@kernel.org>
ARM has a special version of asm/types.h which contains overrides for
certain #define's related to the C types used to back C99 types such as
uint32_t and uintptr_t.
This is only needed when pulling in system headers such as stdint.h
during the build, and this only happens when using NEON intrinsics,
for which there is now a dedicated header file.
So drop this header entirely, and revert to the asm-generic one.
Signed-off-by: Ard Biesheuvel <ardb@kernel.org>
---
arch/arm/include/uapi/asm/types.h | 41 --------------------
1 file changed, 41 deletions(-)
diff --git a/arch/arm/include/uapi/asm/types.h b/arch/arm/include/uapi/asm/types.h
deleted file mode 100644
index 1a667bc26510..000000000000
--- a/arch/arm/include/uapi/asm/types.h
+++ /dev/null
@@ -1,41 +0,0 @@
-/* SPDX-License-Identifier: GPL-2.0 WITH Linux-syscall-note */
-#ifndef _UAPI_ASM_TYPES_H
-#define _UAPI_ASM_TYPES_H
-
-#include <asm-generic/int-ll64.h>
-
-/*
- * The C99 types uintXX_t that are usually defined in 'stdint.h' are not as
- * unambiguous on ARM as you would expect. For the types below, there is a
- * difference on ARM between GCC built for bare metal ARM, GCC built for glibc
- * and the kernel itself, which results in build errors if you try to build with
- * -ffreestanding and include 'stdint.h' (such as when you include 'arm_neon.h'
- * in order to use NEON intrinsics)
- *
- * As the typedefs for these types in 'stdint.h' are based on builtin defines
- * supplied by GCC, we can tweak these to align with the kernel's idea of those
- * types, so 'linux/types.h' and 'stdint.h' can be safely included from the same
- * source file (provided that -ffreestanding is used).
- *
- * int32_t uint32_t uintptr_t
- * bare metal GCC long unsigned long unsigned int
- * glibc GCC int unsigned int unsigned int
- * kernel int unsigned int unsigned long
- */
-
-#ifdef __INT32_TYPE__
-#undef __INT32_TYPE__
-#define __INT32_TYPE__ int
-#endif
-
-#ifdef __UINT32_TYPE__
-#undef __UINT32_TYPE__
-#define __UINT32_TYPE__ unsigned int
-#endif
-
-#ifdef __UINTPTR_TYPE__
-#undef __UINTPTR_TYPE__
-#define __UINTPTR_TYPE__ unsigned long
-#endif
-
-#endif /* _UAPI_ASM_TYPES_H */
--
2.54.0.rc1.555.g9c883467ad-goog
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox