* Re: [PATCHv2] blk-mq: check for stale cached request in blk_mq_submit_bio
From: Ming Lei @ 2026-05-19 3:36 UTC (permalink / raw)
To: Keith Busch; +Cc: Keith Busch, axboe, hch, linux-block
In-Reply-To: <agvT6mMM5Tc2ABbk@kbusch-mbp>
On Mon, May 18, 2026 at 09:07:22PM -0600, Keith Busch wrote:
> On Tue, May 19, 2026 at 09:24:50AM +0800, Ming Lei wrote:
> > On Fri, May 01, 2026 at 10:41:19AM -0700, Keith Busch wrote:
> > >
> > > new_request:
> > > if (rq) {
> > > - blk_mq_use_cached_rq(rq, plug, bio);
> > > + if (!blk_mq_use_cached_rq(rq, plug, bio)) {
> > > + struct bio_integrity_payload *bip = bio_integrity(bio);
> > > +
> > > + if (bip && (bip->bip_flags & BIP_BLOCK_INTEGRITY))
> > > + bio_integrity_free(bio);
> > > + rq = NULL;
> > > + goto retry;
> > > + }
> >
> > This way still looks fragile, given rq isn't guaranteed to be live.
>
> Yeah. The current usage in this patch looks like it should work out, but
> not necessarily by design, so yes, fragile.
>
> > Just wondering why not pop it from plug->cached_rqs from beginning, and
> > move it back in case of `queue_exit:`?
>
> Not sure. I guess it's an optimization when the bio merges with a
> previous request in the plug? If we popped initially, a successful merge
> has to push it back. Maybe there's a better reason I'm not seeing.
Yes, this way may avoid potential perf regression for heavy merge cases.
Thanks,
Ming
^ permalink raw reply
* Re: [PATCHv2] blk-mq: check for stale cached request in blk_mq_submit_bio
From: Keith Busch @ 2026-05-19 3:07 UTC (permalink / raw)
To: Ming Lei; +Cc: Keith Busch, axboe, hch, linux-block
In-Reply-To: <agu74lMg10kxPXrl@fedora>
On Tue, May 19, 2026 at 09:24:50AM +0800, Ming Lei wrote:
> On Fri, May 01, 2026 at 10:41:19AM -0700, Keith Busch wrote:
> >
> > new_request:
> > if (rq) {
> > - blk_mq_use_cached_rq(rq, plug, bio);
> > + if (!blk_mq_use_cached_rq(rq, plug, bio)) {
> > + struct bio_integrity_payload *bip = bio_integrity(bio);
> > +
> > + if (bip && (bip->bip_flags & BIP_BLOCK_INTEGRITY))
> > + bio_integrity_free(bio);
> > + rq = NULL;
> > + goto retry;
> > + }
>
> This way still looks fragile, given rq isn't guaranteed to be live.
Yeah. The current usage in this patch looks like it should work out, but
not necessarily by design, so yes, fragile.
> Just wondering why not pop it from plug->cached_rqs from beginning, and
> move it back in case of `queue_exit:`?
Not sure. I guess it's an optimization when the bio merges with a
previous request in the plug? If we popped initially, a successful merge
has to push it back. Maybe there's a better reason I'm not seeing.
^ permalink raw reply
* Re: [PATCH v13 5/8] blk-mq: use hk cpus only when isolcpus=io_queue is enabled
From: Aaron Tomlin @ 2026-05-19 2:50 UTC (permalink / raw)
To: Daniel Wagner
Cc: Daniel Wagner, axboe, kbusch, hch, sagi, mst, aacraid,
James.Bottomley, martin.petersen, liyihang9, kashyap.desai,
sumit.saxena, shivasharan.srikanteshwara, chandrakanth.patil,
sathya.prakash, sreekanth.reddy, suganath-prabu.subramani,
ranjan.kumar, jinpu.wang, tglx, mingo, peterz, juri.lelli,
vincent.guittot, akpm, maz, ruanjinjie, bigeasy, yphbchou0911,
wagi, frederic, longman, chenridong, hare, kch, ming.lei,
tom.leiming, steve, sean, chjohnst, neelx, mproche, nick.lange,
marco.crivellari, rishil1999, linux-block, linux-kernel
In-Reply-To: <70b4cbf0-9137-41de-881b-ff2a60f084ec@flourine.local>
[-- Attachment #1: Type: text/plain, Size: 1486 bytes --]
On Thu, May 14, 2026 at 12:42:57PM +0200, Daniel Wagner wrote:
> On Wed, May 13, 2026 at 07:30:44PM -0400, Aaron Tomlin wrote:
> > > I think this should also check if the CPU is online. The fallback should
> > > ensure the system will boot.
> >
> > While I completely agree with the goal of ensuring a safe boot, I suspect
> > that we actually must iterate over for_each_possible_cpu(cpu) here to
> > prevent map boundary violations during CPU hotplug events.
>
> I have to check this version of the patch what is happening in detail.
> Originally the idea was that the fallback should only be taken if it's
> not possible to create a valid mapping due to $reason. In this case the
> system should still boot and issue a warning. This is also what the
> comments says in the function, pick one online CPU and map all hctx to
> it. Don't bother to be smart.
Hi Daniel,
Understood.
Previously, heavy isolation could starve a device of its minvec
requirement, forcing the block layer into this fallback.
The next iteration of this series has been updated to ensure a minimum
vector safety net. To prevent fatal -ENOSPC device probe aborts on heavily
isolated systems (i.e., where the housekeeping CPU count might be lower
than a device's structural minimum), the final vector calculation is
safeguarded to never drop below minvec. Queues will safely share the
available housekeeping CPUs instead of failing the probe.
Kind regards,
--
Aaron Tomlin
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCHv2] blk-mq: check for stale cached request in blk_mq_submit_bio
From: Ming Lei @ 2026-05-19 1:24 UTC (permalink / raw)
To: Keith Busch; +Cc: axboe, hch, linux-block, Keith Busch
In-Reply-To: <20260501174120.403960-1-kbusch@meta.com>
On Fri, May 01, 2026 at 10:41:19AM -0700, Keith Busch wrote:
> From: Keith Busch <kbusch@kernel.org>
>
> When submitting a bio to blk-mq, if the task should sleep after peeking
> a cached request, but before it pops it, the plug flushes and calls
> blk_mq_free_plug_rqs, freeing the cached_rqs. This creates a
> use-after-free bug. Fix this by ensuring the cached_rqs still contains
> our peeked request, and retry the bio submission without it if the
> request had been freed.
Yeah, it is one real bug.
>
> The code had already warned of this possibility, and specifically popped
> the request before other known blocking calls, but it didn't handle a
> blocking GFP_NOIO alloc. Under memory pressure, allocating the split bio
> or the integrity payload are two such cases that can block. The blk-mq
> submit_bio function continues using the peeked request that was already
> freed and re-initialized, so the driver receives that request with a
> NULL'ed mq_hctx, and inevitably panics.
>
> Relevant kernel messages if you should encounter this condition, where
> the "WARNING" is the harbinger of the panic about to happen:
>
> ------------[ cut here ]------------
> WARNING: CPU: 4 PID: 80820 at block/blk-mq.c:3071 blk_mq_submit_bio+0x2cf/0x5b0
> ...
> BUG: kernel NULL pointer dereference, address: 0000000000000100
> #PF: supervisor read access in kernel mode
> #PF: error_code(0x0000) - not-present page
> PGD 6b367b067 P4D 6b367b067 PUD 6bb5eb067 PMD 0
> Oops: Oops: 0000 [#1] SMP
> ...
> Call Trace:
> <TASK>
> blk_mq_dispatch_queue_requests+0x46/0x120
> blk_mq_flush_plug_list+0x38/0x130
> blk_add_rq_to_plug+0xa2/0x160
> blk_mq_submit_bio+0x3ab/0x5b0
> __submit_bio+0x3a/0x260
> submit_bio_noacct_nocheck+0xc6/0x2b0
> btrfs_submit_bbio+0x14d/0x520
> ? btrfs_get_extent+0x43f/0x640
> submit_extent_folio+0x31f/0x340
> btrfs_do_readpage+0x2d7/0xac0
> btrfs_readahead+0x142/0x200
> ? clear_state_bit+0x520/0x520
> read_pages+0x57/0x200
> ? folio_alloc_noprof+0x10c/0x310
> page_cache_ra_unbounded+0x28c/0x480
> ? asm_sysvec_call_function+0x16/0x20
> ? blk_cgroup_congested+0xa/0x50
> ? page_cache_sync_ra+0x41/0x2d0
> filemap_get_pages+0x347/0xd50
> filemap_read+0xd3/0x500
> ? 0xffffffff81000000
> __io_read+0x111/0x440
> io_read+0x23/0x90
> __io_issue_sqe+0x40/0x120
> io_issue_sqe+0x3f/0x3a0
> io_submit_sqes+0x2bd/0x790
> __se_sys_io_uring_enter+0x100/0xc10
> ? eventfd_read+0x100/0x1f0
> ? futex_wake+0x1b9/0x260
> ? syscall_trace_enter+0x34/0x1d0
> do_syscall_64+0x6a/0x250
> entry_SYSCALL_64_after_hwframe+0x4b/0x53
>
> Fixes: b0077e269f6c1 ("blk-mq: make sure active queue usage is held for bio_integrity_prep()")
> Fixes: 7b4f36cd22a65 ("block: ensure we hold a queue reference when using queue limits")
> Signed-off-by: Keith Busch <kbusch@kernel.org>
> ---
> v1->v2:
>
> Warn if the cached_rqs is not NULL when the peeked request isn't the
> top. This should never happen, but such a bug would be difficult to
> figure out was happening without the warning. The previous warn was
> essential to figure out the bug this patch is addressing.
>
> If the peeked request was freed, rerun the entire bio setup. The first
> version potentially performed operations outside the queue usage
> counter protection, so may have resulted in an invalid bio if it was
> racing against a driver updating queue limites. This retry also
> required clearing the integrity allocation if it was done since
> updated limits may make any previous setup invalid.
>
> block/blk-mq.c | 29 +++++++++++++++++++++--------
> 1 file changed, 21 insertions(+), 8 deletions(-)
>
> diff --git a/block/blk-mq.c b/block/blk-mq.c
> index 4c5c16cce4f8f..73ef3e4be5123 100644
> --- a/block/blk-mq.c
> +++ b/block/blk-mq.c
> @@ -3096,22 +3096,27 @@ static struct request *blk_mq_peek_cached_request(struct blk_plug *plug,
> return rq;
> }
>
> -static void blk_mq_use_cached_rq(struct request *rq, struct blk_plug *plug,
> +static bool blk_mq_use_cached_rq(struct request *rq, struct blk_plug *plug,
> struct bio *bio)
> {
> - if (rq_list_pop(&plug->cached_rqs) != rq)
> - WARN_ON_ONCE(1);
> -
> /*
> - * If any qos ->throttle() end up blocking, we will have flushed the
> - * plug and hence killed the cached_rq list as well. Pop this entry
> - * before we throttle.
> + * We will have flushed the plug and hence killed the cached_rq list as
> + * well if anything had scheduled. Pop this entry before we throttle if
> + * the entry is still valid.
> */
> + struct request *popped = rq_list_pop(&plug->cached_rqs);
> +
> + if (popped != rq) {
> + WARN_ON_ONCE(popped);
> + return false;
> + }
> +
> rq_qos_throttle(rq->q, bio);
>
> blk_mq_rq_time_init(rq, blk_time_get_ns());
> rq->cmd_flags = bio->bi_opf;
> INIT_LIST_HEAD(&rq->queuelist);
> + return true;
> }
>
> static bool bio_unaligned(const struct bio *bio, struct request_queue *q)
> @@ -3154,6 +3159,7 @@ void blk_mq_submit_bio(struct bio *bio)
> */
> rq = blk_mq_peek_cached_request(plug, q, bio->bi_opf);
>
> +retry:
> /*
> * A BIO that was released from a zone write plug has already been
> * through the preparation in this function, already holds a reference
> @@ -3211,7 +3217,14 @@ void blk_mq_submit_bio(struct bio *bio)
>
> new_request:
> if (rq) {
> - blk_mq_use_cached_rq(rq, plug, bio);
> + if (!blk_mq_use_cached_rq(rq, plug, bio)) {
> + struct bio_integrity_payload *bip = bio_integrity(bio);
> +
> + if (bip && (bip->bip_flags & BIP_BLOCK_INTEGRITY))
> + bio_integrity_free(bio);
> + rq = NULL;
> + goto retry;
> + }
This way still looks fragile, given rq isn't guaranteed to be live.
Just wondering why not pop it from plug->cached_rqs from beginning, and
move it back in case of `queue_exit:`?
Thanks,
Ming
^ permalink raw reply
* Re: [PATCH] blk-cgroup-rwstat: add missing DISCARD counter in blkg_rwstat_total()
From: Tao Cui @ 2026-05-19 1:08 UTC (permalink / raw)
To: Tejun Heo; +Cc: axboe, linux-block
In-Reply-To: <agtmKyx4GqRQrftf@slm.duckdns.org>
在 2026/5/19 3:19, Tejun Heo 写道:
> On Sat, May 16, 2026 at 12:47:52PM +0800, Tao Cui wrote:
>> Since commit 636620b66d5d ("blkcg: Track DISCARD statistics and output
>> them in cgroup io.stat"), blkg_rwstat_add() routes discard IOs into
>> BLKG_RWSTAT_DISCARD, but blkg_rwstat_total() still only sums READ and
>> WRITE. This makes the total inconsistent with the per-type counters
>> whenever discard IOs are present.
>>
>> On mixed read/write/discard workloads, the total undercounts the
>> actual IO activity. If only discard requests are queued, the total
>> returns 0 outright. In BFQ this causes avg_queue_size to be
>> underestimated and the group to be incorrectly marked empty.
>>
>> The affected BFQ code paths are all gated by CONFIG_BFQ_CGROUP_DEBUG
>> and used only for statistics display (bfq.sectors, bfq.avg_queue_size,
>> bfq.empty_time). No scheduling decisions depend on these values.
>
> I don't think including discard in total io amount is an improvement. This
> will throw off a lot of people.
>
Thanks for the review.
You're right - for stats.bytes (displayed as
bfq.sectors), including discard doesn't make semantic sense since
discard doesn't transfer data.
The original motivation was the stats.queued case in BFQ: when only
discard requests are pending, blkg_rwstat_total() returns 0, so
bfqg_stats_set_start_empty_time() incorrectly marks the group as empty.
However, this code path is gated by CONFIG_BFQ_CGROUP_DEBUG and is
purely for debug display, so the practical impact is minimal.
I'll drop this patch.
Thanks.
--
Tao
^ permalink raw reply
* Re: [PATCH v13 6/8] blk-mq: prevent offlining hk CPUs with associated online isolated CPUs
From: Aaron Tomlin @ 2026-05-19 0:51 UTC (permalink / raw)
To: axboe, kbusch, hch, sagi, mst
Cc: aacraid, James.Bottomley, martin.petersen, liyihang9,
kashyap.desai, sumit.saxena, shivasharan.srikanteshwara,
chandrakanth.patil, sathya.prakash, sreekanth.reddy,
suganath-prabu.subramani, ranjan.kumar, jinpu.wang, tglx, mingo,
peterz, juri.lelli, vincent.guittot, akpm, maz, ruanjinjie,
bigeasy, yphbchou0911, wagi, frederic, longman, chenridong, hare,
kch, ming.lei, tom.leiming, steve, sean, chjohnst, neelx, mproche,
nick.lange, marco.crivellari, rishil1999, linux-block,
linux-kernel
In-Reply-To: <20260513005509.135966-7-atomlin@atomlin.com>
[-- Attachment #1: Type: text/plain, Size: 1028 bytes --]
On Tue, May 12, 2026 at 08:55:07PM -0400, Aaron Tomlin wrote:
> + if (fallback_isolated_cpu != -1) {
> + /*
> + * Use READ_ONCE() to prevent compiler double-fetch TOCTOU
> + * issues if the disk is removed concurrently.
> + */
> + disk = READ_ONCE(hctx->queue->disk);
> + pr_warn("%s: trying to offline hctx%d but online isolated CPU %d is still mapped to it\n",
> + disk ? disk->disk_name : "?", hctx->queue_num,
> + fallback_isolated_cpu);
> + return false;
> + }
This looks like a use-after-free BUG.
Consider the following scenario:
1. The per-CPU hotplug thread reads the pointer, and is immediately
preempted.
2. A concurrent SCSI device teardown can invoke disk_release(), which
frees the gendisk structure.
3. When the hotplug thread resumes, it blindly dereferences
disk->disk_name using a pointer to memory that no longer belongs to
it.
I think we should modify the warning message and drop disk->disk_name.
--
Aaron Tomlin
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH v2] loop: Fix NULL pointer dereference by synchronizing lo_release and loop_queue_rq
From: Andrew Morton @ 2026-05-19 0:40 UTC (permalink / raw)
To: Tetsuo Handa
Cc: Bart Van Assche, Jens Axboe, Christoph Hellwig, Damien Le Moal,
linux-block, LKML
In-Reply-To: <9b2032d6-3f36-4d2b-8128-985c08a4fa37@I-love.SAKURA.ne.jp>
On Fri, 15 May 2026 10:38:36 +0900 Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> wrote:
> The loop driver relies on lo_release() to automatically clear the loop
> device via __loop_clr_fd() when the last file descriptor is closed
> (LO_FLAGS_AUTOCLEAR). Although the backing file structure itself remains
> allocated in memory thanks to proper file reference counting (f_count is
> not zero), a severe race condition exists regarding the visibility of
> the lo->lo_backing_file pointer.
>
> This race window was exposed by commit 65565ca5f99b ("block: unify
> the synchronous bi_end_io callbacks"). By unifying and optimizing
> the synchronous I/O completion path, the timing and scheduling behavior of
> the block layer altered significantly.
> As a result, a highly-concurrent execution pipeline emerged where
> lo_release() can progress to __loop_clr_fd() and nullify
> lo->lo_backing_file while an already-scheduled asynchronous I/O work
> (lo_rw_aio) is just about to be executed by a kworker thread.
>
> Since the kworker enters lo_rw_aio() after lo->lo_backing_file has been
> cleared, it attempts to dereference the now-NULL pointer when initializing
> the kiocb, leading to the reported NULL pointer dereference bug.
>
> To close this race safely without introducing heavy fast-path checks,
> we must ensure that any running or scheduled dispatch threads have
> completed before we nullify the pointer. Since loop_queue_rq() operates
> within the block layer's RCU read-side critical section, invoke
> synchronize_rcu() and drain_workqueue() in __loop_clr_fd() prior to
> clearing lo->lo_backing_file.
AI review asked a couple of questions:
https://sashiko.dev/#/patchset/9b2032d6-3f36-4d2b-8128-985c08a4fa37@I-love.SAKURA.ne.jp
^ permalink raw reply
* Re: [PATCH v2] cgroup/rstat: validate cpu before css_rstat_cpu() access
From: Tejun Heo @ 2026-05-18 23:40 UTC (permalink / raw)
To: Shakeel Butt
Cc: Qing Ming, Josef Bacik, Jens Axboe, Johannes Weiner,
Michal Koutný, Michal Hocko, Roman Gushchin, Muchun Song,
Andrew Morton, Alexei Starovoitov, Hao Luo, Yosry Ahmed, cgroups,
linux-block, linux-kernel, linux-mm, bpf
In-Reply-To: <agubZePrBmStHxhH@linux.dev>
Hello,
On Mon, May 18, 2026 at 04:07:44PM -0700, Shakeel Butt wrote:
> On Mon, May 18, 2026 at 09:36:36AM -1000, Tejun Heo wrote:
> > Hello,
> >
> > > Qing Ming (1):
> > > cgroup/rstat: validate cpu before css_rstat_cpu() access
> >
> > Applied to cgroup/for-7.1-fixes.
> >
> > In hindsight, we should have added a separate kfunc wrapper from the
> > start instead of tagging css_rstat_updated() with __bpf_kfunc directly,
> > which would have avoided the rename. Oh well, it is what it is.
>
> Is it frown upon to change the kfunc signature or remove __bpf_kfunc from a
> function? I am assuming we can but better not to, correct?
We can. It's just a bit gratuitous and causes unnecessary churn. Let's just
leave it be for now. We can clean up later when e.g. more meaningful
restructuring is necessary.
Thanks.
--
tejun
^ permalink raw reply
* Re: [PATCH v2] cgroup/rstat: validate cpu before css_rstat_cpu() access
From: Shakeel Butt @ 2026-05-18 23:07 UTC (permalink / raw)
To: Tejun Heo
Cc: Qing Ming, Josef Bacik, Jens Axboe, Johannes Weiner,
Michal Koutný, Michal Hocko, Roman Gushchin, Muchun Song,
Andrew Morton, Alexei Starovoitov, Hao Luo, Yosry Ahmed, cgroups,
linux-block, linux-kernel, linux-mm, bpf
In-Reply-To: <64f59b64664f769661a8b8cd587c85f8@kernel.org>
On Mon, May 18, 2026 at 09:36:36AM -1000, Tejun Heo wrote:
> Hello,
>
> > Qing Ming (1):
> > cgroup/rstat: validate cpu before css_rstat_cpu() access
>
> Applied to cgroup/for-7.1-fixes.
>
> In hindsight, we should have added a separate kfunc wrapper from the
> start instead of tagging css_rstat_updated() with __bpf_kfunc directly,
> which would have avoided the rename. Oh well, it is what it is.
Is it frown upon to change the kfunc signature or remove __bpf_kfunc from a
function? I am assuming we can but better not to, correct?
^ permalink raw reply
* Re: [PATCH RFC 1/2] nvme: downgrade WARN in nvme_setup_rw to pr_debug
From: Keith Busch @ 2026-05-18 22:41 UTC (permalink / raw)
To: Chao S
Cc: linux-nvme, linux-block, hch, sagi, axboe, Sungwoo Kim, Dave Tian,
Weidong Zhu
In-Reply-To: <CACd_6n29MeACfWpqK5jaSPOVK+q5yop+eeuO_Wk0M0E5Wr7ujw@mail.gmail.com>
On Sat, May 16, 2026 at 11:53:54PM -0400, Chao S wrote:
> Not the freeze. The WARN does not depend on q->limits, but on
> ns->head->ms (read live at dispatch, set inside the freeze window) and
> on REQ_INTEGRITY, never set for EXT_LBAS-non-PI. capacity==0 only
> gates submission (bio_check_eod()), not dispatch: a writeback bio that
> passed bio_check_eod() under the old capacity sits on the task plug
> holding no q_usage_counter ref, so it does not block the freeze;
> blk_finish_plug() flushes it after the update committed head->ms != 0
> (dmesg: the capacity-change line prints before the WARN).
I'm also not what you mean about the task plug here. The plug list holds
requests, which hold queue references. The plug does not hold bios. If
you're not using a preallocated cached request, then the bio just gets
stuck on "bio_queue_enter" until the freeze completes with the new
limits, though it does proceed with potentially outdated assumtions.
So I think the problem is that the early checks are done outside the
queue entered context, and some of the following code depends on those
checks being valid.
My suggestion in the other reply isn't quite right for a couple reasons
(partition handling and issues with nvme multipath path failure), so
don't bother trying that.
^ permalink raw reply
* [PATCH v2 07/21] iov_iter: Add a segmented queue of bio_vec[]
From: David Howells @ 2026-05-18 22:29 UTC (permalink / raw)
To: Christian Brauner, Matthew Wilcox, Christoph Hellwig
Cc: David Howells, Paulo Alcantara, Jens Axboe, Leon Romanovsky,
Steve French, ChenXiaoSong, Marc Dionne, Eric Van Hensbergen,
Dominique Martinet, Ilya Dryomov, Trond Myklebust, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, linux-block
In-Reply-To: <20260518222959.488126-1-dhowells@redhat.com>
Add the concept of a segmented queue of bio_vec[] arrays. This allows an
indefinite quantity of elements to be handled and allows things like
network filesystems and crypto drivers to glue bits on the ends without
having to reallocate the array.
The bvecq struct that defines each segment also carries capacity/usage
information along with flags indicating whether the constituent memory
regions need freeing or unpinning and the file position of the first
element in a segment. The bvecq structs are refcounted to allow a queue to
be extracted in batches and split between a number of subrequests.
The bvecq can have the bio_vec[] it manages allocated in with it, but this
is not required. A flag is provided for if this is the case as comparing
->bv to ->__bv is not sufficient to detect this case.
Add an iterator type ITER_BVECQ for it. This is intended to replace
ITER_FOLIOQ (and ITER_XARRAY).
Note that the prev pointer is only really needed for iov_iter_revert() and
could be dispensed with if struct iov_iter contained the head information
as well as the current point.
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Paulo Alcantara (Red Hat) <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: linux-block@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
include/linux/bvecq.h | 56 +++++++
include/linux/iov_iter.h | 69 +++++++-
include/linux/uio.h | 11 ++
lib/iov_iter.c | 322 ++++++++++++++++++++++++++++++++++++-
lib/scatterlist.c | 70 +++++++-
lib/tests/kunit_iov_iter.c | 262 ++++++++++++++++++++++++++++++
6 files changed, 784 insertions(+), 6 deletions(-)
create mode 100644 include/linux/bvecq.h
diff --git a/include/linux/bvecq.h b/include/linux/bvecq.h
new file mode 100644
index 000000000000..15f16f905877
--- /dev/null
+++ b/include/linux/bvecq.h
@@ -0,0 +1,56 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/* Implementation of a segmented queue of bio_vec[].
+ *
+ * Copyright (C) 2026 Red Hat, Inc. All Rights Reserved.
+ * Written by David Howells (dhowells@redhat.com)
+ */
+
+#ifndef _LINUX_BVECQ_H
+#define _LINUX_BVECQ_H
+
+#include <linux/bvec.h>
+
+/*
+ * The type of memory retention used by the elements in bvecq->bv[] and how to
+ * clean it up.
+ */
+enum bvecq_mem {
+ BVECQ_MEM_EXTERNAL, /* Externally retained memory - no freeing */
+ BVECQ_MEM_PAGECACHE, /* Ref'd pagecache pages - must put */
+ BVECQ_MEM_GUP, /* Pinned memory from get_user_pages() - unpin */
+ BVECQ_MEM_ALLOCED, /* Memory alloc'd by bvecq - can be freed/pooled */
+} __mode(byte);
+
+/*
+ * Segmented bio_vec queue.
+ *
+ * These can be linked together to form messages of indefinite length and
+ * iterated over with an ITER_BVECQ iterator. The list is non-circular; next
+ * and prev are NULL at the ends.
+ *
+ * The bv pointer points to the bio_vec array; this may be __bv if allocated
+ * together. The caller is responsible for determining whether or not this is
+ * the case as the array pointed to by bv may be follow on directly from the
+ * bvecq by accident of allocation (ie. ->bv == ->__bv is *not* sufficient to
+ * determine this).
+ *
+ * The file position and discontiguity flag allow non-contiguous data sets to
+ * be chained together, but still teased apart without the need to convert the
+ * info in the bio_vec back into a folio pointer.
+ */
+struct bvecq {
+ struct bvecq *next; /* Next bvec in the list or NULL */
+ struct bvecq *prev; /* Prev bvec in the list or NULL */
+ unsigned long long fpos; /* File position */
+ refcount_t ref;
+ u32 priv; /* Private data */
+ u16 nr_slots; /* Number of elements in bv[] used */
+ u16 max_slots; /* Number of elements allocated in bv[] */
+ enum bvecq_mem mem_type:3; /* What sort of memory and how to free it */
+ bool inline_bv:1; /* T if __bv[] is being used */
+ bool discontig:1; /* T if not contiguous with previous bvecq */
+ struct bio_vec *bv; /* Pointer to array of page fragments */
+ struct bio_vec __bv[]; /* Default array (if ->inline_bv) */
+};
+
+#endif /* _LINUX_BVECQ_H */
diff --git a/include/linux/iov_iter.h b/include/linux/iov_iter.h
index f9a17fbbd398..c19a4c561ab4 100644
--- a/include/linux/iov_iter.h
+++ b/include/linux/iov_iter.h
@@ -10,6 +10,7 @@
#include <linux/uio.h>
#include <linux/bvec.h>
+#include <linux/bvecq.h>
#include <linux/folio_queue.h>
typedef size_t (*iov_step_f)(void *iter_base, size_t progress, size_t len,
@@ -141,6 +142,66 @@ size_t iterate_bvec(struct iov_iter *iter, size_t len, void *priv, void *priv2,
return progress;
}
+/*
+ * Handle ITER_BVECQ.
+ */
+static __always_inline
+size_t iterate_bvecq(struct iov_iter *iter, size_t len, void *priv, void *priv2,
+ iov_step_f step)
+{
+ const struct bvecq *bq = iter->bvecq;
+ unsigned int slot = iter->bvecq_slot;
+ size_t progress = 0, skip = iter->iov_offset;
+
+ do {
+ const struct bio_vec *bvec;
+ struct page *page;
+ size_t poff, plen;
+ void *base;
+
+ if (slot >= bq->nr_slots) {
+ if (!bq->next)
+ break;
+ bq = bq->next;
+ slot = 0;
+ }
+
+ bvec = &bq->bv[slot];
+ page = bvec->bv_page + (bvec->bv_offset + skip) / PAGE_SIZE;
+ poff = (bvec->bv_offset + skip) % PAGE_SIZE;
+ plen = umin(bvec->bv_len - skip, len);
+
+ while (plen > 0) {
+ size_t part, remain, consumed;
+
+ part = umin(plen, PAGE_SIZE - poff);
+ base = kmap_local_page(page) + poff;
+ remain = step(base, progress, part, priv, priv2);
+ kunmap_local(base);
+
+ consumed = part - remain;
+ progress += consumed;
+ skip += consumed;
+ len -= consumed;
+ if (!len || remain)
+ goto stop;
+ page++;
+ poff = 0;
+ plen -= consumed;
+ }
+
+ skip = 0;
+ slot++;
+ } while (len);
+
+stop:
+ iter->bvecq_slot = slot;
+ iter->bvecq = bq;
+ iter->iov_offset = skip;
+ iter->count -= progress;
+ return progress;
+}
+
/*
* Handle ITER_FOLIOQ.
*/
@@ -306,6 +367,8 @@ size_t iterate_and_advance2(struct iov_iter *iter, size_t len, void *priv,
return iterate_bvec(iter, len, priv, priv2, step);
if (iov_iter_is_kvec(iter))
return iterate_kvec(iter, len, priv, priv2, step);
+ if (iov_iter_is_bvecq(iter))
+ return iterate_bvecq(iter, len, priv, priv2, step);
if (iov_iter_is_folioq(iter))
return iterate_folioq(iter, len, priv, priv2, step);
if (iov_iter_is_xarray(iter))
@@ -342,8 +405,8 @@ size_t iterate_and_advance(struct iov_iter *iter, size_t len, void *priv,
* buffer is presented in segments, which for kernel iteration are broken up by
* physical pages and mapped, with the mapped address being presented.
*
- * [!] Note This will only handle BVEC, KVEC, FOLIOQ, XARRAY and DISCARD-type
- * iterators; it will not handle UBUF or IOVEC-type iterators.
+ * [!] Note This will only handle BVEC, KVEC, BVECQ, FOLIOQ, XARRAY and
+ * DISCARD-type iterators; it will not handle UBUF or IOVEC-type iterators.
*
* A step functions, @step, must be provided, one for handling mapped kernel
* addresses and the other is given user addresses which have the potential to
@@ -370,6 +433,8 @@ size_t iterate_and_advance_kernel(struct iov_iter *iter, size_t len, void *priv,
return iterate_bvec(iter, len, priv, priv2, step);
if (iov_iter_is_kvec(iter))
return iterate_kvec(iter, len, priv, priv2, step);
+ if (iov_iter_is_bvecq(iter))
+ return iterate_bvecq(iter, len, priv, priv2, step);
if (iov_iter_is_folioq(iter))
return iterate_folioq(iter, len, priv, priv2, step);
if (iov_iter_is_xarray(iter))
diff --git a/include/linux/uio.h b/include/linux/uio.h
index a9bc5b3067e3..f7cfa6ea8213 100644
--- a/include/linux/uio.h
+++ b/include/linux/uio.h
@@ -26,6 +26,7 @@ enum iter_type {
ITER_IOVEC,
ITER_BVEC,
ITER_KVEC,
+ ITER_BVECQ,
ITER_FOLIOQ,
ITER_XARRAY,
ITER_DISCARD,
@@ -68,6 +69,7 @@ struct iov_iter {
const struct iovec *__iov;
const struct kvec *kvec;
const struct bio_vec *bvec;
+ const struct bvecq *bvecq;
const struct folio_queue *folioq;
struct xarray *xarray;
void __user *ubuf;
@@ -77,6 +79,7 @@ struct iov_iter {
};
union {
unsigned long nr_segs;
+ u16 bvecq_slot;
u8 folioq_slot;
loff_t xarray_start;
};
@@ -145,6 +148,11 @@ static inline bool iov_iter_is_discard(const struct iov_iter *i)
return iov_iter_type(i) == ITER_DISCARD;
}
+static inline bool iov_iter_is_bvecq(const struct iov_iter *i)
+{
+ return iov_iter_type(i) == ITER_BVECQ;
+}
+
static inline bool iov_iter_is_folioq(const struct iov_iter *i)
{
return iov_iter_type(i) == ITER_FOLIOQ;
@@ -295,6 +303,9 @@ void iov_iter_kvec(struct iov_iter *i, unsigned int direction, const struct kvec
void iov_iter_bvec(struct iov_iter *i, unsigned int direction, const struct bio_vec *bvec,
unsigned long nr_segs, size_t count);
void iov_iter_discard(struct iov_iter *i, unsigned int direction, size_t count);
+void iov_iter_bvec_queue(struct iov_iter *i, unsigned int direction,
+ const struct bvecq *bvecq,
+ unsigned int first_slot, unsigned int offset, size_t count);
void iov_iter_folio_queue(struct iov_iter *i, unsigned int direction,
const struct folio_queue *folioq,
unsigned int first_slot, unsigned int offset, size_t count);
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index cac7d7364bc2..63fc75c2bc48 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -538,6 +538,39 @@ static void iov_iter_iovec_advance(struct iov_iter *i, size_t size)
i->__iov = iov;
}
+static void iov_iter_bvecq_advance(struct iov_iter *i, size_t by)
+{
+ const struct bvecq *bq = i->bvecq;
+ unsigned int slot = i->bvecq_slot;
+
+ if (!i->count)
+ return;
+ i->count -= by;
+
+ by += i->iov_offset; /* From beginning of current segment. */
+ do {
+ size_t len;
+
+ while (slot >= bq->nr_slots) {
+ if (!bq->next)
+ break;
+ bq = bq->next;
+ slot = 0;
+ }
+
+ len = bq->bv[slot].bv_len;
+
+ if (likely(by < len))
+ break;
+ by -= len;
+ slot++;
+ } while (by);
+
+ i->iov_offset = by;
+ i->bvecq_slot = slot;
+ i->bvecq = bq;
+}
+
static void iov_iter_folioq_advance(struct iov_iter *i, size_t size)
{
const struct folio_queue *folioq = i->folioq;
@@ -583,6 +616,8 @@ void iov_iter_advance(struct iov_iter *i, size_t size)
iov_iter_iovec_advance(i, size);
} else if (iov_iter_is_bvec(i)) {
iov_iter_bvec_advance(i, size);
+ } else if (iov_iter_is_bvecq(i)) {
+ iov_iter_bvecq_advance(i, size);
} else if (iov_iter_is_folioq(i)) {
iov_iter_folioq_advance(i, size);
} else if (iov_iter_is_discard(i)) {
@@ -591,6 +626,32 @@ void iov_iter_advance(struct iov_iter *i, size_t size)
}
EXPORT_SYMBOL(iov_iter_advance);
+static void iov_iter_bvecq_revert(struct iov_iter *i, size_t unroll)
+{
+ const struct bvecq *bq = i->bvecq;
+ unsigned int slot = i->bvecq_slot;
+
+ for (;;) {
+ size_t len;
+
+ if (slot == 0) {
+ bq = bq->prev;
+ slot = bq->nr_slots;
+ }
+ slot--;
+
+ len = bq->bv[slot].bv_len;
+ if (unroll <= len) {
+ i->iov_offset = len - unroll;
+ break;
+ }
+ unroll -= len;
+ }
+
+ i->bvecq_slot = slot;
+ i->bvecq = bq;
+}
+
static void iov_iter_folioq_revert(struct iov_iter *i, size_t unroll)
{
const struct folio_queue *folioq = i->folioq;
@@ -648,6 +709,9 @@ void iov_iter_revert(struct iov_iter *i, size_t unroll)
}
unroll -= n;
}
+ } else if (iov_iter_is_bvecq(i)) {
+ i->iov_offset = 0;
+ iov_iter_bvecq_revert(i, unroll);
} else if (iov_iter_is_folioq(i)) {
i->iov_offset = 0;
iov_iter_folioq_revert(i, unroll);
@@ -678,9 +742,24 @@ size_t iov_iter_single_seg_count(const struct iov_iter *i)
if (iov_iter_is_bvec(i))
return min(i->count, i->bvec->bv_len - i->iov_offset);
}
+ if (!i->count)
+ return 0;
+ if (unlikely(iov_iter_is_bvecq(i))) {
+ const struct bvecq *bq = i->bvecq;
+ unsigned int slot = i->bvecq_slot;
+ size_t offset = i->iov_offset;
+
+ while (slot >= bq->nr_slots) {
+ bq = bq->next;
+ if (!bq)
+ return 0;
+ slot = 0;
+ offset = 0;
+ }
+ return umin(i->count, bq->bv[slot].bv_len - offset);
+ }
if (unlikely(iov_iter_is_folioq(i)))
- return !i->count ? 0 :
- umin(folioq_folio_size(i->folioq, i->folioq_slot), i->count);
+ return umin(folioq_folio_size(i->folioq, i->folioq_slot), i->count);
return i->count;
}
EXPORT_SYMBOL(iov_iter_single_seg_count);
@@ -717,6 +796,35 @@ void iov_iter_bvec(struct iov_iter *i, unsigned int direction,
}
EXPORT_SYMBOL(iov_iter_bvec);
+/**
+ * iov_iter_bvec_queue - Initialise an I/O iterator to use a segmented bvec queue
+ * @i: The iterator to initialise.
+ * @direction: The direction of the transfer.
+ * @bvecq: The starting point in the bvec queue.
+ * @first_slot: The first slot in the bvec queue to use
+ * @offset: The offset into the bvec in the first slot to start at
+ * @count: The size of the I/O buffer in bytes.
+ *
+ * Set up an I/O iterator to either draw data out of the buffers attached to an
+ * inode or to inject data into those buffers. The pages *must* be prevented
+ * from evaporation, either by the caller.
+ */
+void iov_iter_bvec_queue(struct iov_iter *i, unsigned int direction,
+ const struct bvecq *bvecq, unsigned int first_slot,
+ unsigned int offset, size_t count)
+{
+ WARN_ON(direction & ~(READ | WRITE));
+ *i = (struct iov_iter) {
+ .iter_type = ITER_BVECQ,
+ .data_source = direction,
+ .bvecq = bvecq,
+ .bvecq_slot = first_slot,
+ .count = count,
+ .iov_offset = offset,
+ };
+}
+EXPORT_SYMBOL(iov_iter_bvec_queue);
+
/**
* iov_iter_folio_queue - Initialise an I/O iterator to use the folios in a folio queue
* @i: The iterator to initialise.
@@ -839,6 +947,37 @@ static unsigned long iov_iter_alignment_bvec(const struct iov_iter *i)
return res;
}
+static unsigned long iov_iter_alignment_bvecq(const struct iov_iter *iter)
+{
+ const struct bvecq *bq;
+ unsigned long res = 0;
+ unsigned int slot = iter->bvecq_slot;
+ size_t skip = iter->iov_offset;
+ size_t size = iter->count;
+
+ if (!size)
+ return res;
+
+ for (bq = iter->bvecq; bq; bq = bq->next) {
+ for (; slot < bq->nr_slots; slot++) {
+ const struct bio_vec *bvec = &bq->bv[slot];
+ size_t part = umin(bvec->bv_len - skip, size);
+
+ res |= bvec->bv_offset + skip;
+ res |= part;
+
+ size -= part;
+ if (size == 0)
+ return res;
+ skip = 0;
+ }
+
+ slot = 0;
+ }
+
+ return res;
+}
+
unsigned long iov_iter_alignment(const struct iov_iter *i)
{
if (likely(iter_is_ubuf(i))) {
@@ -854,6 +993,8 @@ unsigned long iov_iter_alignment(const struct iov_iter *i)
if (iov_iter_is_bvec(i))
return iov_iter_alignment_bvec(i);
+ if (iov_iter_is_bvecq(i))
+ return iov_iter_alignment_bvecq(i);
/* With both xarray and folioq types, we're dealing with whole folios. */
if (iov_iter_is_folioq(i))
@@ -1066,6 +1207,36 @@ static int bvec_npages(const struct iov_iter *i, int maxpages)
return npages;
}
+static size_t iov_npages_bvecq(const struct iov_iter *iter, size_t maxpages)
+{
+ const struct bvecq *bq;
+ unsigned int slot = iter->bvecq_slot;
+ size_t npages = 0;
+ size_t skip = iter->iov_offset;
+ size_t size = iter->count;
+
+ for (bq = iter->bvecq; bq; bq = bq->next) {
+ for (; slot < bq->nr_slots; slot++) {
+ const struct bio_vec *bvec = &bq->bv[slot];
+ size_t offs = (bvec->bv_offset + skip) % PAGE_SIZE;
+ size_t part = umin(bvec->bv_len - skip, size);
+
+ npages += DIV_ROUND_UP(offs + part, PAGE_SIZE);
+ if (npages >= maxpages)
+ goto out;
+
+ size -= part;
+ if (!size)
+ goto out;
+ skip = 0;
+ }
+
+ slot = 0;
+ }
+out:
+ return umin(npages, maxpages);
+}
+
int iov_iter_npages(const struct iov_iter *i, int maxpages)
{
if (unlikely(!i->count))
@@ -1080,6 +1251,8 @@ int iov_iter_npages(const struct iov_iter *i, int maxpages)
return iov_npages(i, maxpages);
if (iov_iter_is_bvec(i))
return bvec_npages(i, maxpages);
+ if (iov_iter_is_bvecq(i))
+ return iov_npages_bvecq(i, maxpages);
if (iov_iter_is_folioq(i)) {
unsigned offset = i->iov_offset % PAGE_SIZE;
int npages = DIV_ROUND_UP(offset + i->count, PAGE_SIZE);
@@ -1366,6 +1539,147 @@ void iov_iter_restore(struct iov_iter *i, struct iov_iter_state *state)
i->nr_segs = state->nr_segs;
}
+/*
+ * Count the number of virtually contiguous pages coming up next in an
+ * ITER_BVECQ iterator, up to the specified maxima.
+ */
+static unsigned int iter_count_bvecq_pages(const struct iov_iter *iter,
+ size_t maxsize,
+ unsigned int maxpages)
+{
+ const struct bvecq *bvecq = iter->bvecq;
+ unsigned int slot = iter->bvecq_slot;
+ ssize_t remain = umin(maxsize, iter->count);
+ size_t count = 0, offset = iter->iov_offset;
+
+ for (;; slot++) {
+ const struct bio_vec *bv;
+ size_t boff, blen;
+
+ while (slot >= bvecq->nr_slots) {
+ if (!bvecq->next) {
+ WARN_ON_ONCE(remain > 0);
+ break;
+ }
+ bvecq = bvecq->next;
+ slot = 0;
+ }
+
+ bv = &bvecq->bv[slot];
+ boff = bv->bv_offset;
+ blen = bv->bv_len;
+
+ if (unlikely(!bv->bv_page)) {
+ if (blen && count > 0)
+ break;
+ continue;
+ }
+ if (!PAGE_ALIGNED(boff) && count > 0)
+ break;
+
+ boff += offset;
+ blen -= offset;
+ offset = 0;
+ if (!blen)
+ continue;
+
+ blen = umin(blen, remain);
+ remain -= blen;
+ blen += offset_in_page(boff);
+ count += DIV_ROUND_UP(blen, PAGE_SIZE);
+
+ if (!PAGE_ALIGNED(blen))
+ break;
+ if (remain <= 0)
+ break;
+ if (count >= maxpages)
+ break;
+ }
+
+ return umin(count, maxpages);
+}
+
+/*
+ * Extract a list of virtually contiguous pages from an ITER_BVECQ iterator.
+ * This does not get references on the pages, nor does it get a pin on them.
+ */
+static ssize_t iov_iter_extract_bvecq_pages(struct iov_iter *iter,
+ struct page ***pages, size_t maxsize,
+ unsigned int maxpages,
+ iov_iter_extraction_t extraction_flags,
+ size_t *offset0)
+{
+ const struct bvecq *bvecq;
+ struct page **p;
+ unsigned int slot, nr = 0;
+ size_t extracted = 0, offset;
+
+ /* Count the next run of virtually contiguous pages. */
+ maxpages = iter_count_bvecq_pages(iter, maxsize, maxpages);
+
+ if (!*pages) {
+ *pages = kvmalloc_array(maxpages, sizeof(struct page *), GFP_KERNEL);
+ if (!*pages)
+ return -ENOMEM;
+ }
+
+ p = *pages;
+
+ /* Now transcribe the page pointers. */
+ extracted = 0;
+ bvecq = iter->bvecq;
+ offset = iter->iov_offset;
+ slot = iter->bvecq_slot;
+
+ do {
+ const struct bio_vec *bv;
+ size_t boff, blen;
+
+ while (slot >= bvecq->nr_slots) {
+ if (!bvecq->next) {
+ WARN_ON_ONCE(extracted < iter->count);
+ break;
+ }
+ bvecq = bvecq->next;
+ slot = 0;
+ }
+
+ bv = &bvecq->bv[slot];
+ boff = bv->bv_offset;
+ blen = bv->bv_len;
+
+ if (!bv->bv_page)
+ blen = 0;
+
+ if (offset < blen) {
+ size_t part = umin(maxsize - extracted, blen - offset);
+ size_t poff = (boff + offset) % PAGE_SIZE;
+ size_t pix = (boff + offset) / PAGE_SIZE;
+
+ if (poff + part > PAGE_SIZE)
+ part = PAGE_SIZE - poff;
+
+ if (!extracted)
+ *offset0 = poff;
+
+ p[nr++] = bv->bv_page + pix;
+ offset += part;
+ extracted += part;
+ }
+
+ if (offset >= blen) {
+ offset = 0;
+ slot++;
+ }
+ } while (nr < maxpages && extracted < maxsize);
+
+ iter->bvecq = bvecq;
+ iter->bvecq_slot = slot;
+ iter->iov_offset = offset;
+ iter->count -= extracted;
+ return extracted;
+}
+
/*
* Extract a list of contiguous pages from an ITER_FOLIOQ iterator. This does
* not get references on the pages, nor does it get a pin on them.
@@ -1708,6 +2022,10 @@ ssize_t iov_iter_extract_pages(struct iov_iter *i,
return iov_iter_extract_bvec_pages(i, pages, maxsize,
maxpages, extraction_flags,
offset0);
+ if (iov_iter_is_bvecq(i))
+ return iov_iter_extract_bvecq_pages(i, pages, maxsize,
+ maxpages, extraction_flags,
+ offset0);
if (iov_iter_is_folioq(i))
return iov_iter_extract_folioq_pages(i, pages, maxsize,
maxpages, extraction_flags,
diff --git a/lib/scatterlist.c b/lib/scatterlist.c
index b7fe91ef35b8..b92144659543 100644
--- a/lib/scatterlist.c
+++ b/lib/scatterlist.c
@@ -10,6 +10,7 @@
#include <linux/highmem.h>
#include <linux/kmemleak.h>
#include <linux/bvec.h>
+#include <linux/bvecq.h>
#include <linux/uio.h>
#include <linux/folio_queue.h>
@@ -1267,6 +1268,68 @@ static ssize_t extract_kvec_to_sg(struct iov_iter *iter,
return ret;
}
+/*
+ * Extract up to sg_max folios from an BVECQ-type iterator and add them to
+ * the scatterlist. The pages are not pinned.
+ */
+static ssize_t extract_bvecq_to_sg(struct iov_iter *iter,
+ ssize_t maxsize,
+ struct sg_table *sgtable,
+ unsigned int sg_max,
+ iov_iter_extraction_t extraction_flags)
+{
+ const struct bvecq *bvecq = iter->bvecq;
+ struct scatterlist *sg = sgtable->sgl + sgtable->nents;
+ unsigned int seg = iter->bvecq_slot;
+ ssize_t ret = 0;
+ size_t offset = iter->iov_offset;
+
+ if (seg >= bvecq->nr_slots) {
+ bvecq = bvecq->next;
+ if (WARN_ON_ONCE(!bvecq))
+ return 0;
+ seg = 0;
+ }
+
+ do {
+ const struct bio_vec *bv = &bvecq->bv[seg];
+ size_t blen = bv->bv_len;
+
+ if (!bv->bv_page)
+ blen = 0;
+
+ if (offset < blen) {
+ size_t part = umin(maxsize - ret, blen - offset);
+
+ sg_set_page(sg, bv->bv_page, part, bv->bv_offset + offset);
+ sgtable->nents++;
+ sg++;
+ sg_max--;
+ offset += part;
+ ret += part;
+ }
+
+ if (offset >= blen) {
+ offset = 0;
+ seg++;
+ if (seg >= bvecq->nr_slots) {
+ if (!bvecq->next) {
+ WARN_ON_ONCE(ret < iter->count);
+ break;
+ }
+ bvecq = bvecq->next;
+ seg = 0;
+ }
+ }
+ } while (sg_max > 0 && ret < maxsize);
+
+ iter->bvecq = bvecq;
+ iter->bvecq_slot = seg;
+ iter->iov_offset = offset;
+ iter->count -= ret;
+ return ret;
+}
+
/*
* Extract up to sg_max folios from an FOLIOQ-type iterator and add them to
* the scatterlist. The pages are not pinned.
@@ -1390,8 +1453,8 @@ static ssize_t extract_xarray_to_sg(struct iov_iter *iter,
* addition of @sg_max elements.
*
* The pages referred to by UBUF- and IOVEC-type iterators are extracted and
- * pinned; BVEC-, KVEC-, FOLIOQ- and XARRAY-type are extracted but aren't
- * pinned; DISCARD-type is not supported.
+ * pinned; BVEC-, BVECQ-, KVEC-, FOLIOQ- and XARRAY-type are extracted but
+ * aren't pinned; DISCARD-type is not supported.
*
* No end mark is placed on the scatterlist; that's left to the caller.
*
@@ -1423,6 +1486,9 @@ ssize_t extract_iter_to_sg(struct iov_iter *iter, size_t maxsize,
case ITER_KVEC:
return extract_kvec_to_sg(iter, maxsize, sgtable, sg_max,
extraction_flags);
+ case ITER_BVECQ:
+ return extract_bvecq_to_sg(iter, maxsize, sgtable, sg_max,
+ extraction_flags);
case ITER_FOLIOQ:
return extract_folioq_to_sg(iter, maxsize, sgtable, sg_max,
extraction_flags);
diff --git a/lib/tests/kunit_iov_iter.c b/lib/tests/kunit_iov_iter.c
index 37bd6eb25896..1342487dd211 100644
--- a/lib/tests/kunit_iov_iter.c
+++ b/lib/tests/kunit_iov_iter.c
@@ -12,6 +12,7 @@
#include <linux/mm.h>
#include <linux/uio.h>
#include <linux/bvec.h>
+#include <linux/bvecq.h>
#include <linux/folio_queue.h>
#include <linux/scatterlist.h>
#include <linux/minmax.h>
@@ -545,6 +546,185 @@ static void __init iov_kunit_copy_from_folioq(struct kunit *test)
KUNIT_SUCCEED(test);
}
+static void iov_kunit_destroy_bvecq(void *data)
+{
+ struct bvecq *bq, *next;
+
+ for (bq = data; bq; bq = next) {
+ next = bq->next;
+ for (int i = 0; i < bq->nr_slots; i++)
+ if (bq->bv[i].bv_page)
+ put_page(bq->bv[i].bv_page);
+ kfree(bq);
+ }
+}
+
+static struct bvecq *iov_kunit_alloc_bvecq(struct kunit *test, unsigned int max_slots)
+{
+ struct bvecq *bq;
+
+ bq = kzalloc(struct_size(bq, __bv, max_slots), GFP_KERNEL);
+ KUNIT_ASSERT_NOT_ERR_OR_NULL(test, bq);
+ bq->max_slots = max_slots;
+ bq->bv = bq->__bv;
+ bq->inline_bv = true;
+ return bq;
+}
+
+static struct bvecq *iov_kunit_create_bvecq(struct kunit *test, unsigned int max_slots)
+{
+ struct bvecq *bq;
+
+ bq = iov_kunit_alloc_bvecq(test, max_slots);
+ kunit_add_action_or_reset(test, iov_kunit_destroy_bvecq, bq);
+ return bq;
+}
+
+static void __init iov_kunit_load_bvecq(struct kunit *test,
+ struct iov_iter *iter, int dir,
+ struct bvecq *bq_head,
+ struct page **pages, size_t npages)
+{
+ struct bvecq *bq = bq_head;
+ size_t size = 0;
+
+ for (int i = 0; i < npages; i++) {
+ if (bq->nr_slots >= bq->max_slots) {
+ bq->next = iov_kunit_alloc_bvecq(test, 13);
+ bq->next->prev = bq;
+ bq = bq->next;
+ }
+ bvec_set_page(&bq->bv[bq->nr_slots], pages[i], PAGE_SIZE, 0);
+ bq->nr_slots++;
+ size += PAGE_SIZE;
+ }
+ iov_iter_bvec_queue(iter, dir, bq_head, 0, 0, size);
+}
+
+/*
+ * Test copying to a ITER_BVECQ-type iterator.
+ */
+static void __init iov_kunit_copy_to_bvecq(struct kunit *test)
+{
+ const struct kvec_test_range *pr;
+ struct iov_iter iter;
+ struct bvecq *bq;
+ struct page **spages, **bpages;
+ u8 *scratch, *buffer;
+ size_t bufsize, npages, size, copied;
+ int i, patt;
+
+ bufsize = 0x100000;
+ npages = bufsize / PAGE_SIZE;
+
+ bq = iov_kunit_create_bvecq(test, 13);
+
+ scratch = iov_kunit_create_buffer(test, &spages, npages);
+ for (i = 0; i < bufsize; i++)
+ scratch[i] = pattern(i);
+
+ buffer = iov_kunit_create_buffer(test, &bpages, npages);
+ memset(buffer, 0, bufsize);
+
+ iov_kunit_load_bvecq(test, &iter, READ, bq, bpages, npages);
+
+ i = 0;
+ for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
+ size = pr->to - pr->from;
+ KUNIT_ASSERT_LE(test, pr->to, bufsize);
+
+ iov_iter_bvec_queue(&iter, READ, bq, 0, 0, pr->to);
+ iov_iter_advance(&iter, pr->from);
+ copied = copy_to_iter(scratch + i, size, &iter);
+
+ KUNIT_EXPECT_EQ(test, copied, size);
+ KUNIT_EXPECT_EQ(test, iter.count, 0);
+ i += size;
+ if (test->status == KUNIT_FAILURE)
+ goto stop;
+ }
+
+ /* Build the expected image in the scratch buffer. */
+ patt = 0;
+ memset(scratch, 0, bufsize);
+ for (pr = kvec_test_ranges; pr->from >= 0; pr++)
+ for (i = pr->from; i < pr->to; i++)
+ scratch[i] = pattern(patt++);
+
+ /* Compare the images */
+ for (i = 0; i < bufsize; i++) {
+ KUNIT_EXPECT_EQ_MSG(test, buffer[i], scratch[i], "at i=%x", i);
+ if (buffer[i] != scratch[i])
+ return;
+ }
+
+stop:
+ KUNIT_SUCCEED(test);
+}
+
+/*
+ * Test copying from a ITER_BVECQ-type iterator.
+ */
+static void __init iov_kunit_copy_from_bvecq(struct kunit *test)
+{
+ const struct kvec_test_range *pr;
+ struct iov_iter iter;
+ struct bvecq *bq;
+ struct page **spages, **bpages;
+ u8 *scratch, *buffer;
+ size_t bufsize, npages, size, copied;
+ int i, j;
+
+ bufsize = 0x100000;
+ npages = bufsize / PAGE_SIZE;
+
+ bq = iov_kunit_create_bvecq(test, 13);
+
+ buffer = iov_kunit_create_buffer(test, &bpages, npages);
+ for (i = 0; i < bufsize; i++)
+ buffer[i] = pattern(i);
+
+ scratch = iov_kunit_create_buffer(test, &spages, npages);
+ memset(scratch, 0, bufsize);
+
+ iov_kunit_load_bvecq(test, &iter, READ, bq, bpages, npages);
+
+ i = 0;
+ for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
+ size = pr->to - pr->from;
+ KUNIT_ASSERT_LE(test, pr->to, bufsize);
+
+ iov_iter_bvec_queue(&iter, WRITE, bq, 0, 0, pr->to);
+ iov_iter_advance(&iter, pr->from);
+ copied = copy_from_iter(scratch + i, size, &iter);
+
+ KUNIT_EXPECT_EQ(test, copied, size);
+ KUNIT_EXPECT_EQ(test, iter.count, 0);
+ i += size;
+ }
+
+ /* Build the expected image in the main buffer. */
+ i = 0;
+ memset(buffer, 0, bufsize);
+ for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
+ for (j = pr->from; j < pr->to; j++) {
+ buffer[i++] = pattern(j);
+ if (i >= bufsize)
+ goto stop;
+ }
+ }
+stop:
+
+ /* Compare the images */
+ for (i = 0; i < bufsize; i++) {
+ KUNIT_EXPECT_EQ_MSG(test, scratch[i], buffer[i], "at i=%x", i);
+ if (scratch[i] != buffer[i])
+ return;
+ }
+
+ KUNIT_SUCCEED(test);
+}
+
static void iov_kunit_destroy_xarray(void *data)
{
struct xarray *xarray = data;
@@ -860,6 +1040,85 @@ static void __init iov_kunit_extract_pages_bvec(struct kunit *test)
KUNIT_SUCCEED(test);
}
+/*
+ * Test the extraction of ITER_BVECQ-type iterators.
+ */
+static void __init iov_kunit_extract_pages_bvecq(struct kunit *test)
+{
+ const struct kvec_test_range *pr;
+ struct iov_iter iter;
+ struct bvecq *bq;
+ struct page **bpages, *pagelist[8], **pages = pagelist;
+ ssize_t len;
+ size_t bufsize, size = 0, npages;
+ int i, from;
+
+ bufsize = 0x100000;
+ npages = bufsize / PAGE_SIZE;
+
+ bq = iov_kunit_create_bvecq(test, 13);
+
+ iov_kunit_create_buffer(test, &bpages, npages);
+ iov_kunit_load_bvecq(test, &iter, READ, bq, bpages, npages);
+
+ for (pr = kvec_test_ranges; pr->from >= 0; pr++) {
+ from = pr->from;
+ size = pr->to - from;
+ KUNIT_ASSERT_LE(test, pr->to, bufsize);
+
+ iov_iter_bvec_queue(&iter, WRITE, bq, 0, 0, pr->to);
+ iov_iter_advance(&iter, from);
+
+ do {
+ size_t offset0 = LONG_MAX;
+
+ for (i = 0; i < ARRAY_SIZE(pagelist); i++)
+ pagelist[i] = (void *)(unsigned long)0xaa55aa55aa55aa55ULL;
+
+ len = iov_iter_extract_pages(&iter, &pages, 100 * 1024,
+ ARRAY_SIZE(pagelist), 0, &offset0);
+ KUNIT_EXPECT_GE(test, len, 0);
+ if (len < 0)
+ break;
+ KUNIT_EXPECT_LE(test, len, size);
+ KUNIT_EXPECT_EQ(test, iter.count, size - len);
+ if (len == 0)
+ break;
+ size -= len;
+ KUNIT_EXPECT_GE(test, (ssize_t)offset0, 0);
+ KUNIT_EXPECT_LT(test, offset0, PAGE_SIZE);
+
+ for (i = 0; i < ARRAY_SIZE(pagelist); i++) {
+ struct page *p;
+ ssize_t part = min_t(ssize_t, len, PAGE_SIZE - offset0);
+ int ix;
+
+ KUNIT_ASSERT_GE(test, part, 0);
+ ix = from / PAGE_SIZE;
+ KUNIT_ASSERT_LT(test, ix, npages);
+ p = bpages[ix];
+ KUNIT_EXPECT_PTR_EQ(test, pagelist[i], p);
+ KUNIT_EXPECT_EQ(test, offset0, from % PAGE_SIZE);
+ from += part;
+ len -= part;
+ KUNIT_ASSERT_GE(test, len, 0);
+ if (len == 0)
+ break;
+ offset0 = 0;
+ }
+
+ if (test->status == KUNIT_FAILURE)
+ goto stop;
+ } while (iov_iter_count(&iter) > 0);
+
+ KUNIT_EXPECT_EQ(test, size, 0);
+ KUNIT_EXPECT_EQ(test, iter.count, 0);
+ }
+
+stop:
+ KUNIT_SUCCEED(test);
+}
+
/*
* Test the extraction of ITER_FOLIOQ-type iterators.
*/
@@ -1219,12 +1478,15 @@ static struct kunit_case __refdata iov_kunit_cases[] = {
KUNIT_CASE(iov_kunit_copy_from_kvec),
KUNIT_CASE(iov_kunit_copy_to_bvec),
KUNIT_CASE(iov_kunit_copy_from_bvec),
+ KUNIT_CASE(iov_kunit_copy_to_bvecq),
+ KUNIT_CASE(iov_kunit_copy_from_bvecq),
KUNIT_CASE(iov_kunit_copy_to_folioq),
KUNIT_CASE(iov_kunit_copy_from_folioq),
KUNIT_CASE(iov_kunit_copy_to_xarray),
KUNIT_CASE(iov_kunit_copy_from_xarray),
KUNIT_CASE(iov_kunit_extract_pages_kvec),
KUNIT_CASE(iov_kunit_extract_pages_bvec),
+ KUNIT_CASE(iov_kunit_extract_pages_bvecq),
KUNIT_CASE(iov_kunit_extract_pages_folioq),
KUNIT_CASE(iov_kunit_extract_pages_xarray),
KUNIT_CASE(iov_kunit_iter_to_sg_kvec),
^ permalink raw reply related
* [PATCH v2 06/21] iov_iter: Make iov_iter_get_pages*() wrap iov_iter_extract_pages()
From: David Howells @ 2026-05-18 22:29 UTC (permalink / raw)
To: Christian Brauner, Matthew Wilcox, Christoph Hellwig
Cc: David Howells, Paulo Alcantara, Jens Axboe, Leon Romanovsky,
Steve French, ChenXiaoSong, Marc Dionne, Eric Van Hensbergen,
Dominique Martinet, Ilya Dryomov, Trond Myklebust, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, linux-block
In-Reply-To: <20260518222959.488126-1-dhowells@redhat.com>
Make iov_iter_get_pages*() wrap iov_iter_extract_pages() for kernel
iterator types (e.g. ITER_BVEC, ITER_FOLIOQ, ITER_XARRAY). The pages
obtained have their refcounts incremented afterwards if they're not slab
pages. ITER_KVEC is left returning -EFAULT.
Signed-off-by: David Howells <dhowells@redhat.com>
Reviewed-by: Paulo Alcantara (Red Hat) <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: linux-block@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
lib/iov_iter.c | 164 ++++++-------------------------------------------
1 file changed, 19 insertions(+), 145 deletions(-)
diff --git a/lib/iov_iter.c b/lib/iov_iter.c
index 243662af1af7..cac7d7364bc2 100644
--- a/lib/iov_iter.c
+++ b/lib/iov_iter.c
@@ -910,118 +910,34 @@ static int want_pages_array(struct page ***res, size_t size,
return count;
}
-static ssize_t iter_folioq_get_pages(struct iov_iter *iter,
+/*
+ * Wrap iov_iter_extract_pages() and then pin the non-slab pages we got back.
+ * This only works for non-user iterator types as get_pages uses get_user_pages
+ * not pin_user_pages.
+ */
+static ssize_t iter_get_kernel_pages(struct iov_iter *iter,
struct page ***ppages, size_t maxsize,
unsigned maxpages, size_t *_start_offset)
{
- const struct folio_queue *folioq = iter->folioq;
struct page **pages;
- unsigned int slot = iter->folioq_slot;
- size_t extracted = 0, count = iter->count, iov_offset = iter->iov_offset;
+ ssize_t ret, done;
- if (slot >= folioq_nr_slots(folioq)) {
- folioq = folioq->next;
- slot = 0;
- if (WARN_ON(iov_offset != 0))
- return -EIO;
- }
+ ret = iov_iter_extract_pages(iter, ppages, maxsize, maxpages,
+ 0, _start_offset);
+ if (ret <= 0)
+ return ret;
- maxpages = want_pages_array(ppages, maxsize, iov_offset & ~PAGE_MASK, maxpages);
- if (!maxpages)
- return -ENOMEM;
- *_start_offset = iov_offset & ~PAGE_MASK;
pages = *ppages;
+ for (done = ret + *_start_offset; done > 0; done -= PAGE_SIZE) {
+ struct folio *folio = page_folio(*pages);
- for (;;) {
- struct folio *folio = folioq_folio(folioq, slot);
- size_t offset = iov_offset, fsize = folioq_folio_size(folioq, slot);
- size_t part = PAGE_SIZE - offset % PAGE_SIZE;
-
- if (offset < fsize) {
- part = umin(part, umin(maxsize - extracted, fsize - offset));
- count -= part;
- iov_offset += part;
- extracted += part;
-
- *pages = folio_page(folio, offset / PAGE_SIZE);
- get_page(*pages);
- pages++;
- maxpages--;
- }
-
- if (maxpages == 0 || extracted >= maxsize)
- break;
-
- if (iov_offset >= fsize) {
- iov_offset = 0;
- slot++;
- if (slot == folioq_nr_slots(folioq) && folioq->next) {
- folioq = folioq->next;
- slot = 0;
- }
- }
- }
-
- iter->count = count;
- iter->iov_offset = iov_offset;
- iter->folioq = folioq;
- iter->folioq_slot = slot;
- return extracted;
-}
-
-static ssize_t iter_xarray_populate_pages(struct page **pages, struct xarray *xa,
- pgoff_t index, unsigned int nr_pages)
-{
- XA_STATE(xas, xa, index);
- struct folio *folio;
- unsigned int ret = 0;
-
- rcu_read_lock();
- for (folio = xas_load(&xas); folio; folio = xas_next(&xas)) {
- if (xas_retry(&xas, folio))
- continue;
-
- /* Has the folio moved or been split? */
- if (unlikely(folio != xas_reload(&xas))) {
- xas_reset(&xas);
- continue;
- }
-
- pages[ret] = folio_file_page(folio, xas.xa_index);
- folio_get(folio);
- if (++ret == nr_pages)
- break;
+ if (!folio_test_slab(folio))
+ folio_get(folio);
+ pages++;
}
- rcu_read_unlock();
return ret;
}
-static ssize_t iter_xarray_get_pages(struct iov_iter *i,
- struct page ***pages, size_t maxsize,
- unsigned maxpages, size_t *_start_offset)
-{
- unsigned nr, offset, count;
- pgoff_t index;
- loff_t pos;
-
- pos = i->xarray_start + i->iov_offset;
- index = pos >> PAGE_SHIFT;
- offset = pos & ~PAGE_MASK;
- *_start_offset = offset;
-
- count = want_pages_array(pages, maxsize, offset, maxpages);
- if (!count)
- return -ENOMEM;
- nr = iter_xarray_populate_pages(*pages, i->xarray, index, count);
- if (nr == 0)
- return 0;
-
- maxsize = min_t(size_t, nr * PAGE_SIZE - offset, maxsize);
- i->iov_offset += maxsize;
- i->count -= maxsize;
- return maxsize;
-}
-
/* must be done on non-empty ITER_UBUF or ITER_IOVEC one */
static unsigned long first_iovec_segment(const struct iov_iter *i, size_t *size)
{
@@ -1044,22 +960,6 @@ static unsigned long first_iovec_segment(const struct iov_iter *i, size_t *size)
BUG(); // if it had been empty, we wouldn't get called
}
-/* must be done on non-empty ITER_BVEC one */
-static struct page *first_bvec_segment(const struct iov_iter *i,
- size_t *size, size_t *start)
-{
- struct page *page;
- size_t skip = i->iov_offset, len;
-
- len = i->bvec->bv_len - skip;
- if (*size > len)
- *size = len;
- skip += i->bvec->bv_offset;
- page = i->bvec->bv_page + skip / PAGE_SIZE;
- *start = skip % PAGE_SIZE;
- return page;
-}
-
static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,
struct page ***pages, size_t maxsize,
unsigned int maxpages, size_t *start)
@@ -1095,36 +995,10 @@ static ssize_t __iov_iter_get_pages_alloc(struct iov_iter *i,
iov_iter_advance(i, maxsize);
return maxsize;
}
- if (iov_iter_is_bvec(i)) {
- struct page **p;
- struct page *page;
- page = first_bvec_segment(i, &maxsize, start);
- n = want_pages_array(pages, maxsize, *start, maxpages);
- if (!n)
- return -ENOMEM;
- p = *pages;
- for (int k = 0; k < n; k++) {
- struct folio *folio = page_folio(page + k);
- p[k] = page + k;
- if (!folio_test_slab(folio))
- folio_get(folio);
- }
- maxsize = min_t(size_t, maxsize, n * PAGE_SIZE - *start);
- i->count -= maxsize;
- i->iov_offset += maxsize;
- if (i->iov_offset == i->bvec->bv_len) {
- i->iov_offset = 0;
- i->bvec++;
- i->nr_segs--;
- }
- return maxsize;
- }
- if (iov_iter_is_folioq(i))
- return iter_folioq_get_pages(i, pages, maxsize, maxpages, start);
- if (iov_iter_is_xarray(i))
- return iter_xarray_get_pages(i, pages, maxsize, maxpages, start);
- return -EFAULT;
+ if (iov_iter_is_kvec(i))
+ return -EFAULT;
+ return iter_get_kernel_pages(i, pages, maxsize, maxpages, start);
}
ssize_t iov_iter_get_pages2(struct iov_iter *i, struct page **pages,
^ permalink raw reply related
* [PATCH v2 05/21] Add a function to kmap one page of a multipage bio_vec
From: David Howells @ 2026-05-18 22:29 UTC (permalink / raw)
To: Christian Brauner, Matthew Wilcox, Christoph Hellwig
Cc: David Howells, Paulo Alcantara, Jens Axboe, Leon Romanovsky,
Steve French, ChenXiaoSong, Marc Dionne, Eric Van Hensbergen,
Dominique Martinet, Ilya Dryomov, Trond Myklebust, netfs,
linux-afs, linux-cifs, linux-nfs, ceph-devel, v9fs, linux-erofs,
linux-fsdevel, linux-kernel, linux-block
In-Reply-To: <20260518222959.488126-1-dhowells@redhat.com>
Add a function to kmap one page of a multipage bio_vec by offset (which is
added to the offset in the bio_vec internally). The caller is responsible
for calculating how much of the page is then available.
Signed-off-by: David Howells <dhowells@redhat.com>
Acked-by: Paulo Alcantara (Red Hat) <pc@manguebit.org>
cc: Matthew Wilcox <willy@infradead.org>
cc: Christoph Hellwig <hch@infradead.org>
cc: Jens Axboe <axboe@kernel.dk>
cc: linux-block@vger.kernel.org
cc: netfs@lists.linux.dev
cc: linux-fsdevel@vger.kernel.org
---
include/linux/bvec.h | 17 +++++++++++++++++
1 file changed, 17 insertions(+)
diff --git a/include/linux/bvec.h b/include/linux/bvec.h
index d36dd476feda..9df4a56fef61 100644
--- a/include/linux/bvec.h
+++ b/include/linux/bvec.h
@@ -299,4 +299,21 @@ static inline phys_addr_t bvec_phys(const struct bio_vec *bvec)
return page_to_phys(bvec->bv_page) + bvec->bv_offset;
}
+/**
+ * kmap_local_bvec - Map part of a bvec into the kernel virtual address space
+ * @bvec: bvec to map
+ * @offset: Offset into bvec
+ *
+ * Map the page containing the byte at @offset into the kernel virtual address
+ * space. The caller is responsible for making sure this doesn't overrun.
+ *
+ * Call kunmap_local on the returned address to unmap.
+ */
+static inline void *kmap_local_bvec(struct bio_vec *bvec, size_t offset)
+{
+ offset += bvec->bv_offset;
+
+ return kmap_local_page(bvec->bv_page + offset / PAGE_SIZE) + offset % PAGE_SIZE;
+}
+
#endif /* __LINUX_BVEC_H */
^ permalink raw reply related
* Re: [PATCH] nvme-multipath: set BIO_REMAPPED on bios remapped to per-path namespace disks
From: Keith Busch @ 2026-05-18 21:52 UTC (permalink / raw)
To: Achkinazi, Igor
Cc: hch@lst.de, sagi@grimberg.me, axboe@kernel.dk,
linux-nvme@lists.infradead.org, linux-block@vger.kernel.org,
linux-kernel@vger.kernel.org, stable@vger.kernel.org
In-Reply-To: <MW5PR19MB5484B015B6B1D739D8C5CA2FFD032@MW5PR19MB5484.namprd19.prod.outlook.com>
On Mon, May 18, 2026 at 08:59:09PM +0000, Achkinazi, Igor wrote:
> - submit_bio_noacct_nocheck is block-internal and not exported, so using it
> from NVMe would require a block-layer API change just for this.
That is not a valid reason to not do this. We often export and unexport
APIs as needs evolve. I'm not saying you have to use this API, but I'm
just not yet seeing why we shouldn't.
> - it bypasses more checks than I see we need here (throttling, RO, crypto,
> op-type), I prefer bypassing only the EOD check.
I do not think we need any of those on the hidden device either. The
stacked limits should have caught any problems or handled any policy on
the first pass.
Though I am aware of certain limit checks that go through here are not
done under the queue's entered reference count so there's some racy
things possible, but that's a pre-existing issue and doing
submit_bio_noacct a 2nd time doesn't help.
> - BIO_REMAPPED propagates to split clones, so it covers all resubmissions,
> not just the initial one.
Submitting split clones already uses submit_bio_noacct_nocheck() so the
BIO_REMAPPED flag doesn't come into play there either.
And BIO_REMAPPED applies to when the bio's bd_part is a partition. The
multipath layer overrides the block device with the part0 of the path,
so there is no partition (and we may not have been dealing with a
partition in first place), so this patch is introducing a new implicit
expectation on what this flag means. Consider a real failover going
through the requeue_list using the submit_bio_noacct() again. If you've
already set BIO_REMAPPED, then it skips the checks, but that could have
happened across a controller format change that modified all the limits,
so you probably want the eod checks to happen again on this scenario.
Your suggestion would skip it because you're using BIO_REMAPPED as a
proxy to skip "eod" checks.
^ permalink raw reply
* RE: [PATCH] nvme-multipath: set BIO_REMAPPED on bios remapped to per-path namespace disks
From: Achkinazi, Igor @ 2026-05-18 20:59 UTC (permalink / raw)
To: Keith Busch
Cc: hch@lst.de, sagi@grimberg.me, axboe@kernel.dk,
linux-nvme@lists.infradead.org, linux-block@vger.kernel.org,
linux-kernel@vger.kernel.org, stable@vger.kernel.org
In-Reply-To: <agtnJb5a5uIqH-65@kbusch-mbp>
Keith Busch wrote :
> Any reason nvme multipath can't call submit_bio_noacct_nocheck()
> directly instead? If it's safe to skip the eod check here, then it
> looks safe to skip everything else too.
I'd prefer to keep this internal to nvme and use BIO_REMAPPED rather than
switching to submit_bio_noacct_nocheck:
- submit_bio_noacct_nocheck is block-internal and not exported, so using it
from NVMe would require a block-layer API change just for this.
- it bypasses more checks than I see we need here (throttling, RO, crypto,
op-type), I prefer bypassing only the EOD check.
- BIO_REMAPPED propagates to split clones, so it covers all resubmissions,
not just the initial one.
Internal Use - Confidential
^ permalink raw reply
* Re: [PATCH v2] cgroup/rstat: validate cpu before css_rstat_cpu() access
From: Tejun Heo @ 2026-05-18 19:36 UTC (permalink / raw)
To: Qing Ming
Cc: Josef Bacik, Jens Axboe, Johannes Weiner, Michal Koutný,
Michal Hocko, Roman Gushchin, Shakeel Butt, Muchun Song,
Andrew Morton, Alexei Starovoitov, Hao Luo, Yosry Ahmed, cgroups,
linux-block, linux-kernel, linux-mm, bpf
In-Reply-To: <20260516070849.106141-1-a0yami@mailbox.org>
Hello,
> Qing Ming (1):
> cgroup/rstat: validate cpu before css_rstat_cpu() access
Applied to cgroup/for-7.1-fixes.
In hindsight, we should have added a separate kfunc wrapper from the
start instead of tagging css_rstat_updated() with __bpf_kfunc directly,
which would have avoided the rename. Oh well, it is what it is.
Thanks.
--
tejun
^ permalink raw reply
* Re: [PATCH] nvme-multipath: set BIO_REMAPPED on bios remapped to per-path namespace disks
From: Keith Busch @ 2026-05-18 19:23 UTC (permalink / raw)
To: Achkinazi, Igor
Cc: hch@lst.de, sagi@grimberg.me, axboe@kernel.dk,
linux-nvme@lists.infradead.org, linux-block@vger.kernel.org,
linux-kernel@vger.kernel.org, stable@vger.kernel.org
In-Reply-To: <MW5PR19MB548483D1FAE4F322E4C97352FD032@MW5PR19MB5484.namprd19.prod.outlook.com>
On Mon, May 18, 2026 at 02:52:56PM +0000, Achkinazi, Igor wrote:
> @@ -511,6 +511,13 @@ static void nvme_ns_head_submit_bio(struct bio *bio)
> ns = nvme_find_path(head);
> if (likely(ns)) {
> bio_set_dev(bio, ns->disk->part0);
> + /*
> + * Mark the bio as remapped to the per-path namespace disk so
> + * that bio_check_eod() is skipped on resubmission (e.g. from
> + * bio splitting in blk_mq_submit_bio). The EOD check already
> + * passed on the multipath head disk.
> + */
> + bio_set_flag(bio, BIO_REMAPPED);
Any reason nvme multipath can't call submit_bio_noacct_nocheck()
directly instead? If it's safe to skip the eod check here, then it
looks safe to skip everything else too.
^ permalink raw reply
* Re: [PATCH] blk-cgroup-rwstat: add missing DISCARD counter in blkg_rwstat_total()
From: Tejun Heo @ 2026-05-18 19:19 UTC (permalink / raw)
To: Tao Cui; +Cc: axboe, linux-block
In-Reply-To: <20260516044752.446995-1-cuitao@kylinos.cn>
On Sat, May 16, 2026 at 12:47:52PM +0800, Tao Cui wrote:
> Since commit 636620b66d5d ("blkcg: Track DISCARD statistics and output
> them in cgroup io.stat"), blkg_rwstat_add() routes discard IOs into
> BLKG_RWSTAT_DISCARD, but blkg_rwstat_total() still only sums READ and
> WRITE. This makes the total inconsistent with the per-type counters
> whenever discard IOs are present.
>
> On mixed read/write/discard workloads, the total undercounts the
> actual IO activity. If only discard requests are queued, the total
> returns 0 outright. In BFQ this causes avg_queue_size to be
> underestimated and the group to be incorrectly marked empty.
>
> The affected BFQ code paths are all gated by CONFIG_BFQ_CGROUP_DEBUG
> and used only for statistics display (bfq.sectors, bfq.avg_queue_size,
> bfq.empty_time). No scheduling decisions depend on these values.
I don't think including discard in total io amount is an improvement. This
will throw off a lot of people.
Thanks.
--
tejun
^ permalink raw reply
* Re: [PATCH blktests v2 0/3] introduce command trace feature
From: Bart Van Assche @ 2026-05-18 17:11 UTC (permalink / raw)
To: Shin'ichiro Kawasaki, linux-block; +Cc: Daniel Wagner, John Meneghini
In-Reply-To: <20260516120729.113659-1-shinichiro.kawasaki@wdc.com>
On 5/16/26 5:07 AM, Shin'ichiro Kawasaki wrote:
> Some blktests test cases have deep nesting, making their behavior
> difficult to understand. For example, the nvme test group has many
> helper functions that set sysfs attribute values and call nvme-cli
> commands. Understanding these behaviors is essential for debugging test
> case failures.
For the series:
Reviewed-by: Bart Van Assche <bvanassche@acm.org>
^ permalink raw reply
* [PATCH] nvme-multipath: set BIO_REMAPPED on bios remapped to per-path namespace disks
From: Achkinazi, Igor @ 2026-05-18 14:52 UTC (permalink / raw)
To: kbusch@kernel.org, hch@lst.de, sagi@grimberg.me, axboe@kernel.dk
Cc: linux-nvme@lists.infradead.org, linux-block@vger.kernel.org,
linux-kernel@vger.kernel.org, stable@vger.kernel.org,
Achkinazi, Igor
When nvme_ns_head_submit_bio() remaps a block IO from the multipath
head to a per-path namespace (path of multipath), bio_set_dev() clears
BIO_REMAPPED. Before commit a7c7f7b2b641 ("nvme: use bio_set_dev to
assign ->bi_bdev"), the code used a direct bio->bi_bdev assignment
which did not clear BIO_REMAPPED. The block IO is then
queued on current->bio_list (deferred, not processed inline) and SRCU
read lock is released.
The deferred block IO itself is dispatched directly to
blk_mq_submit_bio() without re-entering submit_bio_noacct(), so it
would be fine on its own. The problem is when the block IO size
exceeds queue limits and blk_mq_submit_bio() needs to split it using
__bio_split_to_limits().
The split remainder is resubmitted through submit_bio_noacct() which
calls bio_check_eod() again because BIO_REMAPPED is not set. This
sometimes races with nvme_ns_remove() zeroing the capacity after
synchronize_srcu(). Result: bio_check_eod() sees zeroed capacity and
fails the IO with "attempt to access beyond end of device" instead of
letting it fail over to another path.
Observed failure scenario during tests:
Setup: NVMe multipath with multiple paths (e.g., controllers nvme0,
nvme1) to the same namespace, exposed as a single multipath block
device (e.g., nvme0n1).
Steps to reproduce:
1. Run sustained IO against the multipath head device (e.g. vdbench)
2. Delete the namespace on one of the paths (e.g., detach the
namespace from NVMe controller on a subsystem on the
target side).
3. The IO that was remapped to the removed path and requires
splitting (exceeds queue limits) hits the race.
Expected behavior:
The IO should fail on the removed path and nvme_failover_req()
should retry it on the remaining healthy path. IO continues
without errors visible to the application.
Actual behavior:
The kernel reports IO errors to the application. dmesg shows:
IO_task /dev/di: attempt to access beyond end of device
nvme1c9n1: rw=33556480, sector=476160, nr_sectors=256 limit=0
IO errors were reported to the testing application, causing
it to stop, despite a healthy path being available. The IO is
rejected by bio_check_eod() on the split remainder before it
ever reaches the NVMe driver, so nvme_failover_req() never gets
a chance to fail over to the other path.
We observed this failure during NVMe multipath failover testing at
Dell, for example, on kernel 5.14.0-570.23.1.el9_6.x86_64 (Red Hat
9.7), kernel 6.4.0-150600.23.53-default (SLES 15.6), and others.
The fix is setting BIO_REMAPPED after bio_set_dev() in
nvme_ns_head_submit_bio(). This skips bio_check_eod() on resubmission
for both the remapped IO and any split clones derived from it. The
EOD check already passed on the multipath head.
This is safe because the individual path for nvme has bd_partno=0
(NVMe per-path namespace device is always a whole disk, not a
partition), so skipping blk_partition_remap() (also gated by
BIO_REMAPPED) has no effect: adjusting bio sector offsets from
partition-relative to whole-disk-relative is not necessary. If the
per-path queue is dead, it fails via GD_DEAD check in
bio_queue_enter(). If the per-path queue is still alive, the
request completes with error Invalid-Namespace coming from nvme target
and nvme_failover_req() handles path failover.
Same approach as commit 3a905c37c351 ("block: skip bio_check_eod for
partition-remapped bios") which solved the same problem for partition
remaps resubmitted after bio splitting.
Fixes: a7c7f7b2b641 ("nvme: use bio_set_dev to assign ->bi_bdev")
Cc: stable@vger.kernel.org
Signed-off-by: Igor Achkinazi <igor.achkinazi@dell.com>
---
drivers/nvme/host/multipath.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c
index 263161cb8ac0..04f7c7e59945 100644
--- a/drivers/nvme/host/multipath.c
+++ b/drivers/nvme/host/multipath.c
@@ -511,6 +511,13 @@ static void nvme_ns_head_submit_bio(struct bio *bio)
ns = nvme_find_path(head);
if (likely(ns)) {
bio_set_dev(bio, ns->disk->part0);
+ /*
+ * Mark the bio as remapped to the per-path namespace disk so
+ * that bio_check_eod() is skipped on resubmission (e.g. from
+ * bio splitting in blk_mq_submit_bio). The EOD check already
+ * passed on the multipath head disk.
+ */
+ bio_set_flag(bio, BIO_REMAPPED);
bio->bi_opf |= REQ_NVME_MPATH;
trace_block_bio_remap(bio, disk_devt(ns->head->disk),
bio->bi_iter.bi_sector);
--
2.43.0
Internal Use - Confidential
^ permalink raw reply related
* Re: [PATCHv2] blk-mq: check for stale cached request in blk_mq_submit_bio
From: Keith Busch @ 2026-05-18 14:44 UTC (permalink / raw)
To: Keith Busch; +Cc: axboe, hch, linux-block
In-Reply-To: <20260501174120.403960-1-kbusch@meta.com>
On Fri, May 01, 2026 at 10:41:19AM -0700, Keith Busch wrote:
> When submitting a bio to blk-mq, if the task should sleep after peeking
> a cached request, but before it pops it, the plug flushes and calls
> blk_mq_free_plug_rqs, freeing the cached_rqs. This creates a
> use-after-free bug. Fix this by ensuring the cached_rqs still contains
> our peeked request, and retry the bio submission without it if the
> request had been freed.
Any thoughts on this one? I know it's an old bug, but I've only recently
started seeing it happen for some reason.
^ permalink raw reply
* Re: [PATCH v3 05/10] lib: add dmabuf token infrastructure
From: Pavel Begunkov @ 2026-05-18 14:23 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Jens Axboe, Keith Busch, Sagi Grimberg, Alexander Viro,
Christian Brauner, Andrew Morton, Sumit Semwal,
Christian König, linux-block, linux-kernel, linux-nvme,
linux-fsdevel, io-uring, linux-media, dri-devel, linaro-mm-sig,
Nitesh Shetty, Kanchan Joshi, Anuj Gupta, Tushar Gohad,
William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <20260518125326.GA5754@lst.de>
On 5/18/26 13:53, Christoph Hellwig wrote:
> On Mon, May 18, 2026 at 11:14:09AM +0100, Pavel Begunkov wrote:
>>> This is about dma-buf based I/O. So I'd expect it to be named dma-buf-io
>>> and no io-dmabuf, and live in drivers/dma-buf and not the unrelated lib/.
>>> But I'd like to hear from the dma-buf maintainers about that.
>>
>> Looking at what Ming is saying, it'd make more sense to keep some of the
>> parts like iterator and the file op more flexible and not automatically
>> imply dma-buf even if it's the main and for now the only medium. I.e.
>> ublk/fuse can use a similar interface for mapping buffers to the server
>> even without dma mappings.
>>
>> I don't know how the API should look like, maybe passing memfd, and dma-buf
>> supports mmap, but I think it's better to call the op something like
>> "register_buffer" instead and keep all it in lib/ for the same reasons.
>
> Let's get the current version landed. If we come up with some kind of
> non-dma dmabuf in the future we can refactor it and move it around.
> I'm a little skeptic we'll be able to share code as long as dmabuf
> is allergic to physical addresses, though.
To be fair, it's not that dma-buf specific. This lib/ code only
does some resv locking, fence waiting and queuing fences, otherwise
all the attaching is done by the driver behind callbacks. Switching
it to some memfd could be pretty simple. But The main thing it'd
need to share is iterator handling like forwarding in the block
layer, and it should be fine as it's already passed as a completely
opaque object with no knowledge about pages / dma / etc. for the
middle layers.
> lib/ is most certainly the wrong place for something that absolutely
> is not library functionality but directly interacts with a few
> subsystems.
It only interacts with dma-buf, and even for dma-buf attachments
are created by the driver. Block, nvme, io_uring are users, either
using the helpers or implementing callbacks.
Ok. Let's assume for the argument's sake it's not dma-buf
specific, if not lib/, where would you put it? I was also
assuming that dma-buf being under drivers/ is rather a relic
of the past rather than the desired location, hmm?
--
Pavel Begunkov
^ permalink raw reply
* Re: [PATCH v3 05/10] lib: add dmabuf token infrastructure
From: Markus Elfring @ 2026-05-18 14:02 UTC (permalink / raw)
To: Pavel Begunkov, linux-block, linux-fsdevel, linux-media, io-uring,
linux-nvme, linaro-mm-sig, dri-devel, Alexander Viro,
Andrew Morton, Christian Brauner, Christian König,
Christoph Hellwig, Jens Axboe, Keith Busch, Sagi Grimberg,
Sumit Semwal
Cc: LKML, Anuj Gupta, Jason Gunthorpe, Kanchan Joshi, Kees Cook,
Nitesh Shetty, Phil Cayton, Tushar Gohad, William Power
In-Reply-To: <c61e6d928f86f4cb253ae350272e6039faefd3a6.1777475843.git.asml.silence@gmail.com>
…
> +++ b/lib/io_dmabuf_token.c
> @@ -0,0 +1,272 @@
…
> +static void io_dmabuf_map_release_work(struct work_struct *work)
> +{
…
> + dma_resv_lock(dmabuf->resv, NULL);
> + token->dev_ops->unmap(token, map);
> + dma_resv_unlock(dmabuf->resv);
…
How do you think about to apply another lock guard?
Regards,
Markus
^ permalink raw reply
* Re: [PATCH v3 04/10] block: introduce dma map backed bio type
From: Pavel Begunkov @ 2026-05-18 13:59 UTC (permalink / raw)
To: Christoph Hellwig
Cc: Christian König, Jens Axboe, Keith Busch, Sagi Grimberg,
Alexander Viro, Christian Brauner, Andrew Morton, Sumit Semwal,
linux-block, linux-kernel, linux-nvme, linux-fsdevel, io-uring,
linux-media, dri-devel, linaro-mm-sig, Nitesh Shetty,
Kanchan Joshi, Anuj Gupta, Tushar Gohad, William Power,
Phil Cayton, Jason Gunthorpe
In-Reply-To: <20260518125713.GC5754@lst.de>
On 5/18/26 13:57, Christoph Hellwig wrote:
> On Mon, May 18, 2026 at 01:40:18PM +0100, Pavel Begunkov wrote:
>>> When that is really a performance critical path then you can use the likely() and unlikely() macros to give the compiler the hint which one to prefer.
>>
>> That might be more penalising than placing them in the right order,
>> and it might be fine as it's new and all that, but it's not a clear
>> cut as it's definitely not created to be a slow path.
>
> Yes. Whatever the caller is using at a given time is the fast path here,
> and dynamic branch prediction in modern cpus handles this really well.
>
>> TBH, not sure
>> why we're bike shedding such things, is it somewhere in the code
>> style?
>
> It makes reading the code annoying, so it better have a good reason.
> Now for a single conditional it's not much of an issue, but these
> things tend to pile up and then start to get really annoying.
> Always write your code the most straight forward way unless you
> have a good reason not to.
For me, having the most common case first is more natural and
readable, but I'm going to flip it just to avoid spending more
time here.
--
Pavel Begunkov
^ permalink raw reply
* Re: [PATCH v6 0/2] blk-mq: introduce tag starvation observability
From: Jens Axboe @ 2026-05-18 13:31 UTC (permalink / raw)
To: Aaron Tomlin, rostedt, mhiramat, mathieu.desnoyers
Cc: bvanassche, johannes.thumshirn, kch, dlemoal, ritesh.list,
loberman, neelx, sean, mproche, chjohnst, linux-block,
linux-kernel, linux-trace-kernel
In-Reply-To: <20260517213614.350367-1-atomlin@atomlin.com>
On 5/17/26 3:36 PM, Aaron Tomlin wrote:
> Hi Jens, Steve, Masami,
>
> In high-performance storage environments, particularly when utilising RAID
> controllers with shared tag sets (BLK_MQ_F_TAG_HCTX_SHARED), severe latency
> spikes can occur when fast devices are starved of available tags.
> Currently, diagnosing this specific queue contention requires deploying
> dynamic kprobes or inferring sleep states, which lacks a simple,
> out-of-the-box diagnostic path.
>
> This short series introduces dedicated, low-overhead observability for tag
> exhaustion events in the block layer:
>
> - Patch 1 introduces the "block_rq_tag_wait" tracepoint in the tag
> allocation slow-path to capture precise, event-based starvation.
>
> - Patch 2 complements this by exposing "wait_on_hw_tag" and
> "wait_on_sched_tag" per-CPU counters via debugfs for quick,
> point-in-time cumulative polling.
>
> Together, these provide storage engineers with zero-configuration
> mechanisms to definitively identify shared-tag bottlenecks.
Why not just issue the trace points? Then there's close to zero
overhead, rather than needing to need added counters for this, and the
kernel to keep track. If you just issue the get/put tag kind of traces,
then userspace can keep track. That's what blktrace has done for decades
for things like inflight/queue depth accounting.
IOW, seems to me, this could be done with basically zero kernel
additions outside of perhaps a trace point or two.
--
Jens Axboe
^ permalink raw reply
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox