* Re: [PATCH v2 2/3] dm-inlinecrypt: add target for inline block device encryption
From: Linlin Zhang @ 2026-04-30 9:21 UTC (permalink / raw)
To: Benjamin Marzinski
Cc: linux-block, ebiggers, mpatocka, gmazyland, linux-kernel,
adrianvovk, dm-devel, quic_mdalam, israelr, hch, axboe
In-Reply-To: <afIqszyKboCEmHDa@redhat.com>
On 4/29/2026 11:58 PM, Benjamin Marzinski wrote:
> On Wed, Apr 29, 2026 at 11:25:04AM -0400, Benjamin Marzinski wrote:
>> On Wed, Apr 29, 2026 at 08:34:00PM +0800, Linlin Zhang wrote:
> [...]
>>>
>>> Thanks for the suggestions.
>>>
>>> Adding a bool need_acct parameter to __blk_crypto_submit_bio() would require
>>> updating all existing callers, which feels rather intrusive given that the
>>> accounting issue only affects the blk‑crypto fallback write slow‑path. I’m a
>>> bit concerned that this would broaden the scope of the change more than
>>> necessary for the problem at hand.
>>
>> I get your concern, and I'd like a second opinion on how much we should
>> care about this, but it doesn't look like there are many other callers
>> that would be effected here. The only existing caller of
>> __blk_crypto_submit_bio() is blk_crypto_submit_bio(), which would just
>> call it with "need_acct=true". Looking at the code path below
>> __blk_crypto_submit_bio() that would need to change for submitting the
>> bios:
>>
>> __blk_crypto_submit_bio() is the only caller of
>> blk_crypto_fallback_bio_prep()
>>
>> blk_crypto_fallback_bio_prep() is the only caller of
>> blk_crypto_fallback_encrypt_bio().
>>
>> blk_crypto_fallback_encrypt_bio() is the only caller of
>> __blk_crypto_fallback_encrypt_bio(), which is the function that would
>> need to choose between submit_bio() and submit_bio_noacct().
>>
>> Doing this would change the crypto API (by necessity, since we're adding
>> a new argument to __blk_crypto_submit_bio() for stacking devices to
>> use), and it is adds a extra argument to a number of functions, just to
>> handle this corner case. But it is still a relatively contained change.
>
> Having discussed this a bit, I'm fine with leaving this as a TODO for
> now. If anyone wants to chime in with an opinion on how acceptable it
> would be to add a new bio flag for skipping accounting, that would be
> great.
Thanks, Ben.
I’d appreciate it if someone else could chime in on these two approaches.
>
> -Ben
>
^ permalink raw reply
* Re: [RFC PATCH v2 0/4] mm/zsmalloc: reduce zs_free() latency on swap release path
From: Kairui Song @ 2026-04-30 8:00 UTC (permalink / raw)
To: Wenchao Hao
Cc: Yosry Ahmed, Andrew Morton, Chengming Zhou, Jens Axboe,
Johannes Weiner, Minchan Kim, Nhat Pham, Sergey Senozhatsky,
linux-block, linux-kernel, linux-mm, Barry Song, Xueyuan Chen,
Wenchao Hao
In-Reply-To: <CAOptpSMLLz0yzg99nkAp7rxO5RphTDU3Cdqd8020mysijN_Bvg@mail.gmail.com>
On Thu, Apr 30, 2026 at 3:43 PM Wenchao Hao <haowenchao22@gmail.com> wrote:
> The data I shared earlier was class_idx-in-obj only — no
> deferred freeing at all.
>
> > I couldn't immediately tell by looking at this vs. the cover letter. I wonder
> > what portion of the improvement comes from the deferred freeing?
>
> On top of that, we added deferred freeing in the zsmalloc
> layer (per-cpu page-pool based buffer swap + WQ_UNBOUND
> drain worker). With both class_idx + deferred:
>
> Test 1: concurrent munmap (256MB/process, RPi 4B):
>
> mode Base Deferred Speedup
> single 56.2ms 17.2ms 3.27x
> multi 3p 153.2ms 51.5ms 2.97x
>
> Test 2: single process munmap (various sizes):
>
> size Base Deferred Speedup
> 64MB 15.0ms 4.3ms 3.47x
> 128MB 28.7ms 8.5ms 3.37x
> 192MB 43.2ms 13.0ms 3.32x
> 256MB 57.0ms 17.3ms 3.30x
> 512MB 114.4ms 38.5ms 2.97x
Hi Wenchao,
One concern here is that the total amount of work is unchanged. I mean
you observe speed up because you offloaded the work to an async
worker. But when under pressure these workers could be a larger
burden. Is it possible for you to measure that part too?
^ permalink raw reply
* Re: [RFC PATCH v2 0/4] mm/zsmalloc: reduce zs_free() latency on swap release path
From: Wenchao Hao @ 2026-04-30 7:38 UTC (permalink / raw)
To: Yosry Ahmed
Cc: Andrew Morton, Chengming Zhou, Jens Axboe, Johannes Weiner,
Minchan Kim, Nhat Pham, Sergey Senozhatsky, linux-block,
linux-kernel, linux-mm, Barry Song, Xueyuan Chen, Wenchao Hao
In-Reply-To: <CAO9r8zPV=c9v4Ar7t+xPxyptSUPrs-shrumsz=hCaNa3anGgAA@mail.gmail.com>
On Thu, Apr 30, 2026 at 6:44 AM Yosry Ahmed <yosry@kernel.org> wrote:
>
> > > How much of the benefit do we get with just these locking improvements
> > > without having to defer any of the freeing work?
> > >
> >
> > Hi Yosry,
> >
> > Thanks for the review. Great question — we tested exactly this.
> >
> > With only the class_idx-in-obj encoding (eliminating pool->lock from
> > zs_free, no deferred freeing), we measured on two platforms.
> >
> > Test: each process independently mmap 256MB, write data, madvise
> > MADV_PAGEOUT to swap out via zram (lzo-rle), then concurrent munmap.
> >
> > Raspberry Pi 4B (4-core ARM64 Cortex-A72):
> >
> > mode Base ClassIdx-only Speedup
> > single 59.0ms 56.0ms 1.05x
> > multi 2p 94.6ms 66.7ms 1.42x
> > multi 4p 202.9ms 110.6ms 1.83x
> >
> > x86 physical machine (4-core Intel i7-12700, 2 rounds averaged):
> >
> > mode Base ClassIdx-only Speedup
> > single 11.7ms 9.8ms 1.19x
> > multi 2p 24.1ms 17.2ms 1.40x
> > multi 4p 63.0ms 45.3ms 1.39x
> >
> > Single-process shows modest improvement. With multiple processes,
> > each read_lock/read_unlock atomically modifies the shared rwlock
> > reader count, and the cost of these atomic operations increases
> > with more CPUs accessing the same cacheline concurrently.
> > Eliminating pool->lock removes this overhead entirely.
> >
> > This only works on 64-bit systems where OBJ_INDEX_BITS has enough
> > spare bits to fit class_idx. 32-bit systems don't have the room.
> > I'm still working on the compile-time gating to properly enable
> > this based on architecture and page size configuration.
> >
> > > As others have pointed out, I don't want to just defer expensive work
> > > without understanding why it's expensive and running into limitations
> > > about why it cannot be improved without deferring.
> >
> > For the deferred freeing part: the class_idx-in-obj optimization
> > addresses the multi-process scenario where concurrent atomic
> > operations on pool->lock become expensive, but does not help
> > single-process munmap. Deferred freeing moves the entire zs_free
> > cost (including class->lock and zspage freeing) off the munmap
> > hot path, which benefits even single-process workloads. The two
> > optimizations are complementary.
>
> What is the extra speedup added by the deferred freeing
> on top of the locking improvements?
The data I shared earlier was class_idx-in-obj only — no
deferred freeing at all.
> I couldn't immediately tell by looking at this vs. the cover letter. I wonder
> what portion of the improvement comes from the deferred freeing?
On top of that, we added deferred freeing in the zsmalloc
layer (per-cpu page-pool based buffer swap + WQ_UNBOUND
drain worker). With both class_idx + deferred:
Test 1: concurrent munmap (256MB/process, RPi 4B):
mode Base Deferred Speedup
single 56.2ms 17.2ms 3.27x
multi 3p 153.2ms 51.5ms 2.97x
Test 2: single process munmap (various sizes):
size Base Deferred Speedup
64MB 15.0ms 4.3ms 3.47x
128MB 28.7ms 8.5ms 3.37x
192MB 43.2ms 13.0ms 3.32x
256MB 57.0ms 17.3ms 3.30x
512MB 114.4ms 38.5ms 2.97x
However, this is not the ceiling. Profiling with perf
shows that after deferred zs_free, zram_slot_free_notify
still accounts for ~65% of munmap time — mostly
slot_trylock/unlock and slot metadata operations.
To understand the theoretical limit, I tested an extreme
version that removes slot_trylock from the hot path
entirely (not safe for production, just benchmarking):
size Base Deferred No-lock Speedup
64MB 15.0ms 4.3ms 2.3ms 6.50x
128MB 28.7ms 8.5ms 4.7ms 6.14x
192MB 43.2ms 13.0ms 6.8ms 6.31x
256MB 57.0ms 17.3ms 9.0ms 6.30x
512MB 114.4ms 38.5ms 33.0ms 3.46x
I'm exploring ways to further reduce or eliminate the lock
from this path, any suggestions on how to approach this
would be appreciated.
Unless otherwise noted, all data is from Raspberry Pi 4B
(4-core ARM64 Cortex-A72, 8GB RAM, zram 2GB, lzo-rle).
Test: mmap + fill + madvise(MADV_PAGEOUT) to swap out
via zram, then measure munmap time.
Thanks,
Wenchao
^ permalink raw reply
* Re: [PATCH v3 01/10] file: add callback for creating long-term dmabuf maps
From: Christian König @ 2026-04-30 6:03 UTC (permalink / raw)
To: Pavel Begunkov, Jens Axboe, Keith Busch, Christoph Hellwig,
Sagi Grimberg, Alexander Viro, Christian Brauner, Andrew Morton,
Sumit Semwal, linux-block, linux-kernel, linux-nvme,
linux-fsdevel, io-uring, linux-media, dri-devel, linaro-mm-sig
Cc: Nitesh Shetty, Kanchan Joshi, Anuj Gupta, Tushar Gohad,
William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <ae941457cf6cacb9d4c16b6ec904da9ef7fed97f.1777475843.git.asml.silence@gmail.com>
On 4/29/26 17:25, Pavel Begunkov wrote:
> Introduce a new file callback that allows creating long-term dma
> mapping. All necessary information together with a dmabuf will be passed
> in the second argument of type struct io_dmabuf_token, which will be
> defined in following patches.
Well first of all the naming is probably not the best. Maybe rather call that dma-buf attachment or context or mappping.
Then the patch should probably define the full interface and not just add the callback here and then the structure in a follow up patch.
Regards,
Christian.
>
> Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
> ---
> include/linux/fs.h | 2 ++
> 1 file changed, 2 insertions(+)
>
> diff --git a/include/linux/fs.h b/include/linux/fs.h
> index b5b01bb22d12..c5558aab4628 100644
> --- a/include/linux/fs.h
> +++ b/include/linux/fs.h
> @@ -1920,6 +1920,7 @@ struct dir_context {
>
> struct io_uring_cmd;
> struct offset_ctx;
> +struct io_dmabuf_token;
>
> typedef unsigned int __bitwise fop_flags_t;
>
> @@ -1967,6 +1968,7 @@ struct file_operations {
> int (*uring_cmd_iopoll)(struct io_uring_cmd *, struct io_comp_batch *,
> unsigned int poll_flags);
> int (*mmap_prepare)(struct vm_area_desc *);
> + int (*create_dmabuf_token)(struct file *, struct io_dmabuf_token *);
> } __randomize_layout;
>
> /* Supports async buffered reads */
^ permalink raw reply
* Re: [PATCH] badblocks: fix infinite loop due to incorrect rounding and overflow
From: kernel test robot @ 2026-04-30 5:09 UTC (permalink / raw)
To: Ramesh Adhikari, axboe, gregkh
Cc: oe-kbuild-all, linux-block, Ramesh Adhikari
In-Reply-To: <20260427151048.756072-1-adhikari.resume@gmail.com>
Hi Ramesh,
kernel test robot noticed the following build errors:
[auto build test ERROR on axboe/for-next]
[also build test ERROR on linus/master v7.1-rc1 next-20260429]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ramesh-Adhikari/badblocks-fix-infinite-loop-due-to-incorrect-rounding-and-overflow/20260429-044425
base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git for-next
patch link: https://lore.kernel.org/r/20260427151048.756072-1-adhikari.resume%40gmail.com
patch subject: [PATCH] badblocks: fix infinite loop due to incorrect rounding and overflow
config: i386-allnoconfig (https://download.01.org/0day-ci/archive/20260430/202604301231.IpPh4AiH-lkp@intel.com/config)
compiler: gcc-14 (Debian 14.2.0-19) 14.2.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260430/202604301231.IpPh4AiH-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202604301231.IpPh4AiH-lkp@intel.com/
All errors (new ones prefixed by >>):
ld: block/badblocks.o: in function `badblocks_check':
badblocks.c:(.text+0x10af): undefined reference to `__umoddi3'
>> ld: badblocks.c:(.text+0x1109): undefined reference to `__umoddi3'
ld: block/badblocks.o: in function `_badblocks_set':
badblocks.c:(.text+0x12b0): undefined reference to `__umoddi3'
ld: badblocks.c:(.text+0x12f4): undefined reference to `__umoddi3'
ld: block/badblocks.o: in function `_badblocks_clear':
badblocks.c:(.text+0x1bfc): undefined reference to `__umoddi3'
ld: block/badblocks.o:badblocks.c:(.text+0x1c1c): more undefined references to `__umoddi3' follow
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* [PATCH] block: blk-mq: fix UAF in blk_mq_tagset_busy_iter
From: l1za0.sec @ 2026-04-30 4:28 UTC (permalink / raw)
To: axboe; +Cc: linux-block, linux-kernel
From: Haocheng Yu <l1za0.sec@gmail.com>
A KASAN: slab-use-after-free Read in blk_mq_tagset_busy_iter is
reported by a modified Syzkaller-based kernel fuzzing tool we
developed.
This problem is caused by a race condition between
block/blk-mq-tag.c/blk_mq_tagset_busy_iter() and
block/blk-mq.c/blk_mq_realloc_tag_set_tags(). In
blk_mq_realloc_tag_set_tags(), set->tags is first freed, and then
new_tags is assigned to set->tags. However, this process is not
protected by synchronization. Therefore, if another process reads
tagset->tags in blk_mq_tagset_busy_iter() between these two steps,
it will cause a use-after-free read problem.
To fix this vulnerability, first save the old set->tags. After
updating set->tags to new_tags, wait for the reading side to exit
before releasing it. This avoids the problem of tagset->tags being
directly released while blk_mq_tagset_busy_iter() is still iterating.
Signed-off-by: Haocheng Yu <l1za0.sec@gmail.com>
---
The full reproducer is attached here:
# {Threaded:true Repeat:true RepeatTimes:0 Procs:8 Slowdown:1 Sandbox:none SandboxArg:0 Leak:false NetInjection:true NetDevices:true NetReset:true Cgroups:true BinfmtMisc:true CloseFDs:true KCSAN:false DevlinkPCI:false NicVF:false USB:true VhciInjection:true Wifi:true IEEE802154:true Sysctl:true Swap:true UseTmpDir:true HandleSegv:true Trace:false CallComments:true LegacyOptions:{Collide:false Fault:false FaultCall:0 FaultNth:0}}
r0 = syz_open_dev$ndb(&(0x7f0000000000), 0x0, 0xc0400)
r1 = syz_open_dev$ndb(&(0x7f0000000000), 0x0, 0x80040)
ioctl$NBD_SET_FLAGS(r1, 0xab0a, 0x9ad)
socketpair$nbd(0x1, 0x1, 0x0, &(0x7f0000000080)={<r2=>0xffffffffffffffff})
r3 = syz_open_dev$dri(&(0x7f0000000100), 0xfffffffffffffffc, 0xc8503)
r4 = syz_open_dev$ndb(&(0x7f0000000000), 0x0, 0xc0400)
socketpair$nbd(0x1, 0x1, 0x0, &(0x7f0000000080)={<r5=>0xffffffffffffffff})
ioctl$NBD_SET_SOCK(r4, 0xab00, r5)
r6 = syz_open_dev$ndb(&(0x7f0000000240), 0x0, 0x12100)
ioctl$NBD_DO_IT(r6, 0xab03)
close_range(r3, 0xffffffffffffffff, 0x0)
ioctl$NBD_SET_SIZE_BLOCKS(0xffffffffffffffff, 0xab07, 0x1)
ioctl$NBD_SET_SOCK(r0, 0xab00, r2)
r7 = syz_open_dev$loop(&(0x7f0000000040), 0x1, 0x200)
ioctl$BLKPG(r7, 0x1269, &(0x7f00000001c0)={0x1, 0x0, 0x98, &(0x7f00000000c0)={0x5, 0x2, 0x10}})
close(0x5)
r8 = syz_open_dev$ndb(&(0x7f0000000000), 0x0, 0x100)
r9 = syz_open_dev$ndb(&(0x7f0000000000), 0x0, 0xc0400)
socketpair$nbd(0x1, 0x1, 0x0, &(0x7f0000000080)={<r10=>0xffffffffffffffff})
ioctl$NBD_SET_SOCK(r9, 0xab00, r10)
close(0x5)
r11 = syz_open_dev$ndb(&(0x7f0000000000), 0x0, 0x100)
ioctl$NBD_DO_IT(r11, 0xab03)
r12 = syz_open_dev$ndb(&(0x7f0000000040), 0x0, 0x0)
ioctl$NBD_CLEAR_SOCK(r12, 0xab04)
ioctl$NBD_DO_IT(r8, 0xab03)
socket$inet6_tcp(0xa, 0x1, 0x0)
syz_open_dev$ndb(&(0x7f0000000000), 0x0, 0xc0400)
block/blk-mq.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/block/blk-mq.c b/block/blk-mq.c
index d626d32f6e57..4357625a512d 100644
--- a/block/blk-mq.c
+++ b/block/blk-mq.c
@@ -4738,6 +4738,7 @@ static int blk_mq_realloc_tag_set_tags(struct blk_mq_tag_set *set,
int new_nr_hw_queues)
{
struct blk_mq_tags **new_tags;
+ struct blk_mq_tags **old_tags;
int i;
if (set->nr_hw_queues >= new_nr_hw_queues)
@@ -4751,8 +4752,10 @@ static int blk_mq_realloc_tag_set_tags(struct blk_mq_tag_set *set,
if (set->tags)
memcpy(new_tags, set->tags, set->nr_hw_queues *
sizeof(*set->tags));
- kfree(set->tags);
+ old_tags = set->tags;
set->tags = new_tags;
+ synchronize_srcu(&set->tags_srcu);
+ kfree(old_tags);
for (i = set->nr_hw_queues; i < new_nr_hw_queues; i++) {
if (!__blk_mq_alloc_map_and_rqs(set, i)) {
base-commit: 7d0a66e4bb9081d75c82ec4957c50034cb0ea449
--
2.51.0
^ permalink raw reply related
* Re: [PATCH v2 00/33] rust: bump minimum Rust and `bindgen` versions
From: patchwork-bot+linux-riscv @ 2026-04-30 3:25 UTC (permalink / raw)
To: Miguel Ojeda
Cc: linux-riscv, nathan, nsc, dakr, a.hindborg, catalin.marinas, will,
pjw, palmer, aou, acourbot, airlied, simona, brendan.higgins,
david, gregkh, arve, tkjos, christian, cmllamas, aliceryhl,
corbet, boqun, gary, bjorn3_gh, lossin, tmgross, rust-for-linux,
linux-kbuild, lorenzo.stoakes, vbabka, Liam.Howlett, urezki,
linux-block, linux-arm-kernel, alex, nouveau, dri-devel,
raemoar63, linux-kselftest, kunit-dev, nick.desaulniers+lkml,
morbo, justinstitt, llvm, linux-kernel, skhan, linux-doc
In-Reply-To: <20260405235309.418950-1-ojeda@kernel.org>
Hello:
This series was applied to riscv/linux.git (fixes)
by Miguel Ojeda <ojeda@kernel.org>:
On Mon, 6 Apr 2026 01:52:36 +0200 you wrote:
> As proposed in the past in e.g. LPC 2025 and the Maintainers Summit [1],
> we are going to follow Debian Stable's Rust versions as our minimum
> supported version.
>
> Debian Trixie was released with a Rust 1.85.0 toolchain [2], which it
> still uses to this day [3] (i.e. no update to Rust 1.85.1).
>
> [...]
Here is the summary with links:
- [v2,01/33] rust: kbuild: remove `--remap-path-prefix` workarounds
https://git.kernel.org/riscv/c/c8cbe2fc22e4
- [v2,02/33] rust: kbuild: remove "`try` keyword" workaround for `bindgen` < 0.59.2
https://git.kernel.org/riscv/c/518b9ad2fab3
- [v2,03/33] rust: kbuild: remove unneeded old `allow`s for generated layout tests
https://git.kernel.org/riscv/c/92cc022f044f
- [v2,04/33] gpu: nova-core: bindings: remove unneeded `cfg_attr`
https://git.kernel.org/riscv/c/341c51ee3293
- [v2,05/33] rust: bump Rust minimum supported version to 1.85.0 (Debian Trixie)
https://git.kernel.org/riscv/c/f32fb9c58a5b
- [v2,06/33] rust: bump Clippy's MSRV and clean `incompatible_msrv` allows
https://git.kernel.org/riscv/c/b6cfba436623
- [v2,07/33] rust: allow globally `clippy::incompatible_msrv`
https://git.kernel.org/riscv/c/7ed188605e1d
- [v2,08/33] rust: simplify `RUSTC_VERSION` Kconfig conditions
https://git.kernel.org/riscv/c/b28711ac98e8
- [v2,09/33] rust: remove `RUSTC_HAS_SLICE_AS_FLATTENED` and simplify code
https://git.kernel.org/riscv/c/9b398d056543
- [v2,10/33] rust: remove `RUSTC_HAS_COERCE_POINTEE` and simplify code
https://git.kernel.org/riscv/c/4ab22c543f18
- [v2,11/33] rust: kbuild: remove skipping of `-Wrustdoc::unescaped_backticks`
https://git.kernel.org/riscv/c/0f6e1e0705f0
- [v2,12/33] rust: kbuild: remove `feature(...)`s that are now stable
https://git.kernel.org/riscv/c/d1aa40daa777
- [v2,13/33] rust: transmute: simplify code with Rust 1.80.0 `split_at_*checked()`
https://git.kernel.org/riscv/c/f309a6edda53
- [v2,14/33] rust: alloc: simplify with `NonNull::add()` now that it is stable
https://git.kernel.org/riscv/c/161dd7b51e96
- [v2,15/33] rust: macros: simplify code using `feature(extract_if)`
https://git.kernel.org/riscv/c/42ec980024f0
- [v2,16/33] rust: block: update `const_refs_to_static` MSRV TODO comment
https://git.kernel.org/riscv/c/961b72d45ae4
- [v2,17/33] rust: bump `bindgen` minimum supported version to 0.71.1 (Debian Trixie)
https://git.kernel.org/riscv/c/c3a00a3f31ff
- [v2,18/33] rust: rust_is_available: remove warning for `bindgen` 0.66.[01]
https://git.kernel.org/riscv/c/41cfbb4295cf
- [v2,19/33] rust: rust_is_available: remove warning for `bindgen` < 0.69.5 && libclang >= 19.1
https://git.kernel.org/riscv/c/ae64324ad5c1
- [v2,20/33] rust: kbuild: update `bindgen --rust-target` version and replace comment
https://git.kernel.org/riscv/c/276ed30c558e
- [v2,21/33] rust: kbuild: remove "dummy parameter" workaround for `bindgen` < 0.71.1
https://git.kernel.org/riscv/c/93553d9922b0
- [v2,22/33] docs: rust: quick-start: openSUSE provides `rust-src` package nowadays
https://git.kernel.org/riscv/c/53c9647c0a48
- [v2,23/33] docs: rust: quick-start: update Ubuntu versioned packages
https://git.kernel.org/riscv/c/982e1aa6de73
- [v2,24/33] docs: rust: quick-start: update minimum Ubuntu version
https://git.kernel.org/riscv/c/6767147cb941
- [v2,25/33] docs: rust: quick-start: add Ubuntu 26.04 LTS and remove subsection title
https://git.kernel.org/riscv/c/780f847e1419
- [v2,26/33] docs: rust: quick-start: remove Gentoo "testing" note
https://git.kernel.org/riscv/c/99c672426aed
- [v2,27/33] docs: rust: quick-start: remove Nix "unstable channel" note
https://git.kernel.org/riscv/c/b69a14650009
- [v2,28/33] docs: rust: quick-start: remove GDB/Binutils mention
https://git.kernel.org/riscv/c/a4392ed1c8b9
- [v2,29/33] docs: rust: general-information: simplify Kconfig example
https://git.kernel.org/riscv/c/9375ea727d7e
- [v2,30/33] docs: rust: general-information: use real example
https://git.kernel.org/riscv/c/86c5d1c6740c
- [v2,31/33] rust: declare cfi_encoding for lru_status
https://git.kernel.org/riscv/c/9e5946de3a38
- [v2,32/33] rust: kbuild: support global per-version flags
https://git.kernel.org/riscv/c/b2aa1535ecdd
- [v2,33/33] rust: kbuild: allow `clippy::precedence` for Rust < 1.86.0
https://git.kernel.org/riscv/c/2e2f8b5a0656
You are awesome, thank you!
--
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html
^ permalink raw reply
* Re: [PATCH v12 02/13] lib/group_cpus: remove dead !SMP code
From: Aaron Tomlin @ 2026-04-29 23:32 UTC (permalink / raw)
To: Sebastian Andrzej Siewior
Cc: axboe, kbusch, hch, sagi, mst, aacraid, James.Bottomley,
martin.petersen, liyihang9, kashyap.desai, sumit.saxena,
shivasharan.srikanteshwara, chandrakanth.patil, sathya.prakash,
sreekanth.reddy, suganath-prabu.subramani, ranjan.kumar,
jinpu.wang, tglx, mingo, peterz, juri.lelli, vincent.guittot,
akpm, maz, ruanjinjie, yphbchou0911, wagi, frederic, longman,
chenridong, hare, kch, ming.lei, tom.leiming, steve, sean,
chjohnst, neelx, mproche, nick.lange, marco.crivellari,
linux-block, linux-kernel, virtualization, linux-nvme, linux-scsi,
megaraidlinux.pdl, mpi3mr-linuxdrv.pdl, MPT-FusionLinux.pdl
In-Reply-To: <20260427152104.WTGAesGs@linutronix.de>
[-- Attachment #1: Type: text/plain, Size: 2466 bytes --]
On Mon, Apr 27, 2026 at 05:21:04PM +0200, Sebastian Andrzej Siewior wrote:
> On 2026-04-22 14:52:04 [-0400], Aaron Tomlin wrote:
> > From: Daniel Wagner <wagi@kernel.org>
> >
> > The support for the !SMP configuration has been removed from the core by
> > commit cac5cefbade9 ("sched/smp: Make SMP unconditional").
> >
> > While one can technically still compile a uniprocessor kernel, the core
> > scheduler now mandates SMP unconditionally, rendering this particular
> > !SMP fallback handling redundant. Therefore, remove the #ifdef CONFIG_SMP
> > guards and the fallback logic.
> >
> > Signed-off-by: Daniel Wagner <wagi@kernel.org>
> > Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
> > Reviewed-by: Hannes Reinecke <hare@suse.de>
> > [atomlin: Updated commit message to clarify !SMP removal context]
>
> This look unchanged vs previous submission. You could explain why you
> want to remove the !SMP case. It looks like the !SMP makes things
> easier ;) I don't know how much of this gets removed because of !SMP
> code elsewhere.
>
> The description still does not make sense/ is accurate.
Hi Sebastian,
Yes, the !SMP path does make things "easier" and lighter for actual UP
builds by bypassing the SMP overhead. However, maintaining two separate
code paths and #ifdef guards for this specific logic adds testing and
maintenance burden that we'd prefer to drop.
The reference to the scheduler commit was meant to highlight a
philosophical alignment-trading a slight performance edge on UP builds in
exchange for a single, unified code path without #ifdef clutter.
How about the following:
The core scheduler recently transitioned to compiling SMP data structures
unconditionally to reduce code complexity (see commit cac5cefbade9
"sched/smp: Make SMP unconditional").
In alignment with this philosophy of reducing dual-path maintenance, this
patch removes the #ifdef CONFIG_SMP guards and the dedicated !SMP fallback
logic here.
While the !SMP path provided a slightly simpler execution flow for
uniprocessor kernels (avoiding SMP-specific overhead), maintaining these
separate code paths adds unnecessary complexity and testing burden.
Removing these guards simplifies the codebase by standardizing entirely on
the SMP logic, which safely resolves to single-CPU operations on UP
configurations.
Kind regards,
--
Aaron Tomlin
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply
* Re: [PATCH] badblocks: fix infinite loop due to incorrect rounding and overflow
From: kernel test robot @ 2026-04-29 23:06 UTC (permalink / raw)
To: Ramesh Adhikari, axboe, gregkh
Cc: oe-kbuild-all, linux-block, Ramesh Adhikari
In-Reply-To: <20260427151048.756072-1-adhikari.resume@gmail.com>
Hi Ramesh,
kernel test robot noticed the following build errors:
[auto build test ERROR on axboe/for-next]
[also build test ERROR on linus/master v7.1-rc1 next-20260429]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]
url: https://github.com/intel-lab-lkp/linux/commits/Ramesh-Adhikari/badblocks-fix-infinite-loop-due-to-incorrect-rounding-and-overflow/20260429-044425
base: https://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux.git for-next
patch link: https://lore.kernel.org/r/20260427151048.756072-1-adhikari.resume%40gmail.com
patch subject: [PATCH] badblocks: fix infinite loop due to incorrect rounding and overflow
config: arm-randconfig-002-20260430 (https://download.01.org/0day-ci/archive/20260430/202604300729.gKrAMFdE-lkp@intel.com/config)
compiler: arm-linux-gnueabi-gcc (GCC) 8.5.0
reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20260430/202604300729.gKrAMFdE-lkp@intel.com/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202604300729.gKrAMFdE-lkp@intel.com/
All errors (new ones prefixed by >>):
arm-linux-gnueabi-ld: block/badblocks.o: in function `badblocks_clear':
>> badblocks.c:(.text+0x1d18): undefined reference to `__aeabi_uldivmod'
>> arm-linux-gnueabi-ld: badblocks.c:(.text+0x1d54): undefined reference to `__aeabi_uldivmod'
arm-linux-gnueabi-ld: block/badblocks.o: in function `_badblocks_set':
badblocks.c:(.text+0x2c90): undefined reference to `__aeabi_uldivmod'
arm-linux-gnueabi-ld: badblocks.c:(.text+0x2ce0): undefined reference to `__aeabi_uldivmod'
arm-linux-gnueabi-ld: block/badblocks.o: in function `badblocks_check':
badblocks.c:(.text+0x3fb8): undefined reference to `__aeabi_uldivmod'
arm-linux-gnueabi-ld: block/badblocks.o:badblocks.c:(.text+0x4018): more undefined references to `__aeabi_uldivmod' follow
--
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki
^ permalink raw reply
* Re: [RFC PATCH v2 0/4] mm/zsmalloc: reduce zs_free() latency on swap release path
From: Yosry Ahmed @ 2026-04-29 22:44 UTC (permalink / raw)
To: Wenchao Hao
Cc: Andrew Morton, Chengming Zhou, Jens Axboe, Johannes Weiner,
Minchan Kim, Nhat Pham, Sergey Senozhatsky, linux-block,
linux-kernel, linux-mm, Barry Song, Xueyuan Chen, Wenchao Hao
In-Reply-To: <CAOptpSMQyyb+q-VJdJ0RzNkjaer1CTePM5WkwEj_7cAO4aUToQ@mail.gmail.com>
> > How much of the benefit do we get with just these locking improvements
> > without having to defer any of the freeing work?
> >
>
> Hi Yosry,
>
> Thanks for the review. Great question — we tested exactly this.
>
> With only the class_idx-in-obj encoding (eliminating pool->lock from
> zs_free, no deferred freeing), we measured on two platforms.
>
> Test: each process independently mmap 256MB, write data, madvise
> MADV_PAGEOUT to swap out via zram (lzo-rle), then concurrent munmap.
>
> Raspberry Pi 4B (4-core ARM64 Cortex-A72):
>
> mode Base ClassIdx-only Speedup
> single 59.0ms 56.0ms 1.05x
> multi 2p 94.6ms 66.7ms 1.42x
> multi 4p 202.9ms 110.6ms 1.83x
>
> x86 physical machine (4-core Intel i7-12700, 2 rounds averaged):
>
> mode Base ClassIdx-only Speedup
> single 11.7ms 9.8ms 1.19x
> multi 2p 24.1ms 17.2ms 1.40x
> multi 4p 63.0ms 45.3ms 1.39x
>
> Single-process shows modest improvement. With multiple processes,
> each read_lock/read_unlock atomically modifies the shared rwlock
> reader count, and the cost of these atomic operations increases
> with more CPUs accessing the same cacheline concurrently.
> Eliminating pool->lock removes this overhead entirely.
>
> This only works on 64-bit systems where OBJ_INDEX_BITS has enough
> spare bits to fit class_idx. 32-bit systems don't have the room.
> I'm still working on the compile-time gating to properly enable
> this based on architecture and page size configuration.
>
> > As others have pointed out, I don't want to just defer expensive work
> > without understanding why it's expensive and running into limitations
> > about why it cannot be improved without deferring.
>
> For the deferred freeing part: the class_idx-in-obj optimization
> addresses the multi-process scenario where concurrent atomic
> operations on pool->lock become expensive, but does not help
> single-process munmap. Deferred freeing moves the entire zs_free
> cost (including class->lock and zspage freeing) off the munmap
> hot path, which benefits even single-process workloads. The two
> optimizations are complementary.
What is the extra speedup added by the deferred freeing on top of the
locking improvements? I couldn't immediately tell by looking at this
vs. the cover letter. I wonder what portion of the improvement comes
from the deferred freeing?
^ permalink raw reply
* Re: [PATCH v12 00/13] blk: honor isolcpus configuration
From: Florian Bezdeka @ 2026-04-29 21:01 UTC (permalink / raw)
To: Daniel Wagner
Cc: Aaron Tomlin, axboe, kbusch, hch, sagi, mst, aacraid,
James.Bottomley, martin.petersen, liyihang9, kashyap.desai,
sumit.saxena, shivasharan.srikanteshwara, chandrakanth.patil,
sathya.prakash, sreekanth.reddy, suganath-prabu.subramani,
ranjan.kumar, jinpu.wang, tglx, mingo, peterz, juri.lelli,
vincent.guittot, akpm, maz, ruanjinjie, bigeasy, yphbchou0911,
wagi, frederic, longman, chenridong, hare, kch, ming.lei,
tom.leiming, steve, sean, chjohnst, neelx, mproche, nick.lange,
marco.crivellari, linux-block, linux-kernel, virtualization,
linux-nvme, linux-scsi, megaraidlinux.pdl, mpi3mr-linuxdrv.pdl,
MPT-FusionLinux.pdl, Jan Kiszka
In-Reply-To: <2d61b1f7-fc06-4fe1-8a6f-cc3a2f114ae1@flourine.local>
On Tue, 2026-04-28 at 15:08 +0200, Daniel Wagner wrote:
> On Mon, Apr 27, 2026 at 12:55:20PM +0200, Florian Bezdeka wrote:
> > This topic reminds me of a discussion started by Tobias [1] some time
> > ago about IRQ spreading of network drivers. The problem was (and still
> > is) that network drivers ignore any CPU isolation when spreading out
> > device IRQs.
> >
> > In general we have two different CPU isolation mechanisms:
> > - The static one, via isolcpus= cmdline parameter
> > - The dynamic one, via cgroups(v2) cpuset controller
> >
> > This series is only taking the static "world" into account, right? Are
> > there any plans to honor the CPU isolations configured the dynamic
> > way?
>
> Dynamic configuration would require every driver to fully support
> reconfiguration during runtime. Only a handful of drivers, such as
> nvme-pci, are currently able to handle this.
>
> The first task, teaching a wide range of drivers to honor CPU isolation
> at boot time, is already going to be a significant amount of work.
>
> > It has been a while since the last investigations on my end. Last time I
> > went through the code, the IRQ core was completely decoupled from the
> > dynamic configuration via cgroups. Are there any plans to fix that gap?
>
> Which use case are you actually aiming to support? While dynamic
> reconfiguration would be ideal, the amount of work to get there is
> significant. I won't be signing up for it.
The use case at hand is a RT enabled platform where the concrete RT
workload is not known at boot time. RT applications are deployed "on-
the-fly", nowadays using the existing container runtimes with some
extended resource management on top.
Applications can request certain resources like isolated CPU cores,
special IRQ affinities, PCI devices to pass through, ..., so that the
resource management on the system can take care of proper system
configuration.
As you said, this requires that drivers are able to honor the system
configuration during runtime. We already identified IRQ spreading of the
network subsystem to be problematic. And yes, that will need some
efforts to fix that. We are still in phase of searching for similar
problems and a starting point for a discussion. Any input in that
direction is highly appreciated.
^ permalink raw reply
* Re: [PATCH 0/9] Support for block device NVMEM providers
From: Loic Poulain @ 2026-04-29 20:20 UTC (permalink / raw)
To: Andrew Lunn
Cc: Ulf Hansson, Rob Herring, Krzysztof Kozlowski, Conor Dooley,
Bjorn Andersson, Konrad Dybcio, Jens Axboe, Johannes Berg,
Jeff Johnson, Bartosz Golaszewski, Marcel Holtmann,
Luiz Augusto von Dentz, Balakrishna Godavarthi, Rocky Liao,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Simon Horman, linux-mmc, devicetree, linux-kernel, linux-arm-msm,
linux-block, linux-wireless, ath10k, linux-bluetooth, netdev,
daniel
In-Reply-To: <b4de6c89-dd89-4e02-8d79-911e4f9f0813@lunn.ch>
Hi Andrew,
On Wed, Apr 29, 2026 at 3:06 AM Andrew Lunn <andrew@lunn.ch> wrote:
>
> > Note that this is currently limited to eMMC-backed block devices, as
> > only the eMMC core associates a firmware node with the block device
> > (add_disk_fwnode). This can be easily extended in the future to
> > support additional block drivers.
>
> Would this be
>
> https://elixir.bootlin.com/linux/v7.0.1/source/drivers/mmc/core/block.c#L2641
>
> Looking at that function, mmc_blk_alloc_req() i don't see it doing
> anything different between an eMMC and MMC.
Yes, that’s correct, in the previous sentence I should have referred
to MMC-backed storage rather than specifically eMMC, since the MMC
scope also includes removable MMC/SD cards. That said, eMMC is more
likely to be used as an NVMEM area.
>
> An eMMC you don't expect to go away, since it is soldered
> down. However an MMC can be ejected. Is the code prepared for that?
I was about to say this wouldn’t be an issue, since the NVMEM device
would be unregistered along with the MMC card removal. However,
looking at nvmem_unregister(), it simply performs a kref_put(). This
means that if a consumer still holds a reference, e.g. through an
nvmem_cell, it might still be able to access the NVMEM reg_read
callback after the NVMEM device has been unregistered and the
NVMEM provider data (bnv) has been freed...
I guess I need to try and dig into this but at first glance it seems to be a
general NVMEM issue, since even though most NVMEM providers are
not hard-removable, there is nothing that fundamentally prevents them
from being logically removed/unbound at runtime.
Regards,
Loic
^ permalink raw reply
* Re: [PATCH 1/1] cdrom, scsi: sr: propagate read-only status to block layer via set_disk_ro()
From: Phillip Potter @ 2026-04-29 19:53 UTC (permalink / raw)
To: Jens Axboe; +Cc: linux-block, daan
In-Reply-To: <8816ec1a-a4cb-4c68-a99a-79e9d59cc288@kernel.dk>
On Mon, Apr 27, 2026 at 03:52:12PM -0600, Jens Axboe wrote:
> On 4/27/26 3:01 PM, Phillip Potter wrote:
> > Fixes: 1da177e4c3f4 ("Linux-2.6.12-rc2")
> > Signed-off-by: Daan De Meyer <daan@amutable.com>
> > Link: https://lore.kernel.org/all/20260422113206.246267-1-daan@amutable.com
> > Reviewed-by: Phillip Potter <phil@philpotter.co.uk>
> > Link: https://lore.kernel.org/all/ae59luYMh6npxD09@equinox
> > Reviewed-by: Martin K. Petersen <martin.petersen@oracle.com>
> > Link: https://lore.kernel.org/all/yq1jytti447.fsf@ca-mkp.ca.oracle.com
> > Signed-off-by: Phillip Potter <phil@philpotter.co.uk>
>
> A few notes:
>
> - Please don't include a ton of Link tags in the commit message. Don't
> add anything unless it's a link to a bug report, and your patch fixes
> that bug.
>
> - Signed-off-by implies Reviewed-by, you don't need both of those.
>
> --
> Jens Axboe
Sorry about that, didn't realise it was bad as I've done it that way
previously without issue. Will make sure not to do it again :-)
Thanks for taking Daan's patch.
Regards,
Phil
^ permalink raw reply
* Re: [PATCH 7/9] Bluetooth: hci_sync: Add NVMEM-backed BD address retrieval
From: Loic Poulain @ 2026-04-29 16:22 UTC (permalink / raw)
To: Andrew Lunn
Cc: Bartosz Golaszewski, Ulf Hansson, Rob Herring,
Krzysztof Kozlowski, Conor Dooley, Bjorn Andersson, Konrad Dybcio,
Jens Axboe, Johannes Berg, Jeff Johnson, Marcel Holtmann,
Luiz Augusto von Dentz, Balakrishna Godavarthi, Rocky Liao,
David S. Miller, Eric Dumazet, Jakub Kicinski, Paolo Abeni,
Simon Horman, linux-mmc, devicetree, linux-kernel, linux-arm-msm,
linux-block, linux-wireless, ath10k, linux-bluetooth, netdev,
daniel
In-Reply-To: <e0adaf64-10d7-4e28-8403-769e3070ec30@lunn.ch>
Hi Bartosz, Andrew,
On Wed, Apr 29, 2026 at 3:15 PM Andrew Lunn <andrew@lunn.ch> wrote:
>
> On Wed, Apr 29, 2026 at 10:53:13AM +0200, Bartosz Golaszewski wrote:
> > On Tue, Apr 28, 2026 at 4:23 PM Loic Poulain
> > <loic.poulain@oss.qualcomm.com> wrote:
> > >
> > > Some devices store the Bluetooth BD address in non-volatile
> > > memory, which can be accessed through the NVMEM framework.
> > > Similar to Ethernet or WiFi MAC addresses, add support for
> > > reading the BD address from a 'local-bd-address' NVMEM cell.
> > >
> > > As with the device-tree provided BD address, add a quirk to
> > > indicate whether a device or platform should attempt to read
> > > the address from NVMEM when no valid in-chip address is present.
> > > Also add a quirk to indicate if the address is stored in
> > > big-endian byte order.
> > >
> > > Signed-off-by: Loic Poulain <loic.poulain@oss.qualcomm.com>
> > > ---
> >
> > Is there any reason why we can't extend the existing
> > of_get_mac_address() with another property name and use it here? It
> > already has support for mac addresses from nvmem.
>
> Does it even need to be a different property name? Is a bluetooth MAC
> address somehow different to an Ethernet MAC address? Isn't it just a
> EUI-48, independent of it being Ethernet, Bluetooth, wifi, fddi, token
> ring, homing pigeon?
In terms of naming, I followed the pattern of the existing DT
properties: for Ethernet/WiFi there is mac-address property, and for
Bluetooth a local-bd-address property. So it seems reasonable to reuse
the same naming for the NVMEM labelling. Also One thing to keep in
mind is that Bluetooth has historically handled addresses in
little-endian format, while Ethernet MAC addresses are typically
stored in big-endian format (as far as I know).
That said, a more generic EUI-48 identifier could also make sense if
we want to go in that direction, with expected property/nvmem-cell
name(s) as parameters?
Regards,
Loic
^ permalink raw reply
* Re: [PATCH v3 07/10] nvme-pci: implement dma_token backed requests
From: Maurizio Lombardi @ 2026-04-29 16:07 UTC (permalink / raw)
To: Pavel Begunkov, Jens Axboe, Keith Busch, Christoph Hellwig,
Sagi Grimberg, Alexander Viro, Christian Brauner, Andrew Morton,
Sumit Semwal, Christian König, linux-block, linux-kernel,
linux-nvme, linux-fsdevel, io-uring, linux-media, dri-devel,
linaro-mm-sig
Cc: Nitesh Shetty, Kanchan Joshi, Anuj Gupta, Tushar Gohad,
William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <5cecb1157ab784f9f303a91449fdf11b03aa6002.1777475843.git.asml.silence@gmail.com>
On Wed Apr 29, 2026 at 5:25 PM CEST, Pavel Begunkov wrote:
> Enable BIO_DMABUF_MAP backed requests. It creates a prp list for the
> dmabuf when it's mapped, which is then used to initialise requests.
>
> Suggested-by: Keith Busch <kbusch@kernel.org>
> Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
> ---
> drivers/nvme/host/pci.c | 282 ++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 282 insertions(+)
>
> diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
> index db5fc9bf6627..d2629853a972 100644
> --- a/drivers/nvme/host/pci.c
> +++ b/drivers/nvme/host/pci.c
> @@ -27,6 +27,8 @@
> #include <linux/io-64-nonatomic-lo-hi.h>
> #include <linux/io-64-nonatomic-hi-lo.h>
> #include <linux/sed-opal.h>
> +#include <linux/io_dmabuf_token.h>
> +#include <linux/dma-resv.h>
>
> #include "trace.h"
> #include "nvme.h"
> @@ -393,6 +395,17 @@ struct nvme_queue {
> struct completion delete_done;
> };
>
> +struct nvme_dmabuf_token {
> + struct dma_buf_attachment *attach;
> +};
> +
> +struct nvme_dmabuf_map {
> + struct io_dmabuf_map base;
> + dma_addr_t *dma_list;
> + struct sg_table *sgt;
> + unsigned nr_entries;
> +};
> +
> /* bits for iod->flags */
> enum nvme_iod_flags {
> /* this command has been aborted by the timeout handler */
> @@ -854,6 +867,134 @@ static void nvme_free_descriptors(struct request *req)
> }
> }
>
> +static void nvme_dmabuf_map_sync(struct nvme_dev *nvme_dev, struct request *req,
> + bool for_cpu)
> +{
> + int length = blk_rq_payload_bytes(req);
> + struct device *dev = nvme_dev->dev;
> + enum dma_data_direction dma_dir;
> + struct bio *bio = req->bio;
> + struct nvme_dmabuf_map *map;
> + dma_addr_t *dma_list;
> + int offset, map_idx;
> +
> + dma_dir = rq_data_dir(req) == READ ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
> + map = container_of(bio->dmabuf_map, struct nvme_dmabuf_map, base);
> + dma_list = map->dma_list;
> +
> + offset = bio->bi_iter.bi_bvec_done;
> + map_idx = offset / NVME_CTRL_PAGE_SIZE;
> + length += offset & (NVME_CTRL_PAGE_SIZE - 1);
> +
> + while (length > 0) {
> + u64 dma_addr = dma_list[map_idx++];
> +
> + if (for_cpu)
> + __dma_sync_single_for_cpu(dev, dma_addr,
> + NVME_CTRL_PAGE_SIZE, dma_dir);
> + else
> + __dma_sync_single_for_device(dev, dma_addr,
> + NVME_CTRL_PAGE_SIZE,
> + dma_dir);
> + length -= NVME_CTRL_PAGE_SIZE;
> + }
> +}
> +
> +static void nvme_rq_clean_dmabuf_map(struct nvme_dev *dev,
> + struct request *req)
> +{
> + struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> +
> + nvme_dmabuf_map_sync(dev, req, true);
> +
> + if (!(iod->flags & IOD_SINGLE_SEGMENT))
> + nvme_free_descriptors(req);
> +}
> +
> +static blk_status_t nvme_rq_setup_dmabuf_map(struct request *req,
> + struct nvme_queue *nvmeq)
> +{
> + struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> + int length = blk_rq_payload_bytes(req);
> + u64 dma_addr, prp1_dma, prp2_dma;
> + struct bio *bio = req->bio;
> + struct nvme_dmabuf_map *map;
> + dma_addr_t *dma_list;
> + dma_addr_t prp_dma;
> + __le64 *prp_list;
> + int i, map_idx;
> + int offset;
> +
> + nvme_dmabuf_map_sync(nvmeq->dev, req, false);
> +
> + map = container_of(bio->dmabuf_map, struct nvme_dmabuf_map, base);
> + dma_list = map->dma_list;
> +
> + offset = bio->bi_iter.bi_bvec_done;
> + map_idx = offset / NVME_CTRL_PAGE_SIZE;
> + offset &= (NVME_CTRL_PAGE_SIZE - 1);
> + prp1_dma = dma_list[map_idx++] + offset;
> +
> + length -= (NVME_CTRL_PAGE_SIZE - offset);
> + if (length <= 0) {
> + prp2_dma = 0;
> + goto done;
> + }
> +
> + if (length <= NVME_CTRL_PAGE_SIZE) {
> + prp2_dma = dma_list[map_idx];
> + goto done;
> + }
> +
> + if (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) <=
> + NVME_SMALL_POOL_SIZE / sizeof(__le64))
> + iod->flags |= IOD_SMALL_DESCRIPTOR;
> +
> + prp_list = dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC,
> + &prp_dma);
> + if (!prp_list)
> + return BLK_STS_RESOURCE;
> +
> + iod->descriptors[iod->nr_descriptors++] = prp_list;
> + prp2_dma = prp_dma;
> + i = 0;
> + for (;;) {
> + if (i == NVME_CTRL_PAGE_SIZE >> 3) {
> + __le64 *old_prp_list = prp_list;
> +
> + prp_list = dma_pool_alloc(nvmeq->descriptor_pools.large,
> + GFP_ATOMIC, &prp_dma);
> + if (!prp_list)
> + goto free_prps;
> + iod->descriptors[iod->nr_descriptors++] = prp_list;
> + prp_list[0] = old_prp_list[i - 1];
> + old_prp_list[i - 1] = cpu_to_le64(prp_dma);
> + i = 1;
> + }
> +
> + dma_addr = dma_list[map_idx++];
> + prp_list[i++] = cpu_to_le64(dma_addr);
> +
> + length -= NVME_CTRL_PAGE_SIZE;
> + if (length <= 0)
> + break;
> + }
> +done:
> + iod->cmd.common.dptr.prp1 = cpu_to_le64(prp1_dma);
> + iod->cmd.common.dptr.prp2 = cpu_to_le64(prp2_dma);
> + return BLK_STS_OK;
> +free_prps:
> + nvme_free_descriptors(req);
> + return BLK_STS_RESOURCE;
> +}
> +
> +static inline bool nvme_rq_is_dmabuf_attached(struct request *req)
> +{
> + if (!IS_ENABLED(CONFIG_DMABUF_TOKEN))
> + return false;
> + return req->bio && bio_flagged(req->bio, BIO_DMABUF_MAP);
> +}
> +
> static void nvme_free_prps(struct request *req, unsigned int attrs)
> {
> struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
> @@ -932,6 +1073,11 @@ static void nvme_unmap_data(struct request *req)
> struct device *dma_dev = nvmeq->dev->dev;
> unsigned int attrs = 0;
>
> + if (nvme_rq_is_dmabuf_attached(req)) {
> + nvme_rq_clean_dmabuf_map(nvmeq->dev, req);
> + return;
> + }
> +
> if (iod->flags & IOD_SINGLE_SEGMENT) {
> static_assert(offsetof(union nvme_data_ptr, prp1) ==
> offsetof(union nvme_data_ptr, sgl.addr));
> @@ -1222,6 +1368,9 @@ static blk_status_t nvme_map_data(struct request *req)
> struct blk_dma_iter iter;
> blk_status_t ret;
>
> + if (nvme_rq_is_dmabuf_attached(req))
> + return nvme_rq_setup_dmabuf_map(req, nvmeq);
> +
> /*
> * Try to skip the DMA iterator for single segment requests, as that
> * significantly improves performances for small I/O sizes.
> @@ -2238,6 +2387,134 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
> return result;
> }
>
> +#ifdef CONFIG_DMABUF_TOKEN
> +static void nvme_dmabuf_invalidate_mappings(struct dma_buf_attachment *attach)
> +{
> + struct io_dmabuf_token *token = attach->importer_priv;
> +
> + io_dmabuf_token_invalidate_mappings(token);
> +}
> +
> +const struct dma_buf_attach_ops nvme_dmabuf_importer_ops = {
> + .invalidate_mappings = nvme_dmabuf_invalidate_mappings,
> + .allow_peer2peer = true,
> +};
> +
> +static struct io_dmabuf_map *nvme_dmabuf_token_map(struct io_dmabuf_token *token)
> +{
> + struct nvme_dmabuf_token *data = token->dev_priv;
> + struct dma_buf_attachment *attach = data->attach;
> + dma_addr_t *dma_list = NULL;
> + unsigned long tmp, i = 0;
> + struct nvme_dmabuf_map *map;
> + struct scatterlist *sg;
> + struct sg_table *sgt;
> + unsigned nr_entries;
> + int ret;
> +
> + dma_resv_assert_held(token->dmabuf->resv);
> +
> + map = kmalloc(sizeof(*map), GFP_KERNEL);
> + if (!map)
> + return ERR_PTR(-ENOMEM);
> +
> + nr_entries = token->dmabuf->size / NVME_CTRL_PAGE_SIZE;
> + dma_list = kmalloc_array(nr_entries, sizeof(dma_list[0]), GFP_KERNEL);
> + if (!dma_list) {
> + ret = -ENOMEM;
> + goto err;
> + }
> +
> + sgt = dma_buf_map_attachment(attach, token->dir);
> + if (IS_ERR(sgt)) {
> + ret = PTR_ERR(sgt);
> + sgt = NULL;
> + goto err;
> + }
> +
> + for_each_sgtable_dma_sg(sgt, sg, tmp) {
> + dma_addr_t dma_addr = sg_dma_address(sg);
> + unsigned long sg_len = sg_dma_len(sg);
> +
> + if (sg_len % NVME_CTRL_PAGE_SIZE) {
> + ret = -EINVAL;
> + goto err;
> + }
> +
> + while (sg_len) {
> + dma_list[i++] = dma_addr;
> + dma_addr += NVME_CTRL_PAGE_SIZE;
> + sg_len -= NVME_CTRL_PAGE_SIZE;
> + }
> + }
> +
> + ret = io_dmabuf_init_map(token, &map->base);
> + if (ret)
> + goto err;
> + map->nr_entries = nr_entries;
> + map->dma_list = dma_list;
> + map->sgt = sgt;
> + return &map->base;
> +err:
> + if (sgt)
> + dma_buf_unmap_attachment(attach, sgt, token->dir);
> + kfree(map);
> + kfree(dma_list);
> + return ERR_PTR(ret);
> +}
> +
> +static void nvme_dmabuf_token_unmap(struct io_dmabuf_token *token,
> + struct io_dmabuf_map *map_base)
> +{
> + struct nvme_dmabuf_token *data = token->dev_priv;
> + struct nvme_dmabuf_map *map = container_of(map_base,
> + struct nvme_dmabuf_map, base);
> +
> + dma_resv_assert_held(token->dmabuf->resv);
> +
> + dma_buf_unmap_attachment(data->attach, map->sgt, token->dir);
> + kfree(map->dma_list);
> +}
> +
> +static void nvme_dmabuf_token_release(struct io_dmabuf_token *token)
> +{
> + struct nvme_dmabuf_token *data = token->dev_priv;
> +
> + dma_buf_detach(token->dmabuf, data->attach);
> + kfree(data);
> +}
> +
> +const struct io_dmabuf_token_dev_ops nvme_dma_token_ops = {
> + .map = nvme_dmabuf_token_map,
> + .unmap = nvme_dmabuf_token_unmap,
> + .release = nvme_dmabuf_token_release,
> +};
> +
> +static int nvme_create_dmabuf_token(struct request_queue *q,
> + struct io_dmabuf_token *token)
> +{
> + struct nvme_dmabuf_token *data;
> + struct dma_buf_attachment *attach;
> + struct nvme_ns *ns = q->queuedata;
> + struct nvme_dev *dev = to_nvme_dev(ns->ctrl);
> + struct dma_buf *dmabuf = token->dmabuf;
> +
> + data = kzalloc(sizeof(data), GFP_KERNEL);
> + if (!data)
> + return -ENOMEM;
Shouldn't this be kzalloc(sizeof(*data)...) ?
Also, checkpatch generates a warning because kzalloc_obj() should be
preferred for this kind of memory allocations over kzalloc().
> +
> + token->dev_priv = data;
> + token->dev_ops = &nvme_dma_token_ops;
> +
> + attach = dma_buf_dynamic_attach(dmabuf, dev->dev,
> + &nvme_dmabuf_importer_ops, token);
> + if (IS_ERR(attach))
> + return PTR_ERR(attach);
Supposing dma_buf_dynamic_attach() returns an error, won't the 'data'
pointer be leaked?
Maurizio
^ permalink raw reply
* Re: [PATCH v2 2/3] dm-inlinecrypt: add target for inline block device encryption
From: Benjamin Marzinski @ 2026-04-29 15:58 UTC (permalink / raw)
To: Linlin Zhang
Cc: linux-block, ebiggers, mpatocka, gmazyland, linux-kernel,
adrianvovk, dm-devel, quic_mdalam, israelr, hch, axboe
In-Reply-To: <afIizQA9kVR_u_2j@redhat.com>
On Wed, Apr 29, 2026 at 11:25:04AM -0400, Benjamin Marzinski wrote:
> On Wed, Apr 29, 2026 at 08:34:00PM +0800, Linlin Zhang wrote:
[...]
> >
> > Thanks for the suggestions.
> >
> > Adding a bool need_acct parameter to __blk_crypto_submit_bio() would require
> > updating all existing callers, which feels rather intrusive given that the
> > accounting issue only affects the blk‑crypto fallback write slow‑path. I’m a
> > bit concerned that this would broaden the scope of the change more than
> > necessary for the problem at hand.
>
> I get your concern, and I'd like a second opinion on how much we should
> care about this, but it doesn't look like there are many other callers
> that would be effected here. The only existing caller of
> __blk_crypto_submit_bio() is blk_crypto_submit_bio(), which would just
> call it with "need_acct=true". Looking at the code path below
> __blk_crypto_submit_bio() that would need to change for submitting the
> bios:
>
> __blk_crypto_submit_bio() is the only caller of
> blk_crypto_fallback_bio_prep()
>
> blk_crypto_fallback_bio_prep() is the only caller of
> blk_crypto_fallback_encrypt_bio().
>
> blk_crypto_fallback_encrypt_bio() is the only caller of
> __blk_crypto_fallback_encrypt_bio(), which is the function that would
> need to choose between submit_bio() and submit_bio_noacct().
>
> Doing this would change the crypto API (by necessity, since we're adding
> a new argument to __blk_crypto_submit_bio() for stacking devices to
> use), and it is adds a extra argument to a number of functions, just to
> handle this corner case. But it is still a relatively contained change.
Having discussed this a bit, I'm fine with leaving this as a TODO for
now. If anyone wants to chime in with an opinion on how acceptable it
would be to add a new bio flag for skipping accounting, that would be
great.
-Ben
^ permalink raw reply
* Re: [PATCH v3 07/10] nvme-pci: implement dma_token backed requests
From: Pavel Begunkov @ 2026-04-29 15:29 UTC (permalink / raw)
To: Jens Axboe, Keith Busch, Christoph Hellwig, Sagi Grimberg,
Alexander Viro, Christian Brauner, Andrew Morton, Sumit Semwal,
Christian König, linux-block, linux-kernel, linux-nvme,
linux-fsdevel, io-uring, linux-media, dri-devel, linaro-mm-sig
Cc: Nitesh Shetty, Kanchan Joshi, Anuj Gupta, Tushar Gohad,
William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <5cecb1157ab784f9f303a91449fdf11b03aa6002.1777475843.git.asml.silence@gmail.com>
On 4/29/26 16:25, Pavel Begunkov wrote:
> Enable BIO_DMABUF_MAP backed requests. It creates a prp list for the
> dmabuf when it's mapped, which is then used to initialise requests.
I left nvme request / map setup as it was in Keith's work for the most
part (apart from rebases, adapting it, etc.). It appears I have some
use for prp lists, and I know Kanchan, Nitesh and Anuj already have some
patches adding sgl support and some other optimisations. Hopefully, I
addressed most all feedback from v2.
--
Pavel Begunkov
^ permalink raw reply
* [PATCH v3 10/10] io_uring/rsrc: add dmabuf backed registered buffers
From: Pavel Begunkov @ 2026-04-29 15:25 UTC (permalink / raw)
To: Jens Axboe, Keith Busch, Christoph Hellwig, Sagi Grimberg,
Alexander Viro, Christian Brauner, Andrew Morton, Sumit Semwal,
Christian König, linux-block, linux-kernel, linux-nvme,
linux-fsdevel, io-uring, linux-media, dri-devel, linaro-mm-sig
Cc: asml.silence, Nitesh Shetty, Kanchan Joshi, Anuj Gupta,
Tushar Gohad, William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <cover.1777475843.git.asml.silence@gmail.com>
Implement dmabuf backed registered buffers. To register them, the user
should specify IO_REGBUF_TYPE_DMABUF for the regitration and pass the
desired dmabuf fd and a file for which it should be registered.
From there, it can be used with io_uring read/write requests
IORING_OP_{READ,WRITE}_FIXED) as normal. The requests should be issued
against the file specified during registration, and otherwise they'll be
failed. The user should also be prepared to handle spurious -EAGAIN by
reissuing the request.
Internally, dmabuf registered buffers is an optin feature for io_uring
request opcodes and they should pass a special flag on import to use it.
Suggested-by: David Wei <dw@davidwei.uk>
Suggested-by: Vishal Verma <vishal1.verma@intel.com>
Suggested-by: Tushar Gohad <tushar.gohad@intel.com>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
include/linux/io_uring_types.h | 5 +
include/uapi/linux/io_uring.h | 6 +-
io_uring/io_uring.c | 3 +-
io_uring/rsrc.c | 163 +++++++++++++++++++++++++++++++--
io_uring/rsrc.h | 30 +++++-
io_uring/rw.c | 4 +-
6 files changed, 200 insertions(+), 11 deletions(-)
diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h
index 7aee83e5ea0e..f9a33099421a 100644
--- a/include/linux/io_uring_types.h
+++ b/include/linux/io_uring_types.h
@@ -10,6 +10,7 @@
struct iou_loop_params;
struct io_uring_bpf_ops;
+struct io_dmabuf_map;
enum {
/*
@@ -567,6 +568,7 @@ enum {
REQ_F_IMPORT_BUFFER_BIT,
REQ_F_SQE_COPIED_BIT,
REQ_F_IOPOLL_BIT,
+ REQ_F_DROP_DMABUF_BIT,
/* not a real bit, just to check we're not overflowing the space */
__REQ_F_LAST_BIT,
@@ -662,6 +664,8 @@ enum {
REQ_F_SQE_COPIED = IO_REQ_FLAG(REQ_F_SQE_COPIED_BIT),
/* request must be iopolled to completion (set in ->issue()) */
REQ_F_IOPOLL = IO_REQ_FLAG(REQ_F_IOPOLL_BIT),
+ /* there is a dma map attached to request that needs to be dropped */
+ REQ_F_DROP_DMABUF = IO_REQ_FLAG(REQ_F_DROP_DMABUF_BIT),
};
struct io_tw_req {
@@ -786,6 +790,7 @@ struct io_kiocb {
/* custom credentials, valid IFF REQ_F_CREDS is set */
const struct cred *creds;
struct io_wq_work work;
+ struct io_dmabuf_map *dmabuf_map;
struct io_big_cqe {
u64 extra1;
diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h
index 05c3fd078767..3cd6ce28f9f5 100644
--- a/include/uapi/linux/io_uring.h
+++ b/include/uapi/linux/io_uring.h
@@ -810,6 +810,7 @@ enum io_uring_rsrc_reg_flags {
enum io_uring_regbuf_type {
IO_REGBUF_TYPE_EMPTY,
IO_REGBUF_TYPE_UADDR,
+ IO_REGBUF_TYPE_DMABUF,
__IO_REGBUF_TYPE_MAX,
};
@@ -819,7 +820,10 @@ struct io_uring_regbuf_desc {
__u32 flags;
__u64 size;
__u64 uaddr;
- __u64 __resv[7];
+
+ __s32 dmabuf_fd;
+ __s32 target_fd;
+ __u64 __resv[6];
};
/* Skip updating fd indexes set to this value in the fd table */
diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c
index 6068448a5aaa..e8a8eef45c3f 100644
--- a/io_uring/io_uring.c
+++ b/io_uring/io_uring.c
@@ -108,7 +108,7 @@
#define IO_REQ_CLEAN_SLOW_FLAGS (REQ_F_REFCOUNT | IO_REQ_LINK_FLAGS | \
REQ_F_REISSUE | REQ_F_POLLED | \
- IO_REQ_CLEAN_FLAGS)
+ IO_REQ_CLEAN_FLAGS | REQ_F_DROP_DMABUF)
#define IO_TCTX_REFS_CACHE_NR (1U << 10)
@@ -1115,6 +1115,7 @@ static void io_free_batch_list(struct io_ring_ctx *ctx,
io_queue_next(req);
if (unlikely(req->flags & IO_REQ_CLEAN_FLAGS))
io_clean_op(req);
+ io_req_drop_dmabuf(req);
}
io_put_file(req);
io_req_put_rsrc_nodes(req);
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index f8696b01cb54..bb61de308543 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -10,6 +10,7 @@
#include <linux/compat.h>
#include <linux/io_uring.h>
#include <linux/io_uring/cmd.h>
+#include <linux/io_dmabuf_token.h>
#include <uapi/linux/io_uring.h>
@@ -789,6 +790,93 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages,
return true;
}
+struct io_regbuf_dma {
+ struct io_dmabuf_token token;
+ struct file *target_file;
+};
+
+static void io_release_reg_dmabuf(void *priv)
+{
+ struct io_regbuf_dma *db = priv;
+
+ fput(db->target_file);
+ io_dmabuf_token_release(&db->token);
+}
+
+static struct io_rsrc_node *io_register_dmabuf(struct io_ring_ctx *ctx,
+ struct io_uring_regbuf_desc *desc)
+{
+ struct io_rsrc_node *node = NULL;
+ struct io_mapped_ubuf *imu = NULL;
+ struct io_regbuf_dma *regbuf = NULL;
+ struct file *target_file = NULL;
+ struct dma_buf *dmabuf = NULL;
+ int ret;
+
+ if (!IS_ENABLED(CONFIG_DMABUF_TOKEN))
+ return ERR_PTR(-EOPNOTSUPP);
+ if (desc->uaddr || desc->size)
+ return ERR_PTR(-EINVAL);
+
+ ret = -ENOMEM;
+ node = io_rsrc_node_alloc(ctx, IORING_RSRC_BUFFER);
+ if (!node)
+ return ERR_PTR(-ENOMEM);
+ imu = io_alloc_imu(ctx, 0);
+ if (!imu)
+ goto err;
+ regbuf = kzalloc(sizeof(*regbuf), GFP_KERNEL);
+ if (!regbuf)
+ goto err;
+
+ ret = -EBADF;
+ target_file = fget(desc->target_fd);
+ if (!target_file)
+ goto err;
+
+ dmabuf = dma_buf_get(desc->dmabuf_fd);
+ if (IS_ERR(dmabuf)) {
+ ret = PTR_ERR(dmabuf);
+ dmabuf = NULL;
+ goto err;
+ }
+ if (dmabuf->size > SZ_1G) {
+ ret = -EINVAL;
+ goto err;
+ }
+
+ ret = io_dmabuf_token_create(target_file, ®buf->token, dmabuf,
+ DMA_BIDIRECTIONAL);
+ if (ret)
+ goto err;
+
+ regbuf->target_file = target_file;
+ imu->nr_bvecs = 1;
+ imu->ubuf = 0;
+ imu->len = dmabuf->size;
+ imu->folio_shift = 0;
+ imu->release = io_release_reg_dmabuf;
+ imu->priv = regbuf;
+ imu->flags = IO_REGBUF_F_DMABUF;
+ imu->dir = IO_BUF_DEST | IO_BUF_SOURCE;
+ refcount_set(&imu->refs, 1);
+ node->buf = imu;
+ dma_buf_put(dmabuf);
+ return node;
+err:
+ kfree(regbuf);
+ if (imu)
+ io_free_imu(ctx, imu);
+ if (node)
+ io_cache_free(&ctx->node_cache, node);
+ if (target_file)
+ fput(target_file);
+ if (dmabuf)
+ dma_buf_put(dmabuf);
+ return ERR_PTR(ret);
+}
+
+
static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
struct io_uring_regbuf_desc *desc,
struct page **last_hpage)
@@ -808,6 +896,12 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
if (!mem_is_zero(&desc->__resv, sizeof(desc->__resv)))
return ERR_PTR(-EINVAL);
+ if (desc->type == IO_REGBUF_TYPE_DMABUF)
+ return io_register_dmabuf(ctx, desc);
+
+ if (desc->dmabuf_fd || desc->target_fd)
+ return ERR_PTR(-EINVAL);
+
if (desc->type == IO_REGBUF_TYPE_EMPTY) {
if (uaddr || size)
return ERR_PTR(-EFAULT);
@@ -1134,9 +1228,57 @@ static int io_import_kbuf(int ddir, struct iov_iter *iter,
return 0;
}
-static int io_import_fixed(int ddir, struct iov_iter *iter,
+void io_drop_dmabuf_node(struct io_kiocb *req)
+{
+ struct io_mapped_ubuf *imu;
+
+ if (!IS_ENABLED(CONFIG_DMABUF_TOKEN))
+ return;
+ if (WARN_ON_ONCE(req->buf_node->type != IORING_RSRC_BUFFER))
+ return;
+ imu = req->buf_node->buf;
+ if (WARN_ON_ONCE(!(imu->flags & IO_REGBUF_F_DMABUF)))
+ return;
+ io_dmabuf_map_drop(req->dmabuf_map);
+}
+
+static int io_import_dmabuf(struct io_kiocb *req,
+ int ddir, struct iov_iter *iter,
struct io_mapped_ubuf *imu,
- u64 buf_addr, size_t len)
+ size_t len, size_t offset,
+ unsigned issue_flags)
+{
+ struct io_regbuf_dma *db = imu->priv;
+ struct io_dmabuf_map *map;
+
+ if (!IS_ENABLED(CONFIG_DMABUF_TOKEN))
+ return -EOPNOTSUPP;
+ if (!len)
+ return -EFAULT;
+ if (req->file != db->target_file)
+ return -EBADF;
+
+ map = io_dmabuf_get_map(&db->token);
+ if (unlikely(!map)) {
+ if (!(issue_flags & IO_URING_F_UNLOCKED))
+ return -EAGAIN;
+ map = io_dmabuf_create_map(&db->token);
+ if (IS_ERR(map))
+ return PTR_ERR(map);
+ }
+
+ req->dmabuf_map = map;
+ req->flags |= REQ_F_DROP_DMABUF;
+ iov_iter_dmabuf_map(iter, ddir, map, offset, len);
+ return 0;
+}
+
+static int io_import_fixed(struct io_kiocb *req,
+ int ddir, struct iov_iter *iter,
+ struct io_mapped_ubuf *imu,
+ u64 buf_addr, size_t len,
+ unsigned issue_flags,
+ unsigned import_flags)
{
const struct bio_vec *bvec;
size_t folio_mask;
@@ -1156,6 +1298,12 @@ static int io_import_fixed(int ddir, struct iov_iter *iter,
offset = buf_addr - imu->ubuf;
+ if (imu->flags & IO_REGBUF_F_DMABUF) {
+ if (!(import_flags & IO_REGBUF_IMPORT_ALLOW_DMABUF))
+ return -EFAULT;
+ return io_import_dmabuf(req, ddir, iter, imu, len, offset,
+ issue_flags);
+ }
if (imu->flags & IO_REGBUF_F_KBUF)
return io_import_kbuf(ddir, iter, imu, len, offset);
@@ -1209,16 +1357,17 @@ inline struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,
return NULL;
}
-int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
+int __io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
u64 buf_addr, size_t len, int ddir,
- unsigned issue_flags)
+ unsigned issue_flags, unsigned import_flags)
{
struct io_rsrc_node *node;
node = io_find_buf_node(req, issue_flags);
if (!node)
return -EFAULT;
- return io_import_fixed(ddir, iter, node->buf, buf_addr, len);
+ return io_import_fixed(req, ddir, iter, node->buf, buf_addr, len,
+ issue_flags, import_flags);
}
/* Lock two rings at once. The rings must be different! */
@@ -1577,7 +1726,9 @@ int io_import_reg_vec(int ddir, struct iov_iter *iter,
iovec_off = vec->nr - nr_iovs;
iov = vec->iovec + iovec_off;
- if (imu->flags & IO_REGBUF_F_KBUF) {
+ if (imu->flags & IO_REGBUF_F_DMABUF) {
+ return -EOPNOTSUPP;
+ } else if (imu->flags & IO_REGBUF_F_KBUF) {
int ret = io_kern_bvec_size(iov, nr_iovs, imu, &nr_segs);
if (unlikely(ret))
diff --git a/io_uring/rsrc.h b/io_uring/rsrc.h
index 8d48195faf9d..005a273ba107 100644
--- a/io_uring/rsrc.h
+++ b/io_uring/rsrc.h
@@ -25,6 +25,11 @@ struct io_rsrc_node {
enum {
IO_REGBUF_F_KBUF = 1,
+ IO_REGBUF_F_DMABUF = 2,
+};
+
+enum {
+ IO_REGBUF_IMPORT_ALLOW_DMABUF = 1,
};
struct io_mapped_ubuf {
@@ -60,9 +65,19 @@ int io_rsrc_data_alloc(struct io_rsrc_data *data, unsigned nr);
struct io_rsrc_node *io_find_buf_node(struct io_kiocb *req,
unsigned issue_flags);
+int __io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
+ u64 buf_addr, size_t len, int ddir,
+ unsigned issue_flags, unsigned import_flags);
+
+static inline
int io_import_reg_buf(struct io_kiocb *req, struct iov_iter *iter,
u64 buf_addr, size_t len, int ddir,
- unsigned issue_flags);
+ unsigned issue_flags)
+{
+ return __io_import_reg_buf(req, iter, buf_addr, len, ddir,
+ issue_flags, 0);
+}
+
int io_import_reg_vec(int ddir, struct iov_iter *iter,
struct io_kiocb *req, struct iou_vec *vec,
unsigned nr_iovs, unsigned issue_flags);
@@ -147,4 +162,17 @@ static inline void io_alloc_cache_vec_kasan(struct iou_vec *iv)
io_vec_free(iv);
}
+void io_drop_dmabuf_node(struct io_kiocb *req);
+
+static inline void io_req_drop_dmabuf(struct io_kiocb *req)
+{
+ if (!IS_ENABLED(CONFIG_DMABUF_TOKEN))
+ return;
+ if (!(req->flags & REQ_F_DROP_DMABUF))
+ return;
+ if (WARN_ON_ONCE(!(req->flags & REQ_F_BUF_NODE)))
+ return;
+ io_drop_dmabuf_node(req);
+}
+
#endif
diff --git a/io_uring/rw.c b/io_uring/rw.c
index 20654deff84d..d50da5fa8bb9 100644
--- a/io_uring/rw.c
+++ b/io_uring/rw.c
@@ -380,8 +380,8 @@ static int io_init_rw_fixed(struct io_kiocb *req, unsigned int issue_flags,
if (io->bytes_done)
return 0;
- ret = io_import_reg_buf(req, &io->iter, rw->addr, rw->len, ddir,
- issue_flags);
+ ret = __io_import_reg_buf(req, &io->iter, rw->addr, rw->len, ddir,
+ issue_flags, IO_REGBUF_IMPORT_ALLOW_DMABUF);
iov_iter_save_state(&io->iter, &io->iter_state);
return ret;
}
--
2.53.0
^ permalink raw reply related
* [PATCH v3 09/10] io_uring/rsrc: extend buffer update
From: Pavel Begunkov @ 2026-04-29 15:25 UTC (permalink / raw)
To: Jens Axboe, Keith Busch, Christoph Hellwig, Sagi Grimberg,
Alexander Viro, Christian Brauner, Andrew Morton, Sumit Semwal,
Christian König, linux-block, linux-kernel, linux-nvme,
linux-fsdevel, io-uring, linux-media, dri-devel, linaro-mm-sig
Cc: asml.silence, Nitesh Shetty, Kanchan Joshi, Anuj Gupta,
Tushar Gohad, William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <cover.1777475843.git.asml.silence@gmail.com>
We need to pass more information to buffer registration than we can fit
into a single struct iovec. This patch allows users to optionally pass
struct io_uring_regbuf_desc. Apart from having more space for future use
cases, it also introduces registration types.
Currently, the type can be either of IO_REGBUF_TYPE_UADDR, which mirrors
the iovec path, or IO_REGBUF_TYPE_EMPTY for leaving a buffer table slot
empty. The next patch introduces a dmabuf backed type, and can be useful
for other extensions like splicing a list of user addresses (i.e.
iovec[]), interoperability with zcrx, kernel allocated memory like was
brough up by Cristoph. Note, the type only represents a registration
option, which is distinct from how io_uring internally stores it.
The flags field is not used yet but always useful to have, e.g. we can
encode read-only / write-only restrictions using it.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
include/uapi/linux/io_uring.h | 27 +++++++++++++-
io_uring/rsrc.c | 69 ++++++++++++++++++++++-------------
2 files changed, 69 insertions(+), 27 deletions(-)
diff --git a/include/uapi/linux/io_uring.h b/include/uapi/linux/io_uring.h
index 17ac1b785440..05c3fd078767 100644
--- a/include/uapi/linux/io_uring.h
+++ b/include/uapi/linux/io_uring.h
@@ -790,13 +790,38 @@ struct io_uring_rsrc_update {
struct io_uring_rsrc_update2 {
__u32 offset;
- __u32 resv;
+ __u32 flags;
__aligned_u64 data;
__aligned_u64 tags;
__u32 nr;
__u32 resv2;
};
+/* struct io_uring_rsrc_update2::flags */
+enum io_uring_rsrc_reg_flags {
+ /*
+ * Use the extended descriptor format for buffer updates,
+ * see struct io_uring_regbuf_desc
+ */
+ IORING_RSRC_UPDATE_EXTENDED = 1U << 1,
+};
+
+/* Buffer registration type, passed in struct io_uring_regbuf_desc::type */
+enum io_uring_regbuf_type {
+ IO_REGBUF_TYPE_EMPTY,
+ IO_REGBUF_TYPE_UADDR,
+
+ __IO_REGBUF_TYPE_MAX,
+};
+
+struct io_uring_regbuf_desc {
+ __u32 type; /* enum io_uring_regbuf_type */
+ __u32 flags;
+ __u64 size;
+ __u64 uaddr;
+ __u64 __resv[7];
+};
+
/* Skip updating fd indexes set to this value in the fd table */
#define IORING_REGISTER_FILES_SKIP (-2)
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index ba00238941ed..f8696b01cb54 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -27,11 +27,6 @@ struct io_rsrc_update {
u32 offset;
};
-struct io_uring_regbuf_desc {
- __u64 uaddr;
- __u64 size;
-};
-
static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
struct io_uring_regbuf_desc *desc,
struct page **last_hpage);
@@ -46,9 +41,12 @@ static void io_iov_to_regbuf_desc(const struct iovec *iov,
struct io_uring_regbuf_desc *desc)
{
*desc = (struct io_uring_regbuf_desc) {
+ .type = IO_REGBUF_TYPE_UADDR,
.uaddr = (u64)iov->iov_base,
.size = iov->iov_len,
};
+ if (!desc->uaddr)
+ desc->type = IO_REGBUF_TYPE_EMPTY;
}
int __io_account_mem(struct user_struct *user, unsigned long nr_pages)
@@ -236,6 +234,8 @@ static int __io_sqe_files_update(struct io_ring_ctx *ctx,
return -ENXIO;
if (up->offset + nr_args > ctx->file_table.data.nr)
return -EINVAL;
+ if (up->flags)
+ return -EINVAL;
for (done = 0; done < nr_args; done++) {
u64 tag = 0;
@@ -292,10 +292,9 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
struct io_uring_rsrc_update2 *up,
unsigned int nr_args)
{
+ bool extended = up->flags & IORING_RSRC_UPDATE_EXTENDED;
u64 __user *tags = u64_to_user_ptr(up->tags);
- struct iovec fast_iov, *iov;
struct page *last_hpage = NULL;
- struct iovec __user *uvec;
u64 user_data = up->data;
__u32 done;
int i, err;
@@ -304,29 +303,49 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
return -ENXIO;
if (up->offset + nr_args > ctx->buf_table.nr)
return -EINVAL;
+ if (up->flags & ~IORING_RSRC_UPDATE_EXTENDED)
+ return -EINVAL;
for (done = 0; done < nr_args; done++) {
struct io_uring_regbuf_desc desc;
struct io_rsrc_node *node;
u64 tag = 0;
- uvec = u64_to_user_ptr(user_data);
- iov = iovec_from_user(uvec, 1, 1, &fast_iov, io_is_compat(ctx));
- if (IS_ERR(iov)) {
- err = PTR_ERR(iov);
- break;
- }
if (tags && copy_from_user(&tag, &tags[done], sizeof(tag))) {
err = -EFAULT;
break;
}
- io_iov_to_regbuf_desc(iov, &desc);
+ if (extended) {
+ if (copy_from_user(&desc, u64_to_user_ptr(user_data),
+ sizeof(desc))) {
+ err = -EFAULT;
+ break;
+ }
+ user_data += sizeof(desc);
+ } else {
+ struct iovec __user *uvec = u64_to_user_ptr(user_data);
+ struct iovec fast_iov, *iov;
+
+ if (io_is_compat(ctx))
+ user_data += sizeof(struct compat_iovec);
+ else
+ user_data += sizeof(struct iovec);
+
+ iov = iovec_from_user(uvec, 1, 1, &fast_iov, io_is_compat(ctx));
+ if (IS_ERR(iov)) {
+ err = PTR_ERR(iov);
+ break;
+ }
+ io_iov_to_regbuf_desc(iov, &desc);
+ }
+
node = io_sqe_buffer_register(ctx, &desc, &last_hpage);
if (IS_ERR(node)) {
err = PTR_ERR(node);
break;
}
+
if (tag) {
if (!node) {
err = -EINVAL;
@@ -337,10 +356,6 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
i = array_index_nospec(up->offset + done, ctx->buf_table.nr);
io_reset_rsrc_node(ctx, &ctx->buf_table, i);
ctx->buf_table.nodes[i] = node;
- if (io_is_compat(ctx))
- user_data += sizeof(struct compat_iovec);
- else
- user_data += sizeof(struct iovec);
}
return done ? done : err;
}
@@ -375,7 +390,7 @@ int io_register_files_update(struct io_ring_ctx *ctx, void __user *arg,
memset(&up, 0, sizeof(up));
if (copy_from_user(&up, arg, sizeof(struct io_uring_rsrc_update)))
return -EFAULT;
- if (up.resv || up.resv2)
+ if (up.resv2)
return -EINVAL;
return __io_register_rsrc_update(ctx, IORING_RSRC_FILE, &up, nr_args);
}
@@ -389,7 +404,7 @@ int io_register_rsrc_update(struct io_ring_ctx *ctx, void __user *arg,
return -EINVAL;
if (copy_from_user(&up, arg, sizeof(up)))
return -EFAULT;
- if (!up.nr || up.resv || up.resv2)
+ if (!up.nr || up.resv2)
return -EINVAL;
return __io_register_rsrc_update(ctx, type, &up, up.nr);
}
@@ -489,12 +504,9 @@ int io_files_update(struct io_kiocb *req, unsigned int issue_flags)
struct io_uring_rsrc_update2 up2;
int ret;
+ memset(&up2, 0, sizeof(up2));
up2.offset = up->offset;
up2.data = up->arg;
- up2.nr = 0;
- up2.tags = 0;
- up2.resv = 0;
- up2.resv2 = 0;
if (up->offset == IORING_FILE_INDEX_ALLOC) {
ret = io_files_update_with_index_alloc(req, issue_flags);
@@ -791,8 +803,13 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
struct io_imu_folio_data data;
bool coalesced = false;
- if (!uaddr) {
- if (size)
+ if (desc->type >= __IO_REGBUF_TYPE_MAX)
+ return ERR_PTR(-EINVAL);
+ if (!mem_is_zero(&desc->__resv, sizeof(desc->__resv)))
+ return ERR_PTR(-EINVAL);
+
+ if (desc->type == IO_REGBUF_TYPE_EMPTY) {
+ if (uaddr || size)
return ERR_PTR(-EFAULT);
/* remove the buffer without installing a new one */
return NULL;
--
2.53.0
^ permalink raw reply related
* [PATCH v3 08/10] io_uring/rsrc: introduce buf registration structure
From: Pavel Begunkov @ 2026-04-29 15:25 UTC (permalink / raw)
To: Jens Axboe, Keith Busch, Christoph Hellwig, Sagi Grimberg,
Alexander Viro, Christian Brauner, Andrew Morton, Sumit Semwal,
Christian König, linux-block, linux-kernel, linux-nvme,
linux-fsdevel, io-uring, linux-media, dri-devel, linaro-mm-sig
Cc: asml.silence, Nitesh Shetty, Kanchan Joshi, Anuj Gupta,
Tushar Gohad, William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <cover.1777475843.git.asml.silence@gmail.com>
In preparation to following changes, instead of passing an iovec for
buffer registration introduce a new structure. It'll be moved to uapi
later, but for now it's initialised early from a user provided iovec.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
io_uring/rsrc.c | 50 +++++++++++++++++++++++++++++++++----------------
1 file changed, 34 insertions(+), 16 deletions(-)
diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c
index c4a7a77d1ee9..ba00238941ed 100644
--- a/io_uring/rsrc.c
+++ b/io_uring/rsrc.c
@@ -27,8 +27,14 @@ struct io_rsrc_update {
u32 offset;
};
+struct io_uring_regbuf_desc {
+ __u64 uaddr;
+ __u64 size;
+};
+
static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
- struct iovec *iov, struct page **last_hpage);
+ struct io_uring_regbuf_desc *desc,
+ struct page **last_hpage);
/* only define max */
#define IORING_MAX_FIXED_FILES (1U << 20)
@@ -36,6 +42,15 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
#define IO_CACHED_BVECS_SEGS 32
+static void io_iov_to_regbuf_desc(const struct iovec *iov,
+ struct io_uring_regbuf_desc *desc)
+{
+ *desc = (struct io_uring_regbuf_desc) {
+ .uaddr = (u64)iov->iov_base,
+ .size = iov->iov_len,
+ };
+}
+
int __io_account_mem(struct user_struct *user, unsigned long nr_pages)
{
unsigned long page_limit, cur_pages, new_pages;
@@ -291,6 +306,7 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
return -EINVAL;
for (done = 0; done < nr_args; done++) {
+ struct io_uring_regbuf_desc desc;
struct io_rsrc_node *node;
u64 tag = 0;
@@ -304,7 +320,9 @@ static int __io_sqe_buffers_update(struct io_ring_ctx *ctx,
err = -EFAULT;
break;
}
- node = io_sqe_buffer_register(ctx, iov, &last_hpage);
+
+ io_iov_to_regbuf_desc(iov, &desc);
+ node = io_sqe_buffer_register(ctx, &desc, &last_hpage);
if (IS_ERR(node)) {
err = PTR_ERR(node);
break;
@@ -760,27 +778,27 @@ bool io_check_coalesce_buffer(struct page **page_array, int nr_pages,
}
static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
- struct iovec *iov,
- struct page **last_hpage)
+ struct io_uring_regbuf_desc *desc,
+ struct page **last_hpage)
{
+ unsigned long uaddr = (unsigned long)desc->uaddr;
+ size_t size = desc->size;
struct io_mapped_ubuf *imu = NULL;
struct page **pages = NULL;
struct io_rsrc_node *node;
unsigned long off;
- size_t size;
int ret, nr_pages, i;
struct io_imu_folio_data data;
bool coalesced = false;
- if (!iov->iov_base) {
- if (iov->iov_len)
+ if (!uaddr) {
+ if (size)
return ERR_PTR(-EFAULT);
/* remove the buffer without installing a new one */
return NULL;
}
- ret = io_validate_user_buf_range((unsigned long)iov->iov_base,
- iov->iov_len);
+ ret = io_validate_user_buf_range(uaddr, size);
if (ret)
return ERR_PTR(ret);
@@ -789,8 +807,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
return ERR_PTR(-ENOMEM);
ret = -ENOMEM;
- pages = io_pin_pages((unsigned long) iov->iov_base, iov->iov_len,
- &nr_pages);
+ pages = io_pin_pages(uaddr, size, &nr_pages);
if (IS_ERR(pages)) {
ret = PTR_ERR(pages);
pages = NULL;
@@ -812,10 +829,9 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
if (ret)
goto done;
- size = iov->iov_len;
/* store original address for later verification */
- imu->ubuf = (unsigned long) iov->iov_base;
- imu->len = iov->iov_len;
+ imu->ubuf = uaddr;
+ imu->len = size;
imu->folio_shift = PAGE_SHIFT;
imu->release = io_release_ubuf;
imu->priv = imu;
@@ -825,7 +841,7 @@ static struct io_rsrc_node *io_sqe_buffer_register(struct io_ring_ctx *ctx,
imu->folio_shift = data.folio_shift;
refcount_set(&imu->refs, 1);
- off = (unsigned long)iov->iov_base & ~PAGE_MASK;
+ off = uaddr & ~PAGE_MASK;
if (coalesced)
off += data.first_folio_page_idx << PAGE_SHIFT;
@@ -878,6 +894,7 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,
memset(iov, 0, sizeof(*iov));
for (i = 0; i < nr_args; i++) {
+ struct io_uring_regbuf_desc desc;
struct io_rsrc_node *node;
u64 tag = 0;
@@ -901,7 +918,8 @@ int io_sqe_buffers_register(struct io_ring_ctx *ctx, void __user *arg,
}
}
- node = io_sqe_buffer_register(ctx, iov, &last_hpage);
+ io_iov_to_regbuf_desc(iov, &desc);
+ node = io_sqe_buffer_register(ctx, &desc, &last_hpage);
if (IS_ERR(node)) {
ret = PTR_ERR(node);
break;
--
2.53.0
^ permalink raw reply related
* [PATCH v3 07/10] nvme-pci: implement dma_token backed requests
From: Pavel Begunkov @ 2026-04-29 15:25 UTC (permalink / raw)
To: Jens Axboe, Keith Busch, Christoph Hellwig, Sagi Grimberg,
Alexander Viro, Christian Brauner, Andrew Morton, Sumit Semwal,
Christian König, linux-block, linux-kernel, linux-nvme,
linux-fsdevel, io-uring, linux-media, dri-devel, linaro-mm-sig
Cc: asml.silence, Nitesh Shetty, Kanchan Joshi, Anuj Gupta,
Tushar Gohad, William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <cover.1777475843.git.asml.silence@gmail.com>
Enable BIO_DMABUF_MAP backed requests. It creates a prp list for the
dmabuf when it's mapped, which is then used to initialise requests.
Suggested-by: Keith Busch <kbusch@kernel.org>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
drivers/nvme/host/pci.c | 282 ++++++++++++++++++++++++++++++++++++++++
1 file changed, 282 insertions(+)
diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c
index db5fc9bf6627..d2629853a972 100644
--- a/drivers/nvme/host/pci.c
+++ b/drivers/nvme/host/pci.c
@@ -27,6 +27,8 @@
#include <linux/io-64-nonatomic-lo-hi.h>
#include <linux/io-64-nonatomic-hi-lo.h>
#include <linux/sed-opal.h>
+#include <linux/io_dmabuf_token.h>
+#include <linux/dma-resv.h>
#include "trace.h"
#include "nvme.h"
@@ -393,6 +395,17 @@ struct nvme_queue {
struct completion delete_done;
};
+struct nvme_dmabuf_token {
+ struct dma_buf_attachment *attach;
+};
+
+struct nvme_dmabuf_map {
+ struct io_dmabuf_map base;
+ dma_addr_t *dma_list;
+ struct sg_table *sgt;
+ unsigned nr_entries;
+};
+
/* bits for iod->flags */
enum nvme_iod_flags {
/* this command has been aborted by the timeout handler */
@@ -854,6 +867,134 @@ static void nvme_free_descriptors(struct request *req)
}
}
+static void nvme_dmabuf_map_sync(struct nvme_dev *nvme_dev, struct request *req,
+ bool for_cpu)
+{
+ int length = blk_rq_payload_bytes(req);
+ struct device *dev = nvme_dev->dev;
+ enum dma_data_direction dma_dir;
+ struct bio *bio = req->bio;
+ struct nvme_dmabuf_map *map;
+ dma_addr_t *dma_list;
+ int offset, map_idx;
+
+ dma_dir = rq_data_dir(req) == READ ? DMA_FROM_DEVICE : DMA_TO_DEVICE;
+ map = container_of(bio->dmabuf_map, struct nvme_dmabuf_map, base);
+ dma_list = map->dma_list;
+
+ offset = bio->bi_iter.bi_bvec_done;
+ map_idx = offset / NVME_CTRL_PAGE_SIZE;
+ length += offset & (NVME_CTRL_PAGE_SIZE - 1);
+
+ while (length > 0) {
+ u64 dma_addr = dma_list[map_idx++];
+
+ if (for_cpu)
+ __dma_sync_single_for_cpu(dev, dma_addr,
+ NVME_CTRL_PAGE_SIZE, dma_dir);
+ else
+ __dma_sync_single_for_device(dev, dma_addr,
+ NVME_CTRL_PAGE_SIZE,
+ dma_dir);
+ length -= NVME_CTRL_PAGE_SIZE;
+ }
+}
+
+static void nvme_rq_clean_dmabuf_map(struct nvme_dev *dev,
+ struct request *req)
+{
+ struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+
+ nvme_dmabuf_map_sync(dev, req, true);
+
+ if (!(iod->flags & IOD_SINGLE_SEGMENT))
+ nvme_free_descriptors(req);
+}
+
+static blk_status_t nvme_rq_setup_dmabuf_map(struct request *req,
+ struct nvme_queue *nvmeq)
+{
+ struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
+ int length = blk_rq_payload_bytes(req);
+ u64 dma_addr, prp1_dma, prp2_dma;
+ struct bio *bio = req->bio;
+ struct nvme_dmabuf_map *map;
+ dma_addr_t *dma_list;
+ dma_addr_t prp_dma;
+ __le64 *prp_list;
+ int i, map_idx;
+ int offset;
+
+ nvme_dmabuf_map_sync(nvmeq->dev, req, false);
+
+ map = container_of(bio->dmabuf_map, struct nvme_dmabuf_map, base);
+ dma_list = map->dma_list;
+
+ offset = bio->bi_iter.bi_bvec_done;
+ map_idx = offset / NVME_CTRL_PAGE_SIZE;
+ offset &= (NVME_CTRL_PAGE_SIZE - 1);
+ prp1_dma = dma_list[map_idx++] + offset;
+
+ length -= (NVME_CTRL_PAGE_SIZE - offset);
+ if (length <= 0) {
+ prp2_dma = 0;
+ goto done;
+ }
+
+ if (length <= NVME_CTRL_PAGE_SIZE) {
+ prp2_dma = dma_list[map_idx];
+ goto done;
+ }
+
+ if (DIV_ROUND_UP(length, NVME_CTRL_PAGE_SIZE) <=
+ NVME_SMALL_POOL_SIZE / sizeof(__le64))
+ iod->flags |= IOD_SMALL_DESCRIPTOR;
+
+ prp_list = dma_pool_alloc(nvme_dma_pool(nvmeq, iod), GFP_ATOMIC,
+ &prp_dma);
+ if (!prp_list)
+ return BLK_STS_RESOURCE;
+
+ iod->descriptors[iod->nr_descriptors++] = prp_list;
+ prp2_dma = prp_dma;
+ i = 0;
+ for (;;) {
+ if (i == NVME_CTRL_PAGE_SIZE >> 3) {
+ __le64 *old_prp_list = prp_list;
+
+ prp_list = dma_pool_alloc(nvmeq->descriptor_pools.large,
+ GFP_ATOMIC, &prp_dma);
+ if (!prp_list)
+ goto free_prps;
+ iod->descriptors[iod->nr_descriptors++] = prp_list;
+ prp_list[0] = old_prp_list[i - 1];
+ old_prp_list[i - 1] = cpu_to_le64(prp_dma);
+ i = 1;
+ }
+
+ dma_addr = dma_list[map_idx++];
+ prp_list[i++] = cpu_to_le64(dma_addr);
+
+ length -= NVME_CTRL_PAGE_SIZE;
+ if (length <= 0)
+ break;
+ }
+done:
+ iod->cmd.common.dptr.prp1 = cpu_to_le64(prp1_dma);
+ iod->cmd.common.dptr.prp2 = cpu_to_le64(prp2_dma);
+ return BLK_STS_OK;
+free_prps:
+ nvme_free_descriptors(req);
+ return BLK_STS_RESOURCE;
+}
+
+static inline bool nvme_rq_is_dmabuf_attached(struct request *req)
+{
+ if (!IS_ENABLED(CONFIG_DMABUF_TOKEN))
+ return false;
+ return req->bio && bio_flagged(req->bio, BIO_DMABUF_MAP);
+}
+
static void nvme_free_prps(struct request *req, unsigned int attrs)
{
struct nvme_iod *iod = blk_mq_rq_to_pdu(req);
@@ -932,6 +1073,11 @@ static void nvme_unmap_data(struct request *req)
struct device *dma_dev = nvmeq->dev->dev;
unsigned int attrs = 0;
+ if (nvme_rq_is_dmabuf_attached(req)) {
+ nvme_rq_clean_dmabuf_map(nvmeq->dev, req);
+ return;
+ }
+
if (iod->flags & IOD_SINGLE_SEGMENT) {
static_assert(offsetof(union nvme_data_ptr, prp1) ==
offsetof(union nvme_data_ptr, sgl.addr));
@@ -1222,6 +1368,9 @@ static blk_status_t nvme_map_data(struct request *req)
struct blk_dma_iter iter;
blk_status_t ret;
+ if (nvme_rq_is_dmabuf_attached(req))
+ return nvme_rq_setup_dmabuf_map(req, nvmeq);
+
/*
* Try to skip the DMA iterator for single segment requests, as that
* significantly improves performances for small I/O sizes.
@@ -2238,6 +2387,134 @@ static int nvme_create_queue(struct nvme_queue *nvmeq, int qid, bool polled)
return result;
}
+#ifdef CONFIG_DMABUF_TOKEN
+static void nvme_dmabuf_invalidate_mappings(struct dma_buf_attachment *attach)
+{
+ struct io_dmabuf_token *token = attach->importer_priv;
+
+ io_dmabuf_token_invalidate_mappings(token);
+}
+
+const struct dma_buf_attach_ops nvme_dmabuf_importer_ops = {
+ .invalidate_mappings = nvme_dmabuf_invalidate_mappings,
+ .allow_peer2peer = true,
+};
+
+static struct io_dmabuf_map *nvme_dmabuf_token_map(struct io_dmabuf_token *token)
+{
+ struct nvme_dmabuf_token *data = token->dev_priv;
+ struct dma_buf_attachment *attach = data->attach;
+ dma_addr_t *dma_list = NULL;
+ unsigned long tmp, i = 0;
+ struct nvme_dmabuf_map *map;
+ struct scatterlist *sg;
+ struct sg_table *sgt;
+ unsigned nr_entries;
+ int ret;
+
+ dma_resv_assert_held(token->dmabuf->resv);
+
+ map = kmalloc(sizeof(*map), GFP_KERNEL);
+ if (!map)
+ return ERR_PTR(-ENOMEM);
+
+ nr_entries = token->dmabuf->size / NVME_CTRL_PAGE_SIZE;
+ dma_list = kmalloc_array(nr_entries, sizeof(dma_list[0]), GFP_KERNEL);
+ if (!dma_list) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ sgt = dma_buf_map_attachment(attach, token->dir);
+ if (IS_ERR(sgt)) {
+ ret = PTR_ERR(sgt);
+ sgt = NULL;
+ goto err;
+ }
+
+ for_each_sgtable_dma_sg(sgt, sg, tmp) {
+ dma_addr_t dma_addr = sg_dma_address(sg);
+ unsigned long sg_len = sg_dma_len(sg);
+
+ if (sg_len % NVME_CTRL_PAGE_SIZE) {
+ ret = -EINVAL;
+ goto err;
+ }
+
+ while (sg_len) {
+ dma_list[i++] = dma_addr;
+ dma_addr += NVME_CTRL_PAGE_SIZE;
+ sg_len -= NVME_CTRL_PAGE_SIZE;
+ }
+ }
+
+ ret = io_dmabuf_init_map(token, &map->base);
+ if (ret)
+ goto err;
+ map->nr_entries = nr_entries;
+ map->dma_list = dma_list;
+ map->sgt = sgt;
+ return &map->base;
+err:
+ if (sgt)
+ dma_buf_unmap_attachment(attach, sgt, token->dir);
+ kfree(map);
+ kfree(dma_list);
+ return ERR_PTR(ret);
+}
+
+static void nvme_dmabuf_token_unmap(struct io_dmabuf_token *token,
+ struct io_dmabuf_map *map_base)
+{
+ struct nvme_dmabuf_token *data = token->dev_priv;
+ struct nvme_dmabuf_map *map = container_of(map_base,
+ struct nvme_dmabuf_map, base);
+
+ dma_resv_assert_held(token->dmabuf->resv);
+
+ dma_buf_unmap_attachment(data->attach, map->sgt, token->dir);
+ kfree(map->dma_list);
+}
+
+static void nvme_dmabuf_token_release(struct io_dmabuf_token *token)
+{
+ struct nvme_dmabuf_token *data = token->dev_priv;
+
+ dma_buf_detach(token->dmabuf, data->attach);
+ kfree(data);
+}
+
+const struct io_dmabuf_token_dev_ops nvme_dma_token_ops = {
+ .map = nvme_dmabuf_token_map,
+ .unmap = nvme_dmabuf_token_unmap,
+ .release = nvme_dmabuf_token_release,
+};
+
+static int nvme_create_dmabuf_token(struct request_queue *q,
+ struct io_dmabuf_token *token)
+{
+ struct nvme_dmabuf_token *data;
+ struct dma_buf_attachment *attach;
+ struct nvme_ns *ns = q->queuedata;
+ struct nvme_dev *dev = to_nvme_dev(ns->ctrl);
+ struct dma_buf *dmabuf = token->dmabuf;
+
+ data = kzalloc(sizeof(data), GFP_KERNEL);
+ if (!data)
+ return -ENOMEM;
+
+ token->dev_priv = data;
+ token->dev_ops = &nvme_dma_token_ops;
+
+ attach = dma_buf_dynamic_attach(dmabuf, dev->dev,
+ &nvme_dmabuf_importer_ops, token);
+ if (IS_ERR(attach))
+ return PTR_ERR(attach);
+ data->attach = attach;
+ return 0;
+}
+#endif
+
static const struct blk_mq_ops nvme_mq_admin_ops = {
.queue_rq = nvme_queue_rq,
.complete = nvme_pci_complete_rq,
@@ -2256,6 +2533,10 @@ static const struct blk_mq_ops nvme_mq_ops = {
.map_queues = nvme_pci_map_queues,
.timeout = nvme_timeout,
.poll = nvme_poll,
+
+#ifdef CONFIG_DMABUF_TOKEN
+ .create_dmabuf_token = nvme_create_dmabuf_token,
+#endif
};
static void nvme_dev_remove_admin(struct nvme_dev *dev)
@@ -4289,5 +4570,6 @@ MODULE_AUTHOR("Matthew Wilcox <willy@linux.intel.com>");
MODULE_LICENSE("GPL");
MODULE_VERSION("1.0");
MODULE_DESCRIPTION("NVMe host PCIe transport driver");
+MODULE_IMPORT_NS("DMA_BUF");
module_init(nvme_init);
module_exit(nvme_exit);
--
2.53.0
^ permalink raw reply related
* [PATCH v3 06/10] block: forward create_dmabuf_token to drivers
From: Pavel Begunkov @ 2026-04-29 15:25 UTC (permalink / raw)
To: Jens Axboe, Keith Busch, Christoph Hellwig, Sagi Grimberg,
Alexander Viro, Christian Brauner, Andrew Morton, Sumit Semwal,
Christian König, linux-block, linux-kernel, linux-nvme,
linux-fsdevel, io-uring, linux-media, dri-devel, linaro-mm-sig
Cc: asml.silence, Nitesh Shetty, Kanchan Joshi, Anuj Gupta,
Tushar Gohad, William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <cover.1777475843.git.asml.silence@gmail.com>
Add a trivial implementation of the create_dmabuf_token call for
block devices that forwards the call to a new blk-mq callback if it's
available.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
block/fops.c | 14 ++++++++++++++
include/linux/blk-mq.h | 9 +++++++++
2 files changed, 23 insertions(+)
diff --git a/block/fops.c b/block/fops.c
index 713a3ba3f457..3d8a48a7d645 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -951,6 +951,19 @@ static int blkdev_mmap_prepare(struct vm_area_desc *desc)
return generic_file_mmap_prepare(desc);
}
+static int blkdev_create_dmabuf_token(struct file *file,
+ struct io_dmabuf_token *token)
+{
+ struct request_queue *q = bdev_get_queue(file_bdev(file));
+
+ if (!(file->f_flags & O_DIRECT))
+ return -EINVAL;
+ if (!q->mq_ops || !q->mq_ops->create_dmabuf_token)
+ return -EINVAL;
+
+ return q->mq_ops->create_dmabuf_token(q, token);
+}
+
const struct file_operations def_blk_fops = {
.open = blkdev_open,
.release = blkdev_release,
@@ -969,6 +982,7 @@ const struct file_operations def_blk_fops = {
.fallocate = blkdev_fallocate,
.uring_cmd = blkdev_uring_cmd,
.fop_flags = FOP_BUFFER_RASYNC,
+ .create_dmabuf_token = blkdev_create_dmabuf_token,
};
static __init int blkdev_init(void)
diff --git a/include/linux/blk-mq.h b/include/linux/blk-mq.h
index 18a2388ba581..ee31fb3ada10 100644
--- a/include/linux/blk-mq.h
+++ b/include/linux/blk-mq.h
@@ -15,6 +15,8 @@ struct blk_mq_tags;
struct blk_flush_queue;
struct io_comp_batch;
+struct io_dmabuf_token;
+
#define BLKDEV_MIN_RQ 4
#define BLKDEV_DEFAULT_RQ 128
@@ -684,6 +686,13 @@ struct blk_mq_ops {
*/
void (*show_rq)(struct seq_file *m, struct request *rq);
#endif
+
+ /**
+ * @create_dma_token: Create a dma token, which will be using to map
+ * a dmabuf for IO requests.
+ */
+ int (*create_dmabuf_token)(struct request_queue *,
+ struct io_dmabuf_token *token);
};
/* Keep hctx_flag_name[] in sync with the definitions below */
--
2.53.0
^ permalink raw reply related
* [PATCH v3 05/10] lib: add dmabuf token infrastructure
From: Pavel Begunkov @ 2026-04-29 15:25 UTC (permalink / raw)
To: Jens Axboe, Keith Busch, Christoph Hellwig, Sagi Grimberg,
Alexander Viro, Christian Brauner, Andrew Morton, Sumit Semwal,
Christian König, linux-block, linux-kernel, linux-nvme,
linux-fsdevel, io-uring, linux-media, dri-devel, linaro-mm-sig
Cc: asml.silence, Nitesh Shetty, Kanchan Joshi, Anuj Gupta,
Tushar Gohad, William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <cover.1777475843.git.asml.silence@gmail.com>
There are two main objects. struct io_dmabuf_token and struct
io_dmabuf_map. The token is used during initial registration and serves
as an interface between the upper layer user like io_uring and to the
importer subsystem / driver. io_dmabuf_map represens the actual dma map
established for the target device[s] with dma_buf_map_attachment() and
stored in a device specific format.
The separation into two different objects exists to support map
invalidation (see dma_buf_invalidate_mappings()). A token can create
multiple maps during its lifetime, but there can only be one (active)
map attached to it. It's aslo possible to not have an active map.
Invalidation drops the active map if present, and the next map will
only be attempted to be created once there is a new request that
wants to use the token.
The primary task of the io_dmabuf_map object is to count all requests
currently using it, which is done with percpu refcounts. When a map is
invalidated, we remove it from the token, so there can be no new
requests, then it adds a fence to the dmabuf reservation object. Once
all the requests complete, we signal the fence and unmap it.
[un]mapping and any work with dma addresses is delegated to the
importer driver via an ops table stored in the token, see struct
io_dmabuf_token_dev_ops. That's required because the generic layer
doesn't have knowledge about the device it's going to be use with,
and there will be more complex use cases with multiple devices.
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
include/linux/io_dmabuf_token.h | 92 +++++++++++
lib/Kconfig | 4 +
lib/Makefile | 2 +
lib/io_dmabuf_token.c | 272 ++++++++++++++++++++++++++++++++
4 files changed, 370 insertions(+)
create mode 100644 include/linux/io_dmabuf_token.h
create mode 100644 lib/io_dmabuf_token.c
diff --git a/include/linux/io_dmabuf_token.h b/include/linux/io_dmabuf_token.h
new file mode 100644
index 000000000000..b94bda684812
--- /dev/null
+++ b/include/linux/io_dmabuf_token.h
@@ -0,0 +1,92 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+#ifndef _LINUX_DMA_TOKEN_H
+#define _LINUX_DMA_TOKEN_H
+
+#include <linux/dma-buf.h>
+
+struct io_dmabuf_fence;
+struct io_dmabuf_token;
+struct io_dmabuf_map;
+
+struct io_dmabuf_token_dev_ops {
+ /*
+ * Create a new map for the given token. It should be initialised
+ * with io_dmabuf_init_map(). The callback is executed with the
+ * reservation lock held.
+ */
+ struct io_dmabuf_map *(*map)(struct io_dmabuf_token *);
+
+ /*
+ * Clean up device specific parts of the map. The callback is
+ * executed with the reservation lock held.
+ */
+ void (*unmap)(struct io_dmabuf_token *, struct io_dmabuf_map *);
+
+ /*
+ * The user tries to destroy the token. Release all device specific
+ * parts of the token.
+ */
+ void (*release)(struct io_dmabuf_token *);
+};
+
+struct io_dmabuf_map {
+ /*
+ * Counts attached requests and other users. Device specific unmapping
+ * is deferred until all refs are dropped.
+ */
+ struct percpu_ref refs;
+
+ struct work_struct release_work;
+ struct io_dmabuf_fence *fence;
+ struct io_dmabuf_token *token;
+};
+
+struct io_dmabuf_token {
+ struct io_dmabuf_map __rcu *map;
+ struct dma_buf *dmabuf;
+ enum dma_data_direction dir;
+
+ atomic_t fence_seq;
+ u64 fence_ctx;
+ struct work_struct release_work;
+ refcount_t refs;
+
+ void *dev_priv;
+ const struct io_dmabuf_token_dev_ops *dev_ops;
+};
+
+int io_dmabuf_token_create(struct file *file,
+ struct io_dmabuf_token *token,
+ struct dma_buf *dmabuf,
+ enum dma_data_direction dir);
+void io_dmabuf_token_release(struct io_dmabuf_token *token);
+
+struct io_dmabuf_map *io_dmabuf_create_map(struct io_dmabuf_token *token);
+
+static inline struct io_dmabuf_map *io_dmabuf_get_map(struct io_dmabuf_token *token)
+{
+ struct io_dmabuf_map *map;
+
+ guard(rcu)();
+
+ map = rcu_dereference(token->map);
+ if (unlikely(!map || !percpu_ref_tryget_live_rcu(&map->refs)))
+ return NULL;
+
+ return map;
+}
+
+static inline void io_dmabuf_map_drop(struct io_dmabuf_map *map)
+{
+ percpu_ref_put(&map->refs);
+}
+
+/*
+ * Device API
+ */
+
+void io_dmabuf_token_invalidate_mappings(struct io_dmabuf_token *token);
+int io_dmabuf_init_map(struct io_dmabuf_token *token, struct io_dmabuf_map *map);
+
+
+#endif
diff --git a/lib/Kconfig b/lib/Kconfig
index 0f2fb9610647..853f10bf8e1a 100644
--- a/lib/Kconfig
+++ b/lib/Kconfig
@@ -636,3 +636,7 @@ config UNION_FIND
config MIN_HEAP
bool
+
+config DMABUF_TOKEN
+ def_bool y
+ depends on DMA_SHARED_BUFFER
diff --git a/lib/Makefile b/lib/Makefile
index ea660cca04f4..4a42cfcaa80c 100644
--- a/lib/Makefile
+++ b/lib/Makefile
@@ -246,6 +246,8 @@ obj-$(CONFIG_IRQ_POLL) += irq_poll.o
obj-$(CONFIG_POLYNOMIAL) += polynomial.o
+obj-$(CONFIG_DMABUF_TOKEN) += io_dmabuf_token.o
+
# stackdepot.c should not be instrumented or call instrumented functions.
# Prevent the compiler from calling builtins like memcmp() or bcmp() from this
# file.
diff --git a/lib/io_dmabuf_token.c b/lib/io_dmabuf_token.c
new file mode 100644
index 000000000000..808b5ad33dbc
--- /dev/null
+++ b/lib/io_dmabuf_token.c
@@ -0,0 +1,272 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Common infrastructure for supporing dma-buf in the I/O path.
+ *
+ * Copyright (C) 2026 Pavel Begunkov <asml.silence@gmail.com>
+ */
+#include <linux/io_dmabuf_token.h>
+#include <linux/dma-resv.h>
+
+struct io_dmabuf_fence {
+ struct dma_fence base;
+ spinlock_t lock;
+};
+
+static const char *io_dmabuf_fence_drv_name(struct dma_fence *fence)
+{
+ /* default fence release kfree's the base pointer */
+ BUILD_BUG_ON(offsetof(struct io_dmabuf_fence, base));
+
+ return "DMABUF token";
+}
+
+static const char *io_dmabuf_fence_timeline_name(struct dma_fence *fence)
+{
+ return "DMABUF token";
+}
+
+const struct dma_fence_ops io_dmabuf_fence_ops = {
+ .get_driver_name = io_dmabuf_fence_drv_name,
+ .get_timeline_name = io_dmabuf_fence_timeline_name,
+};
+
+static void io_dmabuf_token_destroy_work(struct work_struct *work)
+{
+ struct io_dmabuf_token *token = container_of(work, struct io_dmabuf_token,
+ release_work);
+
+ if (WARN_ON_ONCE(refcount_read(&token->refs)))
+ return;
+
+ token->dev_ops->release(token);
+ dma_buf_put(token->dmabuf);
+ kfree(token);
+}
+
+static void io_dmabuf_map_release_work(struct work_struct *work)
+{
+ struct io_dmabuf_map *map = container_of(work, struct io_dmabuf_map,
+ release_work);
+ struct io_dmabuf_fence *fence = map->fence;
+ struct io_dmabuf_token *token = map->token;
+ struct dma_buf *dmabuf = token->dmabuf;
+
+ /* the release path must wait for fences */
+ if (WARN_ON_ONCE(refcount_read(&token->refs) == 0))
+ return;
+
+ /* Prevent from destoying the token while unmapping */
+ refcount_inc(&token->refs);
+
+ /*
+ * There are no more requests using the map, we can signal the fence.
+ * It should be done before taking the resv lock as someone could be
+ * waiting for the fence while holding the lock.
+ */
+ dma_fence_signal(&fence->base);
+
+ dma_resv_lock(dmabuf->resv, NULL);
+ token->dev_ops->unmap(token, map);
+ dma_resv_unlock(dmabuf->resv);
+
+ dma_fence_put(&fence->base);
+ percpu_ref_exit(&map->refs);
+ kfree(map);
+
+ if (refcount_dec_and_test(&token->refs)) {
+ /*
+ * Destruction needs to wait for I/O and dma fences. Defer it to
+ * simplify locking.
+ */
+ INIT_WORK(&token->release_work, io_dmabuf_token_destroy_work);
+ queue_work(system_wq, &token->release_work);
+ }
+}
+
+static void io_dmabuf_map_refs_release(struct percpu_ref *ref)
+{
+ struct io_dmabuf_map *map = container_of(ref, struct io_dmabuf_map, refs);
+
+ /* might sleep, use a worker */
+ INIT_WORK(&map->release_work, io_dmabuf_map_release_work);
+ queue_work(system_wq, &map->release_work);
+}
+
+int io_dmabuf_init_map(struct io_dmabuf_token *token, struct io_dmabuf_map *map)
+{
+ struct io_dmabuf_fence *fence = NULL;
+ int ret;
+
+ fence = kzalloc(sizeof(*fence), GFP_KERNEL);
+ if (!fence)
+ return -ENOMEM;
+
+ ret = percpu_ref_init(&map->refs, io_dmabuf_map_refs_release, 0, GFP_KERNEL);
+ if (ret) {
+ kfree(fence);
+ return ret;
+ }
+
+ spin_lock_init(&fence->lock);
+ dma_fence_init(&fence->base, &io_dmabuf_fence_ops, &fence->lock,
+ token->fence_ctx, atomic_inc_return(&token->fence_seq));
+ map->fence = fence;
+ map->token = token;
+ return 0;
+}
+EXPORT_SYMBOL_NS_GPL(io_dmabuf_init_map, "DMA_BUF");
+
+struct io_dmabuf_map *io_dmabuf_create_map(struct io_dmabuf_token *token)
+{
+ struct dma_buf *dmabuf = token->dmabuf;
+ struct io_dmabuf_map *map;
+ long ret;
+
+retry:
+ /*
+ * ->dmabuf_map() will be calling dma_buf_map_attachment(), for which
+ * we'll need to wait for fences. Do a bit nicer and try to wait
+ * without the resv lock first.
+ */
+ ret = dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_KERNEL,
+ true, MAX_SCHEDULE_TIMEOUT);
+ if (!ret)
+ ret = -EAGAIN;
+ if (ret < 0)
+ return ERR_PTR(ret);
+
+ dma_resv_lock(dmabuf->resv, NULL);
+ map = io_dmabuf_get_map(token);
+ if (map) {
+ ret = 0;
+ goto out;
+ }
+
+ if (dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_KERNEL,
+ true, 0) < 0) {
+ dma_resv_unlock(dmabuf->resv);
+ goto retry;
+ }
+
+ map = token->dev_ops->map(token);
+ if (IS_ERR(map)) {
+ ret = PTR_ERR(map);
+ goto out;
+ }
+
+ percpu_ref_get(&map->refs);
+ rcu_assign_pointer(token->map, map);
+out:
+ dma_resv_unlock(dmabuf->resv);
+ if (ret < 0)
+ return ERR_PTR(ret);
+ return map;
+}
+
+static void io_dmabuf_drop_map(struct io_dmabuf_token *token)
+{
+ struct dma_buf *dmabuf = token->dmabuf;
+ struct io_dmabuf_map *map;
+ int ret;
+
+ dma_resv_assert_held(dmabuf->resv);
+
+ map = rcu_dereference_protected(token->map,
+ dma_resv_held(dmabuf->resv));
+ if (!map)
+ return;
+ rcu_assign_pointer(token->map, NULL);
+
+ ret = dma_resv_reserve_fences(dmabuf->resv, 1);
+ if (WARN_ON_ONCE(ret)) {
+ struct dma_fence *fence = &map->fence->base;
+
+ dma_fence_get(fence);
+ percpu_ref_kill(&map->refs);
+ dma_fence_wait(fence, false);
+ dma_fence_put(fence);
+ return;
+ }
+
+ dma_resv_add_fence(dmabuf->resv, &map->fence->base,
+ DMA_RESV_USAGE_KERNEL);
+ /*
+ * Delay destruction until all inflight requests using the map are
+ * gone. It'll also signal the fence then.
+ */
+ percpu_ref_kill(&map->refs);
+}
+
+void io_dmabuf_token_invalidate_mappings(struct io_dmabuf_token *token)
+{
+ io_dmabuf_drop_map(token);
+}
+EXPORT_SYMBOL_NS_GPL(io_dmabuf_token_invalidate_mappings, "DMA_BUF");
+
+static void io_dmabuf_token_release_work(struct work_struct *work)
+{
+ struct io_dmabuf_token *token = container_of(work, struct io_dmabuf_token,
+ release_work);
+ struct dma_buf *dmabuf = token->dmabuf;
+ long ret;
+
+ dma_resv_lock(dmabuf->resv, NULL);
+ /* Remove the last map, there should be no new ones going forward. */
+ io_dmabuf_drop_map(token);
+ dma_resv_unlock(dmabuf->resv);
+
+ /* Wait until all maps are destroyed. */
+ ret = dma_resv_wait_timeout(dmabuf->resv, DMA_RESV_USAGE_KERNEL,
+ false, MAX_SCHEDULE_TIMEOUT);
+
+ if (WARN_ON_ONCE(ret <= 0))
+ return;
+ if (WARN_ON_ONCE(rcu_dereference_protected(token->map, true)))
+ return;
+
+ if (refcount_dec_and_test(&token->refs))
+ io_dmabuf_token_destroy_work(&token->release_work);
+}
+
+void io_dmabuf_token_release(struct io_dmabuf_token *token)
+{
+ /*
+ * Destruction needs to wait for I/O and dma fences. Defer it to
+ * simplify locking.
+ */
+ INIT_WORK(&token->release_work, io_dmabuf_token_release_work);
+ queue_work(system_wq, &token->release_work);
+}
+
+int io_dmabuf_token_create(struct file *file,
+ struct io_dmabuf_token *token,
+ struct dma_buf *dmabuf,
+ enum dma_data_direction dir)
+{
+ int ret;
+
+ if (!file->f_op->create_dmabuf_token)
+ return -EOPNOTSUPP;
+
+ memset(token, 0, sizeof(*token));
+ token->fence_ctx = dma_fence_context_alloc(1);
+ token->dir = dir;
+ token->dmabuf = dmabuf;
+ refcount_set(&token->refs, 1);
+ get_dma_buf(dmabuf);
+
+ ret = file->f_op->create_dmabuf_token(file, token);
+ if (ret) {
+ memset(token, 0, sizeof(*token));
+ dma_buf_put(dmabuf);
+ return ret;
+ }
+
+ if (WARN_ON_ONCE(!token->dev_ops ||
+ !token->dev_ops->map ||
+ !token->dev_ops->unmap ||
+ !token->dev_ops->release))
+ return -EINVAL;
+
+ return ret;
+}
--
2.53.0
^ permalink raw reply related
* [PATCH v3 04/10] block: introduce dma map backed bio type
From: Pavel Begunkov @ 2026-04-29 15:25 UTC (permalink / raw)
To: Jens Axboe, Keith Busch, Christoph Hellwig, Sagi Grimberg,
Alexander Viro, Christian Brauner, Andrew Morton, Sumit Semwal,
Christian König, linux-block, linux-kernel, linux-nvme,
linux-fsdevel, io-uring, linux-media, dri-devel, linaro-mm-sig
Cc: asml.silence, Nitesh Shetty, Kanchan Joshi, Anuj Gupta,
Tushar Gohad, William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <cover.1777475843.git.asml.silence@gmail.com>
Premapped buffers don't require a generic bio_vec since these have
already been dma mapped. Repurpose the bi_io_vec space to strore dmabuf
maps as they are mutually exclusive.
Suggested-by: Keith Busch <kbusch@kernel.org>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
block/bio.c | 25 ++++++++++++++++++++++++-
block/blk-merge.c | 14 ++++++++++++++
block/blk.h | 3 ++-
block/fops.c | 2 ++
include/linux/bio.h | 19 ++++++++++++++++---
include/linux/blk_types.h | 8 +++++++-
6 files changed, 65 insertions(+), 6 deletions(-)
diff --git a/block/bio.c b/block/bio.c
index 0734b50d4992..bdc91777c288 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -851,7 +851,13 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
bio->bi_write_hint = bio_src->bi_write_hint;
bio->bi_write_stream = bio_src->bi_write_stream;
bio->bi_iter = bio_src->bi_iter;
- bio->bi_io_vec = bio_src->bi_io_vec;
+
+ if (!bio_flagged(bio_src, BIO_DMABUF_MAP)) {
+ bio->bi_io_vec = bio_src->bi_io_vec;
+ } else {
+ bio->dmabuf_map = bio_src->dmabuf_map;
+ bio_set_flag(bio, BIO_DMABUF_MAP);
+ }
if (bio->bi_bdev) {
if (bio->bi_bdev == bio_src->bi_bdev &&
@@ -1183,6 +1189,18 @@ void bio_iov_bvec_set(struct bio *bio, const struct iov_iter *iter)
bio_set_flag(bio, BIO_CLONED);
}
+void bio_dmabuf_map_set(struct bio *bio, struct iov_iter *iter)
+{
+ WARN_ON_ONCE(bio->bi_max_vecs);
+
+ bio->dmabuf_map = iter->dmabuf_map;
+ bio->bi_vcnt = 0;
+ bio->bi_iter.bi_bvec_done = iter->iov_offset;
+ bio->bi_iter.bi_size = iov_iter_count(iter);
+ bio->bi_opf |= REQ_NOMERGE;
+ bio_set_flag(bio, BIO_DMABUF_MAP);
+}
+
/*
* Aligns the bio size to the len_align_mask, releasing excessive bio vecs that
* __bio_iov_iter_get_pages may have inserted, and reverts the trimmed length
@@ -1252,6 +1270,11 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
iov_iter_advance(iter, bio->bi_iter.bi_size);
return 0;
}
+ if (iov_iter_is_dmabuf_map(iter)) {
+ bio_dmabuf_map_set(bio, iter);
+ iov_iter_advance(iter, bio->bi_iter.bi_size);
+ return 0;
+ }
if (iov_iter_extract_will_pin(iter))
bio_set_flag(bio, BIO_PAGE_PINNED);
diff --git a/block/blk-merge.c b/block/blk-merge.c
index fcf09325b22e..fc2c0c428001 100644
--- a/block/blk-merge.c
+++ b/block/blk-merge.c
@@ -348,6 +348,19 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
len_align_mask |= (bc->bc_key->crypto_cfg.data_unit_size - 1);
}
+ if (bio_flagged(bio, BIO_DMABUF_MAP)) {
+ nsegs = 1;
+
+ if ((bio->bi_iter.bi_bvec_done & lim->dma_alignment) ||
+ (bio->bi_iter.bi_size & len_align_mask))
+ return -EINVAL;
+ if (bio->bi_iter.bi_size > max_bytes) {
+ bytes = max_bytes;
+ goto split;
+ }
+ goto out;
+ }
+
bio_for_each_bvec(bv, bio, iter) {
if (bv.bv_offset & start_align_mask ||
bv.bv_len & len_align_mask)
@@ -378,6 +391,7 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
bvprvp = &bvprv;
}
+out:
*segs = nsegs;
bio->bi_bvec_gap_bit = ffs(gaps);
return 0;
diff --git a/block/blk.h b/block/blk.h
index b998a7761faf..b4b09abebce8 100644
--- a/block/blk.h
+++ b/block/blk.h
@@ -424,7 +424,8 @@ static inline struct bio *__bio_split_to_limits(struct bio *bio,
switch (bio_op(bio)) {
case REQ_OP_READ:
case REQ_OP_WRITE:
- if (bio_may_need_split(bio, lim))
+ if (bio_may_need_split(bio, lim) ||
+ bio_flagged(bio, BIO_DMABUF_MAP))
return bio_split_rw(bio, lim, nr_segs);
*nr_segs = 1;
return bio;
diff --git a/block/fops.c b/block/fops.c
index bb6642b45937..713a3ba3f457 100644
--- a/block/fops.c
+++ b/block/fops.c
@@ -349,6 +349,8 @@ static ssize_t __blkdev_direct_IO_async(struct kiocb *iocb,
* bio_iov_iter_get_pages() and set the bvec directly.
*/
bio_iov_bvec_set(bio, iter);
+ } else if (iov_iter_is_dmabuf_map(iter)) {
+ bio_dmabuf_map_set(bio, iter);
} else {
ret = blkdev_iov_iter_get_pages(bio, iter, bdev);
if (unlikely(ret))
diff --git a/include/linux/bio.h b/include/linux/bio.h
index 97d747320b35..0c43fa6b0900 100644
--- a/include/linux/bio.h
+++ b/include/linux/bio.h
@@ -108,16 +108,26 @@ static inline bool bio_next_segment(const struct bio *bio,
#define bio_for_each_segment_all(bvl, bio, iter) \
for (bvl = bvec_init_iter_all(&iter); bio_next_segment((bio), &iter); )
+static inline void bio_advance_iter_dmabuf_map(struct bvec_iter *iter,
+ unsigned int bytes)
+{
+ iter->bi_bvec_done += bytes;
+ iter->bi_size -= bytes;
+}
+
static inline void bio_advance_iter(const struct bio *bio,
struct bvec_iter *iter, unsigned int bytes)
{
iter->bi_sector += bytes >> 9;
- if (bio_no_advance_iter(bio))
+ if (bio_no_advance_iter(bio)) {
iter->bi_size -= bytes;
- else
+ } else if (bio_flagged(bio, BIO_DMABUF_MAP)) {
+ bio_advance_iter_dmabuf_map(iter, bytes);
+ } else {
bvec_iter_advance(bio->bi_io_vec, iter, bytes);
/* TODO: It is reasonable to complete bio with error here. */
+ }
}
/* @bytes should be less or equal to bvec[i->bi_idx].bv_len */
@@ -129,6 +139,8 @@ static inline void bio_advance_iter_single(const struct bio *bio,
if (bio_no_advance_iter(bio))
iter->bi_size -= bytes;
+ else if (bio_flagged(bio, BIO_DMABUF_MAP))
+ bio_advance_iter_dmabuf_map(iter, bytes);
else
bvec_iter_advance_single(bio->bi_io_vec, iter, bytes);
}
@@ -391,7 +403,7 @@ static inline void bio_wouldblock_error(struct bio *bio)
*/
static inline int bio_iov_vecs_to_alloc(struct iov_iter *iter, int max_segs)
{
- if (iov_iter_is_bvec(iter))
+ if (iov_iter_is_bvec(iter) || iov_iter_is_dmabuf_map(iter))
return 0;
return iov_iter_npages(iter, max_segs);
}
@@ -471,6 +483,7 @@ int bio_iov_iter_get_pages(struct bio *bio, struct iov_iter *iter,
unsigned len_align_mask);
void bio_iov_bvec_set(struct bio *bio, const struct iov_iter *iter);
+void bio_dmabuf_map_set(struct bio *bio, struct iov_iter *iter);
void __bio_release_pages(struct bio *bio, bool mark_dirty);
extern void bio_set_pages_dirty(struct bio *bio);
extern void bio_check_pages_dirty(struct bio *bio);
diff --git a/include/linux/blk_types.h b/include/linux/blk_types.h
index 8808ee76e73c..d5ad085b701d 100644
--- a/include/linux/blk_types.h
+++ b/include/linux/blk_types.h
@@ -233,7 +233,12 @@ struct bio {
atomic_t __bi_remaining;
/* The actual vec list, preserved by bio_reset() */
- struct bio_vec *bi_io_vec;
+ union {
+ struct bio_vec *bi_io_vec;
+ /* Driver specific dma map, present only with BIO_DMABUF_MAP */
+ struct io_dmabuf_map *dmabuf_map;
+ };
+
struct bvec_iter bi_iter;
union {
@@ -322,6 +327,7 @@ enum {
BIO_REMAPPED,
BIO_ZONE_WRITE_PLUGGING, /* bio handled through zone write plugging */
BIO_EMULATES_ZONE_APPEND, /* bio emulates a zone append operation */
+ BIO_DMABUF_MAP, /* Using premmaped dma buffers */
BIO_FLAG_LAST
};
--
2.53.0
^ permalink raw reply related
* [PATCH v3 03/10] block: move bvec init into __bio_clone
From: Pavel Begunkov @ 2026-04-29 15:25 UTC (permalink / raw)
To: Jens Axboe, Keith Busch, Christoph Hellwig, Sagi Grimberg,
Alexander Viro, Christian Brauner, Andrew Morton, Sumit Semwal,
Christian König, linux-block, linux-kernel, linux-nvme,
linux-fsdevel, io-uring, linux-media, dri-devel, linaro-mm-sig
Cc: asml.silence, Nitesh Shetty, Kanchan Joshi, Anuj Gupta,
Tushar Gohad, William Power, Phil Cayton, Jason Gunthorpe
In-Reply-To: <cover.1777475843.git.asml.silence@gmail.com>
To quote Cristoph: "Historically __bio_clone itself does not clone the
payload, just the bio. But we got rid of the callers that want to clone
a bio but not the payload long time ago". So let's move ->bi_io_vec
assignment into __bio_clone(), so we have a single point where it's set.
Suggested-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Pavel Begunkov <asml.silence@gmail.com>
---
block/bio.c | 5 ++---
1 file changed, 2 insertions(+), 3 deletions(-)
diff --git a/block/bio.c b/block/bio.c
index 4d46af0cd256..0734b50d4992 100644
--- a/block/bio.c
+++ b/block/bio.c
@@ -851,6 +851,7 @@ static int __bio_clone(struct bio *bio, struct bio *bio_src, gfp_t gfp)
bio->bi_write_hint = bio_src->bi_write_hint;
bio->bi_write_stream = bio_src->bi_write_stream;
bio->bi_iter = bio_src->bi_iter;
+ bio->bi_io_vec = bio_src->bi_io_vec;
if (bio->bi_bdev) {
if (bio->bi_bdev == bio_src->bi_bdev &&
@@ -893,8 +894,6 @@ struct bio *bio_alloc_clone(struct block_device *bdev, struct bio *bio_src,
bio_put(bio);
return NULL;
}
- bio->bi_io_vec = bio_src->bi_io_vec;
-
return bio;
}
EXPORT_SYMBOL(bio_alloc_clone);
@@ -914,7 +913,7 @@ int bio_init_clone(struct block_device *bdev, struct bio *bio,
{
int ret;
- bio_init(bio, bdev, bio_src->bi_io_vec, 0, bio_src->bi_opf);
+ bio_init(bio, bdev, NULL, 0, bio_src->bi_opf);
ret = __bio_clone(bio, bio_src, gfp);
if (ret)
bio_uninit(bio);
--
2.53.0
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox