Re: [PATCH v1 07/30] io_uring/rsrc: add fixed buffer table pinning/unpinning

[Caleb Sander Mateos <csander@purestorage.com>]

On Thu, Dec 4, 2025 at 12:07 PM Joanne Koong <joannelkoong@gmail.com> wrote:
>
> On Wed, Dec 3, 2025 at 5:24 PM Caleb Sander Mateos
> <csander@purestorage.com> wrote:
> >
> > On Wed, Dec 3, 2025 at 2:52 PM Joanne Koong <joannelkoong@gmail.com> wrote:
> > >
> > > On Tue, Dec 2, 2025 at 8:49 PM Caleb Sander Mateos
> > > <csander@purestorage.com> wrote:
> > > >
> > > > On Tue, Dec 2, 2025 at 4:36 PM Joanne Koong <joannelkoong@gmail.com> wrote:
> > > > >
> > > > > Add kernel APIs to pin and unpin the buffer table for fixed buffers,
> > > > > preventing userspace from unregistering or updating the fixed buffers
> > > > > table while it is pinned by the kernel.
> > > > >
> > > > > This has two advantages:
> > > > > a) Eliminating the overhead of having to fetch and construct an iter for
> > > > > a fixed buffer per every cmd. Instead, the caller can pin the buffer
> > > > > table, fetch/construct the iter once, and use that across cmds for
> > > > > however long it needs to until it is ready to unpin the buffer table.
> > > > >
> > > > > b) Allowing a fixed buffer lookup at any index. The buffer table must be
> > > > > pinned in order to allow this, otherwise we would have to keep track of
> > > > > all the nodes that have been looked up by the io_kiocb so that we can
> > > > > properly adjust the refcounts for those nodes. Ensuring that the buffer
> > > > > table must first be pinned before being able to fetch a buffer at any
> > > > > index makes things logistically a lot neater.
> > > >
> > > > Why is it necessary to pin the entire buffer table rather than
> > > > specific entries? That's the purpose of the existing io_rsrc_node refs
> > > > field.
> > >
> > > How would this work with userspace buffer unregistration (which works
> > > at the table level)? If buffer unregistration should still succeed
> > > then fuse would need a way to be notified that the buffer has been
> > > unregistered since the buffer belongs to userspace (eg it would be
> > > wrong if fuse continues using it even though fuse retains a refcount
> > > on it). If buffer unregistration should fail, then we would need to
> > > track this pinned state inside the node instead of relying just on the
> > > refs field, as buffers can be unregistered even if there are in-flight
> > > refs (eg we would need to differentiate the ref being from a pin vs
> > > from not a pin), and I think this would make unregistration more
> > > cumbersome as well (eg we would have to iterate through all the
> > > entries looking to see if any are pinned before iterating through them
> > > again to do the actual unregistration).
> >
> > Not sure I would say buffer unregistration operates on the table as a
> > whole. Each registered buffer node is unregistered individually and
>
> I'm looking at the liburing interface for it and I'm only seeing
> io_uring_unregister_buffers() / IORING_UNREGISTER_BUFFERS which works
> on the entire table, so I'm wondering how that interface would work if
> pinning/unpinning was at the entry level?

IORING_REGISTER_BUFFERS_UPDATE can be used to update individual
registered buffers. For each updated slot, if there is an existing
buffer, it will be unregistered (decrementing the buffer node's
reference count). A new buffer may or may not be registered in its
place; it can be skipped by specifying a struct iovec with a NULL
iov_base.

>
> > stores its own reference count. io_put_rsrc_node() will be called on
> > each buffer node in the table. However, io_put_rsrc_node() just
> > removes the one reference from the buffer node. If there are other
> > references on the buffer node (such as an inflight io_uring request
> > using it), io_free_rsrc_node() won't be called to free the buffer node
> > until all those references are dropped too. So fuse holding a
> > reference on the buffer node would allow it to be unregistered, but
> > prevent it from being freed until fuse dropped its reference.
> > I'm not sure I understand the problem with fuse continuing to hold
> > onto a registered buffer node after userspace has unregistered it from
> > the buffer table. (It looks like the buffer node in question is the
>
> For fuse, it holds the reference to the buffer for the lifetime of the
> connection, which could be a very long time. I'm not seeing how we
> could let userspace succeed in unregistering with fuse continuing to
> hold that reference, since as I understand it conceptually,
> unregistering the buffer should give ownership of the buffer
> completely back to userspace.

I'm not quite sure what you mean by "give ownership of the buffer
completely back to userspace". My understanding is that registering a
buffer with io_uring just pins the physical pages as a perf
optimization. I'm not aware of a way for userspace to observe directly
whether or not certain physical pages are pinned. There's already no
guarantee that the physical pages are unpinned as soon as a buffer is
unregistered; if there are any inflight io_uring requests using the
registered buffer, they will continue to hold a reference count on the
buffer, preventing it from being released. The only guarantee is that
the unregistered slot in the buffer table is now empty.

Presumably fuse must release its reference count (or pin) eventually,
or otherwise there would be a resource leak? I don't see an issue with
holding references to registered buffers for the lifetime of a fuse
connection. As long as there's a way for the fuse server to tell the
kernel to release those resources if they are no longer needed (which
it sounds like already exists from your description of aborting a fuse
connection).

>
> > one at FUSE_URING_FIXED_HEADERS_INDEX?) Wouldn't pinning the buffer
>
> Yep you have that right, the buffer node in question is the one at
> FUSE_URING_FIXED_HEADERS_INDEX which is where all the headers for
> requests are placed.
>
> > table present similar issues? How would userspace get fuse to drop its
>
> I don't think pinning the buffer table has a similar issue because we
> disallow unregistration if it's pinned.

It sounds like you're saying that buffer unregistration just isn't
expected to work together with fuse's use of registered buffers, is
that accurate? Does it matter then whether the buffer unregistration
returns -EBUSY because the buffer table is pinned vs. succeeding but
not actually releasing the buffer resource because fuse still holds a
reference on it? My preference would be to just use the existing
reference counting mechanism rather than introducing another mechanism
if both provide safety equally well.

>
> > pin if it wants to modify the buffer registrations? I would imagine
>
> For the fuse use case, the server never really modifies its buffer
> registrations as it sets up everything before initiating the
> connection. But if it wanted to in the future, the server could send a
> fuse notification to the kernel to unpin the buf table.

This seems to assume that all the registered buffers are used
exclusively for fuse purposes. But it seems plausible that a fuse
server might want to use io_uring for other non-fuse purposes and want
to use registered buffers there too as a perf optimization? Then it
would be nice for the process to be able to update the other
registered buffers even while the kernel pins (or holds a reference
count on) the ones used for fuse.

Best,
Caleb

>
> > the code path that calls io_uring_buf_table_unpin() currently could
> > instead call into io_put_rsrc_node() (maybe by completing an io_uring
> > request that has imported the registered buffer) to release its
> > reference on the buffer node. For ublk, userspace can request to stop
> > a ublk device or the kernel will do so automatically if userspace
> > drops its file handle (e.g. if the process exits), which will release
> > any io_uring resources the ublk device is using.
>
> Fuse has something similar where the server can abort the connection,
> and that will release the pin / other io uring resources.
>
> Thanks,
> Joanne
>
> >
> > >
> > > >
> > > > >