Re: [PATCH 14/15] fuse: Implement multi-threading

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 3749 bytes --]

On Fri, Apr 04, 2025 at 02:49:08PM +0200, Hanna Czenczek wrote:
> On 27.03.25 16:55, Stefan Hajnoczi wrote:
> > On Tue, Mar 25, 2025 at 05:06:54PM +0100, Hanna Czenczek wrote:
> > > FUSE allows creating multiple request queues by "cloning" /dev/fuse FDs
> > > (via open("/dev/fuse") + ioctl(FUSE_DEV_IOC_CLONE)).
> > > 
> > > We can use this to implement multi-threading.
> > > 
> > > Note that the interface presented here differs from the multi-queue
> > > interface of virtio-blk: The latter maps virtqueues to iothreads, which
> > > allows processing multiple virtqueues in a single iothread.  The
> > > equivalent (processing multiple FDs in a single iothread) would not make
> > > sense for FUSE because those FDs are used in a round-robin fashion by
> > > the FUSE kernel driver.  Putting two of them into a single iothread will
> > > just create a bottleneck.
> > This text might be outdated. virtio-blk's new iothread-vq-mapping
> > parameter provides the "array of iothreads" mentioned below and a way to
> > assign virtqueues to those IOThreads.
> 
> Ah, yes.  The difference is still that with FUSE, there is no such
> assignment, because it wouldn’t make sense.  But I can change s/maps
> virtqueues/allows mapping virtqueues/, and s/differs from/is only a subset
> of/, if that’s alright.

Sure, thanks!

> > > Therefore, all we need is an array of iothreads, and we will create one
> > > "queue" (FD) per thread.
> > > 
> > > These are the benchmark results when using four threads (compared to a
> > > single thread); note that fio still only uses a single job, but
> > > performance can still be improved because of said round-robin usage for
> > > the queues.  (Not in the sync case, though, in which case I guess it
> > > just adds overhead.)
> > Interesting. FUSE-over-io_uring seems to be different from
> > FUSE_DEV_IOC_CLONE here. It doesn't do round-robin. It uses CPU affinity
> > instead, handing requests to the io_uring context associated with the
> > current CPU when possible.
> 
> Do you think that should have implications for the QAPI interface?

It would be helpful to document the behavior so users know when
round-robin or CPU affinity are used, but the parameter itself would be
unchanged: an array of IOThreads.

> 
> [...]
> 
> > >   qapi/block-export.json |   8 +-
> > >   block/export/fuse.c    | 214 +++++++++++++++++++++++++++++++++--------
> > >   2 files changed, 179 insertions(+), 43 deletions(-)
> > > 
> > > diff --git a/qapi/block-export.json b/qapi/block-export.json
> > > index c783e01a53..0bdd5992eb 100644
> > > --- a/qapi/block-export.json
> > > +++ b/qapi/block-export.json
> > > @@ -179,12 +179,18 @@
> > >   #     mount the export with allow_other, and if that fails, try again
> > >   #     without.  (since 6.1; default: auto)
> > >   #
> > > +# @iothreads: Enables multi-threading: Handle requests in each of the
> > > +#     given iothreads (instead of the block device's iothread, or the
> > > +#     export's "main" iothread).  For this, the FUSE FD is duplicated so
> > > +#     there is one FD per iothread.  (since 10.1)
> > This option isn't FUSE-specific but FUSE is the first export type to
> > support it. Please add it to BlockExportOptions instead and refuse
> > export creation when the export type only supports 1 IOThread.
> 
> Makes sense.  I’ll try to go with what Kevin suggested, i.e. have @iothread
> be an alternate type.
> 
> Hanna
> 
> > 
> > Eric: Are you interested in implementing support for multiple IOThreads
> > in the NBD export? I remember some time ago we talked about NBD
> > multi-conn support, although maybe that was for the client rather than
> > the server.
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]