Re: [PATCH v2 00/21] export/fuse: Use coroutines and multi-threading

[Stefan Hajnoczi <stefanha@redhat.com>]

[-- Attachment #1: Type: text/plain, Size: 7425 bytes --]

On Wed, Jun 04, 2025 at 03:27:52PM +0200, Hanna Czenczek wrote:
> Hi,
> 
> This series:
> - Fixes some bugs/minor inconveniences,
> - Removes libfuse from the request processing path,
> - Make the FUSE export use coroutines for request handling,
> - Introduces multi-threading into the FUSE export.
> 
> More detail on the v1 cover letter:
> 
> https://lists.nongnu.org/archive/html/qemu-block/2025-03/msg00359.html
> 
> 
> Changes from v1:
> - Patch 1: Clarified “polling” to be `aio_poll()`
> - Patch 11 (new): Pulled out from patch 13 (prev. 11)
> - Patch 12 (new): Suggested by Eric
> - Patch 13 (prev. 11):
>   - Drop false polling handlers
>   - Use qemu_fcntl_addfl() instead of fcntl(F_SETFL) to keep
>     pre-existing FD flags
>   - Add a note that the buffer returned by read needs to be freed via
>     qemu_vfree()
>   - Pulled out a variable rename into the new patch 11
> - Patch 15 (prev. 13):
>   - Simplified the co_read_from_fuse_fd() interface thanks to no longer
>     needing to support poll handlers
>   - Increment in-flight counter before entering the coroutine to make it
>     more obvious how tihs ensures that the export pointer remains valid
>     throughout
> - Patch 16 (new): Add a common multi-threading interface for exports
>   instead of a specific one just for FUSE
> - Patch 17 (new): Test cases for this new interface
> - Patch 18 (prev. 14):
>   - Drop the contrasting with virtio-blk from the commit message;
>     explaining the interface is no longer necessary now that it’s
>     introduced separately in patch 16.
>   - Generally, the interface definition is removed in favor of the new
>     one in patch 16.
>   - Some rebase conflicts (due to other changes earlier in this series).
> - Patch 19 (new): Stefan suggested adding an explicit note for users on
>   how multi-threading behaves with FUSE, not least because in the future
>   this behavior may depend on the specific implementation features
>   chosen (io-uring or not).  Because the actual multi-thread interface
>   is now on the common export options, it is no longer obvious where to
>   put this implementation note; I decided to put it into the general
>   description of the FUSE export options, inside of this dedicated
>   patch.
> - Patch 20 (new): Simple sanity test for FUSE multi-threading (just test
>   that e.g. nothing crashes when running qemu-img bench on top)
> - Patch 21 (prev. 15): Rebase conflict due to the changes in patch 15;
>   kept Stefan’s R-b anyway
> 
> 
> Review notes/suggestions I deliberately did not follow in v2:
> - Stefan suggested to make patch 1 simpler and more robust by allocating
>   a new buffer for each request.  This is indeed a simple change (for
>   patch 1) that I wouldn’t mind, and that I also started to implement.
>   However, eventually I decided against it:
>   The problem doesn’t disappear with the rest of the series, it
>   basically stays the exact same; though instead of an implicit
>   aio_poll() leading to nested polling, it turns into an implicit
>   coroutine yield doing pretty much the same.
>   For performance, it is better not to allocate a new buffer for each
>   request; we only really need a bounce buffer for writes, as there is
>   no other case where we’d continue to read the request buffer after
>   yielding.  Therefore, the final state I would like to have after this
>   series is to use a common request buffer for all requests on a single
>   queue, only using a bounce buffer for writes.
>   With that, I think it’s better to implement exactly that right from
>   the start, rather than introducing a new intermediate state.
> 
> 
> git-backport-diff from v1:
> 
> Key:
> [----] : patches are identical
> [####] : number of functional differences between upstream/downstream patch
> [down] : patch is downstream-only
> The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
> 
> 001/21:[0012] [FC] 'fuse: Copy write buffer content before polling'
> 002/21:[----] [--] 'fuse: Ensure init clean-up even with error_fatal'
> 003/21:[----] [--] 'fuse: Remove superfluous empty line'
> 004/21:[----] [--] 'fuse: Explicitly set inode ID to 1'
> 005/21:[----] [--] 'fuse: Change setup_... to mount_fuse_export()'
> 006/21:[----] [--] 'fuse: Fix mount options'
> 007/21:[----] [--] 'fuse: Set direct_io and parallel_direct_writes'
> 008/21:[----] [--] 'fuse: Introduce fuse_{at,de}tach_handlers()'
> 009/21:[----] [--] 'fuse: Introduce fuse_{inc,dec}_in_flight()'
> 010/21:[----] [--] 'fuse: Add halted flag'
> 011/21:[down] 'fuse: Rename length to blk_len in fuse_write()'
> 012/21:[down] 'block: Move qemu_fcntl_addfl() into osdep.c'
> 013/21:[0077] [FC] 'fuse: Manually process requests (without libfuse)'
> 014/21:[----] [--] 'fuse: Reduce max read size'
> 015/21:[0061] [FC] 'fuse: Process requests in coroutines'
> 016/21:[down] 'block/export: Add multi-threading interface'
> 017/21:[down] 'iotests/307: Test multi-thread export interface'
> 018/21:[0077] [FC] 'fuse: Implement multi-threading'
> 019/21:[down] 'qapi/block-export: Document FUSE's multi-threading'
> 020/21:[down] 'iotests/308: Add multi-threading sanity test'
> 021/21:[0002] [FC] 'fuse: Increase MAX_WRITE_SIZE with a second buffer'
> 
> 
> Hanna Czenczek (21):
>   fuse: Copy write buffer content before polling
>   fuse: Ensure init clean-up even with error_fatal
>   fuse: Remove superfluous empty line
>   fuse: Explicitly set inode ID to 1
>   fuse: Change setup_... to mount_fuse_export()
>   fuse: Fix mount options
>   fuse: Set direct_io and parallel_direct_writes
>   fuse: Introduce fuse_{at,de}tach_handlers()
>   fuse: Introduce fuse_{inc,dec}_in_flight()
>   fuse: Add halted flag
>   fuse: Rename length to blk_len in fuse_write()
>   block: Move qemu_fcntl_addfl() into osdep.c
>   fuse: Manually process requests (without libfuse)
>   fuse: Reduce max read size
>   fuse: Process requests in coroutines
>   block/export: Add multi-threading interface
>   iotests/307: Test multi-thread export interface
>   fuse: Implement multi-threading
>   qapi/block-export: Document FUSE's multi-threading
>   iotests/308: Add multi-threading sanity test
>   fuse: Increase MAX_WRITE_SIZE with a second buffer
> 
>  qapi/block-export.json               |   39 +-
>  include/block/export.h               |   12 +-
>  include/qemu/osdep.h                 |    1 +
>  block/export/export.c                |   48 +-
>  block/export/fuse.c                  | 1181 ++++++++++++++++++++------
>  block/export/vduse-blk.c             |    7 +
>  block/export/vhost-user-blk-server.c |    8 +
>  block/file-posix.c                   |   17 +-
>  nbd/server.c                         |    6 +
>  util/osdep.c                         |   18 +
>  tests/qemu-iotests/307               |   47 +
>  tests/qemu-iotests/307.out           |   18 +
>  tests/qemu-iotests/308               |   55 +-
>  tests/qemu-iotests/308.out           |   61 +-
>  14 files changed, 1213 insertions(+), 305 deletions(-)
> 
> -- 
> 2.49.0
> 

Nice, looks like there are some goodies in here that Brian can use in
his FUSE-over-io_uring project, like the new iothread=['iothread0',
'iothread1', ...] syntax for specifying multiple IOThreads.

Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]