Re: [PATCH 11/15] fuse: Manually process requests (without libfuse)

[Hanna Czenczek <hreitz@redhat.com>]

On 27.03.25 16:35, Stefan Hajnoczi wrote:
> On Tue, Mar 25, 2025 at 05:06:51PM +0100, Hanna Czenczek wrote:
>> Manually read requests from the /dev/fuse FD and process them, without
>> using libfuse.  This allows us to safely add parallel request processing
>> in coroutines later, without having to worry about libfuse internals.
>> (Technically, we already have exactly that problem with
>> read_from_fuse_export()/read_from_fuse_fd() nesting.)
>>
>> We will continue to use libfuse for mounting the filesystem; fusermount3
>> is a effectively a helper program of libfuse, so it should know best how
>> to interact with it.  (Doing it manually without libfuse, while doable,
>> is a bit of a pain, and it is not clear to me how stable the "protocol"
>> actually is.)
>>
>> Take this opportunity of quite a major rewrite to update the Copyright
>> line with corrected information that has surfaced in the meantime.
>>
>> Here are some benchmarks from before this patch (4k, iodepth=16, libaio;
>> except 'sync', which are iodepth=1 and pvsync2):
>>
>> file:
>>    read:
>>      seq aio:   78.6k ±1.3k IOPS
>>      rand aio:  39.3k ±2.9k
>>      seq sync:  32.5k ±0.7k
>>      rand sync:  9.9k ±0.1k
>>    write:
>>      seq aio:   61.9k ±0.5k
>>      rand aio:  61.2k ±0.6k
>>      seq sync:  27.9k ±0.2k
>>      rand sync: 27.6k ±0.4k
>> null:
>>    read:
>>      seq aio:   214.0k ±5.9k
>>      rand aio:  212.7k ±4.5k
>>      seq sync:   90.3k ±6.5k
>>      rand sync:  89.7k ±5.1k
>>    write:
>>      seq aio:   203.9k ±1.5k
>>      rand aio:  201.4k ±3.6k
>>      seq sync:   86.1k ±6.2k
>>      rand sync:  84.9k ±5.3k
>>
>> And with this patch applied:
>>
>> file:
>>    read:
>>      seq aio:   76.6k ±1.8k (- 3 %)
>>      rand aio:  26.7k ±0.4k (-32 %)
>>      seq sync:  47.7k ±1.2k (+47 %)
>>      rand sync: 10.1k ±0.2k (+ 2 %)
>>    write:
>>      seq aio:   58.1k ±0.5k (- 6 %)
>>      rand aio:  58.1k ±0.5k (- 5 %)
>>      seq sync:  36.3k ±0.3k (+30 %)
>>      rand sync: 36.1k ±0.4k (+31 %)
>> null:
>>    read:
>>      seq aio:   268.4k ±3.4k (+25 %)
>>      rand aio:  265.3k ±2.1k (+25 %)
>>      seq sync:  134.3k ±2.7k (+49 %)
>>      rand sync: 132.4k ±1.4k (+48 %)
>>    write:
>>      seq aio:   275.3k ±1.7k (+35 %)
>>      rand aio:  272.3k ±1.9k (+35 %)
>>      seq sync:  130.7k ±1.6k (+52 %)
>>      rand sync: 127.4k ±2.4k (+50 %)
>>
>> So clearly the AIO file results are actually not good, and random reads
>> are indeed quite terrible.  On the other hand, we can see from the sync
>> and null results that request handling should in theory be quicker.  How
>> does this fit together?
>>
>> I believe the bad AIO results are an artifact of the accidental parallel
>> request processing we have due to nested polling: Depending on how the
>> actual request processing is structured and how long request processing
>> takes, more or less requests will be submitted in parallel.  So because
>> of the restructuring, I think this patch accidentally changes how many
>> requests end up being submitted in parallel, which decreases
>> performance.
>>
>> (I have seen something like this before: In RSD, without having
>> implemented a polling mode, the debug build tended to have better
>> performance than the more optimized release build, because the debug
>> build, taking longer to submit requests, ended up processing more
>> requests in parallel.)
>>
>> In any case, once we use coroutines throughout the code, performance
>> will improve again across the board.
>>
>> Signed-off-by: Hanna Czenczek <hreitz@redhat.com>
>> ---
>>   block/export/fuse.c | 793 +++++++++++++++++++++++++++++++-------------
>>   1 file changed, 567 insertions(+), 226 deletions(-)
>>
>> diff --git a/block/export/fuse.c b/block/export/fuse.c
>> index 3dd50badb3..407b101018 100644
>> --- a/block/export/fuse.c
>> +++ b/block/export/fuse.c

[...]

>> +/**
>> + * Check the FUSE FD for whether it is readable or not.  Because we cannot
>> + * reasonably do this without reading a request at the same time, also read and
>> + * process that request if any.
>> + * (To be used as a poll handler for the FUSE FD.)
>> + */
>> +static bool poll_fuse_fd(void *opaque)
>> +{
>> +    return read_from_fuse_fd(opaque);
>> +}
> The other io_poll() callbacks in QEMU peek at memory whereas this one
> invokes the read(2) syscall. Two reasons why this is a problem:
> 1. Syscall latency is too high. Other fd handlers will be delayed by
>     microseconds.
> 2. This doesn't scale. If every component in QEMU does this then the
>     event loop degrades to O(n) of non-blocking read(2) syscalls where n
>     is the number of fds.
>
> Also, handling the request inside the io_poll() callback skews
> AioContext's time accounting because time spent handling the request
> will be accounted as "polling time". The adaptive polling calculation
> will think it polled for longer than it did.
>
> If there is no way to peek at memory, please don't implement the
> io_poll() callback.

Got it, thanks!

Hanna