Re: Large CQE for fuse headers

[Pavel Begunkov <asml.silence@gmail.com>]

On 10/14/24 13:47, Bernd Schubert wrote:
> On 10/14/24 13:10, Miklos Szeredi wrote:
>> On Mon, 14 Oct 2024 at 04:44, Ming Lei <tom.leiming@gmail.com> wrote:
>>
>>> It also depends on how fuse user code consumes the big CQE payload, if
>>> fuse header needs to keep in memory a bit long, you may have to copy it
>>> somewhere for post-processing since io_uring(kernel) needs CQE to be
>>> returned back asap.
>>
>> Yes.
>>
>> I'm not quite sure how the libfuse interface will work to accommodate
>> this.  Currently if the server needs to delay the processing of a
>> request it would have to copy all arguments, since validity will not
>> be guaranteed after the callback returns.  With the io_uring
>> infrastructure the headers would need to be copied, but the data
>> buffer would be per-request and would not need copying.  This is
>> relaxing a requirement so existing servers would continue to work
>> fine, but would not be able to take full advantage of the multi-buffer
>> design.
>>
>> Bernd do you have an idea how this would work?
> 
> I assume returning a CQE is io_uring_cq_advance()?

Yes

> In my current libfuse io_uring branch that only happens when
> all CQEs have been processed. We could also easily switch to
> io_uring_cqe_seen() to do it per CQE.

Either that one.

> I don't understand why we need to return CQEs asap, assuming CQ
> ring size is the same as SQ ring size - why does it matter?

The SQE is consumed once the request is issued, but nothing
prevents the user to keep the QD larger than the SQ size,
e.g. do M syscalls each ending N requests and then wait for
N * M completions.

> If we indeed need to return the CQE before processing the request,
> it indeed would be better to have a 2nd memory buffer associated with
> the fuse request.

With that said, the usual problem is to size the CQ so that it
(almost) never overflows, otherwise it hurts performance. With
DEFER_TASKRUN you can delay returning CQEs to the kernel until
the next time you wait for completions, i.e. do io_uring waiting
syscall. Without the flag, CQEs may come asynchronously to the
user, so need a bit more consideration.

-- 
Pavel Begunkov