Re: Large CQE for fuse headers

[Jens Axboe <axboe@kernel.dk>]

On 10/11/24 1:03 PM, Bernd Schubert wrote:
>>
>> Curious, what's all the space needed for?
> 
> The basic fuse header: struct fuse_in_header -> current 40B
> and per request header headers, I think current max is 64.
> 
> And then some extra compat space for both, so that they can be safely
> extended in the future (which is currently an issue).

So that's 104b, and regular CQE stuff too I presume, so that's 104+16 ==
120 bytes. That'd fit in a 128b CQE, and 256b would be pleeeeenty? Just
squeeze a version field or something in there so you know what the
version is for future proofing? I would strongly recommend making it as
large as you need it for those things, but no longer just for
compat/future reasons. Eg 128b over 256b is a win for sure, and 256b
over 512b is a REALLY nice win.

>>>> Since this obviously means larger CQ rings, one nice side effect is that
>>>> since 6.10 we don't need contig pages to map any of the rings. So should
>>>> work just fine regardless of memory fragmentation, where previously that
>>>> would've been a concern.
>>>>
>>>
>>> Out of interest, what is the change? Up to fuse-io-uring rfc2 I was
>>> vmalloced buffers for fuse that got mmaped - was working fine. Miklos just
>>> wants to avoid that kernel allocates large chunks of memory on behalf of
>>> users.
>>
>> It was the change that got rid of remap_pfn_range() for mapping, and
>> switched to vm_insert_page(s) instead. Memory overhead should generally
>> not be too bad, it's all about sizing the rings appropriately. The much
>> bigger concern is needing contig memory, as that can become scarce after
>> longer uptimes, even with plenty of memory free. This is particularly
>> important if you need 512b CQEs, obviously.
>>
> 
> For sure, I was just curious what you had changed. I think I had looked into
> that io-uring code around 2 years ago.  Going to look into the update
> io-uring code, thanks for the hint.
> For fuse I was just using remap_vmalloc_range().
> 
> https://lore.kernel.org/all/20240529-fuse-uring-for-6-9-rfc2-out-v1-7-d149476b1d65@ddn.com/

That's the one to use, io_uring was just stuck with using the wrong API
for quite a while, but that got sorted.

-- 
Jens Axboe