Re: [fuse-devel] fuse: max_background and congestion_threshold settings

[Nikolaus Rath <Nikolaus@rath.org>]

Hi Maxim,

On Nov 22 2016, Maxim Patlasov <mpatlasov@virtuozzo.com> wrote:
>>>>>>>> Could someone explain to me the meaning of the max_background and
>>>>>>>> congestion_threshold settings of the fuse module?
>>>>>>>>
>>>>>>>> At first I assumed that max_background specifies the maximum number of
>>>>>>>> pending requests (i.e., requests that have been send to userspace but
>>>>>>>> for which no reply was received yet). But looking at fs/fuse/dev.c, it
>>>>>>>> looks as if not every request is included in this number.
>>>>>>> fuse uses max_background for cases where the total number of
>>>>>>> simultaneous requests of given type is not limited by some other
>>>>>>> natural means. AFAIU, these cases are: 1) async processing of direct
>>>>>>> IO; 2) read-ahead. As an example of "natural" limitation: when
>>>>>>> userspace process blocks on a sync direct IO read/write, the number of
>>>>>>> requests fuse consumed is limited by the number of such processes
>>>>>>> (actually their threads). In contrast, if userspace requests 1GB
>>>>>>> direct IO read/write, it would be unreasonable to issue 1GB/128K==8192
>>>>>>> fuse requests simultaneously. That's where max_background steps in.
>>>>>> Ah, that makes sense. Are these two cases meant as examples, or is that
>>>>>> an exhaustive list? Because I would have thought that other cases should
>>>>>> be writing of cached data (when writeback caching is enabled), and
>>>>>> asynchronous I/O from userspace...?
>>>>> I think that's exhaustive list, but I can miss something.
>>>>>
>>>>> As for writing of cached data, that definitely doesn't go through
>>>>> background requests. Here we rely on flusher: fuse will allocate as
>>>>> many requests as the flusher wants to writeback.
>>>>>
>>>>> Buffered AIO READs actually block in submit_io until fully
>>>>> processed. So it's just another example of "natural" limitation I told
>>>>> above.
>>>> Not sure I understand. What is it that's blocking? It can't be the
>>>> userspace process, because then it wouldn't be asynchronous I/O...
>>> Surprise! Alas, Linux kernel does NOT process buffered AIO reads in
>>> async manner. You can verify it yourself by strace-ing a simple
>>> program looping over io_submit + io_getevents: for direct IO (as
>>> expected) io_submit returns immediately while io_getevents waits for
>>> actual IO; in contrast, for buffered IO (surprisingly) io_submit waits
>>> for actual IO while io_getevents returns immediately. Presumably,
>>> people are supposed to use mmap-ed read/writes rather than buffered
>>> AIO.
>> What about buffered, asynchronous writes when writeback cache is
>> disabled? It sounds as if io_submit does not block (so userspace could
>> create an unlimited number), nor can the kernel coalesce them (since
>> writeback caching is disabled).
>
> I've never looked closely at it. Do you have a particular use case or
> concern?

My only concern is to accurately describe the effects of the
"max_background" parameter in the libfuse documentation.

At the moment most FUSE filesystems don't use writeback caching (because
there is no stable libfuse release out that supports it). On the other
hand, most filesystem are probably also not too worried about the
behavior when userspace submits a large number of asynchronous write
requests. But I think it would still be important to correctly describe
this case. If io_submit does not block, and the request does not count
as a background request, wouldn't this be a bug that should be fixed? Or
is there anything else that would limit the number of such requests?

Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«