Should FUSE set IO_FLUSHER for the userspace process?

Should FUSE set IO_FLUSHER for the userspace process?

[Nikolaus Rath]

Hi,

Should the FUSE kernel driver perhaps set PR_SET_IO_FLUSHER for the FUSE
userspace process daemon when a connection is opened?

If I understand correctly, this is necessary to avoid a deadlocks if the
kernel needs to reclaim memory that has to be written back through FUSE.

I don't think it's possible to do this in userspace, since the process
may lack the necessary capabilities.

Best,
-Nikolaus

-- 
GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: Should FUSE set IO_FLUSHER for the userspace process?

[Amir Goldstein]

On Sun, Sep 18, 2022 at 2:12 PM Nikolaus Rath <Nikolaus@rath.org> wrote:
>
> Hi,
>
> Should the FUSE kernel driver perhaps set PR_SET_IO_FLUSHER for the FUSE
> userspace process daemon when a connection is opened?
>

This may have implications.
Not sure it is good to do that unconditionally.
Do you have a reproducer of deadlock?

Note that PR_SET_IO_FLUSHER is set for a thread - not for a process.

> If I understand correctly, this is necessary to avoid a deadlocks if the
> kernel needs to reclaim memory that has to be written back through FUSE.
>

I think the deadlock can happen also when kernel does writeback to FUSE
and then FUSE server ends up trying to do direct reclaim of memory
(not just through FUSE writeback).

In any case, I guess the problem is limited for writeback cache enabled.

> I don't think it's possible to do this in userspace, since the process
> may lack the necessary capabilities.
>

Doing something a bit more subtle would be to write the calling thread
PF_MEMALLOC_NOXX flags in the request and let kernel set the
flags of the thread reading the request, but in libfuse that is not the
worker thread is it?

Need to also check if PF_MEMALLOC_NOFS may also deadlock.
Direct reclaim of inodes can trigger a *lot* of FUSE_FORGET requests
IIRC those requests are one way, but can they block on the request
queue size limit?

Of course it is less common for the forget method to require large
memory allocations, but we cannot assume there is no deadlock
unless we make sure to prevent it.

Thanks,
Amir.

Re: Should FUSE set IO_FLUSHER for the userspace process?

[Miklos Szeredi]

On Sun, 18 Sept 2022 at 13:03, Nikolaus Rath <Nikolaus@rath.org> wrote:
>
> Hi,
>
> Should the FUSE kernel driver perhaps set PR_SET_IO_FLUSHER for the FUSE
> userspace process daemon when a connection is opened?
>
> If I understand correctly, this is necessary to avoid a deadlocks if the
> kernel needs to reclaim memory that has to be written back through FUSE.

The fuse kernel driver is careful to avoid such deadlocks.  When
memory reclaim happens, it copies data to temporary buffers and
immediately finishes the reclaim from the memory management
subsystem's point of view.   The temp buffers are then sent to
userspace and written back without having to worry about deadlocks.
There are lots of details missing from the above description, but this
is the essence of the writeback deadlock avoidance.

Thanks,
Miklos

Re: Should FUSE set IO_FLUSHER for the userspace process?

[Antonio SJ Musumeci]

On 9/19/22 05:20, Miklos Szeredi wrote:
> On Sun, 18 Sept 2022 at 13:03, Nikolaus Rath <Nikolaus@rath.org> wrote:
>> Hi,
>>
>> Should the FUSE kernel driver perhaps set PR_SET_IO_FLUSHER for the FUSE
>> userspace process daemon when a connection is opened?
>>
>> If I understand correctly, this is necessary to avoid a deadlocks if the
>> kernel needs to reclaim memory that has to be written back through FUSE.
> The fuse kernel driver is careful to avoid such deadlocks.  When
> memory reclaim happens, it copies data to temporary buffers and
> immediately finishes the reclaim from the memory management
> subsystem's point of view.   The temp buffers are then sent to
> userspace and written back without having to worry about deadlocks.
> There are lots of details missing from the above description, but this
> is the essence of the writeback deadlock avoidance.
>
> Thanks,
> Miklos

Miklos, does this mean that FUSE servers shouldn't bother setting 
PR_SET_IO_FLUSHER? Are there any benefits to setting it explicitly or 
detriments to not setting it?

Re: Should FUSE set IO_FLUSHER for the userspace process?

[Miklos Szeredi]

On Tue, 25 Oct 2022 at 17:39, Antonio SJ Musumeci <trapexit@spawn.link> wrote:
>
> On 9/19/22 05:20, Miklos Szeredi wrote:
> > On Sun, 18 Sept 2022 at 13:03, Nikolaus Rath <Nikolaus@rath.org> wrote:
> >> Hi,
> >>
> >> Should the FUSE kernel driver perhaps set PR_SET_IO_FLUSHER for the FUSE
> >> userspace process daemon when a connection is opened?
> >>
> >> If I understand correctly, this is necessary to avoid a deadlocks if the
> >> kernel needs to reclaim memory that has to be written back through FUSE.
> > The fuse kernel driver is careful to avoid such deadlocks.  When
> > memory reclaim happens, it copies data to temporary buffers and
> > immediately finishes the reclaim from the memory management
> > subsystem's point of view.   The temp buffers are then sent to
> > userspace and written back without having to worry about deadlocks.
> > There are lots of details missing from the above description, but this
> > is the essence of the writeback deadlock avoidance.
> >
> > Thanks,
> > Miklos
>
> Miklos, does this mean that FUSE servers shouldn't bother setting
> PR_SET_IO_FLUSHER? Are there any benefits to setting it explicitly or
> detriments to not setting it?

PR_SET_IO_FLUHSER internally sets the process flags PF_MEMALLOC_NOIO
and PF_LOCAL_THROTTLE.

The former is clear: don't try to initiate I/O when memory needs to be
reclaimed.  This could be detrimental in low memory situations, since
the kernel has less choice for freeing up memory.

PF_LOCAL_THROTTLE  seems to mean "don't throttle dirtying pages
(writes) by this process, since that would throttle the cleaning of
dirty pages."   This logic seems valid for fuse as well, but it also
upsets the normal dirty throttling mechanisms, so I'm not sure that
there aren't any side effects.

Thanks,
Miklos

Re: Should FUSE set IO_FLUSHER for the userspace process?

[Miklos Szeredi]

On Wed, 26 Oct 2022 at 13:58, Miklos Szeredi <miklos@szeredi.hu> wrote:
>
> On Tue, 25 Oct 2022 at 17:39, Antonio SJ Musumeci <trapexit@spawn.link> wrote:
> >
> > On 9/19/22 05:20, Miklos Szeredi wrote:
> > > On Sun, 18 Sept 2022 at 13:03, Nikolaus Rath <Nikolaus@rath.org> wrote:
> > >> Hi,
> > >>
> > >> Should the FUSE kernel driver perhaps set PR_SET_IO_FLUSHER for the FUSE
> > >> userspace process daemon when a connection is opened?
> > >>
> > >> If I understand correctly, this is necessary to avoid a deadlocks if the
> > >> kernel needs to reclaim memory that has to be written back through FUSE.
> > > The fuse kernel driver is careful to avoid such deadlocks.  When
> > > memory reclaim happens, it copies data to temporary buffers and
> > > immediately finishes the reclaim from the memory management
> > > subsystem's point of view.   The temp buffers are then sent to
> > > userspace and written back without having to worry about deadlocks.
> > > There are lots of details missing from the above description, but this
> > > is the essence of the writeback deadlock avoidance.
> > >
> > > Thanks,
> > > Miklos
> >
> > Miklos, does this mean that FUSE servers shouldn't bother setting
> > PR_SET_IO_FLUSHER? Are there any benefits to setting it explicitly or
> > detriments to not setting it?
>
> PR_SET_IO_FLUHSER internally sets the process flags PF_MEMALLOC_NOIO
> and PF_LOCAL_THROTTLE.
>
> The former is clear: don't try to initiate I/O when memory needs to be
> reclaimed.  This could be detrimental in low memory situations, since
> the kernel has less choice for freeing up memory.
>
> PF_LOCAL_THROTTLE  seems to mean "don't throttle dirtying pages
> (writes) by this process, since that would throttle the cleaning of
> dirty pages."   This logic seems valid for fuse as well, but it also
> upsets the normal dirty throttling mechanisms, so I'm not sure that
> there aren't any side effects.

Also consider:  when a fuse page is under writeback, it's already
accounted as "clean" for the purposes reclaim and for throttling other
dirtiers.  Throttling of fuse fuse writeback pages (NR_WRITEBACK_TEMP)
is done completely separately.

So I'd say, it's better not to set PR_SET_IO_FLUSHER on the fuse
daemon, although there probably wouldn't be any catastrophic
consequences of setting it.

Thanks,
Miklos