fuse: fuse semantics wrt stalled requests

fuse: fuse semantics wrt stalled requests

[Sergey Senozhatsky]

Hello Miklos,

A question: does fuse define any semantics for stalled requests handling?

We are currently looking at a number of hung_task watchdog crashes with
tasks waiting forever in d_wait_lookup() for dentries to lose PAR_LOOKUP
state, and we suspect that those dentries are from fuse mount point
(we also sometimes see hung_tasks in fuse_lookup()->fuse_simple_request()).
Supposedly (a theory) some tasks are in request_wait_answer() under
PAR_LOOKUP, and the rest of tasks are waiting for them to finish and clear
PAR_LOOKUP bit.

request_wait_answer() waits indefinitely, however, the interesting
thing is that it uses wait_event_interruptible() (when we wait for
!fc->no_interrupt request to be processed).  What is the idea behind
interruptible wait?  Is this, may be, for stall requests handling?
Does fuse expect user-space to watchdog or monitor its processes/threads
that issue syscalls on fuse mount points and, e.g., SIGKILL stalled ones?

To make things even more complex, in our particular case fuse mount
point mounts a remote google driver, so it become a network fs in
some sense, which adds a whole new dimension of possibilities for
stalled/failed requests.  How those are expected to be handled?  Should
fuse still wait indefinitely or would it make sense to add a timeout
to request_wait_answer() and FR_INTERRUPTED timeout-ed requests?

Re: fuse: fuse semantics wrt stalled requests

[Bernd Schubert]

On 11/28/24 04:54, Sergey Senozhatsky wrote:
> Hello Miklos,
> 
> A question: does fuse define any semantics for stalled requests handling?
> 
> We are currently looking at a number of hung_task watchdog crashes with
> tasks waiting forever in d_wait_lookup() for dentries to lose PAR_LOOKUP
> state, and we suspect that those dentries are from fuse mount point
> (we also sometimes see hung_tasks in fuse_lookup()->fuse_simple_request()).
> Supposedly (a theory) some tasks are in request_wait_answer() under
> PAR_LOOKUP, and the rest of tasks are waiting for them to finish and clear
> PAR_LOOKUP bit.
> 
> request_wait_answer() waits indefinitely, however, the interesting
> thing is that it uses wait_event_interruptible() (when we wait for
> !fc->no_interrupt request to be processed).  What is the idea behind
> interruptible wait?  Is this, may be, for stall requests handling?
> Does fuse expect user-space to watchdog or monitor its processes/threads
> that issue syscalls on fuse mount points and, e.g., SIGKILL stalled ones?
> 
> To make things even more complex, in our particular case fuse mount
> point mounts a remote google driver, so it become a network fs in
> some sense, which adds a whole new dimension of possibilities for
> stalled/failed requests.  How those are expected to be handled?  Should
> fuse still wait indefinitely or would it make sense to add a timeout
> to request_wait_answer() and FR_INTERRUPTED timeout-ed requests?
> 

Please see here

https://lore.kernel.org/all/20241114191332.669127-1-joannelkoong@gmail.com/



Thanks,
Bernd

Re: fuse: fuse semantics wrt stalled requests

[Sergey Senozhatsky]

On (24/11/28 11:29), Bernd Schubert wrote:
> On 11/28/24 04:54, Sergey Senozhatsky wrote:
> > Hello Miklos,
> > 
> > A question: does fuse define any semantics for stalled requests handling?
> > 
> > We are currently looking at a number of hung_task watchdog crashes with
> > tasks waiting forever in d_wait_lookup() for dentries to lose PAR_LOOKUP
> > state, and we suspect that those dentries are from fuse mount point
> > (we also sometimes see hung_tasks in fuse_lookup()->fuse_simple_request()).
> > Supposedly (a theory) some tasks are in request_wait_answer() under
> > PAR_LOOKUP, and the rest of tasks are waiting for them to finish and clear
> > PAR_LOOKUP bit.
> > 
> > request_wait_answer() waits indefinitely, however, the interesting
> > thing is that it uses wait_event_interruptible() (when we wait for
> > !fc->no_interrupt request to be processed).  What is the idea behind
> > interruptible wait?  Is this, may be, for stall requests handling?
> > Does fuse expect user-space to watchdog or monitor its processes/threads
> > that issue syscalls on fuse mount points and, e.g., SIGKILL stalled ones?
> > 
> > To make things even more complex, in our particular case fuse mount
> > point mounts a remote google driver, so it become a network fs in
> > some sense, which adds a whole new dimension of possibilities for
> > stalled/failed requests.  How those are expected to be handled?  Should
> > fuse still wait indefinitely or would it make sense to add a timeout
> > to request_wait_answer() and FR_INTERRUPTED timeout-ed requests?
> > 
> 
> Please see here
> 
> https://lore.kernel.org/all/20241114191332.669127-1-joannelkoong@gmail.com/

Thanks for the pointers!