Re: [PATCH v2] fuse: permit freezing while waiting for request answer

Re: [PATCH v2] fuse: permit freezing while waiting for request answer

[Miklos Szeredi]

[Cc: fsdevel, Peter Z., Rafael]

On Wed, 19 Aug 2026 at 04:35, Sergey Senozhatsky
<senozhatsky@chromium.org> wrote:
>
> Suspend freezes tasks in random order and doesn't take into
> consideration producer-consumer dependency that may exist
> between tasks.  One example where this can cause issues is:
> fuse server getting frozen ahead of clients, which then get
> stuck waiting for req answers that never come (the server
> is already frozen).
>
> Make all wait-event calls in request_wait_answer() freezer-friendly.
>
> This, however, doesn't address all cases.  E.g. in-place
> PM-freeze of a request_wait_answer() task holding a contended
> VFS lock still will block suspend.
>
> Note: this uses TASK_FREEZABLE, not TASK_FREEZABLE_UNSAFE,
> which may trigger debug_locks warning during suspend (if
> request_wait_answer() task holds some locks at the time
> of freeze.)

This is not okay.  I see the "no new users" warning on
TASK_FREEZABLE_UNSAFE, but this needs further discussion.

Existing users of the _UNSAFE variant are NFS and samba.   I haven't
checked the context where these are called.

Apparently __sb_start_write() uses the safe variant, yet I'm quite
sure it will be called in various locking contexts.

Why is this unsafe exactly?  Does that unsafeness apply to
filesystems?  If so why do we allow freezing while blocked on
sb_start_write()?

Thanks,
Miklos

>
> Signed-off-by: Sergey Senozhatsky <senozhatsky@chromium.org>
> ---
>
>  fs/fuse/dev.c | 13 ++++++++-----
>  1 file changed, 8 insertions(+), 5 deletions(-)
>
> diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
> index 34106f6e66a0..42098ddc2587 100644
> --- a/fs/fuse/dev.c
> +++ b/fs/fuse/dev.c
> @@ -701,8 +701,9 @@ static void request_wait_answer(struct fuse_req *req)
>
>         if (!fch->no_interrupt) {
>                 /* Any signal may interrupt this */
> -               err = wait_event_interruptible(req->waitq,
> -                                       test_bit(FR_FINISHED, &req->flags));
> +               err = wait_event_state(req->waitq,
> +                                      test_bit(FR_FINISHED, &req->flags),
> +                                      (TASK_INTERRUPTIBLE | TASK_FREEZABLE));
>                 if (!err)
>                         return;
>
> @@ -717,8 +718,9 @@ static void request_wait_answer(struct fuse_req *req)
>                 bool removed;
>
>                 /* Only fatal signals may interrupt this */
> -               err = wait_event_killable(req->waitq,
> -                                       test_bit(FR_FINISHED, &req->flags));
> +               err = wait_event_state(req->waitq,
> +                                      test_bit(FR_FINISHED, &req->flags),
> +                                      (TASK_KILLABLE | TASK_FREEZABLE));
>                 if (!err)
>                         return;
>
> @@ -740,7 +742,8 @@ static void request_wait_answer(struct fuse_req *req)
>          * Either request is already in userspace, or it was forced.
>          * Wait it out.
>          */
> -       wait_event(req->waitq, test_bit(FR_FINISHED, &req->flags));
> +       wait_event_state(req->waitq, test_bit(FR_FINISHED, &req->flags),
> +                        (TASK_UNINTERRUPTIBLE | TASK_FREEZABLE));
>  }
>
>  static void __fuse_request_send(struct fuse_req *req)
> --
> 2.55.0.737.g08866a6d13-goog
>

Re: [PATCH v2] fuse: permit freezing while waiting for request answer

[Sergey Senozhatsky]

On (26/08/19 12:03), Miklos Szeredi wrote:
> On Wed, 19 Aug 2026 at 04:35, Sergey Senozhatsky
> <senozhatsky@chromium.org> wrote:
> >
> > Suspend freezes tasks in random order and doesn't take into
> > consideration producer-consumer dependency that may exist
> > between tasks.  One example where this can cause issues is:
> > fuse server getting frozen ahead of clients, which then get
> > stuck waiting for req answers that never come (the server
> > is already frozen).
> >
> > Make all wait-event calls in request_wait_answer() freezer-friendly.
> >
> > This, however, doesn't address all cases.  E.g. in-place
> > PM-freeze of a request_wait_answer() task holding a contended
> > VFS lock still will block suspend.
> >
> > Note: this uses TASK_FREEZABLE, not TASK_FREEZABLE_UNSAFE,
> > which may trigger debug_locks warning during suspend (if
> > request_wait_answer() task holds some locks at the time
> > of freeze.)
> 
> This is not okay.  I see the "no new users" warning on
> TASK_FREEZABLE_UNSAFE, but this needs further discussion.

Sure.  In RFC patch I had TASK_FREEZABLE_UNSAFE but eventually
"don't add new users" won.

> Existing users of the _UNSAFE variant are NFS and samba.   I haven't
> checked the context where these are called.
> 
> Apparently __sb_start_write() uses the safe variant, yet I'm quite
> sure it will be called in various locking contexts.

We also use "safe" variant in fuse_get_req().

> Why is this unsafe exactly?  Does that unsafeness apply to
> filesystems?  If so why do we allow freezing while blocked on
> sb_start_write()?

Right, I don't have much to add to the point that maybe file-systems
can get a waiver.  If we suspend under un-contended VFS lock then it
doesn't look like unsafe here (we similarly can sleep indefinitely
under the same lock waiting for server reply); if the lock is contended
then suspend will fail.

Re: [PATCH v2] fuse: permit freezing while waiting for request answer

[Peter Zijlstra]

On Wed, Aug 19, 2026 at 12:03:32PM +0200, Miklos Szeredi wrote:
> Why is this unsafe exactly?  Does that unsafeness apply to
> filesystems?  If so why do we allow freezing while blocked on
> sb_start_write()?

Getting frozen with lock A held, while another task is blocked on A in
an unfreezable state results in the system not being freezable.

IOW you deadlock the freeze.

Re: [PATCH v2] fuse: permit freezing while waiting for request answer

[Sergey Senozhatsky]

On (26/08/20 10:49), Peter Zijlstra wrote:
> Message-ID: <20260820084933.GA4036497@noisy.programming.kicks-ass.net>
> 
> On Wed, Aug 19, 2026 at 12:03:32PM +0200, Miklos Szeredi wrote:
> > Why is this unsafe exactly?  Does that unsafeness apply to
> > filesystems?  If so why do we allow freezing while blocked on
> > sb_start_write()?
> 
> Getting frozen with lock A held, while another task is blocked on A in
> an unfreezable state results in the system not being freezable.

Right, but then the system says "suspend failed" (tasks refuse to freeze
after 20sec) and just thaws everything back in?

Re: [PATCH v2] fuse: permit freezing while waiting for request answer

[Peter Zijlstra]

On Thu, Aug 20, 2026 at 06:07:18PM +0900, Sergey Senozhatsky wrote:
> On (26/08/20 10:49), Peter Zijlstra wrote:
> > Message-ID: <20260820084933.GA4036497@noisy.programming.kicks-ass.net>
> > 
> > On Wed, Aug 19, 2026 at 12:03:32PM +0200, Miklos Szeredi wrote:
> > > Why is this unsafe exactly?  Does that unsafeness apply to
> > > filesystems?  If so why do we allow freezing while blocked on
> > > sb_start_write()?
> > 
> > Getting frozen with lock A held, while another task is blocked on A in
> > an unfreezable state results in the system not being freezable.
> 
> Right, but then the system says "suspend failed" (tasks refuse to freeze
> after 20sec) and just thaws everything back in?

People don't like suspend failing. People like to close their lid, throw
laptop in bag, and expect laptop to not cook itself to death.

Re: [PATCH v2] fuse: permit freezing while waiting for request answer

[Sergey Senozhatsky]

On (26/08/20 11:10), Peter Zijlstra wrote:
> On Thu, Aug 20, 2026 at 06:07:18PM +0900, Sergey Senozhatsky wrote:
> > On (26/08/20 10:49), Peter Zijlstra wrote:
> > > Message-ID: <20260820084933.GA4036497@noisy.programming.kicks-ass.net>
> > > 
> > > On Wed, Aug 19, 2026 at 12:03:32PM +0200, Miklos Szeredi wrote:
> > > > Why is this unsafe exactly?  Does that unsafeness apply to
> > > > filesystems?  If so why do we allow freezing while blocked on
> > > > sb_start_write()?
> > > 
> > > Getting frozen with lock A held, while another task is blocked on A in
> > > an unfreezable state results in the system not being freezable.
> > 
> > Right, but then the system says "suspend failed" (tasks refuse to freeze
> > after 20sec) and just thaws everything back in?
> 
> People don't like suspend failing. People like to close their lid, throw
> laptop in bag, and expect laptop to not cook itself to death.

Sure, that's exactly the problem I'm looking at.  Throwing TASK_FREEZABLE
addresses some of the cases.  Failing laptop suspend because of uncontended
VFS lock is not an uncommon scenario for us.  Ideally, however, we need
some sort of server/client aware suspend, maybe moving clients to cgroup C
and server to cgroup S, and freezing those in strict order.  Or teaching PM
that some tasks cannot be frozen in random order during suspend (e.g. a
special flag PM_FREEZE_ME_LAST).

fuse is not the only subsystem that doesn't fit current random order suspend.
Another troublemaker for us is notify, which basically has the same server/client
architecture (where both sides are user-space processes).

Re: [PATCH v2] fuse: permit freezing while waiting for request answer

[Peter Zijlstra]

On Thu, Aug 20, 2026 at 06:24:29PM +0900, Sergey Senozhatsky wrote:
> On (26/08/20 11:10), Peter Zijlstra wrote:
> > On Thu, Aug 20, 2026 at 06:07:18PM +0900, Sergey Senozhatsky wrote:
> > > On (26/08/20 10:49), Peter Zijlstra wrote:
> > > > Message-ID: <20260820084933.GA4036497@noisy.programming.kicks-ass.net>
> > > > 
> > > > On Wed, Aug 19, 2026 at 12:03:32PM +0200, Miklos Szeredi wrote:
> > > > > Why is this unsafe exactly?  Does that unsafeness apply to
> > > > > filesystems?  If so why do we allow freezing while blocked on
> > > > > sb_start_write()?
> > > > 
> > > > Getting frozen with lock A held, while another task is blocked on A in
> > > > an unfreezable state results in the system not being freezable.
> > > 
> > > Right, but then the system says "suspend failed" (tasks refuse to freeze
> > > after 20sec) and just thaws everything back in?
> > 
> > People don't like suspend failing. People like to close their lid, throw
> > laptop in bag, and expect laptop to not cook itself to death.
> 
> Sure, that's exactly the problem I'm looking at.  Throwing TASK_FREEZABLE
> addresses some of the cases.  Failing laptop suspend because of uncontended
> VFS lock is not an uncommon scenario for us.  Ideally, however, we need
> some sort of server/client aware suspend, maybe moving clients to cgroup C
> and server to cgroup S, and freezing those in strict order.  Or teaching PM
> that some tasks cannot be frozen in random order during suspend (e.g. a
> special flag PM_FREEZE_ME_LAST).
> 
> fuse is not the only subsystem that doesn't fit current random order suspend.
> Another troublemaker for us is notify, which basically has the same server/client
> architecture (where both sides are user-space processes).

Then propose patches creating freeze order.

PM_FREEZE_ME_LAST is going to be trouble I think, before long you'll
need PM_FREEZE_ME_REALLY_LAST or somesuch nonsense.

Using cgroups for this also doesn't sound right.

Re: [PATCH v2] fuse: permit freezing while waiting for request answer

[Sergey Senozhatsky]

On (26/08/20 11:34), Peter Zijlstra wrote:
> On Thu, Aug 20, 2026 at 06:24:29PM +0900, Sergey Senozhatsky wrote:
> > On (26/08/20 11:10), Peter Zijlstra wrote:
> > > On Thu, Aug 20, 2026 at 06:07:18PM +0900, Sergey Senozhatsky wrote:
> > > > On (26/08/20 10:49), Peter Zijlstra wrote:
> > > > > Message-ID: <20260820084933.GA4036497@noisy.programming.kicks-ass.net>
> > > > > 
> > > > > On Wed, Aug 19, 2026 at 12:03:32PM +0200, Miklos Szeredi wrote:
> > > > > > Why is this unsafe exactly?  Does that unsafeness apply to
> > > > > > filesystems?  If so why do we allow freezing while blocked on
> > > > > > sb_start_write()?
> > > > > 
> > > > > Getting frozen with lock A held, while another task is blocked on A in
> > > > > an unfreezable state results in the system not being freezable.
> > > > 
> > > > Right, but then the system says "suspend failed" (tasks refuse to freeze
> > > > after 20sec) and just thaws everything back in?
> > > 
> > > People don't like suspend failing. People like to close their lid, throw
> > > laptop in bag, and expect laptop to not cook itself to death.
> > 
> > Sure, that's exactly the problem I'm looking at.  Throwing TASK_FREEZABLE
> > addresses some of the cases.  Failing laptop suspend because of uncontended
> > VFS lock is not an uncommon scenario for us.  Ideally, however, we need
> > some sort of server/client aware suspend, maybe moving clients to cgroup C
> > and server to cgroup S, and freezing those in strict order.  Or teaching PM
> > that some tasks cannot be frozen in random order during suspend (e.g. a
> > special flag PM_FREEZE_ME_LAST).
> > 
> > fuse is not the only subsystem that doesn't fit current random order suspend.
> > Another troublemaker for us is notify, which basically has the same server/client
> > architecture (where both sides are user-space processes).
> 
> Then propose patches creating freeze order.
>
> PM_FREEZE_ME_LAST is going to be trouble I think, before long you'll
> need PM_FREEZE_ME_REALLY_LAST or somesuch nonsense.

We probably need more than one, yeah.  This doesn't take into consideration
relations between tasks within a priority group.  If client A holds a lock
and enters freezer, and client B sleeps on that lock, then this group
cannot be suspended.  We need clients to reach some freezer checkpoint,
instead of doing in-place freezing.

> Using cgroups for this also doesn't sound right.

This begins sounding pessimistic.

cgroup-s sounded solid to me, because this moves all the clients to that
save freezer checkpoint when they don't hold any locks.  The only
problem is that if a task never reaches "return from syscall" then
we fail suspend.  (I'm only talking about user-space tasks here, in the
context of fuse or inotify.)  Why don't cgroups sound right to you?

Re: [PATCH v2] fuse: permit freezing while waiting for request answer

[Peter Zijlstra]

On Thu, Aug 20, 2026 at 06:47:03PM +0900, Sergey Senozhatsky wrote:

> > PM_FREEZE_ME_LAST is going to be trouble I think, before long you'll
> > need PM_FREEZE_ME_REALLY_LAST or somesuch nonsense.
> 
> We probably need more than one, yeah.  This doesn't take into consideration
> relations between tasks within a priority group.  If client A holds a lock
> and enters freezer, and client B sleeps on that lock, then this group
> cannot be suspended.  We need clients to reach some freezer checkpoint,
> instead of doing in-place freezing.
> 
> > Using cgroups for this also doesn't sound right.
> 
> This begins sounding pessimistic.
> 
> cgroup-s sounded solid to me, because this moves all the clients to that
> save freezer checkpoint when they don't hold any locks.  The only
> problem is that if a task never reaches "return from syscall" then
> we fail suspend.  (I'm only talking about user-space tasks here, in the
> context of fuse or inotify.)  Why don't cgroups sound right to you?

The cgroup hierarchy is already a mess, and the more different things
you want to stuff in there, the worse it gets.

Also, I still have machines with CGROUP=n.

Also, you'd be putting suspend success in the hands of userspace, that
sounds like a mighty fail right there.

Re: [PATCH v2] fuse: permit freezing while waiting for request answer

[Miklos Szeredi]

On Thu, 20 Aug 2026 at 12:41, Peter Zijlstra <peterz@infradead.org> wrote:

> Also, you'd be putting suspend success in the hands of userspace, that
> sounds like a mighty fail right there.

Freeze ordering is also a hard/impossible problem without the help of
userspace.  One obvious heuristic is if task has a /dev/fuse fd open,
it is a server.  This simple ordering between server and non-server
tasks fails if

 - task X is a server of A, while also doing ops on B

 - task X is server, blocked on task Y which is not a server

What about adding a flag to mutex/rwsem that allows making them
freezable at setup time?  Then fuse fs could set that flag for its
locks (including VFS locks).

Thanks,
Miklos