[PATCH] fuse: mark DAX inode releases as blocking

[PATCH] fuse: mark DAX inode releases as blocking

[Sergio Lopez]

Commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from
fuseblk workers") made fputs on closing files always asynchronous.

As cleaning up DAX inodes may require issuing a number of synchronous
request for releasing the mappings, completing the release request from
the worker thread may lead to it hanging like this:

[   21.386751] Workqueue: events virtio_fs_requests_done_work
[   21.386769] Call trace:
[   21.386770]  __switch_to+0xe4/0x140
[   21.386780]  __schedule+0x294/0x72c
[   21.386787]  schedule+0x24/0x90
[   21.386794]  request_wait_answer+0x184/0x298
[   21.386799]  __fuse_simple_request+0x1f4/0x320
[   21.386805]  fuse_send_removemapping+0x80/0xa0
[   21.386810]  dmap_removemapping_list+0xac/0xfc
[   21.386814]  inode_reclaim_dmap_range.constprop.0+0xd0/0x204
[   21.386820]  fuse_dax_inode_cleanup+0x28/0x5c
[   21.386825]  fuse_evict_inode+0x120/0x190
[   21.386834]  evict+0x188/0x320
[   21.386847]  iput_final+0xb0/0x20c
[   21.386854]  iput+0xa0/0xbc
[   21.386862]  fuse_release_end+0x18/0x2c
[   21.386868]  fuse_request_end+0x9c/0x2c0
[   21.386872]  virtio_fs_request_complete+0x150/0x384
[   21.386879]  virtio_fs_requests_done_work+0x18c/0x37c
[   21.386885]  process_one_work+0x15c/0x2e8
[   21.386891]  worker_thread+0x278/0x480
[   21.386898]  kthread+0xd0/0xdc
[   21.386902]  ret_from_fork+0x10/0x20

Here, the virtio-fs worker_thread is waiting on request_wait_answer()
for a reply from the virtio-fs server that is already in the virtqueue
but will never be processed since it's that same worker thread the one
in charge of consuming the elements from the virtqueue.

To address this issue, when relesing a DAX inode mark the operation as
potentially blocking. Doing this will ensure these release requests are
processed on a different worker thread.

Signed-off-by: Sergio Lopez <slp@redhat.com>
---
 fs/fuse/file.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/fs/fuse/file.c b/fs/fuse/file.c
index 3b2a171e652f..a65c5d32a34b 100644
--- a/fs/fuse/file.c
+++ b/fs/fuse/file.c
@@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)
 			fuse_simple_request(ff->fm, args);
 			fuse_release_end(ff->fm, args, 0);
 		} else {
+			/*
+			 * DAX inodes may need to issue a number of synchronous
+			 * request for clearing the mappings.
+			 */
+			if (ra && ra->inode && FUSE_IS_DAX(ra->inode))
+				args->may_block = true;
 			args->end = fuse_release_end;
 			if (fuse_simple_background(ff->fm, args,
 						   GFP_KERNEL | __GFP_NOFAIL))
-- 
2.52.0

Re: [PATCH] fuse: mark DAX inode releases as blocking

[Darrick J. Wong]

On Mon, Jan 19, 2026 at 12:24:11AM +0100, Sergio Lopez wrote:
> Commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from
> fuseblk workers") made fputs on closing files always asynchronous.
> 
> As cleaning up DAX inodes may require issuing a number of synchronous
> request for releasing the mappings, completing the release request from
> the worker thread may lead to it hanging like this:
> 
> [   21.386751] Workqueue: events virtio_fs_requests_done_work
> [   21.386769] Call trace:
> [   21.386770]  __switch_to+0xe4/0x140
> [   21.386780]  __schedule+0x294/0x72c
> [   21.386787]  schedule+0x24/0x90
> [   21.386794]  request_wait_answer+0x184/0x298
> [   21.386799]  __fuse_simple_request+0x1f4/0x320
> [   21.386805]  fuse_send_removemapping+0x80/0xa0
> [   21.386810]  dmap_removemapping_list+0xac/0xfc
> [   21.386814]  inode_reclaim_dmap_range.constprop.0+0xd0/0x204
> [   21.386820]  fuse_dax_inode_cleanup+0x28/0x5c
> [   21.386825]  fuse_evict_inode+0x120/0x190
> [   21.386834]  evict+0x188/0x320
> [   21.386847]  iput_final+0xb0/0x20c
> [   21.386854]  iput+0xa0/0xbc
> [   21.386862]  fuse_release_end+0x18/0x2c
> [   21.386868]  fuse_request_end+0x9c/0x2c0

Ok, so this is the reply from the async FUSE_RELEASE command.  But then
we iput the inode, which results in fuse issuing a new synchronous
command from within the completion for the first command.

Ouch.

> [   21.386872]  virtio_fs_request_complete+0x150/0x384
> [   21.386879]  virtio_fs_requests_done_work+0x18c/0x37c
> [   21.386885]  process_one_work+0x15c/0x2e8
> [   21.386891]  worker_thread+0x278/0x480
> [   21.386898]  kthread+0xd0/0xdc
> [   21.386902]  ret_from_fork+0x10/0x20
> 
> Here, the virtio-fs worker_thread is waiting on request_wait_answer()
> for a reply from the virtio-fs server that is already in the virtqueue
> but will never be processed since it's that same worker thread the one
> in charge of consuming the elements from the virtqueue.

Yes.  Ow.

> To address this issue, when relesing a DAX inode mark the operation as
> potentially blocking. Doing this will ensure these release requests are
> processed on a different worker thread.

I wonder if you've solved this problem report?
https://github.com/Nevuly/WSL2-Rolling-Kernel-Issue/issues/38

Naturally they reverted the patch, emailed me, and refused to talk about
this on the public list, which is why nobody's heard of this until now.

> Signed-off-by: Sergio Lopez <slp@redhat.com>
> ---
>  fs/fuse/file.c | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/fs/fuse/file.c b/fs/fuse/file.c
> index 3b2a171e652f..a65c5d32a34b 100644
> --- a/fs/fuse/file.c
> +++ b/fs/fuse/file.c
> @@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)
>  			fuse_simple_request(ff->fm, args);
>  			fuse_release_end(ff->fm, args, 0);
>  		} else {
> +			/*
> +			 * DAX inodes may need to issue a number of synchronous
> +			 * request for clearing the mappings.
> +			 */
> +			if (ra && ra->inode && FUSE_IS_DAX(ra->inode))
> +				args->may_block = true;

There's no documentation for what may_block does, but there are so few
uses of it that I can tell that this is kicking the FUSE_RELEASE
completion to a workqueue instead of processing it directly, which
eliminates the livelock.

I wonder if fuse ought to grow the ability to whine when something is
trying to issue a synchronous fuse command while running in a command
queue completion context (aka the worker threads) but I don't know how
difficult that would *really* be.

Insofar as I understand anything in fuse, I think this is the right
thing to do.  Thanks for fixing this.

Reviewed-by: "Darrick J. Wong" <djwong@kernel.org>

--D

>  			args->end = fuse_release_end;
>  			if (fuse_simple_background(ff->fm, args,
>  						   GFP_KERNEL | __GFP_NOFAIL))
> -- 
> 2.52.0
> 
> 

Re: [PATCH] fuse: mark DAX inode releases as blocking

[Jingbo Xu]

On 1/27/26 2:40 AM, Darrick J. Wong wrote:

> I wonder if fuse ought to grow the ability to whine when something is
> trying to issue a synchronous fuse command while running in a command
> queue completion context (aka the worker threads) but I don't know how
> difficult that would *really* be.

I had also observed similar issue where the FUSE daemon thread is
hanging in:

request_wait_answer
fuse_simple_request
fuse_flush_times
fuse write_inode
writeback_single_inode
write_inode_now
fuse_release
_fput

At that time I had no idea how FUSE daemon thread could trigger fuse
file release and thus I didn't dive into this further...

I think commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put
from fuseblk workers") is not adequate in this case, as the commit only
makes FUSE_RELEASE request asynchronously, while in this case the daemon
thread can wait for FUSE_WRITE and FUSE_SETATTR.

Maybe the very first entry i.e. fuse_release() needs to be executed in
an asynchronous context (e.g. workqueue)...


-- 
Thanks,
Jingbo

Re: [PATCH] fuse: mark DAX inode releases as blocking

[Sergio Lopez Pascual]

Jingbo Xu <jefflexu@linux.alibaba.com> writes:

> On 1/27/26 2:40 AM, Darrick J. Wong wrote:
>
>> I wonder if fuse ought to grow the ability to whine when something is
>> trying to issue a synchronous fuse command while running in a command
>> queue completion context (aka the worker threads) but I don't know how
>> difficult that would *really* be.
>
> I had also observed similar issue where the FUSE daemon thread is
> hanging in:
>
> request_wait_answer
> fuse_simple_request
> fuse_flush_times
> fuse write_inode
> writeback_single_inode
> write_inode_now
> fuse_release
> _fput
>
> At that time I had no idea how FUSE daemon thread could trigger fuse
> file release and thus I didn't dive into this further...
>
> I think commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put
> from fuseblk workers") is not adequate in this case, as the commit only
> makes FUSE_RELEASE request asynchronously, while in this case the daemon
> thread can wait for FUSE_WRITE and FUSE_SETATTR.
>
> Maybe the very first entry i.e. fuse_release() needs to be executed in
> an asynchronous context (e.g. workqueue)...

Do you have the rest of the stacktrace? We need to know if it's running
in the context of the worker thread, as otherwise it must be a different
problem.

Thanks,
Sergio.

Re: [PATCH] fuse: mark DAX inode releases as blocking

[Jingbo Xu]

On 1/27/26 7:18 PM, Sergio Lopez Pascual wrote:
> Jingbo Xu <jefflexu@linux.alibaba.com> writes:
> 
>> On 1/27/26 2:40 AM, Darrick J. Wong wrote:
>>
>>> I wonder if fuse ought to grow the ability to whine when something is
>>> trying to issue a synchronous fuse command while running in a command
>>> queue completion context (aka the worker threads) but I don't know how
>>> difficult that would *really* be.
>>
>> I had also observed similar issue where the FUSE daemon thread is
>> hanging in:
>>
>> request_wait_answer
>> fuse_simple_request
>> fuse_flush_times
>> fuse write_inode
>> writeback_single_inode
>> write_inode_now
>> fuse_release
>> _fput
>>
>> At that time I had no idea how FUSE daemon thread could trigger fuse
>> file release and thus I didn't dive into this further...
>>
>> I think commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put
>> from fuseblk workers") is not adequate in this case, as the commit only
>> makes FUSE_RELEASE request asynchronously, while in this case the daemon
>> thread can wait for FUSE_WRITE and FUSE_SETATTR.
>>
>> Maybe the very first entry i.e. fuse_release() needs to be executed in
>> an asynchronous context (e.g. workqueue)...
> 
> Do you have the rest of the stacktrace? We need to know if it's running
> in the context of the worker thread, as otherwise it must be a different
> problem.


This is observed in scenarios of FUSE rather than virtiofs.

The complete stacktrace of FUSE daemon worker thread (in userspace) is like:

 request_wait_answer
 fuse_simple_request
 fuse_flush_times
 fuse write_inode
 writeback_single_inode
 write_inode_now
 fuse_release
 __fput
 ____fput
 task_work_run
 exit_to_user_mode_prepare
 syscall exit_to_user_mode
 do_syscall_64
 entry_SYSCALL_64_after_hwframe



-- 
Thanks,
Jingbo

Re: [PATCH] fuse: mark DAX inode releases as blocking

[Sergio Lopez Pascual]

"Darrick J. Wong" <djwong@kernel.org> writes:

> On Mon, Jan 19, 2026 at 12:24:11AM +0100, Sergio Lopez wrote:
>> Commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from
>> fuseblk workers") made fputs on closing files always asynchronous.
>>
>> As cleaning up DAX inodes may require issuing a number of synchronous
>> request for releasing the mappings, completing the release request from
>> the worker thread may lead to it hanging like this:
>>
>> [   21.386751] Workqueue: events virtio_fs_requests_done_work
>> [   21.386769] Call trace:
>> [   21.386770]  __switch_to+0xe4/0x140
>> [   21.386780]  __schedule+0x294/0x72c
>> [   21.386787]  schedule+0x24/0x90
>> [   21.386794]  request_wait_answer+0x184/0x298
>> [   21.386799]  __fuse_simple_request+0x1f4/0x320
>> [   21.386805]  fuse_send_removemapping+0x80/0xa0
>> [   21.386810]  dmap_removemapping_list+0xac/0xfc
>> [   21.386814]  inode_reclaim_dmap_range.constprop.0+0xd0/0x204
>> [   21.386820]  fuse_dax_inode_cleanup+0x28/0x5c
>> [   21.386825]  fuse_evict_inode+0x120/0x190
>> [   21.386834]  evict+0x188/0x320
>> [   21.386847]  iput_final+0xb0/0x20c
>> [   21.386854]  iput+0xa0/0xbc
>> [   21.386862]  fuse_release_end+0x18/0x2c
>> [   21.386868]  fuse_request_end+0x9c/0x2c0
>
> Ok, so this is the reply from the async FUSE_RELEASE command.  But then
> we iput the inode, which results in fuse issuing a new synchronous
> command from within the completion for the first command.
>
> Ouch.
>
>> [   21.386872]  virtio_fs_request_complete+0x150/0x384
>> [   21.386879]  virtio_fs_requests_done_work+0x18c/0x37c
>> [   21.386885]  process_one_work+0x15c/0x2e8
>> [   21.386891]  worker_thread+0x278/0x480
>> [   21.386898]  kthread+0xd0/0xdc
>> [   21.386902]  ret_from_fork+0x10/0x20
>>
>> Here, the virtio-fs worker_thread is waiting on request_wait_answer()
>> for a reply from the virtio-fs server that is already in the virtqueue
>> but will never be processed since it's that same worker thread the one
>> in charge of consuming the elements from the virtqueue.
>
> Yes.  Ow.
>
>> To address this issue, when relesing a DAX inode mark the operation as
>> potentially blocking. Doing this will ensure these release requests are
>> processed on a different worker thread.
>
> I wonder if you've solved this problem report?
> https://github.com/Nevuly/WSL2-Rolling-Kernel-Issue/issues/38
>
> Naturally they reverted the patch, emailed me, and refused to talk about
> this on the public list, which is why nobody's heard of this until now.

If they're using DAX, that's very likely. I've discovered it while
testing a new kernel with libkrun/muvm, which does use multiple
virtio-fs devices, one of them as root and another one with DAX, which
turns to be a pretty good testbed for virtio-fs (nothing better than
running Steam, and bunch of games under FEX+Proton for stress-testing
multiple subsystems with a real-world workload ;-).

>> Signed-off-by: Sergio Lopez <slp@redhat.com>
>> ---
>>  fs/fuse/file.c | 6 ++++++
>>  1 file changed, 6 insertions(+)
>>
>> diff --git a/fs/fuse/file.c b/fs/fuse/file.c
>> index 3b2a171e652f..a65c5d32a34b 100644
>> --- a/fs/fuse/file.c
>> +++ b/fs/fuse/file.c
>> @@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)
>>  			fuse_simple_request(ff->fm, args);
>>  			fuse_release_end(ff->fm, args, 0);
>>  		} else {
>> +			/*
>> +			 * DAX inodes may need to issue a number of synchronous
>> +			 * request for clearing the mappings.
>> +			 */
>> +			if (ra && ra->inode && FUSE_IS_DAX(ra->inode))
>> +				args->may_block = true;
>
> There's no documentation for what may_block does, but there are so few
> uses of it that I can tell that this is kicking the FUSE_RELEASE
> completion to a workqueue instead of processing it directly, which
> eliminates the livelock.
>
> I wonder if fuse ought to grow the ability to whine when something is
> trying to issue a synchronous fuse command while running in a command
> queue completion context (aka the worker threads) but I don't know how
> difficult that would *really* be.

Perhaps we could check for PF_WQ_WORKER and, at the very least, emit a
warning.

Thanks,
Sergio.

Re: [PATCH] fuse: mark DAX inode releases as blocking

[Darrick J. Wong]

On Tue, Jan 27, 2026 at 05:12:35AM -0600, Sergio Lopez Pascual wrote:
> "Darrick J. Wong" <djwong@kernel.org> writes:
> 
> > On Mon, Jan 19, 2026 at 12:24:11AM +0100, Sergio Lopez wrote:
> >> Commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from
> >> fuseblk workers") made fputs on closing files always asynchronous.
> >>
> >> As cleaning up DAX inodes may require issuing a number of synchronous
> >> request for releasing the mappings, completing the release request from
> >> the worker thread may lead to it hanging like this:
> >>
> >> [   21.386751] Workqueue: events virtio_fs_requests_done_work
> >> [   21.386769] Call trace:
> >> [   21.386770]  __switch_to+0xe4/0x140
> >> [   21.386780]  __schedule+0x294/0x72c
> >> [   21.386787]  schedule+0x24/0x90
> >> [   21.386794]  request_wait_answer+0x184/0x298
> >> [   21.386799]  __fuse_simple_request+0x1f4/0x320
> >> [   21.386805]  fuse_send_removemapping+0x80/0xa0
> >> [   21.386810]  dmap_removemapping_list+0xac/0xfc
> >> [   21.386814]  inode_reclaim_dmap_range.constprop.0+0xd0/0x204
> >> [   21.386820]  fuse_dax_inode_cleanup+0x28/0x5c
> >> [   21.386825]  fuse_evict_inode+0x120/0x190
> >> [   21.386834]  evict+0x188/0x320
> >> [   21.386847]  iput_final+0xb0/0x20c
> >> [   21.386854]  iput+0xa0/0xbc
> >> [   21.386862]  fuse_release_end+0x18/0x2c
> >> [   21.386868]  fuse_request_end+0x9c/0x2c0
> >
> > Ok, so this is the reply from the async FUSE_RELEASE command.  But then
> > we iput the inode, which results in fuse issuing a new synchronous
> > command from within the completion for the first command.
> >
> > Ouch.
> >
> >> [   21.386872]  virtio_fs_request_complete+0x150/0x384
> >> [   21.386879]  virtio_fs_requests_done_work+0x18c/0x37c
> >> [   21.386885]  process_one_work+0x15c/0x2e8
> >> [   21.386891]  worker_thread+0x278/0x480
> >> [   21.386898]  kthread+0xd0/0xdc
> >> [   21.386902]  ret_from_fork+0x10/0x20
> >>
> >> Here, the virtio-fs worker_thread is waiting on request_wait_answer()
> >> for a reply from the virtio-fs server that is already in the virtqueue
> >> but will never be processed since it's that same worker thread the one
> >> in charge of consuming the elements from the virtqueue.
> >
> > Yes.  Ow.
> >
> >> To address this issue, when relesing a DAX inode mark the operation as
> >> potentially blocking. Doing this will ensure these release requests are
> >> processed on a different worker thread.
> >
> > I wonder if you've solved this problem report?
> > https://github.com/Nevuly/WSL2-Rolling-Kernel-Issue/issues/38
> >
> > Naturally they reverted the patch, emailed me, and refused to talk about
> > this on the public list, which is why nobody's heard of this until now.
> 
> If they're using DAX, that's very likely. I've discovered it while
> testing a new kernel with libkrun/muvm, which does use multiple
> virtio-fs devices, one of them as root and another one with DAX, which
> turns to be a pretty good testbed for virtio-fs (nothing better than
> running Steam, and bunch of games under FEX+Proton for stress-testing
> multiple subsystems with a real-world workload ;-).

Hehe. :)

> >> Signed-off-by: Sergio Lopez <slp@redhat.com>
> >> ---
> >>  fs/fuse/file.c | 6 ++++++
> >>  1 file changed, 6 insertions(+)
> >>
> >> diff --git a/fs/fuse/file.c b/fs/fuse/file.c
> >> index 3b2a171e652f..a65c5d32a34b 100644
> >> --- a/fs/fuse/file.c
> >> +++ b/fs/fuse/file.c
> >> @@ -117,6 +117,12 @@ static void fuse_file_put(struct fuse_file *ff, bool sync)
> >>  			fuse_simple_request(ff->fm, args);
> >>  			fuse_release_end(ff->fm, args, 0);
> >>  		} else {
> >> +			/*
> >> +			 * DAX inodes may need to issue a number of synchronous
> >> +			 * request for clearing the mappings.
> >> +			 */
> >> +			if (ra && ra->inode && FUSE_IS_DAX(ra->inode))
> >> +				args->may_block = true;
> >
> > There's no documentation for what may_block does, but there are so few
> > uses of it that I can tell that this is kicking the FUSE_RELEASE
> > completion to a workqueue instead of processing it directly, which
> > eliminates the livelock.
> >
> > I wonder if fuse ought to grow the ability to whine when something is
> > trying to issue a synchronous fuse command while running in a command
> > queue completion context (aka the worker threads) but I don't know how
> > difficult that would *really* be.
> 
> Perhaps we could check for PF_WQ_WORKER and, at the very least, emit a
> warning.

I don't think that will work:

PF_WQ_WORKER only tells us if the current process is a kernel workqueue.
Replies to fuse command are written to the fuse device, which means that
we're running in the process context of the fuse server when we get to
fuse_dev_do_write -> fuse_request_end -> req->args->end ->
fuse_release_end.

--D

> Thanks,
> Sergio.
> 

Re: [PATCH] fuse: mark DAX inode releases as blocking

[Miklos Szeredi]

On Mon, 26 Jan 2026 at 17:29, Sergio Lopez <slp@redhat.com> wrote:
>
> Commit 26e5c67deb2e ("fuse: fix livelock in synchronous file put from
> fuseblk workers") made fputs on closing files always asynchronous.
>
> As cleaning up DAX inodes may require issuing a number of synchronous
> request for releasing the mappings, completing the release request from
> the worker thread may lead to it hanging like this:
>
> [   21.386751] Workqueue: events virtio_fs_requests_done_work
> [   21.386769] Call trace:
> [   21.386770]  __switch_to+0xe4/0x140
> [   21.386780]  __schedule+0x294/0x72c
> [   21.386787]  schedule+0x24/0x90
> [   21.386794]  request_wait_answer+0x184/0x298
> [   21.386799]  __fuse_simple_request+0x1f4/0x320
> [   21.386805]  fuse_send_removemapping+0x80/0xa0
> [   21.386810]  dmap_removemapping_list+0xac/0xfc
> [   21.386814]  inode_reclaim_dmap_range.constprop.0+0xd0/0x204
> [   21.386820]  fuse_dax_inode_cleanup+0x28/0x5c
> [   21.386825]  fuse_evict_inode+0x120/0x190
> [   21.386834]  evict+0x188/0x320
> [   21.386847]  iput_final+0xb0/0x20c
> [   21.386854]  iput+0xa0/0xbc
> [   21.386862]  fuse_release_end+0x18/0x2c
> [   21.386868]  fuse_request_end+0x9c/0x2c0
> [   21.386872]  virtio_fs_request_complete+0x150/0x384
> [   21.386879]  virtio_fs_requests_done_work+0x18c/0x37c
> [   21.386885]  process_one_work+0x15c/0x2e8
> [   21.386891]  worker_thread+0x278/0x480
> [   21.386898]  kthread+0xd0/0xdc
> [   21.386902]  ret_from_fork+0x10/0x20
>
> Here, the virtio-fs worker_thread is waiting on request_wait_answer()
> for a reply from the virtio-fs server that is already in the virtqueue
> but will never be processed since it's that same worker thread the one
> in charge of consuming the elements from the virtqueue.
>
> To address this issue, when relesing a DAX inode mark the operation as
> potentially blocking. Doing this will ensure these release requests are
> processed on a different worker thread.
>
> Signed-off-by: Sergio Lopez <slp@redhat.com>

Applied, thanks.

Miklos