[PATCH 0/2] fuse: io-uring: fix two UAFs in dev_uring.c teardown

[PATCH 0/2] fuse: io-uring: fix two UAFs in dev_uring.c teardown

[Berkant Koc]

Applied on top of 6916d5703ddf. Thanks Greg for asking to take this
on-list; the original off-list report to security@kernel.org included
a defensive third patch that added cancel_delayed_work_sync() in
fuse_conn_put(), but it triggered a WARN_ON in queue_work() because
the work item is not always initialized at that point. That patch is
dropped from this series.

Patch 1/2 is the static-analysis fix: fuse_uring_commit_fetch() leaks
a dangling ent->fuse_req on the set_commit error branch. The patch
routes that branch through the existing fuse_uring_req_end() helper
so ent->fuse_req is cleared under queue->lock.

Patch 2/2 is the KASAN-reproducible fix: fuse_dev_release() on the
last fuse_dev drops the connection ref before the io-uring
async_teardown_work has stopped, so delayed_release() can kfree() ring
entries the work is still walking. Adding fuse_wait_aborted() between
fuse_abort_conn() and fuse_conn_put() drains queue_refs first.

KASAN-tested at HEAD 6916d5703ddf + this series, 50 iterations x 16
worker threads against an io-uring fuse daemon: 0 KASAN trips, 0
warnings. KASAN log on request.

Joanne Koong's [PATCH v2 0/3] (Message-ID
<20260516021138.2759874-1-joannelkoong@gmail.com>) is in flight on a
separate fuse-next base; this series targets mainline 6916d5703ddf and
does not overlap with hers in code or fix scope.

CCing linux-fuse and linux-kernel now per the on-list workflow.

Berkant Koc (2):
  fuse: io-uring: clear ent->fuse_req in commit_fetch error path
  fuse: wait for aborted connection before releasing last fuse_dev

 fs/fuse/dev.c       | 1 +
 fs/fuse/dev_uring.c | 4 +---
 2 files changed, 2 insertions(+), 3 deletions(-)

--
2.47.3

[PATCH 1/2] fuse: io-uring: clear ent->fuse_req in commit_fetch error path

[Berkant Koc]

From: Berkant Koc <me@berkoc.com>

fuse_uring_commit_fetch() locates a request, removes it from the
processing queue, clears req->ring_entry, then calls
fuse_ring_ent_set_commit() under queue->lock. On the error branch
(set_commit returning non-zero because the entry is not in
FRRS_USERSPACE) the function unlocks the queue and ends the request
directly with fuse_request_end(), but it never clears ent->fuse_req.

ent->fuse_req then keeps pointing at the freed fuse_req while the entry
remains on a queue list. Subsequent teardown via
fuse_uring_entry_teardown() reads ent->fuse_req under queue->lock and
hands the dangling pointer to fuse_uring_stop_fuse_req_end(), which
dereferences it and calls fuse_request_end() a second time on freed
memory.

Route the error branch through fuse_uring_req_end() instead. That
helper acquires queue->lock, clears ent->fuse_req under the lock,
removes the request from any list it is still on, drops the lock, sets
req->out.h.error, clears FR_SENT and ends the request. The
ent->fuse_req = NULL store under the lock is what closes the window
for the later teardown reader.

Fixes: c090c8abae4b ("fuse: Add io-uring sqe commit and fetch support")
Cc: stable@vger.kernel.org # 6.14+
Signed-off-by: Berkant Koc <me@berkoc.com>
---
 fs/fuse/dev_uring.c | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
index 7b9822e8837b..7523569ffdce 100644
--- a/fs/fuse/dev_uring.c
+++ b/fs/fuse/dev_uring.c
@@ -924,9 +924,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
 		pr_info_ratelimited("qid=%d commit_id %llu state %d",
 				    queue->qid, commit_id, ent->state);
 		spin_unlock(&queue->lock);
-		req->out.h.error = err;
-		clear_bit(FR_SENT, &req->flags);
-		fuse_request_end(req);
+		fuse_uring_req_end(ent, req, err);
 		return err;
 	}

--
2.47.3

[PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Berkant Koc]

From: Berkant Koc <me@berkoc.com>

fuse_dev_release() on the last fuse_dev of a connection calls
fuse_abort_conn(fc) and then immediately fuse_conn_put(fc). For io-uring
backed connections fuse_abort_conn() reaches fuse_uring_abort(), which
runs fuse_uring_teardown_all_queues() synchronously once and then
schedules ring->async_teardown_work to run after
FUSE_URING_TEARDOWN_INTERVAL (HZ/20). If the synchronous pass left
queue_refs > 0 the work owns further accesses to ring->queues[*]->
ent_avail_queue and ent_in_userspace entries.

Meanwhile fuse_conn_put() can drop the last reference and arm
delayed_release() via call_rcu(). After the RCU grace period
delayed_release() calls fuse_uring_destruct(), which kfree()s the ring
entries on each queue->ent_released list. The previously scheduled
async_teardown_work then runs and walks per-queue lists that contain
freed entries, producing a slab-use-after-free reported by KASAN at
fuse_uring_teardown_all_queues+0xee reading ent->list.next from a
freed kmalloc-192 region.

fuse_wait_aborted() already exists for this purpose: it waits on
fc->blocked_waitq for num_waiting to drain and then calls
fuse_uring_wait_stopped_queues(), which waits for ring->queue_refs to
reach zero. Call it between fuse_abort_conn() and fuse_conn_put() on
the last-device path so the io-uring teardown work has fully drained
before the connection can be torn down.

Fixes: c090c8abae4b ("fuse: Add io-uring sqe commit and fetch support")
Cc: stable@vger.kernel.org # 6.14+
Tested-by: Berkant Koc <me@berkoc.com>
Signed-off-by: Berkant Koc <me@berkoc.com>
---
 fs/fuse/dev.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 5dda7080f4a9..7d9c06654a98 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -2566,6 +2566,7 @@ int fuse_dev_release(struct inode *inode, struct file *file)
 		if (last) {
 			WARN_ON(fc->iq.fasync != NULL);
 			fuse_abort_conn(fc);
+			fuse_wait_aborted(fc);
 		}
 		fuse_conn_put(fc);
 	}
--
2.47.3

Re: [PATCH 0/2] fuse: io-uring: fix two UAFs in dev_uring.c teardown

[Berkant Koc]

Quick correction on the Cc list of this series:

linux-fuse@vger.kernel.org does not exist as a vger list and the
three patch mails bounced from it (550 5.1.1 User unknown).
Per MAINTAINERS, the FUSE list is linux-fsdevel@vger.kernel.org;
adding it now so the series shows up in the lore.kernel.org archive
for the FUSE-fsdevel readership. The original patches and KASAN
context are in this thread via In-Reply-To.

No content change to the patches; this is purely a list-routing fix.

Apologies for the noise.

Berkant

Re: [PATCH 0/2] fuse: io-uring: fix two UAFs in dev_uring.c teardown

[Bernd Schubert]

On 5/17/26 15:14, Berkant Koc wrote:
> Quick correction on the Cc list of this series:
> 
> linux-fuse@vger.kernel.org does not exist as a vger list and the
> three patch mails bounced from it (550 5.1.1 User unknown).

The right list is fuse-devel@lists.linux.dev. MAINTAINERS file is in the
process to get updated.

> Per MAINTAINERS, the FUSE list is linux-fsdevel@vger.kernel.org;
> adding it now so the series shows up in the lore.kernel.org archive
> for the FUSE-fsdevel readership. The original patches and KASAN
> context are in this thread via In-Reply-To.
> 
> No content change to the patches; this is purely a list-routing fix.
> 
> Apologies for the noise.
> 
> Berkant
> 


Thanks,
Bernd

Re: [PATCH 0/2] fuse: io-uring: fix two UAFs in dev_uring.c teardown

[Berkant Koc]

Thanks Bernd, adding fuse-devel@lists.linux.dev to Cc now so this
subthread lands in the right archive. Will use that as the FUSE
mailing list going forward.

Berkant

Re: [PATCH 1/2] fuse: io-uring: clear ent->fuse_req in commit_fetch error path

[Bernd Schubert]

On 5/17/26 14:59, Berkant Koc wrote:
> [You don't often get email from me@berkoc.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> From: Berkant Koc <me@berkoc.com>
> 
> fuse_uring_commit_fetch() locates a request, removes it from the
> processing queue, clears req->ring_entry, then calls
> fuse_ring_ent_set_commit() under queue->lock. On the error branch
> (set_commit returning non-zero because the entry is not in
> FRRS_USERSPACE) the function unlocks the queue and ends the request
> directly with fuse_request_end(), but it never clears ent->fuse_req.
> 
> ent->fuse_req then keeps pointing at the freed fuse_req while the entry
> remains on a queue list. Subsequent teardown via
> fuse_uring_entry_teardown() reads ent->fuse_req under queue->lock and
> hands the dangling pointer to fuse_uring_stop_fuse_req_end(), which
> dereferences it and calls fuse_request_end() a second time on freed
> memory.
> 
> Route the error branch through fuse_uring_req_end() instead. That
> helper acquires queue->lock, clears ent->fuse_req under the lock,
> removes the request from any list it is still on, drops the lock, sets
> req->out.h.error, clears FR_SENT and ends the request. The
> ent->fuse_req = NULL store under the lock is what closes the window
> for the later teardown reader.
> 
> Fixes: c090c8abae4b ("fuse: Add io-uring sqe commit and fetch support")
> Cc: stable@vger.kernel.org # 6.14+
> Signed-off-by: Berkant Koc <me@berkoc.com>
> ---
>  fs/fuse/dev_uring.c | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
> 
> diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
> index 7b9822e8837b..7523569ffdce 100644
> --- a/fs/fuse/dev_uring.c
> +++ b/fs/fuse/dev_uring.c
> @@ -924,9 +924,7 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
>                 pr_info_ratelimited("qid=%d commit_id %llu state %d",
>                                     queue->qid, commit_id, ent->state);
>                 spin_unlock(&queue->lock);
> -               req->out.h.error = err;
> -               clear_bit(FR_SENT, &req->flags);
> -               fuse_request_end(req);
> +               fuse_uring_req_end(ent, req, err);
>                 return err;
>         }
> 
> --
> 2.47.3

We already had a security report for that on Friday


> 
> [You don't often get email from kipreyyy@gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> fuse_uring_commit_fetch() error path calls fuse_request_end(req) without
> clearing ent->fuse_req when fuse_ring_ent_set_commit() fails. The
> still-pending fuse_uring_send_in_task() task-work later dereferences the
> dangling pointer through fuse_uring_prepare_send(), causing a
> use-after-free.
> 
> Clear ent->fuse_req under queue->lock in the error path, matching the
> pattern in fuse_uring_req_end(). Add a NULL check in
> fuse_uring_send_in_task() so the entry is gracefully recycled if the
> request was detached.
> 
> Fixes: c090c8abae4b ("fuse: Add io-uring sqe commit and fetch support")
> Signed-off-by: Zhenghang Xiao
> ---
>  fs/fuse/dev_uring.c | 16 ++++++++++++++++
>  1 file changed, 16 insertions(+)
> 
> diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
> index 7b9822e8837b..5e51c36ae2a0 100644
> --- a/fs/fuse/dev_uring.c
> +++ b/fs/fuse/dev_uring.c
> @@ -921,6 +921,13 @@ static int fuse_uring_commit_fetch(struct io_uring_cmd *cmd, int issue_flags,
> 
>         err = fuse_ring_ent_set_commit(ent);
>         if (err != 0) {
> +               /*
> +                * Entry is not in FRRS_USERSPACE state. Clear the
> +                * back-pointer to prevent the still-pending
> +                * fuse_uring_send_in_task() from dereferencing a request
> +                * that is about to be freed.
> +                */
> +               ent->fuse_req = NULL;
>                 pr_info_ratelimited("qid=%d commit_id %llu state %d",
>                                     queue->qid, commit_id, ent->state);
>                 spin_unlock(&queue->lock);
> @@ -1220,6 +1227,15 @@ static void fuse_uring_send_in_task(struct io_tw_req tw_req, io_tw_token_t tw)
>         int err;
> 
>         if (!tw.cancel) {
> +               /*
> +                * If the request was detached (e.g. by fuse_uring_commit_fetch
> +                * error path), fuse_req will be NULL. Bail out and recycle the
> +                * entry for the next request.
> +                */
> +               if (!ent->fuse_req) {
> +                       fuse_uring_next_fuse_req(ent, queue, issue_flags);
> +                       return;
> +               }
>                 err = fuse_uring_prepare_send(ent, ent->fuse_req);
>                 if (err) {
>                         fuse_uring_next_fuse_req(ent, queue, issue_flags);
> --




I had already replied to Zhenghang on Friday, I don't think it is
enough. This condition

>         err = fuse_ring_ent_set_commit(ent);
>         if (err != 0) {

can also be triggered by a bad fuse-server / user-space and a teardown
race. The tear down race is easy, I think.

Fuse-server that wants to trick us is harder, as checking for
ent->fuse_req == NULL in fuse_uring_send_in_task() is not enough, the
race is valid all over the copy operation (fuse_uring_prepare_send()). I
didn't come to it yesterday (was working for main work), going to look
into it a bit later today.


Thanks,
Bernd

Re: [PATCH 1/2] fuse: io-uring: clear ent->fuse_req in commit_fetch error path

[Berkant Koc]

On 2026-05-17 16:11, Bernd Schubert wrote:
> We already had a security report for that on Friday [...] I had
> already replied to Zhenghang on Friday, I don't think it is enough.
> [...] valid all over the copy operation (fuse_uring_prepare_send())

Thanks for the context. P1 is a duplicate of Zhenghang's Friday report,
please consider it withdrawn.

You are right that clearing ent->fuse_req only in the commit_fetch error
path is not sufficient. The same window is reachable across the whole
copy path in fuse_uring_prepare_send(), so a single-point clear leaves
the race open on the other exits. I will not push a v2 for this one and
leave the scope call to you.

P2 ([PATCH 2/2] serialize ring teardown and per-ent setup against
ent->state writers) is a separate path: ent->state being written without
the queue lock while teardown frees the ring. If that overlaps with what
you are looking at today, I will hold off on P2 as well. If it is out of
scope for your work, a short note is enough and I will keep tracking it
independently.

KASAN config and the repro harness (qemu + libfuse uring example with
abort-on-mount) are set up here, happy to test your fix once it is on
the list.

Thanks,
Berkant

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Bernd Schubert]

On 5/17/26 14:59, Berkant Koc wrote:
> [You don't often get email from me@berkoc.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> From: Berkant Koc <me@berkoc.com>
> 
> fuse_dev_release() on the last fuse_dev of a connection calls
> fuse_abort_conn(fc) and then immediately fuse_conn_put(fc). For io-uring
> backed connections fuse_abort_conn() reaches fuse_uring_abort(), which
> runs fuse_uring_teardown_all_queues() synchronously once and then
> schedules ring->async_teardown_work to run after
> FUSE_URING_TEARDOWN_INTERVAL (HZ/20). If the synchronous pass left
> queue_refs > 0 the work owns further accesses to ring->queues[*]->
> ent_avail_queue and ent_in_userspace entries.
> 
> Meanwhile fuse_conn_put() can drop the last reference and arm
> delayed_release() via call_rcu(). After the RCU grace period
> delayed_release() calls fuse_uring_destruct(), which kfree()s the ring
> entries on each queue->ent_released list. The previously scheduled
> async_teardown_work then runs and walks per-queue lists that contain
> freed entries, producing a slab-use-after-free reported by KASAN at
> fuse_uring_teardown_all_queues+0xee reading ent->list.next from a
> freed kmalloc-192 region.
> 
> fuse_wait_aborted() already exists for this purpose: it waits on
> fc->blocked_waitq for num_waiting to drain and then calls
> fuse_uring_wait_stopped_queues(), which waits for ring->queue_refs to
> reach zero. Call it between fuse_abort_conn() and fuse_conn_put() on
> the last-device path so the io-uring teardown work has fully drained
> before the connection can be torn down.
> 
> Fixes: c090c8abae4b ("fuse: Add io-uring sqe commit and fetch support")
> Cc: stable@vger.kernel.org # 6.14+
> Tested-by: Berkant Koc <me@berkoc.com>
> Signed-off-by: Berkant Koc <me@berkoc.com>
> ---
>  fs/fuse/dev.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
> index 5dda7080f4a9..7d9c06654a98 100644
> --- a/fs/fuse/dev.c
> +++ b/fs/fuse/dev.c
> @@ -2566,6 +2566,7 @@ int fuse_dev_release(struct inode *inode, struct file *file)
>                 if (last) {
>                         WARN_ON(fc->iq.fasync != NULL);
>                         fuse_abort_conn(fc);
> +                       fuse_wait_aborted(fc);
>                 }
>                 fuse_conn_put(fc);
>         }

I might be wrong, but I don't think it is possible, Maybe Pavel or Jens
could help (added to CC). Basically as long as
fuse_uring_async_stop_queues() runs we do not have completed all
io-uring commands via io_uring_cmd_done() and as long as we do not have
completed these io-uring commands.

References are taken here

io_issue_sqe
  io_assign_file
     io_file_get_normal
        fget


  __io_uring_cmd_done()
    req->io_task_work.func = io_req_task_complete
    io_req_task_work_add(req)             io_uring/uring_cmd.c:179



io_req_task_complete
   io_req_complete_defer
     wq_list_add_tail

__io_submit_flush_completions
   io_free_batch_list
     io_put_file
       fput


I also look it up for fixed file and for fixed file it needs
io_ring_ctx_free(), which is then only one after completing all
uring_cmd objects (io_uring_cmd_done).


I don't mind adding fuse_wait_aborted() here, but IMHO it is a no-op and
no security issue.


Thanks
Bernd

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Berkant Koc]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain, Size: 1867 bytes --]

Bernd, thanks for pushing back. Stepping through this against the trace:

fuse_conn_destroy() in fs/fuse/inode.c calls fuse_wait_aborted()
between fuse_abort_conn() and the eventual fuse_conn_put() (from
fuse_sb_destroy). fuse_dev_release() in fs/fuse/dev.c does not wait
between its fuse_abort_conn() and fuse_conn_put(). That asymmetry is
the race.

On topologies where the last fud release IS the last conn ref
(no superblock mount, no other fud open — exactly the PoC setup),
fuse_conn_put() drops the count to zero, call_rcu schedules
delayed_release, and fuse_uring_destruct kfrees ring/queue/ent_released
slabs. async_teardown_work, scheduled by fuse_uring_async_stop_queues
via the teardown-interval delayed_work, then runs on freed memory.

The KASAN trace at top-finding/kasan-trace.txt shows exactly that
interleaving:

  free site: fuse_uring_destruct ← delayed_release ← rcu_core
  use site:  fuse_uring_teardown_all_queues ← async_teardown_work
             (workqueue), reading ent->list.next from
             kmalloc-192 freed by destruct

Your in-flight cmd ref invariant holds on both fixed and non-fixed
paths (non-fixed via per-cmd io_put_file in io_free_batch_list, fixed
via the io_uring file table slot pinning struct file → fud → fuse_conn).
But neither covers the gap between fuse_abort_conn (which schedules
the async work and returns immediately) and the RCU callback. The
PoC topology removes every other ref-holder, so that gap becomes the
last conn ref.

The patch restores symmetry with fuse_conn_destroy by waiting on
ring->queue_refs == 0 (via fuse_wait_aborted → fuse_uring_wait_stopped_queues)
before the put. That guarantees async_teardown_work has finished
before RCU is armed.

The race is reproducible with mdelay-widening; without widening I see
0 trips in 50 iter, but the window is in the code paths.

Berkant

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Pavel Begunkov]

On 5/17/26 16:00, Bernd Schubert wrote:
> On 5/17/26 14:59, Berkant Koc wrote:
>> [You don't often get email from me@berkoc.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> From: Berkant Koc <me@berkoc.com>
>>
>> fuse_dev_release() on the last fuse_dev of a connection calls
>> fuse_abort_conn(fc) and then immediately fuse_conn_put(fc). For io-uring
>> backed connections fuse_abort_conn() reaches fuse_uring_abort(), which
>> runs fuse_uring_teardown_all_queues() synchronously once and then
>> schedules ring->async_teardown_work to run after
>> FUSE_URING_TEARDOWN_INTERVAL (HZ/20). If the synchronous pass left
>> queue_refs > 0 the work owns further accesses to ring->queues[*]->
>> ent_avail_queue and ent_in_userspace entries.
>>
>> Meanwhile fuse_conn_put() can drop the last reference and arm
>> delayed_release() via call_rcu(). After the RCU grace period
>> delayed_release() calls fuse_uring_destruct(), which kfree()s the ring
>> entries on each queue->ent_released list. The previously scheduled
>> async_teardown_work then runs and walks per-queue lists that contain
>> freed entries, producing a slab-use-after-free reported by KASAN at
>> fuse_uring_teardown_all_queues+0xee reading ent->list.next from a
>> freed kmalloc-192 region.
>>
>> fuse_wait_aborted() already exists for this purpose: it waits on
>> fc->blocked_waitq for num_waiting to drain and then calls
>> fuse_uring_wait_stopped_queues(), which waits for ring->queue_refs to
>> reach zero. Call it between fuse_abort_conn() and fuse_conn_put() on
>> the last-device path so the io-uring teardown work has fully drained
>> before the connection can be torn down.
>>
>> Fixes: c090c8abae4b ("fuse: Add io-uring sqe commit and fetch support")
>> Cc: stable@vger.kernel.org # 6.14+
>> Tested-by: Berkant Koc <me@berkoc.com>
>> Signed-off-by: Berkant Koc <me@berkoc.com>
>> ---
>>   fs/fuse/dev.c | 1 +
>>   1 file changed, 1 insertion(+)
>>
>> diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
>> index 5dda7080f4a9..7d9c06654a98 100644
>> --- a/fs/fuse/dev.c
>> +++ b/fs/fuse/dev.c
>> @@ -2566,6 +2566,7 @@ int fuse_dev_release(struct inode *inode, struct file *file)
>>                  if (last) {
>>                          WARN_ON(fc->iq.fasync != NULL);
>>                          fuse_abort_conn(fc);
>> +                       fuse_wait_aborted(fc);
>>                  }
>>                  fuse_conn_put(fc);
>>          }
> 
> I might be wrong, but I don't think it is possible, Maybe Pavel or Jens
> could help (added to CC). Basically as long as
> fuse_uring_async_stop_queues() runs we do not have completed all
> io-uring commands via io_uring_cmd_done() and as long as we do not have
> completed these io-uring commands.

If I understand the question right, yes, fuse io_uring cmd requests hold
a reference to the fuse file, so until you complete them the file will
not get released.

-- 
Pavel Begunkov

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Bernd Schubert]

On 5/18/26 11:06, Pavel Begunkov wrote:
> On 5/17/26 16:00, Bernd Schubert wrote:
>> On 5/17/26 14:59, Berkant Koc wrote:
>>> [You don't often get email from me@berkoc.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>>
>>> From: Berkant Koc <me@berkoc.com>
>>>
>>> fuse_dev_release() on the last fuse_dev of a connection calls
>>> fuse_abort_conn(fc) and then immediately fuse_conn_put(fc). For io-uring
>>> backed connections fuse_abort_conn() reaches fuse_uring_abort(), which
>>> runs fuse_uring_teardown_all_queues() synchronously once and then
>>> schedules ring->async_teardown_work to run after
>>> FUSE_URING_TEARDOWN_INTERVAL (HZ/20). If the synchronous pass left
>>> queue_refs > 0 the work owns further accesses to ring->queues[*]->
>>> ent_avail_queue and ent_in_userspace entries.
>>>
>>> Meanwhile fuse_conn_put() can drop the last reference and arm
>>> delayed_release() via call_rcu(). After the RCU grace period
>>> delayed_release() calls fuse_uring_destruct(), which kfree()s the ring
>>> entries on each queue->ent_released list. The previously scheduled
>>> async_teardown_work then runs and walks per-queue lists that contain
>>> freed entries, producing a slab-use-after-free reported by KASAN at
>>> fuse_uring_teardown_all_queues+0xee reading ent->list.next from a
>>> freed kmalloc-192 region.
>>>
>>> fuse_wait_aborted() already exists for this purpose: it waits on
>>> fc->blocked_waitq for num_waiting to drain and then calls
>>> fuse_uring_wait_stopped_queues(), which waits for ring->queue_refs to
>>> reach zero. Call it between fuse_abort_conn() and fuse_conn_put() on
>>> the last-device path so the io-uring teardown work has fully drained
>>> before the connection can be torn down.
>>>
>>> Fixes: c090c8abae4b ("fuse: Add io-uring sqe commit and fetch support")
>>> Cc: stable@vger.kernel.org # 6.14+
>>> Tested-by: Berkant Koc <me@berkoc.com>
>>> Signed-off-by: Berkant Koc <me@berkoc.com>
>>> ---
>>>   fs/fuse/dev.c | 1 +
>>>   1 file changed, 1 insertion(+)
>>>
>>> diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
>>> index 5dda7080f4a9..7d9c06654a98 100644
>>> --- a/fs/fuse/dev.c
>>> +++ b/fs/fuse/dev.c
>>> @@ -2566,6 +2566,7 @@ int fuse_dev_release(struct inode *inode, struct file *file)
>>>                  if (last) {
>>>                          WARN_ON(fc->iq.fasync != NULL);
>>>                          fuse_abort_conn(fc);
>>> +                       fuse_wait_aborted(fc);
>>>                  }
>>>                  fuse_conn_put(fc);
>>>          }
>>
>> I might be wrong, but I don't think it is possible, Maybe Pavel or Jens
>> could help (added to CC). Basically as long as
>> fuse_uring_async_stop_queues() runs we do not have completed all
>> io-uring commands via io_uring_cmd_done() and as long as we do not have
>> completed these io-uring commands.
> 
> If I understand the question right, yes, fuse io_uring cmd requests hold
> a reference to the fuse file, so until you complete them the file will
> not get released.


Sorry, had totally messed up the phrase, can't read it myself.

What I mean is that the io-uring was set up with /dev/fuse as file and
as long as fuse holds non-completed 'struct io_uring_cmd *cmd' objects
there is a reference on the /dev/fuse fd, which blocks the call of
fuse_dev_release().


Thanks,
Bernd

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Bernd Schubert]

On 5/18/26 03:13, Berkant Koc wrote:
> [You don't often get email from me@berkoc.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
> 
> Bernd, thanks for pushing back. Stepping through this against the trace:
> 
> fuse_conn_destroy() in fs/fuse/inode.c calls fuse_wait_aborted()
> between fuse_abort_conn() and the eventual fuse_conn_put() (from
> fuse_sb_destroy). fuse_dev_release() in fs/fuse/dev.c does not wait
> between its fuse_abort_conn() and fuse_conn_put(). That asymmetry is
> the race.
> 
> On topologies where the last fud release IS the last conn ref
> (no superblock mount, no other fud open — exactly the PoC setup),
> fuse_conn_put() drops the count to zero, call_rcu schedules
> delayed_release, and fuse_uring_destruct kfrees ring/queue/ent_released
> slabs. async_teardown_work, scheduled by fuse_uring_async_stop_queues
> via the teardown-interval delayed_work, then runs on freed memory.
> 
> The KASAN trace at top-finding/kasan-trace.txt shows exactly that
> interleaving:
> 
>   free site: fuse_uring_destruct ← delayed_release ← rcu_core
>   use site:  fuse_uring_teardown_all_queues ← async_teardown_work
>              (workqueue), reading ent->list.next from
>              kmalloc-192 freed by destruct
> 
> Your in-flight cmd ref invariant holds on both fixed and non-fixed
> paths (non-fixed via per-cmd io_put_file in io_free_batch_list, fixed
> via the io_uring file table slot pinning struct file → fud → fuse_conn).
> But neither covers the gap between fuse_abort_conn (which schedules
> the async work and returns immediately) and the RCU callback. The
> PoC topology removes every other ref-holder, so that gap becomes the
> last conn ref.
> 
> The patch restores symmetry with fuse_conn_destroy by waiting on
> ring->queue_refs == 0 (via fuse_wait_aborted → fuse_uring_wait_stopped_queues)
> before the put. That guarantees async_teardown_work has finished
> before RCU is armed.
> 
> The race is reproducible with mdelay-widening; without widening I see
> 0 trips in 50 iter, but the window is in the code paths.

I think I see what the actual issue is, we need an fc (or in linux-next
struct fuse_chan) reference as long as fuse_uring_async_stop_queues()
runs. Patch follows.


Thanks,
Bernd

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Pavel Begunkov]

On 5/18/26 10:50, Bernd Schubert wrote:

>>>> diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
>>>> index 5dda7080f4a9..7d9c06654a98 100644
>>>> --- a/fs/fuse/dev.c
>>>> +++ b/fs/fuse/dev.c
>>>> @@ -2566,6 +2566,7 @@ int fuse_dev_release(struct inode *inode, struct file *file)
>>>>                   if (last) {
>>>>                           WARN_ON(fc->iq.fasync != NULL);
>>>>                           fuse_abort_conn(fc);
>>>> +                       fuse_wait_aborted(fc);
>>>>                   }
>>>>                   fuse_conn_put(fc);
>>>>           }
>>>
>>> I might be wrong, but I don't think it is possible, Maybe Pavel or Jens
>>> could help (added to CC). Basically as long as
>>> fuse_uring_async_stop_queues() runs we do not have completed all
>>> io-uring commands via io_uring_cmd_done() and as long as we do not have
>>> completed these io-uring commands.
>>
>> If I understand the question right, yes, fuse io_uring cmd requests hold
>> a reference to the fuse file, so until you complete them the file will
>> not get released.
> 
> 
> Sorry, had totally messed up the phrase, can't read it myself.
> 
> What I mean is that the io-uring was set up with /dev/fuse as file and
> as long as fuse holds non-completed 'struct io_uring_cmd *cmd' objects
> there is a reference on the /dev/fuse fd, which blocks the call of
> fuse_dev_release().

Yep, that's right

-- 
Pavel Begunkov

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Bernd Schubert]

[-- Attachment #1: Type: text/plain, Size: 2480 bytes --]

On 5/18/26 11:55, Bernd Schubert wrote:
> On 5/18/26 03:13, Berkant Koc wrote:
>> [You don't often get email from me@berkoc.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>
>> Bernd, thanks for pushing back. Stepping through this against the trace:
>>
>> fuse_conn_destroy() in fs/fuse/inode.c calls fuse_wait_aborted()
>> between fuse_abort_conn() and the eventual fuse_conn_put() (from
>> fuse_sb_destroy). fuse_dev_release() in fs/fuse/dev.c does not wait
>> between its fuse_abort_conn() and fuse_conn_put(). That asymmetry is
>> the race.
>>
>> On topologies where the last fud release IS the last conn ref
>> (no superblock mount, no other fud open — exactly the PoC setup),
>> fuse_conn_put() drops the count to zero, call_rcu schedules
>> delayed_release, and fuse_uring_destruct kfrees ring/queue/ent_released
>> slabs. async_teardown_work, scheduled by fuse_uring_async_stop_queues
>> via the teardown-interval delayed_work, then runs on freed memory.
>>
>> The KASAN trace at top-finding/kasan-trace.txt shows exactly that
>> interleaving:
>>
>>   free site: fuse_uring_destruct ← delayed_release ← rcu_core
>>   use site:  fuse_uring_teardown_all_queues ← async_teardown_work
>>              (workqueue), reading ent->list.next from
>>              kmalloc-192 freed by destruct
>>
>> Your in-flight cmd ref invariant holds on both fixed and non-fixed
>> paths (non-fixed via per-cmd io_put_file in io_free_batch_list, fixed
>> via the io_uring file table slot pinning struct file → fud → fuse_conn).
>> But neither covers the gap between fuse_abort_conn (which schedules
>> the async work and returns immediately) and the RCU callback. The
>> PoC topology removes every other ref-holder, so that gap becomes the
>> last conn ref.
>>
>> The patch restores symmetry with fuse_conn_destroy by waiting on
>> ring->queue_refs == 0 (via fuse_wait_aborted → fuse_uring_wait_stopped_queues)
>> before the put. That guarantees async_teardown_work has finished
>> before RCU is armed.
>>
>> The race is reproducible with mdelay-widening; without widening I see
>> 0 trips in 50 iter, but the window is in the code paths.
> 
> I think I see what the actual issue is, we need an fc (or in linux-next
> struct fuse_chan) reference as long as fuse_uring_async_stop_queues()
> runs. Patch follows.
> 

Would it be possible for you to test the attached patch?


Thanks,
Bernd


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: fuse_uring_async_stop_queues.patch --]
[-- Type: text/x-patch; name="fuse_uring_async_stop_queues.patch", Size: 1561 bytes --]

commit b6f63cc1d2f7b8d690835c965d600fda47ffd79b
Author: Bernd Schubert <bernd@bsbernd.com>
Date:   Mon May 18 13:30:30 2026 +0200

    fuse: Avoid use-after-free in fuse_uring_async_stop_queues
    
    fuse_uring_async_stop_queues() might run when the last reference
    on ring->queue_refs was already dropped.
    
    In order to avoid an early destruction a reference on struct fuse_conn
    is now taken before starting fuse_uring_async_stop_queues() and that
    reference is only released when that delayed work queue terminates.
    
    Fixes: 4a9bfb9b6850 ("fuse: {io-uring} Handle teardown of ring entries")
    Cc: stable@kernel.org # 6.14
    Reported-by: Berkant Koc <me@berkoc.com>
    Signed-off-by: Bernd Schubert <bernd@bsbernd.com>

diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
index e467b23e6895..c96d72250830 100644
--- a/fs/fuse/dev_uring.c
+++ b/fs/fuse/dev_uring.c
@@ -465,7 +465,10 @@ static void fuse_uring_async_stop_queues(struct work_struct *work)
 		schedule_delayed_work(&ring->async_teardown_work,
 				      FUSE_URING_TEARDOWN_INTERVAL);
 	} else {
+		struct fuse_chan *chan = ring->chan;
+
 		wake_up_all(&ring->stop_waitq);
+		fuse_conn_put(chan->conn);
 	}
 }
 
@@ -477,6 +480,9 @@ void fuse_uring_stop_queues(struct fuse_ring *ring)
 	fuse_uring_teardown_all_queues(ring);
 
 	if (atomic_read(&ring->queue_refs) > 0) {
+		struct fuse_chan *chan = ring->chan;
+
+		fuse_conn_get(chan->conn);
 		ring->teardown_time = jiffies;
 		INIT_DELAYED_WORK(&ring->async_teardown_work,
 				  fuse_uring_async_stop_queues);

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Berkant Koc]

On Mon, 18 May 2026 11:47:00 +0000, Bernd Schubert <bschubert@ddn.com> wrote:
> Would it be possible for you to test the attached patch?

Reproducer and KASAN harness from the PATCH 2/2 series are staged.
Two-arm plan: revert vs apply, race-widening debug hunk kept in both
arms, 2x50 iterations each against torvalds/master tip, KASAN + lockdep
+ kmemleak enabled. Results back within the day once the base resolves.

Blocker before I build. The patch references ring->chan and chan->conn.
On mainline fs/fuse/dev_uring_i.h declares struct fuse_ring with
struct fuse_conn *fc at line 110, no chan member; grep fuse_chan
across fs/fuse/ returns zero hits. As-is the patch fails to compile
with "struct fuse_ring has no member named chan".

Is this based on a DDN topic branch that introduces a fuse_chan
abstraction not yet upstream? If so, point me at the base tree or
branch URL and I will rebase the test against that. If the references
were meant to be ring->fc and fc against current mainline, confirm and
I will adjust before the run.

Assisted-by: Claude:claude-opus-4-7 berkoc-pipeline

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Bernd Schubert]

On 5/18/26 16:32, Berkant Koc wrote:
> On Mon, 18 May 2026 11:47:00 +0000, Bernd Schubert <bschubert@ddn.com> wrote:
>> Would it be possible for you to test the attached patch?
> 
> Reproducer and KASAN harness from the PATCH 2/2 series are staged.
> Two-arm plan: revert vs apply, race-widening debug hunk kept in both
> arms, 2x50 iterations each against torvalds/master tip, KASAN + lockdep
> + kmemleak enabled. Results back within the day once the base resolves.
> 
> Blocker before I build. The patch references ring->chan and chan->conn.
> On mainline fs/fuse/dev_uring_i.h declares struct fuse_ring with
> struct fuse_conn *fc at line 110, no chan member; grep fuse_chan
> across fs/fuse/ returns zero hits. As-is the patch fails to compile
> with "struct fuse_ring has no member named chan".
> 
> Is this based on a DDN topic branch that introduces a fuse_chan
> abstraction not yet upstream? If so, point me at the base tree or
> branch URL and I will rebase the test against that. If the references
> were meant to be ring->fc and fc against current mainline, confirm and
> I will adjust before the run.
> 
> Assisted-by: Claude:claude-opus-4-7 berkoc-pipeline
> 

Ah, it is based on Miklos' for-next branch, which is also in linux-next
(I think). Yeah, we have a bit back port headache here.


Thanks,
Bernd

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Joanne Koong]

On Mon, May 18, 2026 at 7:46 AM Bernd Schubert <bernd@bsbernd.com> wrote:
>
>
>
> On 5/18/26 16:32, Berkant Koc wrote:
> > On Mon, 18 May 2026 11:47:00 +0000, Bernd Schubert <bschubert@ddn.com> wrote:
> >> Would it be possible for you to test the attached patch?

The fix looks right to me.

Reviewed-by: Joanne Koong <joannelkoong@gmail.com>

> >
> > Reproducer and KASAN harness from the PATCH 2/2 series are staged.
> > Two-arm plan: revert vs apply, race-widening debug hunk kept in both
> > arms, 2x50 iterations each against torvalds/master tip, KASAN + lockdep
> > + kmemleak enabled. Results back within the day once the base resolves.
> >
> > Blocker before I build. The patch references ring->chan and chan->conn.
> > On mainline fs/fuse/dev_uring_i.h declares struct fuse_ring with
> > struct fuse_conn *fc at line 110, no chan member; grep fuse_chan
> > across fs/fuse/ returns zero hits. As-is the patch fails to compile
> > with "struct fuse_ring has no member named chan".
> >
> > Is this based on a DDN topic branch that introduces a fuse_chan
> > abstraction not yet upstream? If so, point me at the base tree or
> > branch URL and I will rebase the test against that. If the references
> > were meant to be ring->fc and fc against current mainline, confirm and
> > I will adjust before the run.

Yes, on mainline the references are meant to be ring->fc, eg

diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
index 5c00fd047c8b..27c417fd9451 100644
--- a/fs/fuse/dev_uring.c
+++ b/fs/fuse/dev_uring.c
@@ -570,6 +570,7 @@ static void fuse_uring_async_stop_queues(struct
work_struct *work)
                                      FUSE_URING_TEARDOWN_INTERVAL);
        } else {
                wake_up_all(&ring->stop_waitq);
+               fuse_conn_put(ring->fc);
        }
 }

@@ -599,6 +600,7 @@ void fuse_uring_stop_queues(struct fuse_ring *ring)

        if (atomic_read(&ring->queue_refs) > 0) {
                ring->teardown_time = jiffies;
+               fuse_conn_get(ring->fc);
                INIT_DELAYED_WORK(&ring->async_teardown_work,
                                  fuse_uring_async_stop_queues);
                schedule_delayed_work(&ring->async_teardown_work,

Thanks,
Joanne

> >
> > Assisted-by: Claude:claude-opus-4-7 berkoc-pipeline
> >
>
> Ah, it is based on Miklos' for-next branch, which is also in linux-next
> (I think). Yeah, we have a bit back port headache here.
>
>
> Thanks,
> Bernd

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Berkant Koc]

On Mon, 18 May 2026 16:46:08 +0200, Bernd Schubert <bernd@bsbernd.com> wrote:
> Ah, it is based on Miklos' for-next branch, which is also in linux-next
> (I think). Yeah, we have a bit back port headache here.

Got it, that resolves the field-name confusion on my side. I will rebase
the test harness onto Miklos' fuse.git for-next tip (and cross-check
against linux-next), then rerun the two-arm comparison against that base:
revert vs apply with the abort/release race-widening kept in place,
2x50 iterations each under KASAN + lockdep + kmemleak, with full
splats and reproducer attached on report-back.

The stable-backport question is parked until the for-next arm lands
clean. Once the trace set is in hand I will fold the v6.14 field-name
delta into a separate backport note rather than mixing it into the
mainline result.

Assisted-by: Claude:claude-opus-4-7 berkoc-pipeline

Re: [PATCH 2/2] fuse: wait for aborted connection before releasing last fuse_dev

[Berkant Koc]

On Mon, 18 May 2026 08:35:16 -0700, Joanne Koong <joannelkoong@gmail.com> wrote:
> Yes, on mainline the references are meant to be ring->fc, eg

Confirmed, ring->fc on mainline. Reviewed-by recorded for Bernd's v6.14 backport path.

Bernd has since published the full 4-patch series via B4-Relay at 18:37 CEST, rebased onto Miklos' for-next (base-commit 040d71ac6470). On that branch the fuse_chan abstraction is in place, so the teardown path uses ring->chan / chan->conn rather than ring->fc. The series stays as-is for for-next.

Your inline diff is the basis for the mainline stable-backport once for-next lands.

I will run the two-arm Tested-by against for-next-base 040d71ac6470 (KASAN + lockdep + kmemleak, async-teardown race + baseline) and report numbers on-thread.

Assisted-by: Claude:claude-opus-4-7 berkoc-pipeline