Re: [PATCH RFC v4 00/15] fuse: fuse-over-io-uring

[Bernd Schubert <bernd.schubert@fastmail.fm>]

On 10/21/24 22:57, David Wei wrote:
> On 2024-10-21 04:47, Bernd Schubert wrote:
>> Hi David,
>>
>> On 10/21/24 06:06, David Wei wrote:
>>> [You don't often get email from dw@davidwei.uk. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>>>
>>> On 2024-10-15 17:05, Bernd Schubert wrote:
>>> [...]
>>>>
>>
>> ...
>>
>>> Hi Bernd, I applied this patchset to io_uring-6.12 branch with some
>>> minor conflicts. I'm running the following command:
>>>
>>> $ sudo ./build/example/passthrough_hp -o allow_other --debug-fuse --nopassthrough \
>>> --uring --uring-per-core-queue --uring-fg-depth=1 --uring-bg-depth=1 \
>>> /home/vmuser/scratch/source /home/vmuser/scratch/dest
>>> FUSE library version: 3.17.0
>>> Creating ring per-core-queue=1 sync-depth=1 async-depth=1 arglen=1052672
>>> dev unique: 2, opcode: INIT (26), nodeid: 0, insize: 104, pid: 0
>>> INIT: 7.40
>>> flags=0x73fffffb
>>> max_readahead=0x00020000
>>>      INIT: 7.40
>>>      flags=0x4041f429
>>>      max_readahead=0x00020000
>>>      max_write=0x00100000
>>>      max_background=0
>>>      congestion_threshold=0
>>>      time_gran=1
>>>      unique: 2, success, outsize: 80
>>>
>>> I created the source and dest folders which are both empty.
>>>
>>> I see the following in dmesg:
>>>
>>> [ 2453.197510] uring is disabled
>>> [ 2453.198525] uring is disabled
>>> [ 2453.198749] uring is disabled
>>> ...
>>>
>>> If I then try to list the directory /home/vmuser/scratch:
>>>
>>> $ ls -l /home/vmuser/scratch
>>> ls: cannot access 'dest': Software caused connection abort
>>>
>>> And passthrough_hp terminates.
>>>
>>> My kconfig:
>>>
>>> CONFIG_FUSE_FS=m
>>> CONFIG_FUSE_PASSTHROUGH=y
>>> CONFIG_FUSE_IO_URING=y
>>>
>>> I'll look into it next week but, do you see anything obviously wrong?
>>
>>
>> thanks for testing it! I just pushed a fix to my libfuse branches to
>> avoid the abort for -EOPNOTSUPP. It will gracefully fall back to
>> /dev/fuse IO now.
>>
>> Could you please use the rfcv4 branch, as the plain uring
>> branch will soon get incompatible updates for rfc5?
>>
>> https://github.com/bsbernd/libfuse/tree/uring-for-rfcv4
>>
>>
>> The short answer to let you enable fuse-io-uring:
>>
>> echo 1 >/sys/module/fuse/parameters/enable_uring
>>
>>
>> (With that the "uring is disabled" should be fixed.)
> 
> Thanks, using this branch fixed the issue and now I can see the dest
> folder mirroring that of the source folder. There are two issues I
> noticed:
> 
> [63490.068211] ---[ end trace 0000000000000000 ]---
> [64010.242963] BUG: sleeping function called from invalid context at include/linux/sched/mm.h:330
> [64010.243531] in_atomic(): 1, irqs_disabled(): 0, non_block: 0, pid: 11057, name: fuse-ring-1
> [64010.244092] preempt_count: 1, expected: 0
> [64010.244346] RCU nest depth: 0, expected: 0
> [64010.244599] 2 locks held by fuse-ring-1/11057:
> [64010.244886]  #0: ffff888105db20a8 (&ctx->uring_lock){+.+.}-{3:3}, at: __do_sys_io_uring_enter+0x900/0xd80
> [64010.245476]  #1: ffff88810f941818 (&fc->lock){+.+.}-{2:2}, at: fuse_uring_cmd+0x83e/0x1890 [fuse]
> [64010.246031] CPU: 1 UID: 0 PID: 11057 Comm: fuse-ring-1 Tainted: G        W          6.11.0-10089-g0d2090ccdbbe #2
> [64010.246655] Tainted: [W]=WARN
> [64010.246853] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.16.0-0-gd239552ce722-prebuilt.qemu.org 04/01/2014
> [64010.247542] Call Trace:
> [64010.247705]  <TASK>
> [64010.247860]  dump_stack_lvl+0xb0/0xd0
> [64010.248090]  __might_resched+0x2f8/0x510
> [64010.248338]  __kmalloc_cache_noprof+0x2aa/0x390
> [64010.248614]  ? lockdep_init_map_type+0x2cb/0x7b0
> [64010.248923]  ? fuse_uring_cmd+0xcc2/0x1890 [fuse]
> [64010.249215]  fuse_uring_cmd+0xcc2/0x1890 [fuse]
> [64010.249506]  io_uring_cmd+0x214/0x500
> [64010.249745]  io_issue_sqe+0x588/0x1810
> [64010.249999]  ? __pfx_io_issue_sqe+0x10/0x10
> [64010.250254]  ? io_alloc_async_data+0x88/0x120
> [64010.250516]  ? io_alloc_async_data+0x88/0x120
> [64010.250811]  ? io_uring_cmd_prep+0x2eb/0x9f0
> [64010.251103]  io_submit_sqes+0x796/0x1f80
> [64010.251387]  __do_sys_io_uring_enter+0x90a/0xd80
> [64010.251696]  ? do_user_addr_fault+0x26f/0xb60
> [64010.251991]  ? __pfx___do_sys_io_uring_enter+0x10/0x10
> [64010.252333]  ? __up_read+0x3ba/0x750
> [64010.252565]  ? __pfx___up_read+0x10/0x10
> [64010.252868]  do_syscall_64+0x68/0x140
> [64010.253121]  entry_SYSCALL_64_after_hwframe+0x76/0x7e
> [64010.253444] RIP: 0033:0x7f03a03fb7af
> [64010.253679] Code: 45 0f b6 90 d0 00 00 00 41 8b b8 cc 00 00 00 45 31 c0 41 b9 08 00 00 00 41 83 e2 01 41 c1 e2 04 41 09 c2 b8 aa 01 00 00 0f 05 <c3> a8 02 74 cc f0 48 83 0c 24 00 49 8b 40 20 8b 00 a8 01 74 bc b8
> [64010.254801] RSP: 002b:00007f039f3ffd08 EFLAGS: 00000246 ORIG_RAX: 00000000000001aa
> [64010.255261] RAX: ffffffffffffffda RBX: 0000561ab7c1ced0 RCX: 00007f03a03fb7af
> [64010.255695] RDX: 0000000000000000 RSI: 0000000000000002 RDI: 0000000000000009
> [64010.256127] RBP: 0000000000000002 R08: 0000000000000000 R09: 0000000000000008
> [64010.256556] R10: 0000000000000000 R11: 0000000000000246 R12: 0000561ab7c1d7a8
> [64010.256990] R13: 0000561ab7c1da00 R14: 0000561ab7c1d520 R15: 0000000000000001
> [64010.257442]  </TASK>

Regarding issue one, does this patch solve it?

diff --git a/fs/fuse/dev_uring.c b/fs/fuse/dev_uring.c
index e518d4379aa1..304919bc12fb 100644
--- a/fs/fuse/dev_uring.c
+++ b/fs/fuse/dev_uring.c
@@ -168,6 +168,12 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
         queue = kzalloc(sizeof(*queue), GFP_KERNEL_ACCOUNT);
         if (!queue)
                 return ERR_PTR(-ENOMEM);
+       pq = kcalloc(FUSE_PQ_HASH_SIZE, sizeof(struct list_head), GFP_KERNEL);
+       if (!pq) {
+               kfree(queue);
+               return ERR_PTR(-ENOMEM);
+       }
+
         spin_lock(&fc->lock);
         if (ring->queues[qid]) {
                 spin_unlock(&fc->lock);
@@ -186,11 +192,6 @@ static struct fuse_ring_queue *fuse_uring_create_queue(struct fuse_ring *ring,
         INIT_LIST_HEAD(&queue->ent_in_userspace);
         INIT_LIST_HEAD(&queue->fuse_req_queue);

-       pq = kcalloc(FUSE_PQ_HASH_SIZE, sizeof(struct list_head), GFP_KERNEL);
-       if (!pq) {
-               kfree(queue);
-               return ERR_PTR(-ENOMEM);
-       }
         queue->fpq.processing = pq;
         fuse_pqueue_init(&queue->fpq);


I think we don't need GFP_ATOMIC, but can do allocations before taking
the lock. This pq allocation is new in v4 and I forgot to put it into
the right place and it slipped through my very basic testing (I'm
concentrating on the design changes for now - testing will come back
with v6).

> 
> If I am already in dest when I do the mount using passthrough_hp and
> then e.g. ls, it hangs indefinitely even if I kill passthrough_hp.

I'm going to check in a bit. I hope it is not a recursion issue.


Thanks,
Bernd