question about fuse livelock situation

question about fuse livelock situation

[박영준]

I found fuse livelock situation and report it for possibility of problem.

[Environment]
22.04 5.15.0-43-generic ubuntu kernel.
ntfs-3g version ntfs-3g 2021.8.22 integrated FUSE 28 - Third
Generation NTFS Driver

[Problem]
I bumped on livelock and analyze it. and concluded that it is needed
to be fixed.
it happends when 3 operation concurrently progressing.

1) usb detach by user. and kernel detect it.
2) mount.ntfs umount request & device release operation
3) pool-udisksd umount operation.

[Conclusion]
1. mounted target device file must be released after /dev/fuse
release. it makes deadlocky scenario.
2. fuse file system "fuse_simple_request" should not be waiting
forever. it is hard to solve this situation by interrupting
application. huntask panic configuration make user kernel panic. user
don't know why.

[Analysis]
I got a kernel crash dump file. and analyze it.

here is the scenario description.
![1](https://user-images.githubusercontent.com/66734313/187164968-70880294-97bb-4ed9-807c-21d76b6f8f01.png)

1. kworker detect usb is detached from computer.
it is blocked by umount operation (pool-udiskd)
PID: 8      TASK: ffff88810029e400  CPU: 0   COMMAND: "kworker/0:1"
 #0 [ffffc9000006f6f0] __schedule at ffffffff81d57c0d
 #1 [ffffc9000006f778] schedule at ffffffff81d57fae
 #2 [ffffc9000006f798] rwsem_down_read_slowpath at ffffffff81d5a2de
 #3 [ffffc9000006f830] down_read at ffffffff81d5a373 // wait s_umount
 #4 [ffffc9000006f848] get_super at ffffffff813799ca
 #5 [ffffc9000006f878] fsync_bdev at ffffffff815c6f46
 #6 [ffffc9000006f8a0] delete_partition at ffffffff815e9328
 #7 [ffffc9000006f8c0] blk_drop_partitions at ffffffff815e9b3e
 #8 [ffffc9000006f8e8] del_gendisk at ffffffff815e75f1 // lock mutex
 #9 [ffffc9000006f930] sd_remove at ffffffff818cd325
#10 [ffffc9000006f958] __device_release_driver at ffffffff8184006f
#11 [ffffc9000006f990] device_release_driver at ffffffff818400a9
#12 [ffffc9000006f9b0] bus_remove_device at ffffffff8183f28e
#13 [ffffc9000006f9d8] device_del at ffffffff818399ac
#14 [ffffc9000006fa28] __scsi_remove_device at ffffffff818c2628
#15 [ffffc9000006fa50] scsi_forget_host at ffffffff818c029f
#16 [ffffc9000006fa70] scsi_remove_host at ffffffff818b3727
#17 [ffffc9000006fa98] usb_stor_disconnect at ffffffffc0659b20 [usb_storage]
#18 [ffffc9000006fac0] usb_unbind_interface at ffffffff8194ef30
#19 [ffffc9000006fb18] __device_release_driver at ffffffff8184006f
#20 [ffffc9000006fb50] device_release_driver at ffffffff818400a9
#21 [ffffc9000006fb70] bus_remove_device at ffffffff8183f28e
#22 [ffffc9000006fb98] device_del at ffffffff818399ac
#23 [ffffc9000006fbe8] usb_disable_device at ffffffff8194cdce
#24 [ffffc9000006fc30] usb_disconnect.cold at ffffffff81d19e09
#25 [ffffc9000006fc80] hub_port_connect at ffffffff81944bb8
#26 [ffffc9000006fd00] hub_port_connect_change at ffffffff819454b1
#27 [ffffc9000006fd70] port_event at ffffffff81945d77
#28 [ffffc9000006fe08] hub_event at ffffffff819460a7
#29 [ffffc9000006fe78] process_one_work at ffffffff810d9e7b
#30 [ffffc9000006fec8] worker_thread at ffffffff810da073
#31 [ffffc9000006ff10] kthread at ffffffff810e1cba
#32 [ffffc9000006ff50] ret_from_fork at ffffffff81004bc2


 #3 [ffffc9000006f830] down_read at ffffffff81d5a373
    ffffc9000006f838: [ffff888154d36800:kmalloc-2k] ffffc9000006f870
    ffffc9000006f848: get_super+154

2. it request umount. and release /dev/sdc file before release
/dev/fuse. but. it is blocked by usb detach.
PID: 18681  TASK: ffff88810e5b8000  CPU: 1   COMMAND: "mount.ntfs"
 #0 [ffffc90000ea7c50] __schedule at ffffffff81d57c0d
 #1 [ffffc90000ea7cd8] schedule at ffffffff81d57fae
 #2 [ffffc90000ea7cf8] schedule_preempt_disabled at ffffffff81d5839e
 #3 [ffffc90000ea7d08] __mutex_lock.constprop.0 at ffffffff81d59053
 #4 [ffffc90000ea7d80] __mutex_lock_slowpath at ffffffff81d59353
 #5 [ffffc90000ea7d90] mutex_lock at ffffffff81d59394 // wait open mutex
 #6 [ffffc90000ea7da8] blkdev_put at ffffffff815c765a
 #7 [ffffc90000ea7de0] blkdev_close at ffffffff815c86e7
 #8 [ffffc90000ea7df8] __fput at ffffffff8137704f
 #9 [ffffc90000ea7e30] ____fput at ffffffff8137724e
#10 [ffffc90000ea7e40] task_work_run at ffffffff810deb6d
#11 [ffffc90000ea7e68] exit_to_user_mode_loop at ffffffff81160fc0
#12 [ffffc90000ea7e90] exit_to_user_mode_prepare at ffffffff8116106c
#13 [ffffc90000ea7ea8] syscall_exit_to_user_mode at ffffffff81d4f327
#14 [ffffc90000ea7ec0] do_syscall_64 at ffffffff81d4b1b9
#15 [ffffc90000ea7ef8] exit_to_user_mode_prepare at ffffffff81161007
#16 [ffffc90000ea7f10] syscall_exit_to_user_mode at ffffffff81d4f327
#17 [ffffc90000ea7f28] do_syscall_64 at ffffffff81d4b1b9
    RIP: 00007f3e315fb117  RSP: 00007fff16bcb628  RFLAGS: 00000246
    RAX: 0000000000000000  RBX: 000055cc1b45c710  RCX: 00007f3e315fb117
    RDX: 0000000000000000  RSI: 0000000000000006  RDI: 0000000000000003
    RBP: 000055cc1b45c970   R8: 000055cc1b476940   R9: 0000000000000000
    R10: 0000000000000005  R11: 0000000000000246  R12: 00007f3e314e36c0
    R13: 000055cc1b45c710  R14: 000055cc1b467030  R15: 00000000ffffffff
    ORIG_RAX: 0000000000000003  CS: 0033  SS: 002b


 #3 [ffffc90000ea7d08] __mutex_lock.constprop.0 at ffffffff81d59053
    ffffc90000ea7d10: [ffff8881caad1880:kmalloc-512] 00000002547b2ff8
    ffffc90000ea7d20: [ffff8881caad1880:kmalloc-512]
[ffff8881caad1880:kmalloc-512]
    ffffc90000ea7d30: [ffff88810e5b8000:task_struct] 0000000008000008
    ffffc90000ea7d40: ffffc90000ea7de0 fcee7cb500545100
    ffffc90000ea7d50: [ffff888110f99f40:bdev_cache]
[ffff8881caad1870:kmalloc-512]
    ffffc90000ea7d60: [ffff8881caad1800:kmalloc-512]
[ffff8881caad1870:kmalloc-512]
    ffffc90000ea7d70: [ffff8881bcd4ccc0:dentry] ffffc90000ea7d88
    ffffc90000ea7d80: __mutex_lock_slowpath+19
 #4 [ffffc90000ea7d80] __mutex_lock_slowpath at ffffff

3. we can know that it is releasing /dev/sdc file and not release /dev/fuse.
crash> files 18681
PID: 18681  TASK: ffff88810e5b8000  CPU: 1   COMMAND: "mount.ntfs"
ROOT: /    CWD: /
 FD       FILE            DENTRY           INODE       TYPE PATH
  0 ffff888102f75200 ffff8881020510c0 ffff888100f84668 CHR  /dev/null
  1 ffff888102f75200 ffff8881020510c0 ffff888100f84668 CHR  /dev/null
  2 ffff888102f75200 ffff8881020510c0 ffff888100f84668 CHR  /dev/null
  4 ffff888154001f00 ffff8881020cce40 ffff888100f90f50 CHR  /dev/fuse
  5 ffff888102f74200 ffff8881c2c5dd80 ffff88815db73e40 SOCK UNIX
  8 ffff888106412800 ffff888104785540 ffff888104a9e470 REG  /proc/804/mountinfo
  9 ffff888106412100 ffff888100443840 ffff8881022a9688 REG  /proc/swaps
 14 ffff88810a720700 ffff888104785540 ffff888104a9e470 REG  /proc/804/mountinfo

4. pool udisksd is progressing umount process (cuz uhelper=udisksd2)
but it is waiting forever cuz mount.ntfs does not progress. (if it is
to be released, /dev/fuse
must be released. but mount.ntfs blocked by kworker)

PI3: 18741  TASK: ffff8881a702b200  CPU: 1   COMMAND: "pool-udisksd"
 #0 [ffffc900022a3be8] __schedule at ffffffff81d57c0d
 #1 [ffffc900022a3c70] schedule at ffffffff81d57fae
 #2 [ffffc900022a3c90] request_wait_answer at ffffffff814ea190
 #3 [ffffc900022a3cf0] fuse_simple_request at ffffffff814ec9c1
 #4 [ffffc900022a3d30] fuse_send_destroy at ffffffff814f743f
 #5 [ffffc900022a3db8] fuse_conn_destroy at ffffffff814f7622
 #6 [ffffc900022a3dd0] fuse_kill_sb_blk at ffffffff814f7d82
 #7 [ffffc900022a3e00] deactivate_locked_super at ffffffff8137863b
 #8 [ffffc900022a3e20] deactivate_super at ffffffff813786f0 // lock s_umount
 #9 [ffffc900022a3e38] cleanup_mnt at ffffffff8139f350
#10 [ffffc900022a3e78] __cleanup_mnt at ffffffff8139f422
#11 [ffffc900022a3e88] task_work_run at ffffffff810deb6d
#12 [ffffc900022a3eb0] exit_to_user_mode_loop at ffffffff81160fc0
#13 [ffffc900022a3ed8] exit_to_user_mode_prepare at ffffffff8116106c
#14 [ffffc900022a3ef0] syscall_exit_to_user_mode at ffffffff81d4f327
#15 [ffffc900022a3f08] do_syscall_64 at ffffffff81d4b1b9
#16 [ffffc900022a3f50] entry_SYSCALL_64_after_hwframe at ffffffff81e0007c
    RIP: 00007f110d17bbeb  RSP: 00007f110beb63b8  RFLAGS: 00000246
    RAX: 0000000000000000  RBX: 00007f10fc0257e0  RCX: 00007f110d17bbeb
    RDX: 0000000000000000  RSI: 0000000000000000  RDI: 00007f10fc012bb0
    RBP: 00007f10fc09f480   R8: 0000000000000000   R9: 00007f10fc05edf0
    R10: 00007f10fc05ee30  R11: 0000000000000246  R12: 0000000000000000
    R13: 00007f10fc012bb0  R14: 00007f10fc09f590  R15: 0000000000000000
    ORIG_RAX: 00000000000000a6  CS: 0033  SS: 002b

Re: question about fuse livelock situation

[Miklos Szeredi]

On Tue, 30 Aug 2022 at 03:58, 박영준 <her0gyugyu@gmail.com> wrote:
>
> I found fuse livelock situation and report it for possibility of problem.
>
> [Environment]
> 22.04 5.15.0-43-generic ubuntu kernel.
> ntfs-3g version ntfs-3g 2021.8.22 integrated FUSE 28 - Third
> Generation NTFS Driver
>
> [Problem]
> I bumped on livelock and analyze it. and concluded that it is needed
> to be fixed.
> it happends when 3 operation concurrently progressing.
>
> 1) usb detach by user. and kernel detect it.
> 2) mount.ntfs umount request & device release operation
> 3) pool-udisksd umount operation.
>
> [Conclusion]
> 1. mounted target device file must be released after /dev/fuse
> release. it makes deadlocky scenario.

Shouldn't this be reported to ntfs-3g developers then?

Thanks,
Miklos

Re: Re: question about fuse livelock situation

[YoungJun.Park]

On Wed, Aug 31, 2022 at 11:58:50AM +0200, Miklos Szeredi wrote:
> On Tue, 30 Aug 2022 at 03:58, 박영준 <her0gyugyu@gmail.com> wrote:
> >
> > I found fuse livelock situation and report it for possibility of problem.
> >
> > [Environment]
> > 22.04 5.15.0-43-generic ubuntu kernel.
> > ntfs-3g version ntfs-3g 2021.8.22 integrated FUSE 28 - Third
> > Generation NTFS Driver
> >
> > [Problem]
> > I bumped on livelock and analyze it. and concluded that it is needed
> > to be fixed.
> > it happends when 3 operation concurrently progressing.
> >
> > 1) usb detach by user. and kernel detect it.
> > 2) mount.ntfs umount request & device release operation
> > 3) pool-udisksd umount operation.
> >
> > [Conclusion]
> > 1. mounted target device file must be released after /dev/fuse
> > release. it makes deadlocky scenario.
> 
> Shouldn't this be reported to ntfs-3g developers then?
> 
> Thanks,
> Miklos

I reported it ntfs-3g and ubuntu bug report channel. 
ntfs-3g does not respond and ubuntu bug report channel response it like below.
(If you want a detail scenario flow picture, calltack etc check the link 
https://github.com/tuxera/ntfs-3g/issues/56)

> Hi

> Thanks for reporting this issue - in general it is better to report bugs
> via launchpad than email (e.g. by running the following command (without
> the quotation marks) in a terminal: "ubuntu-bug ntfs-3g" or by
> https://bugs.launchpad.net/ubuntu/+source/ntfs-3g/+filebug)

> I notice you also appear to have reported this to the upstream nfts-3g
> project at https://github.com/tuxera/ntfs-3g/issues/56 but have had no
> response.

> However, my initial thoughts when looking at this is that it appears you
> can trigger a livelock within the kernel from an unprivileged user in
> userspace - as such I wonder if this is a bug in the FUSE subsystem
> within the Linux kernel and hence whether it should be reported to the
> upstream kernel developers as well? As per
> https://www.kernel.org/doc/html/v4.15/admin-guide/reporting-bugs.html it
> would appear that this should be reported to the following email
> addresses (assuming this is a real kernel bug rather than a bug within
> the ntfs-3g userspace project):

> $ ./scripts/get_maintainer.pl fs/fuse/fuse_i.h
> Miklos Szeredi <miklos@szeredi.hu> (maintainer:FUSE: FILESYSTEM IN USERSPACE)
> linux-fsdevel@vger.kernel.org (open list:FUSE: FILESYSTEM IN USERSPACE)
> linux-kernel@vger.kernel.org (open list)

> Thanks,
> Alex

Could you explan why it shoulde be fixed in userspace?
then I try to fix this issue and to report it one more based on your comment.

Thanks,
YoungJun park. 

Re: Re: question about fuse livelock situation

[Miklos Szeredi]

On Thu, 19 Jan 2023 at 08:05, YoungJun.Park <her0gyugyu@gmail.com> wrote:
>
> On Wed, Aug 31, 2022 at 11:58:50AM +0200, Miklos Szeredi wrote:
> > On Tue, 30 Aug 2022 at 03:58, 박영준 <her0gyugyu@gmail.com> wrote:
> > >
> > > I found fuse livelock situation and report it for possibility of problem.
> > >
> > > [Environment]
> > > 22.04 5.15.0-43-generic ubuntu kernel.
> > > ntfs-3g version ntfs-3g 2021.8.22 integrated FUSE 28 - Third
> > > Generation NTFS Driver
> > >
> > > [Problem]
> > > I bumped on livelock and analyze it. and concluded that it is needed
> > > to be fixed.
> > > it happends when 3 operation concurrently progressing.
> > >
> > > 1) usb detach by user. and kernel detect it.
> > > 2) mount.ntfs umount request & device release operation
> > > 3) pool-udisksd umount operation.
> > >
> > > [Conclusion]
> > > 1. mounted target device file must be released after /dev/fuse
> > > release. it makes deadlocky scenario.
> >
> > Shouldn't this be reported to ntfs-3g developers then?
> >
> > Thanks,
> > Miklos
>
> I reported it ntfs-3g and ubuntu bug report channel.
> ntfs-3g does not respond and ubuntu bug report channel response it like below.
> (If you want a detail scenario flow picture, calltack etc check the link
> https://github.com/tuxera/ntfs-3g/issues/56)
>
> > Hi
>
> > Thanks for reporting this issue - in general it is better to report bugs
> > via launchpad than email (e.g. by running the following command (without
> > the quotation marks) in a terminal: "ubuntu-bug ntfs-3g" or by
> > https://bugs.launchpad.net/ubuntu/+source/ntfs-3g/+filebug)
>
> > I notice you also appear to have reported this to the upstream nfts-3g
> > project at https://github.com/tuxera/ntfs-3g/issues/56 but have had no
> > response.
>
> > However, my initial thoughts when looking at this is that it appears you
> > can trigger a livelock within the kernel from an unprivileged user in
> > userspace - as such I wonder if this is a bug in the FUSE subsystem
> > within the Linux kernel and hence whether it should be reported to the
> > upstream kernel developers as well? As per
> > https://www.kernel.org/doc/html/v4.15/admin-guide/reporting-bugs.html it
> > would appear that this should be reported to the following email
> > addresses (assuming this is a real kernel bug rather than a bug within
> > the ntfs-3g userspace project):
>
> > $ ./scripts/get_maintainer.pl fs/fuse/fuse_i.h
> > Miklos Szeredi <miklos@szeredi.hu> (maintainer:FUSE: FILESYSTEM IN USERSPACE)
> > linux-fsdevel@vger.kernel.org (open list:FUSE: FILESYSTEM IN USERSPACE)
> > linux-kernel@vger.kernel.org (open list)
>
> > Thanks,
> > Alex
>
> Could you explan why it shoulde be fixed in userspace?
> then I try to fix this issue and to report it one more based on your comment.

If the block device close is blocked, then ntfs-3g should reply to the
DESTROY request before it tries to close the block device.

Thanks,
Miklos