Re: Report a fuse deadlock scenario issue

Re: Report a fuse deadlock scenario issue

[Miklos Szeredi]

On Fri, 11 Feb 2022 at 14:55, 陈立新 <clx428@163.com> wrote:
>
> Hi Miklos:
> I meet a dealock scenario on fuse. here are the 4 backtraces:

Which kernel is this?

Commit 4f06dd92b5d0 ("fuse: fix write deadlock") was added in 5.13 and
backported to some (but not all) affected stable trees.

Thanks,
Miklos

Re: Report a fuse deadlock scenario issue

[Miklos Szeredi]

On Fri, 11 Feb 2022 at 14:55, 陈立新 <clx428@163.com> wrote:
>
> Hi Miklos:
> I meet a dealock scenario on fuse. here are the 4 backtraces:
> PID: 301852  TASK: ffff80db78226c80  CPU: 93  COMMAND: "Thread-854"
> #0 [ffff000c1d88b9e0] __switch_to at ffff000080088738
> #1 [ffff000c1d88ba00] __schedule at ffff000080a06f48
> #2 [ffff000c1d88ba90] schedule at ffff000080a07620
> #3 [ffff000c1d88baa0] fuse_wait_on_page_writeback at ffff000001047418 [fuse]
> #4 [ffff000c1d88bb00] fuse_page_mkwrite at ffff000001047538 [fuse]
> #5 [ffff000c1d88bb40] do_page_mkwrite at ffff0000802cb77c
> #6 [ffff000c1d88bb90] do_fault at ffff0000802d1840
> #7 [ffff000c1d88bbd0] __handle_mm_fault at ffff0000802d3574
> #8 [ffff000c1d88bc90] handle_mm_fault at ffff0000802d37c0
> #9 [ffff000c1d88bcc0] do_page_fault at ffff000080a0ef94
> #10 [ffff000c1d88bdc0] do_translation_fault at ffff000080a0f32c
> #11 [ffff000c1d88bdf0] do_mem_abort at ffff0000800812cc
> #12 [ffff000c1d88bff0] el0_da at ffff000080083b20
>
> PID: 400127  TASK: ffff80d1a1c51f00  CPU: 91  COMMAND: "Thread-677"
> #0 [ffff000beb5e3a00] __switch_to at ffff000080088738
> #1 [ffff000beb5e3a20] __schedule at ffff000080a06f48
> #2 [ffff000beb5e3ab0] schedule at ffff000080a07620
> #3 [ffff000beb5e3ac0] fuse_wait_on_page_writeback at ffff000001047418 [fuse]
> #4 [ffff000beb5e3b20] fuse_page_mkwrite at ffff000001047538 [fuse]
> #5 [ffff000beb5e3b60] do_page_mkwrite at ffff0000802cb77c
> #6 [ffff000beb5e3bb0] do_wp_page at ffff0000802d0264
> #7 [ffff000beb5e3c00] __handle_mm_fault at ffff0000802d363c
> #8 [ffff000beb5e3cc0] handle_mm_fault at ffff0000802d37c0
> #9 [ffff000beb5e3cf0] do_page_fault at ffff000080a0ef94
> #10 [ffff000beb5e3df0] do_mem_abort at ffff0000800812cc
> #11 [ffff000beb5e3ff0] el0_da at ffff000080083b20
>
> PID: 178830  TASK: ffff80dc1704cd80  CPU: 64  COMMAND: "kworker/u259:11"
> #0 [ffff0000aab6b6f0] __switch_to at ffff000080088738
> #1 [ffff0000aab6b710] __schedule at ffff000080a06f48
> #2 [ffff0000aab6b7a0] schedule at ffff000080a07620
> #3 [ffff0000aab6b7b0] io_schedule at ffff00008012dbc4
> #4 [ffff0000aab6b7d0] __lock_page at ffff0000802854e0
> #5 [ffff0000aab6b870] write_cache_pages at ffff0000802987e8
> #6 [ffff0000aab6b990] fuse_writepages at ffff00000104ab6c [fuse]
> #7 [ffff0000aab6b9f0] do_writepages at ffff00008029b2e0
> #8 [ffff0000aab6ba70] __writeback_single_inode at ffff00008037f8b4
> #9 [ffff0000aab6bac0] writeback_sb_inodes at ffff000080380150
> #10 [ffff0000aab6bbd0] __writeback_inodes_wb at ffff0000803804c0
> #11 [ffff0000aab6bc20] wb_writeback at ffff000080380880
> #12 [ffff0000aab6bcd0] wb_workfn at ffff000080381470
> #13 [ffff0000aab6bdb0] process_one_work at ffff000080113428
> #14 [ffff0000aab6be00] worker_thread at ffff0000801136c0
> #15 [ffff0000aab6be70] kthread at ffff00008011ab60
>
> PID: 47324  TASK: ffff80db5a038000  CPU: 88  COMMAND: "Thread-2064"
> #0 [ffff000c2114b820] __switch_to at ffff000080088738
> #1 [ffff000c2114b840] __schedule at ffff000080a06f48
> #2 [ffff000c2114b8d0] schedule at ffff000080a07620
> #3 [ffff000c2114b8e0] io_schedule at ffff00008012dbc4
> #4 [ffff000c2114b900] __lock_page at ffff0000802854e0
> #5 [ffff000c2114b9a0] write_cache_pages at ffff0000802987e8
> #6 [ffff000c2114bac0] fuse_writepages at ffff00000104ab6c [fuse]
> #7 [ffff000c2114bb20] do_writepages at ffff00008029b2e0
> #8 [ffff000c2114bba0] __filemap_fdatawrite_range at ffff0000802883f8
> #9 [ffff000c2114bc60] file_write_and_wait_range at ffff0000802886f0
> #10 [ffff000c2114bca0] fuse_fsync_common at ffff0000010491d8 [fuse]
> #11 [ffff000c2114bd90] fuse_fsync at ffff00000104938c [fuse]
> #12 [ffff000c2114bdc0] vfs_fsync_range at ffff000080385938
> #13 [ffff000c2114bdf0] __arm64_sys_msync at ffff0000802dcf8c
> #14 [ffff000c2114be60] el0_svc_common at ffff000080097cbc
> #15 [ffff000c2114bea0] el0_svc_handler at ffff000080097df0
> #16 [ffff000c2114bff0] el0_svc at ffff000080084144
>
> The 4 threads write the same file, and deadlocked:
>   Thread 301852 gets the page 5 lock, and waiting on page 5 writeback is completed;
>   Thread 400127 gets the page 0 lock, and waiting on page 0 writeback is completed;
>   Thread 47324 is waiting page 5 lock, and already set page 0 - 4 to writeback;
>   Thread 178830 is waiting page 0 lock, and already set page 5 - 6 to writeback;

This last is not possible, because write_cache_pages() will always
return when it reached the end of the range and only the next
invocation will wrap around to the zero index page.  See this at the
end of write_cache_pages():

    if (wbc->range_cyclic && !done)
        done_index = 0;

Otherwise index will be monotonic increasing throughout a single
write_cache_pages() call.

That doesn't mean that there's no deadlock, this is pretty complex,
but there must be some other explanation.

Thanks,
Miklos

Re: Re: Report a fuse deadlock scenario issue

[Miklos Szeredi]

On Mon, 21 Feb 2022 at 15:11, 陈立新 <clx428@163.com> wrote:

> >This last is not possible, because write_cache_pages() will always
> >return when it reached the end of the range and only the next
> >invocation will wrap around to the zero index page.  See this at the
> >end of write_cache_pages():
> >
> >    if (wbc->range_cyclic && !done)
> >        done_index = 0;
> I use the kernel version is 4.19.36, which it has no 64081362e8ff4587b4554087f3cfc73d3e0a4cd7 mm/page-writeback.c: fix range_cyclic writeback vs writepages deadlock patch.
> I think this patch can fix this deadlock.

Yes, that explains it.  Thanks for investigating.

Miklos

Re: Re: Report a fuse deadlock scenario issue

[Matthew Wilcox]

On Mon, Feb 21, 2022 at 10:16:49PM +0800, 陈立新 wrote:
> I use the kernel version is 4.19.36, which it has no 64081362e8ff4587b4554087f3cfc73d3e0a4cd7 mm/page-writeback.c: fix range_cyclic writeback vs writepages deadlock patch.
> I think this patch can fix this deadlock.

This patch was added to v4.19.87 which was released in December 2019.
You really should update to the latest v4.19 kernel; you're missing
hundreds, if not thousands of bug-fixes, many of them with CVEs
attached.