Re: [PATCH] fuse: fix deadlock between O_TRUNC open() and non-O_TRUNC open()

[Miklos Szeredi <miklos@szeredi.hu>]

On Thu, 16 Dec 2021 at 15:46, Jiachen Zhang
<zhangjiachen.jaycee@bytedance.com> wrote:
>
> fuse_finish_open() will be called with FUSE_NOWRITE set in case of atomic
> O_TRUNC open(), so commit 76224355db75 ("fuse: truncate pagecache on
> atomic_o_trunc") replaced invalidate_inode_pages2() by truncate_pagecache()
> in such a case to avoid the A-A deadlock. However, we found another A-B-B-A
> deadlock related to the case above, which will cause the xfstests
> generic/464 testcase hung in our virtio-fs test environment.
>
> Consider two processes concurrently open one same file, one with O_TRUNC
> and another without O_TRUNC. The deadlock case is described below, if
> open(O_TRUNC) is already set_nowrite(acquired A), and is trying to lock
> a page (acquiring B), open() could have held the page lock (acquired B),
> and waiting on the page writeback (acquiring A). This would lead to
> deadlocks.
>
> This commit tries to fix it by locking inode in fuse_open_common() even
> if O_TRUNC is not set. This introduces a lock C to protect the area with
> A-B-B-A the deadlock rick.

Okay.

One problem is that this seems to affect a number of other calls to
invalidate_inode_pages2(), specifically those without lock_inode()
protection:

- dmap_writeback_invalidate()
- fuse_file_mmap()
- fuse_change_attributes()
- fuse_reverse_inval_inode()

fuse_change_attributes() is especially problematic because it can be
called with or without the inode lock.

The other issue is that locking the inode may impact performance and
doing it unconditionally for all opens seems excessive.

If there are no better ideas, then the brute force fix is to introduce
another lock (since the inode lock cannot always be used) to protect
fuse_set_nowrite()/fuse_clear_nowrite() racing with
invalidate_inode_pages2().

Thoughts?

Thanks,
Miklos