Re: [fuse-devel] Proxmox + NFS w/ exported FUSE = EIO

[Antonio SJ Musumeci <trapexit@spawn.link>]

On 2/6/24 00:53, Amir Goldstein wrote:
> On Tue, Feb 6, 2024 at 4:52 AM Antonio SJ Musumeci <trapexit@spawn.link> wrote:
>> Hi,
>>
>> Anyone have users exporting a FUSE filesystem over NFS? Particularly
>> from Proxmox (recent release, kernel 6.5.11)? I've gotten a number of
>> reports recently from individuals who have such a setup and after some
>> time (not easily reproducible, seems usually after software like Plex or
>> Jellyfin do a scan of media or a backup process) starts returning EIO
>> errors. Not just from NFS but also when trying to access the FUSE mount
>> as well. One person noted that they had moved from Ubuntu 18.04 (kernel
>> 4.15.0) to Proxmox and on Ubuntu had no problems with otherwise the same
>> settings.
>>
>> I've not yet been able to reproduced this issue myself but wanted to see
>> if anyone else has run into this. As far as I can tell from what users
>> have reported the FUSE server is still running but isn't receiving most
>> requests. I do see evidence of statfs calls coming through but nothing
>> else. Though the straces I've received typically are after the issues start.
>>
>> In an effort to rule out the FUSE server... is there anything the server
>> could do to cause the kernel to return EIO and not forward anything but
>> statfs? Doesn't seem to matter if direct_io is enabled or attr/entry
>> caching is used.
>>
> This could be the outcome of commit 15db16837a35 ("fuse: fix illegal
> access to inode with reused nodeid") in kernel v5.14.
>
> It is not an unintended regression - this behavior replaces what would
> have been a potentially severe security violation with an EIO error.
>
> As the commit says:
> "...With current code, this situation will not be detected and an old fuse
>      dentry that used to point to an older generation real inode, can be used to
>      access a completely new inode, which should be accessed only via the new
>      dentry."
>
> I have made this fix after seeing users get the content of another
> file from the one that they opened in NFS!
>
> libfuse commit 10ecd4f ("test/test_syscalls.c: check unlinked testfiles
> at the end of the test") reproduces this problem in a test.
> This test does not involve NFS export, but NFS export has higher
> likelihood of exposing this issue.
>
> I wonder if the FUSE filesystems that report the errors have
> FUSE_EXPORT_SUPPORT capability?
> Not that this capability guarantees anything wrt to this issue.
>
> IMO, the root of all evil wrt NFS+FUSE is that LOOKUP is by ino
> without generation with FUSE_EXPORT_SUPPORT, but worse
> is that FUSE does not even require FUSE_EXPORT_SUPPORT
> capability to export to NFS, but this is legacy FUSE behavior and
> I am sure that many people export FUSE filesystems, as your
> report proves.
>
> There is now a proposal for opt-out of NFS export:
> https://lore.kernel.org/linux-fsdevel/20240126072120.71867-1-jefflexu@linux.alibaba.com/
> so there will be a way for a FUSE filesystem to prevent misuse.
>
> Some practical suggestions for users running existing FUSE filesystems:
>
> - Never export a FUSE filesystem with a fixed fsid
> - Everytime one wants to export a FUSE filesystem generate
>    a oneshot fsid/uuid to use in exportfs
> - Then restarting/re-exporting the FUSE filesystem will result in
>    ESTALE errors on NFS client, but not security violations and not EIO
>    errors
> - This does not give full guarantee, unlinked inodes could still result
>    in EIO errors, as the libfuse test demonstrates
> - The situation with NFSv4 is slightly better than with NFSv3, because
>     with NFSv3, an open file in the client does not keep the FUSE file
>     open and increases the chance of evicted FUSE inode for an open
>     NFS file
>
> Thanks,
> Amir.

Thank you Amir for such a detailed response. I'll look into this further 
but a few questions. To answer your question: yes, the server is setting 
EXPORT_SUPPORT.

1. The expected behavior, if the above situation occurred, is that the 
whole of the mount would return EIO? All requests going forward? What 
about FUSE_STATFS? From what I saw that was coming through.

2. Regarding the tests. I downloaded the latest libfuse, compiled, and 
ran test_syscalls against the FUSE server. I get no failures when 
running `./test_syscalls /mnt/fusemount :/mnt/ext4mount -u` or 
`./test_syscalls /mnt/fusemount -u` where ext4mount is the underlying 
filesystem and fusemount is the FUSE server's. No error is reported. A 
strace shows the fstat returning ESTALE at the end but the tests all 
pass. The mount continues to work after running the test. This is on 
kernel 6.5.0. Is that expected? It sounds from your description that I 
should be seeing EIOs somewhere.

3. Thank you for the "practical suggestions". I will compare them to 
what my users are doing... but are there specific guidelines somewhere 
for building a FUSE server to ensure NFS export can be supported? This 
topic has had limited details available over the years and I/users have 
had odd behaviors at times that were unclear of the cause. Like this 
situation or when NFS somehow triggered a request for '..' of the root 
nodeid (1). Some questions that come to mind: is the generation strictly 
necessary (practically) for things to work so long as nodeid is unique 
during a session (64bit nodeid space can last a long time)? Is there 
possibility of conflict if multiple fuse servers used the same 
nodeid//gen pairs at the same time? To what degree does the inode value 
matter? Should old node/gen pairs be kept around forever as noforget 
libfuse option suggests for NFS? Perhaps some of this is obvious but 
given changes to FUSE over time and the differences between kernel and 
userspace fs experiences it would be nice to have some of these more 
niche/complicated situations better flushed out in the official docs.

Thanks again.