Re: cannot use btrfs for nfs server

[Forza <forza@tnonline.net>]

On 2021-07-19 15:07, Forza wrote:
> 
> 
> On 2021-07-19 14:06, Forza wrote:
>>
>>
>> On 2021-07-13 09:37, Ulli Horlacher wrote:
>>> On Mon 2021-07-12 (23:56), g.btrfs@cobb.uk.net wrote:
>>>
>>>>> root@tsmsrvj:/etc# du -Hs /nfs/localhost/snapshots
>>>>> du: WARNING: Circular directory structure.
>>>>> This almost certainly means that you have a corrupted file system.
>>>>> NOTIFY YOUR SYSTEM MANAGER.
>>>>> The following directory is part of the cycle:
>>>>>    /nfs/localhost/snapshots/spool
>>>>
>>>> Sure. But it makes the useful operations work. du, find, ls -R, etc all
>>>> work properly on /nfs/localhost/fex.
>>>
>>> Properly on /nfs/localhost/fex : yes
>>> Properly on /nfs/localhost/snapshots : NO
>>>
>>> And the error messages are annoying!
>>>
>>> root@tsmsrvj:/etc# exportfs -v
>>> /data/fex 
>>> localhost.localdomain(rw,async,wdelay,crossmnt,no_root_squash,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash) 
>>>
>>> /data/snapshots 
>>> localhost.localdomain(rw,async,wdelay,crossmnt,no_root_squash,no_subtree_check,sec=sys,rw,secure,no_root_squash,no_all_squash) 
>>>
>>>
>>> root@tsmsrvj:/etc# mount -o vers=3 localhost:/data/fex 
>>> /nfs/localhost/fex
>>> root@tsmsrvj:/etc# mount -o vers=3 localhost:/data/snapshots 
>>> /nfs/localhost/snapshots
>>> root@tsmsrvj:/etc# mount | grep localhost
>>> localhost:/data/fex on /nfs/localhost/fex type nfs 
>>> (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=37961,mountproto=udp,local_lock=none,addr=127.0.0.1) 
>>>
>>> localhost:/data/snapshots on /nfs/localhost/snapshots type nfs 
>>> (rw,relatime,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=127.0.0.1,mountvers=3,mountport=37961,mountproto=udp,local_lock=none,addr=127.0.0.1) 
>>>
>>>
>>
>> What kind of NFS server is this? Isn't UDP mounts legacy and not 
>> normally used by default?
>>
>> Can you switch to an nfs4 server and try again? I also still think you 
>> should use fsid export option.
>>
>>
>>
> I'm replying to myself here because I booted up a VM with Fedora 34 and 
> tested a similar setup as Mr Horlacher's and can reproduce the errors.
> 
> Setup:
> 1) create a subvolume /mnt/rootvol/nfs
> 2) create some snapshots:
> btrfs sub snap /mnt/rootvol/nfs /mnt/rootvol/nfs/.snapshots/nfs-1
> btrfs sub snap /mnt/rootvol/nfs /mnt/rootvol/nfs/.snapshots/nfs-2
> 
> 3) export as:
> /mnt/rootvol/nfs/ *(fsid=1234,no_root_squash)
> 
> 4) mount -o vers=4 localhost:/mnt/rootvol/nfs /media/nfs-mnt/
> 5) "du -sh /media/nfs-mnt" fails with
> "WARNING: Circular directory structure."
> 
> 6) "ls -alR /mnt/nfs-mnt" fails with
> "not listing already-listed directory"
> 
> In addition I have tried with various export options such as crossmnt, 
> nohide and subtree_check. They do not improve the situation.
> 
> Also the behaviour is the same with nfs3 as with nfs4.
> 
> Full outputs are available at https://paste.ee/p/pkHLh

Perhaps the problem is that inode numbers are re-used inside snapshots 
and that nfsd doesn't understand how to handle this properly?

# ls -ila /media/nfs-mnt/
total 0
256 drwxr-xr-x. 1 root root 80 Jul 19 14:17 .
270 drwxr-xr-x. 1 root root 14 Jul 19 14:21 ..
259 -rw-r--r--. 1 root root  0 Jul 19 14:17 bar
261 -rw-r--r--. 1 root root  0 Jul 19 14:17 file1
262 -rw-r--r--. 1 root root  0 Jul 19 14:17 file2
263 -rw-r--r--. 1 root root  0 Jul 19 14:17 file3
258 -rw-r--r--. 1 root root  0 Jul 19 14:17 foo
257 drwxr-xr-x. 1 root root 30 Jul 19 15:02 .snapshots
260 -rw-r--r--. 1 root root  0 Jul 19 14:17 somefiles

# ls -ila /media/nfs-mnt/.snapshots/nfs-2/
total 0
256 drwxr-xr-x. 1 root root 80 Jul 19 14:17 .
257 drwxr-xr-x. 1 root root 30 Jul 19 15:02 ..
259 -rw-r--r--. 1 root root  0 Jul 19 14:17 bar
261 -rw-r--r--. 1 root root  0 Jul 19 14:17 file1
262 -rw-r--r--. 1 root root  0 Jul 19 14:17 file2
263 -rw-r--r--. 1 root root  0 Jul 19 14:17 file3
258 -rw-r--r--. 1 root root  0 Jul 19 14:17 foo
257 drwxr-xr-x. 1 root root 10 Jul 19 14:17 .snapshots
260 -rw-r--r--. 1 root root  0 Jul 19 14:17 somefiles


Using nfs4 exports and specifying each snapshot as its own fsid does not 
work either.

### /etc/exports
/mnt/rootvol/nfs/ *(fsid=root,no_root_squash,no_subtree_check)
/mnt/rootvol/nfs/.snapshots/nfs-1 
*(fsid=1000,no_root_squash,no_subtree_check)
/mnt/rootvol/nfs/.snapshots/nfs-2 
*(fsid=2000,no_root_squash,no_subtree_check)
/mnt/rootvol/nfs/.snapshots/nfs-3 
*(fsid=3000,no_root_squash,no_subtree_check)

# ls -laRi nfs-mnt/
nfs-mnt/:
total 0
256 drwxr-xr-x. 1 root root 80 Jul 19 14:17 .
270 drwxr-xr-x. 1 root root 14 Jul 19 14:21 ..
259 -rw-r--r--. 1 root root  0 Jul 19 14:17 bar
261 -rw-r--r--. 1 root root  0 Jul 19 14:17 file1
262 -rw-r--r--. 1 root root  0 Jul 19 14:17 file2
263 -rw-r--r--. 1 root root  0 Jul 19 14:17 file3
258 -rw-r--r--. 1 root root  0 Jul 19 14:17 foo
257 drwxr-xr-x. 1 root root 30 Jul 19 15:02 .snapshots
260 -rw-r--r--. 1 root root  0 Jul 19 14:17 somefiles

nfs-mnt/.snapshots:
total 0
257 drwxr-xr-x. 1 root root 30 Jul 19 15:02 .
256 drwxr-xr-x. 1 root root 80 Jul 19 14:17 ..
256 drwxr-xr-x. 1 root root 56 Jul 19 15:02 nfs-1
256 drwxr-xr-x. 1 root root 80 Jul 19 14:17 nfs-2
256 drwxr-xr-x. 1 root root 86 Jul 19 15:03 nfs-3
ls: nfs-mnt/.snapshots/nfs-1: not listing already-listed directory
ls: nfs-mnt/.snapshots/nfs-2: not listing already-listed directory
ls: nfs-mnt/.snapshots/nfs-3: not listing already-listed directory