Re: btrfs blocks root from writing even when there is plenty of statfs->f_bfree

[Jeff Layton <jlayton@kernel.org>]

On Fri, 2023-08-25 at 10:12 +0800, Qu Wenruo wrote:
> 
> On 2023/8/25 09:23, Qu Wenruo wrote:
> > 
> > 
> > On 2023/8/25 04:33, Jeff Layton wrote:
> > > I've been doing some testing with btrfs exported via nfsd and hit this
> > > bug. root is unable to write to the filesystem, even though statfs
> > > reports that there should be 400k+ blocks available.
> > > 
> > > The reproducer is pretty simple. First I fill up the filesystem as an
> > > unprivileged user (let this run until I hit ENOSPC):
> > > 
> > > [vagrant@kdevops-nfs-default btrfs]$ dd if=/dev/urandom
> > > of=/media/btrfs/bigfile bs=1M
> > > 
> > > ...and then try to write to a file on the fs as root:
> > > 
> > > [root@kdevops-nfs-default btrfs]# strace -f -e statfs df .
> > > statfs(".", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
> > > f_blocks=26214400, f_bfree=452204, f_bavail=0, f_files=0, f_ffree=0,
> > > f_fsid={val=[0xa9d43863, 0xb08b34cc]}, f_namelen=255, f_frsize=4096,
> > > f_flags=ST_VALID|ST_RELATIME}) = 0
> > 
> > Note that, unlike XFS/EXT4, btrfs goes strict limits between data and
> > metadata.
> > 
> > Metadata space can only be used to store tree blocks, never data.
> > And both metadata and data space are allocated dynamically, thus we have
> > have problems when the workload is unbalanced.
> > (This is a little like the inodes limits on XFS/EXT4, but more dynamic)
> > 
> > In your particular case, the data space is all used up, but metadata
> > space still has some free space.
> > 
> > And f_btree returned by btrfs is including the metadata space, as
> > although that's inaccurate, it's the best what we can do to report.
> > 
> > 
> > > Filesystem     1K-blocks      Used Available Use% Mounted on
> > > /dev/nvme2n1   104857600 103048784         0 100% /media/btrfs
> > > +++ exited with 0 +++
> > > [root@kdevops-nfs-default btrfs]# xfs_io -f -c "pwrite -b 4096 4096
> > > 4096" ./file1
> > > pwrite: No space left on device
> > > 
> > > This works on ext4 and xfs. The kernel is Linus' master branch as of a
> > > couple of days ago:
> > > 
> > >      6.5.0-rc7-g081b0d4bef5d
> > > 
> > > Let me know if you need more info.
> > 
> > That's why for btrfs we recommend to go "btrfs fi usage" to get a more
> > comprehensive understand on the space usage.
> > 
> > In your case, Data usage should be 100%, without any spare ones.
> 
> Another thing is, btrfs doesn't have any extra data space reserved for
> root user, instead btrfs has extra space reserved for metadata, as
> metadata COW will cost extra space, even doing things like unlinking a file.
> 
> Thus this would be a behavior difference here.
> 


Got it, thanks for the explanation.

FWIW, I noticed this because generic/186 and generic/187 were failing on
NFS exported btrfs, but was passing with XFS, they get _notrun:

    generic/186       [not run] Can't fragment free space on btrfs.
    generic/187       [not run] Can't fragment free space on btrfs.

I'll probably just skip this test on NFS since it only reliably passes
on certain exported filesystems.

Thanks again!
-- 
Jeff Layton <jlayton@kernel.org>