btrfs blocks root from writing even when there is plenty of statfs->f_bfree

btrfs blocks root from writing even when there is plenty of statfs->f_bfree

[Jeff Layton]

I've been doing some testing with btrfs exported via nfsd and hit this
bug. root is unable to write to the filesystem, even though statfs
reports that there should be 400k+ blocks available.

The reproducer is pretty simple. First I fill up the filesystem as an
unprivileged user (let this run until I hit ENOSPC):

[vagrant@kdevops-nfs-default btrfs]$ dd if=/dev/urandom of=/media/btrfs/bigfile bs=1M

...and then try to write to a file on the fs as root:

[root@kdevops-nfs-default btrfs]# strace -f -e statfs df .
statfs(".", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096, f_blocks=26214400, f_bfree=452204, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0xa9d43863, 0xb08b34cc]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/nvme2n1   104857600 103048784         0 100% /media/btrfs
+++ exited with 0 +++
[root@kdevops-nfs-default btrfs]# xfs_io -f -c "pwrite -b 4096 4096 4096" ./file1
pwrite: No space left on device

This works on ext4 and xfs. The kernel is Linus' master branch as of a
couple of days ago:

    6.5.0-rc7-g081b0d4bef5d

Let me know if you need more info.
-- 
Jeff Layton <jlayton@kernel.org>

Re: btrfs blocks root from writing even when there is plenty of statfs->f_bfree

[Qu Wenruo]

On 2023/8/25 04:33, Jeff Layton wrote:
> I've been doing some testing with btrfs exported via nfsd and hit this
> bug. root is unable to write to the filesystem, even though statfs
> reports that there should be 400k+ blocks available.
>
> The reproducer is pretty simple. First I fill up the filesystem as an
> unprivileged user (let this run until I hit ENOSPC):
>
> [vagrant@kdevops-nfs-default btrfs]$ dd if=/dev/urandom of=/media/btrfs/bigfile bs=1M
>
> ...and then try to write to a file on the fs as root:
>
> [root@kdevops-nfs-default btrfs]# strace -f -e statfs df .
> statfs(".", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096, f_blocks=26214400, f_bfree=452204, f_bavail=0, f_files=0, f_ffree=0, f_fsid={val=[0xa9d43863, 0xb08b34cc]}, f_namelen=255, f_frsize=4096, f_flags=ST_VALID|ST_RELATIME}) = 0

Note that, unlike XFS/EXT4, btrfs goes strict limits between data and
metadata.

Metadata space can only be used to store tree blocks, never data.
And both metadata and data space are allocated dynamically, thus we have
have problems when the workload is unbalanced.
(This is a little like the inodes limits on XFS/EXT4, but more dynamic)

In your particular case, the data space is all used up, but metadata
space still has some free space.

And f_btree returned by btrfs is including the metadata space, as
although that's inaccurate, it's the best what we can do to report.


> Filesystem     1K-blocks      Used Available Use% Mounted on
> /dev/nvme2n1   104857600 103048784         0 100% /media/btrfs
> +++ exited with 0 +++
> [root@kdevops-nfs-default btrfs]# xfs_io -f -c "pwrite -b 4096 4096 4096" ./file1
> pwrite: No space left on device
>
> This works on ext4 and xfs. The kernel is Linus' master branch as of a
> couple of days ago:
>
>      6.5.0-rc7-g081b0d4bef5d
>
> Let me know if you need more info.

That's why for btrfs we recommend to go "btrfs fi usage" to get a more
comprehensive understand on the space usage.

In your case, Data usage should be 100%, without any spare ones.

Thanks,
Qu

Re: btrfs blocks root from writing even when there is plenty of statfs->f_bfree

[Qu Wenruo]

On 2023/8/25 09:23, Qu Wenruo wrote:
>
>
> On 2023/8/25 04:33, Jeff Layton wrote:
>> I've been doing some testing with btrfs exported via nfsd and hit this
>> bug. root is unable to write to the filesystem, even though statfs
>> reports that there should be 400k+ blocks available.
>>
>> The reproducer is pretty simple. First I fill up the filesystem as an
>> unprivileged user (let this run until I hit ENOSPC):
>>
>> [vagrant@kdevops-nfs-default btrfs]$ dd if=/dev/urandom
>> of=/media/btrfs/bigfile bs=1M
>>
>> ...and then try to write to a file on the fs as root:
>>
>> [root@kdevops-nfs-default btrfs]# strace -f -e statfs df .
>> statfs(".", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
>> f_blocks=26214400, f_bfree=452204, f_bavail=0, f_files=0, f_ffree=0,
>> f_fsid={val=[0xa9d43863, 0xb08b34cc]}, f_namelen=255, f_frsize=4096,
>> f_flags=ST_VALID|ST_RELATIME}) = 0
>
> Note that, unlike XFS/EXT4, btrfs goes strict limits between data and
> metadata.
>
> Metadata space can only be used to store tree blocks, never data.
> And both metadata and data space are allocated dynamically, thus we have
> have problems when the workload is unbalanced.
> (This is a little like the inodes limits on XFS/EXT4, but more dynamic)
>
> In your particular case, the data space is all used up, but metadata
> space still has some free space.
>
> And f_btree returned by btrfs is including the metadata space, as
> although that's inaccurate, it's the best what we can do to report.
>
>
>> Filesystem     1K-blocks      Used Available Use% Mounted on
>> /dev/nvme2n1   104857600 103048784         0 100% /media/btrfs
>> +++ exited with 0 +++
>> [root@kdevops-nfs-default btrfs]# xfs_io -f -c "pwrite -b 4096 4096
>> 4096" ./file1
>> pwrite: No space left on device
>>
>> This works on ext4 and xfs. The kernel is Linus' master branch as of a
>> couple of days ago:
>>
>>      6.5.0-rc7-g081b0d4bef5d
>>
>> Let me know if you need more info.
>
> That's why for btrfs we recommend to go "btrfs fi usage" to get a more
> comprehensive understand on the space usage.
>
> In your case, Data usage should be 100%, without any spare ones.

Another thing is, btrfs doesn't have any extra data space reserved for
root user, instead btrfs has extra space reserved for metadata, as
metadata COW will cost extra space, even doing things like unlinking a file.

Thus this would be a behavior difference here.

Thanks,
Qu
>
> Thanks,
> Qu

Re: btrfs blocks root from writing even when there is plenty of statfs->f_bfree

[Jeff Layton]

On Fri, 2023-08-25 at 10:12 +0800, Qu Wenruo wrote:
> 
> On 2023/8/25 09:23, Qu Wenruo wrote:
> > 
> > 
> > On 2023/8/25 04:33, Jeff Layton wrote:
> > > I've been doing some testing with btrfs exported via nfsd and hit this
> > > bug. root is unable to write to the filesystem, even though statfs
> > > reports that there should be 400k+ blocks available.
> > > 
> > > The reproducer is pretty simple. First I fill up the filesystem as an
> > > unprivileged user (let this run until I hit ENOSPC):
> > > 
> > > [vagrant@kdevops-nfs-default btrfs]$ dd if=/dev/urandom
> > > of=/media/btrfs/bigfile bs=1M
> > > 
> > > ...and then try to write to a file on the fs as root:
> > > 
> > > [root@kdevops-nfs-default btrfs]# strace -f -e statfs df .
> > > statfs(".", {f_type=BTRFS_SUPER_MAGIC, f_bsize=4096,
> > > f_blocks=26214400, f_bfree=452204, f_bavail=0, f_files=0, f_ffree=0,
> > > f_fsid={val=[0xa9d43863, 0xb08b34cc]}, f_namelen=255, f_frsize=4096,
> > > f_flags=ST_VALID|ST_RELATIME}) = 0
> > 
> > Note that, unlike XFS/EXT4, btrfs goes strict limits between data and
> > metadata.
> > 
> > Metadata space can only be used to store tree blocks, never data.
> > And both metadata and data space are allocated dynamically, thus we have
> > have problems when the workload is unbalanced.
> > (This is a little like the inodes limits on XFS/EXT4, but more dynamic)
> > 
> > In your particular case, the data space is all used up, but metadata
> > space still has some free space.
> > 
> > And f_btree returned by btrfs is including the metadata space, as
> > although that's inaccurate, it's the best what we can do to report.
> > 
> > 
> > > Filesystem     1K-blocks      Used Available Use% Mounted on
> > > /dev/nvme2n1   104857600 103048784         0 100% /media/btrfs
> > > +++ exited with 0 +++
> > > [root@kdevops-nfs-default btrfs]# xfs_io -f -c "pwrite -b 4096 4096
> > > 4096" ./file1
> > > pwrite: No space left on device
> > > 
> > > This works on ext4 and xfs. The kernel is Linus' master branch as of a
> > > couple of days ago:
> > > 
> > >      6.5.0-rc7-g081b0d4bef5d
> > > 
> > > Let me know if you need more info.
> > 
> > That's why for btrfs we recommend to go "btrfs fi usage" to get a more
> > comprehensive understand on the space usage.
> > 
> > In your case, Data usage should be 100%, without any spare ones.
> 
> Another thing is, btrfs doesn't have any extra data space reserved for
> root user, instead btrfs has extra space reserved for metadata, as
> metadata COW will cost extra space, even doing things like unlinking a file.
> 
> Thus this would be a behavior difference here.
> 


Got it, thanks for the explanation.

FWIW, I noticed this because generic/186 and generic/187 were failing on
NFS exported btrfs, but was passing with XFS, they get _notrun:

    generic/186       [not run] Can't fragment free space on btrfs.
    generic/187       [not run] Can't fragment free space on btrfs.

I'll probably just skip this test on NFS since it only reliably passes
on certain exported filesystems.

Thanks again!
-- 
Jeff Layton <jlayton@kernel.org>