Re: Understanding "Used" in df

[Martin Steigerwald <martin@lichtvoll.de>]

Zygo Blaxell - 23.07.20, 06:51:06 CEST:
> On Wed, Jul 22, 2020 at 05:10:19PM +0200, Martin Steigerwald wrote:
> > I have:
> > 
> > % LANG=en df -hT /home
> > Filesystem            Type   Size  Used Avail Use% Mounted on
> > /dev/mapper/sata-home btrfs  300G  175G  123G  59% /home
> > 
> > And:
> > 
> > merkaba:~> btrfs fi sh /home
> > Label: 'home'  uuid: […]
> > 
> >         Total devices 2 FS bytes used 173.91GiB
> >         devid    1 size 300.00GiB used 223.03GiB path
> >         /dev/mapper/sata-home
> >         devid    2 size 300.00GiB used 223.03GiB path
> >         /dev/mapper/msata-home
> > 
> > merkaba:~> btrfs fi df /home
> > Data, RAID1: total=218.00GiB, used=171.98GiB
> > System, RAID1: total=32.00MiB, used=64.00KiB
> > Metadata, RAID1: total=5.00GiB, used=1.94GiB
> > GlobalReserve, single: total=490.48MiB, used=0.00B
> > 
> > As well as:
> > 
> > merkaba:~> btrfs fi usage -T /home
> > 
> > Overall:
> >     Device size:                 600.00GiB
> >     Device allocated:            446.06GiB
> >     Device unallocated:          153.94GiB
> >     Device missing:                  0.00B
> >     Used:                        347.82GiB
> >     Free (estimated):            123.00GiB      (min: 123.00GiB)
> >     Data ratio:                       2.00
> >     Metadata ratio:                   2.00
> >     Global reserve:              490.45MiB      (used: 0.00B)
> >     Multiple profiles:                  no
> >     
> >                           Data      Metadata System
> > 
> > Id Path                   RAID1     RAID1    RAID1    Unallocated
> > -- ---------------------- --------- -------- -------- -----------
> > 
> >  1 /dev/mapper/sata-home  218.00GiB  5.00GiB 32.00MiB    76.97GiB
> >  2 /dev/mapper/msata-home 218.00GiB  5.00GiB 32.00MiB    76.97GiB
> > 
> > -- ---------------------- --------- -------- -------- -----------
> > 
> >    Total                  218.00GiB  5.00GiB 32.00MiB   153.94GiB
> >    Used                   171.97GiB  1.94GiB 64.00KiB
> > 
> > I think I understand all of it, including just 123G instead of
> > 300 - 175 = 125 GiB "Avail" in df -hT.
> > 
> > But why 175 GiB "Used" in 'df -hT' when just 173.91GiB (see 'btrfs
> > fi sh') is allocated *within* the block group / chunks?
> 
> statvfs (the 'df' syscall) does not report a "used" number, only total
> and available btrfs data blocks (no metadata blocks are counted).
> 'df' computes "used" by subtracting f_blocks - f_bavail.
> 
> 	122.99875 = 300 - 171.97 - 5 - .03125
> 
> 	df_free = total - data_used - metadata_allocated - system_allocated

I get that one. That is for what is still free.

But I do not understand "Used" in df as.

1) It it would be doing 300 GiB - what is still available, it would do 300-122.99 = 177.01

2) If it would add together all allocated within a chunk… 

171.98 GiB used in data + 64 KiB used in system + 1,94 GiB used in metadata ~= 174 GiB

3) It may consider all allocated system and metadata chunks as lost for writing
data:

171.98 used in date + 32 MiB allocated in system + 5 GiB allocated in metadata ~= 176.98 GiB

4) It may consider 2 of those 5 GiB chunks for metadata as reclaimable and
then it would go like this:

171.98 used in date + 32 MiB allocated in system + 3 GiB metadata ~= 116.98 GiB = 174.98 GiB

That would be about right, but also as unpredictable as it can get.

> Inline files count as metadata instead of data, so even when you are
> out of data blocks (zero blocks free in df), you can sometimes still
> write small files.  Sometimes, when you write one small file, 1GB of
> available space disappears as a new metadata block group is allocated.
> 
> 'df' doesn't take metadata or data sharing into account at all, or
> the space required to store csums, or bursty metadata usage workloads.
> 'df' can't predict these events, so its accuracy is limited to no
> better than about 0.5% of the size of the filesystem or +/- 1GB,
> whichever is larger.

So just assume that df output can be +/- 1 GiB off?

I am just wondering cause I aimed to explaining this to participants of
my Linux courses… and for now I have the honest answer that I have
no clue why df displays "175 GiB" as used.

> > Does this have something to do with that global reserve thing?
> 
> 'df' currently tells you nothing about metadata (except in kernels
> before 5.6, when you run too low on metadata space, f_bavail is
> abruptly set to zero).  That's about the only impact global reserve
> has on 'df'.

But it won't claim used or even just allocated metadata space as available
for writing data?

> Global reserve is metadata allocated-but-unused space, and all
> metadata is not visible to df.  The reserve ensures that critical
> btrfs metadata operations can complete without running out of space,
> by forcing non-critical long-running operations to commit
> transactions when no metadata space is available outside the reserved
> pool.  It mostly works, though there are still a few bugs left that
> lead to EROFS when metadata runs low.

Hmmm, thanks.

But as far as I understood also from the other post, Global Reserve is
reserved but not reported as used in df?

I am not sure whether I am getting it though.

Best,
-- 
Martin