Understanding "btrfs filesystem usage"

Understanding "btrfs filesystem usage"

[Ulli Horlacher]

I want to know how many free space is left and have problems in
interpreting the output of: 

btrfs filesystem usage
btrfs filesystem df
btrfs filesystem show


In detail:


root@diaspora:~# btrfs filesystem usage /local
Overall:
    Device size:                 883.51GiB
    Device allocated:            825.51GiB
    Device unallocated:           58.00GiB
    Device missing:                  0.00B
    Used:                        494.45GiB
    Free (estimated):            387.94GiB      (min: 387.94GiB)
    Data ratio:                       1.00
    Metadata ratio:                   1.00
    Global reserve:              320.00MiB      (used: 0.00B)

Data,single: Size:823.47GiB, Used:493.53GiB
   /dev/sda4     823.47GiB

Metadata,single: Size:2.01GiB, Used:938.67MiB
   /dev/sda4       2.01GiB

System,single: Size:32.00MiB, Used:128.00KiB
   /dev/sda4      32.00MiB

Unallocated:
   /dev/sda4      58.00GiB


root@diaspora:~# btrfs filesystem df /local
Data, single: total=823.47GiB, used=493.53GiB
System, single: total=32.00MiB, used=128.00KiB
Metadata, single: total=2.01GiB, used=938.67MiB
GlobalReserve, single: total=320.00MiB, used=0.00B


root@diaspora:~# btrfs filesystem show /local
Label: 'local'  uuid: 11faaa39-5805-4e92-a891-e8ceb4afa9f7
        Total devices 1 FS bytes used 494.45GiB
        devid    1 size 883.51GiB used 825.51GiB path /dev/sda4


What does the last line mean?
825.51GiB used = allocated by (file) data?
In this case I would have only 58 GB free disk space left?

"btrfs filesystem usage" says 494 GB used, but
"btrfs filesystem show" says 825 GB used?!

The man pages do not explain the different meanings of "used" and
"allocated"  

And at last:

root@diaspora:~# df -HT /local
Filesystem     Type   Size  Used Avail Use% Mounted on
/dev/sda4      btrfs  949G  532G  417G  57% /local



-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<20181029181141.GA31256@rus.uni-stuttgart.de>

Re: Understanding "btrfs filesystem usage"

[Remi Gauvin]

[-- Attachment #1: Type: text/plain, Size: 1513 bytes --]

On 2018-10-29 02:11 PM, Ulli Horlacher wrote:
> I want to know how many free space is left and have problems in
> interpreting the output of: 
> 
> btrfs filesystem usage
> btrfs filesystem df
> btrfs filesystem show
> 
>

In my not so humble opinion, the filesystem usage command has the
easiest to understand output.  It' lays out all the pertinent information.

You can clearly see 825GiB is allocated, with 494GiB used, therefore,
filesystem show is actually using the "Allocated" value as "Used".
Allocated can be thought of "Reserved For".  As the output of the Usage
command and df command clearly show, you have almost 400GiB space available.

Note that the btrfs commands are clearly and explicitly displaying
values in Binary units, (Mi, and Gi prefix, respectively).  If you want
df command to match, use -h instead of -H (see man df)

An observation:

The disparity between 498GiB used and 823Gib is pretty high.  This is
probably the result of using an SSD with an older kernel.  If your
kernel is not very recent, (sorry, I forget where this was fixed,
somewhere around 4.14 or 4.15), then consider mounting with the nossd
option.  You can improve this by running a balance.

Something like:
btrfs balance start -dusage=55

You do *not* want to end up with all your space allocated to Data, but
not actually used by data.  Bad things can happen if you run out of
Unallocated space for more metadata. (not catastrophic, but awkward and
unexpected downtime that can be a little tricky to sort out.)



[-- Attachment #2: remi.vcf --]
[-- Type: text/x-vcard, Size: 203 bytes --]

begin:vcard
fn:Remi Gauvin
n:Gauvin;Remi
org:Georgian Infotech
adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada
email;internet:remi@georgianit.com
tel;work:226-256-1545
version:2.1
end:vcard

Re: Understanding "btrfs filesystem usage"

[Hugo Mills]

[-- Attachment #1: Type: text/plain, Size: 2312 bytes --]

On Mon, Oct 29, 2018 at 05:57:10PM -0400, Remi Gauvin wrote:
> On 2018-10-29 02:11 PM, Ulli Horlacher wrote:
> > I want to know how many free space is left and have problems in
> > interpreting the output of: 
> > 
> > btrfs filesystem usage
> > btrfs filesystem df
> > btrfs filesystem show
> > 
> >
> 
> In my not so humble opinion, the filesystem usage command has the
> easiest to understand output.  It' lays out all the pertinent information.

   Opinions are divided. I find it almost impossible to read, and
always use btrfs fi df and btrfs fi show together.

   There's short tutorials of how to read the output in both cases in
the FAQ, which is where I start out by directing people in this
instance.

   Hugo.

> You can clearly see 825GiB is allocated, with 494GiB used, therefore,
> filesystem show is actually using the "Allocated" value as "Used".
> Allocated can be thought of "Reserved For".  As the output of the Usage
> command and df command clearly show, you have almost 400GiB space available.
> 
> Note that the btrfs commands are clearly and explicitly displaying
> values in Binary units, (Mi, and Gi prefix, respectively).  If you want
> df command to match, use -h instead of -H (see man df)
> 
> An observation:
> 
> The disparity between 498GiB used and 823Gib is pretty high.  This is
> probably the result of using an SSD with an older kernel.  If your
> kernel is not very recent, (sorry, I forget where this was fixed,
> somewhere around 4.14 or 4.15), then consider mounting with the nossd
> option.  You can improve this by running a balance.
> 
> Something like:
> btrfs balance start -dusage=55
> 
> You do *not* want to end up with all your space allocated to Data, but
> not actually used by data.  Bad things can happen if you run out of
> Unallocated space for more metadata. (not catastrophic, but awkward and
> unexpected downtime that can be a little tricky to sort out.)
> 
> 

> begin:vcard
> fn:Remi Gauvin
> n:Gauvin;Remi
> org:Georgian Infotech
> adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada
> email;internet:remi@georgianit.com
> tel;work:226-256-1545
> version:2.1
> end:vcard
> 


-- 
Hugo Mills             | Great oxymorons of the world, no. 8:
hugo@... carfax.org.uk | The Latest In Proven Technology
http://carfax.org.uk/  |
PGP: E2AB1DE4          |

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]

Re: Understanding "btrfs filesystem usage"

[Eli V]

On Mon, Oct 29, 2018 at 6:46 PM Hugo Mills <hugo@carfax.org.uk> wrote:
>
> On Mon, Oct 29, 2018 at 05:57:10PM -0400, Remi Gauvin wrote:
> > On 2018-10-29 02:11 PM, Ulli Horlacher wrote:
> > > I want to know how many free space is left and have problems in
> > > interpreting the output of:
> > >
> > > btrfs filesystem usage
> > > btrfs filesystem df
> > > btrfs filesystem show
> > >
> > >
> >
> > In my not so humble opinion, the filesystem usage command has the
> > easiest to understand output.  It' lays out all the pertinent information.
>
>    Opinions are divided. I find it almost impossible to read, and
> always use btrfs fi df and btrfs fi show together.

I find the tabular output via -T makes btrfs file usage much easier to
read, and it's now the only command I use to look at it space usage on
btrfs.

>
>    There's short tutorials of how to read the output in both cases in
> the FAQ, which is where I start out by directing people in this
> instance.
>
>    Hugo.
>
> > You can clearly see 825GiB is allocated, with 494GiB used, therefore,
> > filesystem show is actually using the "Allocated" value as "Used".
> > Allocated can be thought of "Reserved For".  As the output of the Usage
> > command and df command clearly show, you have almost 400GiB space available.
> >
> > Note that the btrfs commands are clearly and explicitly displaying
> > values in Binary units, (Mi, and Gi prefix, respectively).  If you want
> > df command to match, use -h instead of -H (see man df)
> >
> > An observation:
> >
> > The disparity between 498GiB used and 823Gib is pretty high.  This is
> > probably the result of using an SSD with an older kernel.  If your
> > kernel is not very recent, (sorry, I forget where this was fixed,
> > somewhere around 4.14 or 4.15), then consider mounting with the nossd
> > option.  You can improve this by running a balance.
> >
> > Something like:
> > btrfs balance start -dusage=55
> >
> > You do *not* want to end up with all your space allocated to Data, but
> > not actually used by data.  Bad things can happen if you run out of
> > Unallocated space for more metadata. (not catastrophic, but awkward and
> > unexpected downtime that can be a little tricky to sort out.)
> >
> >
>
> > begin:vcard
> > fn:Remi Gauvin
> > n:Gauvin;Remi
> > org:Georgian Infotech
> > adr:;;3-51 Sykes St. N.;Meaford;ON;N4L 1X3;Canada
> > email;internet:remi@georgianit.com
> > tel;work:226-256-1545
> > version:2.1
> > end:vcard
> >
>
>
> --
> Hugo Mills             | Great oxymorons of the world, no. 8:
> hugo@... carfax.org.uk | The Latest In Proven Technology
> http://carfax.org.uk/  |
> PGP: E2AB1DE4          |

Re: Understanding "btrfs filesystem usage"

[Ulli Horlacher]

On Mon 2018-10-29 (17:57), Remi Gauvin wrote:
> On 2018-10-29 02:11 PM, Ulli Horlacher wrote:
> 
> > I want to know how many free space is left and have problems in
> > interpreting the output of: 
> > 
> > btrfs filesystem usage
> > btrfs filesystem df
> > btrfs filesystem show
> 
> In my not so humble opinion, the filesystem usage command has the
> easiest to understand output.  It' lays out all the pertinent information.
> 
> You can clearly see 825GiB is allocated, with 494GiB used, therefore,
> filesystem show is actually using the "Allocated" value as "Used".
> Allocated can be thought of "Reserved For".

And what is "Device unallocated"? Not reserved?


> As the output of the Usage command and df command clearly show, you have
> almost 400GiB space available.

This is the good part :-)


> The disparity between 498GiB used and 823Gib is pretty high.  This is
> probably the result of using an SSD with an older kernel.  If your
> kernel is not very recent, (sorry, I forget where this was fixed,
> somewhere around 4.14 or 4.15), then consider mounting with the nossd
> option.

I am running kernel 4.4 (it is a Ubuntu 16.04 system)
But /local is on a SSD. Should I really use nossd mount option?!



> You can improve this by running a balance.
> 
> Something like:
> btrfs balance start -dusage=55

I run balance via cron weekly (adapted 
https://software.opensuse.org/package/btrfsmaintenance)


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<85a63523-7e77-f4ca-9947-2c957c5c577a@georgianit.com>

Re: Understanding "btrfs filesystem usage"

[Austin S. Hemmelgarn]

On 10/30/2018 12:10 PM, Ulli Horlacher wrote:
> 
> On Mon 2018-10-29 (17:57), Remi Gauvin wrote:
>> On 2018-10-29 02:11 PM, Ulli Horlacher wrote:
>>
>>> I want to know how many free space is left and have problems in
>>> interpreting the output of:
>>>
>>> btrfs filesystem usage
>>> btrfs filesystem df
>>> btrfs filesystem show
>>
>> In my not so humble opinion, the filesystem usage command has the
>> easiest to understand output.  It' lays out all the pertinent information.
>>
>> You can clearly see 825GiB is allocated, with 494GiB used, therefore,
>> filesystem show is actually using the "Allocated" value as "Used".
>> Allocated can be thought of "Reserved For".
> 
> And what is "Device unallocated"? Not reserved?
> 
> 
>> As the output of the Usage command and df command clearly show, you have
>> almost 400GiB space available.
> 
> This is the good part :-)
> 
> 
>> The disparity between 498GiB used and 823Gib is pretty high.  This is
>> probably the result of using an SSD with an older kernel.  If your
>> kernel is not very recent, (sorry, I forget where this was fixed,
>> somewhere around 4.14 or 4.15), then consider mounting with the nossd
>> option.
> 
> I am running kernel 4.4 (it is a Ubuntu 16.04 system)
> But /local is on a SSD. Should I really use nossd mount option?!
Probably, and you may even want to use it on newer (patched) kernels.

This requires some explanation though.

SSD's are write limited media (write to them too much, and they stop 
working).  This is generally a pretty well known fact, and while it is 
true, it's not anywhere near as much of an issue on modern SSD"s as 
people make it out to be (pretty much, if you've got an SSD made in the 
last 5 years, you almost certainly don't have to worry about this).  The 
`ssd` code in BTRFS behaves as if this is still an issue (and does so in 
a way that doesn't even solve it well).

Put simply, when BTRFS goes to look for space, it treats requests for 
space that ask for less than a certain size as if they are that minimum 
size, and only tries to look for smaller spots if it can't find one at 
least that minimum size.  This has a couple of advantages in terms of 
write performance, especially in the common case of a mostly empty 
filesystem.

For the default (`nossd`) case, that minimum size is 64kB.  So, in most 
cases, the potentially wasted space actually doesn't matter much (most 
writes are bigger than 64k) unless you're doing certain things.

For the old (`ssd`) case, that minimum size is 2MB.  Even with the 
common cases that would normally not have an issue with the 64k default, 
this ends up wasting a _huge_ amount of space.

For the new `ssd` behavior, the minimum is different for data and 
metadata (IIRC, metadata uses the 64k default, while data still uses the 
2M size).  This solves the biggest issues (which were seen with 
metadata), but doesn't completely remove the problem.

Expanding on this further, some unusual workloads actually benefit from 
the old `ssd` behavior, so on newer kernels `ssd_spread` gives that 
behavior.  However, many workloads actually do better with the `nossd` 
behavior (especially the pathological worst case stuff like databases 
and VM disk images), so if you have a recent SSD, you probably want to 
just use that.
> 
> 
>> You can improve this by running a balance.
>>
>> Something like:
>> btrfs balance start -dusage=55
> 
> I run balance via cron weekly (adapted
> https://software.opensuse.org/package/btrfsmaintenance)
> 
> 

Re: Understanding "btrfs filesystem usage"

[Chris Murphy]

On Tue, Oct 30, 2018 at 10:10 AM, Ulli Horlacher
<framstag@rus.uni-stuttgart.de> wrote:
>
> On Mon 2018-10-29 (17:57), Remi Gauvin wrote:
>> On 2018-10-29 02:11 PM, Ulli Horlacher wrote:
>>
>> > I want to know how many free space is left and have problems in
>> > interpreting the output of:
>> >
>> > btrfs filesystem usage
>> > btrfs filesystem df
>> > btrfs filesystem show
>>
>> In my not so humble opinion, the filesystem usage command has the
>> easiest to understand output.  It' lays out all the pertinent information.
>>
>> You can clearly see 825GiB is allocated, with 494GiB used, therefore,
>> filesystem show is actually using the "Allocated" value as "Used".
>> Allocated can be thought of "Reserved For".
>
> And what is "Device unallocated"? Not reserved?

That's a reasonable interpretation. Unallocated space is space that's
not used for anything: no data, no metadata, and isn't reference by
any block group.

It's not a relevant number day to day, I'd say it's advanced leaning
toward esoteric knowledge of Btrfs internals. At this point I'd like
to see a simper output by default, and have a verbose option for
advanced users, and an export option that spits out a superset of all
available information in a format parsable for scripts. But I know
there are other project that depend on btrfs user space output, rather
than having something specifically invented for them that's easily
parsed, and can be kept consistent and extendible, separate from human
user consumption. Oh well!




>> The disparity between 498GiB used and 823Gib is pretty high.  This is
>> probably the result of using an SSD with an older kernel.  If your
>> kernel is not very recent, (sorry, I forget where this was fixed,
>> somewhere around 4.14 or 4.15), then consider mounting with the nossd
>> option.
>
> I am running kernel 4.4 (it is a Ubuntu 16.04 system)
> But /local is on a SSD. Should I really use nossd mount option?!

Yes. But it's not a file system integrity suggestion, it's an optimization.


>
>
>
>> You can improve this by running a balance.
>>
>> Something like:
>> btrfs balance start -dusage=55
>
> I run balance via cron weekly (adapted
> https://software.opensuse.org/package/btrfsmaintenance)

With a newer kernel you can probably reduce this further depending on
your workload and use case. And optionally follow it up with executing
fstrim, or just enable fstrim.timer (we don't recommend using discard
mount option for most use cases as it too aggressively discards very
recently stale Btrfs metadata and can make recovery from crashes
harder).

There is a trim bug that causes FITRIM to only get applied to
unallocated space on older file systems, that have been balanced such
that block group logical addresses are outside the physical address
space of the device which prevents the free space inside of such block
groups to be passed over for FITRIM. Looks like this will be fixed in
kernel 4.20/5.0




-- 
Chris Murphy

Re: Understanding "btrfs filesystem usage"

[Ulli Horlacher]

On Tue 2018-10-30 (17:10), Ulli Horlacher wrote:
> 
> On Mon 2018-10-29 (17:57), Remi Gauvin wrote:
> 
> > On 2018-10-29 02:11 PM, Ulli Horlacher wrote:
> > 
> > > I want to know how many free space is left and have problems in
> > > interpreting the output of: 
> > > 
> > > btrfs filesystem usage
> > > btrfs filesystem df
> > > btrfs filesystem show
> > 
> > In my not so humble opinion, the filesystem usage command has the
> > easiest to understand output.  It' lays out all the pertinent information.
> > 
> > You can clearly see 825GiB is allocated, with 494GiB used, therefore,
> > filesystem show is actually using the "Allocated" value as "Used".
> > Allocated can be thought of "Reserved For".
> 
> And what is "Device unallocated"? Not reserved?

The answer is in the FAQ and I am stupid not to looked at it, first :-}

https://btrfs.wiki.kernel.org/index.php/FAQ#How_much_free_space_do_I_have.3F


-- 
Ullrich Horlacher              Server und Virtualisierung
Rechenzentrum TIK         
Universitaet Stuttgart         E-Mail: horlacher@tik.uni-stuttgart.de
Allmandring 30a                Tel:    ++49-711-68565868
70569 Stuttgart (Germany)      WWW:    http://www.tik.uni-stuttgart.de/
REF:<20181030161003.GA25765@rus.uni-stuttgart.de>