* BTRF - Storage Usage
@ 2012-09-27 10:44 Sébastien Maury
2012-09-27 11:09 ` Hugo Mills
2012-09-27 20:39 ` [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage] Goffredo Baroncelli
0 siblings, 2 replies; 21+ messages in thread
From: Sébastien Maury @ 2012-09-27 10:44 UTC (permalink / raw)
To: linux-btrfs; +Cc: sebastien.maury
Hi,
I've installed a new server using btrfs for my root partition ("/").
It uses snapper for snapshots management and all seems to work pretty fine.
My problem is to be able to know the remaining REAL free space in my
partition.
Using different commands, i have different results, and i don't know
how to interpret them correctly :
poivron:~ # btrfs filesystem df /
Data: total=4.01GB, used=2.16GB
System, DUP: total=8.00MB, used=4.00KB
System: total=4.00MB, used=0.00
Metadata, DUP: total=3.00GB, used=429.16MB
Metadata: total=8.00MB, used=0.00
poivron:~ # df -hP /
Filesystem Size Used Avail Use% Mounted on
/dev/sda3 132G 3.0G 124G 3% /
poivron:~ # btrfs filesystem show /dev/sda3
Label: none uuid: 9e68b667-f9f9-490f-9da1-ae4e91558212
Total devices 1 FS bytes used 2.58GB
devid 1 size 131.64GB used 10.04GB path /dev/sda3
Btrfs v0.19+
poivron:~ # du -sh /.snapshots
40G /.snapshots
===========
Please help me understand and interpret those information to know the
most accurately as possible what is my real remaining space, and what
space is used by what.
Although, i don't really understand the output of the command "btrfs
filesystem df /" : what are exactly "Data", "System DUP", "System
total", "Metadata DUP" and "Metadata total" ?
==========
Here are some complementary informations :
poivron:~ # uname -a
Linux poivron 3.0.26-0.7-default #1 SMP Tue Apr 17 10:27:57 UTC 2012
(3829766) x86_64 x86_64 x86_64 GNU/Linux
poivron:~ # snapper list-configs
Config | Subvolume
--------------+------------------------
root | /
poivron:~ # cat /etc/snapper/configs/root
# subvolume to snapshot
SUBVOLUME="/"
# filesystem type
FSTYPE="btrfs"
# run daily number cleanup
NUMBER_CLEANUP="yes"
# limit for number cleanup
NUMBER_MIN_AGE="1800"
NUMBER_LIMIT="100"
# create hourly snapshots
TIMELINE_CREATE="yes"
# cleanup hourly snapshots after some time
TIMELINE_CLEANUP="yes"
# limits for timeline cleanup
TIMELINE_MIN_AGE="1800"
TIMELINE_LIMIT_HOURLY="10"
TIMELINE_LIMIT_DAILY="10"
TIMELINE_LIMIT_MONTHLY="10"
TIMELINE_LIMIT_YEARLY="10"
# cleanup empty pre-post-pairs
EMPTY_PRE_POST_CLEANUP="yes"
# limits for empty pre-post-pair cleanup
EMPTY_PRE_POST_MIN_AGE="1800"
Cordialement,
Sébastien MAURY
Responsable d'exploitation du site de Montpellier
Équipe DBA
___________________________________
INSERM - DSI - Pôle Infrastructures
Délégation régionale Languedoc Roussillon
60, rue de Navacelles
34394 Montpellier Cedex 5
Mob : 06 31 51 42 18
Fixe : 04 67 63 61 43
Fax : 04 67 63 70 25
Mél : sebastien.maury@inserm.fr
___________________________________
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: BTRF - Storage Usage
2012-09-27 10:44 BTRF - Storage Usage Sébastien Maury
@ 2012-09-27 11:09 ` Hugo Mills
2012-09-27 11:25 ` Sébastien Maury
2012-09-27 20:39 ` [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage] Goffredo Baroncelli
1 sibling, 1 reply; 21+ messages in thread
From: Hugo Mills @ 2012-09-27 11:09 UTC (permalink / raw)
To: Sébastien Maury; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 4166 bytes --]
On Thu, Sep 27, 2012 at 12:44:27PM +0200, Sébastien Maury wrote:
> I've installed a new server using btrfs for my root partition ("/").
>
> It uses snapper for snapshots management and all seems to work pretty fine.
>
> My problem is to be able to know the remaining REAL free space in my
> partition.
This is in the FAQ: https://btrfs.wiki.kernel.org/index.php/FAQ#Why_are_there_so_many_ways_to_check_the_amount_of_free_space.3F
Short answer: you can't know in general.
Longer answer -- see below.
> Using different commands, i have different results, and i don't know
> how to interpret them correctly :
> poivron:~ # btrfs filesystem show /dev/sda3
> Label: none uuid: 9e68b667-f9f9-490f-9da1-ae4e91558212
> Total devices 1 FS bytes used 2.58GB
> devid 1 size 131.64GB used 10.04GB path /dev/sda3
You have 131.64 GiB of raw storage in your filesystem. Of that,
10.04 GiB is currently allocated for use by the FS (and it will take
more as it needs it).
> poivron:~ # btrfs filesystem df /
> Data: total=4.01GB, used=2.16GB
4.01 GiB of the 10.04 GiB allocation is assigned for use by data,
and 2.16 GiB of that allocation actually contains data.
> System, DUP: total=8.00MB, used=4.00KB
16 MiB (=2*8.00 MiB) of the 10.04 GiB allocation is assigned for
use as two copies of the system data. There is 4 KiB of system data
actually used.
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=3.00GB, used=429.16MB
6 GiB (=2*3.00 GiB) of your 10.04 GiB allocation is assigned for
use as metadata, with two copies (DUP) being kept. 429.16 MiB of the
3.00 GiB is currently in use.
> Metadata: total=8.00MB, used=0.00
> poivron:~ # df -hP /
> Filesystem Size Used Avail Use% Mounted on
> /dev/sda3 132G 3.0G 124G 3% /
Plain old df can't handle the truth, so this is at best only a hint
at what's actually happening. When "Avail" reaches zero, your FS is
probably full. Other than that, you can't necessarily say very much.
> ===========
>
> Please help me understand and interpret those information to know the
> most accurately as possible what is my real remaining space, and what
> space is used by what.
>
> Although, i don't really understand the output of the command "btrfs
> filesystem df /" : what are exactly "Data", "System DUP", "System
> total", "Metadata DUP" and "Metadata total" ?
This should all be covered in the glossary on the website:
https://btrfs.wiki.kernel.org/index.php/Glossary
Data is the contents of your files. Metadata is all the other stuff
that the FS needs in order to store your files -- directory
structures, permissions, locations of the file data, that kind of
thing. System is a particular bit of the metadata (the chunk tree)
which governs an internal physical/virtual mapping, and which needs to
be read before anything else can make any kind of sense.
DUP is a bit like RAID-1: anything stored in a DUP chunk is
actually written to two different places on the disk, and can help
recovery in the case of physical disk corruption (e.g. bad blocks,
head crash).
> ==========
>
> Here are some complementary informations :
> poivron:~ # uname -a
> Linux poivron 3.0.26-0.7-default #1 SMP Tue Apr 17 10:27:57 UTC 2012
> (3829766) x86_64 x86_64 x86_64 GNU/Linux
You [probably(*)] need to upgrade your kernel as soon as possible.
btrfs code moves very fast, and 3.0 has significant bugs in it. You
should be running the latest released kernel -- right now, that's 3.5,
or 3.6-rc7. Next week, it will probably change to 3.6 when Linus makes
the next release. Most distributions have a repository somewhere which
will give you access to new kernels without too much trouble.
Hugo.
(*) Some of the enterprise distributions do have backported btrfs
fixes in their apparently older kernels.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- __(_'> Squeak! ---
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: BTRF - Storage Usage
2012-09-27 11:09 ` Hugo Mills
@ 2012-09-27 11:25 ` Sébastien Maury
2012-09-27 11:43 ` Hugo Mills
0 siblings, 1 reply; 21+ messages in thread
From: Sébastien Maury @ 2012-09-27 11:25 UTC (permalink / raw)
To: Hugo Mills; +Cc: linux-btrfs, sebastien.maury
Hi,
Thanks for the quick reply, this clarify me lots of things.
I've had read the articles you mentioned, but i must admit that your
explanations based on my examples makes things even more clearer.
Also, if i understand things properly, snaphots size aren't included
in the "btrfs filesystem show" command output ?
So, the use, for example, of a "du -sh /.snapshots" is correct to
determine the disk usage of my snapshots ?
I will see with the people of my company in charge of maintaining
distributions to provide us a more recent kernel.
PS : I use SLES 11 SP2 distribution.
Hugo Mills <hugo@carfax.org.uk> a écrit :
> On Thu, Sep 27, 2012 at 12:44:27PM +0200, Sébastien Maury wrote:
>> I've installed a new server using btrfs for my root partition ("/").
>>
>> It uses snapper for snapshots management and all seems to work pretty fine.
>>
>> My problem is to be able to know the remaining REAL free space in my
>> partition.
>
> This is in the FAQ:
> https://btrfs.wiki.kernel.org/index.php/FAQ#Why_are_there_so_many_ways_to_check_the_amount_of_free_space.3F
>
> Short answer: you can't know in general.
>
> Longer answer -- see below.
>
>> Using different commands, i have different results, and i don't know
>> how to interpret them correctly :
>
>> poivron:~ # btrfs filesystem show /dev/sda3
>> Label: none uuid: 9e68b667-f9f9-490f-9da1-ae4e91558212
>> Total devices 1 FS bytes used 2.58GB
>> devid 1 size 131.64GB used 10.04GB path /dev/sda3
>
> You have 131.64 GiB of raw storage in your filesystem. Of that,
> 10.04 GiB is currently allocated for use by the FS (and it will take
> more as it needs it).
>
>> poivron:~ # btrfs filesystem df /
>> Data: total=4.01GB, used=2.16GB
>
> 4.01 GiB of the 10.04 GiB allocation is assigned for use by data,
> and 2.16 GiB of that allocation actually contains data.
>
>> System, DUP: total=8.00MB, used=4.00KB
>
> 16 MiB (=2*8.00 MiB) of the 10.04 GiB allocation is assigned for
> use as two copies of the system data. There is 4 KiB of system data
> actually used.
>
>> System: total=4.00MB, used=0.00
>> Metadata, DUP: total=3.00GB, used=429.16MB
>
> 6 GiB (=2*3.00 GiB) of your 10.04 GiB allocation is assigned for
> use as metadata, with two copies (DUP) being kept. 429.16 MiB of the
> 3.00 GiB is currently in use.
>
>> Metadata: total=8.00MB, used=0.00
>
>> poivron:~ # df -hP /
>> Filesystem Size Used Avail Use% Mounted on
>> /dev/sda3 132G 3.0G 124G 3% /
>
> Plain old df can't handle the truth, so this is at best only a hint
> at what's actually happening. When "Avail" reaches zero, your FS is
> probably full. Other than that, you can't necessarily say very much.
>
>> ===========
>>
>> Please help me understand and interpret those information to know the
>> most accurately as possible what is my real remaining space, and what
>> space is used by what.
>>
>> Although, i don't really understand the output of the command "btrfs
>> filesystem df /" : what are exactly "Data", "System DUP", "System
>> total", "Metadata DUP" and "Metadata total" ?
>
> This should all be covered in the glossary on the website:
> https://btrfs.wiki.kernel.org/index.php/Glossary
>
> Data is the contents of your files. Metadata is all the other stuff
> that the FS needs in order to store your files -- directory
> structures, permissions, locations of the file data, that kind of
> thing. System is a particular bit of the metadata (the chunk tree)
> which governs an internal physical/virtual mapping, and which needs to
> be read before anything else can make any kind of sense.
>
> DUP is a bit like RAID-1: anything stored in a DUP chunk is
> actually written to two different places on the disk, and can help
> recovery in the case of physical disk corruption (e.g. bad blocks,
> head crash).
>
>> ==========
>>
>> Here are some complementary informations :
>> poivron:~ # uname -a
>> Linux poivron 3.0.26-0.7-default #1 SMP Tue Apr 17 10:27:57 UTC 2012
>> (3829766) x86_64 x86_64 x86_64 GNU/Linux
>
> You [probably(*)] need to upgrade your kernel as soon as possible.
> btrfs code moves very fast, and 3.0 has significant bugs in it. You
> should be running the latest released kernel -- right now, that's 3.5,
> or 3.6-rc7. Next week, it will probably change to 3.6 when Linus makes
> the next release. Most distributions have a repository somewhere which
> will give you access to new kernels without too much trouble.
>
> Hugo.
>
> (*) Some of the enterprise distributions do have backported btrfs
> fixes in their apparently older kernels.
>
> --
> === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
> PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
> --- __(_'> Squeak! ---
>
Cordialement,
Sébastien MAURY
Responsable d'exploitation du site de Montpellier
Équipe DBA
___________________________________
INSERM - DSI - Pôle Infrastructures
Délégation régionale Languedoc Roussillon
60, rue de Navacelles
34394 Montpellier Cedex 5
Mob : 06 31 51 42 18
Fixe : 04 67 63 61 43
Fax : 04 67 63 70 25
Mél : sebastien.maury@inserm.fr
___________________________________
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: BTRF - Storage Usage
2012-09-27 11:25 ` Sébastien Maury
@ 2012-09-27 11:43 ` Hugo Mills
2012-09-27 11:52 ` Sébastien Maury
0 siblings, 1 reply; 21+ messages in thread
From: Hugo Mills @ 2012-09-27 11:43 UTC (permalink / raw)
To: Sébastien Maury; +Cc: linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 6006 bytes --]
On Thu, Sep 27, 2012 at 01:25:58PM +0200, Sébastien Maury wrote:
> Hi,
>
> Thanks for the quick reply, this clarify me lots of things.
> I've had read the articles you mentioned, but i must admit that your
> explanations based on my examples makes things even more clearer.
>
> Also, if i understand things properly, snaphots size aren't included
> in the "btrfs filesystem show" command output ?
> So, the use, for example, of a "du -sh /.snapshots" is correct to
> determine the disk usage of my snapshots ?
"Disk usage of a snapshot" has two different answers:
1) The total size of the files listed in the snapshot, which you can
get from du.
2) The amount of space that would be freed up by deleting the
snapshot, which isn't currently available, but probably will be
soon. (The additional bookkeeping code was part of the qgroups
patches, which are in 3.6).
> I will see with the people of my company in charge of maintaining
> distributions to provide us a more recent kernel.
>
> PS : I use SLES 11 SP2 distribution.
OK, that one's actually one of the few that does keep proper
backports: https://btrfs.wiki.kernel.org/index.php/Getting_started#Distro_support
That said, I don't know how good they are at keeping up -- probably
pretty good, but other people here may be able to answer that better.
Hugo.
> Hugo Mills <hugo@carfax.org.uk> a écrit :
>
> > On Thu, Sep 27, 2012 at 12:44:27PM +0200, Sébastien Maury wrote:
> >> I've installed a new server using btrfs for my root partition ("/").
> >>
> >> It uses snapper for snapshots management and all seems to work pretty fine.
> >>
> >> My problem is to be able to know the remaining REAL free space in my
> >> partition.
> >
> > This is in the FAQ:
> > https://btrfs.wiki.kernel.org/index.php/FAQ#Why_are_there_so_many_ways_to_check_the_amount_of_free_space.3F
> >
> > Short answer: you can't know in general.
> >
> > Longer answer -- see below.
> >
> >> Using different commands, i have different results, and i don't know
> >> how to interpret them correctly :
> >
> >> poivron:~ # btrfs filesystem show /dev/sda3
> >> Label: none uuid: 9e68b667-f9f9-490f-9da1-ae4e91558212
> >> Total devices 1 FS bytes used 2.58GB
> >> devid 1 size 131.64GB used 10.04GB path /dev/sda3
> >
> > You have 131.64 GiB of raw storage in your filesystem. Of that,
> > 10.04 GiB is currently allocated for use by the FS (and it will take
> > more as it needs it).
> >
> >> poivron:~ # btrfs filesystem df /
> >> Data: total=4.01GB, used=2.16GB
> >
> > 4.01 GiB of the 10.04 GiB allocation is assigned for use by data,
> > and 2.16 GiB of that allocation actually contains data.
> >
> >> System, DUP: total=8.00MB, used=4.00KB
> >
> > 16 MiB (=2*8.00 MiB) of the 10.04 GiB allocation is assigned for
> > use as two copies of the system data. There is 4 KiB of system data
> > actually used.
> >
> >> System: total=4.00MB, used=0.00
> >> Metadata, DUP: total=3.00GB, used=429.16MB
> >
> > 6 GiB (=2*3.00 GiB) of your 10.04 GiB allocation is assigned for
> > use as metadata, with two copies (DUP) being kept. 429.16 MiB of the
> > 3.00 GiB is currently in use.
> >
> >> Metadata: total=8.00MB, used=0.00
> >
> >> poivron:~ # df -hP /
> >> Filesystem Size Used Avail Use% Mounted on
> >> /dev/sda3 132G 3.0G 124G 3% /
> >
> > Plain old df can't handle the truth, so this is at best only a hint
> > at what's actually happening. When "Avail" reaches zero, your FS is
> > probably full. Other than that, you can't necessarily say very much.
> >
> >> ===========
> >>
> >> Please help me understand and interpret those information to know the
> >> most accurately as possible what is my real remaining space, and what
> >> space is used by what.
> >>
> >> Although, i don't really understand the output of the command "btrfs
> >> filesystem df /" : what are exactly "Data", "System DUP", "System
> >> total", "Metadata DUP" and "Metadata total" ?
> >
> > This should all be covered in the glossary on the website:
> > https://btrfs.wiki.kernel.org/index.php/Glossary
> >
> > Data is the contents of your files. Metadata is all the other stuff
> > that the FS needs in order to store your files -- directory
> > structures, permissions, locations of the file data, that kind of
> > thing. System is a particular bit of the metadata (the chunk tree)
> > which governs an internal physical/virtual mapping, and which needs to
> > be read before anything else can make any kind of sense.
> >
> > DUP is a bit like RAID-1: anything stored in a DUP chunk is
> > actually written to two different places on the disk, and can help
> > recovery in the case of physical disk corruption (e.g. bad blocks,
> > head crash).
> >
> >> ==========
> >>
> >> Here are some complementary informations :
> >> poivron:~ # uname -a
> >> Linux poivron 3.0.26-0.7-default #1 SMP Tue Apr 17 10:27:57 UTC 2012
> >> (3829766) x86_64 x86_64 x86_64 GNU/Linux
> >
> > You [probably(*)] need to upgrade your kernel as soon as possible.
> > btrfs code moves very fast, and 3.0 has significant bugs in it. You
> > should be running the latest released kernel -- right now, that's 3.5,
> > or 3.6-rc7. Next week, it will probably change to 3.6 when Linus makes
> > the next release. Most distributions have a repository somewhere which
> > will give you access to new kernels without too much trouble.
> >
> > Hugo.
> >
> > (*) Some of the enterprise distributions do have backported btrfs
> > fixes in their apparently older kernels.
> >
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- A clear conscience. Where did you get this taste ---
for luxuries, Bernard?
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: BTRF - Storage Usage
2012-09-27 11:43 ` Hugo Mills
@ 2012-09-27 11:52 ` Sébastien Maury
0 siblings, 0 replies; 21+ messages in thread
From: Sébastien Maury @ 2012-09-27 11:52 UTC (permalink / raw)
To: Hugo Mills; +Cc: linux-btrfs, sebastien.maury
Hi,
Thanks a lot for your time and answers.
Things look pretty clear now for me.
I'm monitoring my systems using nagios, and i was annoyed about the
disk usage monitoring.
Thanks to your answers, i should be able to developp a rather accurate script.
Or so i hope :)
Regards,
Sebastien.
Hugo Mills <hugo@carfax.org.uk> a écrit :
> On Thu, Sep 27, 2012 at 01:25:58PM +0200, Sébastien Maury wrote:
>> Hi,
>>
>> Thanks for the quick reply, this clarify me lots of things.
>> I've had read the articles you mentioned, but i must admit that your
>> explanations based on my examples makes things even more clearer.
>>
>> Also, if i understand things properly, snaphots size aren't included
>> in the "btrfs filesystem show" command output ?
>> So, the use, for example, of a "du -sh /.snapshots" is correct to
>> determine the disk usage of my snapshots ?
>
> "Disk usage of a snapshot" has two different answers:
>
> 1) The total size of the files listed in the snapshot, which you can
> get from du.
>
> 2) The amount of space that would be freed up by deleting the
> snapshot, which isn't currently available, but probably will be
> soon. (The additional bookkeeping code was part of the qgroups
> patches, which are in 3.6).
>
>> I will see with the people of my company in charge of maintaining
>> distributions to provide us a more recent kernel.
>>
>> PS : I use SLES 11 SP2 distribution.
>
> OK, that one's actually one of the few that does keep proper
> backports:
> https://btrfs.wiki.kernel.org/index.php/Getting_started#Distro_support
>
> That said, I don't know how good they are at keeping up -- probably
> pretty good, but other people here may be able to answer that better.
>
> Hugo.
>
>> Hugo Mills <hugo@carfax.org.uk> a écrit :
>>
>> > On Thu, Sep 27, 2012 at 12:44:27PM +0200, Sébastien Maury wrote:
>> >> I've installed a new server using btrfs for my root partition ("/").
>> >>
>> >> It uses snapper for snapshots management and all seems to work
>> pretty fine.
>> >>
>> >> My problem is to be able to know the remaining REAL free space in my
>> >> partition.
>> >
>> > This is in the FAQ:
>> >
>> https://btrfs.wiki.kernel.org/index.php/FAQ#Why_are_there_so_many_ways_to_check_the_amount_of_free_space.3F
>> >
>> > Short answer: you can't know in general.
>> >
>> > Longer answer -- see below.
>> >
>> >> Using different commands, i have different results, and i don't know
>> >> how to interpret them correctly :
>> >
>> >> poivron:~ # btrfs filesystem show /dev/sda3
>> >> Label: none uuid: 9e68b667-f9f9-490f-9da1-ae4e91558212
>> >> Total devices 1 FS bytes used 2.58GB
>> >> devid 1 size 131.64GB used 10.04GB path /dev/sda3
>> >
>> > You have 131.64 GiB of raw storage in your filesystem. Of that,
>> > 10.04 GiB is currently allocated for use by the FS (and it will take
>> > more as it needs it).
>> >
>> >> poivron:~ # btrfs filesystem df /
>> >> Data: total=4.01GB, used=2.16GB
>> >
>> > 4.01 GiB of the 10.04 GiB allocation is assigned for use by data,
>> > and 2.16 GiB of that allocation actually contains data.
>> >
>> >> System, DUP: total=8.00MB, used=4.00KB
>> >
>> > 16 MiB (=2*8.00 MiB) of the 10.04 GiB allocation is assigned for
>> > use as two copies of the system data. There is 4 KiB of system data
>> > actually used.
>> >
>> >> System: total=4.00MB, used=0.00
>> >> Metadata, DUP: total=3.00GB, used=429.16MB
>> >
>> > 6 GiB (=2*3.00 GiB) of your 10.04 GiB allocation is assigned for
>> > use as metadata, with two copies (DUP) being kept. 429.16 MiB of the
>> > 3.00 GiB is currently in use.
>> >
>> >> Metadata: total=8.00MB, used=0.00
>> >
>> >> poivron:~ # df -hP /
>> >> Filesystem Size Used Avail Use% Mounted on
>> >> /dev/sda3 132G 3.0G 124G 3% /
>> >
>> > Plain old df can't handle the truth, so this is at best only a hint
>> > at what's actually happening. When "Avail" reaches zero, your FS is
>> > probably full. Other than that, you can't necessarily say very much.
>> >
>> >> ===========
>> >>
>> >> Please help me understand and interpret those information to know the
>> >> most accurately as possible what is my real remaining space, and what
>> >> space is used by what.
>> >>
>> >> Although, i don't really understand the output of the command "btrfs
>> >> filesystem df /" : what are exactly "Data", "System DUP", "System
>> >> total", "Metadata DUP" and "Metadata total" ?
>> >
>> > This should all be covered in the glossary on the website:
>> > https://btrfs.wiki.kernel.org/index.php/Glossary
>> >
>> > Data is the contents of your files. Metadata is all the other stuff
>> > that the FS needs in order to store your files -- directory
>> > structures, permissions, locations of the file data, that kind of
>> > thing. System is a particular bit of the metadata (the chunk tree)
>> > which governs an internal physical/virtual mapping, and which needs to
>> > be read before anything else can make any kind of sense.
>> >
>> > DUP is a bit like RAID-1: anything stored in a DUP chunk is
>> > actually written to two different places on the disk, and can help
>> > recovery in the case of physical disk corruption (e.g. bad blocks,
>> > head crash).
>> >
>> >> ==========
>> >>
>> >> Here are some complementary informations :
>> >> poivron:~ # uname -a
>> >> Linux poivron 3.0.26-0.7-default #1 SMP Tue Apr 17 10:27:57 UTC 2012
>> >> (3829766) x86_64 x86_64 x86_64 GNU/Linux
>> >
>> > You [probably(*)] need to upgrade your kernel as soon as possible.
>> > btrfs code moves very fast, and 3.0 has significant bugs in it. You
>> > should be running the latest released kernel -- right now, that's 3.5,
>> > or 3.6-rc7. Next week, it will probably change to 3.6 when Linus makes
>> > the next release. Most distributions have a repository somewhere which
>> > will give you access to new kernels without too much trouble.
>> >
>> > Hugo.
>> >
>> > (*) Some of the enterprise distributions do have backported btrfs
>> > fixes in their apparently older kernels.
>> >
>
> --
> === Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
> PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
> --- A clear conscience. Where did you get this taste ---
> for luxuries, Bernard?
>
Cordialement,
Sébastien MAURY
Responsable d'exploitation du site de Montpellier
Équipe DBA
___________________________________
INSERM - DSI - Pôle Infrastructures
Délégation régionale Languedoc Roussillon
60, rue de Navacelles
34394 Montpellier Cedex 5
Mob : 06 31 51 42 18
Fixe : 04 67 63 61 43
Fax : 04 67 63 70 25
Mél : sebastien.maury@inserm.fr
___________________________________
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
^ permalink raw reply [flat|nested] 21+ messages in thread
* [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-27 10:44 BTRF - Storage Usage Sébastien Maury
2012-09-27 11:09 ` Hugo Mills
@ 2012-09-27 20:39 ` Goffredo Baroncelli
2012-09-27 21:02 ` Goffredo Baroncelli
1 sibling, 1 reply; 21+ messages in thread
From: Goffredo Baroncelli @ 2012-09-27 20:39 UTC (permalink / raw)
To: Sébastien Maury; +Cc: linux-btrfs, Hugo Mills
On 09/27/2012 12:44 PM, Sébastien Maury wrote:
> Hi,
>
> I've installed a new server using btrfs for my root partition ("/").
>
> It uses snapper for snapshots management and all seems to work pretty fine.
>
> My problem is to be able to know the remaining REAL free space in my
> partition.
>
> Using different commands, i have different results, and i don't know how
> to interpret them correctly :
> poivron:~ # btrfs filesystem df /
> Data: total=4.01GB, used=2.16GB
> System, DUP: total=8.00MB, used=4.00KB
> System: total=4.00MB, used=0.00
> Metadata, DUP: total=3.00GB, used=429.16MB
> Metadata: total=8.00MB, used=0.00
In effect the output of "btrfs filesystem df /" is not very friendly.
What about changing the output as below:
$ btrfs filesystem disk-free /
Summary:
Total: 135.00GB
Allocated: 10.51GB
Unallocated: 124.49GB
Free_(Estimated) 86.56GB
Average_disk_efficiency: 62 %
Details:
Chunk-type Mode Allocated Used Free
---------- ---- --------- -------- ---------
Data Single 4.01GB 2.16GB 1.87GB
System DUP 16.00MB 4.00KB 7.99MB
System Single 4.00MB 0.00 4.00MB
Metadata DUP 6.00GB 429.16MB 2.57GB
Metadata Single 8.00MB 0.00 8.00MB
Where the "Free_(Estimated)" and "Average_disk_efficency" are computed as:
Average_disk_efficency = ratio of average disk usage =
(sum(ChunkUsed)+sum(ChunkFree))/sum(ChunkAllocated)
Estimated_available = Average_disk_efficency *
Unallocated+sum(ChunkFree)
I am open to suggestion about the terms: Used vs Allocated and Free vs
Available, or a better description of "Average disk efficiency"
BR
G.Baroncelli
P.S. the source could be find at
http://cassiopea.homelinux.net/git/btrfs-progs-unstable.git
branch
disk_free
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-27 20:39 ` [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage] Goffredo Baroncelli
@ 2012-09-27 21:02 ` Goffredo Baroncelli
2012-09-28 3:17 ` Roman Mamedov
0 siblings, 1 reply; 21+ messages in thread
From: Goffredo Baroncelli @ 2012-09-27 21:02 UTC (permalink / raw)
Cc: Sébastien Maury, linux-btrfs, Hugo Mills
Sorry for the space error:
Below a more correct example
$ btrfs filesystem disk-free /
Summary:
Total: 135.00GB
Allocated: 10.51GB
Unallocated: 124.49GB
Free_(Estimated) 86.56GB
Average_disk_efficiency: 62 %
Details:
Chunk-type Mode Allocated Used Free
---------- ---- --------- -------- ---------
Data Single 4.01GB 2.16GB 1.87GB
System DUP 16.00MB 4.00KB 7.99MB
System Single 4.00MB 0.00 4.00MB
Metadata DUP 6.00GB 429.16MB 2.57GB
Metadata Single 8.00MB 0.00 8.00MB
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-27 21:02 ` Goffredo Baroncelli
@ 2012-09-28 3:17 ` Roman Mamedov
2012-09-28 8:58 ` Hugo Mills
2012-09-28 16:44 ` Goffredo Baroncelli
0 siblings, 2 replies; 21+ messages in thread
From: Roman Mamedov @ 2012-09-28 3:17 UTC (permalink / raw)
To: kreijack; +Cc: kreijack, Sébastien Maury, linux-btrfs, Hugo Mills
[-- Attachment #1: Type: text/plain, Size: 1482 bytes --]
On Thu, 27 Sep 2012 23:02:35 +0200
Goffredo Baroncelli <kreijack@libero.it> wrote:
> Sorry for the space error:
> Below a more correct example
>
> $ btrfs filesystem disk-free /
> Summary:
> Total: 135.00GB
> Allocated: 10.51GB
> Unallocated: 124.49GB
> Free_(Estimated) 86.56GB
> Average_disk_efficiency: 62 %
How do you estimate "Free" here? Sorry I didn't check the source code in git,
but from the "Details" below nothing leads me to believe that this FS is
doomed to only be able to usefully utilize only ~86GB of the partition, and not
more.
Are you ready to answer the flood of questions from people why their disk is
only 62% efficient, and how to tune it to 100%? :-)
Why use underscores instead of spaces?
>
> Details:
> Chunk-type Mode Allocated Used Free
> ---------- ---- --------- -------- ---------
> Data Single 4.01GB 2.16GB 1.87GB
> System DUP 16.00MB 4.00KB 7.99MB
> System Single 4.00MB 0.00 4.00MB
> Metadata DUP 6.00GB 429.16MB 2.57GB
> Metadata Single 8.00MB 0.00 8.00MB
--
With respect,
Roman
~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-28 3:17 ` Roman Mamedov
@ 2012-09-28 8:58 ` Hugo Mills
2012-09-28 17:27 ` Goffredo Baroncelli
2012-11-12 18:16 ` Jan Engelhardt
2012-09-28 16:44 ` Goffredo Baroncelli
1 sibling, 2 replies; 21+ messages in thread
From: Hugo Mills @ 2012-09-28 8:58 UTC (permalink / raw)
To: Roman Mamedov; +Cc: kreijack, kreijack, Sébastien Maury, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 3989 bytes --]
On Fri, Sep 28, 2012 at 09:17:59AM +0600, Roman Mamedov wrote:
> On Thu, 27 Sep 2012 23:02:35 +0200
> Goffredo Baroncelli <kreijack@libero.it> wrote:
>
> > Sorry for the space error:
> > Below a more correct example
> >
> > $ btrfs filesystem disk-free /
> > Summary:
> > Total: 135.00GB
> > Allocated: 10.51GB
> > Unallocated: 124.49GB
> > Free_(Estimated) 86.56GB
> > Average_disk_efficiency: 62 %
>
> How do you estimate "Free" here? Sorry I didn't check the source code in git,
> but from the "Details" below nothing leads me to believe that this FS is
> doomed to only be able to usefully utilize only ~86GB of the partition, and not
> more.
>
> Are you ready to answer the flood of questions from people why their disk is
> only 62% efficient, and how to tune it to 100%? :-)
Data_to_disk_ratio, maybe?
> Why use underscores instead of spaces?
So that you can use, say, "read" in the shell to extract data from
each line. To that end, there should be a space between the value and
the unit throughout.
> > Details:
> > Chunk-type Mode Allocated Used Free
> > ---------- ---- --------- -------- ---------
Minor thing: The underlines are largely superfluous. Few basic CL
tools I can think of use them.
> > Data Single 4.01GB 2.16GB 1.87GB
> > System DUP 16.00MB 4.00KB 7.99MB
> > System Single 4.00MB 0.00 4.00MB
> > Metadata DUP 6.00GB 429.16MB 2.57GB
> > Metadata Single 8.00MB 0.00 8.00MB
I think we need another column here, to indicate how much *actual*
disk space is used by each row, so adding up that column will give you
the "Allocated" value in the first clause. I think that's probably the
biggest cause of confusion. "Raw alloc.", maybe, and use the term
"raw" somewhere in the first clause to hammer the point home.
My only concern here is that we're a bit too close to the existing
solution (albeit merging the two sets of output), which has proven
itself over time to be somewhat confusing. I think the Alloc_Raw
column is the minimum necessary to link the two in some easily
determinable way. Adding totals to Alloc_Raw, and Used (but not Free
or Alloc) would help, I think. I don't think it's useful to add them
to the Free or Alloc columns, because those figures change as the FS
allocates chunks, and we'll end up with people querying the fact that
the total of Free doesn't add up to any of the figures in the
summary.
Say, something like this:
Summary_(Raw):
Total: 135.00 GiB
Allocated: 10.51 GiB
Unallocated: 124.49 GiB
Free_(Estimated): 86.56 GiB
Average_disk_efficiency: 62 %
Details:
Chunk_type Mode Alloc_Raw Alloc Used Free
Data Single 4.01 GiB 4.01 GiB 2.16 GiB 1.87 GiB
System DUP 32.00 MiB 16.00 MiB 4.00 KiB 7.99 MiB
System Single 4.00 MiB 4.00 MiB 0.00 B 4.00 MiB
Metadata DUP 12.00 GiB 6.00 GiB 429.16 MiB 2.57 GiB
Metadata Single 8.00 MiB 8.00 MiB 0.00 B 8.00 MiB
Total 16.04 GiB 2.59 GiB
The other thing is that there should be a switch (or possibly two)
to give highly machine-readable versions of the output -- no units
(units as bytes by default, with other units settable by a switch),
tab-separated, possibly a different option for each of the above
output clauses.
Ultimately, I think the bikeshed should be turquoise.
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Python is executable pseudocode; perl ---
is executable line-noise.
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-28 3:17 ` Roman Mamedov
2012-09-28 8:58 ` Hugo Mills
@ 2012-09-28 16:44 ` Goffredo Baroncelli
2012-09-28 18:02 ` Roman Mamedov
1 sibling, 1 reply; 21+ messages in thread
From: Goffredo Baroncelli @ 2012-09-28 16:44 UTC (permalink / raw)
To: Roman Mamedov; +Cc: kreijack, Sébastien Maury, linux-btrfs, Hugo Mills
On 09/28/2012 05:17 AM, Roman Mamedov wrote:
> On Thu, 27 Sep 2012 23:02:35 +0200
> Goffredo Baroncelli<kreijack@libero.it> wrote:
>
>> Sorry for the space error:
>> Below a more correct example
>>
>> $ btrfs filesystem disk-free /
>> Summary:
>> Total: 135.00GB
>> Allocated: 10.51GB
>> Unallocated: 124.49GB
>> Free_(Estimated) 86.56GB
>> Average_disk_efficiency: 62 %
>
> How do you estimate "Free" here? Sorry I didn't check the source code in git,
> but from the "Details" below nothing leads me to believe that this FS is
> doomed to only be able to usefully utilize only ~86GB of the partition, and not
> more.
The estimation is made on the basis of the real allocated space on the
disk and the available space.
In the example we know that BTRFS allocate:
- 4GB in Single mode (4GB available, 2.16GB used)
- 16MB in DUP mode (so 16/2=8MB available, 4kb used)
- 4MB in Single mode (4MB available)
- 6GB in DUP mode (6/2=3GB available, 429MB used)
- 8MB in Single mode (8MB available)
So BTRFS allocated on disk 4GB+16MB+4MB+6GB+8MB = ~10GB, but the space
availabled (regarding these allocated chunks) is 4GB+8MB+4MB+3GB+8MB = ~7GB.
This means that the ration of space physically allocated on the disk and
the space available is 7GB/10GB = 0.7 . So on 135GB of disk, only 94GB
are available.
Yes my previous 0.62 was wrong. The real ratio is 0.7.
> Are you ready to answer the flood of questions from people why their disk is
> only 62% efficient, and how to tune it to 100%? :-)
I don't understand your question: by default BTRFS store all metadata
DUP-ed, this means that on the disk the space allocated are 2 times the
space required. Because on BTRFS the metadata are a lot, this means that
BTRFS is not so efficiency as other file-systems. This is a well know fact.
If you want to use all the space with the maximum efficiency, you could
format the filesystem with the options "-m single".
>
> Why use underscores instead of spaces?
Simplify the parsing in scripts
>
>>
>> Details:
>> Chunk-type Mode Allocated Used Free
>> ---------- ---- --------- -------- ---------
>> Data Single 4.01GB 2.16GB 1.87GB
>> System DUP 16.00MB 4.00KB 7.99MB
>> System Single 4.00MB 0.00 4.00MB
>> Metadata DUP 6.00GB 429.16MB 2.57GB
>> Metadata Single 8.00MB 0.00 8.00MB
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-28 8:58 ` Hugo Mills
@ 2012-09-28 17:27 ` Goffredo Baroncelli
2012-09-28 20:13 ` Hugo Mills
2012-11-12 18:16 ` Jan Engelhardt
1 sibling, 1 reply; 21+ messages in thread
From: Goffredo Baroncelli @ 2012-09-28 17:27 UTC (permalink / raw)
To: Hugo Mills, Roman Mamedov, kreijack, Sébastien Maury,
linux-btrfs
Hi Hugo,
On 09/28/2012 10:58 AM, Hugo Mills wrote:
> On Fri, Sep 28, 2012 at 09:17:59AM +0600, Roman Mamedov wrote:
>> On Thu, 27 Sep 2012 23:02:35 +0200
>> Goffredo Baroncelli<kreijack@libero.it> wrote:
>>
[...]
> So that you can use, say, "read" in the shell to extract data from
> each line. To that end, there should be a space between the value and
> the unit throughout.
>
>>> Details:
>>> Chunk-type Mode Allocated Used Free
>>> ---------- ---- --------- -------- ---------
>
> Minor thing: The underlines are largely superfluous. Few basic CL
> tools I can think of use them.
Ok
>
>>> Data Single 4.01GB 2.16GB 1.87GB
>>> System DUP 16.00MB 4.00KB 7.99MB
>>> System Single 4.00MB 0.00 4.00MB
>>> Metadata DUP 6.00GB 429.16MB 2.57GB
>>> Metadata Single 8.00MB 0.00 8.00MB
>
> I think we need another column here, to indicate how much *actual*
> disk space is used by each row, so adding up that column will give you
> the "Allocated" value in the first clause. I think that's probably the
> biggest cause of confusion. "Raw alloc.", maybe, and use the term
> "raw" somewhere in the first clause to hammer the point home.
I think that there is a little misunderstanding. We are saying the same
thing. Only I call "allocated" what you call "raw alloc"
>
> My only concern here is that we're a bit too close to the existing
> solution (albeit merging the two sets of output), which has proven
> itself over time to be somewhat confusing. I think the Alloc_Raw
> column is the minimum necessary to link the two in some easily
> determinable way. Adding totals to Alloc_Raw, and Used (but not Free
> or Alloc) would help, I think. I don't think it's useful to add them
> to the Free or Alloc columns, because those figures change as the FS
> allocates chunks, and we'll end up with people querying the fact that
> the total of Free doesn't add up to any of the figures in the
> summary.
>
> Say, something like this:
>
> Summary_(Raw):
> Total: 135.00 GiB
> Allocated: 10.51 GiB
> Unallocated: 124.49 GiB
> Free_(Estimated): 86.56 GiB
> Average_disk_efficiency: 62 %
>
> Details:
> Chunk_type Mode Alloc_Raw Alloc Used Free
> Data Single 4.01 GiB 4.01 GiB 2.16 GiB 1.87 GiB
> System DUP 32.00 MiB 16.00 MiB 4.00 KiB 7.99 MiB
> System Single 4.00 MiB 4.00 MiB 0.00 B 4.00 MiB
> Metadata DUP 12.00 GiB 6.00 GiB 429.16 MiB 2.57 GiB
> Metadata Single 8.00 MiB 8.00 MiB 0.00 B 8.00 MiB
> Total 16.04 GiB 2.59 GiB
>
> The other thing is that there should be a switch (or possibly two)
> to give highly machine-readable versions of the output -- no units
> (units as bytes by default, with other units settable by a switch),
> tab-separated, possibly a different option for each of the above
> output clauses.
I fully Agree. But my first concern was about the wording (if fact even
though we are saying the same thing you didn't understood me).
Let me propose the following:
Summary:
Disk_size: 135.00 GiB
Disk_allocated: 10.51 GiB
Disk_unallocated: 124.49 GiB
Used: 2.59 GiB
Free_(Estimated): 91.93 GiB
Average_disk_efficiency: 70 %
Details:
Chunk-type Mode Disk-allocated Used Available
Data Single 4.01GB 2.16GB 1.87GB
System DUP 16.00MB 4.00KB 7.99MB
System Single 4.00MB 0.00 4.00MB
Metadata DUP 6.00GB 429.16MB 2.57GB
Metadata Single 8.00MB 0.00 8.00MB
Where:
Disk-allocated -> space used on the disk by the chunk
Disk-size -> size of the disk
Disk-unallocated -> disk not used in any chunk
Used -> space used by the files/metadata
Available -> space available in the *allocated* chunk
Free_(Estimated) -> Theoretical free space for files (Disk_size
* Average_disk_efficiency - Used)
>
> Ultimately, I think the bikeshed should be turquoise.
? :-)
>
> Hugo.
>
Ciao
Goffredo
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-28 16:44 ` Goffredo Baroncelli
@ 2012-09-28 18:02 ` Roman Mamedov
2012-09-28 19:38 ` Goffredo Baroncelli
2012-09-28 20:20 ` Hugo Mills
0 siblings, 2 replies; 21+ messages in thread
From: Roman Mamedov @ 2012-09-28 18:02 UTC (permalink / raw)
To: kreijack; +Cc: kreijack, Sébastien Maury, linux-btrfs, Hugo Mills
[-- Attachment #1: Type: text/plain, Size: 3264 bytes --]
On Fri, 28 Sep 2012 18:44:07 +0200
Goffredo Baroncelli <kreijack@inwind.it> wrote:
> This means that the ration of space physically allocated on the disk and
> the space available is 7GB/10GB = 0.7 . So on 135GB of disk, only 94GB
> are available.
You assume metadata allocation will always grow linearly with data, which is
not true. So in my opinion it is not a good estimate.
> > Are you ready to answer the flood of questions from people why their disk is
> > only 62% efficient, and how to tune it to 100%? :-)
>
> I don't understand your question
You mentioned that the aim was to make the output more friendly, i.e. to make
it less confusing. But I find this percentage and the way it is labeled likely
to achieve the opposite effect, causing a lot of new questions on what does
this mean (while the percentage reported is likely not even being correct),
how to improve it, etc.
> Because on BTRFS the metadata are a lot
Keep in mind that there is also inlining; so even if the space is allocated
for metadata, it will be used to store small files. So it might be not
completely fair to count the metadata allocated space as unusable space.
> > Why use underscores instead of spaces?
>
> Simplify the parsing in scripts
I think it looks awkward and is not warranted since this is a primarily
user-facing utility. Also none of the other similar tools shy from having
spaces anywhere they need to, e.g.
# mdadm --detail /dev/md0
/dev/md0:
Version : 1.2
Creation Time : Wed May 25 00:07:38 2011
Raid Level : raid5
Array Size : 3907003136 (3726.01 GiB 4000.77 GB)
Used Dev Size : 976750784 (931.50 GiB 1000.19 GB)
Raid Devices : 5
Total Devices : 5
Persistence : Superblock is persistent
Intent Bitmap : Internal
Update Time : Fri Sep 28 21:20:51 2012
State : active
Active Devices : 5
Working Devices : 5
Failed Devices : 0
Spare Devices : 0
Layout : left-symmetric
Chunk Size : 64K
Name : avdeb:0 (local to host avdeb)
UUID : b99961fb:ed1f76c8:ec2dad31:6db45332
Events : 14254
Number Major Minor RaidDevice State
7 8 17 0 active sync /dev/sdb1
6 8 33 1 active sync /dev/sdc1
3 8 65 2 active sync /dev/sde1
4 8 49 3 active sync /dev/sdd1
5 8 81 4 active sync /dev/sdf1
# lvdisplay
--- Logical volume ---
LV Path /dev/alpha/lv1
LV Name lv1
VG Name alpha
LV UUID HP19fU-oMhM-sdqN-yFWa-N3Rs-ktBw-21GSD2
LV Write Access read/write
LV Creation host, time ,
LV Status available
# open 0
LV Size 3.52 TiB
Current LE 115431
Segments 3
Allocation inherit
Read ahead sectors auto
- currently set to 4096
Block device 252:0
--
With respect,
Roman
~~~~~~~~~~~~~~~~~~~~~~~~~~~
"Stallman had a printer,
with code he could not see.
So he began to tinker,
and set the software free."
[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 198 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-28 18:02 ` Roman Mamedov
@ 2012-09-28 19:38 ` Goffredo Baroncelli
2012-09-28 20:20 ` Hugo Mills
1 sibling, 0 replies; 21+ messages in thread
From: Goffredo Baroncelli @ 2012-09-28 19:38 UTC (permalink / raw)
To: Roman Mamedov; +Cc: Sébastien Maury, linux-btrfs, Hugo Mills
On 09/28/2012 08:02 PM, Roman Mamedov wrote:
> On Fri, 28 Sep 2012 18:44:07 +0200
> Goffredo Baroncelli<kreijack@inwind.it> wrote:
>
>> This means that the ration of space physically allocated on the disk and
>> the space available is 7GB/10GB = 0.7 . So on 135GB of disk, only 94GB
>> are available.
>
> You assume metadata allocation will always grow linearly with data, which is
> not true. So in my opinion it is not a good estimate.
I am open to accept suggestion on how improve the algorithm. Today we
have only ... nothing. If I elaborate the output of btrfs fi show I can
estimate the best-case (i.e. the data have no further redundancy); my
algorithm is a bit smarter. However I repeat: please suggest us a better
algorithm.
Regarding the assumption about the ratio data/metadata is constant, yes
I assumed that. Why this should change ? Of course could change, but
which would be a better estimation ?
My algorithm is not perfect, but better than nothing.
>
>>> Are you ready to answer the flood of questions from people why their disk is
>>> only 62% efficient, and how to tune it to 100%? :-)
>>
>> I don't understand your question
>
> You mentioned that the aim was to make the output more friendly, i.e. to make
> it less confusing. But I find this percentage and the way it is labeled likely
> to achieve the opposite effect, causing a lot of new questions on what does
> this mean (while the percentage reported is likely not even being correct),
> how to improve it, etc.
These questions already are there, because the free space estimation in
BTRFS is
a) very complex
b) "btrfs fi df" and "btrfs fi show" don't help to measure ( nor
estimate) the space available.
>
>> Because on BTRFS the metadata are a lot
>
> Keep in mind that there is also inlining; so even if the space is allocated
> for metadata, it will be used to store small files. So it might be not
> completely fair to count the metadata allocated space as unusable space.
I never told that the metadata space is unusable space. Is true the
opposite: I don't differentiate data/metadata/system.... I only consider
the RAID/DUP/Single in terms of disk-space/available-space.
>
>>> Why use underscores instead of spaces?
>>
>> Simplify the parsing in scripts
>
> I think it looks awkward and is not warranted since this is a primarily
> user-facing utility. Also none of the other similar tools shy from having
> spaces anywhere they need to, e.g.
We could improve on this side. However these utilities are often used in
scripts
>
> # mdadm --detail /dev/md0
> /dev/md0:
> Version : 1.2
> Creation Time : Wed May 25 00:07:38 2011
> Raid Level : raid5
> Array Size : 3907003136 (3726.01 GiB 4000.77 GB)
> Used Dev Size : 976750784 (931.50 GiB 1000.19 GB)
> Raid Devices : 5
> Total Devices : 5
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Fri Sep 28 21:20:51 2012
> State : active
> Active Devices : 5
> Working Devices : 5
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Name : avdeb:0 (local to host avdeb)
> UUID : b99961fb:ed1f76c8:ec2dad31:6db45332
> Events : 14254
>
> Number Major Minor RaidDevice State
> 7 8 17 0 active sync /dev/sdb1
> 6 8 33 1 active sync /dev/sdc1
> 3 8 65 2 active sync /dev/sde1
> 4 8 49 3 active sync /dev/sdd1
> 5 8 81 4 active sync /dev/sdf1
>
> # lvdisplay
> --- Logical volume ---
> LV Path /dev/alpha/lv1
> LV Name lv1
> VG Name alpha
> LV UUID HP19fU-oMhM-sdqN-yFWa-N3Rs-ktBw-21GSD2
> LV Write Access read/write
> LV Creation host, time ,
> LV Status available
> # open 0
> LV Size 3.52 TiB
> Current LE 115431
> Segments 3
> Allocation inherit
> Read ahead sectors auto
> - currently set to 4096
> Block device 252:0
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-28 17:27 ` Goffredo Baroncelli
@ 2012-09-28 20:13 ` Hugo Mills
2012-09-28 21:26 ` Goffredo Baroncelli
2012-09-29 7:19 ` Goffredo Baroncelli
0 siblings, 2 replies; 21+ messages in thread
From: Hugo Mills @ 2012-09-28 20:13 UTC (permalink / raw)
To: Goffredo Baroncelli
Cc: Roman Mamedov, kreijack, Sébastien Maury, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 5847 bytes --]
Hi, Goffredo,
On Fri, Sep 28, 2012 at 07:27:16PM +0200, Goffredo Baroncelli wrote:
> On 09/28/2012 10:58 AM, Hugo Mills wrote:
> >On Fri, Sep 28, 2012 at 09:17:59AM +0600, Roman Mamedov wrote:
> >>On Thu, 27 Sep 2012 23:02:35 +0200
> >>Goffredo Baroncelli<kreijack@libero.it> wrote:
> >>
> [...]
[...]
> >>>Details:
> >>> Chunk-type Mode Allocated Used Free
> >>> ---------- ---- --------- -------- ---------
[...]
> >>> Data Single 4.01GB 2.16GB 1.87GB
> >>> System DUP 16.00MB 4.00KB 7.99MB
> >>> System Single 4.00MB 0.00 4.00MB
> >>> Metadata DUP 6.00GB 429.16MB 2.57GB
> >>> Metadata Single 8.00MB 0.00 8.00MB
> >
> > I think we need another column here, to indicate how much *actual*
> >disk space is used by each row, so adding up that column will give you
> >the "Allocated" value in the first clause. I think that's probably the
> >biggest cause of confusion. "Raw alloc.", maybe, and use the term
> >"raw" somewhere in the first clause to hammer the point home.
>
> I think that there is a little misunderstanding. We are saying the
> same thing. Only I call "allocated" what you call "raw alloc"
OK, I think we need both. We need to indicate somewhere (in the
"Details" section in my version) both the total number of bits of rust
used and the amount of data stored. It's not good to ask the user to
know that they need to multiply/divide by two for certain storage
modes (or even more complicated for RAID-5/6). Somewhere, they will
find that values change twice as fast as they expect (or at half the
speed), and that causes problems. We need to find some way of
connecting the two in a way that makes it reasonably obvious where the
figures come from..
> > My only concern here is that we're a bit too close to the existing
> >solution (albeit merging the two sets of output), which has proven
> >itself over time to be somewhat confusing. I think the Alloc_Raw
> >column is the minimum necessary to link the two in some easily
> >determinable way. Adding totals to Alloc_Raw, and Used (but not Free
> >or Alloc) would help, I think. I don't think it's useful to add them
> >to the Free or Alloc columns, because those figures change as the FS
> >allocates chunks, and we'll end up with people querying the fact that
> >the total of Free doesn't add up to any of the figures in the
> >summary.
> >
> > Say, something like this:
> >
> >Summary_(Raw):
> > Total: 135.00 GiB
> > Allocated: 10.51 GiB
> > Unallocated: 124.49 GiB
> > Free_(Estimated): 86.56 GiB
> > Average_disk_efficiency: 62 %
> >
> >Details:
> > Chunk_type Mode Alloc_Raw Alloc Used Free
> > Data Single 4.01 GiB 4.01 GiB 2.16 GiB 1.87 GiB
> > System DUP 32.00 MiB 16.00 MiB 4.00 KiB 7.99 MiB
> > System Single 4.00 MiB 4.00 MiB 0.00 B 4.00 MiB
> > Metadata DUP 12.00 GiB 6.00 GiB 429.16 MiB 2.57 GiB
> > Metadata Single 8.00 MiB 8.00 MiB 0.00 B 8.00 MiB
> > Total 16.04 GiB 2.59 GiB
> >
> > The other thing is that there should be a switch (or possibly two)
> >to give highly machine-readable versions of the output -- no units
> >(units as bytes by default, with other units settable by a switch),
> >tab-separated, possibly a different option for each of the above
> >output clauses.
> I fully Agree. But my first concern was about the wording (if fact
> even though we are saying the same thing you didn't understood me).
>
> Let me propose the following:
>
> Summary:
> Disk_size: 135.00 GiB
> Disk_allocated: 10.51 GiB
> Disk_unallocated: 124.49 GiB
> Used: 2.59 GiB
> Free_(Estimated): 91.93 GiB
> Average_disk_efficiency: 70 %
>
> Details:
> Chunk-type Mode Disk-allocated Used Available
> Data Single 4.01GB 2.16GB 1.87GB
> System DUP 16.00MB 4.00KB 7.99MB
> System Single 4.00MB 0.00 4.00MB
> Metadata DUP 6.00GB 429.16MB 2.57GB
> Metadata Single 8.00MB 0.00 8.00MB
>
>
>
> Where:
> Disk-allocated -> space used on the disk by the chunk
> Disk-size -> size of the disk
> Disk-unallocated -> disk not used in any chunk
> Used -> space used by the files/metadata
The problem here is that if you're using raw storage, the Used
value in the second stanza grows twice as fast as the user expects. I
think this second stanza should at minimum include the "cooked" values
used in btrfs fi df, because those reflect the user's experience. Then
adding [some of?] the raw values you've got here to help connect the
values to the raw data in the first stanza of output.
As I said above, it's the connection between "I wrote a 1GiB file
to my filesystem" and "why have my numbers increased/decreased by
2GiB(*)/1.2GiB(**)?"
(*) RAID-1
(**) RAID-5-ish
> Available -> space available in the *allocated* chunk
> Free_(Estimated) -> Theoretical free space for files (Disk_size
> * Average_disk_efficiency - Used)
> >
> > Ultimately, I think the bikeshed should be turquoise.
> ? :-)
http://www.freebsd.org/doc/en_US.ISO8859-1/books/faq/misc.html#bikeshed-painting
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- And what rough beast, its hour come round at last / slouches ---
towards Bethlehem, to be born?
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-28 18:02 ` Roman Mamedov
2012-09-28 19:38 ` Goffredo Baroncelli
@ 2012-09-28 20:20 ` Hugo Mills
2012-09-28 21:26 ` Wade Cline
1 sibling, 1 reply; 21+ messages in thread
From: Hugo Mills @ 2012-09-28 20:20 UTC (permalink / raw)
To: Roman Mamedov; +Cc: kreijack, kreijack, Sébastien Maury, linux-btrfs
[-- Attachment #1: Type: text/plain, Size: 3468 bytes --]
On Sat, Sep 29, 2012 at 12:02:23AM +0600, Roman Mamedov wrote:
> On Fri, 28 Sep 2012 18:44:07 +0200
> Goffredo Baroncelli <kreijack@inwind.it> wrote:
>
> > This means that the ration of space physically allocated on the disk and
> > the space available is 7GB/10GB = 0.7 . So on 135GB of disk, only 94GB
> > are available.
>
> You assume metadata allocation will always grow linearly with data, which is
> not true. So in my opinion it is not a good estimate.
No, but it's the best model we have right now. (And probably about
the best model we will have, without knowledge of the future
intentions of the user). Without inlining file data, the metadata is
dominated by checksums, which is a linear relationship (approx
1000:1). With inlining file data, metadata is probably dominated by
inline data; assuming the ratio of small-to-large files on the FS
remains unchanged in future, a linear relationship also applies. For
general usage, I'm happy to assume that the current ratio of data to
metadata will remain largely unchanged over the lifetime of the FS.
> > > Why use underscores instead of spaces?
> >
> > Simplify the parsing in scripts
>
> I think it looks awkward and is not warranted since this is a primarily
> user-facing utility. Also none of the other similar tools shy from having
> spaces anywhere they need to, e.g.
>
> # mdadm --detail /dev/md0
> /dev/md0:
> Version : 1.2
> Creation Time : Wed May 25 00:07:38 2011
> Raid Level : raid5
> Array Size : 3907003136 (3726.01 GiB 4000.77 GB)
> Used Dev Size : 976750784 (931.50 GiB 1000.19 GB)
> Raid Devices : 5
> Total Devices : 5
> Persistence : Superblock is persistent
>
> Intent Bitmap : Internal
>
> Update Time : Fri Sep 28 21:20:51 2012
> State : active
> Active Devices : 5
> Working Devices : 5
> Failed Devices : 0
> Spare Devices : 0
>
> Layout : left-symmetric
> Chunk Size : 64K
>
> Name : avdeb:0 (local to host avdeb)
> UUID : b99961fb:ed1f76c8:ec2dad31:6db45332
> Events : 14254
>
> Number Major Minor RaidDevice State
> 7 8 17 0 active sync /dev/sdb1
> 6 8 33 1 active sync /dev/sdc1
> 3 8 65 2 active sync /dev/sde1
> 4 8 49 3 active sync /dev/sdd1
> 5 8 81 4 active sync /dev/sdf1
>
> # lvdisplay
> --- Logical volume ---
> LV Path /dev/alpha/lv1
> LV Name lv1
> VG Name alpha
> LV UUID HP19fU-oMhM-sdqN-yFWa-N3Rs-ktBw-21GSD2
> LV Write Access read/write
> LV Creation host, time ,
> LV Status available
> # open 0
> LV Size 3.52 TiB
> Current LE 115431
> Segments 3
> Allocation inherit
> Read ahead sectors auto
> - currently set to 4096
> Block device 252:0
... and I've always found those hard to deal with in scripts. :)
(But they do have "plumbing" options, to use the git terminology,
so I'd be happy with having a parsable output option).
Hugo.
--
=== Hugo Mills: hugo@... carfax.org.uk | darksatanic.net | lug.org.uk ===
PGP key: 515C238D from wwwkeys.eu.pgp.net or http://www.carfax.org.uk
--- Hey, Virtual Memory! Now I can have a *really big* ramdisk! ---
[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 828 bytes --]
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-28 20:20 ` Hugo Mills
@ 2012-09-28 21:26 ` Wade Cline
0 siblings, 0 replies; 21+ messages in thread
From: Wade Cline @ 2012-09-28 21:26 UTC (permalink / raw)
To: Hugo Mills, Roman Mamedov, kreijack, kreijack,
Sébastien Maury, linux-btrfs
On 09/28/2012 01:20 PM, Hugo Mills wrote:
> On Sat, Sep 29, 2012 at 12:02:23AM +0600, Roman Mamedov wrote:
>> On Fri, 28 Sep 2012 18:44:07 +0200
>> Goffredo Baroncelli<kreijack@inwind.it> wrote:
>>
>>> This means that the ration of space physically allocated on the disk and
>>> the space available is 7GB/10GB = 0.7 . So on 135GB of disk, only 94GB
>>> are available.
>> You assume metadata allocation will always grow linearly with data, which is
>> not true. So in my opinion it is not a good estimate.
> No, but it's the best model we have right now. (And probably about
> the best model we will have, without knowledge of the future
> intentions of the user). Without inlining file data, the metadata is
> dominated by checksums, which is a linear relationship (approx
> 1000:1). With inlining file data, metadata is probably dominated by
> inline data; assuming the ratio of small-to-large files on the FS
> remains unchanged in future, a linear relationship also applies. For
> general usage, I'm happy to assume that the current ratio of data to
> metadata will remain largely unchanged over the lifetime of the FS.
Since there really isn't a simple answer to how much free-space,
why not have the command print an upper and lower estimate and let
the user figure out how to interpret the numbers? This would inform
the user that there is some guesswork inherent in the estimation and
also provide an educated user with more exact numbers. Something
containing information such as:
Total: 135.00 GiB
Allocated: 10.51 GiB
Unallocated: 124.49 GiB
Free_Upper_Est: 130.00 GiB
Free_Lower_Est: 62.45 GiB
The main idea is that an informed user would know that the
upper-estimation would be for only writing, say, new data, while
the lower-estimation would be for writing everything in, say, a
RAID-1 subvolume. An uninformed user would (hopefully) realize
that he needs to read the Wiki's FAQ.
> ... and I've always found those hard to deal with in scripts. :)
>
> (But they do have "plumbing" options, to use the git terminology,
> so I'd be happy with having a parsable output option).
>
> Hugo.
>
In 'df'/'du', -h is used for human-readable output while no options
is for easily parsable output.
Basically, I think that the bikeshed should be green. ;)
-Wade
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-28 20:13 ` Hugo Mills
@ 2012-09-28 21:26 ` Goffredo Baroncelli
2012-09-29 7:19 ` Goffredo Baroncelli
1 sibling, 0 replies; 21+ messages in thread
From: Goffredo Baroncelli @ 2012-09-28 21:26 UTC (permalink / raw)
To: Hugo Mills, Roman Mamedov, Sébastien Maury, linux-btrfs
On 09/28/2012 10:13 PM, Hugo Mills wrote:
>> Summary:
>> > Disk_size: 135.00 GiB
>> > Disk_allocated: 10.51 GiB
>> > Disk_unallocated: 124.49 GiB
>> > Used: 2.59 GiB
>> > Free_(Estimated): 91.93 GiB
>> > Average_disk_efficiency: 70 %
>> >
>> > Details:
>> > Chunk-type Mode Disk-allocated Used Available
>> > Data Single 4.01GB 2.16GB 1.87GB
>> > System DUP 16.00MB 4.00KB 7.99MB
>> > System Single 4.00MB 0.00 4.00MB
>> > Metadata DUP 6.00GB 429.16MB 2.57GB
>> > Metadata Single 8.00MB 0.00 8.00MB
>> >
>> >
>> >
>> > Where:
>> > Disk-allocated -> space used on the disk by the chunk
>> > Disk-size -> size of the disk
>> > Disk-unallocated -> disk not used in any chunk
>> > Used -> space used by the files/metadata
> The problem here is that if you're using raw storage, the Used
> value in the second stanza grows twice as fast as the user expects.
This is the misunderstanding whom I talked before.
If you give a look at the line "Metadata DUP", you can see that the
disk-allocated are about 6GB, instead if you sum Used and Available you
got 3GB.
I.e. if you create a 1GB file, "Used" ever increased of 1GB, and
Available ever decrease 1GB, whichever you are using DUP or Single or RAID*
I
> think this second stanza should at minimum include the "cooked" values
> used in btrfs fi df, because those reflect the user's experience. Then
> adding [some of?] the raw values you've got here to help connect the
> values to the raw data in the first stanza of output.
The only raw values are the one "prefixed" with disk. The other ones are
at the net of the DUP/Single/Raid....
>
> As I said above, it's the connection between "I wrote a 1GiB file
> to my filesystem" and "why have my numbers increased/decreased by
> 2GiB(*)/1.2GiB(**)?"
I repeat, if the chunk is DUP-ed, if you create 1GB file:
- Disk-allocate increase 2GB (supposing that all the chunks are full)
- Used increase 1GB
- Available decrease 1GB
>
> (*) RAID-1
> (**) RAID-5-ish
>
Ciao
Goffredo
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-28 20:13 ` Hugo Mills
2012-09-28 21:26 ` Goffredo Baroncelli
@ 2012-09-29 7:19 ` Goffredo Baroncelli
2012-09-29 9:59 ` Sébastien Maury
1 sibling, 1 reply; 21+ messages in thread
From: Goffredo Baroncelli @ 2012-09-29 7:19 UTC (permalink / raw)
To: Hugo Mills, Roman Mamedov, Sébastien Maury, linux-btrfs
On 09/28/2012 10:13 PM, Hugo Mills wrote:
>> Summary:
>> > Disk_size: 135.00 GiB
>> > Disk_allocated: 10.51 GiB
>> > Disk_unallocated: 124.49 GiB
>> > Used: 2.59 GiB
>> > Free_(Estimated): 91.93 GiB
>> > Average_disk_efficiency: 70 %
>> >
>> > Details:
>> > Chunk-type Mode Disk-allocated Used Available
>> > Data Single 4.01GB 2.16GB 1.87GB
>> > System DUP 16.00MB 4.00KB 7.99MB
>> > System Single 4.00MB 0.00 4.00MB
>> > Metadata DUP 6.00GB 429.16MB 2.57GB
>> > Metadata Single 8.00MB 0.00 8.00MB
>> >
>> >
>> >
>> > Where:
>> > Disk-allocated -> space used on the disk by the chunk
>> > Disk-size -> size of the disk
>> > Disk-unallocated -> disk not used in any chunk
>> > Used -> space used by the files/metadata
> The problem here is that if you're using raw storage, the Used
> value in the second stanza grows twice as fast as the user expects.
This is the misunderstanding whom I talked before.
If you give a look at the line "Metadata DUP", you can see that the
disk-allocated are about 6GB, instead if you sum Used and Available you
got 3GB.
I.e. if you create a 1GB file, "Used" ever increased of 1GB, and
Available ever decrease 1GB, whichever you are using DUP or Single or RAID*
I
> think this second stanza should at minimum include the "cooked" values
> used in btrfs fi df, because those reflect the user's experience. Then
> adding [some of?] the raw values you've got here to help connect the
> values to the raw data in the first stanza of output.
The only raw values are the one "prefixed" with disk. The other ones are
at the net of the DUP/Single/Raid....
>
> As I said above, it's the connection between "I wrote a 1GiB file
> to my filesystem" and "why have my numbers increased/decreased by
> 2GiB(*)/1.2GiB(**)?"
I repeat, if the chunk is DUP-ed, if you create 1GB file:
- Disk-allocate increase 2GB (supposing that all the chunks are full)
- Used increase 1GB
- Available decrease 1GB
>
> (*) RAID-1
> (**) RAID-5-ish
>
Ciao
Goffredo
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-29 7:19 ` Goffredo Baroncelli
@ 2012-09-29 9:59 ` Sébastien Maury
2012-09-29 11:51 ` Goffredo Baroncelli
0 siblings, 1 reply; 21+ messages in thread
From: Sébastien Maury @ 2012-09-29 9:59 UTC (permalink / raw)
To: Goffredo Baroncelli, Hugo Mills, Roman Mamedov
Cc: linux-btrfs, sebastien.maury
Hi,
First of all, i've to say that i'm not a linux specialist, so that
means my point of view is balanced between a linux admin and a user.
I may also say "stupid" things, so pleas excuse me in advance :p
The first difference between the original command and the discussed
one is on the value for the DUP parts (one has to be multiplied by 2,
whereas the other is already multiplied by 2).
I think this should be indicated somewhere in order to avoid confusion.
This has been pointed already, but whatever the output is, it is
essential to know if the value is raw or not, if it has to be
multiplied or divided.
Also, i do agree with Hugo concerning the output to make it easier to
parse through scripting.
The units should also be settable in order to have the same units for
all values.
Basically, this new output is more explicit for me and remove a bit of
confusion.
Although, the part "Average_disk_efficiency" seems confusing as i'm
not sure the term "efficiency" is correct in that part.
That makes me ask some questions : why this much allocated ? when will
it allocate more ? how much might be allocated ? ...
So, this percentage doesn't indicate an efficient usage of disk space
or not ... for me, it indicates that it needed to allocated that
(depending on the chunk size).
In this example there's indeed 30% of the allocation that is unused,
but it will be used as data will grow on the disk.
For me it's similar as a LUN created in thick provisioning ... i might
not need all the space, but i don't want to be stuck if i'll need it.
(dunno if i'm clear on that part)
Am i wrong in saying that "Free_(Estimated)" is a false value as the
snapshots size isn't included ?
Let's say i've like 10 GB of snapshots ... then
Free_(Estimated)=Free_(Estimated)-snaps size ? no ?
Is it possible to include those snaps size somewhere (maybe not to
include in the summary or details, but to add another section or
option allowing to have that info) ?
Finally, i do agree about the linearly growth as the best model currently.
For several reasons, some already explained by Hugo, and because as
far as i understood, there is no "single" way to know very accurately
how your disk is used. That said, the point is at least to give the
most accurate data as possible and to be able to interpret them.
In a production environment, i can't afford to say "sorry, the app is
crashed because my disk is full". So i need a view on what's happening
on my disk.
Even if it lacks perfect accuracy, i can place thresholds to avoid any
problem (70% of disk full as a warning for example).
So, i would change some terms i guess indicating more precisely the
"raw" data and the already computed ones.
I would also not use the term efficiency as people may wonder at some
point if they didn't make a mistake using btrfs seeing a % never near
from 100.
The "Data_to_disk_ratio" seems preferable for me.
Cordialement,
Sébastien
Goffredo Baroncelli <kreijack@gmail.com> a écrit :
> On 09/28/2012 10:13 PM, Hugo Mills wrote:
>>> Summary:
>>>> Disk_size: 135.00 GiB
>>>> Disk_allocated: 10.51 GiB
>>>> Disk_unallocated: 124.49 GiB
>>>> Used: 2.59 GiB
>>>> Free_(Estimated): 91.93 GiB
>>>> Average_disk_efficiency: 70 %
>>>>
>>>> Details:
>>>> Chunk-type Mode Disk-allocated Used Available
>>>> Data Single 4.01GB 2.16GB 1.87GB
>>>> System DUP 16.00MB 4.00KB 7.99MB
>>>> System Single 4.00MB 0.00 4.00MB
>>>> Metadata DUP 6.00GB 429.16MB 2.57GB
>>>> Metadata Single 8.00MB 0.00 8.00MB
>>>>
>>>>
>>>>
>>>> Where:
>>>> Disk-allocated -> space used on the disk by the chunk
>>>> Disk-size -> size of the disk
>>>> Disk-unallocated -> disk not used in any chunk
>>>> Used -> space used by the files/metadata
>> The problem here is that if you're using raw storage, the Used
>> value in the second stanza grows twice as fast as the user expects.
>
> This is the misunderstanding whom I talked before.
>
> If you give a look at the line "Metadata DUP", you can see that the
> disk-allocated are about 6GB, instead if you sum Used and Available you
> got 3GB.
>
> I.e. if you create a 1GB file, "Used" ever increased of 1GB, and
> Available ever decrease 1GB, whichever you are using DUP or Single or
> RAID*
>
>
> I
>> think this second stanza should at minimum include the "cooked" values
>> used in btrfs fi df, because those reflect the user's experience. Then
>> adding [some of?] the raw values you've got here to help connect the
>> values to the raw data in the first stanza of output.
>
> The only raw values are the one "prefixed" with disk. The other ones
> are at the net of the DUP/Single/Raid....
>
>>
>> As I said above, it's the connection between "I wrote a 1GiB file
>> to my filesystem" and "why have my numbers increased/decreased by
>> 2GiB(*)/1.2GiB(**)?"
>
> I repeat, if the chunk is DUP-ed, if you create 1GB file:
> - Disk-allocate increase 2GB (supposing that all the chunks are full)
> - Used increase 1GB
> - Available decrease 1GB
>
>
>>
>> (*) RAID-1
>> (**) RAID-5-ish
>>
> Ciao
> Goffredo
----------------------------------------------------------------
This message was sent using IMP, the Internet Messaging Program.
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-29 9:59 ` Sébastien Maury
@ 2012-09-29 11:51 ` Goffredo Baroncelli
0 siblings, 0 replies; 21+ messages in thread
From: Goffredo Baroncelli @ 2012-09-29 11:51 UTC (permalink / raw)
To: Sébastien Maury; +Cc: Hugo Mills, Roman Mamedov, linux-btrfs, Wade Cline
Hi Sébastien,
On 09/29/2012 11:59 AM, Sébastien Maury wrote:
> Hi,
>
> First of all, i've to say that i'm not a linux specialist, so that means
> my point of view is balanced between a linux admin and a user.
> I may also say "stupid" things, so pleas excuse me in advance :p
>
> The first difference between the original command and the discussed one
> is on the value for the DUP parts (one has to be multiplied by 2,
> whereas the other is already multiplied by 2).
> I think this should be indicated somewhere in order to avoid confusion.
> This has been pointed already, but whatever the output is, it is
> essential to know if the value is raw or not, if it has to be multiplied
> or divided.
> Also, i do agree with Hugo concerning the output to make it easier to
> parse through scripting.
> The units should also be settable in order to have the same units for
> all values.
I have added a "-k" switch, so the output is in KiB unit (I tried bytes
but so the line will became very long: 1<<64 is about 20 digits in
decimal form)
> Basically, this new output is more explicit for me and remove a bit of
> confusion.
Great I reached my 1st goal !
>
> Although, the part "Average_disk_efficiency" seems confusing as i'm not
> sure the term "efficiency" is correct in that part.
> That makes me ask some questions : why this much allocated ? when will
> it allocate more ? how much might be allocated ? ...
> So, this percentage doesn't indicate an efficient usage of disk space or
> not ... for me, it indicates that it needed to allocated that (depending
> on the chunk size).
> In this example there's indeed 30% of the allocation that is unused, but
> it will be used as data will grow on the disk.
The 30% of the disk is/will be used for redundancy purpose. Moreover
there are the chunk that are "pre-allocated" area, which could influence
the free space estimation...
> For me it's similar as a LUN created in thick provisioning ... i might
> not need all the space, but i don't want to be stuck if i'll need it.
> (dunno if i'm clear on that part)
>
> Am i wrong in saying that "Free_(Estimated)" is a false value as the
> snapshots size isn't included ?
> Let's say i've like 10 GB of snapshots ... then
> Free_(Estimated)=Free_(Estimated)-snaps size ? no ?
> Is it possible to include those snaps size somewhere (maybe not to
> include in the summary or details, but to add another section or option
> allowing to have that info) ?
Free_(Estimated) takes in account also the snapshot. The point is
another one: the user has to know that updating (i.e. changing part of
file without increasing its size) a snapshoted file requires space. But
Used part takes in account all the space used. So Free_(Estimated) it is
accurate.
> Finally, i do agree about the linearly growth as the best model currently.
> For several reasons, some already explained by Hugo, and because as far
> as i understood, there is no "single" way to know very accurately how
> your disk is used. That said, the point is at least to give the most
> accurate data as possible and to be able to interpret them.
> In a production environment, i can't afford to say "sorry, the app is
> crashed because my disk is full". So i need a view on what's happening
> on my disk.
> Even if it lacks perfect accuracy, i can place thresholds to avoid any
> problem (70% of disk full as a warning for example).
>
> So, i would change some terms i guess indicating more precisely the
> "raw" data and the already computed ones.
I would like to uses the "Disk" prefix. "Raw" to me creates more
confusions. However we should highlight that the disk occupation is
related to the chunks, which means basically a "pre-allocation" and not
an "using". For example a my filesystem has:
ghigo@venice:~$ btrfs/btrfs-progs/btrfs fi disk /mnt/old-btrfs/
Summary:
Path: /mnt/old-btrfs/
Disk_size: 232.11GB
Disk_allocated: 150.29GB
Disk_unallocated: 81.82GB
Used: 19.94GB
Free_(Estimated): 201.16GB
Average_disk_efficiency: 95 %
Details:
Chunk-type Mode Disk-allocated Used Available
Data Single 136.01GB 18.84GB 117.17GB
System DUP 16.00MB 28.00KB 7.97MB
System Single 4.00MB 0.00 4.00MB
Metadata DUP 14.25GB 1.10GB 6.03GB
Metadata Single 8.00MB 0.00 8.00MB
Note that I have 136GB of chunk, but only 18GB are used.
After a "btrfs balance start" I got a different picture:
Summary:
Path: /mnt/old-btrfs/
Disk_size: 232.11GB
Disk_allocated: 34.13GB
Disk_unallocated: 197.98GB
Used: 19.94GB
Free_(Estimated): 177.74GB
Average_disk_efficiency: 85 %
Details:
Chunk-type Mode Disk-allocated Used Available
Data Single 24.00GB 18.84GB 5.16GB
System DUP 128.00MB 4.00KB 64.00MB
System Single 4.00MB 0.00 4.00MB
Metadata DUP 10.00GB 1.10GB 3.90GB
The allocated chunk decreases, this impacts also on
Average_disk_efficiency (or data_to_disk_ratio).
> I would also not use the term efficiency as people may wonder at some
> point if they didn't make a mistake using btrfs seeing a % never near
> from 100.
> The "Data_to_disk_ratio" seems preferable for me.
I like your idea; however I would also like the Wade's suggestion to
show also the max and the min
>
> Cordialement,
>
> Sébastien
>
> Goffredo Baroncelli <kreijack@gmail.com> a écrit :
>
>> On 09/28/2012 10:13 PM, Hugo Mills wrote:
>>>> Summary:
>>>>> Disk_size: 135.00 GiB
>>>>> Disk_allocated: 10.51 GiB
>>>>> Disk_unallocated: 124.49 GiB
>>>>> Used: 2.59 GiB
>>>>> Free_(Estimated): 91.93 GiB
>>>>> Average_disk_efficiency: 70 %
>>>>>
>>>>> Details:
>>>>> Chunk-type Mode Disk-allocated Used Available
>>>>> Data Single 4.01GB 2.16GB 1.87GB
>>>>> System DUP 16.00MB 4.00KB 7.99MB
>>>>> System Single 4.00MB 0.00 4.00MB
>>>>> Metadata DUP 6.00GB 429.16MB 2.57GB
>>>>> Metadata Single 8.00MB 0.00 8.00MB
>>>>>
>>>>>
>>>>>
>>>>> Where:
>>>>> Disk-allocated -> space used on the disk by the chunk
>>>>> Disk-size -> size of the disk
>>>>> Disk-unallocated -> disk not used in any chunk
>>>>> Used -> space used by the files/metadata
>>> The problem here is that if you're using raw storage, the Used
>>> value in the second stanza grows twice as fast as the user expects.
>>
>> This is the misunderstanding whom I talked before.
>>
>> If you give a look at the line "Metadata DUP", you can see that the
>> disk-allocated are about 6GB, instead if you sum Used and Available you
>> got 3GB.
>>
>> I.e. if you create a 1GB file, "Used" ever increased of 1GB, and
>> Available ever decrease 1GB, whichever you are using DUP or Single or
>> RAID*
>>
>>
>> I
>>> think this second stanza should at minimum include the "cooked" values
>>> used in btrfs fi df, because those reflect the user's experience. Then
>>> adding [some of?] the raw values you've got here to help connect the
>>> values to the raw data in the first stanza of output.
>>
>> The only raw values are the one "prefixed" with disk. The other ones
>> are at the net of the DUP/Single/Raid....
>>
>>>
>>> As I said above, it's the connection between "I wrote a 1GiB file
>>> to my filesystem" and "why have my numbers increased/decreased by
>>> 2GiB(*)/1.2GiB(**)?"
>>
>> I repeat, if the chunk is DUP-ed, if you create 1GB file:
>> - Disk-allocate increase 2GB (supposing that all the chunks are full)
>> - Used increase 1GB
>> - Available decrease 1GB
>>
>>
>>>
>>> (*) RAID-1
>>> (**) RAID-5-ish
>>>
>> Ciao
>> Goffredo
>
>
> ----------------------------------------------------------------
> This message was sent using IMP, the Internet Messaging Program.
>
>
>
^ permalink raw reply [flat|nested] 21+ messages in thread
* Re: [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage]
2012-09-28 8:58 ` Hugo Mills
2012-09-28 17:27 ` Goffredo Baroncelli
@ 2012-11-12 18:16 ` Jan Engelhardt
1 sibling, 0 replies; 21+ messages in thread
From: Jan Engelhardt @ 2012-11-12 18:16 UTC (permalink / raw)
To: Hugo Mills
Cc: Roman Mamedov, kreijack, kreijack, Sébastien Maury,
linux-btrfs
On Friday 2012-09-28 10:58, Hugo Mills wrote:
>
> Data_to_disk_ratio, maybe?
>
>> Why use underscores instead of spaces?
>
> So that you can use, say, "read" in the shell to extract data from
>each line. To that end, there should be a space between the value and
>the unit throughout.
Eww. Having a special single-line output mode would be much better
for these kinds of integration.
Is it too far fetched to make the info available through sysfs?
space_used=$(cat /sys/.../space_used)
is so much preferable than an awkful
space_used=$(btrfs fi df | awk ...)
and hope for that the line is actually in the df output.
^ permalink raw reply [flat|nested] 21+ messages in thread
end of thread, other threads:[~2012-11-12 18:16 UTC | newest]
Thread overview: 21+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2012-09-27 10:44 BTRF - Storage Usage Sébastien Maury
2012-09-27 11:09 ` Hugo Mills
2012-09-27 11:25 ` Sébastien Maury
2012-09-27 11:43 ` Hugo Mills
2012-09-27 11:52 ` Sébastien Maury
2012-09-27 20:39 ` [RFC] btrfs fi df output [Was Re: BTRF - Storage Usage] Goffredo Baroncelli
2012-09-27 21:02 ` Goffredo Baroncelli
2012-09-28 3:17 ` Roman Mamedov
2012-09-28 8:58 ` Hugo Mills
2012-09-28 17:27 ` Goffredo Baroncelli
2012-09-28 20:13 ` Hugo Mills
2012-09-28 21:26 ` Goffredo Baroncelli
2012-09-29 7:19 ` Goffredo Baroncelli
2012-09-29 9:59 ` Sébastien Maury
2012-09-29 11:51 ` Goffredo Baroncelli
2012-11-12 18:16 ` Jan Engelhardt
2012-09-28 16:44 ` Goffredo Baroncelli
2012-09-28 18:02 ` Roman Mamedov
2012-09-28 19:38 ` Goffredo Baroncelli
2012-09-28 20:20 ` Hugo Mills
2012-09-28 21:26 ` Wade Cline
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).