How does "btrfs fi show" show full?

How does "btrfs fi show" show full?

[Marc MERLIN]

More slides, more questions, sorry :)
(thanks for the other answers, I'm still going through them)

If I have:
gandalfthegreat:~# btrfs fi show
Label: 'btrfs_pool1'  uuid: 873d526c-e911-4234-af1b-239889cd143d
	Total devices 1 FS bytes used 214.44GB
	devid    1 size 231.02GB used 231.02GB path /dev/dm-0

I'm a bit confused.

It tells me
1) FS uses 214GB out of 231GB
2) Device uses 231GB out of 231GB

I understand how the device can use less than the FS if you have
multiple devices that share a filesystem.
But I'm not sure how a filesystem can use less than what's being used on
a single device.

Similarly, my current laptop shows:
legolas:~# btrfs fi show
Label: btrfs_pool1  uuid: 4850ee22-bf32-4131-a841-02abdb4a5ba6
	Total devices 1 FS bytes used 442.17GiB
	devid    1 size 865.01GiB used 751.04GiB path /dev/mapper/cryptroot

So, am I 100GB from being full, or am I really only using 442GB out of 865GB?

If so, what does the device used value really mean if it can be that
much higher than the filesystem used value?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: How does "btrfs fi show" show full?

[Brendan Hide]

On 2014/05/05 02:54 AM, Marc MERLIN wrote:
> More slides, more questions, sorry :)
> (thanks for the other answers, I'm still going through them)
>
> If I have:
> gandalfthegreat:~# btrfs fi show
> Label: 'btrfs_pool1'  uuid: 873d526c-e911-4234-af1b-239889cd143d
> 	Total devices 1 FS bytes used 214.44GB
> 	devid    1 size 231.02GB used 231.02GB path /dev/dm-0
>
> I'm a bit confused.
>
> It tells me
> 1) FS uses 214GB out of 231GB
> 2) Device uses 231GB out of 231GB
>
> I understand how the device can use less than the FS if you have
> multiple devices that share a filesystem.
> But I'm not sure how a filesystem can use less than what's being used on
> a single device.
>
> Similarly, my current laptop shows:
> legolas:~# btrfs fi show
> Label: btrfs_pool1  uuid: 4850ee22-bf32-4131-a841-02abdb4a5ba6
> 	Total devices 1 FS bytes used 442.17GiB
> 	devid    1 size 865.01GiB used 751.04GiB path /dev/mapper/cryptroot
>
> So, am I 100GB from being full, or am I really only using 442GB out of 865GB?
>
> If so, what does the device used value really mean if it can be that
> much higher than the filesystem used value?
>
> Thanks,
> Marc
The "per-device" used amount refers to the amount of space that has been 
allocated to chunks. That first one probably needs a balance. Btrfs 
doesn't behave very well when available diskspace is so low due to the 
fact that it cannot allocate any new chunks. An attempt to allocate a 
new chunk will result in ENOSPC errors.

The "Total" bytes used refers to the total actual data that is stored.

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: How does "btrfs fi show" show full?

[Marc MERLIN]

On Mon, May 05, 2014 at 06:11:28AM +0200, Brendan Hide wrote:
> The "per-device" used amount refers to the amount of space that has
> been allocated to chunks. That first one probably needs a balance.
> Btrfs doesn't behave very well when available diskspace is so low
> due to the fact that it cannot allocate any new chunks. An attempt
> to allocate a new chunk will result in ENOSPC errors.
> 
> The "Total" bytes used refers to the total actual data that is stored.

Right. So 'Total used' is what I'm really using, whereas 'devid used' is
actually what is being used due to the way btrfs doesn't seem to reclaim
chunks after they're not used anymore, or some similar problem.

In the second FS:
> >Label: btrfs_pool1  uuid: 4850ee22-bf32-4131-a841-02abdb4a5ba6
> >	Total devices 1 FS bytes used 442.17GiB
> >	devid    1 size 865.01GiB used 751.04GiB path /dev/mapper/cryptroot

The difference is huge between 'Total used' and 'devid used'.

Is btrfs going to fix this on its own, or likely not and I'm stuck doing
a full balance (without filters since I'm balancing data and not
metadata)?

If that helps.
legolas:~# btrfs fi df /mnt/btrfs_pool1
Data, single: total=734.01GiB, used=435.29GiB
System, DUP: total=8.00MiB, used=96.00KiB
System, single: total=4.00MiB, used=0.00
Metadata, DUP: total=8.50GiB, used=6.74GiB
Metadata, single: total=8.00MiB, used=0.00

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: How does "btrfs fi show" show full?

[Brendan Hide]

On 05/05/14 07:50, Marc MERLIN wrote:
> On Mon, May 05, 2014 at 06:11:28AM +0200, Brendan Hide wrote:
>> The "per-device" used amount refers to the amount of space that has
>> been allocated to chunks. That first one probably needs a balance.
>> Btrfs doesn't behave very well when available diskspace is so low
>> due to the fact that it cannot allocate any new chunks. An attempt
>> to allocate a new chunk will result in ENOSPC errors.
>>
>> The "Total" bytes used refers to the total actual data that is stored.
> Right. So 'Total used' is what I'm really using, whereas 'devid used' is
> actually what is being used due to the way btrfs doesn't seem to reclaim
> chunks after they're not used anymore, or some similar problem.
>
> In the second FS:
>>> Label: btrfs_pool1  uuid: 4850ee22-bf32-4131-a841-02abdb4a5ba6
>>> 	Total devices 1 FS bytes used 442.17GiB
>>> 	devid    1 size 865.01GiB used 751.04GiB path /dev/mapper/cryptroot
> The difference is huge between 'Total used' and 'devid used'.
>
> Is btrfs going to fix this on its own, or likely not and I'm stuck doing
> a full balance (without filters since I'm balancing data and not
> metadata)?
>
> If that helps.
> legolas:~# btrfs fi df /mnt/btrfs_pool1
> Data, single: total=734.01GiB, used=435.29GiB
> System, DUP: total=8.00MiB, used=96.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, DUP: total=8.50GiB, used=6.74GiB
> Metadata, single: total=8.00MiB, used=0.00
>
> Thanks,
> Marc
What I typically do in snapshot cleanup scripts is to use the usage= 
filter with the percentage being a simple algorithm of "halfway between 
actual data and maximum chunk allocation I'm comfortable with". As an 
example, one of my servers' diskspace was at 65% and last night the 
chunk allocation reached the 90% mark. So it automatically ran the 
balance with "dusage=77". This would have cleared out about half of the 
"unnecessary" chunks.

Balance causes a lot of load with spinning rust - of course, 
after-hours, nobody really cares. With SSDs it causes wear. This is just 
one method I felt was a sensible way for me to avoid ENOSPC issues while 
also ensuring I'm not rebalancing the entire system unnecessarily.

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: How does "btrfs fi show" show full?

[Duncan]

Marc MERLIN posted on Sun, 04 May 2014 22:50:29 -0700 as excerpted:

> In the second FS:
>> >Label: btrfs_pool1  uuid: [...]
>> >	Total devices 1 FS bytes used 442.17GiB
>> >	devid    1 size 865.01GiB used 751.04GiB path [...]
> 
> The difference is huge between 'Total used' and 'devid used'.
> 
> Is btrfs going to fix this on its own, or likely not and I'm stuck doing
> a full balance (without filters since I'm balancing data and not
> metadata)?
> 
> If that helps.
> legolas:~# btrfs fi df /mnt/btrfs_pool1
> Data, single: total=734.01GiB, used=435.29GiB
> System, DUP: total=8.00MiB, used=96.00KiB
> System, single: total=4.00MiB, used=0.00
> Metadata, DUP: total=8.50GiB, used=6.74GiB
> Metadata, single: total=8.00MiB, used=0.00

Definitely helps.  The spread is in data.

Try

btrfs balance start -dusage=20 /mnt/btrfs_pool1

You still have plenty of unused (if allocated) space available, so you 
can play around with the usage= a bit.  -dusage=20 will be faster than 
something like -dusage=50 or -dusage=80, likely MUCH faster, but will 
return less chunks to unallocated, as well.  Still, your spread between 
data-total and data-used is high enough, I expect -dusage=20 will give 
you pretty good results.

Since show says you still have ~100 GiB unallocated in df, there's no 
real urgency, and again I'd try -dusage=20 the first time.  If that 
doesn't cut it you can of course try bumping the usage= as needed, but 
because you still have 100 GiB unallocated and because the data used vs. 
total spread is so big, I really do think -dusage=20 will do it for you.
As your actual device usage goes up the spread between used and size will 
go down, meaning more frequent balances to keep some reasonable 
unallocated space available, and you'll either need to actually delete 
some stuff or to bump up those usage= numbers as well, but usage=20 is 
very likely to be sufficient at this point.

I hadn't seen anyone try an actual formula as Brendan suggests in his 
post, and I'm not actually sure that formula will apply well in all use-
cases as I think fragmentation and fill-pattern will have a lot to do 
with it, but based on his post it does apply for his use-case, and the 
same general principle if not the specific formula should apply 
everywhere and is what I'm doing above, only simply eyeballing it, not 
using a specific formula.

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman

Re: How does "btrfs fi show" show full?

[Marc MERLIN]

On Tue, May 06, 2014 at 08:10:00PM +0000, Duncan wrote:
> Marc MERLIN posted on Sun, 04 May 2014 22:50:29 -0700 as excerpted:
> 
> > In the second FS:
> >> >Label: btrfs_pool1  uuid: [...]
> >> >	Total devices 1 FS bytes used 442.17GiB
> >> >	devid    1 size 865.01GiB used 751.04GiB path [...]
> > 
> > The difference is huge between 'Total used' and 'devid used'.
> > 
> > Is btrfs going to fix this on its own, or likely not and I'm stuck doing
> > a full balance (without filters since I'm balancing data and not
> > metadata)?
> > 
> > If that helps.
> > legolas:~# btrfs fi df /mnt/btrfs_pool1
> > Data, single: total=734.01GiB, used=435.29GiB
> > System, DUP: total=8.00MiB, used=96.00KiB
> > System, single: total=4.00MiB, used=0.00
> > Metadata, DUP: total=8.50GiB, used=6.74GiB
> > Metadata, single: total=8.00MiB, used=0.00
> 
> Definitely helps.  The spread is in data.
> 
> Try
> 
> btrfs balance start -dusage=20 /mnt/btrfs_pool1

So, I had already tried -dusage=50 yesterday, and I'm now reasonable:
Label: btrfs_pool1  uuid: 4850ee22-bf32-4131-a841-02abdb4a5ba6
	Total devices 1 FS bytes used 443.22GiB
	devid    1 size 865.01GiB used 514.04GiB path /dev/mapper/cryptroot


> something like -dusage=50 or -dusage=80, likely MUCH faster, but will 
> return less chunks to unallocated, as well.  Still, your spread between 

(fewer)

> data-total and data-used is high enough, I expect -dusage=20 will give 
> you pretty good results.

So, on
On http://marc.merlins.org/perso/btrfs/post_2014-05-04_Fixing-Btrfs-Filesystem-Full-Problems.html
I wrote
In the case above, because the filesystem is only 55% full, I can ask
balance to rewrite all chunks that are more than 55% full:
legolas:~# btrfs balance start -dusage=55 /mnt/btrfs_pool1

Did I get this right?
I'm not sure I did, since it seems the bigger the -dusage number, the
more work balance has to do.

If I asked -dsuage=85, it would do all chunks that are more than 15%
full?
So, do I need to change the text above to say "more than 45% full" ?

More generally, does it not make sense to just use the same percentage
in -dusage than the percentage of total filesytem full?

Thanks,
Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: How does "btrfs fi show" show full?

[Brendan Hide]

On 2014/05/07 09:59 AM, Marc MERLIN wrote:
> [snip]
>
> Did I get this right?
> I'm not sure I did, since it seems the bigger the -dusage number, the
> more work balance has to do.
>
> If I asked -dsuage=85, it would do all chunks that are more than 15%
> full?

-dusage=85 balances all chunks that up to 85% full. The higher the 
number, the more work that needs to be done.
> So, do I need to change the text above to say "more than 45% full" ?
>
> More generally, does it not make sense to just use the same percentage
> in -dusage than the percentage of total filesytem full?
>
> Thanks,
> Marc

Separately, Duncan has made me realise my "halfway up" algorithm is not 
very good - it was probably just "good enough" at the time and worked 
"well enough" that I wasn't prompted to analyse it further.

Doing a simulation with randomly-semi-filled chunks, "df" at 55%, and 
chunk utilisation at 86%, -dusage=55 balances 30% of the chunks, almost 
perfectly bringing chunk utilisation down to 56%. In my algorithm I 
would have used -dusage=70 which in my simulation would have balanced 
34% of the chunks - but bringing chunk utilisation down to 55% - a bit 
of wasted effort and unnecessary SSD wear.

I think now that I need to experiment with a much lower -dusage value 
and perhaps to repeat the balance with the df value (55 in the example) 
if the chunk usage is still too high. Getting an optimal first value 
algorithmically might prove a challenge - I might just end up picking 
some arbitrary percentage point below the df value.

Pathological use-cases still apply however (for example if all chunks 
except one are exactly 54% full). The up-side is that if the algorithm 
is applied regularly (as in scripted and scheduled) then the situation 
will always be that the majority of chunks are going to be relatively 
full, avoiding the pathological use-case.

-- 
__________
Brendan Hide
http://swiftspirit.co.za/
http://www.webafrica.co.za/?AFF1E97

Re: How does "btrfs fi show" show full?

[Marc MERLIN]

On Wed, May 07, 2014 at 12:53:11PM +0200, Brendan Hide wrote:
> On 2014/05/07 09:59 AM, Marc MERLIN wrote:
> >[snip]
> >
> >Did I get this right?
> >I'm not sure I did, since it seems the bigger the -dusage number, the
> >more work balance has to do.
> >
> >If I asked -dsuage=85, it would do all chunks that are more than 15%
> >full?
> 
> -dusage=85 balances all chunks that up to 85% full. The higher the
> number, the more work that needs to be done.

Aah, right. I see why it's more work.
=20 only makes is process the few chunks that are up to 20% full which
won't be many if your FS is almost full.


> Doing a simulation with randomly-semi-filled chunks, "df" at 55%,
> and chunk utilisation at 86%, -dusage=55 balances 30% of the chunks,
> almost perfectly bringing chunk utilisation down to 56%. In my

Oh good, so my guess of using the same number for FS used and dusage=
was good, glad to know it worked well for you too.

> Pathological use-cases still apply however (for example if all
> chunks except one are exactly 54% full). The up-side is that if the
> algorithm is applied regularly (as in scripted and scheduled) then
> the situation will always be that the majority of chunks are going
> to be relatively full, avoiding the pathological use-case.

Sounds good. Also, the filesystem itself should ideally do this in the
background soon. What we're doing here is just a stopgap.

Marc
-- 
"A mouse is a device used to point at the xterm you want to type in" - A.S.R.
Microsoft is to operating systems ....
                                      .... what McDonalds is to gourmet cooking
Home page: http://marc.merlins.org/                         | PGP 1024R/763BE901

Re: How does "btrfs fi show" show full?

[Duncan]

On Wed, 7 May 2014 04:30:30 -0700
Marc MERLIN <marc@merlins.org> wrote:

> > -dusage=85 balances all chunks that up to 85% full. The higher the
> > number, the more work that needs to be done.  
> 
> Aah, right. I see why it's more work. =20 only makes is process the
> few chunks that are up to 20% full which won't be many if your FS
> is almost full.

It's actually even less work than you imply.

Balance only has to rewrite the actual content, not the empty space in
the chunk.  So 20% full means it's only writing 20% of the
(possible/full) content, thus only taking 20% of the time to rewrite
that chunk that it'd take to rewrite a full chunk.

Which is why a usage=5 or 20 goes so fast, even if the system's
actually mostly empty but is all allocated.  With a 20% full chunk it's
rewriting five chunks into one; at 5%, it's rewriting 20 chunks into
one.  That goes pretty fast, even if there's a bunch of them to write!

-- 
Duncan - No HTML messages please, as they are filtered as spam.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman