Never balance metadata?

Never balance metadata?

[waxhead]

I have read several places, including on this mailing list that metadata 
is not supposed to be balanced unless converting between profiles.

Interestingly enough there is nothing mentioned about this in the docs...
https://btrfs.readthedocs.io/en/latest/btrfs-balance.html

Should one still NOT balance metadata? If so - please update the docs 
with a explanation to why one should not do that.

Re: Never balance metadata?

[Martin Steigerwald]

waxhead - 11.02.23, 02:36:26 CET:
> I have read several places, including on this mailing list that
> metadata is not supposed to be balanced unless converting between
> profiles.
> 
> Interestingly enough there is nothing mentioned about this in the
> docs... https://btrfs.readthedocs.io/en/latest/btrfs-balance.html
> 
> Should one still NOT balance metadata? If so - please update the docs
> with a explanation to why one should not do that.

If that is still the case, it may also be good to by default omit 
metadata on a balance without parameters.

A long, long time ago I experienced a much longer boot time after a 
balance, I decided to refrain from balancing my BTRFS file systems. 
Especially as long as there is still unallocated space on them. Cause as 
long as BTRFS can still allocate a chunk, I thought that is fine. I 
thought I would be getting a faster system, but it was much much slower. 
That was on a SATA SSD. 

An additional reason for not, or at least not regularly balancing for me 
is avoiding needless writes to flash storage like SATA or NVMe SSDs. 
Usually they can take it, but why age them more than needed?

Maybe all or some of this is not accurate anymore and one should balance 
BTRFS regularly like default in SUSE distributions, maybe regular 
balancing even reduces writes on other occasions, however I never fully 
understood the performance impact of a regular balance. However in a 
virtual environment with lots of virtual machines it may not be best to 
start balance on a lot of them at same or similar times either, 
considering the load it creates for the underlying storage.

I know balancing not the same as defragmenting files in other filesystems, 
but I still remember the strong warnings of XFS developers not to ever 
just defragment a XFS filesystem just cause you feel like it. The 
reasoning behind these warnings was that defragmenting files tends to 
fragment free space. But balancing actually defragments free space as 
far as I understood and thus is different.

Anyway, I am keen to read on more insight on this from people who 
understand better what impact a BTRFS balance really has.

Thanks,
-- 
Martin

Re: Never balance metadata?

[Andrei Borzenkov]

On 11.02.2023 11:38, Martin Steigerwald wrote:
> waxhead - 11.02.23, 02:36:26 CET:
>> I have read several places, including on this mailing list that
>> metadata is not supposed to be balanced unless converting between
>> profiles.
>>
>> Interestingly enough there is nothing mentioned about this in the
>> docs... https://btrfs.readthedocs.io/en/latest/btrfs-balance.html
>>
>> Should one still NOT balance metadata? If so - please update the docs
>> with a explanation to why one should not do that.
> 
> If that is still the case, it may also be good to by default omit
> metadata on a balance without parameters.
> 
> A long, long time ago I experienced a much longer boot time after a
> balance, I decided to refrain from balancing my BTRFS file systems.
> Especially as long as there is still unallocated space on them. Cause as
> long as BTRFS can still allocate a chunk, I thought that is fine. I
> thought I would be getting a faster system, but it was much much slower.
> That was on a SATA SSD.
> 
> An additional reason for not, or at least not regularly balancing for me
> is avoiding needless writes to flash storage like SATA or NVMe SSDs.
> Usually they can take it, but why age them more than needed?
> 
> Maybe all or some of this is not accurate anymore and one should balance
> BTRFS regularly like default in SUSE distributions, maybe regular
> balancing even reduces writes on other occasions, however I never fully
> understood the performance impact of a regular balance. However in a

Balancing creates larger free space pool which allows large writes which 
is faster and requires less metadata. If every second 4K block in each 
chunk is free, you have exactly 50% free space like if half of chunks 
are free. But in the former case each write will be split into 4K 
pieces, each 4K becomes separate extent and needs additional metadata. 
Whether you will actually observe impact depends heavily on your workload.

Secondary (btrfs specific) consideration is that balancing creates free 
space that can be used both for data and metadata, while each chunk is 
(normally) dedicated to either one. So if your metadata is full you 
/may/ get more space for metadata by compacting data chunks (except 
balancing needs more metadata, at least temporary, an so my fail under 
these conditions).

This is also the reason why balancing metadata is not recommended today. 
Fragmented metadata chunks do not have as much impact on performance, 
but space that is freed by balancing may be allocated to data chunks and 
so unavailable to metadata anymore. As Zygo put it once, if btrfs ever 
needed as much metadata, there are good chances it will need it again.

> virtual environment with lots of virtual machines it may not be best to
> start balance on a lot of them at same or similar times either,
> considering the load it creates for the underlying storage.
> 
> I know balancing not the same as defragmenting files in other filesystems,
> but I still remember the strong warnings of XFS developers not to ever
> just defragment a XFS filesystem just cause you feel like it. The
> reasoning behind these warnings was that defragmenting files tends to
> fragment free space. But balancing actually defragments free space as
> far as I understood and thus is different.
> 
> Anyway, I am keen to read on more insight on this from people who
> understand better what impact a BTRFS balance really has.
> 
> Thanks,

Re: Never balance metadata?

[Martin Steigerwald]

Andrei Borzenkov - 11.02.23, 12:42:49 CET:
> Balancing creates larger free space pool which allows large writes
> which is faster and requires less metadata. If every second 4K block
> in each chunk is free, you have exactly 50% free space like if half
> of chunks are free. But in the former case each write will be split
> into 4K pieces, each 4K becomes separate extent and needs additional
> metadata. Whether you will actually observe impact depends heavily on
> your workload.

Okay so it can save metadata size. And on a full BTRFS filesystem with no 
unallocated space it can reduce fragmentation of new writes. I get that 
much.

But what if I have for example

% btrfs filesystem usage -T /home
Overall:
    Device size:                 400.00GiB
    Device allocated:            300.02GiB
    Device unallocated:           99.98GiB
[…]
    Free (estimated):            113.60GiB      (min: 113.60GiB)
    Free (statfs, df):           113.60GiB
[…]

(metadata usage is 3.52GiB of 5.01 GiB)

on a filesystem that is heavily written to and

% btrfs filesystem usage -T /some/dir
Overall:
    Device size:                   1.20TiB
    Device allocated:              1.19TiB
    Device unallocated:            8.78GiB
[…]
    Free (estimated):             44.77GiB      (min: 44.77GiB)
    Free (statfs, df):            44.77GiB
[…]

(metadata usage is 2.88GiB of 4.01 GiB)

on a filesystem that is used mostly for the purpose of placing larger 
files there to archive them? Both are using single profiles. 

Why would I balance those? They are stored on the same 2TB NVMe SSD in 
this laptop and I did not yet see any performance related issues with 
them so far.

What I went with is this: If it is heavily written to leave enough space 
free to begin with. And if its just for storing larger files without 
receiving any regular writes by applications then I can fill it a bit 
more. Just similar to what I would do with XFS filesystem for example as 
well. Having "/home" on a 95% full XFS filesystem would not be a good 
idea either – mind you I had something like that with a BTRFS filesystem 
before and before kernel 4.5 or 4.6 actually sometimes the kernel came 
to a halt while with BTRFS searching for free space. These some times 
were related to no unallocated space free and then I balanced until next 
time it happened, but since kernel 4.5 or 4.6 this issue was gone for 
good. And before that as long as unallocated space was free it was 
working fine.

So if I plan like that, I still do not get why I would balance my 
filesystems regularly. And if I do not plan like that and let /home usage 
grow to 95% or even more… I am not sure how much "relief" balancing 
would really give to the filesystem. Is it more than the amount of work 
done during the regular balances?

Well probably that is the "depends heavily on your workload" thing you 
wrote about :)

Thanks,
-- 
Martin

Re: Never balance metadata?

[Forza]

On 2023-02-11 02:36, waxhead wrote:
> I have read several places, including on this mailing list that metadata 
> is not supposed to be balanced unless converting between profiles.
> 
> Interestingly enough there is nothing mentioned about this in the docs...
> https://btrfs.readthedocs.io/en/latest/btrfs-balance.html
> 
> Should one still NOT balance metadata? If so - please update the docs 
> with a explanation to why one should not do that.

There is no direct issue with balancing metadata chunks.

The recommendation to not balance metadata is because it can contribute 
to a ENOSPC (no free space left) error. This happens when Btrfs needs to 
allocate another metadata chunk, but there is no unallocated space 
available.

Because metadata usually only is a small percentage of the filesystem, 
it makes sense to simply leave it unless you need to convert between 
profiles or add/remove devices.

Btrfs uses a two-stage allocator. The first stage allocates large 
regions of space known as chunks for specific types of data, then the 
second stage allocates blocks like a regular (old-fashioned) filesystem 
within these larger regions.

There are three different types of block groups/chunks that Btrfs uses:
DATA: Stores normal user file data.
METADATA: Stores internal metadata. Small files can also stored inline.
SYSTEM: Stores mapping between physical devices and the logical space 
representing the filesystem.
UNALLOCATED: Any unallocated space. Can be used to allocate any of the 
three data types.

Balancing means compacting chunks to make them fully utilised by moving 
data from other chunks and release the empty chunks back as unallocated 
space. This has the effect of defragmenting the free space which 
improves sequential write speeds, especially on spinning HDDs.

The problem with balancing metadata (that leads to the recommendation to 
not balance metadata) is that when metadata chunks are full, Btrfs needs 
to allocate new metadata chunks to write additional metadata in. Many 
disk operations, including removing files and snapshots requires 
additional metadata. If this happens when there is not enough 
unallocated space to allocate another metadata chunk the filesystem with 
fail with ENOSPC. By letting metadata chunks be underutilised, this risk 
is lower, especially on very full filesystems.

- Forza

Re: Never balance metadata?

[Martin Steigerwald]

Forza - 11.02.23, 18:18:50 CET:
> On 2023-02-11 02:36, waxhead wrote:
> > I have read several places, including on this mailing list that
> > metadata is not supposed to be balanced unless converting between
> > profiles.
> > 
> > Interestingly enough there is nothing mentioned about this in the
> > docs... https://btrfs.readthedocs.io/en/latest/btrfs-balance.html
> > 
> > Should one still NOT balance metadata? If so - please update the
> > docs
> > with a explanation to why one should not do that.
> 
> There is no direct issue with balancing metadata chunks.
> 
> The recommendation to not balance metadata is because it can
> contribute to a ENOSPC (no free space left) error. This happens when
> Btrfs needs to allocate another metadata chunk, but there is no
> unallocated space available.

Thanks for that very good and thorough explanation.

Something like this in the official documentation would be great!

Best,
-- 
Martin

Re: Never balance metadata?

[Jukka Larja]

Andrei Borzenkov kirjoitti 11.2.2023 klo 13.42:

> Fragmented metadata chunks do not have as much impact on performance,

Although this may be very much true in general case, I do have a counter 
example:

I have a 64 TB Btrfs filesystem (RAID1, so 32 TB available) made up of eight 
HDDs of varying size (all 7200 RPM NAS disks though, so pretty similar 
performance). Every night, backups of hundreds of thousands of files are 
rsynced to file system, most of course unchanged. Some new multi-gigabyte 
video files are also backed up to another filesystem on the same computer, 
making it so that any disk caches are in unhelpful state by the morning.

Now, when I fire up my MythTV DVR computer, which likes to check if there 
are any changes to media files (some hundreds of folders with thousands of 
files, nothing major). That check used to took dozens of seconds. So long 
that MythTV sometimes gave up and reported "no files". Few months ago I 
decided to move to RAID1C3 for metadata. After balance, the everyday 
scanning for changes only takes couple of seconds. Backups are also much 
faster, though as they run during night, it doesn't much matter.

The filesystem was created June 2010 (probably) and every one of the 
original disks and some that came after have been replaced. This was the 
first time I balanced metadata, as far as I can remember. I think it does 
make sense to balance every now and then, if only once a decade :D .

-JLarja

-- 
      ...Elämälle vierasta toimintaa...
     Jukka Larja, Roskakori@aarghimedes.fi

<saylan> I just set up port forwards to defense.gov
<saylan> anyone scanning me now will be scanning/attacking the DoD :D
<renderbod> O.o
<bolt> that's... not exactly how port forwarding works
<saylan> ?
- Quote Database, http://www.bash.org/?954232 -

Re: Never balance metadata?

[Zygo Blaxell]

On Sun, Feb 12, 2023 at 12:10:16PM +0200, Jukka Larja wrote:
> After balance, the everyday
> scanning for changes only takes couple of seconds. Backups are also much
> faster, though as they run during night, it doesn't much matter.

It would be interesting to compare the results of a metadata balance with
a metadata defrag ('btrfs fi defrag /path/to/subvolume' _without_ -r).

I suspect a metadata defrag has much the same side-effect as the metadata
balance (reduce average seek distance between pages), but metadata defrag
won't reduce spare metadata allocations to zero.

> The filesystem was created June 2010 (probably) and every one of the
> original disks and some that came after have been replaced. This was the
> first time I balanced metadata, as far as I can remember. I think it does
> make sense to balance every now and then, if only once a decade :D .
> 
> -JLarja
> 
> -- 
>      ...Elämälle vierasta toimintaa...
>     Jukka Larja, Roskakori@aarghimedes.fi
> 
> <saylan> I just set up port forwards to defense.gov
> <saylan> anyone scanning me now will be scanning/attacking the DoD :D
> <renderbod> O.o
> <bolt> that's... not exactly how port forwarding works
> <saylan> ?
> - Quote Database, http://www.bash.org/?954232 -