Re: balancing every night broke balancing so now I can't balance anymore?

[Hugo Mills <hugo@carfax.org.uk>]

[-- Attachment #1: Type: text/plain, Size: 3728 bytes --]

On Sun, May 14, 2017 at 01:15:09PM -0700, Marc MERLIN wrote:
> On Sun, May 14, 2017 at 09:13:35PM +0200, Hans van Kranenburg wrote:
> > On 05/13/2017 10:54 PM, Marc MERLIN wrote:
> > > Kernel 4.11, btrfs-progs v4.7.3
> > > 
> > > I run scrub and balance every night, been doing this for 1.5 years on this
> > > filesystem.
> > 
> > What are the exact commands you run every day?
>  
> http://marc.merlins.org/perso/btrfs/post_2014-03-19_Btrfs-Tips_-Btrfs-Scrub-and-Btrfs-Filesystem-Repair.html
> (at the bottom)
> every night:
> 1) scrub
> 2) balance -musage=0
> 3) balance -musage=20

   In most cases, this is going to make ENOSPC problems worse, not
better. The reason for doign this kind of balance is to recover unused
space and allow it to be reallocated. The typical behaviour is that
data gets overallocated, and it's metadata which runs out. So, the
last thing you want to be doing is reducing the metadata allocation,
because that's the scarce resource.

   Also, I'd usually recommend using limit=n, where n is approximately
the amount of data overallcation (allocated space less used
space). It's much more controllable than usage.

   Hugo.

> 4) balance -dusage=0
> 5) balance -dusage=20
> 
> > > How did I get into such a misbalanced state when I balance every night?
> > 
> > I don't know, since I don't know what you do exactly. :)
>  
> Now you do :)
> 
> > > My filesystem is not full, I can write just fine, but I sure cannot
> > > rebalance now.
> > 
> > Yes, because you have quite some allocated but unused space. If btrfs
> > cannot just allocate more chunks, it starts trying a bit harder to reuse
> > all the empty spots in the already existing chunks.
> 
> Ok. shouldn't balance fix problems just like this?
> I have 60GB-ish free, or in this case that's also >25%, that's a lot
> 
> Speaking of unallocated, I have more now:
>     Device unallocated:		 993.00MiB
> 
> This kind of just magically fixed itself during snapshot rotation and
> deletion I think.
> Sure enough, balance works again, but this feels pretty fragile.
> Looking again:
>     Device size:		 228.67GiB
>     Device allocated:		 227.70GiB
>     Device unallocated:		 993.00MiB
>     Free (estimated):		  58.53GiB	(min: 58.53GiB)
> 
> You're saying that I need unallocated space for new chunks to be
> created, which is required by balance.
> Should btrfs not take care of keeping some space for me?
> Shoudln't a nigthly balance, which I'm already doing, help even more
> with this?
> 
> > > Besides adding another device to add space, is there a way around this
> > > and more generally not getting into that state anymore considering that
> > > I already rebalance every night?
> > 
> > Add monitoring and alerting on the amount of unallocated space.
> > 
> > FWIW, this is what I use for that purpose:
> > 
> > https://packages.debian.org/sid/munin-plugins-btrfs
> > https://packages.debian.org/sid/monitoring-plugins-btrfs
> > 
> > And, of course the btrfs-heatmap program keeps being a fun tool to
> > create visual timelapses of your filesystem, so you can learn how your
> > usage pattern is resulting in allocation of space by btrfs, and so that
> > you can visually see what the effect of your btrfs balance attempts is:
> 
> That's interesting, but ultimately, users shoudln't have to micromanage
> their filesystem to that level, even btrfs.
> 
> a) What is wrong in my nightly script that I should fix/improve?
> b) How do I recover from my current state?
> 
> Thanks,
> Marc

-- 
Hugo Mills             | You stay in the theatre because you're afraid of
hugo@... carfax.org.uk | having no money? There's irony...
http://carfax.org.uk/  |
PGP: E2AB1DE4          |                                     Slings and Arrows

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 836 bytes --]