Re: Rebalancing raid1 after adding a device

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2019-06-18 14:26, Stéphane Lesimple wrote:
> Hello,
> 
> I've been a btrfs user for quite a number of years now, but it seems I 
> need the wiseness of the btrfs gurus on this one!
> 
> I have a 5-hdd btrfs raid1 setup with 4x3T+1x10T drives.
> A few days ago, I replaced one of the 3T by a new 10T, running btrfs 
> replace and then resizing the FS to use all the available space of the 
> new device.
> 
> The filesystem was 90% full before I expanded it so, as expected, most 
> of the space on the new device wasn't actually allocatable in raid1, as 
> very few available space was available on the 4 other devs.
> 
> Of course the solution is to run a balance, but as the filesystem is now 
> quite big, I'd like to avoid running a full rebalance. This would be 
> quite i/o intensive, would be running for several days, and putting and 
> unecessary stress on the drives. This also seems excessive as in theory 
> only some Tb would need to be moved: if I'm correct, only one of two 
> block groups of a sufficient amount of chunks to be moved to the new 
> device so that the sum of the amount of available space on the 4 
> preexisting devices would at least equal the available space on the new 
> device, ~7Tb instead of moving ~22T.
> I don't need to have a perfectly balanced FS, I just want all the space 
> to be allocatable.
> 
> I tried using the -ddevid option but it only instructs btrfs to work on 
> the block groups allocated on said device, as it happens, it tends to 
> move data between the 4 preexisting devices and doesn't fix my problem. 
> A full balance with -dlimit=100 did no better.
> 
> Is there a way to ask the block group allocator to prefer writing to a 
> specific device during a balance? Something like -ddestdevid=N? This 
> would just be a hint to the allocator and the usual constraints would 
> always apply (and prevail over the hint when needed).
> 
> Or is there any obvious solution I'm completely missing?

Based on what you've said, you may actually not have enough free space 
that can be allocated to balance things properly.

When a chunk gets balanced, you need to have enough space to create a 
new instance of that type of chunk before the old one is removed.  As 
such, if you can't allocate new chunks at all, you can't balance those 
chunks either.

So, that brings up the question of how to deal with your situation.

The first thing I would do is multiple compaction passes using the 
`usage` filter.  Start with:

     btrfs balance -dusage=0 -musage=0 /wherever

That will clear out any empty chunks which haven't been removed (there 
shouldn't be any if you're on a recent kernel, but it's good practice 
anyway).  After that, repeat the same command, but with a value of 10 
instead of 0, and then keep repeating in increments of 10 up until 50. 
Doing this will clean up chunks that are more than half empty (making 
multiple passes like this is a bit more reliable, and in some cases also 
more efficient), which should free up enough space for balance to work 
with (as well as probably moving most of the block groups it touches to 
use the new disk).