Re: Rebalancing raid1 after adding a device

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2019-06-18 14:45, Hugo Mills wrote:
> On Tue, Jun 18, 2019 at 08:26:32PM +0200, Stéphane Lesimple wrote:
>> I've been a btrfs user for quite a number of years now, but it seems
>> I need the wiseness of the btrfs gurus on this one!
>>
>> I have a 5-hdd btrfs raid1 setup with 4x3T+1x10T drives.
>> A few days ago, I replaced one of the 3T by a new 10T, running btrfs
>> replace and then resizing the FS to use all the available space of
>> the new device.
>>
>> The filesystem was 90% full before I expanded it so, as expected,
>> most of the space on the new device wasn't actually allocatable in
>> raid1, as very few available space was available on the 4 other
>> devs.
>>
>> Of course the solution is to run a balance, but as the filesystem is
>> now quite big, I'd like to avoid running a full rebalance. This
>> would be quite i/o intensive, would be running for several days, and
>> putting and unecessary stress on the drives. This also seems
>> excessive as in theory only some Tb would need to be moved: if I'm
>> correct, only one of two block groups of a sufficient amount of
>> chunks to be moved to the new device so that the sum of the amount
>> of available space on the 4 preexisting devices would at least equal
>> the available space on the new device, ~7Tb instead of moving ~22T.
>> I don't need to have a perfectly balanced FS, I just want all the
>> space to be allocatable.
>>
>> I tried using the -ddevid option but it only instructs btrfs to work
>> on the block groups allocated on said device, as it happens, it
>> tends to move data between the 4 preexisting devices and doesn't fix
>> my problem. A full balance with -dlimit=100 did no better.
> 
>     -dlimit=100 will only move 100 GiB of data (i.e. 200 GiB), so it'll
> be a pretty limited change. You'll need to use a larger number than
> that if you want it to have a significant visible effect.
Last I checked, that's not how the limit filter works.  AFAIUI, it's an 
upper limit on how full a chunk can be to be considered for the balance 
operation.  So, balancing with only `-dlimit=100` should actually 
balance all data chunks (but only data chunks, because you haven't asked 
for metadata balancing).
> 
>     The -ddevid=<old_10T> option would be my recommendation. It's got
> more chunks on it, so they're likely to have their copies spread
> across the other four devices. This should help with the
> balance.
> 
>     Alternatively, just do a full balance and then cancel it when the
> amount of unallocated space is reasonably well spread across the
> devices (specifically, the new device's unallocated space is less than
> the sum of the unallocated space on the other devices).
> 
>> Is there a way to ask the block group allocator to prefer writing to
>> a specific device during a balance? Something like -ddestdevid=N?
>> This would just be a hint to the allocator and the usual constraints
>> would always apply (and prevail over the hint when needed).
> 
>     No, there isn't. Having control over the allocator (or bypassing
> it) would be pretty difficult to implement, I think.
> 
>     It would be really great if there was an ioctl that allowed you to
> say things like "take the chunks of this block group and put them on
> devices 2, 4 and 5 in RAID-5", because you could do a load of
> optimisation with reshaping the FS in userspace with that. But I
> suspect it's a long way down the list of things to do.
> 
>> Or is there any obvious solution I'm completely missing?
> 
>     I don't think so.
> 
>     Hugo.
>