Re: Rebalancing raid1 after adding a device

["Stéphane Lesimple" <stephane_btrfs@lesimple.fr>]

June 18, 2019 8:45 PM, "Hugo Mills" <hugo@carfax.org.uk> wrote:

> On Tue, Jun 18, 2019 at 08:26:32PM +0200, Stéphane Lesimple wrote:
>> [...]
>> Of course the solution is to run a balance, but as the filesystem is
>> now quite big, I'd like to avoid running a full rebalance. This
>> would be quite i/o intensive, would be running for several days, and
>> putting and unecessary stress on the drives. This also seems
>> excessive as in theory only some Tb would need to be moved: if I'm
>> correct, only one of two block groups of a sufficient amount of
>> chunks to be moved to the new device so that the sum of the amount
>> of available space on the 4 preexisting devices would at least equal
>> the available space on the new device, ~7Tb instead of moving ~22T.
>> I don't need to have a perfectly balanced FS, I just want all the
>> space to be allocatable.
>> 
>> I tried using the -ddevid option but it only instructs btrfs to work
>> on the block groups allocated on said device, as it happens, it
>> tends to move data between the 4 preexisting devices and doesn't fix
>> my problem. A full balance with -dlimit=100 did no better.
> 
> -dlimit=100 will only move 100 GiB of data (i.e. 200 GiB), so it'll
> be a pretty limited change. You'll need to use a larger number than
> that if you want it to have a significant visible effect.

Yes of course, I wasn't clear here but what I meant to do when starting
a full balance with -dlimit=100 was to test under a reasonable amount of
time whether the allocator would prefer to fill the new drive. I observed
after those 100G (200G) of data moved that it wasn't the case at all.
Specifically, no single allocation happened on the new drive. I know this
would be the case at some point, after Terabytes of data would have been
moved, but that's exactly what I'm trying to avoid.

> The -ddevid=<old_10T> option would be my recommendation. It's got
> more chunks on it, so they're likely to have their copies spread
> across the other four devices. This should help with the
> balance.

Makes sense. That's probably what I'm going to do if I don't find
a better solution. That's a bit frustrating because I know exactly
what I want btrfs to do, but I have no way to make it do that.

> Alternatively, just do a full balance and then cancel it when the
> amount of unallocated space is reasonably well spread across the
> devices (specifically, the new device's unallocated space is less than
> the sum of the unallocated space on the other devices).

I'll try with the old 10T and cancel it when I get 0 unallocatable
space, if that happens before all the data is moved around.

>> Is there a way to ask the block group allocator to prefer writing to
>> a specific device during a balance? Something like -ddestdevid=N?
>> This would just be a hint to the allocator and the usual constraints
>> would always apply (and prevail over the hint when needed).
> 
> No, there isn't. Having control over the allocator (or bypassing
> it) would be pretty difficult to implement, I think.
> 
> It would be really great if there was an ioctl that allowed you to
> say things like "take the chunks of this block group and put them on
> devices 2, 4 and 5 in RAID-5", because you could do a load of
> optimisation with reshaping the FS in userspace with that. But I
> suspect it's a long way down the list of things to do.

Exactly, that would be awesome. I would probably even go as far as
writing some C code myself to call this ioctl to do this "intelligent"
balance on my system!

--
Stéphane.