Re: Growing RAID10 with active XFS filesystem

[Dave Chinner <david@fromorbit.com>]

On Tue, Jan 09, 2018 at 09:36:49AM +0000, Wols Lists wrote:
> On 08/01/18 22:01, Dave Chinner wrote:
> > Yup, 21 devices in a RAID 10. That's a really nasty config for
> > RAID10 which requires an even number of disks to mirror correctly.
> > Why does MD even allow this sort of whacky, sub-optimal
> > configuration?
> 
> Just to point out - if this is raid-10 (and not raid-1+0 which is a
> completely different beast) this is actually a normal linux config. I'm
> planning to set up a raid-10 across 3 devices. What happens is that is
> that raid-10 writes X copies across Y devices. If X = Y then it's a
> normal mirror config, if X > Y it makes good use of space (and if X < Y
> it doesn't make sense :-)
> 
> SDA: 1, 2, 4, 5
> 
> SDB: 1, 3, 4, 6
> 
> SDC: 2, 3, 5, 6

It's nice to know that MD has redefined RAID-10 to be different to
the industry standard definition that has been used for 20 years and
optimised filesystem layouts for.  Rotoring data across odd numbers
of disks like this is going to really, really suck on filesystems
that are stripe layout aware..

For example, XFS has hot-spot prevention algorithms in it's
internal physical layout for striped devices. It aligns AGs across
different stripe units so that metadata and data doesn't all get
aligned to the one disk in a RAID0/5/6 stripe. If the stripes are
rotoring across disks themselves, then we're going to end up back in
the same position we started with - multiple AGs aligned to the
same disk.

The result is that many XFS workloads are going to hotspot disks and
result in unbalanced load when there are an odd number of disks in a
RAID-10 array.  Actually, it's probably worse than having no
alignment, because it makes hotspot occurrence and behaviour very
unpredictable.

Worse is the fact that there's absolutely nothing we can do to
optimise allocation alignment or IO behaviour at the filesystem
level. We'll have to make mkfs.xfs aware of this clusterfuck and
turn off stripe alignment when we detect such a layout, but that
doesn't help all the existing user installations out there right
now.

IMO, odd-numbered disks in RAID-10 should be considered harmful and
never used....

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com