Re: Understanding BTRFS storage

[Duncan <1i5t5.duncan@cox.net>]

Austin S Hemmelgarn posted on Thu, 27 Aug 2015 08:01:58 -0400 as
excerpted:

>> Someone (IIRC it was Austin H) posted what I thought was an extremely
>> good setup, a few weeks ago.  Create two (or more) mdraid0s, and put
>> btrfs raid1 (or raid5/6 when it's a bit more mature, I've been
>> recommending waiting until 4.4 and see what the on-list reports for it
>> look like then) on top.  The btrfs raid on top lets you use btrfs' data
>> integrity features, while the mdraid0s beneath help counteract the fact
>> that btrfs isn't well optimized for speed yet, the way mdraid has been.
>> And the btrfs raid on top means all is not lost with a device going bad
>> in the mdraid0, as would normally be the case, since the other
>> raid0(s),
>> functioning as the remaining btrfs devices, let you rebuild the missing
>> btrfs device, by recreating the missing raid0.
>>
>> Normally, that sort of raid01 is discouraged in favor of raid10, with
>> raid1 at the lower level and raid0 on top, for more efficient rebuilds,
>> but btrfs' data integrity features change that story entirely. =:^)
>>
> Two additional things:
> 1. If you use MD RAID1 instead of RAID0, it's just as fast for reads, no
> slower than on top of single disks for writes, and get's you better data
> safety guarantees than even raid6 (if you do 2 MD RAID 1 devices with
> BTRFS raid1 on top, you can lose all but one disk and still have all
> your data).

My hesitation for btrfs raid1 on top of mdraid1, is that a btrfs scrub 
doesn't scrub all the mdraid component devices.

Of course if btrfs scrub finds an error, it will try to rewrite the bad 
copy from the (hopefully good) other btrfs raid1 copy, and that will 
trigger a rewrite of both/all copies on that underlying mdraid1, which 
should catch the bad one in the process no matter which one it was.

But if one of the lower level mdraid1 component devices is bad while the 
other(s) are good, and mdraid happens to pick the good device, it won't 
even see and thus can't scrub the bad lower-level copy.

To avoid that problem, one can of course do an mdraid level scrub 
followed by a btrfs scrub.  The mdraid level scrub won't tell bad from 
good but will simply ensure they match, and if it happens to pick the bad 
one at that level, the followon btrfs level scrub will detect that and 
trigger a rewrite from its other copy, which again, will rewrite both/all 
the underlying mdraid1 component devices on that btrfs raid1 side, but 
that still wouldn't ensure that the rewrite actually happened properly, 
so then you're left redoing both levels yet again, to ensure that.

Which in theory can work, but in practice, particularly on spinning rust, 
you pretty quickly reach a point when you're running 24/7 scrubs, which, 
again particularly on spinning rust, is going to kill throughput for 
pretty much any other IO going on at the same time.


Which is one of the reasons I found btrfs raid1 on mdraid0 so appealing 
in comparison -- raid0 has only the single copy, which is either correct 
or incorrect, and if the btrfs scrub turns up a problem, it does the 
rewrite, and a single second pass of that btrfs scrub can verify that the 
rewrite happened correctly, because there's no hidden copies being picked 
more or less randomly at the mdraid level, only the single copy, which is 
either correct or incorrect.  I like that determinism! =:^)

-- 
Duncan - List replies preferred.   No HTML msgs.
"Every nonfree program has a lord, a master --
and if you use the program, he is your master."  Richard Stallman