Re: Is it safe to use btrfs on top of different types of devices?

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2017-10-17 16:21, Adam Borowski wrote:
> On Tue, Oct 17, 2017 at 03:19:09PM -0400, Austin S. Hemmelgarn wrote:
>> On 2017-10-17 13:06, Adam Borowski wrote:
>>> The thing is, reliability guarantees required vary WILDLY depending on your
>>> particular use cases.  On one hand, there's "even an one-minute downtime
>>> would cost us mucho $$$s, can't have that!" -- on the other, "it died?
>>> Okay, we got backups, lemme restore it after the weekend".
>> Yes, but if you are in the second case, you arguably don't need replication,
>> and would be better served by improving the reliability of your underlying
>> storage stack than trying to work around it's problems. Even in that case,
>> your overall reliability is still constrained by the least reliable
>> component (in more idiomatic terms 'a chain is only as strong as it's
>> weakest link').
> 
> MD can handle this case well, there's no reason btrfs shouldn't do that too.
> A RAID is not akin to serially connected chain, it's a parallel connected
> chain: while pieces of the broken second chain hanging down from the first
> don't make it strictly more resilient than having just a single chain, in
> general case it _is_ more reliable even if the other chain is weaker.
My chain analogy is supposed to be relating to the storage stack as a 
whole, RAID is a single link in the chain, with whatever filesystem 
above it, and whatever storage drivers and hardware below.
> 
> Don't we have a patchset that deals with marking a device as failed at
> runtime floating on the mailing list?  I did not look at those patches yet,
> but they are a step in this direction.
There were some disagreements on whether the device should be released 
(that is, the node closed) immediately when we know it's failed, or 
should be held open until remount.
> 
>> Using replication with a reliable device and a questionable device is
>> essentially the same as trying to add redundancy to a machine by adding an
>> extra linkage that doesn't always work and can get in the way of the main
>> linkage it's supposed to be protecting from failure.  Yes, it will work most
>> of the time, but the system is going to be less reliable than it is without
>> the 'redundancy'.
> 
> That's the current state of btrfs, but the design is sound, and reaching
> more than parity with MD is a matter of implementation.
Indeed, however MD is still not perfectly reliable in this situation 
(though they are exponentially better than BTRFS at the moment).
> 
>>> Thus, I switched the machine to NBD (albeit it sucks on 100Mbit eth).  Alas,
>>> the network driver allocates memory with GFP_NOIO which causes NBD
>>> disconnects (somehow, this doesn't ever happen on swap where GFP_NOIO would
>>> be obvious but on regular filesystem where throwing out userspace memory is
>>> safe).  The disconnects happen around once per week.
>> Somewhat off-topic, but you might try looking at ATAoE as an alternative,
>> it's more reliable in my experience (if you've got a reliable network),
>> gives better performance (there's less protocol overhead than NBD, and it
>> runs on top of layer 2 instead of layer 4)
> 
> I've tested it -- not on the Odroid-U2 but on Pine64 (fully working GbE).
> NBD delivers 108MB/sec in a linear transfer, ATAoE is lucky to break
> 40MB/sec, same target (Qnap-253a, spinning rust), both in default
> configuration without further tuning.  NBD is over IPv6 for that extra 20
> bytes per packet overhead.
Interesting, I've seen the the exact opposite in terms of performance.
> 
> Also, NBD can be encrypted or arbitrarily routed.
Yes, though if you're on a local network, neither should matter :).
> 
>>> It's a single-device filesystem, thus disconnects are obviously fatal.  But,
>>> they never caused even a single bit of damage (as scrub goes), thus proving
>>> btrfs handles this kind of disconnects well.  Unlike times past, the kernel
>>> doesn't get confused thus no reboot is needed, merely an unmount, "service
>>> nbd-client restart", mount, restart the rebuild jobs.
>> That's expected behavior though.  _Single_ device BTRFS has nothing to get
>> out of sync most of the time, the only time there's any possibility of an
>> issue is when you die after writing the first copy of a block that's in a
>> dup profile chunk, but even that is not very likely to cause problems
>> (you'll just lose at most the last <commit-time> worth of data).
> 
> How come?  In a DUP profile, the writes are: chunk 1, chunk2, barrier,
> superblock.  The two prior writes may be arbitrarily reordered -- both
> between each other or even individual sectors inside the chunks, but unless
> the disk lies about barriers, there's no way to have any corruption, thus
> running scrub is not needed.
If the device dies after writing chunk 1 but before the barrier, you end 
up needing scrub.  How much of a failure window is present is largely a 
function of how fast the device is, but there is a failure window there.
> 
>> The moment you add another device though, that simplicity goes out the
>> window.
> 
> RAID1 doesn't seem less simple to me: if the new superblock has been
> successfully written on at least one disk, barriers imply that at least one
> copy is correct.  If the other disk was out to lunch before the final
> unmount, those blocks will be degraded, but that's no different from one
> pair of DUP blocks being corrupted.
It's not guaranteed to be just those blocks though, it could be anything 
up to and including the superblock.  It's the handling needed for that 
situation, as well as possibly being so out of sync that generations on 
the old device don't have any data at all any more on the new one, that 
makes replication complex in BTRFS.
> 
> With RAID5, there be dragons, but that's due to implementation deficiencies;
> if an upper layer says "hey you downstairs, the block you gave me has a
> wrong csum/generation, try to recover it", there's no reason it shouldn't
> be able to reliably recover it in all cases that don't involve a double
> (RAID5) or triple (RAID6) failure.