Re: Is it safe to use btrfs on top of different types of devices?

["Austin S. Hemmelgarn" <ahferroin7@gmail.com>]

On 2017-10-16 21:14, Adam Borowski wrote:
> On Mon, Oct 16, 2017 at 01:27:40PM -0400, Austin S. Hemmelgarn wrote:
>> On 2017-10-16 12:57, Zoltan wrote:
>>> On Mon, Oct 16, 2017 at 1:53 PM, Austin S. Hemmelgarn wrote:
>> In an ideal situation, scrubbing should not be an 'only if needed' thing,
>> even for a regular array that isn't dealing with USB issues. From a
>> practical perspective, there's no way to know for certain if a scrub is
>> needed short of reading every single file in the filesystem in it's
>> entirety, at which point, you're just better off running a scrub (because if
>> you _do_ need to scrub, you'll end up reading everything twice).
> 
>> [...]  There are three things to deal with here:
>> 1. Latent data corruption caused either by bit rot, or by a half-write (that
>> is, one copy got written successfully, then the other device disappeared
>> _before_ the other copy got written).
>> 2. Single chunks generated when the array is degraded.
>> 3. Half-raid1 chunks generated by newer kernels when the array is degraded.
> 
> Note that any of the above other than bit rot affect only very recent data.
> If we keep record of the last known-good generation, all of that can be
> enumerated, allowing us to make a selective scrub that checks only a small
> part of the disk.  A linear read a 8TB disk takes 14 hours...
> 
> If we ever get auto-recovery, this is a fine candidate.
Indeed, and in fact I think that generational filtering may in fact be 
one of the easier performance improvements here too.
> 
>> Scrub will fix problem 1 because that's what it's designed to fix.  it will
>> also fix problem 3, since that behaves just like problem 1 from a
>> higher-level perspective.  It won't fix problem 2 though, as it doesn't look
>> at chunk types (only if the data in the chunk doesn't have the correct
>> number of valid copies).
> 
> Here not even tracking generations is required: a soft convert balance
> touches only bad chunks.  Again, would work well for auto-recovery, as it's
> a no-op if all is well.
However, it would require some minor differences from the current 
balance command, as newer kernels (are supposed to) generate half-raid1 
chunks instead of single chunks, though that can also be fixed by scrub.
> 
>> In contrast, the balance command you quoted won't fix issue 1 (because it
>> doesn't validate checksums or check that data has the right number of
>> copies), or issue 3 (because it's been told to only operate on non-raid1
>> chunks), but it will fix issue 2.
>>
>> In comparison to both of the above, a full balance without filters will fix
>> all three issues, although it will do so less efficiently (in terms of both
>> time and disk usage) than running a soft-conversion balance followed by a
>> scrub.
> 
> "less efficiently" is an understatement.  Scrub gets a good part of
> theoretical linear speed, while I just had a single metadata block take
> 14428 seconds to balance.
Yeah, the metadata especially can get pretty bad.
> 
>> In the case of normal usage, device disconnects are rare, so you should
>> generally be more worried about latent data corruption.
> 
> Yeah, but certain setups (like anything USB) gets disconnect quite often.
> It would be nice to get them right.  MD thanks to write-intent bitmap can
> recover almost instantly, btrfs could do it better -- the code to do so
> isn't written yet.
The write intent bitmap is also exponentially easier to implement than 
what's be needed for BTRFS.
> 
>> monitor the kernel log to watch for device disconnects, remount the
>> filesystem when the device reconnects, and then run the balance command
>> followed by a scrub.  With most hardware I've seen, USB disconnects tend to
>> be relatively frequent unless you're using very high quality cabling and
>> peripheral devices.  If, however, they happen less than once a day most of
>> the time, just set up the log monitor to remount, and set the balance and
>> scrub commands on the schedule I suggested above for normal usage.
> 
> A day-long recovery for an event that happens daily isn't a particularly
> enticing prospect.
I forget sometimes that people insist on storing large volumes of data 
on unreliable storage...